RustyCSV

Ultra-fast CSV parsing and encoding for Elixir. A purpose-built Rust NIF with SIMD acceleration, parallel parsing, and bounded-memory streaming. Drop-in replacement for NimbleCSV.

Why RustyCSV?

The Problem: CSV parsing in Elixir can be optimized further:

1. Speed: Pure Elixir parsing, while well-optimized, can't match native code with SIMD acceleration for large files.

2. Flexibility: Different workloads benefit from different strategies—parallel processing for huge files, streaming for unbounded data.

3. Direct binary chunk streaming: RustyCSV's parse_stream/2 processes arbitrary binary chunks without first converting them to a line-oriented stream.

Why not wrap an existing Rust CSV library? The excellent csv crate is designed for Rust workflows, not BEAM integration. Wrapping it would require serializing data between Rust and Erlang formats—adding overhead and losing the benefits of direct term construction.

RustyCSV's approach: The Rust NIF is built specifically for BEAM integration. It parses directly into BEAM terms and keeps optional features modular:

1. Bounded-memory streaming - Process multi-GB files without retaining the full input; memory is bounded by the configured buffer and longest row
2. Sub-binary field references - Near-zero BEAM allocation; fields reference the input binary directly
3. Multiple strategies - Choose SIMD, parallel, or streaming based on your workload
4. Reduced scheduler load - Parsing NIFs run on dirty CPU schedulers
5. NimbleCSV-compatible API - Supported public functions are covered by the upstream semantic test suite

Feature Comparison

Purpose-Built for Elixir

RustyCSV isn't a wrapper around an existing Rust CSV library. It is built specifically for Elixir/BEAM integration:

• Boundary-based sub-binary fields - SIMD scanner finds field boundaries, then creates BEAM sub-binary references directly (zero-copy for clean fields, copy only when unescaping "" → ")
• Dirty scheduler aware - parsing NIFs run on dirty CPU schedulers rather than normal BEAM schedulers
• ResourceArc-based streaming - stateful parser properly integrated with BEAM's garbage collector
• Direct term building - parsing results go straight to BEAM terms; encoding builds one binary per row without per-field BEAM terms

Parsing Strategies

Choose the right tool for the job:

Parsing is strict by default, matching NimbleCSV's malformed-quote behavior. Existing RustyCSV applications that intentionally accept dirty exports can opt back into the pre-0.4 behavior:

CSV.parse_string(csv, strict: false)
CSV.parse_stream(chunks, strict: false)

Memory Model:

All batch strategies use boundary-based sub-binaries — the SIMD scanner finds field boundaries, then creates BEAM sub-binary references that point into the original input binary. Only fields requiring quote unescaping ("" → ") are copied.

# Strategy selection
CSV.parse_string(data)                           # Uses :simd (default)
CSV.parse_string(huge_data, strategy: :parallel) # Parallel field extraction via rayon
File.stream!("huge.csv") |> CSV.parse_stream()   # Bounded memory

Installation

def deps do
  [{:rusty_csv, "~> 0.4.2"}]
end

Precompiled NIFs are provided for supported targets through RustlerPrecompiled, so normal installation does not require Rust. Building locally or targeting an unsupported platform requires Rust nightly for std::simd; see SIMD and Rust Nightly.

Quick Start

alias RustyCSV.RFC4180, as: CSV

# Parse CSV (skips headers by default, like NimbleCSV)
CSV.parse_string("name,age\njohn,27\njane,30\n")
#=> [["john", "27"], ["jane", "30"]]

# Include headers
CSV.parse_string(csv, skip_headers: false)
#=> [["name", "age"], ["john", "27"], ["jane", "30"]]

# Stream large files with bounded memory
"huge.csv"
|> File.stream!()
|> CSV.parse_stream()
|> Stream.each(&process_row/1)
|> Stream.run()

# Parse to maps with headers
CSV.parse_string("name,age\njohn,27\njane,30\n", headers: true)
#=> [%{"name" => "john", "age" => "27"}, %{"name" => "jane", "age" => "30"}]

# With atom keys
CSV.parse_string("name,age\njohn,27\n", headers: [:name, :age])
#=> [%{name: "john", age: "27"}]

# Dump back to CSV
CSV.dump_to_iodata([["name", "age"], ["john", "27"]])
#=> [["name,age\r\n"], ["john,27\r\n"]]

Drop-in NimbleCSV Replacement

# Before
alias NimbleCSV.RFC4180, as: CSV

# After
alias RustyCSV.RFC4180, as: CSV

# That's it. Same API, 3-9x faster on typical workloads.

All NimbleCSV functions are supported:

Benchmarks

The most recent published benchmark run used RustyCSV 0.3.6, Elixir 1.19, OTP 28, and an Apple M1 Pro. It measured 3.5x faster parsing for simple synthetic CSV and 18.6x faster for quote-heavy CSV.

The same run measured 13-28% faster parsing on Amazon settlement-report TSV files with roughly 10,000 or more rows. Results vary with data shape and hardware.

mix run bench/decode_bench.exs

See the benchmark methodology and results.

When to Use RustyCSV

Custom Parsers

Define parsers with custom separators and options:

# TSV parser
RustyCSV.define(MyApp.TSV,
  separator: "\t",
  escape: "\"",
  line_separator: "\n"
)

# Pipe-separated
RustyCSV.define(MyApp.PSV,
  separator: "|",
  escape: "\"",
  line_separator: "\n"
)

MyApp.TSV.parse_string("a\tb\tc\n1\t2\t3\n")
#=> [["1", "2", "3"]]

Define Options

Multi-Separator Support

For files with inconsistent delimiters (common in European locales), specify multiple separators:

# Accept both comma and semicolon as delimiters
RustyCSV.define(MyApp.FlexibleCSV,
  separator: [",", ";"],
  escape: "\""
)

# Parse files with mixed separators
MyApp.FlexibleCSV.parse_string("a,b;c\n1;2,3\n", skip_headers: false)
#=> [["a", "b", "c"], ["1", "2", "3"]]

# Dumping uses only the FIRST separator
MyApp.FlexibleCSV.dump_to_iodata([["x", "y", "z"]]) |> IO.iodata_to_binary()
#=> "x,y,z\n"

Separators and escape sequences can be multi-byte:

# Double-colon separator
RustyCSV.define(MyApp.DoubleColon,
  separator: "::",
  escape: "\""
)

# Multi-byte escape
RustyCSV.define(MyApp.DollarEscape,
  separator: ",",
  escape: "$$"
)

# Mix single-byte and multi-byte separators
RustyCSV.define(MyApp.Mixed,
  separator: [",", "::"],
  escape: "\""
)

Headers-to-Maps

Return rows as maps instead of lists using the :headers option:

# First row becomes string keys
CSV.parse_string("name,age\njohn,27\njane,30\n", headers: true)
#=> [%{"name" => "john", "age" => "27"}, %{"name" => "jane", "age" => "30"}]

# Explicit atom keys (first row skipped by default)
CSV.parse_string("name,age\njohn,27\n", headers: [:name, :age])
#=> [%{name: "john", age: "27"}]

# Explicit string keys
CSV.parse_string("name,age\njohn,27\n", headers: ["n", "a"])
#=> [%{"n" => "john", "a" => "27"}]

# Works with streaming too
"huge.csv"
|> File.stream!()
|> CSV.parse_stream(headers: true)
|> Stream.each(&process_map/1)
|> Stream.run()

Edge cases: fewer columns than headers fills with nil, extra columns are ignored, duplicate headers use last value, empty headers become "".

Key interning is done Rust-side for parse_string — header terms are allocated once and reused across all rows. Streaming uses Elixir-side Stream.transform for map conversion.

Encoding Support

RustyCSV supports character encoding conversion:

# UTF-16 Little Endian (Excel/Windows exports)
RustyCSV.define(MyApp.Spreadsheet,
  separator: "\t",
  encoding: {:utf16, :little},
  trim_bom: true,
  dump_bom: true
)

# Or use the pre-defined spreadsheet parser
alias RustyCSV.Spreadsheet
Spreadsheet.parse_string(utf16_data)

RFC 4180 Compliance

RustyCSV is fully RFC 4180 compliant and validated against industry-standard test suites:

See Compliance and Validation for full details.

How It Works

Why Not Wrap the Rust csv Crate?

The Rust ecosystem has excellent CSV libraries like csv and polars. But wrapping them for Elixir has overhead:

1. Parse CSV → Rust data structures (allocation)
2. Convert Rust structs → Erlang terms (allocation + serialization)
3. Return to BEAM

RustyCSV eliminates the middle step by parsing directly into BEAM terms:

1. Parse CSV → Erlang terms directly (single pass)
2. Return to BEAM

Strategy Implementations

All batch strategies share a single-pass SIMD structural scanner that finds field boundaries, then create BEAM sub-binary references directly.

Shared across batch strategies (:simd, :parallel):

• Single-pass SIMD structural scanner (finds all unquoted separators and row endings in one sweep)
• Boundary-based sub-binary field references (near-zero BEAM allocation)
• Hybrid unescaping: sub-binaries for clean fields, copy only when "" → " unescaping needed
• Direct Erlang term construction via Rustler (no serde)
• mimalloc high-performance allocator

:parallel specifics:

• Runs on dirty CPU schedulers to avoid blocking BEAM
• Rayon workers compute boundary pairs (pure index arithmetic on the shared structural index) — no data copying
• Main thread builds BEAM sub-binary terms from boundaries (Env is not thread-safe, so term construction is serial)

:streaming specifics:

• ResourceArc integrates parser state with BEAM GC
• Tracks quote state across chunk boundaries
• Copies field data (since input chunks are temporary)

NIF-Accelerated Encoding

RustyCSV returns one top-level iodata list per row from dump_to_iodata/2 and list-shaped row iodata from dump_to_stream/1, matching NimbleCSV's public shape. Each row is one contiguous binary, avoiding one BEAM term per field. Use IO.iodata_to_binary/1 when a flat binary is required.

See the encoding benchmark results for throughput and memory numbers.

Encoding Strategies

dump_to_iodata/2 accepts a :strategy option:

# Default: single-threaded encoder returning list-shaped iodata.
# SIMD scan for quoting, writes directly to a reusable row buffer.
# Best for most workloads.
CSV.dump_to_iodata(rows)

# Parallel: multi-threaded encoding via rayon.
# Copies field data into Rust-owned memory, encodes rows across worker threads.
# Faster when fields frequently need quoting (commas, quotes, newlines in values).
CSV.dump_to_iodata(rows, strategy: :parallel)

High-Throughput Concurrent Exports

RustyCSV's encoding NIF runs on BEAM dirty CPU schedulers with per-thread mimalloc arenas, making it well-suited for concurrent export workloads (e.g., thousands of users downloading CSV reports simultaneously in a Phoenix application):

# Phoenix controller — concurrent CSV download
def export(conn, %{"id" => id}) do
  rows = MyApp.Reports.fetch_rows(id)
  csv = MyCSV.dump_to_iodata(rows)

  conn
  |> put_resp_content_type("text/csv")
  |> put_resp_header("content-disposition", ~s(attachment; filename="report.csv"))
  |> send_resp(200, csv)
end

For very large exports where you want bounded memory, use chunked NIF encoding:

# Chunked encoding — bounded memory with NIF speed
def stream_export(conn, %{"id" => id}) do
  conn = conn
  |> put_resp_content_type("text/csv")
  |> put_resp_header("content-disposition", ~s(attachment; filename="report.csv"))
  |> send_chunked(200)

  MyApp.Reports.stream_rows(id)
  |> Stream.chunk_every(5_000)
  |> Stream.each(fn chunk ->
    csv = MyCSV.dump_to_iodata(chunk)
    {:ok, _conn} = Plug.Conn.chunk(conn, csv)
  end)
  |> Stream.run()

  conn
end

Key characteristics for concurrent workloads:

• Each NIF call is independent — no shared mutable state between requests
• Dirty CPU schedulers prevent encoding from blocking normal BEAM schedulers
• mimalloc's per-thread arenas avoid allocator contention under concurrency
• The real bottleneck is typically DB queries and connection pool sizing, not CSV encoding

Architecture

RustyCSV is built with a modular Rust architecture:

native/rustycsv/src/
├── lib.rs                 # NIF entry points, separator/escape decoding, dispatch
├── core/
│   ├── simd_scanner.rs    # Single-pass SIMD structural scanner (prefix-XOR quote detection)
│   ├── simd_index.rs      # StructuralIndex, RowIter, RowFieldIter, FieldIter
│   ├── scanner.rs         # Byte-level helpers (separator matching)
│   ├── field.rs           # Field extraction, quote handling
│   └── newlines.rs        # Custom newline support
├── strategy/
│   ├── direct.rs          # Basic + SIMD strategies (single-byte)
│   ├── two_phase.rs       # Indexed strategy (single-byte)
│   ├── streaming.rs       # Stateful streaming parser (single-byte)
│   ├── parallel.rs        # Rayon-based parallel parsing (single-byte)
│   ├── zero_copy.rs       # Sub-binary reference parsing (single-byte)
│   ├── general.rs         # Multi-byte separator/escape (all strategies)
│   ├── encode.rs          # SIMD field scanning, quoting helpers
│   └── encoding.rs        # UTF-8 → target encoding converters (UTF-16, Latin-1, etc.)
├── term.rs                # BEAM term building (sub-binary + copy fallback)
└── resource.rs            # ResourceArc for streaming state

See the architecture documentation for detailed implementation notes.

Memory Efficiency

The streaming parser does not retain the full input. Its memory use is bounded by the configured buffer limit, the longest buffered row, and rows retained by the caller:

# Process a 10GB file without loading it all into memory
File.stream!("huge.csv", [], 65_536)
|> CSV.parse_stream()
|> Stream.each(&process/1)
|> Stream.run()

Streaming Buffer Limit

The streaming parser enforces a maximum internal buffer size (default 256 MB) to prevent unbounded memory growth when parsing data without newlines or with very long rows. If a feed exceeds this limit, a :buffer_overflow exception is raised.

To adjust the limit, pass :max_buffer_size (in bytes):

# Increase for files with very long rows
CSV.parse_stream(stream, max_buffer_size: 512 * 1024 * 1024)

# Decrease to fail fast on malformed input
CSV.parse_stream(stream, max_buffer_size: 10 * 1024 * 1024)

# Also works on direct streaming APIs
RustyCSV.Streaming.stream_file("data.csv", max_buffer_size: 1_073_741_824)

High-Performance Allocator

RustyCSV uses mimalloc as the default allocator, providing:

• 10-20% faster allocation for many small objects
• Reduced memory fragmentation
• Zero tracking overhead in default configuration

The distributed NIFs include mimalloc. To test a local source build without it, temporarily set default = [] in native/rustycsv/Cargo.toml, then rebuild:

FORCE_RUSTYCSV_BUILD=true mix compile --force

Optional Memory Tracking (Benchmarking Only)

For profiling Rust-side memory during development and benchmarking. Not intended for production — it wraps every allocation with atomic counter updates, adding overhead. This is also the only source of unsafe in the codebase (required by the GlobalAlloc trait). Temporarily add memory_tracking to the default features in native/rustycsv/Cargo.toml, then force a local build:

# In native/rustycsv/Cargo.toml
[features]
default = ["mimalloc", "memory_tracking"]

FORCE_RUSTYCSV_BUILD=true mix compile --force

Then use:

RustyCSV.Native.reset_rust_memory_stats()
result = CSV.parse_string(large_csv)
peak = RustyCSV.Native.get_rust_memory_peak()
IO.puts("Peak Rust memory: #{peak / 1_000_000} MB")

When disabled (default), these functions return 0 with zero overhead.

SIMD and Rust Nightly

RustyCSV uses Rust's std::simd portable SIMD, which currently requires nightly via #![feature(portable_simd)]. However, RustyCSV only uses the stabilization-safe subset of the API:

• Simd::from_slice, splat, simd_eq, bitwise ops (&, |, !)
• Mask::to_bitmask() for extracting bit positions

We deliberately avoid the APIs that are blocking stabilization: swizzle, scatter/gather, and lane-count generics (LaneCount<N>: SupportedLaneCount). The items blocking the portable_simdtracking issue — mask semantics, supported vector size limits, and swizzle design — are unrelated to the operations we use. When std::simd stabilizes, RustyCSV will work on stable Rust with no code changes.

The prefix-XOR quote detection uses a portable shift-and-xor cascade rather than architecture-specific intrinsics, keeping the entire scanner free of unsafe code. Benchmarks show no measurable difference for the 16/32-bit masks used in CSV scanning.

Development

# Install dependencies
mix deps.get

# Compile (includes Rust NIF)
mix compile

# Run tests (494 ExUnit tests)
mix test

# Run benchmarks
mix run bench/decode_bench.exs
mix run bench/encode_bench.exs

# Code quality
mix credo --strict
mix dialyzer

License

MIT License - see LICENSE file for details.

RustyCSV - Purpose-built Rust NIF for ultra-fast CSV parsing in Elixir.