Polars 2.0 Just Made DuckDB and DataFusion Look Slow, Here’s the Architecture That Did It

Your Spark cluster is overkill. There, I said it.

While you’re provisioning nodes and debugging executor failures, the Rust-native DataFrame library Polars just shipped version 2.0 with a set of architectural changes that make it the default choice for high-performance data processing, and the benchmarks are frankly uncomfortable for the incumbents.

The release post, authored by Polars creator Ritchie Vink, dropped on October 6th, 2026, and it’s not messing around. Streaming engine as default. Out-of-core spill-to-disk enabled out of the box. SQL as a first-class citizen. A native Map dtype. Stricter type handling.

The headline numbers? Polars beat both DuckDB 1.5.6 and DataFusion 54.0.0 across nearly all TPC-H and TPC-DS derived benchmarks on both a 16-core and 192-core AWS instance. On the 16-core machine, Polars finished TPC-H SF10 in 3.22 seconds versus DuckDB’s 4.44 and DataFusion’s 7.41. At SF100 TPC-DS, Polars clocked 85.83 seconds while DataFusion limped in at 198.48.

Let’s dig into what actually changed and why it matters.

The Streaming Engine Is No Longer Optional

The most consequential change in 2.0 is that calling collect() on a LazyFrame now defaults to the streaming engine. Previously, you had to opt in with .collect(streaming=True). Now it’s the default path, and that’s a major version bump precisely because it changes observable behavior.

Here’s the catch: the streaming engine doesn’t guarantee row order for certain operations, specifically join, group_by, and unpivot. If your downstream code depends on the order rows appear after those operations, you need to explicitly set maintain_order=True.

This is the kind of breaking change that will surprise teams upgrading without reading the migration guide. The trade-off is real: streaming execution dramatically reduces memory footprint and improves performance on most queries. But if you’ve been relying on implicit ordering, your pipeline might produce subtly different results after upgrade.

The community reception on Hacker News has been largely positive, with the 416-point discussion focusing on the benchmark methodology and what it means for the broader data engineering ecosystem. One recurring theme: this is getting harder to dismiss as a niche tool.

For teams still comparing Polars with Pandas for high-performance data processing, the streaming default closes the memory gap that was Pandas’ last advantage.

What’s Driving the Performance: Architecture Under the Hood

The GitDiagram analysis of the pola-rs/polars repository breaks down the architecture into clean layers:

• User APIs: Python DataFrame API, LazyFrame API, Expressions, Rust API, SQL context
• Query Planning: Subquery rewrites, query optimizer with projection pushdown
• Execution Runtime: Engine selection, execution engines, streaming execution, compute kernels
• Data and I/O: DataFrame core, Arrow arrays, Data I/O, Parquet support
• Interoperability: Python bindings, Arrow FFI, remote engine

The key architectural decisions:

1. Arrow as the native memory format, Polars doesn’t convert data structures, it operates directly on Arrow arrays. This eliminates serialization overhead and enables zero-copy sharing across the Rust/Python boundary via Arrow FFI.
2. Lazy evaluation by default, The LazyFrame API builds a query plan that gets optimized before execution. Projection pushdown, predicate pushdown, and common-subplan elimination happen before any data is touched.
3. Streaming execution with morsels, The streaming engine processes data in small chunks (morsels) that can be distributed across threads without loading everything into memory.
4. Join reordering and dynamic predicates/bloom filters, The optimizer improvements in 2.0 make complex joins significantly faster, which is where the TPC-DS gains come from.
5. Common-subplan elimination, Repeated computations in a query plan get computed once and reused instead of recomputed per occurrence.

These aren’t incremental tweaks. This is a fundamentally different design philosophy from Pandas, which materializes everything eagerly. And it’s different from Spark, which adds network serialization overhead even for single-node workloads.

For teams contrasting Polars and Spark in the context of lightweight, high-performance ETL, the streaming improvements in 2.0 make the case stronger: why spin up a cluster when a single machine with Polars handles your workload faster?

df = pl.DataFrame(
    {
        "user": ["alice", "bob", "carol"],
        "scores": pl.Series(
            [{"math": 90, "art": 75}, {"math": 60}, {}],
            dtype=pl.Map(pl.String, pl.Int64),
        ),
        "subject": ["art", "art", "math"],
    }
)

df.select(
    "user",
    pl.col("scores").map.get("math").alias("math"),
    pl.col("scores").map.get(pl.col("subject")).alias("by_subject"),
    pl.col("scores").map.contains_key("art").alias("has_art"),
    pl.col("scores").map.len().alias("n"),
    pl.col("scores").map.keys().alias("keys"),
    pl.col("scores").map.values().alias("values"),
)

Stricter Polars: Fail Fast, Fail Early

The push toward stricter type handling is the least glamorous but arguably most important change, especially in the age of AI-driven development.

Polars aims to raise errors up front rather than 20 minutes into a pipeline. Implicit behavior on data mismatches is now opt-in, not default. This is a deliberate design choice that prioritizes correctness feedback over convenience.

The collect_schema() method lets you validate a query’s structure without materializing any data. For AI agents iterating on queries, this means schema errors get caught instantly instead of after a full execution cycle. The release notes explicitly frame this as a feature for “faster AI iteration.”

This is the kind of thing that doesn’t show up in benchmarks but matters enormously in practice. Evaluating over-engineered enterprise platforms versus lightweight, efficient tools like Polars often comes down to iteration speed, and stricter validation is a meaningful contributor.

The Verdict: What Should You Do?

Polars 2.0 is a significant step forward, but the upgrade isn’t frictionless.

Do this first:

1. Read the migration guide
2. Audit your pipelines for row-order dependencies, especially after joins and group-bys
3. Test with maintain_order=True if you find ordering issues
4. Check memory thresholds, the 80% spill trigger may need tuning for your workloads

Watch for:

• Out-of-core joins and group-bys (roadmap, not shipped)
• The 192-thread overhead fix in the next release
• GeoPolars, announced as in progress

The benchmark numbers deserve healthy skepticism until independently replicated. Vendor-run benchmarks have a way of favoring the vendor. But the architecture is sound, the methodology was transparent, and the code is public. That’s more than most projects offer.

The data engineering landscape is shifting. Spark isn’t dead, Pandas isn’t irrelevant, and DuckDB won’t disappear. But the rise of DuckDB and Polars represents a fundamental realignment: for a growing class of workloads, the best tool isn’t a distributed cluster, it’s a well-architected single-node engine that uses your hardware efficiently.

Polars 2.0 makes that case harder to ignore.

Benchmark data and methodology details are derived from the official release post. The benchmarks are derived from TPC-H and TPC-DS and are not comparable to published TPC results.