TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Polars 2.0 makes its streaming engine the default for LazyFrame collection and enables initial spill-to-disk support, changing memory use and row-order behavior for some operations. The project also reports improved SQL benchmark results, though those comparisons are vendor-run and have stated limits.
Polars has released version 2.0, making its streaming engine the default when users collect a LazyFrame and enabling initial spill-to-disk support for operations that can exceed available memory. The release also expands SQL support and introduces a Map data type, but the new execution default can change observable row order for some operations unless users opt in to preserving it.
The project says that calling collect on a LazyFrame now uses the streaming engine by default, which processes data in batches rather than requiring the full working set in memory. Polars says this can reduce memory use and improve performance on many queries. Because streaming does not guarantee row order for operations including joins, group-bys and unpivots, users who depend on that order must set maintain_order=True where supported.
Version 2.0 also enables out-of-core execution by default. According to the release post, supported operations such as sorts, window functions and many expressions can spill data to disk once usage reaches about 80% of RAM. The default disk budget is 64 GB. The project says joins and group-bys are not yet supported for this feature, though it plans to add them.
Other changes include expanded first-class SQL support, engine and optimizer work, a Map dtype that represents Arrow MapType directly, and stricter handling of data types and explicitness. The release post points to join reordering, common-subplan elimination and dynamic predicates or bloom filters among the performance improvements. It does not provide a full API migration guide in the supplied material.
Memory and Ordering Change in 2.0
The default switch matters to teams running queries on datasets larger than their available memory. Disk spilling can let supported workloads complete instead of failing when memory pressure rises, while streaming can lower memory requirements. Those benefits are conditional: the operation must be supported, and the project’s current list does not include joins or group-bys for out-of-core execution.
There is also a compatibility consideration. A query that previously produced a particular row order may not preserve it under streaming for certain operations. Applications that rely on output order should test their results and set maintain_order=True where needed. SQL improvements may broaden Polars’ use in existing analytics workflows, but the benchmark results should be treated as the project’s own measurements rather than an independent comparison.
As an affiliate, we earn on qualifying purchases.
How Polars Benchmarked SQL
Polars says it tested SQL queries derived from TPC-H and TPC-DS against DuckDB 1.5.6, DuckDB 2.0 alpha and DataFusion 54.0.0. Tests ran on two AWS machine types: a 16-vCPU, 32-GB instance and a 192-vCPU, 384-GB instance. The benchmark used Parquet data stored on EBS, ran each query five times in a hot setting, and compared the best run by total time and geometric mean.
In its report, Polars said its default configuration was fastest in all but one of the benchmarks it presented. The project also described a scaling overhead on the 192-thread machine that hurt smaller queries, and said a 32-core Polars configuration was competitive or winning across the reported benchmarks. DataFusion timed out on TPC-DS query 72, once on query 67, and ran out of memory on TPC-H query 18 on the smaller machine; the affected queries were excluded for all engines. Polars published a repository so others can replicate its tests.
“Calling collect on a LazyFrame will now default to the streaming engine.”
— Polars, in its version 2.0 release post
As an affiliate, we earn on qualifying purchases.
Limits of Current Support
The release material does not specify a calendar release date, nor does the supplied source include the earlier post explaining why Polars chose a major version bump. It also does not set out a detailed compatibility guide for every behavior change. Users will need to check their workloads, particularly where output order is relied on or queries use operations not yet covered by disk spilling.
The benchmark findings have not been independently verified in the source material. Polars documented its hardware, data generation, query selection and run procedure, but noted that one scaling issue remains to be fixed in a future release. Results may differ with other data, storage, settings or workload mixes.
As an affiliate, we earn on qualifying purchases.
Upcoming Engine Work
Polars says it intends to extend out-of-core execution to joins and group-bys, which would increase the range of workloads able to spill to disk. It also says it has diagnosed the overhead seen when scaling to 192 threads and hopes to address it in the next release; no release date or fix commitment is given in the source.
For now, users can consult the project’s benchmark repository to reproduce the SQL comparisons and test their own workloads against the new defaults. Teams upgrading should verify row-order assumptions and confirm that their key operations are supported by the current spill-to-disk implementation.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main change in Polars 2.0?
LazyFrame collection now defaults to the streaming engine, and initial spill-to-disk support is enabled by default for selected operations.
Can Polars 2.0 change query row order?
Yes. The project says streaming does not guarantee observable row order for some operations, including joins, group-bys and unpivots. Set maintain_order=True where needed and supported.
Which operations can spill to disk?
The release post lists sorts, window functions and many expressions. It says joins and group-bys are not yet supported for out-of-core execution.
Did Polars independently prove it is faster than DuckDB and DataFusion?
No independent verification is included in the source. Polars reports that its default configuration led most of the benchmarks it ran, and has published a repository to help others reproduce the comparisons.
Source: hn
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
