Polars 2.0

It is optimized for analytic workloads (default in-mem columnar layout), whereas PG and sqlite are OLTP DBs. It's going to be insanely faster for those use cases.

Actual databases pay up to two orders of magnitude of speed for durability. If you can regenerate your dataset in case the DB drops it (or there is a power outage, or whatever), being able to complete a complex, write-heavy query in 10s instead of 15 minutes is actually very useful when doing analysis.

Because those are databases? Polars is a data processing engine, not a database. They have different uses.

If you want to use a database instead of Polars, for similar use cases, duckdb is a much better bet than postgressql or sqlite.

I've nearly entirely switched to DuckDB for anything more than like 500 or 1,000 rows or if there are a tonne of columns.

Polars is great, but I'm just too used to the Pandas API to use it as a replacement for the cases where DuckDB is overkill.

it’s on the correct path. i use rust for geo spatial and the gap with c, c++ closing rapidly or negligible in most cases

from https://github.com/pola-rs/geopolars/tree/main

Comparison with GeoPandas

Imitation is the sincerest form of flattery! GeoPandas — and its underlying libraries of shapely and GEOS — is an incredible production-ready tool.

GeoPolars is nowhere near the functionality or stability of GeoPandas, but competition is good and, due to its pure-Rust core, GeoPolars will be much easier to use in WebAssembly.

There is a geospatial package for duckdb though which is pretty slick. It is actually how I first learned of duckdb. We were dealing with nationwide parcel datasets and need to apply transformations nationwide and save out to more parquet files. It was easier and cheaper to replace all of the pandas workloads with duckdb.

> its not replacing the reason why pandas was popular ie data scientists

Why not? Our data science team uses it exclusively.

>Pandas wins if you want to use what the vast majority of folks are using

Vast majority of skilled developers are now using Polars, unless they are constrained by lack of Narwhals support in their third-party library of choice (e.g. Great Expectations, SHAP). That's the more important trend to follow.

It’s the API that gave me the push to leave Pandas. 10 or more years of occasional Pandas use and I still had to google for any non-trivial queries.

In that regard, I’m still waiting for a credible jq replacement…

Avoiding pandas developers is a great reason to use polars imo

It kind of reminds me of Dungeons and Dragons, in a sense. Pandas is so popular that people learn/use it because of its popularity, rather than because it's the best for any one use-case. Polars is cleaner, faster, and can easily be scaled up for production use-cases so your little POC script for local analysis can be productionized really easily, but there are holdouts still on Pandas because it was so complicated to learn with so many little extra rules to learn to avoid paper cuts that they feel like learning another data manipulation tool would be really hard.

DnD does the same thing: it's the most popular but its rules are this awkward hybrid of legacy cruft and some modern ideas, so learning it a huge effort, which means most people who play it aren't willing to try any other RPG systems even though most of them are dramatically easier to learn because they were built with a clean design from the ground-up.

It's the sunk-cost fallacy as applied to learning something complex, combined with something like the horn effect (inverse of the halo effect) making any competitors look equally complex even if they're not, causing long-time Pandas users/DnD players to strongly resist even looking at other options. Basically the frustration of learning these older systems seemingly traumatizes some people into never straying.

I'm not sure this is true e.g. you can specify schema `pl.Datetime("ns")`. That survives roundtrip to/from parquet in my experience. True though that the default is `us` whereas pandas defaults to `ns`.

https://docs.pola.rs/api/python/stable/reference/expressions...

Feels like a feature to me. You could just keep the original timestamp as a string.

I'm not sure what you tried. Is this not what you want?

    >>> df = pl.DataFrame({"x": ["a", "b", "c"], "y": ["d", "e", "f"]})
    >>> df.with_columns(new=pl.col.x + " text " + pl.col.y)
    shape: (3, 3)
    ┌─────┬─────┬──────────┐
    │ x   ┆ y   ┆ new      │
    │ --- ┆ --- ┆ ---      │
    │ str ┆ str ┆ str      │
    ╞═════╪═════╪══════════╡
    │ a   ┆ d   ┆ a text d │
    │ b   ┆ e   ┆ b text e │
    │ c   ┆ f   ┆ c text f │
    └─────┴─────┴──────────┘

Agreed. For its time, R was a lot of fun to play with data pipelines, visuals, Quarto, and frontier stats. It's a shame it's so hard to make it work for a large swath of production use cases.

Depends on need. We started using PyArrow on a reporting microservice when we realized we needed no additional functionality that pandas provided since it has better data type ergonomics. DuckDb is a great go-to for SQL based transformations when working with parquet files outside a managed system like Databricks. I want to actually test duckdb versus polars with a few lower level places like iceberg on S3.

I think the person you're replying to is more so saying that by choosing the right queries, machine, disk setup, caching, settings, thread count, RAM amount, etc, you can get quite different results. There is a reason everyone always wins their own benchmarks, and it doesn't even have to be dishonest - you optimize and iterate for your own benchmark whereas everyone else just gets one shot to do well out of the box.

I tried my best to be as transparent and fair as possible, running everyone with out-of-the-box settings on a third party's queries (DuckDB), a third party's data generator (tpcgen-rs, from the DataFusion guys), on a stock setup available to everyone (AWS machines).

The one exception is that we also ran Polars locked to 32 threads on the large machine (in addition to the out-of-the-box setup), which was to highlight we can do a lot better on small data. We still suffer from a relatively high constant overhead on very high CPU count machines if the data isn't large enough, but I'm working on fixing that. It's possible that DuckDB / DataFusion have similar scaling issues with high thread counts and would do better with 32 threads as well, I didn't test that.

It is absolutely without a doubt 100% not diverse enough.

I've spent years of my life running TPC benchmarks. They're useful guidelines, but they're simply not diverse enough to show the whole picture unless your picture is very simple. Certainly no single TPC benchmark in isolation. Maybe you can get a better idea by running all of them.

The largest gaps are around non-relational style data, JSON querying and the like. Nearly every organisation has something in that space now, and TPC-H/TPC-DS don't touch it at all.

Doing the benchmarks for 2.0 on the large AWS metal machines at small data sizes (SF=10) really opened my eyes that we have some low-hanging fruit in Polars when it comes to optimizing our constant overhead for smaller queries.

For example our join currently does a full partition into T partitions, for each of the T threads. Overall we create T^2 partitions, which on a 192-core machine is non-trivial. Great if you have a ton of data to feed that with, but if you 'only' have a few dozen million rows it becomes rather small. This is the primary reason we saw in the benchmarks that Polars pinned to 32 threads beats 192 thread Polars at SF=10.

I'll be working on improving that soon. I expect that to have a big impact on SF=10, and a decent impact on ClickBench, which sits between SF=10 and SF=100 in terms of rows.

I think it always has been? All of these data frame libraries are very much akin to olap databases.

Love competition in the local data SQL space, makes everyone better

> first class SQL support, which together with the performance improvements has Polars leading DataFusion and DuckDB in TPC-H and TPC-DS1 benchmarks,

Apparently about something ))

Yes, but why?

I mean, it's very clear that the projects have a lot in common, even if building one on top of another directly is not a good idea.

I use asof join frequently in polars! Happy to open the issue.

I think what you might consider the usual - Pandas, matplotlib etc in straight up Python. Some people still used R. But once folks saw the value, they all started making a painful switch over.

use a higher timeframe to understand the market phase, and a lower timeframe for entries.

when I started, I tried mimicking a human trader. if you want to expand into full quant, be aware of overengineering (past mistake of mine).

you can copy or mimic institutional desk strategies. most of the concepts are fine, but the devil is in the details.

sometimes you open too early or too late. finding that sweet spot, where you don’t want a lot of drawdown, requires a lot of fine-tuning.

I mainly trade xauusd and fx

more predictable