Why I Switched From Pandas to Polars | TDE Workshop
HTML-код
- Опубликовано: 26 июн 2024
- In this workshop, Ben breaks down the 3 reasons he has permanently switched from Pandas to Polars and goes through a demo with code.
Led by: Ben Feifke
More Resources
TDS Article: / the-3-reasons-why-i-ha...
Example Code: github.com/bfeif/personal-web...
Udemy Course: www.udemy.com/course/data-ana...
Podcast Episode: www.inspiringcomputing.com/21...
Connect with Ben
LinkedIn: / benjamin-feifke
Medium: / benfeifke
Twitter/X: / benfeifke
Homepage: benfeifke.com/
Keep up with The Data Entrepreneurs!
🎥 RUclips: / @thedataentrepreneurs
👉 Discord: / discord
📰 Medium: / the-data
📅 Events: lu.ma/tde
🗞️ Newsletter: the-data-entrepreneurs.ck.pag...
Overview - 0:00
About Ben - 0:29
What's Polars - 3:19
Reason 1: .list namespace - 5:30
Reason 2: .scan_parquet() and .sink_parquet() - 11:48
Reason 3: Data-oriented Programming - 16:16
Demo - 20:58
Some Questions - 34:52
What’s Next? - 36:48
Q&A - 39:00
Great Job Ben! 👍
Thanks Ilia!
It is a shame that the lazy API is so entangled into the API. Might be nice to write generic code which then has the option to switch on the lazy API with one single change. I don't like the idea of having to rewrite the whole codebase to switch between lazy and eager. I question if that was a good design decision from polars. - Side note please always upload videos in 1080p
Here's my solution: cmd+f "scan_" replace with "read_" 😂
P.S. I'm on Mac
Although I see the benefits of Polars. I haven't had enough obstacle with Pandas for my workflows. I don't deal datasets that exceed memory and I think currently I can extend my memory limit using Dask ... but looking forward to the development of Polars and will likely adopt once it has more support!
Same situation here. I find Pandas and Dask to be sufficient tools for my workflows
Chatgpt knows pandas much better. For exploratory work probably not an issue. But if shopping something to prod and want to keep it very fast and minimise system resource then polars seems a better choice.
To me the appeal is the coherence of the API and the superior execution model. But the ecosystem disadvantages associated with using a much less popular library are substantial.
@@samuelswatson but polars has a to_pandas() method, so the disadvantages is easily overcome, so its more like if you your doing simple things, then its unneccesary to bring in polars. We replaced a lot advanced elt(not etl) with polars. 16 000 lines of sql code done with the main transfroms done in polars instead, for this task it's excellent and translated really well, and a lot of stuff is easier to to in polars than in sql for example. Doing the same in pandas is a nightmare. Translating advanced sql code to pandas is a hard job.
@@signoc1964 That seems to me to be the best use case for Polars (replacing complex SQL in transformation pipelines, especially because of its composability), so it's cool to hear another testimonial for its success in that context.
Can Polars replace pyspark Or hadoop?
Good question. Here’s a response from Ben.
“I’m not entirely sure tbh. i'm pretty sure pyspark is more scalable (e.g. > 1 TB data), but polars is better for data processing on your local machine (e.g. < 1 TB). i don't think Polars has so much stuff yet like pyspark does for distributed computing, whereas that is pretty much what pyspark was built for afaik.”
man, we can read.
why read everything on the screen?
Pandas is a massive mess. It's very easy to write very poor code in Pandas but to write it well is not that intuitive, Matt Harrison has written some great stuff, but it's not that easy to learn.
Polars is way better and improving all the time. It's much easier to write and way quicker. The lazy API is a thing of beauty.
I'm looking forward to using Polars more in my own workflow -Shaw
I agree, Pandas' learning curve was steeper for me than R. But I've kind of settled on a consistent workflow(.loc[:,['col']] instead of ['col'], and dotchaining) that minimizes the surprises.
One problem with polars though is that "pandas" developers then to write "polars" code like they write pandas code, and to some extent it is possible which gives people a bad example, since a couple of those. Polars becomes like pandas then executing in serial instead of parallell.