It does projection pushdown via the columns parameter (pandas.pydata.org/docs/reference/api/pandas.read_parquet.html) And I think if you'd use the pyarrow engine you would get predicate pushdown too. Pandas delegates the Parquet reading to fastparquet or pyarrow under the covers, it doesn't have a reader itself as far as I'm aware.
Enjoying videos Mark, keep going. One suggestion - slow down a bit and highlight the code/ explain the code a bit (not in detail) so we get context.
Thanks and I'll try! Although I have been told to slow down for many years and I'm clearly not great at doing that!
can you provide examples where predicate pushdown is not possible with parquet?
Can you do a tutorial of Duckbd + ibis, I am totally new to databases, I am actually familiar with excel, pandas and Polars, just starting
Hi! Thanks for your comment. I haven't used Ibis before, but I'm gonna take a look and will try to make an intro tutorial 🙂
does the pandas API also support predicate pushdown?
It does projection pushdown via the columns parameter (pandas.pydata.org/docs/reference/api/pandas.read_parquet.html)
And I think if you'd use the pyarrow engine you would get predicate pushdown too.
Pandas delegates the Parquet reading to fastparquet or pyarrow under the covers, it doesn't have a reader itself as far as I'm aware.
@@learndatawithmark great insights Mark, just slowdown the explanation in the main part of the code.
Where may I find pqrs?
Sorry forgot to put the link. Here we go: github.com/manojkarthick/pqrs