Querying Parquet files on S3 with DuckDB

In this video, we'll learn how to query Apache Parquet files on Amazon S3, using DuckDB.
#duckdb #s3 #apacheparquet
Resources
► DuckDB - duckdb.org/
► Apache Parquet - parquet.apache...
► Lightweight Compression in DuckDB - duckdb.org/202...
► Build a poor man’s data lake from scratch with DuckDB - dagster.io/blo...

[@tributarydata]

This is the most concise and comprehensive DuckDB video I've seen so far. Great job, Mark!

[@michaelsimons2560]

I love your information density.
The 4secs for getting the first 10 results over the wire is ofc because you selected a bunch of different columns that are spread across the Parquet. With remove CSV it would have been faster, I guess.

[@FranciscoRuizA]

This is great, thanks. Is ADLS in Azure supported?

[@learndatawithmark]

I don't think it does out of the box, but I just learnt about support for fsspec, which does support ADLS. So I need to give that a try! duckdb.org/docs/guides/python/filesystems

[@benjaminwootton]

What's happening with the data transfer when we query a view? Do only the relevant columns get pulled back to the local DuckDb process? Im surprised performance is as good as it is here when we need to do the full table scan.

[@learndatawithmark]

On the full table scan we have a LIMIT on the query though, so my understanding is it'd only be reading the first row group that contains that data. I think you can read parts of files on S3 without needing to download the whole file, but I'm not an expert in that!

[@lizardogre]

Were the queries running on a single parquet file, or across all the parquet files in the bucket?

[@learndatawithmark]

They were running across all the files in the bucket.

[@CaribouDataScience]

It's not butter it's Parquet!

[@Phoenixspin]

I see what you did there.