Build a poor man’s data lake from scratch with DuckDB

DuckDB is so hot right now. Could it replace our cloud data warehouses or data lakes?
Pete Hunt builds a data lake from scratch with DuckDB and Dagster. Follow the tutorial here: dagster.io/blog/duckdb-data-lake
Combined with Dagster, S3, and Apache Parquet, DuckDB can become a powerful, multiplayer data lake that can serve the needs of many organizations with very little effort. Think of it as a free, simple SQLite.
We can see the rise of DuckDB for subsets of workloads that don’t have massive scale and can take advantage of its simplicity and fast performance.
Give it a go!
Try Dagster for free for 30 days: dagster.io/lp/dagster-cloud-t...

[@vikramtatke5930]

As a person with just 2 years of experience my mind was blown watching this.
I am a single person writing code in my department so I don't have any seniors to learn from but I'm leading a data engineering project that deals with terabytes of data and each request is multiple times larger than the server's RAM and multiple such requests need to be processed in parallel to complete stuff in time.
Also, we have the tiniest possible budget to aggregate 25 to 30 columns and billions of rows every day. Also, we need to cut down on costs.
This was super helpful.

[@tobiaspucher9597]

Awesome!!! Please more!

[@michaelayoub2211]

Great video, thanks!

[@marcosoliveira8731]

Really good stuff! A lot of good ideas.

[@ImperialTerrain]

thank you pete

[@gauravlotekar660]

aawwwseome.

[@kalidsherefuddin]

Thanks for

[@gw1284]

Thanks for this demo. Can you comment on what role polars may play in this?

[@hwy9nightkid]

polars is akin to pandas or spark dataframes.. a way to organize your tables of data , if im not mistaken

[@marcosoliveira8731]

As pandas alternative.