Thinking in Partitions (A Dagster Deep Dive)

By embracing partitioning, you can speed up execution and greatly reduce the cost of running pipelines. Gain control over backfills and observe your data assets at a higher level of granularity.
You can catch all Dagster Deep Dives in the series here: dagster.io/events/dagster-dee...
You will find the Dagster repo for these sessions here: bit.ly/DagDemo
Be sure to bookmark (star) the Dagster repo for future reference: github.com/dagster-io/dagster
00:00 Intro
01:22 Why partition your pipeline?
05:06 An intro to Dagster's partitions API
06:37 Partitioning by category
11:24 Partition an asset by time
15:23 Recap of partitioning and partitioning patterns.
22:00 wrap up and other resources
22:40 Q&A

[@shaounakn]

I like what you folks have done with this product.

[@dagsterio]

Thanks - there is a lot more in store coming next month!

[@quinnherden]

+1 I am rooting for you guys. Thank you for all of your hard work

[@dagsterio]

We appreciate it - thanks @@quinnherden !

[@shaounakn]

@@dagsterio Sure, thanks for making these sessions, these are really helpful.

[@jordanfox470]

Is there native support for mapping time based partitions to static partitions defined like "today", "rest of month", "rest of year", "rest of history"? This is a common setup for power bi datasets, which can be represented as assets in dagster. Would be nice to take advantage of auto materialize policies.

[@dagsterio]

Dagster does not natively support mapping time-based partitions to static partitions like "today," "rest of month," "rest of year," and "rest of history" directly out of the box. However, you can achieve similar functionality by defining custom partitioning schemes and using the appropriate partition mappings. You can define custom partitions using StaticPartitionsDefinition for static and TimeWindowPartitionsDefinition for time-based partitions.

[@user-yh1hx9pe7u]

@dagsterio Do you have the source of the demo avaialble somewhere?

[@dagsterio]

All the code for the demos from the deep dives are in this repository ( github.com/dagster-io/devrel-project-demos )! This one in particular is in the partitions directory.

[@Jahaniam]

@@dagsterio unfortunately it is private/ link is broken.

[@dagsterio]

@@Jahaniam Sorry, the final parenthesis got included by RUclips in the URL - try this:
github.com/dagster-io/devrel-project-demos