Orchestrating Flexible Compute for ML with Dagster and Modal

Project Repository: github.com/dag...
In this Deep Dive, we showcase how Dagster and Modal combine to orchestrate scalable workloads with an intuitive developer experience to build scalable, robust pipelines.
We discuss how Dagster and Modal can be used together to accelerate machine learning workflows. Dagster, a data pipeline orchestration tool, manages pipeline state and leverages Modal's scalable infrastructure for GPU usage and compute scaling. Modal, an infrastructure platform, offers a self-provisioning runtime, provides pipeline components, and allows for autoscaling distributed apps without the need for Kubernetes YAML or GPU operators.
Both tools have a good developer experience, observability, and resiliency.
We share a personal experience of building a podcast summary application using Modal and Dagster, discussing the benefits of using Dagster to manage components and the ease of deploying on Modal. We also discuss using Cloudflare R2 buckets, transcribing podcasts using the Whisper model on Modal, and using Dagster cursors and sensors for managing RSS feeds.
Modal with Dagster can be used to access information from external processes and send events and logs back.

[@seanpool9125]

This is so cool, excited to tinker with this design pattern!

[@Mandalorx10]

This was a real treat to watch gang, you wouldn't happen to be able to post the repo in the description... and maybe next time have a higher contrast theme on the editor for us with poor eyesight watching on our phones :D

[@dagsterio]

Thanks, I'm glad you enjoyed this deep dive.
Good call on the contrast of the editor's theme, this is something we'll improve for next time.
If you are still looking for the link to the code repository, you can find it here:
github.com/dagster-io/dagster-modal-demo