Configuration & Resources (A Dagster Deep Dive)
HTML-код
- Опубликовано: 26 июл 2024
- Are you done with Airflow? If you've ever had to install Kubernetes locally just to test a simple pipeline, or have resorted to the push-to-prod-and-pray method, it's time to take a look at how Dagster's configuration and resource systems allow you to develop locally and ship confidently.
Colton Padden leads us on a deep dive into a critical part of the Dagster framework.
Follow along using the demo repo: github.com/dagster-io/devrel-...
Start a free trial for 30 days to try this out: bit.ly/DagCloud
Bookmark/Star the Dagster repo: bit.ly/DagDemo
00:00 introduction
03:33 What we will build in this demo
10:25 Hands-on coding demo
22:22 Further resources
24:30 Q&A 
Наука
I really want to love Dagster but watching this video reminded me of why I stopped using Dagster for moving data from point A to point B. There are so, so many layers of configuration and plain infrastructure all over the place that kind of just needs to be there that the actual business logic (you know, the valuable part of the code that defines the data product) gets completely buried.
IMO, one of the most confusing concept and unnecessarily convoluted item in Dagster (which is otherwise amazing). Eg what’s with RunConfig that has references to `ops` but then things have to be keyed/named by asset name. You totally glossed over the global config item (eg s3 bucket that is common to everyone ) then you have to use an awkward resource that doesn’t really do anything other than holds some fields (ahem config). I really wish this would get cleaned up.
Hey @Amapramaadhy, what you’re expressing is totally valid. The concepts of Assets, Ops, and Jobs and how to compose them can be a bit convoluted - this has become more noticeable as our APIs evolve. We’re aware of this, and it’s on our roadmap to improve. Thanks for taking the time to respond and sharing your thoughts.
At around 8:20 you mention it's vulnerable to SQL injection - could I get more detail on that?
In 7:47 of the video you show using the Launchpad to configure assets... I can't figure out how to access this page?
Hi @user-hs9lo5gh3r, the most common way to bring up this menu is to select an asset from the global asset lineage, and then in the top right where it says "Materialize selected...", open the dropdown menu and select "Open launchpad". Hope this helps!
link repo please.
Sorry, one of our redirects got broken - here is the link: github.com/dagster-io/devrel-project-demos/tree/main
More specifically for this session: github.com/dagster-io/devrel-project-demos/tree/main/dagster-deep-dives/dagster_deep_dives/resources_and_configurations