Видео

please tell us how to add dbt to existing pipeline.

Please tell us how to insert with an existing pipeline.. DBT for me is a second step , first step being ingestion.

This is so cool, excited to tinker with this design pattern!

This was a real treat to watch gang, you wouldn't happen to be able to post the repo in the description... and maybe next time have a higher contrast theme on the editor for us with poor eyesight watching on our phones :D

but sir i have a dagster-dbt project and when i test my dbt runs it's working perfectly, but the integration is where i have a problem, could you help me?

"Obviously we have to have code. It's 2024, we're doing code. It's over, no more no code, no low code. It's just not going to work." 15:55

When you write ACCESS_TOKEN, what does that stand for? Is it your password? Is it the end of a link, why is it so bad to put it in your source code?-

If I understand this correctly this is a very impressive framework. I'll try it out with our dbt project since it seems to integrate with dbt as well and consider it for dagster at a later point. Is there a Slack/discord community as well? If you include a link to your website in the description it saves us the detour via google :D

Hello Ampcus Inc. This is IBU agency a managing and consulting agency that help businesses manage all their social media handles. So we simply post content, handle clients interactions across all platforms and a bonus package that we will give more details about when we schedule a meeting. Please let us know if you have any questions or concerns.

I'm always looking forward to these Dagster videos. The fastest way to get the new things that they did. I was also a lot curious about this SDF

I have to ask, how do you get in the Terminal the full path by just typing the beginning of each folder name?

Is dlt embedded in Dagster now?

Integrate or replace slurm?

badly explained

instead of dagit install dagster-webserver from now on

I love using Dagster, especially when I can juggle the assets in my mind before writing any code. With Airflow, I have no mental picture of that.

What is the difference between dagit and dagster-webserver?

Is FreshnessPolicy being deprecated?

Came here excited to learn about new features in the latest Dagster version. But it looks like you've decided to widen the feature-gap between the open source offering and the enterprise offering... even though this will be a maintenance burden on your team... causing delays in "backporting" features and bugfixes to the open-source version going forward. Kinda disappointed...

Is there native support for mapping time based partitions to static partitions defined like "today", "rest of month", "rest of year", "rest of history"? This is a common setup for power bi datasets, which can be represented as assets in dagster. Would be nice to take advantage of auto materialize policies.

Awesome!!! Please more!

I'm fairly sure this sales guy never used Airflow

If the requirement is to get the data from S3 files into a BQ table but perform some validations on those files before inserting into the table, how would we do it with Embedded ELT? We are using Dagster OSS heavily and looking to use embedded-elt for getting data from files, tables and APIs..

As a person with just 2 years of experience my mind was blown watching this. I am a single person writing code in my department so I don't have any seniors to learn from but I'm leading a data engineering project that deals with terabytes of data and each request is multiple times larger than the server's RAM and multiple such requests need to be processed in parallel to complete stuff in time. Also, we have the tiniest possible budget to aggregate 25 to 30 columns and billions of rows every day. Also, we need to cut down on costs. This was super helpful.

anyone notice silicon valley reference in screenshots

Seems to be an alternative to dbt docs and dbt Cloud Explorer?

You lost me at "cloud".

Where is Nick Shrock ?

What does Dagster+ mean for the open source version?

Exciting

Great presentation.

Please update the gh repo when possible with the data mesh example. Multiple code locations seem super useful. Thanks

Hi Dagster team, great stuff here! I really enjoyed watching this!! is the demo code available in github?

nice talk but the slides are hard to follow on here. Would be better if recorded with autofocus off, and white balanced to the projector screen.

100%

This is the coolest tech demo I've ever seen. I have wanted for so long to see an end-to-end analytics stack demo, or tutorial, and never found it. You just did it in 15 minutes, using free, open source tools I can run locally on my laptop. Absolutely incredible!

At around 8:20 you mention it's vulnerable to SQL injection - could I get more detail on that?

fk the learning curve on this shit.

That's interesting. Do you expand on this somewhere?

I work in a financial institution and there is definitely a need for a reliable and resilient data process. Look forward to finding out more about Dagster. I also agree, no point building something flaky and have it barf 🤢

Yes I’m excited. Thanks

link repo please.

I don’t know… this video is one year old, but still uses the legacy DAG syntax from Airflow 1, rather than the TaskFlow API from Airflow 2. So the syntax doesn’t make a difference anymore. Regarding the coupling to environment: Airflow has different executors. The KubernetesPodOperator is not the only way to run on a Kubernetes environment. The rest may or may not be true. Probably there are many things that Dagster does better than Airflow. But I’m disappointed that you would publish such a biased comparison.

@dagsterio Do you have the source of the demo avaialble somewhere?

I like what you folks have done with this product.

Please send me link Git repo on video

Joining other comments, I'd love to see more step-by-step tutorials and use cases. It took a few videos to grasp the concepts, and this one is a good one to start with. Docs are good, but videos are even better. I would love to see more of duckdb / dagster and ingestion cases.

In 7:47 of the video you show using the Launchpad to configure assets... I can't figure out how to access this page?

What’s with these shorts? Feels like a kid got hold of your social account! Stick to real content