Partitioned Data Pipelines in Data Engineering

Поделиться
HTML-код
  • Опубликовано: 2 ноя 2024

Комментарии • 9

  • @JohnoScott
    @JohnoScott Год назад +4

    Excellent. I just began learning partitions in Dagster yesterday, so the timing of this video is perfect.

  • @ledinhanhtan
    @ledinhanhtan 3 месяца назад

    I love using Dagster, especially when I can juggle the assets in my mind before writing any code. With Airflow, I have no mental picture of that.

  • @MrMal0w
    @MrMal0w Год назад +3

    Love this kind of video, very helpful, thank you ! :)

  • @JohnMcclaned
    @JohnMcclaned Год назад +5

    Using dagster is a literally a competitive advantage

  • @edancoben5784
    @edancoben5784 Год назад

    Great video! Is there a built in definition to partition assets by year? If not what's the best way to do that?

  • @dwieszopy
    @dwieszopy Год назад +1

    What’s that SensorResult class you’re using in the dynamic partitions part? I don’t see it in the latest release (1.2.3)

    • @s_ryz
      @s_ryz Год назад

      This was a tiny bit ahead of ourselves. It was added in a release soon after.

  • @jordanfox470
    @jordanfox470 Год назад

    Is there a partitioning strategy where I can partition a table by say, day, but anytime it refreshes, it refreshes the last 90 partitions?

  • @huyhoangnguyen7465
    @huyhoangnguyen7465 8 месяцев назад

    dagster._core.errors.DagsterInvariantViolationError: Cannot access partition_key for a non-partitioned run