How to Automate Event-based End-to-End ETL Pipeline using AWS Glue & AWS Lambda | Data Engineering

Поделиться
HTML-код
  • Опубликовано: 2 окт 2024

Комментарии • 21

  • @keshavamugulursrinivasiyen5502
    @keshavamugulursrinivasiyen5502 3 месяца назад +1

    Impressive Demo

  • @data_bracket
    @data_bracket  7 месяцев назад +3

    Dont miss out on checking the Code snippets and interesting topics on my Newsletter.
    Substack➔ databracket.substack.com
    jayachandra27.github.io/databracket.ai/

  • @SAURABHKUMAR-uk5gg
    @SAURABHKUMAR-uk5gg 2 месяца назад +1

    Bro, use a good microphone. You'll have more attention to your channel

    • @data_bracket
      @data_bracket  2 месяца назад

      Thank you for the feedback :)
      I am working on improving the quality of videos.
      Future videos will be better for sure :)

  • @Cherupakstmt
    @Cherupakstmt 7 месяцев назад +3

    voice is too low. Can't hear you

    • @data_bracket
      @data_bracket  7 месяцев назад

      Thank you for the feedback.
      I'll try to produce better quality audio and visuals going forward.
      Really appreciate your inputs🙏❤️

  • @prabhashswain1878
    @prabhashswain1878 3 месяца назад +1

    Nice explanation❤

  • @shubhammahure3530
    @shubhammahure3530 5 месяцев назад +1

    Fix the issue of job failure when all the market zip files are placed in cft folder
    Some of the jobs will fail due to concurrency issue Exceeded maximum concurrent capacity for your account:500.
    How i add Queue for this jobs and delay so that it cannot exceed 500 DPUs.How check how many files are running.

    • @data_bracket
      @data_bracket  5 месяцев назад

      Have you thought of considering step functions?
      You can use initiate start job run sync functionality which will sit in a wait status until completed.
      If the glue trigger is happening through boto3, pull the job run ID from the response of the start job run.
      Create a while not true loop with sleep, then call get job run and check the JobRunState, if completed initiate the next job run.
      for parallel run, check this
      docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-map-state.html

  • @MrAnildas007
    @MrAnildas007 5 месяцев назад +1

    Good session. But voice is not audible

    • @data_bracket
      @data_bracket  5 месяцев назад

      Yes,
      My bad.
      I’ll improve the quality of my upcoming content.
      Thanks for the feedback 🙂

  • @shubhammahure3530
    @shubhammahure3530 5 месяцев назад +1

    you are teaching very well. Only thing you can increase the volume option.

    • @data_bracket
      @data_bracket  5 месяцев назад

      thank you for your feedback.
      I will work towards improving the audio quality.

  • @jiaweihe1279
    @jiaweihe1279 5 месяцев назад +1

    what a pity for such good tutorial without good voice

    • @data_bracket
      @data_bracket  5 месяцев назад

      Unfortunately yes😔
      I regret it
      I will try to generate better quality content going forward.
      Thanks for the feedback 🙂

  • @shubhammahure3530
    @shubhammahure3530 5 месяцев назад +1

    please tell me solution for this

  • @kreddy8621
    @kreddy8621 4 месяца назад +1

    Brilliant

    • @data_bracket
      @data_bracket  4 месяца назад

      Thank you for the comment.
      Hope this was useful!