Kafka Connect in Action : S3 Sink

Поделиться
HTML-код
  • Опубликовано: 24 ноя 2024

Комментарии • 37

  • @BrettRandall
    @BrettRandall 4 года назад +1

    Thanks Robin - an easy to follow walk-through of creating a useful SMT and running it in Kafka Connect.

  • @DenysonData
    @DenysonData 4 года назад +1

    Love the way you work your way through the hiccups

    • @rmoff
      @rmoff  4 года назад

      😊 thank you

  • @LMGaming0
    @LMGaming0 6 месяцев назад +1

    4 years later, still usefull

  • @victorbecerra6737
    @victorbecerra6737 4 года назад +1

    great stuff, finally I could understand the diff of use PUT instead of POST, I saw it before but really didn't know why. This is a kind of art, you make me remember to Bob Ross !

    • @rmoff
      @rmoff  4 года назад

      thank you :)

    • @harleyanderson1866
      @harleyanderson1866 3 года назад

      i know it is kinda off topic but do anyone know a good site to watch new tv shows online ?

    • @graysencasey301
      @graysencasey301 3 года назад

      @Harley Anderson I watch on Flixzone. You can find it by googling :)

    • @laytonrowan9172
      @laytonrowan9172 3 года назад

      @Graysen Casey definitely, I've been watching on FlixZone for years myself :D

    • @harleyanderson1866
      @harleyanderson1866 3 года назад

      @Graysen Casey Thanks, I signed up and it seems like a nice service :D I really appreciate it!!

  • @lalitshukla6585
    @lalitshukla6585 3 месяца назад

    Thanks for this video it really helped.
    Just one thing I’m curious to know…is this S3 connector free to use ? I meant that sink connector is open source?

  • @LMGaming0
    @LMGaming0 6 месяцев назад

    Just wondering if there is already a video of creating a custom partionner and use it in my connector? I've been struggling to day to implement that since I'm not a Java guy .. thanks

  • @jorgeluiscarhuaricaaguilar4849
    @jorgeluiscarhuaricaaguilar4849 4 месяца назад

    Hello Robin I was trying following your video but the voluble connector is not longer supported. So the step in 9 min is not possible to follow. Can you give another options to produce data like that connector. I also was trying to use the kafka-console-producer command but I could not get data inserted into my bucket yet.

    • @rmoff
      @rmoff  4 месяца назад

      Check out shadowtraffic.io - it's what Voluble was initially conceived as.

  • @sohiljain7973
    @sohiljain7973 4 года назад +1

    Very helpful. Thanks Robin

    • @rmoff
      @rmoff  4 года назад

      Glad it helped!

  • @provaas7
    @provaas7 3 года назад +1

    Hi Robin, great demo, thank you so much, but I'm getting error: "io.confluent.ksql.util.KsqlServerException: org.apache.http.conn.HttpHostConnectException: Connect to kafka-connect:8083 [kafka-connect/172.20.0.5] failed: Connection refused (Connection refused)" when trying show connectors through ksqldb, any idea?

    • @rmoff
      @rmoff  3 года назад

      Hi Prasanna, head over to forum.confluent.io/ and post your question there :)

  • @djdjoko
    @djdjoko 3 года назад +1

    great video! thanks for the step by step guide.

    • @rmoff
      @rmoff  3 года назад +1

      Glad you enjoyed it!

  • @surajchauhan-zj9eb
    @surajchauhan-zj9eb 2 года назад

    Hello robin , I am stuck with some problems in kafka connect and S3,is there anyway we can coonect ?

  • @JordanCricketMoore
    @JordanCricketMoore 4 года назад

    I would recommend using Apache Pig, Spark, or Drill for reading Avro from S3 without downloading the files.

    • @JordanCricketMoore
      @JordanCricketMoore 4 года назад

      @@rmoff drill.apache.org/docs/running-drill-on-docker/

  • @ryandaniels9756
    @ryandaniels9756 4 года назад +1

    Great video!

    • @rmoff
      @rmoff  4 года назад

      Thanks!

  • @djdjoko
    @djdjoko 3 года назад

    Is it possible for a consumer to receive data by offset for data that is stored in s3 via kafka?

    • @rmoff
      @rmoff  3 года назад +1

      Hi, I'm not quite sure I follow. Can you post more details at forum.confluent.io/ and I will try to help out there.

  • @pramodniralakeri2338
    @pramodniralakeri2338 3 года назад

    I'm getting curl: (52) Empty reply from server.
    any help, appreciated.

    • @rmoff
      @rmoff  3 года назад

      The best place to ask this, with full details of your environment, is at forum.confluent.io/

  • @gustavoanjos3548
    @gustavoanjos3548 3 года назад +1

    Amazing video

    • @rmoff
      @rmoff  3 года назад

      Thanks!

  • @ryandaniels9756
    @ryandaniels9756 4 года назад

    For future videos, can you zoom the web pages a bit so that the fonts are larger? Not referring to the momentary zooms, but rather the base zoom when viewing the page normally.

    • @rmoff
      @rmoff  4 года назад

      Yes happy to - Can you link to a timecode of an example where it's too small?

    • @ryandaniels9756
      @ryandaniels9756 4 года назад

      @@rmoff 21:23 is a bit small for me.

    • @rmoff
      @rmoff  4 года назад

      @@ryandaniels9756 ACK, and completely agree! This was my first ever RUclips and stuff like this shows :) Thanks for pointing it out.

    • @ryandaniels9756
      @ryandaniels9756 4 года назад

      @@rmoff No problem!