Running Kafka on Docker with Compose
HTML-код
- Опубликовано: 16 ноя 2024
- Subscribe to show your support! goo.gl/1Ty1Q2 .
Patreon 👉🏽 / marceldempers
In this video we extend what we achieved on our Introduction to Kafka video and this time bootstrapping it all on top of Docker compose.
This will make it easier to start up the environment locally as we progress through the series.
If you missed the Introduction to Kafka, check it out here 👇🏽
• What is Kafka | Tutori...
Checkout the source code below 👇🏽 and follow along 🤓
Also if you want to support the channel further, become a member 😎
marceldempers....
Checkout "That DevOps Community" too
marceldempers....
Source Code 🧐
--------------------------------------------------------------
github.com/mar...
If you are new to Kubernetes, check out my getting started playlist on Kubernetes below :)
Kubernetes Guide for Beginners:
---------------------------------------------------
• Kubernetes development...
Kubernetes Monitoring Guide:
-----------------------------------------------
• Kubernetes Monitoring ...
Kubernetes Secret Management Guide:
--------------------------------------------------------------
• Kubernetes Secret Mana...
Like and Subscribe for more :)
Follow me on socials!
marceldempers.dev
Twitter | / marceldempers
GitHub | github.com/mar...
Facebook | thatdevop...
LinkedIn | / marceldempers
Instagram | / thatdevopsguy
Music:
Track: Sappheiros - Affection | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / affection
Track: J3bii - camel toe [ Prod. J3bi ] | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / camel-toe-prod-j3bi
Track: Reckoner - lofi hip hop chill beats for study~game~sleep | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / reckoner-lofi-hip-hop-...
Track: souKo - souKo - Parallel | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / parallel
Previous mission:
Introduction to Kafka! Check the link in the description
Today's mission:
We move on to docker-compose!
Next mission:
We'll be building a consumer and learn
how messages are structured, distributed and more advanced
message distributed streaming!
Future mission: Kubernetes!
Best practical tutorial on Kafka! Thanks
3 days grid locked till I found this video, thank you soooo much
Love hearing a South African accent explain these hot topics!
Wow, thank you! You are very strong men💪 hello from Russia
Can't believe this channel is only at 36K. Great stuff.
Wow Awesome...!
This is the best Kafka tutorial I have ever seen in RUclips. I'm a beginner to Kafka and was looking for the basic knowledge and installation steps. And luckily have found this amazing video... Thanks a ton!! 👍👌👏
Love your style of video making. Inspires me. Thank you. Please keep making such awesome content. On zookeeper maybe next
In love with that doll above :D BTW great tutorial ! THANKS
Awesome video as always ❤️ Please do a video on how to build a Kubernetes controller with go 🙏🏽
Thank you for this detailed video
Thanks for the wonderful tutorial. I have a question regarding the same.
How to connect to the kafka cluster from outside ? for eg, from windows cmd, can I connect to the kafka cluster, running on docker on same machine.
It's a brilliant walk though! Haven't tried it along, but should work I believe :-)
Thank you all your effort
Nice informative video. Waiting for the next video
Thanks mercel
I am getting below error while running docker-compose up, any idea on this?
Attaching to kafka-consumer, kafka-consumer-go, zookeeper-1, kafka-producer, kafka-3, kafka-1, kafka-2
: invalid option | /bin/bash: -
: invalid option | /bin/bash: -
: invalid option | /bin/bash: -
: invalid option | /bin/bash: -
kafka-3 exited with code 1
kafka-1 exited with code 1
kafka-2 exited with code 1
zookeeper-1 exited with code 1
kafka-consumer-go | Failed to init Kafka consumer group: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
kafka-consumer-go |
kafka-consumer-go | goroutine 1 [running]:
kafka-consumer-go | main.main()
kafka-consumer-go | /app/main.go:36 +0x75b
kafka-consumer-go exited with code 2
DO have have a video on connecting golang and postgres using docker?
Thanks for the content.could you please explain bind mount ( data volumes) on ecs fargate? Especially when a container needs some datas from another container that belongs to the same task definition?
Thanks
Awesome!
Amazing demo, but am just wondering why you didnt deploy Kafka on K8s!!
Can we get something around Kafka on kubernetes...
I would really love to see Kubernetes on Kafka...?
Wonderful tutorial !👍
How to create API endpoints to communicate with kafka producers? By exposing port 80
Request tutorial how to install kafka on kubernetes
💪
Why do you have to go into the consumer to create topic?
This is awesome but still did not have the answers to my problems.
Wonder if you would kindly give any opinions on my question :
So I created producer on docker and the Kafka brokers are oustide of the docker (different environment and defnitely not on docker). I'm pretty sure I can connect my producer on docker to this Kafka brokers by configuring the docker-compose.yml. Is it the kafka_advertised_listeners or from how? Do I still need to configure the zookeeper?
Yes the application can point to the Kafka brokers via environment variables (KAFKA_PEERS in my example go application, see consumer in source code). You will need zookeeper only for Kafka high availability as it helps manage Kafka clusters
@@MarcelDempers not sure I understand, wdym "in my example go application"? is it on this video or do I have to look somewhere on your github?
Correct, its in the source code, this is part 2. In part 3, we build a consumer in go that talks to kafka brokers. See ruclips.net/video/xNDwt1tEZkw/видео.html for that content
Source code is in the description of each video too if you need it
Amazing video! Extremely well explained. But I am having a big issue! Bringing down the containers, then bringing them back up, I'd assume they'd come back no problem thanks to the persistent storage of the data. But when brought back to life, Kafka1-3 immediately die again!
The error looks something like this: ERROR Error while creating ephemeral at /brokers/ids/2, node already exists and owner '3423423423423' does not match current session '2423432423432'
But for all three brokers.
Only way to get Docker-compose up to work again is to delete the /data directory entirely, then they come back with no problem.
Any reccs?