Event sourcing anyone? How to install Apache Kafka in production

We look into how to install Apache Kafka in production using ZooKeeper for redundancy and working with topics. We also setup a producer for syslogs and checking the consumer for all the logs to appear.
India Exclusive: For a limited time, you can get an annual subscription to Coursera Plus for just INR 7999 (normally INR 33,0001). That’s over 75% in savings for unlimited access.
imp.i384100.net/c/3402234/207...
Expand your career horizons with 40% off Coursera Plus annual subscription
imp.i384100.net/c/3402234/207...
Get Proficient knowledge in Azure services and infrastructures, managing cloud resources, cloud security with our Cloud Support Associate Certificate !
imp.i384100.net/c/3402234/203...
Join the channel to get access to more perks:
/ @danielpersson
Or visit my blog at:
danielpersson.dev
Outro music: Sanaas Scylla
#zookeeper #kafka #apache

[@aaronperl]

When we introduced Kafka a few years ago on the project I was working on, it took me a long time to wrap my head around how it's different from RabbitMQ (which we were also using), and why the extra headache would be worth it. Then we started introducing geo-redundancy, and Kafka allowed each location to read the same incoming data, and it finally started to make sense.
For component, integration, and end-to-end testing, we we had to deploy local Kafka instances in Docker containers ... and what a headache. I never understood why Kafka needs to know the external port that its container is being mapped to, but when using random port assignment, it made it unnecessarily difficult to automate, because we had to bring up the containers, then check which port got assigned, then update the docker-compose.yml with the exposed port, and restart the containers. Insanity. Eventually we reworked our configuration so we could use pre-defined ports, which made things a lot more manageable.
Then they introduced and started enforcing Avro...

[@johnjcenapolis]

First time commenting, I've pretty much watched all your ceph videos and it seems like you make a video at the exact time I start working on something. Any chance of getting a part 2 with Kafka as the message queue and clickhouse for log storage?
Keep up the great work!

[@DanielPersson]

Hi John.
Are we working for the same company ? :)
I had to learn Kafka so I could maintain a solution for my company as some developers wanted to use it for statistics. Not sure if message queue is the best application for Kafka but I could look into the term clickhouse, never heard about it.
Thank you for watching my videos.
Best regards
Daniel