Kafka Basics

Get a Free System Design PDF with 158 pages by subscribing to our weekly newsletter: bytebytego.ck....
Animation tools: Adobe Illustrator and After Effects.
Checkout our bestselling System Design Interview books:
Volume 1: amzn.to/3Ou7gkd
Volume 2: amzn.to/3HqGozy
The digital version of System Design Interview books: bit.ly/3mlDSk9
ABOUT US:
Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview series.

[@tomydurazno6243]

I used Kafka in a project some years ago. The idea behind a durable log not coupled with producers or consumers was a really eye opener

[@maheshsrocks]

? 🤔

[@honor9lite1337]

Indeed

[@gus473]

👍 Glad to see your channel do Shorts! Great stuff! 😎✌️

[@pcodes]

Fun fact: It was initially developed by LinkedIn employees.

[@ihsanpro9406]

Interesting. They made it open source ? Wonder why aws has exclusive rights to it

[@KinreeveNaku]

⁠@@ihsanpro9406they don’t. Kafka is managed by the ASF. It’s generally open source and free use in most implementations. AWS has its own implementation wrapper labeled as MSK but that’s kinda different

[@shadowpenguin3482]

What do you mean AWS has exclusive rights to it? Kafka is licensed with Apache 2.0

[@ihsanpro9406]

@@shadowpenguin3482 my bad I commented on the wrong video, was referring to ELK stack where they had a dispute with AWs

[@DanWalshTV]

​@@ihsanpro9406They don't, it's open source.

[@hasanhanzalah9799]

Your videos and shorts are really helpful, thank you. It is making real difference.

[@ericstephenson9924]

I think this could be explained in more detail with examples. It's still too abstract to get a handle on what it does and how it's used.

[@fabio.1]

I'd also like more details

[@Trekiros]

Shorts can only be up to one minute long, but see that play button that says "System Design, why is.." above the short's title? That's a link to a full length video that goes into more detail

[@jgeoffrey1]

Let me google that for you😅

[@lugebeatzz8747]

Simple concepts are useful and this video does that. From this video you can search for the relevant information you need and some code.

[@Lima3578user]

You are the best trainer in RUclips

[@blender_wiki]

Kafka is very useful also for small projects that need strong orchestration

[@Kelz_369]

You're a gem 💎 thank you for all the content

[@alexXove]

This channel is great ! Thanks!

[@guptaachin]

You are amazing.!!

[@SajjadAhmad-zq2jb]

need a longer version for kafka for pub/sub & queues. Show how this works for different designs.

[@shrine010]

sure but partitions not migrating to other nodes automatically and if more brokers fails, durability too

[@gro967]

You forgot to mention, that 99,9% of companies will never ever need Kafka, so don't even try to use it, it's not the right solution for your use case.

[@Trekiros]

I used it in a tiny French start-up with just a couple hundred customers
It was definitely super overkill, but hey, looks great on the resume :p

[@benisrood]

True... And yet every job listing asks for experience with it 😂

[@gro967]

@@benisrood that are the same ones that asked for 10 years of kubernetes experience 5 years ago...

[@alonsoruiz6315]

We’ve used it for a large XDR platform. It was literally our backbone. Everything worked as a charm. The downside, nevertheless, is that you might need specific data engineers maintaining the Kafka cluster.

[@gro967]

@@alonsoruiz6315 you have one of the rare use cases where it is a great choice, it's still not the right one for 99,9% of the other cases, especially in a time, where people think they are data engineers because they let ChatGPT write some sql...

[@MaddyIndia]

What a wonderful explanation! Understood in under a minute brother

[@michaelkoukoui1235]

Can anyone explain why given netflix microservice architecture why not use Nats jetstream instead of kafka?

[@wendysanarwanto6817]

I wonder which one that is more performant between Kafka & Rabbit MQ ?

[@hellamaxine]

they are not exactly the same. rabbitmq is a message broker vs kafka a streaming platform. though you can use both of them for similar usecases

[@luuc]

Awesome, keep doing these!

[@supercompooper]

I love kafka. I dont love having to rewire junior devs world view to change their paradigms

[@quintesse]

Sooo why is this tagged #javascript and #python when Kafka is written in #java and #scala? ^^

[@codeforbreakfast]

Views obv

[@rockymaivia787]

Kafka and Cribl solve the same use cases??

[@ihsanpro9406]

You should make a video on NATs which is a cheaper alternative to Kafka

[@jaimeduncan6167]

What do you man cheaper , Kafka if free and open source.

[@ihsanpro9406]

@@jaimeduncan6167 I meant infrastructure ways. It’s more complex to maintain and cost can go up. Many companies switched to NATs from Kafka but of course Kafka has more features, always trades off.

[@bjo004]

@@ihsanpro9406looks like NATS is deprecated
This component is deprecated because the NATS Streaming Server was deprecated in June 2023 and no longer receives updates. Users are encouraged to switch to using JetStream as an alternative.

[@v1shn3vsk72]

We were using Kafka with our golang microservices. There is no official golang client for Kafka, sometimes Kafka did lose our messages, and somehow it writes message (

[@v1shn3vsk72]

@@ihsanpro9406 yeah. We send messages to subscribers with retryers (to prevent network errors). When consumer receives the message it tries to process it. If it was successful we use Nats DoubleAck (also with retryers). If something went wrong during processing we are using NackWithDelay/Term/Nack

[@Kenbomp]

Million dollar hardware also helps as well

[@JimSendre]

Ironic name for a system where stuff doesn't get lost.

[@sumitsp01]

I thought kafka just streams messages in the form of keys and values.
Did not know video data can also be streamed through it.

[@chirayumishra1185]

Why everyone only created basics videos. Experts should make videos with designs to support Netflix level system and Best practices to like Schema Registry, How to centralized Or generalized designs, Apply different pattern, there so many thi gs to be taken care beyond basics

[@rishiraj2548]

Superpower

[@StLouis-bi3fi]

Yea Kafka is amazing and fun to work with, Redis is like it’s sidekick in someways for ultra cache collection and very efficient real time logging of data. I used both at my current internship and they’re fun to work with

[@miristegal]

so, the same as Kinesis?

[@fernandoangelrodriguezgonz1393]

🙌🙌

[@ehtasamgamer4477]

thought it was Kafaka hebino from Kaiju no.8

[@satish1012]

But we can't straem the videos byitself by Kafka.

[@maphilak]

Thought they used torrents

[@TheMegaMrMe]

Isn't the analogy to superhighways overrated in 2024? Highways are just expensive crappy slow methods of transportation vulnerable to induced demand.

[@liyuling1984]

Is he Alex Xu?

[@smit359]

Big giants needs big giant 😂

[@king-cog]

You were on point until you said... "lightning speed"... 😅

[@boxingexpert9065]

Sup Satoshi

[@phrous]

No longer secret

[@linuxuser2928]

I wonder how I could use this for a mc server.

[@kayakMike1000]

This is exactly what I would expect someone from Apache would say....

[@0ninetyseven97]

His videos is very advanced for a beginner like me 🤣

[@thianngunsang]

I learned literally nothing 👍

[@movax20h]

Hahaha. No.

[@Mikkihiiri27]

Really bad name. Kafka is overrated nonsense.