3 Important Use Cases Why The Industry Is Using Kafka
HTML-код
- Опубликовано: 29 сен 2024
- ► Join my Discord community for free education 👉 / discord
► Exclusive Videos, Mini courses, Resume reviews, and coaching 👉 thetotalcoder.com
► 33% OFF on my Go + HTMX + Templ Course 👉bit.ly/3UFruxO
► Enjoy a 60% Black Friday Discount on My Golang Course 👉 fulltimegodev.com
► Follow me on Twitter 👉 / anthdm
► Follow me on GitHub 👉 github.com/anthdm
SUBSCRIBE OR NO MARGARITAS
╔═╦╗╔╦╗╔═╦═╦╦╦╦╗╔═╗
║╚╣║║║╚╣╚╣╔╣╔╣║╚╣═╣
╠╗║╚╝║║╠╗║╚╣║║║║║═╣
╚═╩══╩═╩═╩═╩╝╚╩═╩═╝
► 33% OFF on my Go + HTMX + Templ Course PRESALE bit.ly/3UFruxO
► Exclusive Videos, Mini courses, Resume reviews, and coaching 👉 thetotalcoder.com
► Join my Discord community for free education discord.com/invite/Ac7CWREe58
► 60% OFF on my Golang course fulltimegodev.com
Thanks for watching
As someone who has heard about Kafka but no idea how it worked, this was amazing, i understood everything, please trickle in more videos like this
My problem, as someone who has only been part of big project using kafka for a month, was never kafka itself. The concept is pretty simple, but the problem is when you have extremely rushed development, bad documentation, over 40 topics, each having 50 partitions, bambilion producers and consumers that are actually multiple companies, coworkers that don't have time for daily 2 hour long calls to explain stuff, man it's hard. And I'm sure this is pretty "common" and that it can be soooo much worse
I feel you
Hey what are some of the differences between SQS and Kafka ? 🤔what are some of the points you should use one over the other ?
NATS > Kafka for me
Don’t be a Timmy at least give us the reasons as to why NATS works well for your set of problems.
@@none_the_less 1. It's written in Go. I can read the source and even embed NATS server into my own binary. 2. More flexible and has distributed storage beyond just queues.. 3. It's very light and consumes less resources.
@@none_the_less 1. Written in Go. You can even NATS embed server into your own application.
2. It's light and consumes less resources. You can deploy nodes absolutely anywhere, including Edge, IoT devices, etc...
3. It's pretty flexible and has features not only for queues but also K/V that is distributed with full history.
4. And finally it's I feel like it's more stable than Kafka. I have been using it in prod for 4 years and haven't had a single issue or weird crash.
@@dyto2287 Thanks mate.
I learnt some stuff I didn't know about NATS, because I used to think that it was just another broker.
Do you also use jetstream? Because we faced some weird issues with JS.
also it allows a distributed event-based system that used to be event-based monolith.
True
Last advice is actually pretty awesome.
So Kafka is somewhat like a buffered channel but instead of a couple of concurrent goroutines we have separate services?
I already knew all this but still watched entire video, I wish I got this video back when I started using Kafka.. great explanation Anthony
I am using Kafka in production. For log analysis and for IOt device sensor processing. Maybe kafka makes sense for some uses cases. But for the most of the cases where the folks kakfa is for, other systems like nats fits better. Kafka is not a broker. Kafka is a distrubuted log. That is my one billion dollar mistake that I understand after going to production.
1. Every Tranformation on a IOT device strem ( re partioning ) will multiply your storage.
2. Cluster management is complicated
3. You have to persist data. You cannot work inmem
4. Data adations means that you have process gigs of data agin. ( Stoooorage that you don't have on an edge device )
5. Reboot of the whlo kafka cluster can take hours if the election is loosed ( I deon't how it is after they replaced zookeeper )
The stregth on the other hand is rthat you can write in transactional manner so that consumers do not process messages until the producer commits and the java streaming api.
If you will do messageing and stream processing. Take NATS at first.
NATS and edge go hand in hand
Exactly, Kafka is more of a fast as fuck distributed queue, but it’s not a broker. If you need routing, multiple consumers and all that, NATS or RabbitMQ should be better.
I have now started saying "Headesh" instead of "Headache".
interesting.. i basically use a "journal" table to track of jobs and process them in order.
practically the same thing, except much less complex.
the replay functionality is very nice, especially if you have the situation where you introduced a bug in the latest release a couple of days ago and just released a hotfix, you can command a "replay" or recalculation from the offset before the bug was released. kind of neat!
my implementation is not a distributed service, i use it within my monolith!
So kafka helps if service 2 goes down. But what if the kafka service itself goes down? They’re all just services after all right? And if kafka instances are replicated, so could service 2 instances. This leaves just the second benefit mentioned, which is that multiple downstream services can subscribe to the same topic and hence won’t interfere with each other
The thing is Kafka is already made, it works, and it’s freaking performant, you don’t want to write it yourself. Also, it’s easy for every service to write to Kafka and be done. If every service has to implement the queueing you end up reinventing the wheel many times and introducing bugs and complexity. Also, those services may be in many different languages and you bet theres a Kafka library for that language.
Great work boss you are really making us excellent developers.
please can you create a simple application or api using Kafka ?
What are the pros/cons of Kafka vs Rabbitmq?
Company I work for uses Kafka
Anthony I am a big fan of your nugget videos. Straight to the point short and packed! 💪
have you seen AWS SQS used in place ok Kafka in a production trading environment?
You'd need both SNS and SQS to fully replace Kafka I think. Both are heavily used in production.
Antony, i think you should make a video on how to make a crypto casino in golang or any other language of your choice. Sounds crazy... I love your videos
Wow, I always thought Kafka was useful, but every system I make caters for every aspect Kafka offers. Thx a lot!
Are you Johnny Sins's brother ?
Sometimes
Everyone talks about kafka but no one talks about NATs why??
I never understood why I needed a message queue instead of a simple in memory stack/linked list. So basically it's for fault tolerance and replaying the events whenever you want?
If your data fits in a linked list, you don’t need Kafka.
Is not really a message queue that's a common misconception. Its a distributed append only log with publish subscribe capabilities
A message queue is a queue with messages. Lets keep it simple for people.
what is a message queue then ??
@@kamalkumar1729 as Anthony said, a message queue is a queue with messages. It doesn't need to be persistent or a log, or be distributed, or partitioned, or support pub/sub. It just needs to have a protocol/API to consume and push messages (which is not the same as pub/sub since there doesn't have to be a fan out pattern common to subscriptions).
Of course the capabilities of kaka can be used as a queue, but it seems and overkill and there are simpler queues.