Golang Microservices: Events Streaming using Apache Kafka
HTML-код
- Опубликовано: 25 июл 2024
- Let's add support for Apache Kafka! Apache Kafka is an event streaming platform that allows you to:
* Publish and subscribe to streams of events,
* Store streams of events for as long as you want, and
* Process streams of events as they occur.
In Go to interact with Kafka, we can use:
* github.com/confluentinc/confl... OR
* github.com/Shopify/sarama
I personally prefer using Confluent's package because they are the one behind the scenes providing commercial support for Kafka.
Learning Kafka takes some time and even if you're familiar with the tools I highly encourage you to take advantage of managed services like Amazon Managed Streaming for Apache Kafka (Amazon MSK) it will simplify your live when working with Kafka clusters!
---
* Example code: github.com/MarioCarrion/todo-...
* Blog: mariocarrion.com/2021/06/03/g...
* Previous Episode: "Golang Microservices: Golang Microservices: Events and Background Jobs using RabbitMQ": • Golang Microservices: ...
* Playlist "Building Microservices in Go/Golang": • Building Microservices...
* Playlist "Golang Tools and Packages": • Golang/Go Tools and Pa...
* Playlist "Testing in Go": • Testing in Golang/Go
00:00 Start
00:12 What is Apache Kafka?
01:30 How does Apache Kafka work?
03:46 Apache Kafka + Golang Demo starts
10:00 Conclusions
---
Who am I:
Hello👋🏼! I'm Mario, a Hands-on Software Architect and Lead Backend Engineer with more than 16 years of professional experience building all kinds of software including on-premise Industrial Automation Systems, Linux Accessibility Desktop and Browser Components as well as Distributed Advertising Microservices.
Every week I will share with you different topics I've learned while working for small startups and large companies including the processes I've followed over the years for successfully delivering complex enterprise systems from start to end.
Subscribe if you like Software Development, Software Architecture and Systems Design!
Keep it up. Don't give up!
#golang #microservices #programming
--- Our affiliate links below
Shop our favorite products → www.amazon.com/shop/rubycarrion
Vlogging setup → rubycarrion.com/camerasforvlo...
Get a 30 day FREE trial of Epidemic Sound → www.epidemicsound.com/referra...
Try Amazon Prime 30-Day FREE trial → amzn.to/3s0el1R
I love getting Cash Back and think you will too! Join for free and get $20 when you spend $20 → www.rakuten.com/r/RUBYRA132?e...
Sign up to start using vidIQ to grow your RUclips channel for FREE → vidiq.com/rubycarrion
--- Our Vlog Channel
/ rubycarrion 
Наука
Thanks for making this video. Really helpful
Thanks Husein. Take care.
Nice Mario, thx for explanation!
Thanks Rodrigo! Cheers!
Thank you for all this content!
Thanks for watching! Cheers
@@MarioCarrion I was wondering if you would do another step by step from scratch tutorial for Kafka & microservices
That's not a bad idea, a lot of things have changed since I recorded this video.
Great video!
Will you be making a video on how to implement SAGA pattern for microservices?
Thanks Lau. Yes, I'm planning to covert different patters and Sagas are going to be included.
@@MarioCarrion nice! Looking forward to it :)
Hi Mario!
Thank you for this content, it's very well explained.
Would it be possible for you to make a tutorial about adding gocron package in this microservice series?
From a quick search "gocron" seems to emulate the original cron; I don't know you're use case but I'd keep it simple and try using whatever containerization option you currently use for running scheduled jobs, that's typically the simplest way to do things.
Have you tried kafka, KSQL and scylladb? Can you make a video on that subject?
I haven't tried scylladb, but I'll look into it. Thanks for the suggestion.
sir can u make an entire golang project with apis and microservices and deployments using k8 docker...i love ur channel
Thanks for the idea. I'll see what I can do.
Hi mario, i have a use case. Imagine our web server is exposing some rest endpoints (gorila mux) that have long time to process (external systems) so we publish a message on kafka or nats and just give result back to user that we promise when we have result available u will see it in web ui. Cosumers that are console apps written in go consume event, process and publish a new event with result.
Now we want to consume those result events inside same gorilla web server and through websocket or server sent event wana update user that their long async task is completed.
My question is if we are running a infinite loop and our gorilla mux router endpoints still work ? i am guessing since this loop is not sleeping request will not be handled and user will face timeouts
Hi! You're correct, your client will eventually time out; however what if you approach it in a different way? Let your customers know (like you said) "we are processing your request come back later", and in the backend do what you said (consume and process of the event) and then let the customer know "your result is available" maybe using an email or other alert mechanism.
Other way would be the frontend to poll the status of the process from the backend from time to time, to indicate the progress being made, but I don't like that approach because the client could leave a browser open all day, and if the process takes long time like you said you are going to be wasting backend resources.