CTO walks into the office tomorrow CTO: We are moving from rabbitmq to kafka Engineer: why is the move necessary? CTO: Doordash says rabbitmq is bad. Let's start moving.
I like how pretty much every article in the style of "moving from X to Y" initially sounds like "Y is betterh than X, but later it turns out that the system designers either never thought that theywould grow to this size, or they never knew that X had a limitation, or some other random thing happened that makes Y more suitable than X, for *their* specific setup. It's interesting to learn why they wanted to move and hw they moved, but as you said at the beginning these articles do not tell you that one is better than the other, only that the other was more suitable for that particular application..
I love the videos that discuss others companies decisions. Thank you. Keep it up 👍🏾. My point of view Let’s not forget that these companies work with A LOT of asynchronous tasks. If they have decided to move from one technology to another, it just means that the current technologies do not meet their criteria anymore. For common life engineers and developers, these current technologies can work very fine. RabbitMQ is simple and easy for simple task like sending emails, sending notifications, image processing, and others.
These case studies are awesome. It really helps to bridge that transition between knowing the concepts and being able o apply them successfully in production. More informacion please!
Minor correction - Celery does not open&close connection for each task request submission. The high conn. churn rate could be because high rate of server (producers) churn - when they have surge (auto-scaling kicking in) or something - or some sort of process-recycling feature (which would recycle the conn. as well) with server (uWSGI has this feature) which could be often and (worst if) at the same time (as this as often not randomised). This is guess-work by me - with whatever info available.
25:40 Depending on the number of partitions and the number of consumers, a consumer can be assigned multiple partitions or even zero partitions (if I am not mistaken, 2 partition 3 consumers for example). So maybe „exactly one“ is unfortunate use of wording here. Nevertheless a partition is always assigned to exactly one consumer. EDIT: In context of a consumer group.
Building it yourself slows you down, and Celery is a very strong task package. It allows you to get up and running quickly. And then developing tests for your celery tasks is much cleaner than building out a pub/sub package. I'm currently dealing with this nightmare. Celery allows you to bypass the message broker and call your methods directly. So you can run tests using pytest without running a rabbit machine. Some say that doesn't really matter, but to me, having to connect to a message broker for your tests to run is a pain in the ass at times. Also, flower is a UI that is built for celery so you can run your tasks and see what is happening on the backend. Again, not a lot of data there, but theres enough to know your system is running properly. If you NEED to build it yourself, then do so. But if you need speed, python probably isn't your move anyway
Asynchronous doesn’t necessarily mean things are queued, that’s only true if asynchronous is implemented using event-loop. In a multi-threading environment, the async task will be separated to a different thread away from the main thread, and both of them will be processing at the same time.
My question is the churnout problem they were facing with celery and how they resolved that with kafka is having multiple exectuors in parallel, behind kafka consumer, why couldnt do the same with celery?Why custom executors are working better than celery, whats the point of onboarding celery then?
@@hnasr thanks for the reply, my question is basically with celery. Given that rabbitmq + celery is a popular combo ,how come they have scaling issue?Is that because of the no of topics or something very specific to their business that celery just cant hanlde
Vaishnav Sreekanth Menon It is hard to find to be honest.. but I tailored google news so it serve me stuff I like.. and I report on the ones I like.. some news come from twitter
Sudha Rajamanickam Not basic at all question is valid The 6-10 connection is just a browser arbitrarily limit to enable parallel request sending You can open as many connections as you want to a single host as long as 1) the client has enough memory 2) the server has enough memory 3) the client doesn’t exceed 2^16 source ports 4) the server doesn’t exceed 2^16 dest port This number is per source ip and destination ip So the server can open up 2^16 for every client But the server doesn’t usually have a single client it has many... the total number or connections for the server for all clients is limited to memory and cpu usage. Whatsapp single server handles 3 million tcp connections for example WhatsApp handles 3 MILLION TCP Connections Per Server! How do they do it? Let us discuss ruclips.net/video/vQ5o4wPvUXg/видео.html
Thank you Hussein !! Celery has configuration called broker_pool_limit, If set to None or 0 the connection pool will be disabled and connections will be established and closed for every use. Check docs for configuration option docs.celeryproject.org/en/stable/userguide/configuration.html#broker-pool-limit
Good find!! Thanks for sharing.. so you would need to set this connection pool to an appropriate number so it preheats and reuse connections.. I wonder if doordash team disabled this for a particular reason
Hussein Nasser I hope, they might have disabled in the early stage of the project like any another mis-configuration settings or some other reason too.
How does the request and response work while using message queue? Using REST it is easy. Suppose ServiceA called via REST ServiceB. And after processing the request ServiceB response with the process result. But when communicating between ServiceA and ServiceB using message queue how does it work? Because there is no response.
CTO walks into the office tomorrow
CTO: We are moving from rabbitmq to kafka
Engineer: why is the move necessary?
CTO: Doordash says rabbitmq is bad. Let's start moving.
That is not correct, DoorDash doesn't say that, they said they have some requirements that RabbitMQ doesn't fit for
@@hassanharera I think the guys' being sarcastic....
I like how pretty much every article in the style of "moving from X to Y" initially sounds like "Y is betterh than X, but later it turns out that the system designers either never thought that theywould grow to this size, or they never knew that X had a limitation, or some other random thing happened that makes Y more suitable than X, for *their* specific setup.
It's interesting to learn why they wanted to move and hw they moved, but as you said at the beginning these articles do not tell you that one is better than the other, only that the other was more suitable for that particular application..
I love the videos that discuss others companies decisions. Thank you. Keep it up 👍🏾.
My point of view
Let’s not forget that these companies work with A LOT of asynchronous tasks. If they have decided to move from one technology to another, it just means that the current technologies do not meet their criteria anymore.
For common life engineers and developers, these current technologies can work very fine.
RabbitMQ is simple and easy for simple task like sending emails, sending notifications, image processing, and others.
Agreed.
Yes realtime problems help more than inverting binary tree on leetcode :p
These case studies are awesome. It really helps to bridge that transition between knowing the concepts and being able o apply them successfully in production. More informacion please!
I totally agree! You can see the concepts in real production
Great video Hussein. Your videos are so informative! Could you please make a video on Event Sourcing and CQRS?
Thanks Hussein!! This is cool webside . Such Like :"Scaling Splunk Securely by Building a Custom Terraform Provider"
Minor correction - Celery does not open&close connection for each task request submission.
The high conn. churn rate could be because high rate of server (producers) churn - when they have surge (auto-scaling kicking in) or something - or some sort of process-recycling feature (which would recycle the conn. as well) with server (uWSGI has this feature) which could be often and (worst if) at the same time (as this as often not randomised). This is guess-work by me - with whatever info available.
25:40 Depending on the number of partitions and the number of consumers, a consumer can be assigned multiple partitions or even zero partitions (if I am not mistaken, 2 partition 3 consumers for example). So maybe „exactly one“ is unfortunate use of wording here. Nevertheless a partition is always assigned to exactly one consumer.
EDIT:
In context of a consumer group.
Super video! I applauded for $2.00 👏
❤️❤️
Building it yourself slows you down, and Celery is a very strong task package. It allows you to get up and running quickly. And then developing tests for your celery tasks is much cleaner than building out a pub/sub package. I'm currently dealing with this nightmare. Celery allows you to bypass the message broker and call your methods directly. So you can run tests using pytest without running a rabbit machine. Some say that doesn't really matter, but to me, having to connect to a message broker for your tests to run is a pain in the ass at times. Also, flower is a UI that is built for celery so you can run your tasks and see what is happening on the backend. Again, not a lot of data there, but theres enough to know your system is running properly. If you NEED to build it yourself, then do so. But if you need speed, python probably isn't your move anyway
Great video. Thanks for sharing knowledge.
Asynchronous doesn’t necessarily mean things are queued, that’s only true if asynchronous is implemented using event-loop. In a multi-threading environment, the async task will be separated to a different thread away from the main thread, and both of them will be processing at the same time.
Very good spanish my friend 😂
I don't think closing a queue is a solution. You don't want the queue to be opened forever as that queue cannot process tasks related to another user
My question is the churnout problem they were facing with celery and how they resolved that with kafka is having multiple exectuors in parallel, behind kafka consumer, why couldnt do the same with celery?Why custom executors are working better than celery, whats the point of onboarding celery then?
Siddiqui Sarik my understanding is that celery doesn’t support kafka
@@hnasr thanks for the reply, my question is basically with celery. Given that rabbitmq + celery is a popular combo ,how come they have scaling issue?Is that because of the no of topics or something very specific to their business that celery just cant hanlde
Quick question: how do you come by all these articles? :thinking
Vaishnav Sreekanth Menon It is hard to find to be honest.. but I tailored google news so it serve me stuff I like.. and I report on the ones I like.. some news come from twitter
Sometimes I can't even understand his videos. May be because I'm a fresher to this job
Sorry for the basic question. active connections is always confusing to me..How many tcp connetions a client can have at a time with server(
Sudha Rajamanickam Not basic at all question is valid
The 6-10 connection is just a browser arbitrarily limit to enable parallel request sending
You can open as many connections as you want to a single host as long as
1) the client has enough memory
2) the server has enough memory
3) the client doesn’t exceed 2^16 source ports
4) the server doesn’t exceed 2^16 dest port
This number is per source ip and destination ip
So the server can open up 2^16 for every client
But the server doesn’t usually have a single client it has many... the total number or connections for the server for all clients is limited to memory and cpu usage. Whatsapp single server handles 3 million tcp connections for example
WhatsApp handles 3 MILLION TCP Connections Per Server! How do they do it? Let us discuss
ruclips.net/video/vQ5o4wPvUXg/видео.html
Thank you for the video! Do you know what tool they used to make those architecture diagrams?
I know they are pretty good not sure what they use. Probably Google Draw? I use gliffy for my diagrams when I do any
Excellent bro... Please keep up the good work
Thank you Hussein !!
Celery has configuration called broker_pool_limit, If set to None or 0 the connection pool will be disabled and connections will be established and closed for every use.
Check docs for configuration option
docs.celeryproject.org/en/stable/userguide/configuration.html#broker-pool-limit
Good find!! Thanks for sharing.. so you would need to set this connection pool to an appropriate number so it preheats and reuse connections.. I wonder if doordash team disabled this for a particular reason
Hussein Nasser I hope, they might have disabled in the early stage of the project like any another mis-configuration settings or some other reason too.
How does the request and response work while using message queue? Using REST it is easy. Suppose ServiceA called via REST ServiceB. And after processing the request ServiceB response with the process result. But when communicating between ServiceA and ServiceB using message queue how does it work? Because there is no response.
Muztaba Hasanat i do discuss the same question here
Publish-Subscribe Architecture (Explained by Example)
ruclips.net/video/O1PgqUqZKTA/видео.html
You can store the task status in db. And have a GET api that will provide the status of that task.
Meow
agreed
this guy literally didn’t know anything about this stack. why do a video in it?
I also felt the same. I could not learn anything from this video.