One other major toolkit item worth mentioning might be object storage (S3, minio, etc.) which might also be a good segue into discussing consistency/CAP theorem, depending on the use case. You might want to mention object storage for things like user uploads (videos, images, or whatever). Digging deeper into object storage could include storage tiering (hot/warm/cold storage).
This was an awesome video!! It was a great refresher/summary and I even learned a couple new things :D I’d definitely love to see more stuff like this-you have a really great, non judgmental, way of explaining these things!
Thanks for the video. Actually it is a really good question. I usually use Kafka but if I need some lightweight I create these by myself. What would you recommend to use or what are you used to use instead of Kafka for simple projects?
tbh for the simplest of things I'll (ab)use my database for most things. sqlite / postgres can be a kv store, document store / queue / message bus in a pinch though they aren't optimized for the usecase
I use NATS, which is a message broker that doubles down as a k-v store. It's much faster than the alternatives like RabbitMQ, and it's fully real-time, unlike Kafka(I know nothing about Kafka). I am also planning to use it internally instead of HTTP APIs, because it supports request-reply also. But if you can use SQLite for everything, do so.
2:49 instead of "consistency, availability, partition tolerance, pick 2" isn't it more like "here's your partition tolerance, now choose if you want it to be consistent or available"?
One other major toolkit item worth mentioning might be object storage (S3, minio, etc.) which might also be a good segue into discussing consistency/CAP theorem, depending on the use case. You might want to mention object storage for things like user uploads (videos, images, or whatever). Digging deeper into object storage could include storage tiering (hot/warm/cold storage).
I'd like a more in depth per segement overview, could even be a series. (tldr my DevOps interview was very much dumber'ed down than this example)
Very helpful. Would love more high level overview stuff like this
Loving your channel. Keep it up!
Never thought of discussing the Service Discovery system!
This was an awesome video!! It was a great refresher/summary and I even learned a couple new things :D
I’d definitely love to see more stuff like this-you have a really great, non judgmental, way of explaining these things!
very good content!
Great video, esp. the still at 28:40 is helpful. More breakdown along common types/optimization needs of systems would be great.
This was amazing thank you !
More system design videos please.
Can you go more in depth about the columnar store in this system? I didn't get it what kind of data would be stored in it and its purpose.
Thanks for the video. Actually it is a really good question. I usually use Kafka but if I need some lightweight I create these by myself. What would you recommend to use or what are you used to use instead of Kafka for simple projects?
tbh for the simplest of things I'll (ab)use my database for most things. sqlite / postgres can be a kv store, document store / queue / message bus in a pinch though they aren't optimized for the usecase
Rabbit is nice and standard which can easily be pulled and setup quickly using docker
I use NATS, which is a message broker that doubles down as a k-v store. It's much faster than the alternatives like RabbitMQ, and it's fully real-time, unlike Kafka(I know nothing about Kafka). I am also planning to use it internally instead of HTTP APIs, because it supports request-reply also.
But if you can use SQLite for everything, do so.
Didn’t you have a video on this already? Or maybe I caught it on stream?
Awesome content as always!
the notes I was working from were from a livetream yeah!
2:49 instead of "consistency, availability, partition tolerance, pick 2" isn't it more like "here's your partition tolerance, now choose if you want it to be consistent or available"?
not always. I've had a few where partition tolerance wasn't super necessary (but it wasn't your traditional web app sort of thing)
Amazing content, love it!
Great video. Some of this was new to me. If you could go in depth about each of these someday that would be helpful.
Greate episode! Sounds like an idea for eight more videos.
13:56 envoy hon hon hon