Finding your channel feels like finding gold! There are ton of SD videos on youtube with shallow content basically exactly like you mention what a junior or mid level candidate would do. Going indepth for senior and staff is one of the highlight of your content. Please continue doing that. Also please don't worry about length of the video. Keep the gold coming :)
Watch this and then imagine if Evan puts together a System Design Learning Course. Just image that !!!! I mean we (the learners) will jump on it sooo quickly. This is just absolutely amazing. This is combining years of experience with hands on actual approach that works along with book contents presented in a very professional manner. Evan, think about it 🙂
I have purchased so far Alex Xu grokking system design by Mikhail Smarshchok i would say i learned a LOT from the Mikhail Smarshchok as far as internals go and then i bumped into your Hello Interview ABSOLUTE BEST stuff that can replace Alex,grokking for me. I was ALWAYS looking for how to connect the functional requirements into the later high level design/ deep dives but both Alex, Grok fail many times in connecting the dots. Subscribed and a big fan of you. I will consider purchasing coaching or mocks once I am through your videos and feel confident.
The content is great as always. However I found this system a bit frustrating, there is so much specific knowledge one should know to really end up with a good design for this kind of problem. I wish I don't get a system like this in an interview.
Awesome walkthrough! As a junior engineer I learned a lot. Near the end you said you wanted to keep it short but I appreciate the nuances you point out for your decisions including design choices between mid and higher level candidates. Time didn't faze me at all, I watched every second!
You are the best at what you do, after listening to your videos I cannot now like and tolerate other system design videos, please make more content for system design
Really helpful videos - especially the breakdown for different expectations at mid/senior/staff levels, common things you see from candidates, and context into the actual systems like the shard limits for events streams. I used to work on Kinesis - happy you chose it!
Great job with creating this content to help us prep for interviews! Just one thing to note, you can't send a 302 redirect for POST requests, it has to be a GET request.
I rarely leave comments but this is BY FAR the best system design video I have seen on youtube. The thing I like most is that it differentiates from common textbook solutions that we see everywhere and you explain why certain choices were made. Thanks!
This video helped me ace the system design interview. The detailed explanations provided in-depth knowledge of various components, which was extremely helpful for answering follow-up questions during my interview.
Thank you so much for these videos! They’re absolute goldmines-I’m genuinely amazed at the depth and clarity they offer. I’ve learned more from these than from many of the books I’ve read. Truly invaluable!
These interview preps make you feel like if you know enough of system design knowledge, have good cross team examples for bq, and can solve leetcode medium-hard fast, you can get to higher level quicker than going through internal promotions.
Ah, nice, you re-uploaded! Thanks a lot for taking the feedback and acting quickly on this. And, sorry if it caused inconvenience for you 😄Thanks a lot for all of your hard work. 🙏
For hot shard problem in Kafka, you can salt things but in Flink, we will no longer have all the events aggregated in one Flink task. In the case where we are sinking to Redshift, we could have aggregate it there. but if we want to access it in stream, maybe we need a secondary stream that aggregates everything? So that added salt needs to be handled one way or the other
Thank you for a great video. For a senior candidate it will be helpful, in my opinion, to narrate the data structures that underpin these solutions in addition to the supporting vendor products/technologies. In that, for fast aggregation of counters, one could demonstrate the use of fast but slightly imprecise counters using a count-min-sketch-like data structure, and for a slower but more accurate counter the use of a Map Reduce jobs. Aggregates and statistics over varying windows are almost a necessity for contemporary Monitoring, Analytics and ML Systems. And in such cases retaining data in-memory backed by persistent storage in the form of tree data structures keyed by aggregation windows are useful for range queries at varying granularities. For e.g.: root window [0-100). Immediate child node windows [0-50), [50-100) etc. It could be helpful to talk about idempotency within queue consumers. And also out-of-sequence arrival of events in the queue (handled through watermarking)
Excellent, as usual! Thanks so much 🙏 While concluding, you mentioned you want to keep the videos short. Please don't reduce the length. 1 hour is a sweet spot and it's necessary to capture all the important "juicy" tidbits and details you highlight. Please keep it coming 🙌
I discovered your channel a few days and love all the videos so far. I love the solution enforce idempotent click tracking. There are a lot of SD videos but only your SD videos provide guidelines on the expectations based on candidates' levels. So far, I watch one video per day so I will run out of videos to watch very soon xD
you are honestly the best content on system design , can you do some playlist on the system design topics themselves ? i mean a video where you discuss replications in depth , concurrency , etc.
This was such a pleasure to watch. Thank You. I would love to see a video on a metrics monitoring system. There will be some common components with ad-click aggregators.
This is great content. However, I would like to point out that the Lambda architecture includes both batch and speed layers, which process historical and real-time data in parallel. It’s not solely reliant on batch processing.
Thanks a lot ! This video is super helpful. It not only helped me understand the key components of ad click aggregator, but more importantly it taught me what does the interviewer expect for different level SDEs. One question --- do we need to dive into the aggregation logic details in Flink?
Great video! I learnt a lot! One question I have is regarding the hot shard handling. When the click processor detects there is a hot shard and decides to change the key from AdId to AdId:n, how would it let Flink know that it now needs to change the aggregation logic (and sharding logic) for that particular AdId? (I believe this would also have a race condition when it changes within a minute window, but any data integrity issue that arise from it should get resolved by the batch job)
Stellar video. Only suggestion for the next ones: your drawing tool seem to have keyboard shortcuts for those diagrams and other options on the toolbar. It'll greatly improve your quality of life! Keep it up!!
Awesome content!! I was hoping to get more clarity on how same flink node determines the partitions it need to consume from in case of hot shard issue which we are effectively handling by adding a number b/w 0-n to the partition key which leads to same adId clicks published to multiple partitions in the stream.
these are so good , such deep dive and so clear ! Thanks because they cover so many different aspects . i am looking forward to many more videos . I am currently preparing for interviews and these are so helpful ..
Thank you so much for the explanation. However I more like you write the functional and non functional requirements instead of paste from the clipboard. The main reason is that it's make me thinking about what should be the requirements. It's feels more like code pairing
Thanks for the content; It's great; One query on 1. As we compute 10sec and write it to OLAP DBs ; The deep dive on how the user queries for 1min window, 1hr window, may be 1 day and all time window would be great info; 2. And what could be that OLAP database choice would be and what factors to consider. 3. Another aspect is; Freshness of the data; What does each component in the system contributes to the latency to make the event available for aggregates; Other aspects are great; thanks
Thanks for the video, it is really in-depth and informative. You covered the flink + streaming part really well and why not to use checkpointing was really a good point. Saw some videos in the past that mentioned let's do checkpointing and never explained why it will work or not work. For the streaming components both streams (kinesis or kafka) and stream processor (Flink/kafka-streams) I got following questions ( (I was asked similar questions in an interview.) 1. we have said that we will partition the shards or streams using adId but we have 10M ads at any given point of time. In this case, if I consider other things like replicas in kafka topic or even kinesis shards, don't we have to create lots of shards? 2. The max traffic is 10k cps so most of the ads are not clicked and won't have active traffic so most of the shards will be empty, in this case what should be the approach? 3. If we group some ads based on let's say advertiserId than we introduce the noisy neighbour problem where one ad generates lots of traffic and blocks the other ads.
Those questions are great and definitely come up often with streaming setups. I'm no expert but ... Partitioning with 10M Ads ... Yeah, if we just went with adId for partitioning, it would mean loads of shards or partitions, which isn’t practical. Instead, you can use consistent hashing or modulo partitioning to map multiple adIds to a smaller, manageable number of partitions.
I like your videos, I have learned a lot. A couple comments on this video: a. I think the system would benefit from a Redis in the click processor service, not the idempotency lock but a redirectUrl cache {adId: redirectUrl} to reduce reads in the Ad DB. It might be a MRU cache to avoid overloading the Redis. b. I'm not sure why you are pushing Kinesis so hard in this solution, I mean yes I learned something about Kinesis, but it would be more practical just to place a Kafka that can handle the load peaks and has event history as well so it is possible to write the reconciliation procedures from it. c. I learned about Flink, thanks. I used a redis aggregator in my own solution. Thank you so much for your work!
Hey Evan, your videos have been the best, seriously! The only suggestion I have is if you could use a dedicated mic that would be wonderful. The volume is too low even on max volume on my earphones.
Thank you for such a fantastic video! I have a quick question regarding the deduplication process. When using Redis to deduplicate based on impression IDs, would the entire operation of "check Redis -> populate Redis -> submit click to event stream" be atomic? If it's not atomic, there could be cases where Redis is updated after a click arrives, but before the click is submitted to the event stream, and if the server crashes, this could cause an issue. Even if we change the order to "check Redis -> submit click to event stream -> populate Redis," there's still a chance that two impressions could arrive at the same time, find no record in Redis, and both submit the same click to the event stream. Could you please clarify how this is typically handled? Thank you!
Question: For the sharding while processing the events through Kinesis, the adId was suggested as the sharding key. This doesn't look like the best approach. At scale, millions of ads are being run on the platform and a good share of them have high enough volume. Going by the presented logic, the number of shards would explode. What do you think about this?
Hi Evan, Absolutely gold content. I have one doubt here. For a really hot ad, you are adding some random no (1-n) to build the partition key before adding it to the kinesis stream. Now this particular ad can land into multiple spark consumers. How the different spark consumers will aggregate this data for this ad. Is there anything that i am missing? Are you referring to keyBy(adId) in Flink?
Hey, I love these videos. I only used your videos and designing data intensive applications and that was enough for an E4 offer at Meta, I love the advice you give and common pitfalls you provide.
Amazing system design. I've been searching for something exactly like that, which is interview driven, not "let me show you all the ways you can do it" driven. May I just make a small suggestion: please use "etcetera", not "eccetera"...
Few questions: - Even in spark solution, how would you know the keys of which one to aggregate? Either you have to emit keys of which has changed or scan the db by time and get keys which is again costly. - Aren't this falls under lambda architecture as we are using both realtime stream processing and batch processing to ensure data integrity?
Question: Why do we need the reconciliation mechanism? And why use a batch processing for the purpose? Do we know/assume stream processing would accumulate errors compared to batch processing (the latter being accurate?)?
Amazing content! Very much appreciate you posting these 🙌 System design padawan here. I have a question about the hybrid approach .. what makes us trust the "Kinesis -> Connector -> S3 -> Spark -> Worker -> OLAP" results more than the "Kinesis -> Flink -> OLAP" results? Is it a guarantee where the connector always successfully writes the data to S3? or does Flink make some kind of tradeoff for speed? kind of confused about that piece and figured i'd ask. thanks again!
+1 on this question. IMO, Spark is useful when you have really out-of-order events, like events arriving half an hour late or something. Then, by using Spark, you can reorder the events and get a more accurate count. On the other hand, for events that are only a few minutes late, you can configure Flink with the allowed lateness of a few minutes. That being said, the cost of maintaining 2 code bases and ensuring that they are updated at the same time (to avoid triggering discrepancy alerts) doesn't seem worth it for such edge cases. I'd be interested to hear @hello_interview's thoughts on this though.
> Is it a guarantee where the connector always successfully writes the data to S3? Yes. That is something provided by AWS as a managed service, and it should meet their SLAs. AWS would've created fallbacks and fault tolerance to ensure that all events that are in Kinesis reach S3. I don't see there being any compromise due to speed. Kinesis has retention policy, so the data isn't going anywhere, and S3 is highly available, so the data shall be written there as well.
Great video, covers so many new topics. Is the use of a time-series database practical for storing click events by timestamp? By design, that would help with aggregation in 1 minute windows, no?
We initially said that we want to perform complex queries. In addition to time range, we want to check who were the users, and maybe where the clicks were from, etc. So that means we need more types of aggregations, for example by user id, by location, etc. How do we handle that? What if we get a query that we didn't prepare an aggregation for in advance?
Love the content! Watched so many videos and they have really helped me in interviews. I had one questions. Instead of dumbing data from Kinesis to S3 to be read by spark. Why not just have spark batch processing read right off of Kinesis? If you are doing reconciliation every hour, day, or whatever. If is is smaller than the retention policy on Kinesis, I thought you could just do it from there. What am I missing?
I know you called out the issue of contention as a reason for having two db in the beginning. What makes the OLAP db different so that we don't need to worry about that for read and write queries?
I believe the checkpoint is still necessary so it avoids data loss. If the server crashed, the checkpoint stores what the offset in kafka/kinesis stream is at and restart from the offset in the checkpoint. Without the checkpoint, flink has no idea where it should be restart from and the data will be lost between the last time it send data to the OLAP and the the it crashed.
For Kinesis hot shards, we don't know if an ad is hot beforehand. So are these ad_id 0-N always active? Is it ok to use x10 the amount of streams we need under normal circumstances? For Flask, we have the same amount of flask servers as the Kinesis shards right? If the server dies, how will the new server keep track of the pointer from the old server? Are they statefull backups instead of stateless
This is a great question. In reality you can make predictions here. We know based on budget and historical performance which ads we’d need to be worried about before hand
Thank you for sharing! I got a question about the decision to not use checkpointing in Flink. If you don’t enable that then where would you store the Kafka consumer offsets for recovering?
I saw some videos and your content is so great. Thank you so much for clarifying the SQL vs NoSQL debate. I always thought that bringing that into an interview was irrelevant but was afraid to do it. 😅 Keep up the amazing work.
This is awesome. Thanks Evan. I have a question on the usage of blob storage. Aren't those supposed to be used for storing unstructured data like images, audio & video files? Could you please elaborate on 1) how Spark task reads data from S3 2) how Spark job would sustain reading one day's data? Is checkpointing used in this context?
Thanks for the great videos - they are extremely helpful. I noticed at around 24 mins in you mention querying for the number of distinct userIDs. I don't think you're going to be able to serve queries like that using the pre-aggregation you suggest doing with Flink. I don't know a good solution to this problem other than dumping the list of userIDs for each minute window into your OLAP DB. You might be able to use HLL hashes for this, but depending on how granular the dimensions are in your DB, it may not be worth it.. I think it's at least worth mentioning this if we think users care about unique counts.
Why did we not use a Redis instead of using Flink? Redis is also and in memory structure, that will meet our write throughput demand and we should scale it by sharding on Ad id?
Hi this was super helpful! My question is how would you handle the offline channels case? i.e. how would you aggregate data for one ad shown across multiple channels? I feel like the design wouldn't have to change that much because the adId could just remain the same and you can just add a "channel" metadata field for where it was shown.
Hi Great explanation for a complex topic in such a easy way.Is Once only processing also critical when we are passing messages from kafka to flink.If we want one to enable once only processing on flink then checkpointing will be required juts a though
You can set it per message! From the docs, “Every Amazon SQS queue has the default visibility timeout setting of 30 seconds. You can change this setting for the entire queue. Typically, you should set the visibility timeout to the maximum time that it takes your application to process and delete a message from the queue. When receiving messages, you can also set a special visibility timeout for the returned messages without changing the overall queue timeout.”
This is another great video!! Please keep it coming. Can we use mirroring in Kafka and have spark read from the mirror and provide data to reconciliation service?
I rewatched and had some new thoughts. Wonder what are the costs of using streaming solution? I seems like the database for clicks that was used in batching solution is completely replaced by the streaming components, so benefits from having the previous database queries are lost? 34:52 streaming solution real time is by dumping to OLAP?
Please can you clarify this? You mentioned the count query on cassandra will be really slow. Would it really be slow? If the partition key is ad_id and the sort key is timestamp. I assume all the data for the same id will be on the same partition sorted by timestamp. Why would it be slow?
The best system design content. Thanks alot for helping me to prepare for my upcoming interview s. Can you please clarify the difference between product design and system design at Meta?
When we do the idempotency, could we save the last time we see the ads in the redis for the adId and every later time, it will compare the current ads request to see if it has been a few days? So it knows that it should be a different impression id? impression id 1 and impression id 2 will be 3 days apart. Is it valid?
hi, I may missed this: is it possible to put Apache Kafka + ksqlDB to build aggregations with Materialized Windowed Tables, where you can also use flash interval? Is it acceptable for such interview?
I honestly feel you should hire @Jordan Has No Life as a system design expert on your channel. The depth of system design in his videos his quite good and honestly it makes up for a senior engineer. As what's the case with Staff SWE Expectations well that depends honestly on the individual. I think It can only come from experience or reading books such as Database Internals and/or DDIA. No amount of videos can make up for the Staff SWE expectations in System Design.
This kind of content can make someone fall in love with software engineering.
Finding your channel feels like finding gold!
There are ton of SD videos on youtube with shallow content basically exactly like you mention what a junior or mid level candidate would do.
Going indepth for senior and staff is one of the highlight of your content. Please continue doing that.
Also please don't worry about length of the video. Keep the gold coming :)
@hello_interview waiting eagerly for next video
This is by far the best system design interview ever seen on the internet. Keep doing the great work sir...
Watch this and then imagine if Evan puts together a System Design Learning Course. Just image that !!!! I mean we (the learners) will jump on it sooo quickly. This is just absolutely amazing. This is combining years of experience with hands on actual approach that works along with book contents presented in a very professional manner. Evan, think about it 🙂
Maybe one day! For now just happy with all the people learning about hello interview and getting tons of free value
Literally recommended Hello Interview to everyone I've mocked interviewed with
I have purchased so far
Alex Xu
grokking
system design by Mikhail Smarshchok
i would say i learned a LOT from the Mikhail Smarshchok as far as internals go
and then i bumped into your Hello Interview
ABSOLUTE BEST stuff that can replace Alex,grokking for me. I was ALWAYS looking for how to connect the functional requirements into the later high level design/ deep dives but both Alex, Grok fail many times in connecting the dots.
Subscribed and a big fan of you. I will consider purchasing coaching or mocks once I am through your videos and feel confident.
You somehow managed to make preparing for system design interviews really fun. Massively underrated channel
You’re the best!
This is not just interview prep, this is some serious stuff here. Thanks a lot!
🫡
Honestly, it is the best SD showcase I’ve ever seen. You are the best. I watched all your videos and whiteboard them myself then. Thank you!
So glad you like them and very smart to try them yourself and not just blindly consume!
The content is great as always. However I found this system a bit frustrating, there is so much specific knowledge one should know to really end up with a good design for this kind of problem. I wish I don't get a system like this in an interview.
Awesome walkthrough! As a junior engineer I learned a lot. Near the end you said you wanted to keep it short but I appreciate the nuances you point out for your decisions including design choices between mid and higher level candidates. Time didn't faze me at all, I watched every second!
Hell yah!
You are the best at what you do, after listening to your videos I cannot now like and tolerate other system design videos, please make more content for system design
Coming soon! 🫡
Really helpful videos - especially the breakdown for different expectations at mid/senior/staff levels, common things you see from candidates, and context into the actual systems like the shard limits for events streams. I used to work on Kinesis - happy you chose it!
How cool! That must’ve been fun to work on :)
Great job with creating this content to help us prep for interviews!
Just one thing to note, you can't send a 302 redirect for POST requests, it has to be a GET request.
I rarely leave comments but this is BY FAR the best system design video I have seen on youtube. The thing I like most is that it differentiates from common textbook solutions that we see everywhere and you explain why certain choices were made. Thanks!
Thanks for commenting!! Glad you enjoyed it
This video helped me ace the system design interview. The detailed explanations provided in-depth knowledge of various components, which was extremely helpful for answering follow-up questions during my interview.
Really good! Listened 3 times to pick up every single detail. Thanks.
Thank you so much for these videos! They’re absolute goldmines-I’m genuinely amazed at the depth and clarity they offer. I’ve learned more from these than from many of the books I’ve read. Truly invaluable!
So glad you like them!
These interview preps make you feel like if you know enough of system design knowledge, have good cross team examples for bq, and can solve leetcode medium-hard fast, you can get to higher level quicker than going through internal promotions.
One of the best channels on system design! Please keep going!
Ah, nice, you re-uploaded! Thanks a lot for taking the feedback and acting quickly on this. And, sorry if it caused inconvenience for you 😄Thanks a lot for all of your hard work. 🙏
Thanks so much for calling that out! Glad to get it fixed within the first day :)
Is e-commerce (design amazon / ebay) not as common as it once was?
For hot shard problem in Kafka, you can salt things but in Flink, we will no longer have all the events aggregated in one Flink task. In the case where we are sinking to Redshift, we could have aggregate it there. but if we want to access it in stream, maybe we need a secondary stream that aggregates everything?
So that added salt needs to be handled one way or the other
Thank you for a great video.
For a senior candidate it will be helpful, in my opinion, to narrate the data structures that underpin these solutions in addition to the supporting vendor products/technologies. In that, for fast aggregation of counters, one could demonstrate the use of fast but slightly imprecise counters using a count-min-sketch-like data structure, and for a slower but more accurate counter the use of a Map Reduce jobs. Aggregates and statistics over varying windows are almost a necessity for contemporary Monitoring, Analytics and ML Systems. And in such cases retaining data in-memory backed by persistent storage in the form of tree data structures keyed by aggregation windows are useful for range queries at varying granularities. For e.g.: root window [0-100). Immediate child node windows [0-50), [50-100) etc.
It could be helpful to talk about idempotency within queue consumers. And also out-of-sequence arrival of events in the queue (handled through watermarking)
Can have some future videos which go deeper on probabilistic data structures or other more foundational topics.
Excellent, as usual! Thanks so much 🙏 While concluding, you mentioned you want to keep the videos short. Please don't reduce the length. 1 hour is a sweet spot and it's necessary to capture all the important "juicy" tidbits and details you highlight. Please keep it coming 🙌
If you HAVE to reduce something, please reduce the time between videos to 1 week 😛
No, seriously, thanks so much.
Haha trying 😝
I discovered your channel a few days and love all the videos so far. I love the solution enforce idempotent click tracking. There are a lot of SD videos but only your SD videos provide guidelines on the expectations based on candidates' levels. So far, I watch one video per day so I will run out of videos to watch very soon xD
I felt this was much better than the Alex Xu System Design Vol 2 on the same topic. Great Job1
High praise!
you are honestly the best content on system design , can you do some playlist on the system design topics themselves ?
i mean a video where you discuss replications in depth , concurrency , etc.
Will definitely consider this!
This is the best system design video I have seen on youtube till now. Really loved the in depth discussion. Would love to see more videos. 👍
need more like this man!
Amazing explanation skill you have, OMG.
Thanks a lot for uploading these videos. They are very informative. Keep doing the good work.
This was such a pleasure to watch. Thank You. I would love to see a video on a metrics monitoring system. There will be some common components with ad-click aggregators.
Amazing, really better than other stuff I found on internet!
Kudos to you!
This is great content. However, I would like to point out that the Lambda architecture includes both batch and speed layers, which process historical and real-time data in parallel. It’s not solely reliant on batch processing.
Thanks a lot ! This video is super helpful. It not only helped me understand the key components of ad click aggregator, but more importantly it taught me what does the interviewer expect for different level SDEs. One question --- do we need to dive into the aggregation logic details in Flink?
The final solution _is_ literally lambda architecture.
Great video! I learnt a lot! One question I have is regarding the hot shard handling. When the click processor detects there is a hot shard and decides to change the key from AdId to AdId:n, how would it let Flink know that it now needs to change the aggregation logic (and sharding logic) for that particular AdId? (I believe this would also have a race condition when it changes within a minute window, but any data integrity issue that arise from it should get resolved by the batch job)
Stellar video. Only suggestion for the next ones: your drawing tool seem to have keyboard shortcuts for those diagrams and other options on the toolbar. It'll greatly improve your quality of life! Keep it up!!
Awesome content!! I was hoping to get more clarity on how same flink node determines the partitions it need to consume from in case of hot shard issue which we are effectively handling by adding a number b/w 0-n to the partition key which leads to same adId clicks published to multiple partitions in the stream.
these are so good , such deep dive and so clear ! Thanks because they cover so many different aspects . i am looking forward to many more videos . I am currently preparing for interviews and these are so helpful ..
Very useful video and its best among others. I got this same question twice in my loop interview.very happy how I answered
Nice. Good job!
Thank you so much for the explanation. However I more like you write the functional and non functional requirements instead of paste from the clipboard. The main reason is that it's make me thinking about what should be the requirements. It's feels more like code pairing
Thanks for the content; It's great; One query on
1. As we compute 10sec and write it to OLAP DBs ; The deep dive on how the user queries for 1min window, 1hr window, may be 1 day and all time window would be great info;
2. And what could be that OLAP database choice would be and what factors to consider.
3. Another aspect is; Freshness of the data; What does each component in the system contributes to the latency to make the event available for aggregates;
Other aspects are great; thanks
Thanks for the video, it is really in-depth and informative. You covered the flink + streaming part really well and why not to use checkpointing was really a good point. Saw some videos in the past that mentioned let's do checkpointing and never explained why it will work or not work.
For the streaming components both streams (kinesis or kafka) and stream processor (Flink/kafka-streams) I got following questions ( (I was asked similar questions in an interview.)
1. we have said that we will partition the shards or streams using adId but we have 10M ads at any given point of time. In this case, if I consider other things like replicas in kafka topic or even kinesis shards, don't we have to create lots of shards?
2. The max traffic is 10k cps so most of the ads are not clicked and won't have active traffic so most of the shards will be empty, in this case what should be the approach?
3. If we group some ads based on let's say advertiserId than we introduce the noisy neighbour problem where one ad generates lots of traffic and blocks the other ads.
Those questions are great and definitely come up often with streaming setups. I'm no expert but ...
Partitioning with 10M Ads ... Yeah, if we just went with adId for partitioning, it would mean loads of shards or partitions, which isn’t practical. Instead, you can use consistent hashing or modulo partitioning to map multiple adIds to a smaller, manageable number of partitions.
I like your videos, I have learned a lot.
A couple comments on this video:
a. I think the system would benefit from a Redis in the click processor service, not the idempotency lock but a redirectUrl cache {adId: redirectUrl} to reduce reads in the Ad DB. It might be a MRU cache to avoid overloading the Redis.
b. I'm not sure why you are pushing Kinesis so hard in this solution, I mean yes I learned something about Kinesis, but it would be more practical just to place a Kafka that can handle the load peaks and has event history as well so it is possible to write the reconciliation procedures from it.
c. I learned about Flink, thanks. I used a redis aggregator in my own solution.
Thank you so much for your work!
Hey Evan, your videos have been the best, seriously! The only suggestion I have is if you could use a dedicated mic that would be wonderful. The volume is too low even on max volume on my earphones.
Updates in latest video! Have a nice mix now :)
@hello_interview yay 😀
Beautiful Design and Amazing explanation - just impressed with the elegance of the design and the beauty of software engineering.
😍
Looking for your next videos. Pls upload more design problems. It almost 1month you have not uploaded. Love your content.
Sorry, was traveling. Recording a video today! Up by EOW
Thank you for such a fantastic video! I have a quick question regarding the deduplication process. When using Redis to deduplicate based on impression IDs, would the entire operation of "check Redis -> populate Redis -> submit click to event stream" be atomic?
If it's not atomic, there could be cases where Redis is updated after a click arrives, but before the click is submitted to the event stream, and if the server crashes, this could cause an issue.
Even if we change the order to "check Redis -> submit click to event stream -> populate Redis," there's still a chance that two impressions could arrive at the same time, find no record in Redis, and both submit the same click to the event stream.
Could you please clarify how this is typically handled? Thank you!
Question: For the sharding while processing the events through Kinesis, the adId was suggested as the sharding key. This doesn't look like the best approach. At scale, millions of ads are being run on the platform and a good share of them have high enough volume. Going by the presented logic, the number of shards would explode. What do you think about this?
Love these! And can't recommend the Hello Interview mock interviews enough!
Wahoo thanks Ben!
You are a legend man. Make some more videos which are mentioned on your websites. Search, E-commerce , Hotel Booking system etc.
Thanks so much for doing this! Greatly appreciated! By far the best system design videos I've seen.
Hi Evan, Absolutely gold content. I have one doubt here.
For a really hot ad, you are adding some random no (1-n) to build the partition key before adding it to the kinesis stream. Now this particular ad can land into multiple spark consumers. How the different spark consumers will aggregate this data for this ad. Is there anything that i am missing?
Are you referring to keyBy(adId) in Flink?
Hey, I love these videos. I only used your videos and designing data intensive applications and that was enough for an E4 offer at Meta, I love the advice you give and common pitfalls you provide.
Crushed it. Congrats on your offer!
Amazing system design. I've been searching for something exactly like that, which is interview driven, not "let me show you all the ways you can do it" driven.
May I just make a small suggestion: please use "etcetera", not "eccetera"...
Few questions:
- Even in spark solution, how would you know the keys of which one to aggregate? Either you have to emit keys of which has changed or scan the db by time and get keys which is again costly.
- Aren't this falls under lambda architecture as we are using both realtime stream processing and batch processing to ensure data integrity?
Question: Why do we need the reconciliation mechanism? And why use a batch processing for the purpose? Do we know/assume stream processing would accumulate errors compared to batch processing (the latter being accurate?)?
Amazing content! Very much appreciate you posting these 🙌
System design padawan here. I have a question about the hybrid approach .. what makes us trust the "Kinesis -> Connector -> S3 -> Spark -> Worker -> OLAP" results more than the "Kinesis -> Flink -> OLAP" results? Is it a guarantee where the connector always successfully writes the data to S3? or does Flink make some kind of tradeoff for speed? kind of confused about that piece and figured i'd ask. thanks again!
I am also curious about this
+1 on this question.
IMO, Spark is useful when you have really out-of-order events, like events arriving half an hour late or something. Then, by using Spark, you can reorder the events and get a more accurate count. On the other hand, for events that are only a few minutes late, you can configure Flink with the allowed lateness of a few minutes.
That being said, the cost of maintaining 2 code bases and ensuring that they are updated at the same time (to avoid triggering discrepancy alerts) doesn't seem worth it for such edge cases.
I'd be interested to hear @hello_interview's thoughts on this though.
> Is it a guarantee where the connector always successfully writes the data to S3?
Yes. That is something provided by AWS as a managed service, and it should meet their SLAs. AWS would've created fallbacks and fault tolerance to ensure that all events that are in Kinesis reach S3.
I don't see there being any compromise due to speed. Kinesis has retention policy, so the data isn't going anywhere, and S3 is highly available, so the data shall be written there as well.
this one had a diff approach i dint see b4. nice design.
I love this channel. Very good job sir, your strategy is really good a comprehensive. Straight to the main points. Bravo
Great video, covers so many new topics. Is the use of a time-series database practical for storing click events by timestamp? By design, that would help with aggregation in 1 minute windows, no?
Why this is not lambda architecture? It has both realtime and batch processing, so what is the difference?
Thank you, btw
We initially said that we want to perform complex queries. In addition to time range, we want to check who were the users, and maybe where the clicks were from, etc. So that means we need more types of aggregations, for example by user id, by location, etc. How do we handle that? What if we get a query that we didn't prepare an aggregation for in advance?
Thanks for the detailed explanation! Definitely learned some new things in this video.
The final design has both real time data processing and batched processing. Why is it not lambda architecture?
Hi Evan, Thanks for the great video. I have a query. Can we use time series database here?
I think bloom filter would be a good choice to check on duplicate impression id. I think, it is also supported by redis.
Love the content! Watched so many videos and they have really helped me in interviews.
I had one questions. Instead of dumbing data from Kinesis to S3 to be read by spark. Why not just have spark batch processing read right off of Kinesis? If you are doing reconciliation every hour, day, or whatever. If is is smaller than the retention policy on Kinesis, I thought you could just do it from there. What am I missing?
I know you called out the issue of contention as a reason for having two db in the beginning.
What makes the OLAP db different so that we don't need to worry about that for read and write queries?
I believe the checkpoint is still necessary so it avoids data loss. If the server crashed, the checkpoint stores what the offset in kafka/kinesis stream is at and restart from the offset in the checkpoint.
Without the checkpoint, flink has no idea where it should be restart from and the data will be lost between the last time it send data to the OLAP and the the it crashed.
absolutely brilliant content mate. keep em coming. only channel for which I have a notification on.
For Kinesis hot shards, we don't know if an ad is hot beforehand. So are these ad_id 0-N always active? Is it ok to use x10 the amount of streams we need under normal circumstances?
For Flask, we have the same amount of flask servers as the Kinesis shards right? If the server dies, how will the new server keep track of the pointer from the old server? Are they statefull backups instead of stateless
This is a great question. In reality you can make predictions here. We know based on budget and historical performance which ads we’d need to be worried about before hand
Thank you for the video! Isn't this exactly what Lambda architecture is and not a "hybrid between lambda and kappa"?
Simply the best resource !!!!
Thank you for sharing!
I got a question about the decision to not use checkpointing in Flink. If you don’t enable that then where would you store the Kafka consumer offsets for recovering?
I saw some videos and your content is so great. Thank you so much for clarifying the SQL vs NoSQL debate. I always thought that bringing that into an interview was irrelevant but was afraid to do it. 😅
Keep up the amazing work.
Yah funny how that was evangelized in a couple books and then just stuck
This is a very high quality video.
This is awesome. Thanks Evan. I have a question on the usage of blob storage. Aren't those supposed to be used for storing unstructured data like images, audio & video files? Could you please elaborate on 1) how Spark task reads data from S3 2) how Spark job would sustain reading one day's data? Is checkpointing used in this context?
Can I use this aggregator flow for designing the top-K RUclips videos system? What are the major differences except ranking?
Thanks for the great videos - they are extremely helpful.
I noticed at around 24 mins in you mention querying for the number of distinct userIDs. I don't think you're going to be able to serve queries like that using the pre-aggregation you suggest doing with Flink. I don't know a good solution to this problem other than dumping the list of userIDs for each minute window into your OLAP DB. You might be able to use HLL hashes for this, but depending on how granular the dimensions are in your DB, it may not be worth it..
I think it's at least worth mentioning this if we think users care about unique counts.
Incredible video with excellent drawing and explanation.
Why did we not use a Redis instead of using Flink? Redis is also and in memory structure, that will meet our write throughput demand and we should scale it by sharding on Ad id?
Hi this was super helpful! My question is how would you handle the offline channels case? i.e. how would you aggregate data for one ad shown across multiple channels? I feel like the design wouldn't have to change that much because the adId could just remain the same and you can just add a "channel" metadata field for where it was shown.
Hi Great explanation for a complex topic in such a easy way.Is Once only processing also critical when we are passing messages from kafka to flink.If we want one to enable once only processing on flink then checkpointing will be required juts a though
You can set it per message! From the docs, “Every Amazon SQS queue has the default visibility timeout setting of 30 seconds. You can change this setting for the entire queue. Typically, you should set the visibility timeout to the maximum time that it takes your application to process and delete a message from the queue. When receiving messages, you can also set a special visibility timeout for the returned messages without changing the overall queue timeout.”
when will you post the next interview video? waiting for it about 1 month!!! really appreciate the effort.
Tomorrow!!
This is another great video!! Please keep it coming. Can we use mirroring in Kafka and have spark read from the mirror and provide data to reconciliation service?
you know i have less familiarity there. potentially, but not totally sure.
21:48 I like how DB can be used for simplest case consistently in these approaches
Why do we use nosql for the write heavy db? Since the data is very structured and you might want complex queries…
Why do we have to rely on Kinesis retention policy, when Flink ensures fault tolerance using check-pointing?
I rewatched and had some new thoughts. Wonder what are the costs of using streaming solution? I seems like the database for clicks that was used in batching solution is completely replaced by the streaming components, so benefits from having the previous database queries are lost?
34:52 streaming solution real time is by dumping to OLAP?
Please can you clarify this? You mentioned the count query on cassandra will be really slow. Would it really be slow? If the partition key is ad_id and the sort key is timestamp. I assume all the data for the same id will be on the same partition sorted by timestamp. Why would it be slow?
The best The best.. Loved it. Thanks for doing this. ❤
My pleasure 😊
The best system design content.
Thanks alot for helping me to prepare for my upcoming interview s.
Can you please clarify the difference between product design and system design at Meta?
www.hellointerview.com/blog/meta-system-vs-product-design :)
How can you identify if interviewer is asking Product vs Infrastructure system design question?
When we do the idempotency, could we save the last time we see the ads in the redis for the adId and every later time, it will compare the current ads request to see if it has been a few days? So it knows that it should be a different impression id? impression id 1 and impression id 2 will be 3 days apart. Is it valid?
If we use Cassandra with adId as primary key and timeStamp as the sort key, will the read be fast enough?
Not to aggregate over large periods
hi, I may missed this: is it possible to put Apache Kafka + ksqlDB to build aggregations with Materialized Windowed Tables, where you can also use flash interval? Is it acceptable for such interview?
Love the content! Thank you for making these!
I am learning so much, my god
I honestly feel you should hire @Jordan Has No Life as a system design expert on your channel. The depth of system design in his videos his quite good and honestly it makes up for a senior engineer. As what's the case with Staff SWE Expectations well that depends honestly on the individual. I think It can only come from experience or reading books such as Database Internals and/or DDIA. No amount of videos can make up for the Staff SWE expectations in System Design.
We love Jordan ♥️
@@hello_interview Me Too!! That guy's an OG in System Design.