Видео

I was thinking about the recently Apache Fury vs protobuf

@9:30 you said if we want to insert at x and x is occupied we do x+1 . What of x+1 is out of range will we move to next partition ?

for hot assets like AAPL, GOOG, do they have dedicated pricing servers? for example, one layer-1 AAPL pricing server directly connect to publisher, multiple layer-2 AAPL pricing server connected to the layer-1 AAPL pricing server, user server will connect any one of the layer-2 pricing server

Right wrist and no tissues... Due to apache arrow?

Thanks for suffering through your sore throat to deliver this great lesson to us

how easy is to register a spark consumer to zookeeper? Because spark consumers are configured by and are managed by spark driver program. Does spark controller provide this info via any API? And even if it provides, since the executors number is dynamic, how will flink manage the merge sort of list?

I think that you can't control which HDFS node to write the top K list. The distribution is handled by HDFS and all that is visible to you is a directory structure. Same directory may be distributed across different data nodes.

Hi Jordan, Thanks for the amazing video! I had a few questions around the posts DB. 1) As we are using Cassandra here which is a leaderless replication DB.Lets say a user uploads a post, and the user immediately wants to update the post. Because we are using a leaderless replication, its possible that the user may/may-not be able to read their own writes here. Does it make sense to have some kind of Write-back caches: which can provide read-your-writes consistency so that a user can update a very recently uploaded post and then flush these writes/updates to Cassandra (assuming most of the users only update a new posts within 5-10 minutes from the time of the upload)? 2) Follow up to the first question, if we have a write-back cache (assuming we use a distributed cache) and if the cache goes down which would lead to the posts getting dropped as they were not committed to the DB. In this case would a Write Ahead Log with cache help us make this more fault tolerant?

I know it’s 1 year old and probably you know this already, but DBs like Postgres use MVCC, so you don’t need to lock rows that you update in your transaction.

correct me if I'm wrong but we don't need serializable transactions for booking one seat itself(i am not talking about claiming e.t.c). if we have a lock. Let's suppose that you acquire a distributed lock when you try to create/update the booking. Even if you decide to have a separate payment entity, most probably you don't need to update the balance, only create it with a reference to the booking id enough. You are gonna use a 3rd party vendor to send the data, so if the data is successfully transferred you just commit the transaction. Atomicity is provided by default. There are no 2 users that can have a lock at the same time (provided by consensus algorithm), and you just release the lock when a transaction is committed.

At first I thought you rated Meghan _Trainor_ a 10 and I almost threw up

Arrow Flight is a bit like cpnproto or flatbuffers

Hey Jordan! I love your simplified explanation of Advanced concepts. Kudos to your oratory skills!!!! It would be great if you can make a video on Elastic block storage(AWS) or Persistent Disk(GCP). Much Appreciated!!!!!

Hey Jordan! I love your simplified explanation of Advanced concepts. Kudos to your oratory skills!!!! It would be great if you can make a video on Elastic block storage(AWS) or Persistent Disk(GCP). Much Appreciated!!!!!

Dude I loooove arrow and what it has to offer for future of open source data engineering, so glad you also did a video on it!

lets goo

Can we partition in single node? Like create partition in mysql. Whats the benefit like just going directly to the Hard drive section where the data is? Because I assume the db files are already sized limited and in the B tree we know where they are

0:40 wierd correlation indeed 😂

Hey Jordan! This is super helpful. Thanks so much! Quick question about ensuring jobs only run once - the retry logic via the scheduling node a la timestamps is intentional right? As in, we intend to run the same job more than once so was a bit confused by that concern at around 23:05. Even if an executor goes down, we want to retry that job right? I guess I was bit confused why "running jobs once" is even a concern as that is expected behavior.

Didn't know Kylo Ren do tech stuffs

video so good, liked in the first 1 minute

Hey Jordan this is awesome content and you're a great teacher. Listening to it in the gym from India and I can follow everything.

Hey Jordan, thanks for the amazing system design video as usual !! I have one doubt on usage of Flink. Whenever atleast one flink compute node goes down or restarts, then the flink job fails and it has to restore the entire state across all the nodes from S3. So, this whole restoration process can take few minutes. So, our message delievery will be delayed for that many minutes, affecting the entire user base. Is that understanding correct ?

I had a hard time reading alex petrov's database internals book, so I quit after few days. But this is informative and engaging at same time. Looking forward to entire series.

Chapters please 🙏

One question regarding Fan out approach. While pushing posts to each followers, we are pushing that to a particular News Feed Caches corresponding to that user. My doubt is whether these news feed caches are just an another caching layer sharded on user id(let's say 10 caching servers sharded on userid for 100 users), or they specific to the user(100 users 100 caches in that case)?

Hi Jordan, just wondering if the mutual connection databases in 14:14 are the same as the mutual cache table?

Jordan sat alone in his dimly lit room, eyes fixed on the screen. Kafka Streams flowed before him like a slow, seductive dance. His fingers moved over the keyboard, sending commands that made the data bend to his will-smooth, precise, and totally in his control. “Processing this much data feels like... processing my love life,” he chuckled, leaning in closer. “A little messy at first, but once I get my hands on it, everything falls into place... perfectly.” The logs rolled in real time, a rhythm that matched his heartbeat. Who needed real life when the streams responded to him this way? Controlled. Obedient. Alive. Jordan didn’t just process data-he made it swoon.

cool

Hey Jordan, do you have any course on Udemy ? If not highly recommend u do that

great

Thank you Jordan for this series. In the paper when comparing b/w 3 partitions approaches its written that for fixed size 3rd strategy, the membership information stored at each node is reduced by three orders of magnitude. Whereas in previous paras, its mentioned that 3rd strategy stores not only no of tokens (servers) hash as in 1st but also which partition info stored on each node. Isn’t that contradictory info or am misunderstanding something. Ideally third partition scheme should contain more membership information per node. Assuming that they have not changed request forwarding from O(1) hops to logn dht like chord or pastry like routing whereas each node stores only limited number of nodes information sacrificing direct hops.

Hey Jordan, I just wanted to take a moment to express my deep gratitude. I recently received 5 offers from tier-1 companies, including an offer from Google for a Staff Engineer role. Your videos have been an absolute game-changer for me throughout this journey. I can't thank you enough for the insights and guidance you've shared-it's made a world of difference. Please keep up the amazing work, you're truly making an impact!

some other videos talked about recommendation engine from ML aspect, content filtering, collaborative filtering. What is the relationship between them and this embedding approach

besides vector db, which database should we choose for other data including entity history db, neighbor index. Can we use Cassandra because for history db because it is write heavy and append only. KV store for neighbor index.

11:00 good summary Can use hash index when you want fast reads and writes but you can’t do efficient range queries. Hash index is also kept in memory not on disk

Thank You!

Thank you! The videos and the discussions in Comments make this channel the best source for system design.

14:02 isn't it the same UUId and different timestamp for replacing the message?

another comment on this awesome video after rewatching it bunch of times. so to conclude, how did u say we achieve idempotency since there seems to be no best option?

Thank you!

Assigning N seats in a section to a group of k people each requesting {s_1, ..., s_k} seats is a Bin-packing problem. Simply allocating seats in FIFO order may lead to unfairness and sub-optimal allocation (people at the end of the list and requesting higher number of seats will be most impacted; and we want to sell as many tickets as possible). Although, it's not practically possible to solve Bin-packing in real-time, a simple optimization would be to maintain the sum of total seats requested by people for every section. In real-time, while the bookings are on-going, that sum must not exceed N. The actual seat allocation may be done offline e.g., seats can be finalized after the booking closes, and the users can be sent their tickets via mail/phone [AWS SNS] with actual seat numbers.

Jordan would HBase be a better alternative to MySQL here? You get write ahead log style indexing which can then be leveraged to build our heap as well and expire elements as they are consumed.

Thank you!

If we take the example of creating a search on top of chats and if we partition it at chat_id wont that lead to an uneven distribution of data? Given elastic search has these shards and it tries to distribute the data evenly across all the shards but if we explicitly route our data to a specific shard (using chat_id in our example) it can lead to uneven distribution of data across shards where one chat might have active and other might be dormant. Just thinking out loud how we would solve for this :P (Probably distribute it evenly by using some composite key but that would defy the purpose to just search chats from one partition)

Why do you hashing on the user id and just store the session state in a separate storage like redis ? Sticky session will not guarantee the even distribution of traffic and it might be a bottleneck for very high amount of traffic

All those channels are more redundant than replicated database !! True that 😂

Thank you for the series, finally finished it! You're a literal chad, I can only hope to grow up to be like you one day.

Hi Jordan - In order service - the second Flink which is sharded by product ID --how does it know that it has received all the products of the cart before sending email to the user ?

great video. thanks for the incredible contribution. 2 questions 1) how do you notify users who aren’t online about unpopular posts. i would assume you need some sort of user specific queue after you fanout. the notifications table is sharded by topic id so it possibly won’t be queried directly for unpopular posts. 2) how do you notify users who are online about popular posts since i don’t see it connected to the web socket flow. i assume you expect them to poll periodically to see if they have any popular posts?