Видео

all your videos are awesome, This one is little fast run or complicated or felt something is broken . Would be great if you can revise after 1 yr. something you might have gotten more info during this period, obviously NO LIFE>

this would have been really useful last night

Excellent video as always Some insights: In most cases single leader replication and leader only read is sufficient. For example, Kafka by default uses single leader replication and leader read, and despite this it offers fantastic throughput. Tests have shown that Kafka can offer up to 70k reads / second, or about 4 million reads per minute! So, in most cases, the benefits of strong consistency from single leader setups will outweigh the read performance advantage of replica reads. I can imagine unless you are dealing with software like Discord or WhatsApp, in most other cases just go with single leader setup. This is something that can be clarified during interviews, to show the interviewer that you know your stuff, but also have the technical sense to simplify design by favoring strong consistency.

Jordan with no life is out here saving lives, and had to do it while looking awesome because the ladies like it that way. Thanks Champ.

Is there a reason not to use Redis Cluster and rely on it doing the sharding and replicating?

@~14:27 Jordan starts talking about the need to do range queries and the need for predicate locks, hash indexes and whether my interviewer will let me get away with that. Instead she may ask for an on-disk database. This all moved a little fast. Could someone please explain this a little bit more?

at ~9:30 you say that since we are storing ranges on each db, and in case of each collision we can just probe for the next n + 1 hash on the same partition, "this in theory should increase latency". Didn't you meant to say "...should improve latency"?

Unmatched content brother! Bless you. Keep enlightening those interested. 🙏🏾

Awesome video Jordan. So much to learn here. God knowns how the Route Calculation is done in India. Here the transition probability is 100% between any two continuous locations. 😂

When you mention that the reader schema will ignore net worth, because its not on the reader schema, do you mean like ignore the valus/ not fetch or like dont care what the values would be? What i mean is, if the value is ignored, then all the data of server A will not be fetched right? We will only get the data which are common between server A and B?

one thing i find interesting is that this uber matching problem is also bounded by how fast rider/driver can physically move. for example rider is likely by foot so if location update freq is , say, every 60 second, that's at most 500 ft for each update and it's very unlikely this person can walk from one partition to another, same with driver, if drives at 100mph(and not caught by cops), that's at most 10 miles moved within 5min. i understand this is an extreme example but i also don't think time square would ever warrant it's own partition, the reason being it's still bounded by physics, see this silly example: Scenario: 100k people calling ride at the same time and matched with 100k drivers. Why it's not true because 1. you can't possibly fit 100k cars in that space, so you have to search within a bigger geo region anyway, and 2. you can't have 100k people calling ride simultaneously because times square is small, you can only fit like 150k people max there. So it would be funny if all of them decide to take out their phone and decide to call a ride at the same time. Unless we humans can transcend physics, i feel lucky for those SREs at Uber since their server load is bound by physics haha

Incredible series! By far the best on youtube. Looking forward to going through the interview questions now.

Hi Jordan, I feel using Write ahead log for ordering transactions can cause issues since only writes (and not reads) would reach the write ahead log. I have added a broader list of questions around this in my other comment on this same video: ruclips.net/video/Tgpa9TrxsfU/видео.html&lc=UgxcOA38-dmUMdvCKQF4AaABAg&si=RHwLnmnEiYSPlBJj Please go through it when you get some time and let me know your thoughts. Thank you!

ISO8583 would be neat to mention at the point you mention Thrift and Protobufs. Very similar approach.

JSON : and it uses the ',' to delimit fields which could occur inside of a lot of data, but uses colon to delimit between key and value; and the type of collection is hard to distinguish between; etc.

that is soo detailed, i had to take a break from learning about it for a few mins!!!

Can someone tell me all of his Collabs? Only know the one with Neetcode and that one was hilarious

Great rundown, thanks

leader election: if the candidate node is more upto date than local log, and its not voted for any other candidate, it votes for the candidate - > shouldn;t it be if the candidate log is NOT as upto date as local log, it will vote for the candidate? if the logs are the same point, it should still vote for the candidate right?

one thing to add is to pre-compute a subset of popular time ranges, I''m sure 90% of the folks would just be reading the billboard of yesterday/last week/YTD/etc. So these should take advantage of the power of batch compute + cache another thing is I doubt many would query with granularity of hour, the older the time ranges the less they care about granularity. So from UX POV we should restrict users to query by hour/minutes if it's over, say, 7 days. This way, we can move the columnar stored data into an OLAP db such as Pinot, which can make exact query a lot faster than say running ad-hoc Spark jobs against Parquet files or Iceberg tables.

Jordan gives so many hearts to comments, we're starting to wonder if his fictional gf is feeling a bit of jealousy. Send me one too bro

Would there be a queue between the streamer and encoder?

Hey how are you? Great video! You explained everything pretty easily. One confusion I have htough: You mentioned to store each rows in LSM tree to overcome the downsides of column oriented storage... Here, do we store the values each column into a different LSM tree?( Different LSM tree for different columns)

I'm glad those sauce bottles behind you are all still full.

No offense to them, but I'm at rewriting their VDL definition as such, "The DB can further constrain LSNs which are free from truncation by tagging some as consistency point LSNs or CPLs. The Volume Durable Point LSN will be the CPL that is the less than or equal to the VCL. (CPLs constituting complete transactions it seems)."

We are migrating some applications to Spanner at work. This gave me a much better understanding of why it's able to do what it does. Thank you!

Great video. Small correction at the 10:20. You said leaderless replication improves write throughput. Actually: -Leaderless replication trades write performance and consistency for improved availability. -Multileader replication trades consistency for increased write performance and availability. -Single leader replication trades write performance and availability for improved consistency This is because in leaderless replication, every node in a quorum is read and written from. This is like single leader replication since all servers are being hit for each request, so doesn't increase write throughput by letting different read/writes hit different servers. Often times, deployments will mix and match leaderless replication with multileader replication. In this case, you have different quorums for different sections of your application - you hinted upon this in sloppy quorum (profile quorum is different from message quorum)

So, you're saying master leases increase read throughput by always having a master to read from even though a prior master has been separated from the network?

Hey jordan, in which case we used region wise sharding? like the country specific requests should go to the local servers right? and we might be sharding dbs on region basis

Hey thank you for the video, really appreciate it. Just the audio was bit out of sync from barrier injection part ...

This channel might very well be the best system design channel on RUclips today. Honored to be learning from you Jordan!

hey jordan! first of all you rock! and you explain things better than literally any teacher I ever had.. but I have a question - you didn't dive into the followers-following mechanism at all, now that you moved from SQL to Cassandra and there's no many-to-many table anymore, where is that data stored? in lists under each user? there's a table for every user with all his followers?

Materializing conflicts is not something which the Database does by itself, right? It would be something which the person deciding the schema and building the application should think about. Something like "Looks like I will run into phantoms here. Let me add default entries for all possible values so that it would be easier to acquire locks"

Thanks for this vid on Hadoop. I've been wondering about it.

In snapshot isolation, while reading the data, the transaction should know its transaction number, right? Otherwise it wouldn't know which snapshot to read. For each row, it should read data updated by the latest preceding transaction which updated that row. You've said that for a single node DB, Write Ahead Log can be used to order transactions. So whichever transaction hits the Write Ahead Log gets the earlier transaction number. But what if reads happen before a write in a transaction? How would the reads know their transaction number if the transaction hasn't hit the WAL yet? Do they just kind of query the WAL at the beginning of their execution to get the latest transaction number? Also, I checked with Google Gemini that Snapshot Isolation is the minimum level which prevents lost updates. I believe this can only happen if while writing the updated data to the counter, the DB knows which snapshot of the data was read by this transaction. This way it will abort one of the two transactions trying to make an incorrect update. How does the DB know which snapshot was read by the transaction making this write? I think this will again be possible if any transaction gets the latest transaction entry in the WAL while starting its execution and then adds itself as the latest entry in the WAL irrespective of whether it has a read or write operation at the beginning of its own transaction. But in that case should it still be called a Write Ahead Log? It would be something like a Transaction log.

I ddint understand the db scehma. How does chat id helsp to find other user in non group chat session. A chat need atleast 2 users

Важно ли нам вообще посчитать ТОЧНОЕ число пользователей, а не +-100 если у нас всё такое высоконагружееное, что пользователей миллионы?

i think I've seen this content in DDIA chapter 2. Maybe good to put a reference link.

Hey, thank you for the video. Very nicely explained.Out of curiosity which podcast you referred at the start? And do you have any tech podcasts recommendation? Thank you in advance

Really appreciate Jordan for bringing this on youtube. I have some fundamental questions around generating unique shortURLs. Why not just generate random numbers vs hashing the long url ? Are there any security/performance implications ?

What is the podcast you mentioned?

Thank you Jordan. Vector embeddings is done using it's own database optimised for this case usually or maybe stored as a json or list for product? I think I got the tech lead job here in Thailand that I was aiming for. (apparently I'm an interview rockstar from the feedback) and I think it's partly because of binge watching your videos. I'm totally here for the tech of course😅

The next great distributed DB : "Wal Eye!"

can anyone enlighten me about the last example ? If we real all values from snapshot 15 and before, wouldn't the total value be different than 1 Million, and we will have an inconsistent read ?

Good video. Maybe they could also add a heartbeat between shards / systems to get a hop length and use that as the margin of error eliminating the dependency on clock accuracy. They could also watch for drift in that margin and do qc on it. Or do they already do all that.

thank you for great video solution: I am just trying to practice math here. But the storage requirements come out to be 10 PB. cross checking with ChatGPT "If you calculate 1 billion × 100 × 100 KB, it equals 10,000,000,000,000 KB. To convert this into TB, divide by 1,000,000,000 (since 1 TB = 1 billion KB), which gives 10,000 TB. Further converting to PB, divide by 1,000 (since 1 PB = 1,000 TB), resulting in 10 PB. So the final answer is 10,000 TB or 10 PB."

What podcast if you don't mind me asking? just curious

Great illustration. You get me interested into architecture. Very underrated channel.

Regarding the paper’s sparsity of detail on generating embeddings, if you’re interested in going down the RecSys rabbit hole, there are some famous models for multi-stage ranking/recommendation. i.e. Two Towers Neural Networks, Deep Factorization Machines, Deep and Cross Networks (DCN V2), and Deep Hierarchical Ensemble Network (DHEN)

That opening comment is crazy