What is CAP Theorem? Degrees of CAP theorem, use cases | System Design Tutorials | Lecture 20 | 2020
HTML-код
- Опубликовано: 13 июл 2024
- Errata: The middleware in the nodes diagram actually is network. Please read it network and not middleware.
Also, quite an important detail between 16:00 to 17:30. I promise it's not fake engagement, just want to highlight it since I couldn't break the video skip ahead tabs.
Useful links:
dzone.com/articles/understand...
mwhittaker.github.io/blog/an_...
www.infoq.com/articles/cap-tw...
groups.csail.mit.edu/tds/paper...
blog.softwaremill.com/eric-br...
This is the twentieth video in the series of System Design Primer Course. We talk about one more important component of System Design: CAP Theorem.
We want software engineers and aspiring software engineers to develop basics and get ready for the world of interviews as well as excelling as a Software Engineer.
Reference: ksat.me/a-plain-english-introd...
------------------------------------------------------------------
Recommendations
------------------------------------------------------------------
Our full courses on youtube:
✒ System Design Primer Course: • System Design Primer C...
✒ REST APIs made easy: • REST APIs MADE EASY
Some paid courses that we recommend:
✒Educative.io: bit.ly/3qnW5ku
✒Interviewready.io: get.interviewready.io/ (Use coupon code SUDOCODE for extra discount)
------------------------------------------------------------------
About Us
------------------------------------------------------------------
Created and Instructed by:
Yogita Sharma
✒ LinkedIn - / yogita-sharma-83400b55
✒ Instagram - / sudo.code1
✒ Facebook - / sudo.code
✒ Medium - / yogita088
Post-production(editing, thumbnail etc) managed by:
CiKi
✒ Website: www.ciki.co.in
✒ LinkedIn: / 74735937
Colors and design by:
Naini Todi
✒ LinkedIn - / nainitodi
Both Arpit and Yogita are software engineers and want to help other software engineers become better by providing high quality and well researched content by adding their creativity and teaching twist.
------------------------------------------------------------------
Join Us
------------------------------------------------------------------
Hangout with sudoCode:
✒Discord Server: / discord
For business:
✒Email: sudocode.yogita@gmail.com
------------------------------------------------------------------------------------------------------------------------------------
Timestamps:
0:00 - Intro
0:58 - C, A and P in distributed systems
4:43 - What is CAP theorem ?
6:21 - Disproving CAP theorem ?
9:53 - Degrees of Consistency and availability
13:25 - What to sacrifice: C, A or P ?
14:30 - What have we learnt so far ?
15:35 - Tweaking partition tolerance
Very well explained with examples. Easy to understand these concepts. One of the best content on System Design i have came across. As a beginner in System Design it really helping me to connect with day to day challenges in software system.
One important thing here to avoid confusion - the CAP theorem is stated for Distributed Data Stores.
Ant why we can't sacrifice 'P' because no distributed system is safe from network failures, so we need a way to ensure the system continues to operate i.e the system has to be Partition tolerant
You are doing great help to job seekers. Keep up the good work!
you rock !!! I am looking for Architect roles and as a Lead/Developer was not aware about Architect.You gave me confidence..Thanks a lot !!!
Great explanation of CAP. Made it so easy to understand.
Best description of difference between consistency and availability. I can not thank u enough Yogita
Thanks a lot. Concept is very well explained.
This is very useful video, you have the explanation in very detail
Thank you so much
Wonderful video, really enjoyed it and understood well
This is the best, most thorough explanation I have seen. Thank you!
This is really the best explanation on CAP theorem!
Today i understood CAP 😌
Clear explaination with real world examples
Thanks for your efforts to making things simpler
loved this video
Clean and Concise. Also, the Links are helpful too!
Thanks for describing it so nicely
wonderful explanation !
awesome explanation in both videos.
Some quality content right there. Great job!
Girl - great video, super clear and concise!!!
Thanks girl 👧
Awesome 😍
you are very knowledgeable.
Thank you very much
Although i m pretty new to CAP theorem, but the introduction given during the first few minutes or so by you, goes to show simply how greatly you know about it. As i have been reading about it of late & i was like ok, this is another debatable topic, as ppl are fighting over getting their concepts rightly understood. But i think this video nails it...
Thanks a lot Akash. Much appreciated. All the best for your system design prep 🙂
Nicely explained 😀
thanks for explaining
By far the best explanation on cap theorem available.
While doing a project on distributed systems I came across many explanations of CAP theorem, none of them explained it as effectively as you did.
Thanks Mrityunjay. We appreciate your letting us know of our quality very much ☺️
It's really clear explanation!
Glad you think so!
hello sudoCode, can you explain how distributed transaction works and 2PC and 3PC in terms of Distributed Transactions ?
Respect!
Thank you :)
Also, it is really good to see mam that you provide links to references
Glad you find it useful Akash!
Good Explanation!! Thanks
Glad it was helpful!
nice and informative
Appears, you started just post 7.51..... thanks for educating
Thank you for the explanation! Would be nice to have more graphics displayed like in the beginning of the video vs the white board. The markers seemed dried out and hard to take notes due to rewinding and squinting eyes to see what values you're updating.
You are wonderful
In morning, I saw your CAP video not the theorem ;) , and thought that when the next video will come. and see here I am watching your second video. Thank you for such amazing content.
Cool, thanks
Awesome
great work. i have one small suggestion: please use a dark marker.
Your explanation is great, keep the great work.
Thanks Zeyad
Excellent explanation
Thanks Anuradha
Great Explanation ....
Glad it was helpful!
thanks
When there is network problem and both dB can't communicate and we are allowing reads on one(b) and reads and writes on other(a) , we don't get latest data from A right? Until and unless reconciliation happens? So this will also lead to inconsistency?
Could you please create a playlist for LLDs
How do we come to know which node giving us correct result, and How can we decide at run time which node should accept write also ,in case of CA?
Great explanation as always. At 2:51 there's a mistake in reading about availability defn.
Very good attempt to explain the concept. Was very clear, thank you. At 11:50 you mention that in order to achieve some level of availability, B can continue to write even when the communication is Broken. But wouldn't it lead to loss of Consistency because after T2, a read from A will still return x=1 and not x=2 because that update has not been received by A?
That is what she said. We cannot have both together if there is a partition.
Looks like the example was not accurate. Instead of allowing B to only accept writes..you can let it do both as before. However, chances are fraction of requests will get accurate value if they are redirected to the same instance. But once the partition is recovered, we can have eventual consistency. Note that there is 100% availability even though there is lesser degree of consistency.
@@akshayshah8264 you're correct...the example is not correct because if you're favouring 100% consistency but allowing B to take write requests, then any read requests served by A will return stale data i.e. not consistent.
Therefore we could rather say that allowing B to take write requests and also serve requests (A is already serving both read and write), makes the cluster Available but NOT Consistent.
I think there is no such thing as some degree of consistency as the technical definition also says that all the reads after the latest write should give out the same result. You either have full consistency or none.
Good going. In distributed system some of the problem solve by redundant component and it is upto the business case to tolerate that cost or less in terms of overall system.
Absolutely!
Amazing video ! while watching this video In-between i also got a thought that what if we can have backups of these network like what we have in data that is data backup right. and in the end of the video got my answer. Thankyou and KEEP UPLOADING videos.
That's amazing Kunal 👍
In the degree (CP) and node B is writing and eventually read happens in node a it will return the wrong data right, how it is consistent than
Example at 11.57 - It won't be consistent right even if the system is taking all the 100 writes. Because until they reconcile, node A might have incorrect data, because what if the latest write happened in node B and not node A.
I didn't understand why we considered read to be only taken by one node. It's anyway inconsistent (eventual consistency though). So Node A would also give wrong data, and Node B will also give wrong data.
Eventual consistency is a type of consistency.
At 11:50 how will all the reads serve consistent data? 'A' wont be aware of the writes on 'B' bcoz of partitioning.
Hi Yogita, Thank you for explaining in details. I want to confirm something.
In the case of going for eventual consistency, we are choosing 'AP' right?
I mean you mentioned that one node is giving consistent output - which I think, is not the case. As there is something written in Node B, which is not transferred to Node A until it is reconciled. Resulting in inconsitent behavior.
Please add some more details on partition in this video.
Hi yogita, you mentioned if we allow writes on the node b we will have comparatively high availability than completely turning it off. what if latest write is on node b and later read is happening on node a? in that case read value is wrong. so is it fair to say in case of consistency we can have wrong value as long as it's consistent in all our reads?
I had same question as well.
Same question.
Same question
Man I had the exact same question and I came to comment section to ask but you already did.
if partition tolerance not occur then till that moment all cap are there but as per definition only 2
Thanks a lot for such an amazing and detailed explanation. Will there be any full system design video on system such as facebook, whatspp or uber? That will be really quite informational. Thanks
Thanks Chandra. That's in the pipeline. We will release such videos
Great video Yogita. Thank you for the free content :) . Let's say we construct 3 DB instead of 2 DB in your example, is it possible to achieve all 3 in CAP theoram. node A, node B, node C - Any Write will write on all 3 nodes to achieve consistency. (write could take bit longer). Reads will be consistent. If there is a partition, between A & B, B will stop serving traffic but A & C are there for availability. In addition to this, we can configure multiple networks between each node to avoid partition as you mentioned in the last part of video.
What if node A fails?
Hi Yogita, thanks for amazing videos. I had a doubt in degree of availability at 10:55 - if A gets x=3 at t3 and B gets x=2 at t2 since both of them can perform write operations then wouldn't the system become inconsistent as A never had the updated value i.e. x=2? So both availability and consistency are compromised. Please let me know if I'm not thinking correctly.
Both A & B can perform write operations but since only A could perform read operations so i'll always get latest value of x it wrote hence we can say that system is consistent with certain degree of availability in regards to write operations.
Consider an example that you are using a shared google doc with one of your friend. Now both of you are updating on the doc & your wifi goes down but the app instead of crashing (ie no availabilty) still allows you to write and whenever wifi goes up the doc is synced with your friend.
@@medharai5914 it'll not always be latest value of x, although the system will still be called "consistent" as only A performs the read operation and is the sole node responsible for responding with the read requests.
Very well explained
I watch all three video but still not understand .. please answer .. can we have system which is highly available and highly consistance without being Replicated database that is Partition tollerant ?
did you get your answer, pls share if yes....
So if system is P is broken as C will not be there at all, it so interconnected. Can system will be CA if P is compromised. CA is myth if P is compromised.
MAJOR DOUBT: Lets say we want 100% consistency and partial availability and consider the value of x = 1. Now in case of partition if we allow B to serve writes, my write request could go to B, so it will update its local value to 2. And immediately if I send read request which is supposed to be served by A will return 1. So in this case, in trying to provide partial availability we are voiding our 100% consistency requirement.
you're correct...the example is not correct because if you're favouring 100% consistency but allowing B to take write requests, then any read requests served by A will return stale data i.e. not consistent.
Therefore we could rather say that allowing B to take write requests and also serve requests (A is already serving both read and write), makes the cluster Available but NOT Consistent.
it's some degree of both at same time and we shouldn't forget that it's only for the time when network is disrupted, once it resumes your data will be consistent again..Time in which network resumes is crucial in this.
And again as explained in last section, we will only do this, when being consistent 100% all time is not our priority, if it is then the other node should become unavailable.
So this is a master - master architecture what you are saying. So here if you want Strong consistency then you have to introduce a third node which will help to maintain the state of the other two nodes. If I am correct, this problem is called a split brain problem. Whenever a write happens Node A will update it's state to Node c(our newly introduced third node) which will in turn update Node B.
first time saw someone so obsessed with black color as me :P
Can you please explain what happens if we can have different modes of communication between servers for replication/consistency? As in that case, even if there's one network failure between nodes, other modes can be up and running and thus facilitating consistency. Basically, it's expanding on "Availability" of communication methods between nodes.
Damn it! You have explained this one at the end. I commented without watching full video 😅 I'm not gonna delete this comment as this would be a lesson for others as well to finish the video first and only then comment the doubts 😄
😅😅
If the partition network breaks down, then the system would be available but we would not be able to maintain consistency as writes/updates couldn't be updated due to lack of communication channel, right? Then here we are not able to maintain two(C and P) of the three features. How is CAP theorem applicable here then?
Because you are able to maintain C and P. That's what CAP theorem says! At most 2 out of 3 properties can be achieved.
@@sudocode But in the case @harshit pandey , C & P wont be maintained.
If the system partitioned, you can compromise on consistency by providing the lastest update that you have not caring about other nodes, or you can compromise on availability by simply waiting for connection recovery and then responding after ensuring that you have the latest update received by the system. Right?
Hey, Thanks a lot for the awesome video. I have one small doubt. In the section on degrees of availability, you said that we can have writes on system B as well while restricting reads on A only, which will improve the availability. But what would happen if we read a value from A after we have updated its value on system B? won't this affect the consistency?
Yes, correct it will impact consistency but we sacrifice some degree of consistency over a high degree of availability. Some reads will get lates value, some will not but no read would fail because both nodes are "available". Definition of availability is giving a non error response with no guarantee of returning latest write. That's where eventual consistency comes into play!
@@sudocode ok thank you very much for clarifying my doubt :)
@@sudocode Quick question for you, in response to your answer, if we have decided to sacrifice consistency altogether and why limit "B" to write's alone? Why not allow the user to read from node "B" as well? Thanks, Shankar
You mentioned that if I choose to compromise on availability, then the system can be consistent even when the two nodes are partitioned, one of them is accepting only reads, and both are accepting write requests. But if the system is partitioned and the other node has received last write, how can this node answer a read request while being consistent as the other node could have the most recent write.
The other node cannot answer correctly and that is what makes it not available.
That also makes it inconsistent, right? Because it gave the incorrect response as it couldn't communicate with the other node that had the recent update
This video s little confusing. But she has attempted sincerely.
Yeah , same doubt. Even if Availability is compromised , I can’t understand how Consistency is maintained .
Learned that by giving an error, the system is still consistent. But, my first question is still valid. If the two nodes are partitioned, then both cannot accept writes, right? The possible case, per my understanding, is a scenario where Node1 is master and accepts both reads and writes. Node2 is slave and accepts only reads. If the system gets partitioned, slave stops responding to reads. Master remains consistent only it can writes.
Disclaimer - I am "0" in technicals and a Sales guy, just got interested accidently watching your 3rd video yesterday. Here is my question on time stamp 7.51 ..... if B has not acknowledged then how come "A" registered the latest write............ in case, this happens, then CAP is absolutely right, if not then CAP is wrong.... Kindly excuse my ignorance
In case, system to be consistent. And partition happens, then how come the 2nd system ll come to know, it has not the latest value?
Ohh ok, May be its tracking about partitioning. If someone reads and there is partition then return 'value many be incorrect, i ll not return in this case'. i.e Avail forfeits. I.e can't have a system CAP.
CAP- Can't provide 100% consistency and 100% availablity and system is 100%fault tolerant
For a distributed system.
please explain what is exactly meaning of partitioning
Two machines connected by network get partitioned if network is broken.
please use a darker pen if you can
2:50 You made a little error in your reading of availability my dear.
Seems to be LEVI'S is coming for brand endorsement🤔
I am sorry that this was your key take away from a lecture on CAP theorem. I will try to do better next time ;)
full till 4:30 ruclips.net/video/kwCFHLbIhak/видео.html
I believe your Partition Tolerance explanation is not correct. Partition means if your node is disconnected from other nodes due to network issues. Partition tolerance means the ability to recover all the data that was lost during the connection/network issue.