This video proves two important things to me: 1. Quality is far over quantity in hiring for engineers. 2. Simplicity principles like KISS is a blessing in any form of engineering
@yestechguy-fj9tm im a software engineer and sometimes there are alot of features to implement which are pretty basic. Ofcourse designing a scalable application isn’t one of them
Absolutely, the book The Mythical Man-Month written by Fred Brooks talks about this, I recommend a lot reading this book, it helps you prevent erros that when you know it, it seems obvious, but when you don't know, it leads to a catastrophe
As a very new developer... I do not understand many things he is saying.... I am just amazed by the ingenuity of engineers... I hope to be like one one day
Heyy don't worry if you don't understand these concepts as you're starting out! I honestly didn't know much myself when I first started. Had to spend a lot of time studying system design for interviews and that's how I started to learn more about system architecture. If you're interested github.com/donnemartin/system-design-primer is a really good resource.
me neither, lol. done projects using postgresql for about 3 years, but haven't used any of these tools or faced problems like this due to projects being built for low user counts (local industrial facilities etc)
Best tech channel I've seen in a long time, great thumbails and content, you're bound to blow up! I would say to keep going on the company architecture case studies with practical takeaways, they're super insightful as a viewer. Also don't be afraid to ask people to subscribe if they enjoyed the video at the end, I think it's been proven to actually work.
I showed your comment to my gf who is my thumbnail designer and she was very happy to hear this! Thank you for making her day! And yeah I'll definitely be giving your suggestion a try in the next video.
I have such a hard time trying to not think "OK so all they needed was two larger, but not large, AIX boxes or RAC and instead they engineered around" till they had finally cleaned up things back to a manageable design.
3 questions: 1. How many years of experience did your professor have in the industry? 2. Am curious, how exactly did the video explain it better? 3.. Which university was this if you don't mind sharing?
@@skyhappy1. Ten years 2. Well, now that I think about it, I think I just needed examples. He did a good job of explaining the conflicts one might have with distributed databases when there are lag/shutdowns, but he didn't cover sharding very clearly. But yeah, personally I learn a lot through examples and he didn't give many. 3. University of Manchester
@@altertopias Examples are super important, I like them as well. One of my grips with professors at my uni, uni of Toronto, is that they give textbook definitions but barely any examples. This happened in the SQL databases course, operating systems, networking, and more. I think it's a sign of poor knowledge on their part, as they are just regurgitating what the textbook says. Only someone who understands it well is able to give good examples. Also, what year did you take the class? Wondering how recent it was.
the sharding part is very complex topic. I have never done sharding the database. I read somewhere that sharding is probably the last option you should do when it comes to scaling the database because of how difficult it is
It's probably not as bad as you're imagining in most cases. So long as you can logically delineate the data then you won't have an issue. You can think of it as adding a layer of indirection, before you fetch data you need to know which database the data is in. If you can find a simple way to solve that problem in your application, then you won't have any issues.
My guy. Great use case. Would love to hear more abstract strategies how to solve general problems. But this was neat and awesome. Great entertainment plus could use it for my work. Ps. : an other suggestion would be to evaluate if sharding with consistent hashing would be a good avenue to follow. Generally, I might be great solution
When you upload a new image into an albums. Then immediately when you come back to the album, the new image shows up wrong. And only after the 2nd refresh, would it show the correct new image. I noticed this as of 2023/4
@@kikisbytes Some of the projects take a lot of time. I imagine your elaborated videos also must take a lot of time and effort to complete (research, script, animations, etc) I hope I had more time to make videos more often but at the moment I'm very busy with uni and some web dev and ml projects. I hope this summer I will have more time to release new content again. Keep up the good work!
The original article in the description if you want to learn more. It was quite hard to follow so that I had to spend a bit of time trying to understand what they were doing.
hahaha no worries!! And yeah I use motion canvas for the animations. I was inspired by the creator of motion canvas to start using it. His channel on youtube is `aarthificial` if you're curious.
Software doesn't require that many engineers if you put quality first. Once something is done it doesn't require change unless you did something wrong.
Everything except nginx is a AWS Service with a fancy name, oversimplified the services are as follows: EC2 -> A virtual private server (VPS) S3 -> Some kind of external HDD with a fancy api CloudFront -> A CDN, Webservers serving static files like images but from a server which is physically as close to you as possible Nginx -> A webserver, compare it to apache2
heyy first of all thank you for taking the time to watch the video! What are your thoughts if I add some brief descriptions of these technologies in the description? Would that be helpful in future videos?
the original blog is in the description which I used to understand what was done and I fill in missing details as needed (how some technology work etc).
Loved the videos and as a cs student, I hope to become a problem solver as these guys. I am also interested in learning database design, would you recommend any course(preferable) or books?
thank you and you will eventually!!! Hmm I recommend checking out: 1. Hussein Nasser's channel on youtube. He has some really cool videos on backend. 2. Alex Xu's system design books (honestly helped soo much back then) 3. Designing Data-Intensive Applications (book) Hope these help you on your journey!
@@kikisbytesThese sound good for future but I was actually asking for some core database resources like once that teaches the core concepts like ACID principles, normalisation, etc. I had one database book during my uni but that was dense theory and not very practical. If you have any recs for that then I would appreciate you sharing them
@@dickgrayson4237 ahh I get you. Yeah sorry I don't really have any specific course that I can recommend :( I learned about these concepts through interview prep, work and blogs that I come across.
A place with a lot of cr*p pictures ? I often have to remove pininterest results from the search to avoid irrelevant data. Regrettably, trash is getting popular these days.
They never ran PHP. The video mentioned the LAMP stack only as an example of what other comanies were using. They used mysql like in LAMP but they used python instead, with a mix of modified Django for their initial service and other frameworks for specific parts. They also adopted react early to try to make the frontend avoid repeated requests to the server if it had already fetched the relevant data.
that's a good question and I don't know the answer to but I'm sure they have load balancing, service discovery, auto scaling and some monitoring to handle the scale.
Pinterest has a very specific focus and feature set, where as other platforms are often doing several dozen things, like a phone system, a Messenger system, video Infrastructure and in some cases being outright full blown ISPs
i mean, the site doesnt do anything useful so im not surprised they managed. the amount of times ive looked for an image only to find frikkin pinterests thumbnail-sized (resized) crap was what lead me to make the google chrome extension that removes all pinterest results from your google searches
@@kikisbytes Ofcause those extensions are open source just like any Javascript program. Those extensions to prorect people from malicious Pinterest's practices were available for all the browsers in various forms.
I know the answer from experience: Spam. They somehow convinced Google to show them whenever I search for pictures. None of the pictures they host are available for free, so people who are desperate become customers. That's not a model I'd recommend anyone else reproduce and it's certainly nothing to admire.
This video proves two important things to me:
1. Quality is far over quantity in hiring for engineers.
2. Simplicity principles like KISS is a blessing in any form of engineering
@yestechguy-fj9tm Depends on work in quantity vs quality.
@yestechguy-fj9tm im a software engineer and sometimes there are alot of features to implement which are pretty basic. Ofcourse designing a scalable application isn’t one of them
agreed and even more critical if you're just starting out!
The less 'moving parts' there are, the less prone to failure something will be.
Absolutely, the book The Mythical Man-Month written by Fred Brooks talks about this, I recommend a lot reading this book, it helps you prevent erros that when you know it, it seems obvious, but when you don't know, it leads to a catastrophe
As a very new developer... I do not understand many things he is saying.... I am just amazed by the ingenuity of engineers... I hope to be like one one day
read the first few chapters in Designing data intensive application
Heyy don't worry if you don't understand these concepts as you're starting out! I honestly didn't know much myself when I first started. Had to spend a lot of time studying system design for interviews and that's how I started to learn more about system architecture.
If you're interested github.com/donnemartin/system-design-primer is a really good resource.
Thanks
me neither, lol.
done projects using postgresql for about 3 years, but haven't used any of these tools or faced problems like this due to projects being built for low user counts (local industrial facilities etc)
Read up on scaling up a database....that's the basics basically then a lot of it is system design
As an ops Person, these software devs did a solid job of designing their infrastructure
very beautifully made video, i wonder how it has only 2.7k views with such high quality
haha thank you! I just like making these type of videos and I just hope that my viewers learn a thing a two :)
the views are coming
Way interessting that primegean screaming like an idiot and giving hot takes on everything
Please continue to do these types of videos it’s such an important insight that isn’t common on youtube
Best tech channel I've seen in a long time, great thumbails and content, you're bound to blow up! I would say to keep going on the company architecture case studies with practical takeaways, they're super insightful as a viewer. Also don't be afraid to ask people to subscribe if they enjoyed the video at the end, I think it's been proven to actually work.
I showed your comment to my gf who is my thumbnail designer and she was very happy to hear this! Thank you for making her day!
And yeah I'll definitely be giving your suggestion a try in the next video.
Interesting case study! Thanks for the great explanation and engaging visuals; keep it up :)
I'm glad the visuals were engaging!
Amazing video. I appreciate every bit of effort you put into creating this video.
Thank you for taking the time to watch this video! :)
Using a cache to compensate for replication lag is brilliant
This channel is a god send. As i am moving into more senior roles, these videos are gold mine.
That's awesome hopefully you will stick around for more videos like this!
@@kikisbytes you bet. Notifications turned on 🫡
Damn 🔥, you'll react 1M soon. The quality of video and explanation is quite smooth.
Great content, very well explained with nice animations.
Can you tell which tool you are using for this animations?
It’s a lot of data engineering and data infra work, which not many people appreciate. Great content.
Just stumbled upon this channel.
This video is top tier!
---
Earned a sub
Great content, definitely need more stories like this!
I have such a hard time trying to not think "OK so all they needed was two larger, but not large, AIX boxes or RAC and instead they engineered around" till they had finally cleaned up things back to a manageable design.
This was great, thank you for sharing!
thank you for taking the time to watch!
This is gold! Thank you!!
Glad it was helpful!
3:20 not to be confused with sharting, which is *much* less favorable.
Great content and nice story telling...
bUt wHy NoT JuST haVE a MONOl1tH
Explained distributed databases better than my professor! Thank you
3 questions:
1. How many years of experience did your professor have in the industry?
2. Am curious, how exactly did the video explain it better?
3.. Which university was this if you don't mind sharing?
@@skyhappy1. Ten years
2. Well, now that I think about it, I think I just needed examples. He did a good job of explaining the conflicts one might have with distributed databases when there are lag/shutdowns, but he didn't cover sharding very clearly. But yeah, personally I learn a lot through examples and he didn't give many.
3. University of Manchester
@@altertopias Examples are super important, I like them as well. One of my grips with professors at my uni, uni of Toronto, is that they give textbook definitions but barely any examples. This happened in the SQL databases course, operating systems, networking, and more. I think it's a sign of poor knowledge on their part, as they are just regurgitating what the textbook says. Only someone who understands it well is able to give good examples.
Also, what year did you take the class? Wondering how recent it was.
This is GOLD, thank you for sharing!
Where do you get this information?
Thank you for watching! The original blog is in the description if you want to check it out :)
great content, we need more of this.
Thank you and more are coming!
the sharding part is very complex topic. I have never done sharding the database. I read somewhere that sharding is probably the last option you should do when it comes to scaling the database because of how difficult it is
It's probably not as bad as you're imagining in most cases. So long as you can logically delineate the data then you won't have an issue. You can think of it as adding a layer of indirection, before you fetch data you need to know which database the data is in. If you can find a simple way to solve that problem in your application, then you won't have any issues.
They basically created MongoDB with MySQL
My guy. Great use case. Would love to hear more abstract strategies how to solve general problems. But this was neat and awesome. Great entertainment plus could use it for my work.
Ps. : an other suggestion would be to evaluate if sharding with consistent hashing would be a good avenue to follow. Generally, I might be great solution
When you upload a new image into an albums. Then immediately when you come back to the album, the new image shows up wrong. And only after the 2nd refresh, would it show the correct new image. I noticed this as of 2023/4
Very valuable content! Love this case studies
Heyy thank you very much! Also I checked out your channel and damn some very interesting videos! How long did it take to build one of those lego cars?
@@kikisbytes Some of the projects take a lot of time. I imagine your elaborated videos also must take a lot of time and effort to complete (research, script, animations, etc)
I hope I had more time to make videos more often but at the moment I'm very busy with uni and some web dev and ml projects. I hope this summer I will have more time to release new content again.
Keep up the good work!
looking forward to seeing you publish your next video!
teaching style very cool and easy thx
Bro how do you get all this operational detail. Great stuff
The original article in the description if you want to learn more. It was quite hard to follow so that I had to spend a bit of time trying to understand what they were doing.
Hi Kiki, the video was very informative and I really enjoyed the visuals. What do you use to make them?
Motion canvas. Sorry for bothering you before checking other comments!
hahaha no worries!! And yeah I use motion canvas for the animations. I was inspired by the creator of motion canvas to start using it. His channel on youtube is `aarthificial` if you're curious.
Another great video!
Glad you enjoyed it!
When you said Membase, is that the same thing as Couchbase?
Software doesn't require that many engineers if you put quality first. Once something is done it doesn't require change unless you did something wrong.
lmao okay. I wish I had that experience.
Yep one of the takeaway that they mentioned which really resonates here
i love denormalization
After watching your other videos of companies growing to scale it seems like the engineers at pinterest always chose the most complicated solutions 😭
it's called job security 😉
Great video, thank you very much. Subbed.
much appreciated!! Stay tune for more!
nice and clean presentation. What app/solution did you use for the visualization?
I did most of the animation using motion canvas!
"Everything will fail, so try to keep it simple." - wise engineer
yeah that's why I will integrate AI, ML, deep learning, blockchain, cryptocurrency into my todo app.
Great video.
This was great. Could you also make a video explaining the working of sites like Canva? Thank you
I just sharded my pants
LOOOL what. At least you can clean each part of your pants individually :P
Why didn't they just use nosql(Cassandra) then? It had all these capabilities built-in
great video
What tool is used to create these animations?
Im new to web development so i dont understand anything here. What is EC2, S3, CloudFront, NGINX? What even are they used for?
Everything except nginx is a AWS Service with a fancy name, oversimplified the services are as follows:
EC2 -> A virtual private server (VPS)
S3 -> Some kind of external HDD with a fancy api
CloudFront -> A CDN, Webservers serving static files like images but from a server which is physically as close to you as possible
Nginx -> A webserver, compare it to apache2
Cool. Thank you for taking the time to respond. All of this is just so confusing.@@soundzthevibewalker
heyy first of all thank you for taking the time to watch the video!
What are your thoughts if I add some brief descriptions of these technologies in the description? Would that be helpful in future videos?
@kikisbytes Yes. That would be tremendous help. Especially for beginners like me to help us go along with your videos and not get lost.
Which technologies did you use to create this awesome video? haha
thank you for the compliment! I use motion canvas for the animations
nice,keeeeeeeeeeeeeep going 🚀🚀🚀
good job
Thank you!
Wow!
How do you know this stuff? How programs worked and how they changed and what software they used?
the original blog is in the description which I used to understand what was done and I fill in missing details as needed (how some technology work etc).
Pinterest is a blight on the internet. Utterly pointless website that pollutes search engines.
Loved the videos and as a cs student, I hope to become a problem solver as these guys.
I am also interested in learning database design, would you recommend any course(preferable) or books?
thank you and you will eventually!!!
Hmm I recommend checking out:
1. Hussein Nasser's channel on youtube. He has some really cool videos on backend.
2. Alex Xu's system design books (honestly helped soo much back then)
3. Designing Data-Intensive Applications (book)
Hope these help you on your journey!
@@kikisbytesThese sound good for future but I was actually asking for some core database resources like once that teaches the core concepts like ACID principles, normalisation, etc.
I had one database book during my uni but that was dense theory and not very practical. If you have any recs for that then I would appreciate you sharing them
@@dickgrayson4237 ahh I get you. Yeah sorry I don't really have any specific course that I can recommend :( I learned about these concepts through interview prep, work and blogs that I come across.
Thank you algorithm for bringing me this amazing and hidden content
awhhh 😭 thank you so much for the kind words!!!
How do you know the issues they faced with such detail?!
you can check out the original blog in the description!
A place with a lot of cr*p pictures ? I often have to remove pininterest results from the search to avoid irrelevant data. Regrettably, trash is getting popular these days.
very good very nice
Thanks!
Thank you for watching!
Are they still running php?
They never ran PHP. The video mentioned the LAMP stack only as an example of what other comanies were using. They used mysql like in LAMP but they used python instead, with a mix of modified Django for their initial service and other frameworks for specific parts.
They also adopted react early to try to make the frontend avoid repeated requests to the server if it had already fetched the relevant data.
@BosonCollider they would have been better off with PHP as it's orders of magnitude faster, requires less resources, and highly scalable.
How do they manage that lots of instances?
that's a good question and I don't know the answer to but I'm sure they have load balancing, service discovery, auto scaling and some monitoring to handle the scale.
The real answer is SEO spam and making image search unusable
what Pinterest whatsap and discord have to do with erlang?
sharting
sharting!
2:15 Alogirthm?? ALOGIRTHM??!
omfg I can't believe I missed this 🙈
very nice video subscribed if you keep the graphics same and use the same topics like these
thank you! Def will be seeing a bunch more type of videos coming soon
What happened since 2012? :)
Shardt
Only 6 engineers?
That costs a million dollars a year In the US.
Why would you need a hundred devs.
Small teams get stuff done.
Pinterest has a very specific focus and feature set, where as other platforms are often doing several dozen things, like a phone system, a Messenger system, video Infrastructure and in some cases being outright full blown ISPs
so they had 40 engineers 10+ years ago, guess they have thousands now.
yeah I wouldn't be surprised if their employee number is in the thousands.
sharding not to be confused with sharting
🙈
similar concept though. Your digestive system gets over loaded and you shart out a mess.
Bcz they didn’t use Rust
Rust didn't exist and would have slowed build time wayyy down. Go or PHP would have been far better than Python
Pinterest loading is built like garbage, not like doomscroll addicts seem to car about their pictures taking 30 seconds to load
You kidding? It takes like a second max to load everything in every page. It just seems slower because no lazy loading elements
haha, Alogirthm
so embarrassing haha 🙈
i mean, the site doesnt do anything useful so im not surprised they managed. the amount of times ive looked for an image only to find frikkin pinterests thumbnail-sized (resized) crap was what lead me to make the google chrome extension that removes all pinterest results from your google searches
that sounds interesting, is your extension open source by any chance? Would love to check it out.
@@kikisbytes Ofcause those extensions are open source just like any Javascript program. Those extensions to prorect people from malicious Pinterest's practices were available for all the browsers in various forms.
tl;dr they stole the internet
I know the answer from experience: Spam. They somehow convinced Google to show them whenever I search for pictures. None of the pictures they host are available for free, so people who are desperate become customers. That's not a model I'd recommend anyone else reproduce and it's certainly nothing to admire.
they made a worse version of cassandra
That's nothing. Whatspp got to billions of users with just 35 employees.
That's one of the case study that I want to cover in the future :)
So sharding, eww.
pinterest is the cancer of internet and i'm baffled that it existed for so long time
Go cry that you can't do anything like it 😂
I could tell you're asian without seeing your face
It's your accent
guess you'll have to stick around to find out :p
If they started using Parquet and did not just copy and paste tech stacks it would only need 1 engineer