How Discord Stores TRILLIONS of Messages
HTML-код
- Опубликовано: 9 июн 2024
- Have you ever wondered how Discord is able to store trillions of user messages without breaking a sweat? In this video, we'll take a deep dive into the technologies and strategies that Discord's software engineers use to manage such a massive amount of data.
SOURCE: / discord
Discord is one of the most popular communication apps in the world, with over 250 million users and counting. With such a large user base, Discord's servers receive an enormous amount of messages every day. To store all these messages, Discord employs a distributed database system that runs on hundreds of servers worldwide.
This system uses a variety of modern database technologies, including Cassandra, Redis, Elasticsearch and most recently: ScyllaDB to ensure that messages are stored reliably and efficiently. These databases are designed to handle huge amounts of data and provide fast read and write capabilities, making it possible for Discord to scale to millions of users without compromising performance.
So if you've ever wondered how your messages are stored on Discord, or if you're just curious about how modern database technology works, this video is a must-watch. Join us as we explore the world of distributed databases and learn how Discord's software engineers keep trillions of messages organized and accessible for millions of users around the world.
👻 MY NEW STARTUP: www.caspr.ai/
MY NEWSLETTER 💌
thebetter.dev
------
CONNECT WITH ME ON SOCIAL
📸 Instagram:
/ lewismenelaws
🎚TikTok:
/ lewismenelaws
🐣 Twitter:
/ lewismenelaws
--
My gear 💻
liinks.co/lewismenelaws
-----
TIMESTAMPS ⏰
0:00 Intro
0:38 The BackStory
1:46 Cassandra Troubles
4:09 Change Architectures
6:28 Data Services
8:57 The BIG Migration
10:35 Where It's at now
11:40 What we can learn from this
12:46 Thanks for watching and subscribe :) 
Наука
Thanks for watching this video :) What company would you like me to cover next?
Reddit
Do How ExplodeCode is cool. naa do snapchat or instagram or something based with pictures
Rust roadmap
How telegram makes money and how it stores as much as data like discord and much more (you too will get more questions once you start to think how telegram is able to do it 😅)
And 😅 I want you and forrest knight to have a video together 🥺
How the hell is discord even making money. That infrastructure sounds pretty expensive...
Nitro 😭
Bloody expensive too
Data
Venture capital is one hell of a drug
They are not making money. Welcome to capitalism
This types of videos are great. It's not common to face scenarios that have this massive scale and it's great to get to know how they have been handled before by other engineers
The engineering was truly awesome. Have to salute the engineers as well as SyllaDB team for such a great product.
No © symbol next to “Rust”?
Lawsuit is already being filled
This is amazing, please keep going.
Such an amazing source of knowledge and inspiration in a perfect method for better engineers
11:24 Poor Descord engineers😢
They are stressing even HARD
lmao
Nice! I love databases and learning more about them, great video.
😂 same
Hi Lewis, I discovered your channel thru YT Shorts, but I like this format a little better, not only it dives into architectural problems and solutions, in my opinion it is not rushed. Also, awesome graphics! Thanks a lot and more power to your channel.
The type of content I love to watch thank you lewis!
Had this question for Reddit how reddit saved and would love to know how discord works
as a dev, just thinking about a migration of that scale is anxiety inducing
Kudos to the ScyllaDB team! I've been a fan of their product since 2019, and in my opinion, it's better than Apache Cassandra. However, their open-source license limits usage to only five nodes.
the ydeserve being paid for that product
Not really. The 5 nodes limit applies only to ScyllaDB Manger, not the OSS database.
72 nodes per 140mln users = ~1.9 users per node. 5 nodes limit = ~7.72mln users = you must have money for DB licenses 😅
@pietrodcfab but that's not how it scales
Wow discord are really committed to keeping their app up to date
I was today years old when I learned that Lewis makes long form content.
This is fantastic! Congratulations to all those amazing engineers and congratulations to you for making this great video. Thank you for bringing this content to us.
i love the way the subtitles interpreted scylladb: s db, mes db, solid b, sillydb, facilitydb, silladb, sil a db
still a DB
vs
no longer a DB
Really fascinating video and well made. Thank you!
I like your funny words magic man.
Definitely didn’t understand about 95% of this video but it was really fascinating and well presented
This video was perfect omg!! Liked & subscribed, keep it up ❤️❤️
I'm waiting for the day discord switches to vector databases when they have to store zetabytes of messages 😂
I can see it happening. I’m sure the singularity will shake up technology a lot, and it does seem like vector databases will continue to become more popular
Whoa! This is so enlightening, thanks Lewis! The engineers are doing really great work bts
great breakdown and description, thank you!
I am so happy 😁 to see how it goes with your channel and how much your topic is related to more advanced topics
I can imagine reading discord messages from 47 years ago.
Everything is perfect. Thanks much Lewis!
Your channel needs to be more popular. This content is the only thing that motivates me to write some code after running into an error.
This types of videos are great, and your animations are really on point
Very cool format and interesting stories!
I wish Discord to find frontend devs as good as their backend team
Bold of you to mention the Rust language and use it's logo in this video. Considering what the Rust Foundation has been up to recently.... AND YOU SHOW A PICTURE OF THE LOGO IN A DIFFERENT COLOR... oh my god, I hope the foundation doesn't see this.
🤣🤣🤣
This was an incredibly well made video!
Love the video and also the short takeaways at the end
This is my first video on your channel. Loving it. Awesome video :)
agreed, my first video too! :D
Aha so now we know how! Great video man!
The concern you described at the end is a well known anti-pattern called the Golden Hammer. Don't get too comfortable with the tools you know and use.
Never realized how incredible the engineering is on Discord
Great video. Fascinating to see how they solved this!
Thank god yt recommended this. I learned a lot.
Thank you for this video! Please create more
Sir you're my inspiration these types are useful please bring more like this like how big system works so that we can have real-time understanding
This is better than ASMR. Fell asleep in 8 minutes!
Great content, please I want to learn about the internet from a developer's perspective what book are you going to recommend for me?
Excellent video 👏🏼👍🏼
Very nice video Lewis
This guy basically turned Discord's blog into an interesting video 10/10.
What a nice video. Thank you very much :)
love the animations on these videos
Great video Lewis 😃. Got to know so much information about Discord databases. Thankyou!
Great video ❤
This was very fascinating to watch
Wow I really enjoyed the video, so insightful.
Great video. Subbed.
I wonder if the issue with the normal Cassandra is really the GC, or the memory pressure Java currently has due go reference types. I suspect user defined primitives and fibers will really make Java a stronger contender in the high performance space
Great video!
wow very very insightfull !!! thanks for making this video
Could you do "A roadpath to become a software engineer" some time soon?
this story are amazing, great motivation to improve myself!
Awesome dude!!!
Today i learned a lot about database management how behind the scenes work
Great video. Great content. Thank you.
I may be wrong but I heard from someone in a livestream that data service layer can be implemented in golang and the latency difference would still be unnoticeable
This shows that Discord developers are really capable, yet the auto update on Linux doesn't work.
Because Linux is literally completely irrelevant, atleast on the consumer market.
@@Tudas lol so dumb
@@Tudas You would think Discord, a company that relies on 100% of their own servers running Linux and is a consumer of open-source products like Apache Cassandra, would put just a tad more care into the linux version.
@@TudasI mean, there's the Steam Deck now. It definitely has become a LOT less irrelevant compared to a couple of years ago.
@@Tudaswhat are you talking about. The entire consumer market is reliant on Linux. Facts.
the editing on this channel has exponenetially increased in quality 🤩
Really enjoyed this one ☝️
That was cool… awesome breakdown…🔥🔥
Love this channel
Discord is the perfect combination of Twitter, Facebook, an in-game chat bar, and even Instagram. If you pay them a little you can even treat it like RUclips or Twitch.
Cool video, earned a sub
I read the title and immediately thought, “I wonder how much storage space I can take up by just spamming emojis…”
So informative and entertaining.
have never put much thought into discord but they really are at the peak of social media, almost 0 downtime ever and always provides a consistent streamlined experience. I can imagine that they will end up doing many many more database migrations in the future as discord dominates the social media space. Wonder if there will ever be a permanent solution of a database which is truely infinitely expandable.
Great video, and I'm left with one question how to find and use the right tools at the right place and at the right time 😅
That something comes with experience or any beginning can also master that?
A few years ago, I don't remember why exactly but I had issues my phone plan and I was stuck with worse than dial up speeds for a while. Im talking low kilobytes per second, maybe even bytes. Discord was literally the only "online" app that still worked pretty much normally (besides embeds). That thing is optimized to death
yeah that's not it. the size of the data being sent and received has no correlation between how optimized the app is. your internet connection works for discord because the amount of data sent by discord is very small. it has nothing to do with app optimization.
Love these videos
Lewis: *thinking he says NoSQL bad*
Me: *says NoSquirrel*
every time I think about discord I get a horrendously awkward flashback to my first situationship :/ when he asked “do u have discord” I replied “about you staying or about you leaving?” 💀
what a video! learning + entertainment
Please Keep making such system design videos because I don’t like to read long articles 😂
You just altered the color of the rust logo. That is in violation of their trademark!!! 😂
Interesting colour choice on the Coalesce... What an interesting hypothesis.
This could've been the perfect video to advertise your dc
He's got his own data center? Nice
Interesting video. I feel sorry for that underused blackboard though… it looked like it was going to be written on about 5 times with not a single mark ever made on it!
Wow this is an awesome informative video.
Please do keep them coming!!
I love discord. Best comunicator out there
"2017 was 6 years ago" nooooooo😭
I love how you made the letters of coalescence alternate purple and yellow... like Moira's ultimate in Overwatch. This was intentional, right?
Hmmmmm 🤔
Subtitles: "Hi, my name is, and"
Video: "Hi my name is Lewis"
discord working their ass off just to allow me and my friends to write the dodgiest messages ever (we're on a government watchlist)
😂😂
thanks for the video!!! the question was in my mind this morning and, what a great recommendation, you gave me a full answer! but I have one more question why discord mobile app is so laggy and slow, while its web and desktop versions works fine
Can you go into a little more details as to why they were facing difficulties?
Make a video on how Google sheets can handle Sheets that are thousands of rows long without even the user noticing it. For example How it can load the data faster(even with my crappy internet). I know that they will be using compressions and things. But a video or a short would be nice explaining it.
It’s just text. Your average RUclips video is harder to load
@@martinmc0950 It's not just text lmao, did you ever see Sheets that are fking huge? But Excel does that even better, NASA literally uses Excel for huge ass calculations.
@@Tudas just because the sheets are big doesn’t mean it’s not text??? Although I see where you’re coming from with the whole calculation thing, using a computer for math is always going to be effective
great video
Thanks for your sharing
I have a questuon ... If they had written the data services first , would they still migrate from cassandra ? Considering that daya services was their bigger improvement ?
Great video.
Got urself a sub sir
"what any programmer would want to do when they see something new: changing architectures"
I'm guilty.
That's awesome
More than awesome!
Nice Bro :D
I feel like you are trying to target newer programmers when you explain very simply concepts but then at the same time you gloss over extremely advanced concepts as if we understand them already. I'm not really learning much. I would love like a 5 part series of this whole thing in extreme detail. How does scaling work? How does spinning up clusters/nodes work, how are requests routed? How does GRPC work.
9:10 during soviet era migrating entire buildings during night was a thing. And utilities(electricity, telephone, water, sewage) kept working.