This video took 6,089 lines of code to bring to life. Here’s the breakdown if you’re curious: S1: 507 S2: 386 s3: 285 S4: 546 S5: 771 s6: 414 S7: 531 s8: 572 s9: 783 s10: 598 S11: 256 S12: 219 s13: 221 And it's built using Motion Canvas - motioncanvas.io/
They literally get paid with stock, if discord makes more money their stock is worth more. They can then sell it when discord goes public and get paid. smh
@@Reyhz I'd say banning 1/2 of the predators would cut more than just 1/2 of the revenue, more like 3/4+ ya assume predators and non-predators buy Nitro at (roughly, we're all doing very vague estimations here, so it's not about the precision) the same rate, but I'd say: what normal person buys Nitro? it seems useless for normal people who just want to have a chat with friends or participate in a kinda forum every once in a while, you need to be a bit of a no-life to pay for fluff like extra emojis or whatever, it's really for the minority of users for whom Discord is their whole life AKA mostly predators, lol - now that I think about it, the profit cuts would prolly be even worse, it's probably more like "1% of predators make up 90% of the Nitro purchases", lol [I'm just joking, ofc, no hard feelings]
Its fascinating to see what a big company does to implement a new feature or a fix. So many factors, so many elements to check for. Great video. Thanks.
No. It's fascinating to see how people are unaware of zstd existing that was released 10 years ago and I haven't seen anyone who has every worked with logs being unaware of it and actually using gz/zlib for anything where bandwidth/compressed file size is actually important
@@noobgam6331 nah, I just send the full raw data, all the music is in WAV, images in BMP (hell, the background is just one color, but I don't just send CSS with background color, you get an 8k uncompressed BMP image of #ffffff, you can check every pixel individually, each one was served with love and care - hell, my engineers are working on a new system where we'll send everything as a JSON/plaintext file in the background, images will just be a giant array with all RGBA values going like [[255, 255, 255, 1], [255, 255, 255, 1]] and so on and then a giant JS script with lots of dependencies will render the images on runtime, it'll be awesome, we have the best pixels out there, it improves accessibility cos every user can just open their Dev Tools and have all the pixels laid out in a human-readable format (images aren't human-readable, they make my eyes hurt), they can read it like a novel, ofc we send all the multimedia first and only once the user loads everything (we'll test it with a checksum), we send the actual content of the website - I need to be sure users experience everything in all its glory, not loading everything ruins the user experience, the new system helps to fight all the hacks like using wget or telling the browser to just download the plain HTML file, now it'll all be sent as JSON on fly, you need to have JS enabled and actually load everything, the script keeps sending randomized unique hashes to check if you're really downloading everything and established a proper session instead of trying to just download the text, ruining your user experience in the process), all with very high bitrates and resolution - my users deserve only the best: the most pixels, the most waves, the most data! if you can't access the site, tough luck bucko, buy a better internet connection or if it's on the server due to the load, wait for another time of the day, check the downtime info, estimate based on the timezones of the users and try to access it during the few hours in a day when the servers aren't overwhelmed - the very limited exclusive access only adds to the perceived luxury, it's only accessible to the people with the best internet connection during only a few exclusive hours of the day (if you're from the US, tough luck bucko, you'll need to stay up all night and keep refreshing; in fact, I'm considering closing off the website for the Shabbat [from Friday's sunset to Saturday's sunset, according to the Israel timezone (the servers will be down) + an additional JS script blocking it according to your local timezone] to be respectful towards religious Jews), people like the feeling of being the select few who managed to access and see the website (all bots are blocked, ofc, no archiving, no cache), the exclusivity makes it precious, we're the Apple of the webdev world: only the select few, dedicated rich people, get to see it, but once you access it, you get the best experience, the website runs super smoothly cos no decompressing is happening on the user's side, everything's pre-rendered and sent as raw data, so it's rendered instantly, a super smooth user experience
@@kikisbytesI didn’t find it particularly hard to understand, and I feel like you did a pretty decent job of explaining compression. I do that wish you went into more depth about what context is and how it’s used to improve the efficiency of streaming compression, though. Otherwise I thought the video was great!
@@kikisbytes randoguy7488 is right on the money. Your video is great, but not very engaging. The lack of pauses and monotone voice give off a school presentation vibe (no offense). Other than that, the content of the video itself was fantastic and I love seeing deep dives into postmortems like this.
5:48 Woooooow, almost like reducing the amount of data you send in the first place is more important than the compression you apply after the fact. Who could have guessed? Truly the most incredible engineering feat of all time.
@@wormsquirm2707 a little. I take issue with the likes of discord taking away what was once a popular paradigm - people self hosting their VCs with software like ventrillo or team speak. Now all of this user data is going to a company - who knows how it will be used against us long term.
I love the quality and pace of this video - engaging explanation of the underlying concepts, use case story, test outcomes, deployment strategy...all explained in 8 minutes! Edit: Plus citations and Chapter markers in description too! 🥳
One thing discord could do to save bandwidth would be to hash all files and checking the hash against a file database before actually sending the file to the server. Firstly it would reduce bandwidth usage and secondly it would reduce storage usage.
Thank you for the feedback! I usually make the animations of how I envision the scene to look like then I try to match it with the voice recording. I find that this 2-3x my development speed. This means that sometimes the animation can be fast because the voice recording section is a lot shorter than I had anticipated. Will try my best to leave another second or two for you to absorb the graphs.
@@kikisbytes I thought it was perfect, even at 2x. Honestly really good pace - usually I start a video and scroll to the commemts while the idiot talking takes 9x longer than he needs to to say anything, but this had me actually watching the video. Thank you.
0:08 costs per what? Week? Day? Hour? It's not cumulative, it's not monthly, the points don't line up with months and looks like it's every 2 weeks but that's a weird unit of time
for the record, zip drives from the 90s went all the way up to 750mb, a floppy esk disk from 30 years ago can hold more than a discord attatchment, and nitro's 150mb limit isnt much of an improvement, I still have some photos considered too large to send.
1:35 That's not exactly how it would compress, interestingly. The more efficient way is A B C (3, 6, EOF) This looks like "Write A, then B, then C. For the next 6 characters, copy the character that's 3 characters back from the position you're writing." (0 indexed) When you write the [3] letter, you're copying whatever letter is 3 positions back, [0]. Then for [4], [1] is 3 positions back. [5] is [2], and this is key: [6] is copied from [3], data that has not been written when we started copying!
They could reduce traffic bandwidth by even further with p2p networking. Just send CRC of an image, and then force clients to find the image by themselves with a timeout, if there's no image among the other peers, send one from server. So in a large servers, the more people saw some image, the more you have peers for this image
Some day discord will managed to reduce their bandwidth usage by 100% by making their clients so bloated and with every feature behind a paywall so everyone will just use something else.
Sure that’s how we solve problems, everybody should have their own company. That’s a great Idea, everybody should have their own country too, so they can make their own laws. At that point just have your own planet. That’s a stupid way to think. Just accepting obvious injustices and not unionising is the first mistake.
This video took 6,089 lines of code to bring to life. Here’s the breakdown if you’re curious:
S1: 507
S2: 386
s3: 285
S4: 546
S5: 771
s6: 414
S7: 531
s8: 572
s9: 783
s10: 598
S11: 256
S12: 219
s13: 221
And it's built using Motion Canvas - motioncanvas.io/
What program and language are you using?
@@faytruefireside good question
Manim
@NiFreZ yeah I'm also interested
I use motion canvas to make my animation!
motioncanvas.io/
Trust me, engineers don't get the bonus for this
youre right, they get paid 300k a year. this is literally their job. a surgeon doesnt get a bonus at every successful heart transplant
one can dream hehe. Exceeded performance and get some extra cash into 401k / RRSP
I saved my company 30k a year in cost with some optimizations and I all got a nice job.
@@HolyOllie Who the hell gets paid 300k unless they're staff+?
They literally get paid with stock, if discord makes more money their stock is worth more. They can then sell it when discord goes public and get paid. smh
"I hope the engineers get a big bonus for this" - Yeah probably, a big bonus time off to look for a new job after the layoffs.
that would be very upsetting :(
True story.
@@kikisbytes "I love capitalisim" tech bros hgahahahahaha
The bonus went to management for sure 😂
hahah yea....one can dream
If they ban half the predators there it would also half the traffic lmao
And half those pradators pays for nitro, they'd also halve their revenues
They might end up banning themselves
@@simdimdim LOL
So you mean everyone on Discord is predators?
@@Reyhz I'd say banning 1/2 of the predators would cut more than just 1/2 of the revenue, more like 3/4+
ya assume predators and non-predators buy Nitro at (roughly, we're all doing very vague estimations here, so it's not about the precision) the same rate, but I'd say: what normal person buys Nitro? it seems useless for normal people who just want to have a chat with friends or participate in a kinda forum every once in a while, you need to be a bit of a no-life to pay for fluff like extra emojis or whatever, it's really for the minority of users for whom Discord is their whole life AKA mostly predators, lol - now that I think about it, the profit cuts would prolly be even worse, it's probably more like "1% of predators make up 90% of the Nitro purchases", lol
[I'm just joking, ofc, no hard feelings]
Its fascinating to see what a big company does to implement a new feature or a fix. So many factors, so many elements to check for.
Great video. Thanks.
No. It's fascinating to see how people are unaware of zstd existing that was released 10 years ago and I haven't seen anyone who has every worked with logs being unaware of it and actually using gz/zlib for anything where bandwidth/compressed file size is actually important
thank you for watching!
@@noobgam6331 nah, I just send the full raw data, all the music is in WAV, images in BMP (hell, the background is just one color, but I don't just send CSS with background color, you get an 8k uncompressed BMP image of #ffffff, you can check every pixel individually, each one was served with love and care - hell, my engineers are working on a new system where we'll send everything as a JSON/plaintext file in the background, images will just be a giant array with all RGBA values going like [[255, 255, 255, 1], [255, 255, 255, 1]] and so on and then a giant JS script with lots of dependencies will render the images on runtime, it'll be awesome, we have the best pixels out there, it improves accessibility cos every user can just open their Dev Tools and have all the pixels laid out in a human-readable format (images aren't human-readable, they make my eyes hurt), they can read it like a novel, ofc we send all the multimedia first and only once the user loads everything (we'll test it with a checksum), we send the actual content of the website - I need to be sure users experience everything in all its glory, not loading everything ruins the user experience, the new system helps to fight all the hacks like using wget or telling the browser to just download the plain HTML file, now it'll all be sent as JSON on fly, you need to have JS enabled and actually load everything, the script keeps sending randomized unique hashes to check if you're really downloading everything and established a proper session instead of trying to just download the text, ruining your user experience in the process), all with very high bitrates and resolution - my users deserve only the best: the most pixels, the most waves, the most data!
if you can't access the site, tough luck bucko, buy a better internet connection or if it's on the server due to the load, wait for another time of the day, check the downtime info, estimate based on the timezones of the users and try to access it during the few hours in a day when the servers aren't overwhelmed - the very limited exclusive access only adds to the perceived luxury, it's only accessible to the people with the best internet connection during only a few exclusive hours of the day (if you're from the US, tough luck bucko, you'll need to stay up all night and keep refreshing; in fact, I'm considering closing off the website for the Shabbat [from Friday's sunset to Saturday's sunset, according to the Israel timezone (the servers will be down) + an additional JS script blocking it according to your local timezone] to be respectful towards religious Jews), people like the feeling of being the select few who managed to access and see the website (all bots are blocked, ofc, no archiving, no cache), the exclusivity makes it precious, we're the Apple of the webdev world: only the select few, dedicated rich people, get to see it, but once you access it, you get the best experience, the website runs super smoothly cos no decompressing is happening on the user's side, everything's pre-rendered and sent as raw data, so it's rendered instantly, a super smooth user experience
exceptional video quality, unfortunately i think i'm too orange cat brain to understand compression 💀
haha ty for taking the time to watch this video though! Any suggestions on how I can make it easier to understand?
@@kikisbytesI didn’t find it particularly hard to understand, and I feel like you did a pretty decent job of explaining compression. I do that wish you went into more depth about what context is and how it’s used to improve the efficiency of streaming compression, though. Otherwise I thought the video was great!
@@kikisbytestoo many technical terms were left unexplained to non-programmers i think.
@@Skaffa Also there weren't enough pauses. Felt like someone was reading a wall of text without line breaks or paragraphs
@@kikisbytes randoguy7488 is right on the money. Your video is great, but not very engaging. The lack of pauses and monotone voice give off a school presentation vibe (no offense). Other than that, the content of the video itself was fantastic and I love seeing deep dives into postmortems like this.
5:48
Woooooow, almost like reducing the amount of data you send in the first place is more important than the compression you apply after the fact. Who could have guessed? Truly the most incredible engineering feat of all time.
This was my takeaway as well lol
You sound salty
@@wormsquirm2707 a little. I take issue with the likes of discord taking away what was once a popular paradigm - people self hosting their VCs with software like ventrillo or team speak. Now all of this user data is going to a company - who knows how it will be used against us long term.
I love the quality and pace of this video - engaging explanation of the underlying concepts, use case story, test outcomes, deployment strategy...all explained in 8 minutes!
Edit: Plus citations and Chapter markers in description too! 🥳
Thank you! Definitely a lot to cover so hopefully I was able to explain the most important concepts.
Man please upload more regularly if possible, just love your content, especially the breakdown of architecture and design decisions of tech companies
I'll try my best!! Might be a little slow with a full time job though, but definitely more to come!
I really like those videos, very informative and interesting. You got a new subscriber.
Really nice video!
big bonuses -> large pizza party hahaha
the goofy ahh sound effects are too contrasting with your bored/tired voice tbh
Thank you for the feedback
The animations look awesome!
One thing discord could do to save bandwidth would be to hash all files and checking the hash against a file database before actually sending the file to the server. Firstly it would reduce bandwidth usage and secondly it would reduce storage usage.
2:20 the resulting huffman codes for a and c are swapped
ahhhh fk me. Thank you for catching this!
Another great video! Keep up the good work!
Thank you!
The video seems to intentionally skip fast while showing text content? You show a graph and then before even i could glance at the axes, it's gone.
Thank you for the feedback! I usually make the animations of how I envision the scene to look like then I try to match it with the voice recording. I find that this 2-3x my development speed. This means that sometimes the animation can be fast because the voice recording section is a lot shorter than I had anticipated.
Will try my best to leave another second or two for you to absorb the graphs.
@@kikisbytes I'm glad to know it's not intended. Great content 👍
@@kikisbytes I thought it was perfect, even at 2x. Honestly really good pace - usually I start a video and scroll to the commemts while the idiot talking takes 9x longer than he needs to to say anything, but this had me actually watching the video. Thank you.
really good creation! I just love such types of videos
Will definitely!! Thank you for tuning in!
Top quality content, do make more like this bro
❤
Thanks! Glad you enjoyed it, more coming soon!
Yeah, engineers won’t be getting bonuses for doing their job.
Such an amazing Video! Thanks
Please make more and more videos of big tech company's system design architecture 🎉🎉🎉🎉🎉
will do!
Great video. I loved the little animations. What software do you use to make them?
Motion Canvas!
they reduced traffic by making the app shitty 👍
This is cool, but I’m still upset. I can’t upload 25 MB files anymore.
i cant even send an pic from my phone without compressing it first now
Good video. But, I thought youll would talk more than the compression algorithms, aren't they only used for the files? Just a question.
I ain't no math guy but that thumbnail makes it look like more than 40%
They got a pizza party and a nice work pal🎉
It's a shame discord is crap now, especially the android app
I get so many glitches during voice calls it's ridiculous. UI buggy as hell
They did it by failing to send 40% of my messages lately
ahahahah I had a good laugh at this!
By getting banned in Russia, haha!
Great video, thank you
good content!
Thank you for watching!
The timing of this right after Russia ban of Discord is just perfect
oh actually? TBH I didn't even know that was happening.
tldr: They switched libraries and removed unnecessary payload.
Classic W switching from zlib to zstd
0:08 costs per what? Week? Day? Hour? It's not cumulative, it's not monthly, the points don't line up with months and looks like it's every 2 weeks but that's a weird unit of time
please make more and more, big tech company videos
Good video, great hook
Appreciate it!
Wait, people still optimize their electron apps? Truly a company of cuture.
isn't all this compressions are built in already in the streaming frameworks / protocols?
6:06 yeah i realized that years before.
i was like, "why?"
And we're stuck to showing clips of maximum 7 seconds long instead of 15 -_-
5:18 company of culture with no linux support ;-;
I'm not smart enough to be here 😄
Well, your curiosity already proves you're more than smart enough to be here! 😊
HEEELLL YEAHH got me rolling for some reason
i'm here till the end although i don't understand a single thing 🙃😅
6:17 will save you guys some time watching the first 6 min: they were sending useless crap to clients. They then stopped.
hahha maybe this should be a short :p
real clap. excellent
Thank you!
40% reduction in traffic huh?
Can I have my 25mb upload limit back discord?
They forkwd it to github?? ON public repo??? They are angels!
cant even send photos anymore, 10mb is too small for most modern camera photos. that change sucks.
for the record, zip drives from the 90s went all the way up to 750mb, a floppy esk disk from 30 years ago can hold more than a discord attatchment, and nitro's 150mb limit isnt much of an improvement, I still have some photos considered too large to send.
the first part of the video makes a lot of sense to me but 5:46 is baffling, why did they do it that way to begin with? seems wasteful and unnecessary
I wish I could improve by 40%
1:35 That's not exactly how it would compress, interestingly. The more efficient way is
A B C (3, 6, EOF)
This looks like "Write A, then B, then C. For the next 6 characters, copy the character that's 3 characters back from the position you're writing."
(0 indexed)
When you write the [3] letter, you're copying whatever letter is 3 positions back, [0]. Then for [4], [1] is 3 positions back. [5] is [2], and this is key:
[6] is copied from [3], data that has not been written when we started copying!
How to reduce bandwith:
Ban more users
Common zstd W
discord lost all of its turkish users that can also skew data
They banned PDF files? 🤔
Interesting
Neetcode pog
hahha love him!
Bc Discord banned in Turkey and Russia 😢
They could reduce traffic bandwidth by even further with p2p networking.
Just send CRC of an image, and then force clients to find the image by themselves with a timeout, if there's no image among the other peers, send one from server.
So in a large servers, the more people saw some image, the more you have peers for this image
2:21 I bet
Some day discord will managed to reduce their bandwidth usage by 100% by making their clients so bloated and with every feature behind a paywall so everyone will just use something else.
So thats why my discord randomly disconnects and is pure TRASH lately
oh is that a consistent issue? Haven't experienced any disconnects
Anyone complaining about the engineers not getting their share... Go start your own company or accept that you're owned by someone else.
Sure that’s how we solve problems, everybody should have their own company. That’s a great Idea, everybody should have their own country too, so they can make their own laws. At that point just have your own planet. That’s a stupid way to think. Just accepting obvious injustices and not unionising is the first mistake.
Please make more and more videos of big tech company's system design architecture 🎉🎉🎉🎉🎉