Open source developers: "Hey guys check out this thing I built in my spare time! It's not perfect but I'm making it freely available so other people can learn from--" Large corporations: "FREE?? 👀👀🥵🥵👀👀"
every time I hear that someones hobby project caused a major outage somewhere I get the feeling that maybe big corporations should maybe check what software they are built on and support it's development/maintenance
Except that software relies on another software, which then relies on another software, which then relies on another software.... It can turn into an endless loop
@@humza890 it can't circular dependencies are usually rare and you can stop looking for dependencies once you've seen it. I also didn't mean, that every company has to look through all of their dependencies and maintain them all, but maybe picking a few or doing an audit of some of them every now and then would be beneficial to not only them, but the world as a whole
I worked at a global e-commerce company a year ago and their platform infrastructure is pretty similar, down to their use of etcd and go channel spaghetti 💀
I still remember the day that it went down, people were blaming Chipotle (american fast casual chain) because they had an event that same day where you could claim a free burrito. People suspected that it was due to a mass influx of people, I knew (and a bunch of people too) that this wasn't an issue with influx of people. At the end of the day, it was a fun journey (more or less with the conspiracies, guessing correctly that it went down for 3 days months before this outage, and youtubers just milking on the outage). Thank you for making a video about this.
@@frezzingaces it was a sort of partnership between chipotle and roblox, so if you installed roblox and did a bunch of stuff you'd get a free burrito. I think that's what it was, roblox has done tons of these
@@baribari1000 Yeah, it was super easy too, you could do it in like 2 minutes on a new account, it gave you a free entrée instead of a free burrito, so you could actually choose most meals you wanted. The few times they did the event with chipotle, I probably earned like 35 or so free entrees, which is pretty decent!
Hi Kevin, amazing content as always! One minor correction @9:54 tho: Go unbuffered channel's length is 0, instead of 1, and it means the sender will get blocked until a receiver receives the value. What the video @9:54 showcase is actually a buffered channel with length 1 (e.g. result of make(chan string, 1)).
This is one of the biggest challenges of modern programming, depending on various 3rd party packages, not knowing what that package is, what it does, or whether it's even reliable, and moreover knowing what are the dependency of that 3rd party package and whether they are safe or not.
Or HashiCorp, being a multi-billion dollar company, could just maintain the fucking project themselves instead of blindly using a 4-year-old abandoned pet project from some random person's GitHub page and trusting it to work in a large production environment.
And that's before the issues of relying on additional 3rd party companies to supply the correct 3rd party packages. Supply chain issues the whole way down.
Imagine how it must feel, starting a free project just as a hobby, and planning to abandon it eventually, then pretty much half the internet starts using it as an important building block to support the web. Now you're just sitting there, and have a choice to make. Stop maintaining the software, and pretty much break half the internet or keep going, getting zero thanks, and zero dollars for your work.
I'd imagine programmer Hell is just a bug like this which takes all of Eternity to fix, also it takes down the company's internal issue tracker and communication system.
5:04 I also heard avatars broke before the whole game went out, and some players were able to play roblox but most of the scripts were missing so it was pretty unplayable. Is it because the game couldnt fetch those? Wow
This happened in the middle of my friends sleepover, when we were COMPLETELY into Roblox. He pretty much just came to play it. We checked like every 5 minutes if it got better. We eventually just slept. THROUGH THE WHOLE THING Edit: Are some of you really watching videos on Roblox and just hate people in the comment section who used to like the game? Find something better to do jeez
@@tbuk8350 Tbh nothing is correct here. Minetest is not trying to be minecraft it's trying to be a general purpose voxel game engine (check out it's other gamemodes there's some pretty unique cool stuff in there)
iirc i was 10 years old when this happened, roblox was the only game i played and i only had my moms phone because i was in a different city (it was painful)
9:45 "A default channel can only hold one piece of data at a time" It's actually even worse than this: an unbuffered channel also requires that this piece of data is received before a send can complete (!)
I love this series. It's like true crime or airplane disaster videos, but it can be fun, because nobody really gets hurt. Except for big corporations and Roblox players, and well... screw them.
3:33 Roblox has a whole dedicated Fast Flag system including Dynamic Fast Flags which can be updated while servers and clients are live and will apply within around 10 minutes. The infustrucutre for this is quite interesting and you may want to touch on it
I’m getting ready to launch my own web application and love watching these videos to learn what not to do. Thanks for making these vids they are very good and informative!
Another great video, I really enjoyed it. There's probably heaps of outages you can do next, but perhaps you could do a video on the "OpenOffice can't print on Tuesdays" bug?
Ah yes, that day in 2021 that i was working in Studio and the toolbox stopped working, and my ass almost had a heart attack because i though i got banned.
I wa sthe first person to play on roblox technically. I was the first person to play because of studio. When roblox was down i was sad and for 2 days i didnt notice that i had roblox studio opened. I had a map opened with cars and buildings. It was uncanny but awesome that i was the first person to play. I had footage on a phone and specifically galaxy fold 3 but the screen broke and i didnt upload the footage. I didnt have an avatar and i couldnt place anything that was free models or paid models but i still had fun. Knowing that no one else could play. Lesson i learned was to always keep studio opened.
Funnily enough, this is probably the most reasonable bug you've covered yet. Who'd think that _Roblox_ of all tech companies would have the most sane architecture
I remember this happening, it ruined arsenal’s Halloween update.. a 2 part update with a boss fight and over 20 items to collect for free became a 1 part update with no boss fight and only 12 items to get. (also for free but the ones we got took awhile to get)
Imagine just doing a Hobby project to understand a piece of Software and suddenly the complete Roblox Infrastrucure is build on it.
Open source developers: "Hey guys check out this thing I built in my spare time! It's not perfect but I'm making it freely available so other people can learn from--"
Large corporations: "FREE?? 👀👀🥵🥵👀👀"
i thought this was a joke,
xkcd 2347
@@xelspeth what is xkcd
@@shantilkhadatkar1195 webcomic. type that text into goold and youll get the comic.
every time I hear that someones hobby project caused a major outage somewhere I get the feeling that maybe big corporations should maybe check what software they are built on and support it's development/maintenance
XKCD 2347
Except that software relies on another software, which then relies on another software, which then relies on another software....
It can turn into an endless loop
@@humza890 it can't circular dependencies are usually rare and you can stop looking for dependencies once you've seen it. I also didn't mean, that every company has to look through all of their dependencies and maintain them all, but maybe picking a few or doing an audit of some of them every now and then would be beneficial to not only them, but the world as a whole
The Unix philosophy of "do one thing" and link against a ton of dependencies was a mistake.
It's what FUTO stands for.
Turns out, this video could be a great introduction to modern backend architecture and development.
I think all of his videos are a good resource for understanding different architectures and subsequently how fragile they can be lol
I worked at a global e-commerce company a year ago and their platform infrastructure is pretty similar, down to their use of etcd and go channel spaghetti 💀
and a great counter-example for troubleshooting....
the leaks are too
the negative 900 million dollars hits hard 😭
Why? Trash game has trash income
@@_GhostMiner its not a game tho
its a game engine and hoster
@@Luna5829 **H O S T*
@@Luna5829 ackhually
@@_GhostMiner plenty of trash games hosted on roblox, plenty of great ones
I still remember the day that it went down, people were blaming Chipotle (american fast casual chain) because they had an event that same day where you could claim a free burrito. People suspected that it was due to a mass influx of people, I knew (and a bunch of people too) that this wasn't an issue with influx of people. At the end of the day, it was a fun journey (more or less with the conspiracies, guessing correctly that it went down for 3 days months before this outage, and youtubers just milking on the outage). Thank you for making a video about this.
Wait. How in tf could Chipolte's traffic affect Roblox's servers. Whats the theoretical connection?
@@frezzingaces it was a sort of partnership between chipotle and roblox, so if you installed roblox and did a bunch of stuff you'd get a free burrito. I think that's what it was, roblox has done tons of these
Oh this happened during that time? Man the memes about the roblox crashes during its downtime were so enjoyable
@@baribari1000 Yeah, it was super easy too, you could do it in like 2 minutes on a new account, it gave you a free entrée instead of a free burrito, so you could actually choose most meals you wanted. The few times they did the event with chipotle, I probably earned like 35 or so free entrees, which is pretty decent!
wasnt there also a massive adopt me update at the time which also probably caused a large increase of active accounts
Hi Kevin, amazing content as always! One minor correction @9:54 tho: Go unbuffered channel's length is 0, instead of 1, and it means the sender will get blocked until a receiver receives the value. What the video @9:54 showcase is actually a buffered channel with length 1 (e.g. result of make(chan string, 1)).
This is one of the biggest challenges of modern programming, depending on various 3rd party packages, not knowing what that package is, what it does, or whether it's even reliable, and moreover knowing what are the dependency of that 3rd party package and whether they are safe or not.
Also never update anything
*If its not a security fix @@Paulo27
Or HashiCorp, being a multi-billion dollar company, could just maintain the fucking project themselves instead of blindly using a 4-year-old abandoned pet project from some random person's GitHub page and trusting it to work in a large production environment.
And that's before the issues of relying on additional 3rd party companies to supply the correct 3rd party packages. Supply chain issues the whole way down.
@@Paulo27 well you want to update everything as much as possible but not too much and with ready rollbacks on hand
This is like the XKCD of all of the world depending on a toy project someone abandoned 10 years ago
probably 2347... as someone mentioned in some comment above.....
"A massive company with ... -$924 million net income" 💀
"Each minute of downtime costs us negative $1750, this must be fixed ASAP!"
@@klafbanglolol
That is absurd lmao
@@klafbang So does that mean they were earning money when they were down? 🤔
@@AlbatrossCommandono, because they still had expenses (developers trying to fix the servers) but reduced income (no micro transactions available).
github repo: "it was a toy project never meant for production"
multibillion dollar company: "YAYEET"
"multibillion dollar company"
_-$924 million net income_
@bruhmoment5974Free Palestine and long live mother anarchy 🏴🚩
@@Mihacappy it is a multibillion dollar company, it's just the multiplier is negitive
@@tirushone6446 i forgot i had even made that comment
@@Mihacappy you're welcome
It’s crazy how much of the internet as a whole is in the hands of solo developers who made a thing in their spare time for fun
xkcd 2347 at its finest
Half of them are furries
@@thewhitefalcon8539 They're braver than any US marine
Oh shit I was gonna suggest this as an idea, awesome to see that you did it!
Lol nice you're here
is that
@@glefyrhello call of duty black ops guy
This is by far my favorite documentary channel on yt
Imagine how it must feel, starting a free project just as a hobby, and planning to abandon it eventually, then pretty much half the internet starts using it as an important building block to support the web. Now you're just sitting there, and have a choice to make. Stop maintaining the software, and pretty much break half the internet or keep going, getting zero thanks, and zero dollars for your work.
the trick to this is option 3, make money of of it
@@Willimann The trick to that is finding out who's even using your stuff.
@@efulmer8675 yea, i guess so
@@efulmer8675That’s not hard. Break it intentionally and see who complains
Option 4: publish all updated versions under AGPL
Thank you for all the work you put into making this!!
Whatever it took to make a video about a Roblox server crash and not use the "oof" SFX even once... I salute it.
It's on 6:45
@@MartijnvanBerkel gottem
6:45
@@MartijnvanBerkel I stand corrected. Frankly, using it only once is even more impressive.
i read this 2 seconds before the oof sound played, well done sir
the kids enter angry
the kids leave confused
Roblox is actually a bigger company than most think. Thanks for doing a video on it.
I'd imagine programmer Hell is just a bug like this which takes all of Eternity to fix, also it takes down the company's internal issue tracker and communication system.
IT global outage vid gonna go crazy
Abstain
Can't wait for the CrowdStrike episode 😀
daily appreciation of kevin's visual style, i love how you're able to break down the language i might take for granted and make it easily followable
Haven't finished the video yet, but this already makes me feel better about the half-day internet outage I fixed at work
Nice technical aspect of the outage!
Well.. now we know the next video
Waiting for the Crowdstrike outage video!
5:04 I also heard avatars broke before the whole game went out, and some players were able to play roblox but most of the scripts were missing so it was pretty unplayable. Is it because the game couldnt fetch those? Wow
KEVIN FANG JUST DROPPED A VIDEO ABOUT THE HALLOWS OUTAGE OH MY GOD
The burrito incident
@@jakfjfrgnei the slippery cord incident
new kevin fang video
today is a good day
This happened in the middle of my friends sleepover, when we were COMPLETELY into Roblox. He pretty much just came to play it. We checked like every 5 minutes if it got better.
We eventually just slept. THROUGH THE WHOLE THING
Edit: Are some of you really watching videos on Roblox and just hate people in the comment section who used to like the game? Find something better to do jeez
Seems like you guys need to find better games
@@3kh0alt Isn't your avatar from a ripoff of Minetest?
@@N30ZUK1 minetest is a clone of minecraft...
@@N30ZUK1 Calling Minecraft a ripoff of Minetest is the most sweaty nerd Redditor thing you could do
@@tbuk8350 Tbh nothing is correct here. Minetest is not trying to be minecraft it's trying to be a general purpose voxel game engine (check out it's other gamemodes there's some pretty unique cool stuff in there)
Roblox players figuring out about the DNS steering and sharing ips for early access is kinda crazy 💀
Crowdstrike video when?
ive heard about that but i dont know exactly what happened so i hope he makes that
@@animating_painssame
Saddest day ever for 7 Y.O i hope they can recovery from this 😢
😂😂😂😂😂😂😂😂😂😂😂
developers probably missed out on millions of dollars too!
“3 days no roblox 😭😭😭😭😭😭😭😭😭😭😭”
iirc i was 10 years old when this happened, roblox was the only game i played and i only had my moms phone because i was in a different city (it was painful)
Thank you for this, been waiting for this one for awhile now!
You gotta make a video about the CrowdStrike outage
I'm glad you made a video on that. I had no idea how it went down behind the scenes! :D
there was an blog post made after the outage
Thanks! I submitted this in as a suggestion a while ago, never thought it’d be published.
yo when is the CrowdStrike video coming
13:00 Love the summoning salt background music when talking about things being slow.
9:45 "A default channel can only hold one piece of data at a time" It's actually even worse than this: an unbuffered channel also requires that this piece of data is received before a send can complete (!)
I love this series. It's like true crime or airplane disaster videos, but it can be fun, because nobody really gets hurt. Except for big corporations and Roblox players, and well... screw them.
That’s a bit harsh on Roblox players… I mean most of them are like 9 years old
@@hagangray8006if they aren’t 9 there’s a 50% chance they’re a predator or another kind of scum
@@absoultethings4213 or… just normal people. Big shocker I know
@@apersoniguess_impossible😱😱😱😱😱
unti money some rando get involved, yeahhh its really fun
yo honey wake up, new Kevin Fang video to watch while at work
Came here to look for crowdstrike, seems like im way too late🤣
I love the way that you explain these complex incidents. You deserve a 冰淇淋🍦
Crowdstrike video incoming in 2 years
its a good day when there ia a new kevin fang video
Kevin Fang, big fan here. Please cover the clownstrike incident
Thank you for the great information and entertainment video like always😊
Damn I was waiting for this one
The Crowdstrike video is going to hit pretty hard
Man I love your videos, this was a particularly technical one, but still really well presented and interesting.
Love these videos please keep them coming!
Great great video, I seriously love the format and I learn so much
> And probably some machine learning and block chain for good measure
lmao nice
The oof sound was a chef kiss to this master piece of video. Great work as always.
Great. Hopefully you'll make a video about Windows bsod due to CrowdStrike
Thanks mate, this is some seriously quality content
can you do a video about the current CrowdStrike Outage?
Waiting for the crowdstrike video
why do i like binging these videos so much
Amazing CGI as always, thanks !!!
"random program some guy made to mess around currently being used by the absolute market top of companies" is a gift that keeps on giving
Very well explained. amazing content. Thanks
This video is very well executed!
Kevin, get busy and make the Crowdstrike video 😂😢
Let’s go bro. CrowdStrike is giving free material to your next video.
Good work, these are both interesting from the tech perspective and just plain fun hah
these videos make me feel like im watching a some type of CSI crime documentary
3:33 Roblox has a whole dedicated Fast Flag system including Dynamic Fast Flags which can be updated while servers and clients are live and will apply within around 10 minutes. The infustrucutre for this is quite interesting and you may want to touch on it
Nice! I wish Roblox never recover from that!
Tons of love for captioning your videos❤❤
In life... you have roblox
(another BANGER kevin fang video, cant wait for the next)
These server incident always feels like a surgery where you have to save a person in its current form ASAP
Awesome summary, as always. Thank you! :D
I’m getting ready to launch my own web application and love watching these videos to learn what not to do. Thanks for making these vids they are very good and informative!
Another great video, I really enjoyed it.
There's probably heaps of outages you can do next, but perhaps you could do a video on the "OpenOffice can't print on Tuesdays" bug?
I like your stuff keep it up make more security related stuff!
Your videos are awesome Kevin!
3 days to figure out that they turned on a bad feature a day ago is actually insane
After this, no one knew how a similar incident affected the entire world 3 years later.
WHENS THE CLOUDSTRIKE EPISODE COMING OUT??? 😂😂😂
Everyone's waiting for Crowdstrike vid
Next video on Crowd Strike update causing global outage!!
It was Crowdstrike, not Microsoft
love your vids! please make more
Ah yes, that day in 2021 that i was working in Studio and the toolbox stopped working, and my ass almost had a heart attack because i though i got banned.
your videos are quality over quantity
Wasn't expecting him to talk about this but man I remember when this happened
finally, a good video on the infamous outage
How fast can you pop out a video? I think there might be something video worthy.
Baby, wake up!
New Kevin Fang lore just dropped 🎉
I think my favorite gag in this video is the "San Mateo" label at 11:20.
I wa sthe first person to play on roblox technically. I was the first person to play because of studio. When roblox was down i was sad and for 2 days i didnt notice that i had roblox studio opened. I had a map opened with cars and buildings. It was uncanny but awesome that i was the first person to play. I had footage on a phone and specifically galaxy fold 3 but the screen broke and i didnt upload the footage. I didnt have an avatar and i couldnt place anything that was free models or paid models but i still had fun. Knowing that no one else could play. Lesson i learned was to always keep studio opened.
Finally, a Kevin Fang video about an outage I was witness to.
Listening to this whilst driving and all of a sudden summoning salt music plays and here I am hoping for a cameo
Great video!
My friend was stuck in a game during the roblox outage and saw lots of things happen
i remember this outage, me and my friends kept playing muck while waiting
Funnily enough, this is probably the most reasonable bug you've covered yet. Who'd think that _Roblox_ of all tech companies would have the most sane architecture
I remember this happening, it ruined arsenal’s Halloween update.. a 2 part update with a boss fight and over 20 items to collect for free became a 1 part update with no boss fight and only 12 items to get. (also for free but the ones we got took awhile to get)
I remember that weekend. Worst weekend of my life.