Knowing how governments work, US probably hired some intern to make a bluetooth API for the famous football (remote launcher that is supposed to be always close to the president)..
The US is replacing all its ground based nuclear ICBMs and there is a LOT of controversy and debate about how computerized their control should be for this very reason. The current control system is to analog it is impossible to hack remotely but IMHO using TCP/IP to launch nukes would be a really bad idea.
It uses basic authentication and the password is just 15 of the "*" character because the guy in charge thought "no one would expect something THAT stupid".
Plowing through can avoid some other classes of errors. Its the fail-open he explains at length though I don't think he noticed he was looking at it. Meant to think as you write scripts. That's also why pipefail or e need to be applied with thought, and not to replace thought. set -u is the only thing that can be expected to fail safe and only for good causes. Still in the end, most of these features ought to never be hit, the script's flow control should be handling errors and not the shell running the script, and design / pre/post conditions should handle data. Blaming a pipe in that case means you just ignore that you wrote a trash script. Poop code caused by thinking shell scripts are not worthy of thought, while they are critical systems code after all. If thinking about this thing, you should not see a cron job for a task like this, you'd want to see another daemon. At the end - learn use pre/post conditions, let your code measure its data quality, learn to write state machines. And then rely on set mechanisms on top. Don't replace good design with it. Shell scripts are important because they can condense various tasks of multiple sources etc., and they are a lot better at error handling, you just mustn't take too many shortcuts. Like tee in a data processing step. Noone on our team would have let you wrote something like that twice.
@@udirt I think cron is a perfectly valid use case for this. While cron is an old tool its a very mature and good tool for what it does. You denying cron kind of reminds me of a coworker that "solved" a problem of docker not running cron correctly by replacing a cronjob with "while true {runscript; sleep X}" loop and causing more problems due to the script not being executed at the beginning of the minute. (The script I made had an assumption that it is executed at the beginning of a minute or else there might be a race condition due to time being checked twice and having to match in both checks).
The way you cover and present these is awesome. I always look forward to new videos from you. I've been in FAANG and adjacent companies, including being on-call during one of the incidents you've done a video on. I wish we could present the post-mortem/COE/whatever by just playing one of your videos.
Blowing up the service while doing batched rollbacks on a known good commit is like the worst nightmare I can imagine😅. I can’t blame them for looking for potentials hackers before anything else.
Imagine writing a whole service in Rust and spend half your code just handling errors and result types, just to call unwrap() and panic on bad input anyway.
That's kinda one of the reasons to use Rust, though. A reliable panic is A LOT better than undefined behavior you're likely to have in C or C++ in a similar situation.
the reason rust is safer is because in rust that's a panic and the software is guaranteed to either terminate or perform a desired action, but in C it's an undefined behavior and the whole software could be vulnerable to a code injection exploit
Love the way you walked through the initial problem, explained what the engineers looked at to troubleshoot and what they did to eventually fix the issue. This is the kind of thing we deal with frequently at work when I am working On-Call, so it was really interesting! Gives people who do not work in this industry more insight into how it all goes down.
It's ironically insightful in the sense that it shows how people that call themselves engineers are now allowing their practices to be informed by totally uncritical viewing of clickbait media.
I'm a simple man. I see a new Kevin Fang video-I click. Serious segment: One of the ideas I had was to not rollback all of the servers simultaneously. That was clearly a bit much.
I'm glad to see that even deep inside one if the world's biggest IT companies there are cron and bash-related bugs just like the ones I make every other month lol
I’m a senior back-end engineer and this is the only RUclips channel I love. Kevin, please, keep doing your great work. There are no alternatives to your content, not even a single channel I can name
@@alex84632Use strict mode in bash or leave pipe for daily terminal usage. Scripts without strict bash options can lead to hard debug sessions like that. For example that issue could be avoided with basic habit of avoid single line / pipes in scritps (same apply for deep sub shells) Example: $ dos-make-addr-conf > config.toml $ dosctl set template_vars config.toml Without pipes an error in the code will trigger an error in the first line avoiding fulfilling cache with an empty value.
Use strict mode in bash or leave pipe for daily terminal usage. Scripts without strict bash options can lead to hard debug sessions like that. For example that issue could be avoided with basic habit of avoid single line / pipes in scritps (same apply for deep sub shells) Example: $ dos-make-addr-conf > config.toml $ dosctl set template_vars config.toml Without pipes an error in the code will trigger an error in the first line avoiding fulfilling cache with an empty value.
00:49 bro got hit hard by those auto updates and had to throw shade at Microsoft 😂😂. To be honest, have had similar experience and haven’t had auto updates enabled in a long time
I think there's a mistake a 8:11. Enabling errexit would not cause the script fail if the pipeline fails because errexit works based on the last exit status of a compound command (i.e. a command using "&&" or "||" among other things). If you run `bash -c 'set -e; (false) && true; echo OK'` you'll see that "OK" is still displayed even though the first command, "false", fails. Compare that to `bash -c 'set -e; (false); true; echo OK'` in which "OK" is not printed.
Additionally, the `… | tee config.toml` part does NOT get “skipped” as the video overlay text suggests. It actually starts up at the same time as the first command, just with its input hooked up to the first command’s output. And because the first command fails, its output is empty, and then `tee` **successfully** overwrites the config file with its (empty) input. `pipefail` does not enable any short circuiting - it just tells bash to consider exit codes for **all** jobs to determine the exit code of a pipeline, not just the exit code of the last job (the default behavior).
In an ordinary command context `&&` and `||` are list operators which form command lists, not compound commands. Compound commands are always surrounded by a pair of either keywords like `if` and `fi` or metacharacters in the case of `()` and `(())` . This aspect isn't particularly relevant to the issue. As a side-note, `&&` and `||` have numerous context-dependent functions. The sections of a `[[ ]]` compound command are delimited by keywords that may include `&&` and `||` and the arithmetic expression grammar has its own entirely separate `&&` and `||` operators that have yet another function.
@@Ormaaj looking at Bash manual, you're right -- compound command isn't the right term, but errexit still operates based on the last command in the series, so the behavior described/shown is the same.
@@ericpruitt2558 The best solution is to never use errexit. pipefail on its own is an ok solution but it's better to avoid needing pipefail if possible. A better non-portable solution is `cmd1 | cmd2; ((!PIPESTATUS[0]))`. Another is to use `
Actually I lied there is one shell that does pipes the right way and it's execline (not even a shell). An execline chain using `pipeline` `piperw` and `wait` can handle this situation beautifully (but it's a little verbose). It is very funny that the simple execline toolset of all things has no problem here while none of the big languages can deal with such simple problems in anything close to an ideal way.
I kind of didn't expect the solution to the rollback causing slowdowns to just be scaling it out (throw more hardware at it), but yeah, that's a fairly reasonable step to take.
I had a team on our Kubernetes cluster configure their liveness and readiness probes with a "; exit 0" at the end of the executed command. Then they called me trying to find out why any pod that stopped responding is not being seen as dead and automatically restarted by Kubernetes. /smh
do note that `pipefail` also means that a pipeline containing `| head` will always fail, because `head` closes stdin after enough lines and most programs handle `SIGPIPE` by hard-crashing
9:13 Or better yet have a 'last known good configuration' fallback as part of the error correction handling. Hypothetical: 0. Frontline is setup as an N-1 Configuration (always one configuration behind DoSD and other daemons.) and will verify legitimacy of the most current config files received before overwriting one in service Redundancy is established for DoSD server and servers running the Kubernetes Script and the Addressing APIs. 1. DoSD detects an empty or malformed quicksilver key -> DoSd throws an alarm and sends a command to Frontline not to overwrite existing config files until service is restored -> Command sent to Kubernetes daemon to restart 2. Kubernetes script needs it's own output sanitatization/error handling and will throw an alarm if the return from the addressing API return is empty or malformed. 3. Addressing API needs redundant servers and Kubernetes will throw an alert if the data from one server varies significantly (or doesn't exist) and run with the majority-agreed output.
The moral of the story is to always think about the edge case when writing code. Error handling has been neglected for so long in the search for “clean looking code” and / or implementing features as fast as possible.
Glad to see you back, king 👑 Love your breakdown videos of tech companies being one inch away from complete disaster - makes me feel like the entire internet is more fragile than it seems (specially with a few companies, like Cloudflare, routing one fifth of the entire global web traffic).
4:04 bad rust developers! Rust specifically makes you handle those types of things, of course, you can ignore that and just panic on invalid data, which is bad practice and should never be pushed to production in that stage
How about I fire the whole team and I’ll implement the shit I want not what the “community” wants. I’m not part of a single one yet it seems like everyone wants to micro manage my life but that’s gonna be stopped.
I always heard it pronounced "day-mon", _specifically_ to avoid semantic overlap with the other kind (something Dwarf Fortress actually exploits, as the ascii tileset character for a... denizen of the circus, is the & symbol, the same thing that denotes a daemon).
I would also argue that the command which writes a value to Quiksilver should not be able to set a blank value. If a value needs to be blank, there ahould be a flag which causes the value to be blanked out. This would make it more difficult to accidentally blank out a value, because you have to explicitly change the command in order to blank the value.
bash pipe always scared me. there is one time I make backup script that works for years, and it suddenly creating 0 byte backup. it took me hours debugging it, not realizing it simply because one part of the pipe broke.
I wonder if pipefail shouldn't be the default. It makes more sense for a command returning an error code to kill the pipeline, but unless each command writes its output to a file somewhere you can't easily chain them to stop at failure points. Also, it kind of sucks that you even need `tee` in the first place because you can't use the shell to redirect to files if you don't have permission to write to those files and tee obviates the need to acquire higher permissions. This is where having a proper programming language would be better, because you could deal with everything in one step, but then it would be harder to write glue code for automation.
5:30 If dos-make-addr-conf returns with an error value, then dosctl shouldn't run. Granted, there will still be a corrupt conf file since tee will execute, but it shouldn't do anything until the dosctl command is run. So I take it that the dos-make-conf command always returned true? Edit 8:04 my question was answered 🤦♂️
And what a great idea to use bash to validate inputs... Seriously, it's just lazy work by the cloudflare developers. This bash command is a shortcut to something which should have way more thoughts into it. Each deployment step is a potential failure, and even more with bash and all theses broken tools
0:39 Not sure if this is an intentional inside joke or genuine misunderstanding, but the "ae" in "daemon" should be pronounced as a long "a" like 'day-mon' and is not meant to be related at all to an evil "supernatural being"; as "daemon" predates "demon" and is more like a guardian angel
Contested, if you look around there are competing valid pronunciation rules; demon, dēm(ə)n, dee·muhn, daymon etc etc are valid in different schools of English. So virtually any reasonable pronunciation is valid unless you are trying to enforce a particular dialect/ruleset.
I completely disagree that a shell configuration was the core fix. The update to the dosd configuration wasn't robust. It should have written configuration to a new variable, verified the configuration, and then atomically swapped out the running config for the new config. Relying on the shell exiting properly on subshell error papers over the fact that the downstream of the pipe was dumb and couldn't detect obvious failures.
Why is the timing of the rollback of the second batch overlapping with the timing of the scheduled cronjob not included as a lesson/potential fix? Shouldn't they have had some sort of preventative measure to account for such a situation, beyond the 4 lessons already described (i.e. some sort of change/release management procedure to catch this)?
As a software dev "nuclear launch code API" is one of the scariest things I can imagine.
I’ll be sure to show you the demo first. -dev
Knowing how governments work, US probably hired some intern to make a bluetooth API for the famous football (remote launcher that is supposed to be always close to the president)..
The US is replacing all its ground based nuclear ICBMs and there is a LOT of controversy and debate about how computerized their control should be for this very reason. The current control system is to analog it is impossible to hack remotely but IMHO using TCP/IP to launch nukes would be a really bad idea.
It uses basic authentication and the password is just 15 of the "*" character because the guy in charge thought "no one would expect something THAT stupid".
@@the-answer-is-42 The password is "password"!
Love that bash by default just doesn't give 2 Fs about your errors and just ploughs on regardless
Just modify the strings to remove all the error codes. Then any shell will plough through
Plowing through can avoid some other classes of errors. Its the fail-open he explains at length though I don't think he noticed he was looking at it.
Meant to think as you write scripts. That's also why pipefail or e need to be applied with thought, and not to replace thought. set -u is the only thing that can be expected to fail safe and only for good causes.
Still in the end, most of these features ought to never be hit, the script's flow control should be handling errors and not the shell running the script, and design / pre/post conditions should handle data. Blaming a pipe in that case means you just ignore that you wrote a trash script.
Poop code caused by thinking shell scripts are not worthy of thought, while they are critical systems code after all.
If thinking about this thing, you should not see a cron job for a task like this, you'd want to see another daemon.
At the end - learn use pre/post conditions, let your code measure its data quality, learn to write state machines. And then rely on set mechanisms on top. Don't replace good design with it.
Shell scripts are important because they can condense various tasks of multiple sources etc., and they are a lot better at error handling, you just mustn't take too many shortcuts. Like tee in a data processing step. Noone on our team would have let you wrote something like that twice.
wait until you learn about sh, then you'll apreciate bash for all the good it does
@@udirt I think cron is a perfectly valid use case for this.
While cron is an old tool its a very mature and good tool for what it does.
You denying cron kind of reminds me of a coworker that "solved" a problem of docker not running cron correctly by replacing a cronjob with "while true {runscript; sleep X}" loop and causing more problems due to the script not being executed at the beginning of the minute. (The script I made had an assumption that it is executed at the beginning of a minute or else there might be a race condition due to time being checked twice and having to match in both checks).
@@udirt I can see that the default behavior would be useful in some situations but you have to acknowledge it's an insane default
The way you cover and present these is awesome. I always look forward to new videos from you. I've been in FAANG and adjacent companies, including being on-call during one of the incidents you've done a video on. I wish we could present the post-mortem/COE/whatever by just playing one of your videos.
God damn the Teams notification
wasn't it slack?
Thou shalt not take the name of the Lord in vain
yeah 😭
@@RanCham727 god damn
Skype too! I miss 2016!
Blowing up the service while doing batched rollbacks on a known good commit is like the worst nightmare I can imagine😅. I can’t blame them for looking for potentials hackers before anything else.
More shit to be broken.
Yeah, I can 100% guarantee that everybody involved said "wtf" when they saw a rollback to good code failing
Imagine writing a whole service in Rust and spend half your code just handling errors and result types, just to call unwrap() and panic on bad input anyway.
That's kinda one of the reasons to use Rust, though. A reliable panic is A LOT better than undefined behavior you're likely to have in C or C++ in a similar situation.
@@Sierra410 I completely agree.
"Oh, I add match here"
.....
"Is that it?"
Good luck debugging same thing in C
the reason rust is safer is because in rust that's a panic and the software is guaranteed to either terminate or perform a desired action, but in C it's an undefined behavior and the whole software could be vulnerable to a code injection exploit
thats actually big chunky safe guard.
Your editing is always absolutely amazing and one of the main reasons I love basically every video on this channel
Yes, so many hilarious small details
putting all those notification sounds in a video dropping on a Saturday is dastardly work
Love the way you walked through the initial problem, explained what the engineers looked at to troubleshoot and what they did to eventually fix the issue. This is the kind of thing we deal with frequently at work when I am working On-Call, so it was really interesting! Gives people who do not work in this industry more insight into how it all goes down.
It's ironically insightful in the sense that it shows how people that call themselves engineers are now allowing their practices to be informed by totally uncritical viewing of clickbait media.
this was very well explained and animated, and funny, without overly intense sound effects or jokes. Got a new subscriber!
I'm a simple man. I see a new Kevin Fang video-I click.
Serious segment:
One of the ideas I had was to not rollback all of the servers simultaneously. That was clearly a bit much.
I'm glad to see you're well Mister Fang.
I'm glad to see that even deep inside one if the world's biggest IT companies there are cron and bash-related bugs just like the ones I make every other month lol
F*cking up a shell command? Couldn't be me!
I’m a senior back-end engineer and this is the only RUclips channel I love.
Kevin, please, keep doing your great work. There are no alternatives to your content, not even a single channel I can name
I've leaned a long time ago to not have stacked pipe commands in scripts. It makes for a fun troubleshooting time..
So what do you use instead? Process substitution?
@@alex84632Use strict mode in bash or leave pipe for daily terminal usage.
Scripts without strict bash options can lead to hard debug sessions like that.
For example that issue could be avoided with basic habit of avoid single line / pipes in scritps (same apply for deep sub shells)
Example:
$ dos-make-addr-conf > config.toml
$ dosctl set template_vars config.toml
Without pipes an error in the code will trigger an error in the first line avoiding fulfilling cache with an empty value.
Use strict mode in bash or leave pipe for daily terminal usage.
Scripts without strict bash options can lead to hard debug sessions like that.
For example that issue could be avoided with basic habit of avoid single line / pipes in scritps (same apply for deep sub shells)
Example:
$ dos-make-addr-conf > config.toml
$ dosctl set template_vars config.toml
Without pipes an error in the code will trigger an error in the first line avoiding fulfilling cache with an empty value.
00:49 bro got hit hard by those auto updates and had to throw shade at Microsoft 😂😂. To be honest, have had similar experience and haven’t had auto updates enabled in a long time
Group policy ftw
Babe wake up new site event just dropped
Not anymore.
Thank you for the opportunity.
I smashed that like button really hard.
Your channel is a true gem 💎 I celebrate every new upload - keep it up!
I think there's a mistake a 8:11. Enabling errexit would not cause the script fail if the pipeline fails because errexit works based on the last exit status of a compound command (i.e. a command using "&&" or "||" among other things). If you run `bash -c 'set -e; (false) && true; echo OK'` you'll see that "OK" is still displayed even though the first command, "false", fails. Compare that to `bash -c 'set -e; (false); true; echo OK'` in which "OK" is not printed.
Additionally, the `… | tee config.toml` part does NOT get “skipped” as the video overlay text suggests. It actually starts up at the same time as the first command, just with its input hooked up to the first command’s output. And because the first command fails, its output is empty, and then `tee` **successfully** overwrites the config file with its (empty) input. `pipefail` does not enable any short circuiting - it just tells bash to consider exit codes for **all** jobs to determine the exit code of a pipeline, not just the exit code of the last job (the default behavior).
In an ordinary command context `&&` and `||` are list operators which form command lists, not compound commands. Compound commands are always surrounded by a pair of either keywords like `if` and `fi` or metacharacters in the case of `()` and `(())` . This aspect isn't particularly relevant to the issue.
As a side-note, `&&` and `||` have numerous context-dependent functions. The sections of a `[[ ]]` compound command are delimited by keywords that may include `&&` and `||` and the arithmetic expression grammar has its own entirely separate `&&` and `||` operators that have yet another function.
@@Ormaaj looking at Bash manual, you're right -- compound command isn't the right term, but errexit still operates based on the last command in the series, so the behavior described/shown is the same.
@@ericpruitt2558 The best solution is to never use errexit. pipefail on its own is an ok solution but it's better to avoid needing pipefail if possible. A better non-portable solution is `cmd1 | cmd2; ((!PIPESTATUS[0]))`. Another is to use `
Actually I lied there is one shell that does pipes the right way and it's execline (not even a shell). An execline chain using `pipeline` `piperw` and `wait` can handle this situation beautifully (but it's a little verbose). It is very funny that the simple execline toolset of all things has no problem here while none of the big languages can deal with such simple problems in anything close to an ideal way.
OUR YEARLY KEVIN FANG UPLOAD!
Great video, very well produced, and with a quick explanation of some informations, perfect!
Excellent analysis. Thanks for the detailed breakdown.
Great to see you back with new video, commenting before watching it 😊
Learned more about CS from Kevin Fang's videos than I ever did in college
You definitely weren't paying attention boss
04:30 I love how the nuke has the cloudflare logo in it.
I was desperate for more videos. Great job and thank you!
I kind of didn't expect the solution to the rollback causing slowdowns to just be scaling it out (throw more hardware at it), but yeah, that's a fairly reasonable step to take.
no fucking way new upload
I had a team on our Kubernetes cluster configure their liveness and readiness probes with a "; exit 0" at the end of the executed command. Then they called me trying to find out why any pod that stopped responding is not being seen as dead and automatically restarted by Kubernetes.
/smh
do note that `pipefail` also means that a pipeline containing `| head` will always fail, because `head` closes stdin after enough lines and most programs handle `SIGPIPE` by hard-crashing
9:13 Or better yet have a 'last known good configuration' fallback as part of the error correction handling. Hypothetical:
0. Frontline is setup as an N-1 Configuration (always one configuration behind DoSD and other daemons.) and will verify legitimacy of the most current config files received before overwriting one in service
Redundancy is established for DoSD server and servers running the Kubernetes Script and the Addressing APIs.
1. DoSD detects an empty or malformed quicksilver key -> DoSd throws an alarm and sends a command to Frontline not to overwrite existing config files until service is restored -> Command sent to Kubernetes daemon to restart
2. Kubernetes script needs it's own output sanitatization/error handling and will throw an alarm if the return from the addressing API return is empty or malformed.
3. Addressing API needs redundant servers and Kubernetes will throw an alert if the data from one server varies significantly (or doesn't exist) and run with the majority-agreed output.
Really nice visuals!!!
I love that Bloons metaphor and repressing actors as amogus
That PagerDuty ship bells sound almost triggered my PTSD.
2:56 Lol, gotta love pop. "How to tell if a girl likes you" top suggestion
The moral of the story is to always think about the edge case when writing code. Error handling has been neglected for so long in the search for “clean looking code” and / or implementing features as fast as possible.
Cant wait for him to cover the wayback machine incident
Waiting for Crowdstrike outrage video :D
The second image of something broke is insanely crazy
lol the part when typing for search results in bing, love the funny video editing
Automatic P0 Incident is a trigger I didn't know I had
Glad to see you back, king 👑
Love your breakdown videos of tech companies being one inch away from complete disaster - makes me feel like the entire internet is more fragile than it seems (specially with a few companies, like Cloudflare, routing one fifth of the entire global web traffic).
most of the time we actually have no idea what we're doing, or what the last guy did
This reminds me of what I learned a few years ago from taking some Google Cloud Platform and Google Kubernetes courses
4:04 bad rust developers! Rust specifically makes you handle those types of things, of course, you can ignore that and just panic on invalid data, which is bad practice and should never be pushed to production in that stage
nice video 👍 and editing
That PagerDuty alarm spiked my blood pressure
@6:03 right in the C64 nostalgia feels
???
@@8bits59 Sound effects from the old game, Boulder Dash :-)
How about keeping the current cache if the new value is "null" too as a simple precaution ?
How about I fire the whole team and I’ll implement the shit I want not what the “community” wants. I’m not part of a single one yet it seems like everyone wants to micro manage my life but that’s gonna be stopped.
Dude I love these vids, please make more!!!
I always heard it pronounced "day-mon", _specifically_ to avoid semantic overlap with the other kind (something Dwarf Fortress actually exploits, as the ascii tileset character for a... denizen of the circus, is the & symbol, the same thing that denotes a daemon).
OMG he finally CAME back.
I hope we get episode on the crowdstrike f-up soon
Dinners gonna taste real nice with this one 🔥
The Slack knock-knock-knock has scared me 🤧
So glad this guy returned
Miss these videos!
HES BACK
Love how global companies run solutions similar to what I do at home, bunch of scripts in Kubernetes cronjons. 😅
We are prepared for the job already 😂
This reminds me of a time I was asked to remove error handling because it made the code "too cumbersome". Please handle your errors 🙏
amazing content. love the humour
kevin vid just dropped🗣🗣🔥
I would also argue that the command which writes a value to Quiksilver should not be able to set a blank value. If a value needs to be blank, there ahould be a flag which causes the value to be blanked out. This would make it more difficult to accidentally blank out a value, because you have to explicitly change the command in order to blank the value.
Wow, i really shiuld order quicksilver now!
hell yeah new kevin fang video
bash pipe always scared me.
there is one time I make backup script that works for years, and it suddenly creating 0 byte backup. it took me hours debugging it, not realizing it simply because one part of the pipe broke.
How dare you flash a Teams notification, I had a heart attack bruh
Excellent, let me get my popcorn!
I got the butter right here *pulls out the spot between his thighs and scrota* extra buttery just like you like.
Oh thank God, I've had food in front of me for like 20 minutes just looking for a video to watch.
Have fun fasting like it’s Ramadan.
HE'S BACK! I'LL GET THE POPCORN 🍿
That mushroom cloudflare looks awesome.
new kevin fang video just dropped
Bloons mentioned!!!! 🎈🎈🎈🙊🙈
2:22 Swear I get PTSD with that pager duty sound 😂
Babe wake up, Kevin just dropped a new video🗣️
Thank you Kevin
HELL YEAH KEVIN FANG UPLOAD
I wonder if pipefail shouldn't be the default. It makes more sense for a command returning an error code to kill the pipeline, but unless each command writes its output to a file somewhere you can't easily chain them to stop at failure points. Also, it kind of sucks that you even need `tee` in the first place because you can't use the shell to redirect to files if you don't have permission to write to those files and tee obviates the need to acquire higher permissions. This is where having a proper programming language would be better, because you could deal with everything in one step, but then it would be harder to write glue code for automation.
He's back!!
Yay, the funny videos are back!
5:30 If dos-make-addr-conf returns with an error value, then dosctl shouldn't run. Granted, there will still be a corrupt conf file since tee will execute, but it shouldn't do anything until the dosctl command is run. So I take it that the dos-make-conf command always returned true?
Edit 8:04 my question was answered 🤦♂️
MY GOAT POSTED AGAIN 🗣️📢🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥💯💯💯💯💯💯💯💯💯
Invalid input data is always the problem.
Please validate your inputs, all of them.
And by validate we mean let this third party app/extension do it for you. Not anymore fags.
Ever, the async job putting empty objects is a old friend.
And what a great idea to use bash to validate inputs...
Seriously, it's just lazy work by the cloudflare developers.
This bash command is a shortcut to something which should have way more thoughts into it.
Each deployment step is a potential failure, and even more with bash and all theses broken tools
0:39 Not sure if this is an intentional inside joke or genuine misunderstanding, but the "ae" in "daemon" should be pronounced as a long "a" like 'day-mon' and is not meant to be related at all to an evil "supernatural being"; as "daemon" predates "demon" and is more like a guardian angel
Contested, if you look around there are competing valid pronunciation rules; demon, dēm(ə)n, dee·muhn, daymon etc etc are valid in different schools of English. So virtually any reasonable pronunciation is valid unless you are trying to enforce a particular dialect/ruleset.
latin æ is traditionally pronounced /iː/ in english, from middle english /ɛː/ as in meat (now /iː/)
We don't have a lot of bash but when we do I start with `set -eu` probably time to add pipefail to that
Oh wow I loved watching this
So for 30 minutes, ddos attacks were possible?
NEW KEVIN FANG VIDEO DROPPED!
yay new upload from the legend
4:26 The sleeping dosd is so cute 😆
new kevin fang video 🎉
I completely disagree that a shell configuration was the core fix. The update to the dosd configuration wasn't robust. It should have written configuration to a new variable, verified the configuration, and then atomically swapped out the running config for the new config. Relying on the shell exiting properly on subshell error papers over the fact that the downstream of the pipe was dumb and couldn't detect obvious failures.
Babe, wake up, Kevin posted a new video
Woohoo more Kevin Fang!
Why does Kevins last name have an N in it?
Very good video.
KEVIN! YOUR BACK!
Imma watch it first but I am sure it is still as great as your other vids going in depth to the technical details
what about his back?
I hope his back is ok.
kevin fang is KING
Yessss, New Kevin Fang
Love this videos.
Why is the timing of the rollback of the second batch overlapping with the timing of the scheduled cronjob not included as a lesson/potential fix? Shouldn't they have had some sort of preventative measure to account for such a situation, beyond the 4 lessons already described (i.e. some sort of change/release management procedure to catch this)?