My heart breaks when hearing all this new info about Lasse Collin. His companions betrayed him and now he is there alone, unable to trust anyone, battling mental issues and still trying to clean up one of the biggest security fiascos of the decade :(
@@lfarrocodevvery likely and hopefully this spurs more active efforts in taming the community around maintership. Not just allowing giant companies to freeride on over worked underpaid contributors. Same for people who use these projects on smaller scales but abuse and exploit the work of maintainers. It’s literally a security risk now if there was a coordinated social engineering attack on this poor maintainer of you allow these projects to go on with shoe string support
@@lfarrocodevthat’s very likely. Most of those accounts were just disposable accounts created with the sole purpose of posting the nasty comments to pressure the original maintainer
They probably fucked with him behind the scenes on alt accounts etc. If they go this deep not beyond messing with his socials and so on, to cultivate him as a vulnerable target
Hot take: Everyone has been saying that this is proving the short comings of OSS, I think the opposite. If some idiot got themself a job at MS and did something like this, you would see that PS-Remote or perhaps RDP takes an extra half a second and say "what did MS do now??" and move on with your day. The beauty of OSS, the ability for someone to look at the code did what it needed to do: Someone who had nothing to do with the project was able to look at the code and sound the alarm.
exactly. you cant peek behind the curtain with proprietary OS'es. With linux and OSS, anyone can view or modify code. Thats the whole reason this was even found in the first place.
💯💯💯💯 people failing to see how good this is. How would MS even be able to tell if someone put in a backdoor in anything closed source, it would be impossible to spot ten seconds of performance lag on anything from MS, let alone half a second.. and noticing one more backdoor among the dozens of 'telemetry' backdoors they already put in there? Nah.
Unrelated, but temple OS is actually a lot cooler than a people think and it is basically the perfect OS for learning about hardware, as it's complete lack of permissions and it's unique paging setup makes working with hardware very easy, also the fact you have the holyC REPL, you can experiment without friction. Seriously go take a look at zeal OS (a modern port) and start messing around with it
Having been a DB Engineer and having been blamed many times for poor performance when it actually it was some half-bright code monkey we get very thorough and very cranky.
the scariest part is the social engineering did on Lasse. this person was manipulated for YEARS and the team (probably) behind it saw the opportunity and exploited it. exploiting Lasse's mental health, trust and desire to pass on the torch. this is actually evil
@@SpiDey1500 my god, didn't even think about it but they could totally have had accounts sending him hate comments over the xz utils not getting enough updates, which caused him to want to find someone else to take it over.
@@themodfather9382 absolutely no evidence Lasse Collin did anything wrong, while I see Andres Freund being congratulated with quips like "lifetime free drinks", which makes treatment of Collin even more stark contrast. No it's not "life" to throw wild accusations. It's just more prejudice against "mental health issues" being not real, so he must be suspect. Really uncool to suggest this.
This whole situation just feels like a movie. The fact that this is real is insane. And I can't decide what's more impressive - developing this backdoor or finding the backdoor ... this just shows me how little I actually know. I feel vulnerable ... just let me cry...
It feels like someone held a genius coder hostage to develop the backdoor and then the criminal half-assed the distribution of the backdoor (using accounts that were just created to push for inclusion, come on! That's like composing Beethoven's Tenth and then playing it on a glass bottle.).
Putting aside the likely-nefarious intentions of the chef, I appreciate that there are minds out there that can think like this and produce technical artwork of this caliber with such a niche medium, it’s really impressive.
The pushing might be because there are 2 other things happening that are each likely to kill the attack chain. 1. openssh was already working on their own method for calling systemd-notify without linking it (upto now they did not link it due to them being very careful on dependency checking). Debian, Fedora and OpenSuse were patching sshd to do this linking. This is how liblzma got linked to openssh at all, this wouldn't be done anymore. 2. systemd is looking at better isolating and reducing their dependencies, especially for more critical parts of the system themselves and liblzma is looking to be dropped as a dependency. Given these things, this backdoor may have been on a sudden clock where it is get it in next release or it is likely to be several years of setup for nothing.
I was also thinking once in the wild they're on a time limit before someone notices it so as soon as the malicious code was merged they need to infect as many machines as possible before that happens.
@@chilverscthat is always a risk, there is a chance that the moment this backdoor is actually first used in earnest it would set off some intrustion detection of why are we suddenly getting an ssh connectiong from some foreign country where we don't have any offices or something.
28:25 The obvious reason for the rush is probably a branch+ticket+PR in systemd repo to switch the library loading to runtime and be optional, and this looks almost ready. Just imagine, the hard work of many years to be flushed down the sink.
There's an episode of The Sopranos where the FBI spends the entire ep putting a bug into a desk lamp and then planting the lamp in Tony's basement where he talks business with his associates. They only capture a single conversation, of Tony talking to a plumber about his water heater, before the whole scheme is undone by Meadow grabbing the lamp and taking it to her dorm room. Feels like a good metaphor for this guy's exploit getting caught so quickly.
Or that time the CIA spent millions putting a microphone and radio transmitter into a cat, deployed the cat, and it immediately ran into a road and got hit by a car.
@@superscatboyhey, it was a decent PoC. The good thing about this situation is now people can really see some techniques one might use to obfuscate and maybe people will be able to look out for similar situations. However there's also tons of different commands and techniques to do essentially the same thing, but it's all patterns. Also people might take testing code more seriously.
I think this backdoor ultimately is going to do more good than harm, because now people are on the lookout for backdoors in tests and similar wild exploits.
You dang right about that. My last 3 days have been auditing and catching up on dependencies and scraping the git projects' comments. So many eyes are on it now.
My speculation is that the person is not Chinese. The information that the name had mixes of Mandarin and Cantonese makes it sound more likely that it’s a non-Chinese person attempting to create a Chinese identity. I think it’s a very clever ploy to leave digital breadcrumbs that align with people’s existing beliefs. People want it to be a big grand Chinese cyberattack, so by intentionally choosing a Chinese sounding username people will immediately jump to that conclusion. We obviously can’t rule out the potential of it being a state sponsored cyberattack and perhaps even a CCP coordinated attack. But I think it’s important to be aware of our existing confirmation bias
I think they picked the name as another test of how easy it would be to sneak in malicious stuff. People in the US would be very suspicious of Chinese contributors, so an actual attempt to be as sneaky as possible would probably use a French or Swedish name. Look at the first vulnerability in 2021, replaced a secure function with an insecure one while having an apparent Chinese name, and it got through. That was part of the test.
@@magicmulder I like this theory, it makes a lot of sense. It’s like the Nigerian Prince emails where they are sending obvious scams to filter out people that are smart enough to recognize the scam
@@magicmulder Makes a lot of sense. This theory is similar to how email scammers purposefully make the scam more obvious to filter out the people smart enough to avoid getting scammed. So this could be a similar thing where they tested the waters before commiting the time needed to create the backdoor
Having extremely complicated bash scripts that modify files during the build step is kind of wild in 2024. I'm not sure why you'd even set a build system up like this, seems like hell to work with, let alone audit for security.
Let's be very clear: he already was the active maintainer of the project, if he wasn't the code would probably not have been accepted. This was only possible because he played the long game. Assuming it's even a he, not a she or a group, etc.
@ChuckNorris-lf6vo yeah I just asked chatgpt about how to fix the current state of the Open Source Community and yeah I totally agree these guys are wasting their time AI
if you were given 20 issues from 20 different repos and asked to address them in a week, realistically speaking you might get maybe 20%-25% done in that amount of time if you are that good. And that's probably the best you can ever do, but then imagine the 13% accurate guy can one day do 50% or more with an upgraded model
@@bugzpodderBut then you waste 87% of the maintainer's time. Because they can only tell if a contribution is bad when someone looks into it deeply. It isn't better for open source if (even *if* the model is 50% accurate) half of the PRs maintainers need to go through are plausible looking but actually don't work
@ChuckNorris-lf6vo they're being sarcastic. Both NeverTrust298 and mhdmuzaffar-vr5mb. I don't think that counts as trolling. Unless you were replying originally to mhdmuz* in earnest, in which case NeverTrust may be trolling you in specific.
@@edgars9581 sure. but the submitter doesn't have to be a bot that submits everything the AI cooks up. it could be a dev who is just using it as a tool to surface potential bugs, then only submits the legit ones. they could also use the AI to help them come up with legit repro cases, and possibly a new test case. it isn't really much different than fuzz testing in that way. or plain old static analysis/compiler warnings.
For me, the big problem that this has exposed is the vulnerability inherent to the OSS / Linux / GNU building and packaging systems. It's an arcane mess of Makefiles, Bash scripts, ad hoc patches, and tar-ballz inside tar-ballz. It's long overdue for some security to be built into all that, like properly sandboxing builds versus tests, and having verifiable steps. In this particular exploit, it looks like a crazy mess of bash magic, but ultimately it's scary because of how easy it was.
it exposes a psychological weakness in test code really. It's mind tormentingly bureaucratic and boring and our mind just defaults to "looks good to me" ... "ah yes, random shit... that looks like good random shit to me" and "tests PASSED".
@@jamesnewman9547 Pressuring open source maintainers, seems like a great idea. Fundamentally, the software is provided _as is_ (as is stated in the MIT and GPL licenses). Solving this problem starts with finding a solution to better compensate OSS developers for their work and getting more eyeballs on barely maintained software
The way that open source projects are worked on; at least you can find them in open source code. This compromise began when the attacker did a Social Engineering attack to get onto the project and add these commits In closed source code you'll never know you were compromised
7 месяцев назад
Imagine! I remember that years ago the curl author shared a tweet about some guy saying something like "Thanks to curl (codebase) I can enjoy all my CVEs bounties"...
Almost definitely a nation state. Lots of carefully crafted obfuscation & social engineering. I think over all this is a very strong argument for reducing our reliance on shit build systems
We're not going to narrow it down until we do heuristics on the accounts. Right now the bad guys are scrubbing and even scrubbing sticks out like a sore thumb. If it is a state or gov't sponsored effort, then a deal will be made behind closed doors.
Compression algorithms do a lot of data deduplication so a real test file will have duplicated data to prove that the algorithm actually works. Files with high entropy don’t benefit much from compression. Just noting this as it would be expected for test files on a compression library to have that kind of repeated/duplicated data. If I were an attacker I might theorise that adding data to a test file would be less conspicuous if the added data compressed effectively 🧐
Attackers didnt have much time left, as xz dependency was about to be removed/lazy loaded from libsystemd, breaking the backdoor. Might be the reason why they pushed for it.
All major security agencies should be after the perpetrator(s). The caliber is HUGE. If those guys aren't caught and the whole thing is silenced then it must have been state sponsored.
Suspending the original maintainer with appropriate explanation could be net positive regardless of if he was intentionally involved. Sometimes a forced break from things is good (also might keep him from getting hate mail while things are hot)
This opens up a whole new world of attack vectors. Even just the proliferation of this one aside, we have no way of knowing just how broad the compromise is. Scary shit.
not really, first of all stop building half the operating system off of tarballs which arent peer-reviewed, OR actually inspect and scrutinize what's in them, especially some big "testfile_good-trust-me" binary which is loaded during the build process, for absolutely no reason at all.
41:03 One note on chinese name things: many groups do speak multiple dialects of chinese, particularly in areas like Malaysia or Singapore (where the Tan last name would be used in Hokkien communities) or other areas with large dispersed chinese populations. My fiancée’s family, for example, primarily speak Mandarin and pronounce their chinese names in Mandarin, but use the Hokkien anglicization of their surname. So, while it’s a good to look at, it’s not necessarily indicative that the Jia Cheong Tan name is fake.
Being this backdoor so much complex, I highly doubt it is being implemented for the first time. From start to end, everything seems well crafted and maybe improved on the possible previous iterations.
Some other comment pointed out that some lazyloading dependency of XZ in systemd would soon have been removed, and this is most likely what they were interested in.
Honestly, seeing how much effort was put into this makes me think the guy who did is simply a madman. Like lots of steps could be skipped with same effect. But guy wanted to prove a point and flex his genius on everyone
Also zero preparation for the actual push to get it included in distributions. 3 years of backdoor preparation and then they use two freshly created accounts to push distributors? Sounds incongruous to me.
exactly, theres gaurenteed to be russian and chinese spies in every major US tech company. No doubt about that. Who knows what sort of damage they are doing. Netflix probably has a spy as well, probably a streamer too, as a disguise.
@@gileeeDave Plumber had made at least 1 video about this, there are significant QC checks in place at Microsoft. You’d need multiple people on the inside, which is possible. But much much much easier in open source
@@lucasjames8281 the thing is microsoft can spare like what a couple thousand engineers? open source you have a much bigger population, it doesn't matter what the backdoor is, it'll get found out. good luck finding this kind of issues with just a few hundred to thousands. Then there is the elephant in the room, what of microsoft approved backdoors? they're not your friends either
If this happened inside of a large proprietary C/C++ code base, for example a foreign independent contractor with a fake identity at Microsoft or Riot Games was compromised and committed a malicious tar ball, most of the country would be compromised and almost no one would have the ability to find the issue. I don't think businesses are immune from attacks as sophisticated as this. At least with open source we have a chance to find the backdoors.
Great analysis, thanks for going through it! This is truly scary stuff! it really makes you think how much stuff is out there actually compromising open source software that we're not aware of... 😢 Consider the following: this was only caught because of increased delay introduced by the exploit code. Now, what would have happened if whatever actors who cooked up this mess added a simple delayed activation logic? The exploit would be everywhere and likely no one would have been the wiser Scary scary shit
This is so obscure I'm getting paranoid about the guy who even found the bug in the first place... my brain is like, "oh HOW CONVENIENT, you just simply stumbled on that!?" but then just has nothing to put after that. ... like maybe this was a compromised APT that was already under observation and "discovering" the backdoor was just a parallel construction--a way to expose it without exposing that they have a peep-hole into the APT's activities.
@@IronicHavocIt's OK bro. I've learned to type real quietly so they can't hear my keystrokes through the matrix. It's everyone else I'm worried about. TRUST NO ONE.
if the NSA wanted to expose it they wouldn't need to find some guy to claim he found it.. they could do the most Chad git move ever and make a new anonymous account and submit a PR on the repo titled "Fix backdoor introduced by.." That'd be the most epic commit ever..
Seeing a lot of commentary on this issue pointing out how catastrophic this *could* have ended if it weren't for Andres' diligence. While that is of course true, the takeaway from this cannot be the story of how one very knowledgeable and detail oriented man saved the world. The discovery of a sophisticated, catastrophic RCE like this *necessarily* requires unlikely circumstances. If the attack was not discovered through these unlikely circumstances, we would never know how sophisticated and catastrophic the attack is. Conversely, if the attack was not sophisticated, it would not require unlikely circumstances to discover. Therefore, it is expected that catastrophic and sophisticated attacks will be discovered through unlikely circumstances. This is something like the anthropic principle for cybersecurity. The real takeaway here is that the more effective and catastrophic an attack is, the more unlikely you are to discover it.
@@kiwikemist Apparently you don't know what Russia did then or what "lib" even refers to. But no, this is espionage by an organized group and it is targeted. we don't know yet who made it, but the list isn't that long. Learn what words mean and try again.
I think it was Richard Stallman who warned us about this kind of thing the 1960s!. It’s one of the things that is supposed to make Open Source software more secure than proprietary software. But the price is eternal vigilance.
On the other hand, If something like that happened in a proprietary codebase, nobody would even notice because they wouldn't have access to the source code
Some binary can't be reproduced with code, like image files for example. Those were never compiled, they were just created. But even an image could have code in it.
I believe GH repo was blocked so that automatic build systems don't pull tars from there. Despite that Lasse Collin stated that GH repo is unaffected, who knows?
Would isolating the build and test environments (ie via containers) limit this class of attack? Might take longer to build, but if the test suites can't touch the binary that is going out, then the injection should be impossible, no?
I get the "no comments in my code" policy, but whenever I see expressions like this -> (49:10) -> it kinda starts falling apart for me... I think in such a case it really starts being nearly crucial to comment. Not even necessarily "what" you're doing, but more importantly "WHY"!
12:40 this case may or may not be state affiliated. but it's extremely obvious there are state actors who are intended to deploy subtle bugs into widely used software.
Question: would the backdoor still be relevant if SSH is disabled? Most linux Desktop users do not have SSH enabled, so this would mean the target was entirely servers.
48:59 In the team where I work, when you do code review and accept the code, you'll be the one fixing the bugs when the original author is on the holiday. That results in automatic "nope" whenever some piece of code cannot be understood. A code like the crap in this m4-file is clearly either an exploit or totally unmaintainable. Either way, it doesn't get to live in the official master branch. I'm not sure if I'm just old enough but the bash scripts seemed easy to understand compared to the m4-stuff.
The thing is precisely that it did not live in the master branch, it was *only* present (added by hand) in the source tarballs. And I guess nobody bothered to check if the source tarballs actually matched the source code from the repository.
"The Three Body Problem" is the best sci-fi book I've read in years, hands down. Also, I refused to get the sequels because the first book freaked me out so much, and I know things don't actually get serious until books 2 and 3.
I don't think The Primagean is right about compression code being inscrutable. I think it's investigatable in it's raw form, and LLL is right that that's how it should be committed, or at least SHA256 hash addressable back to compilable code.
Shouldnt `binary_blob | manipulation | eval` be a red flag that could maybe be scanned for somewhat automagically? Not sure if I understand everything here though.
Simple proposition: Make things be more human friendly for some crucial items. For eg, at vulnerable/failure points have understandable code, which I prioritised as chip verification engineer. Everything we do has to send data as binary & get it as binary, so its good practice to have understandable code for someone else
I watched his video, some of his assessments in his video are definitely wrong, but I think it all comes down to: you are accepting code from random person on the Internet. That code possibly needs more attention than some of that in your own company. But notice how with the XZ hack they played the long game, to become the new main contributor. NSA does the same when they infiltrate a company. The difference is basically 0.
This is why I leave my repos on private most of the time. I've never been totally sold on open source; I have been on the bad end of unreasonable expectations too many times.
@@IronicHavoc even if they were individuals themselves, it would still be like 3 at most lol, nothing to cry over. ive seen women get more hate under instagram comments.
Whilst not specifically this exploit, this is why I have been so anti binary repositories for years. That does not mean always building from source is safe; clearly this could have been put into source and still have been missed. However, the main takeaway is not the details of the exploit itself but how once again "social engineering / psychological" attacks against one or a limited number of dedicated individuals enables bad-actors to fly under the radar. Small cogs in a big machine are important. We need to support them!
Agree that any binaries should not be included true and if they do, they should not try to take data from it. It should just be: test is OK if it can be decoded. Test fails if it can't be decoded. But let's be very clear, they played the long game and were the active maintainer. So they set the policy of the project. Sadly, this should mean changes to the processes of the Linux distributions. They should look at the parent project and point out problems that don't match the policy of the Linux distribution which is to not take in stuff which can't easily be audited/simple aka fit for purpose.
I think the key takeaway of this situation should not be "Oh don't commit binary data to a repository" but rather "how do you enforce integrity of the released build to the actual source code + don't do magic in build systems". 1. For the integrity part, this exploit only worked this way by modifying the build file where the archive got depbfuscated, so if the release tarballs would have been exactly the same as the repos, the payload for the exploit would have only been gibberish to begin with. 2. I think they would also get away with it if the build script would have been committed to the source code. Yes, it would be more obvious, but realistically, nobody would have audited a complicated build like this where sed, tr and other magic commands are normal.
They really shouldnt be normal parts of a build though. There's essentially 0 reason to have those as part of a normal build. The only time I can think of is "here is a plaintext/image file that is perfectly legible, this command will compress and corrupt the file" for the bad case instead of submitting a binary blob
@@TurtleKwitty You are right, modifying a good file would be the best if possible. But still only solves one of the holes, because the release tarball still contained changes to the original source and could have made any change to any file, from build scripts to source itself
I don't think this is related to Open source specifically. This could happen even in commercial software. Nothing in the source, everything is split between the tests and the build system!
Agreed, I do think maybe the git repo with the normal code should be separate from the repo with the test-code. And both should not be run in the same environment. Only the result of the build (without test-cases) should be packaged.
@@khalilzakariazemmoura8995 the scary part is, is the real issue: the active maintainer was the compromise. I really hope Linux distributions and package maintainers take a couple of lessons out of this. They are the most important barrier after code review by the people directly involved in the project itself.
This has been happening for some time. There was a case where a group at a university tried to sneak in a backdoor into the Linux Kernel and got dang close before someone found it and Linus then went back and pulled ALL code that came from that University and banned them from any and all commits going forward. It was much more complex than this condition, but interesting it wasn't as popular because it wasn't on the twitters.
Others might get scared by this. I, on the other hand, am getting reassured a bit. There's bound to be backdoors, the fact people are finding some, means there's one less backdoor to worry about.
The main scary part for me is that it was barely found and that it was found accidentally. I’m very glad that it did get found, since this will likely lead to developers scanning through tons of OSS libs
I would guess the 5 checks for Linux has something to do with finding where you are in the memory. You land somewhere in the checks, go until you find the last open/close square brackets, and then you know where you are. You could probably find the checks for Linux being passed to the OS for evaluation.
So this one got caught, but do we really expect this to be the only one? Just imagine how many of these clever backdoors slipped through the cracks because there wasn’t a clever Andres paying attention…
I love LLL, but ngl, everytime LLL goes online, I get the biggest imposter syndrome anxiety I ever had after my previous biggest Like i'm also doing cybersecurity and software development, so it doesnt help
`sudo apt list --installed | grep xz` That's what the above guy meant to say. Run this instead of asking the malicious program its version. Because that means you are executing the program, potentially running it.
@@abbe9641 I do only that command because I know 5.6.1 is only available on testing and sid and I was just curious to know how old my version was. Stable doesn't get updates that fast
The repetition in the bash script (as well as the binary comment) sounds like an attempt to ensure certain attributes in the compressed data. Padding length and possibly targeting a certain CRC, or maybe to set conditions for the tr replacement to work properly.
There was something about moving to zstd from xz in this video, but looking at what the xz package is required by on my system, zstd is one of them (along with rustup and the base package)... Kinda goes back to being able to scarily run arbitrary code at build time in stuff like build(dot)rs (which I remember Jon Gjengset talking about), I guess being more readable/auditable than some of the arcane build systems is one step, but yeah, some sandboxing, like even having all the features (like network or filesystem access), but having to turn them on one by one as needed, and having to justify turning them on to maintainers... because otherwise it all comes back to trusting upstream. I mean given that they set the scene for themselves, by patching the fuzzing library and what not, could still potentially be bypassed, but the more steps a bad actor would have to go through (so long as it doesn't add much more steps to normal users), the less likely.
I think it _looks_ complex from the outside - but if you were _creating_ this kind of hack, it's not too hard, I think. You're _setting_ the rules, not trying to figure them out.
agreed, it's really not that complex, which leads me to believe it wasn't even state sponsored if it was state sponsored it would probably be the tiniest security bugs that allow for a full exploit chain, not an obvious backdoor
For testing of things like decompression, it's hard to get away from binary blobs. I think the remedy is to treat binary blobs with a lot of suspicion and look carefully at exactly how they're used. For fuzz testing input... Just use a deterministic random number generator and provide the seeds that have caused problems in the past.
@@Omnifarious0 I mean its mere presence in the release tarball should've raised flags. "WTF are these large 'test files' doing in a production release?" (Though of course it would probably easy to hide nefarious stuff in image files that appear to belong in production.)
Binary blobs are only a problem if you get no info of how they were created. The fact that the bad.xz got in without any receipts of how it was created/getting creation parameters set by a second source is really the big takeaway no-no for me. Funny thing is, you could probably do similar things with more 'truly' random blobs, but reconstructing those would then be a (de)compression problem
My heart breaks when hearing all this new info about Lasse Collin. His companions betrayed him and now he is there alone, unable to trust anyone, battling mental issues and still trying to clean up one of the biggest security fiascos of the decade :(
I do suspect that some of the pressure that he received was part of a coordinate attack
@@lfarrocodevvery likely and hopefully this spurs more active efforts in taming the community around maintership. Not just allowing giant companies to freeride on over worked underpaid contributors. Same for people who use these projects on smaller scales but abuse and exploit the work of maintainers. It’s literally a security risk now if there was a coordinated social engineering attack on this poor maintainer of you allow these projects to go on with shoe string support
@@lfarrocodevthat’s very likely. Most of those accounts were just disposable accounts created with the sole purpose of posting the nasty comments to pressure the original maintainer
@@lfarrocodev of course it is. The moment I saw these messages about progress and stuff I thought about coordinated attack
They probably fucked with him behind the scenes on alt accounts etc. If they go this deep not beyond messing with his socials and so on, to cultivate him as a vulnerable target
Hot take: Everyone has been saying that this is proving the short comings of OSS, I think the opposite. If some idiot got themself a job at MS and did something like this, you would see that PS-Remote or perhaps RDP takes an extra half a second and say "what did MS do now??" and move on with your day. The beauty of OSS, the ability for someone to look at the code did what it needed to do: Someone who had nothing to do with the project was able to look at the code and sound the alarm.
Preach brother, this is a strength, not a weakness
exactly. you cant peek behind the curtain with proprietary OS'es. With linux and OSS, anyone can view or modify code. Thats the whole reason this was even found in the first place.
💯💯💯💯 people failing to see how good this is.
How would MS even be able to tell if someone put in a backdoor in anything closed source, it would be impossible to spot ten seconds of performance lag on anything from MS, let alone half a second.. and noticing one more backdoor among the dozens of 'telemetry' backdoors they already put in there? Nah.
The irony here is that this exploit was discovered by an MS employee.
@@evancombs5159 sometimes you gotta work for the devil to get the bag
Gladly TempleOS is doing just fine.
So you know that from FIRESHIP 🔥
Everybody is fine, only an idiot would be pulling from git to build a library for server deployment and most servers are on 5.4.x.
@@MuhammadYusuf-nz5nj Or we know it from.. TempleOS
Unrelated, but temple OS is actually a lot cooler than a people think and it is basically the perfect OS for learning about hardware, as it's complete lack of permissions and it's unique paging setup makes working with hardware very easy, also the fact you have the holyC REPL, you can experiment without friction. Seriously go take a look at zeal OS (a modern port) and start messing around with it
@@orbatos God bless
What's the lesson here? Don't get between a DB engineer and performance.
Trust me, don't
Having been a DB Engineer and having been blamed many times for poor performance when it actually it was some half-bright code monkey we get very thorough and very cranky.
The real lesson, working on open source projects doesn’t pay
@@vitalis the real leson is use winrar
the scariest part is the social engineering did on Lasse. this person was manipulated for YEARS and the team (probably) behind it saw the opportunity and exploited it. exploiting Lasse's mental health, trust and desire to pass on the torch. this is actually evil
You guys are awfully quick to clear this guy's name, it's sad when people get falsely accused, but that's life.
They may also caused the mental health problems…
Maybe you guys forgot about the covid psyop. Everyone is prone to be manipulated.
@@SpiDey1500 my god, didn't even think about it but they could totally have had accounts sending him hate comments over the xz utils not getting enough updates, which caused him to want to find someone else to take it over.
@@themodfather9382 absolutely no evidence Lasse Collin did anything wrong, while I see Andres Freund being congratulated with quips like "lifetime free drinks", which makes treatment of Collin even more stark contrast. No it's not "life" to throw wild accusations. It's just more prejudice against "mental health issues" being not real, so he must be suspect. Really uncool to suggest this.
Freund isn’t even a security engineer (disclaimer at the end of the post on openwall). Man is just that big of a gigachad.
He is now I guess
That's what freunds are for 🎶
Freundlich neighboorhood engineer 🫡
Database engineers are from a different breed man
«Freund» is also German for «Friend», which is very fitting here
This whole situation just feels like a movie. The fact that this is real is insane. And I can't decide what's more impressive - developing this backdoor or finding the backdoor ... this just shows me how little I actually know. I feel vulnerable ... just let me cry...
It feels like someone held a genius coder hostage to develop the backdoor and then the criminal half-assed the distribution of the backdoor (using accounts that were just created to push for inclusion, come on! That's like composing Beethoven's Tenth and then playing it on a glass bottle.).
And they're guessing there is more backdoors
Putting aside the likely-nefarious intentions of the chef, I appreciate that there are minds out there that can think like this and produce technical artwork of this caliber with such a niche medium, it’s really impressive.
You and lowlevellearning have really good energy together. Great video. More collabs please.
The pushing might be because there are 2 other things happening that are each likely to kill the attack chain.
1. openssh was already working on their own method for calling systemd-notify without linking it (upto now they did not link it due to them being very careful on dependency checking). Debian, Fedora and OpenSuse were patching sshd to do this linking. This is how liblzma got linked to openssh at all, this wouldn't be done anymore.
2. systemd is looking at better isolating and reducing their dependencies, especially for more critical parts of the system themselves and liblzma is looking to be dropped as a dependency.
Given these things, this backdoor may have been on a sudden clock where it is get it in next release or it is likely to be several years of setup for nothing.
I was also thinking once in the wild they're on a time limit before someone notices it so as soon as the malicious code was merged they need to infect as many machines as possible before that happens.
@@chilverscthat is always a risk, there is a chance that the moment this backdoor is actually first used in earnest it would set off some intrustion detection of why are we suddenly getting an ssh connectiong from some foreign country where we don't have any offices or something.
Yea this makes a lot of sense especially given how ham fisted the push was compared to the slow preparation.
28:25 The obvious reason for the rush is probably a branch+ticket+PR in systemd repo to switch the library loading to runtime and be optional, and this looks almost ready. Just imagine, the hard work of many years to be flushed down the sink.
There's an episode of The Sopranos where the FBI spends the entire ep putting a bug into a desk lamp and then planting the lamp in Tony's basement where he talks business with his associates. They only capture a single conversation, of Tony talking to a plumber about his water heater, before the whole scheme is undone by Meadow grabbing the lamp and taking it to her dorm room. Feels like a good metaphor for this guy's exploit getting caught so quickly.
Or that time the CIA spent millions putting a microphone and radio transmitter into a cat, deployed the cat, and it immediately ran into a road and got hit by a car.
@@superscatboyhey, it was a decent PoC.
The good thing about this situation is now people can really see some techniques one might use to obfuscate and maybe people will be able to look out for similar situations. However there's also tons of different commands and techniques to do essentially the same thing, but it's all patterns. Also people might take testing code more seriously.
Saved by some random engineer benchmarking postgres which 99.9% of SE engineers won't even have time to do :D
ssh is very widely used so yeah people will benchmark commonly used tools, it was a weird mistake
It took me over an hour to realize that this wasn't an April fools.
I don't know how developers are so smart that they can find this shit.
Blows my mind
yeah so what hasn't been found yet that's out there right now. Don't trust the computers!
This is not doing good things for my imposter syndrome
Wait until you hear about mathematicians
@@allsunday1485 what the hell does mathematicians have to do with any of this
I imagine the guy discovering this was just saying "Wtf is this" the entire time as he unravelled the shit storm.
I think this backdoor ultimately is going to do more good than harm, because now people are on the lookout for backdoors in tests and similar wild exploits.
You dang right about that. My last 3 days have been auditing and catching up on dependencies and scraping the git projects' comments. So many eyes are on it now.
My speculation is that the person is not Chinese.
The information that the name had mixes of Mandarin and Cantonese makes it sound more likely that it’s a non-Chinese person attempting to create a Chinese identity.
I think it’s a very clever ploy to leave digital breadcrumbs that align with people’s existing beliefs. People want it to be a big grand Chinese cyberattack, so by intentionally choosing a Chinese sounding username people will immediately jump to that conclusion.
We obviously can’t rule out the potential of it being a state sponsored cyberattack and perhaps even a CCP coordinated attack. But I think it’s important to be aware of our existing confirmation bias
I think they picked the name as another test of how easy it would be to sneak in malicious stuff. People in the US would be very suspicious of Chinese contributors, so an actual attempt to be as sneaky as possible would probably use a French or Swedish name. Look at the first vulnerability in 2021, replaced a secure function with an insecure one while having an apparent Chinese name, and it got through. That was part of the test.
@@magicmulder I like this theory, it makes a lot of sense. It’s like the Nigerian Prince emails where they are sending obvious scams to filter out people that are smart enough to recognize the scam
@@magicmulder Makes a lot of sense. This theory is similar to how email scammers purposefully make the scam more obvious to filter out the people smart enough to avoid getting scammed. So this could be a similar thing where they tested the waters before commiting the time needed to create the backdoor
Having extremely complicated bash scripts that modify files during the build step is kind of wild in 2024. I'm not sure why you'd even set a build system up like this, seems like hell to work with, let alone audit for security.
Let's be very clear: he already was the active maintainer of the project, if he wasn't the code would probably not have been accepted. This was only possible because he played the long game. Assuming it's even a he, not a she or a group, etc.
It's called GNU Autotools and there are still huge amount of projects using it. Migrating to something else like Meson or CMake takes a long time.
You can do it in makefiles too. And if the project uses scons (python based build tool) it's even easier.
Where is the 13% accurate guy who was going to solve Open Source Issues, weren't he supposed to take our jobs??
@ChuckNorris-lf6vo yeah I just asked chatgpt about how to fix the current state of the Open Source Community and yeah I totally agree these guys are wasting their time AI
if you were given 20 issues from 20 different repos and asked to address them in a week, realistically speaking you might get maybe 20%-25% done in that amount of time if you are that good. And that's probably the best you can ever do, but then imagine the 13% accurate guy can one day do 50% or more with an upgraded model
@@bugzpodderBut then you waste 87% of the maintainer's time. Because they can only tell if a contribution is bad when someone looks into it deeply.
It isn't better for open source if (even *if* the model is 50% accurate) half of the PRs maintainers need to go through are plausible looking but actually don't work
@ChuckNorris-lf6vo they're being sarcastic. Both NeverTrust298 and mhdmuzaffar-vr5mb. I don't think that counts as trolling.
Unless you were replying originally to mhdmuz* in earnest, in which case NeverTrust may be trolling you in specific.
@@edgars9581 sure. but the submitter doesn't have to be a bot that submits everything the AI cooks up. it could be a dev who is just using it as a tool to surface potential bugs, then only submits the legit ones. they could also use the AI to help them come up with legit repro cases, and possibly a new test case.
it isn't really much different than fuzz testing in that way. or plain old static analysis/compiler warnings.
For me, the big problem that this has exposed is the vulnerability inherent to the OSS / Linux / GNU building and packaging systems. It's an arcane mess of Makefiles, Bash scripts, ad hoc patches, and tar-ballz inside tar-ballz. It's long overdue for some security to be built into all that, like properly sandboxing builds versus tests, and having verifiable steps. In this particular exploit, it looks like a crazy mess of bash magic, but ultimately it's scary because of how easy it was.
it exposes a psychological weakness in test code really. It's mind tormentingly bureaucratic and boring and our mind just defaults to "looks good to me" ... "ah yes, random shit... that looks like good random shit to me" and "tests PASSED".
GNU package installer when? Gotta admit, build and package process is whack, each dev does their own random shit
@@onewildviktor i, too, think that OSS is simply the least bad option out there, and thats demonstrated perfectly with this story now.
whats the alternative?
@@jamesnewman9547 Pressuring open source maintainers, seems like a great idea. Fundamentally, the software is provided _as is_ (as is stated in the MIT and GPL licenses). Solving this problem starts with finding a solution to better compensate OSS developers for their work and getting more eyeballs on barely maintained software
I got hit with skill issue every line of the article
Really? Where?
Imagine all the potencial back doors we still dont know about
The way that open source projects are worked on; at least you can find them in open source code. This compromise began when the attacker did a Social Engineering attack to get onto the project and add these commits
In closed source code you'll never know you were compromised
Imagine! I remember that years ago the curl author shared a tweet about some guy saying something like "Thanks to curl (codebase) I can enjoy all my CVEs bounties"...
No. It seems pointless.
No way this is the first time. It's too sophisticated and well thought out to be an opportunistic attempt.
@@ThePlayerOfGames exactly. i think this being found and explained the way it is now is actually great for open source, isnt it?
Almost definitely a nation state. Lots of carefully crafted obfuscation & social engineering. I think over all this is a very strong argument for reducing our reliance on shit build systems
I think it is unlikely to be an individual, but it could be any large nefarious organization not just a nation state.
@@evancombs5159 To me it almost feels like an org that had one genius coder and then total doofuses trying to actually get the code published.
We're not going to narrow it down until we do heuristics on the accounts. Right now the bad guys are scrubbing and even scrubbing sticks out like a sore thumb. If it is a state or gov't sponsored effort, then a deal will be made behind closed doors.
I'm an industry veteran of 15 years.
I understand some of these words.
Things that run COBOL don't have native xz libraries. /s
@@chupasaurusCOBOL, lol wut? 15 years was not as long ago as you surmise. Learn to history. Probably the guy was writing JQuery 15 years ago.
@@JeremyAndersonBoise /s stands for SARCASM.
skill issue
“Reproduce the binary via the source code.” Npm just a giant binary basically at this point. Needs to be compiled by an independent source.
Compression algorithms do a lot of data deduplication so a real test file will have duplicated data to prove that the algorithm actually works.
Files with high entropy don’t benefit much from compression.
Just noting this as it would be expected for test files on a compression library to have that kind of repeated/duplicated data.
If I were an attacker I might theorise that adding data to a test file would be less conspicuous if the added data compressed effectively 🧐
Attackers didnt have much time left, as xz dependency was about to be removed/lazy loaded from libsystemd, breaking the backdoor.
Might be the reason why they pushed for it.
smart
*laptop bag with stickers all over it lid opens*
How do you do fellow open source maintainers?
All major security agencies should be after the perpetrator(s). The caliber is HUGE. If those guys aren't caught and the whole thing is silenced then it must have been state sponsored.
19:26 "fork yourself" lol. new insult dropped
I've been saying "fork" and "shirt" ever since watching The Good Place.
Not really a new thing. I saw someone walking around in a "Go fork yourself" tshirt before.
Suspending the original maintainer with appropriate explanation could be net positive regardless of if he was intentionally involved. Sometimes a forced break from things is good (also might keep him from getting hate mail while things are hot)
This opens up a whole new world of attack vectors.
Even just the proliferation of this one aside, we have no way of knowing just how broad the compromise is.
Scary shit.
not really, first of all stop building half the operating system off of tarballs which arent peer-reviewed, OR actually inspect and scrutinize what's in them, especially some big "testfile_good-trust-me" binary which is loaded during the build process, for absolutely no reason at all.
"I'm receiving 16$ a week from my patrons, my goal is 20$ a week". Open source culture right there.
It's someone's personal blog, no? Making $2k+ a year from a blog sounds very reasonable.
@@alexnoman1498Especially if you're in a non-western country.
@@complexity5545 if you live off that money in any developed country, you will starve.
Exploit discovered because some guy on the internet didn't like the noise his fans were making.
Head cannon.
after watching for 56 minutes i was already at " i am too stupid for this " however hearing the priameagean say it made me LoL
Flip is my favorite editor.
Neovim is my favorite editor.
But Flip is cool too.
prime and flip W right there
this is my favorite comment
41:03 One note on chinese name things: many groups do speak multiple dialects of chinese, particularly in areas like Malaysia or Singapore (where the Tan last name would be used in Hokkien communities) or other areas with large dispersed chinese populations. My fiancée’s family, for example, primarily speak Mandarin and pronounce their chinese names in Mandarin, but use the Hokkien anglicization of their surname. So, while it’s a good to look at, it’s not necessarily indicative that the Jia Cheong Tan name is fake.
Being this backdoor so much complex, I highly doubt it is being implemented for the first time. From start to end, everything seems well crafted and maybe improved on the possible previous iterations.
but it is to specific to xz that it is to easy i think
The sudden rush to get it done after taking 3 years to set it up sounds a lot like management interference, like there's a boss demanding results.
Some other comment pointed out that some lazyloading dependency of XZ in systemd would soon have been removed, and this is most likely what they were interested in.
It sounds like the coder selling his backdoor and the buyer being incompetent in getting it out.
@@NoidoDevI read that too from some BSD guys. This is so big that even the BSD security nut guys are on it (and the gov't).
Honestly, seeing how much effort was put into this makes me think the guy who did is simply a madman. Like lots of steps could be skipped with same effect. But guy wanted to prove a point and flex his genius on everyone
Also zero preparation for the actual push to get it included in distributions. 3 years of backdoor preparation and then they use two freshly created accounts to push distributors? Sounds incongruous to me.
Exploiter: i would have gotten away with it if it weren't for those meddling Microsoft guys
*Exploiter - I would have gotten away with it if it weren't for those meddling friend guy
It’s because it’s open source that we’ve discovered this. Had it been hidden, we would’ve never known about it
exactly, theres gaurenteed to be russian and chinese spies in every major US tech company. No doubt about that. Who knows what sort of damage they are doing. Netflix probably has a spy as well, probably a streamer too, as a disguise.
Had it been hidden the backdoor probably wouldn't be introduced in the first place lol. This xz situation definitely complicates stuff.
@@alpacamax3404 Not true. If he had a job for some Microsoft team he could have slipped the same code through.
@@gileeeDave Plumber had made at least 1 video about this, there are significant QC checks in place at Microsoft. You’d need multiple people on the inside, which is possible. But much much much easier in open source
@@lucasjames8281 the thing is microsoft can spare like what a couple thousand engineers? open source you have a much bigger population, it doesn't matter what the backdoor is, it'll get found out. good luck finding this kind of issues with just a few hundred to thousands. Then there is the elephant in the room, what of microsoft approved backdoors? they're not your friends either
Just an FYI: Lasse is pronounced ”Las-eh”, not ”Las”
If this happened inside of a large proprietary C/C++ code base, for example a foreign independent contractor with a fake identity at Microsoft or Riot Games was compromised and committed a malicious tar ball, most of the country would be compromised and almost no one would have the ability to find the issue. I don't think businesses are immune from attacks as sophisticated as this. At least with open source we have a chance to find the backdoors.
Where was Devin when we needed him?!!!?
How can you be sure it's not Devin?
@@michaelb4727 the backdoor works
?
too busy inserting print statements into some random python scripts
Great analysis, thanks for going through it!
This is truly scary stuff! it really makes you think how much stuff is out there actually compromising open source software that we're not aware of... 😢
Consider the following: this was only caught because of increased delay introduced by the exploit code. Now, what would have happened if whatever actors who cooked up this mess added a simple delayed activation logic? The exploit would be everywhere and likely no one would have been the wiser
Scary scary shit
This is so obscure I'm getting paranoid about the guy who even found the bug in the first place... my brain is like, "oh HOW CONVENIENT, you just simply stumbled on that!?" but then just has nothing to put after that. ... like maybe this was a compromised APT that was already under observation and "discovering" the backdoor was just a parallel construction--a way to expose it without exposing that they have a peep-hole into the APT's activities.
Dude chill
@@IronicHavocIt's OK bro. I've learned to type real quietly so they can't hear my keystrokes through the matrix. It's everyone else I'm worried about. TRUST NO ONE.
Exactly. If the NSA discovered it, they would find a misdirected way to disclose it.
if the NSA wanted to expose it they wouldn't need to find some guy to claim he found it..
they could do the most Chad git move ever and make a new anonymous account and submit a PR on the repo titled "Fix backdoor introduced by.."
That'd be the most epic commit ever..
Meds now
Seeing a lot of commentary on this issue pointing out how catastrophic this *could* have ended if it weren't for Andres' diligence. While that is of course true, the takeaway from this cannot be the story of how one very knowledgeable and detail oriented man saved the world. The discovery of a sophisticated, catastrophic RCE like this *necessarily* requires unlikely circumstances. If the attack was not discovered through these unlikely circumstances, we would never know how sophisticated and catastrophic the attack is. Conversely, if the attack was not sophisticated, it would not require unlikely circumstances to discover. Therefore, it is expected that catastrophic and sophisticated attacks will be discovered through unlikely circumstances. This is something like the anthropic principle for cybersecurity. The real takeaway here is that the more effective and catastrophic an attack is, the more unlikely you are to discover it.
Potential State Actor behind this attack
Clearly, it's flat out espionage
PSA about a PSA
LMAO is it Russia Gate 2.0 for you libs
@@kiwikemist Apparently you don't know what Russia did then or what "lib" even refers to. But no, this is espionage by an organized group and it is targeted. we don't know yet who made it, but the list isn't that long. Learn what words mean and try again.
@@orbatos lmao this is funny like the new Havana syndrome hysteria.
I think it was Richard Stallman who warned us about this kind of thing the 1960s!. It’s one of the things that is supposed to make Open Source software more secure than proprietary software. But the price is eternal vigilance.
On the other hand, If something like that happened in a proprietary codebase, nobody would even notice because they wouldn't have access to the source code
And stop using blobs. I hope it makes true open source instead of binaries sometimes (for drivers (cough cough Nvidia and broadcom)).
Some binary can't be reproduced with code, like image files for example. Those were never compiled, they were just created. But even an image could have code in it.
Wasn't there a recent attack on the boot process using a replaced boot loader graphic?
I believe GH repo was blocked so that automatic build systems don't pull tars from there. Despite that Lasse Collin stated that GH repo is unaffected, who knows?
low level learning is lock picking lawyer of software, they would a neat team
Would isolating the build and test environments (ie via containers) limit this class of attack? Might take longer to build, but if the test suites can't touch the binary that is going out, then the injection should be impossible, no?
I get the "no comments in my code" policy, but whenever I see expressions like this -> (49:10) -> it kinda starts falling apart for me... I think in such a case it really starts being nearly crucial to comment. Not even necessarily "what" you're doing, but more importantly "WHY"!
0:44 welcome to Costco I love you
12:40 this case may or may not be state affiliated. but it's extremely obvious there are state actors who are intended to deploy subtle bugs into widely used software.
The guy that discovered the backdoor and "got suspicious" needs approx. $100 million deposited in his account and be bought a beer.
Because of the number of heads in this command, I've been calling this The Hydra.
amazon,twitch,google ,youtube gotta retro actively pay open source creators going back to at least 95
The ingenuity of humans is amazing and sometimes scary. We did manage to harness the power of the atom in nuclear bombs decades ago after all.
imagine someone inject a crypto mining code into you CI pipeline.
This seems so obvious once you hear it, I'm surprised it hasn't happened yet/wasn't reported widely.
LLL got me hip to the importance of C, i friggin love the latest Prime collabs!!!!!!!!!
Question: would the backdoor still be relevant if SSH is disabled? Most linux Desktop users do not have SSH enabled, so this would mean the target was entirely servers.
Yeah it was backdooring the OpenSSH server process, if you aren't running that you are good
Amazing how the person who miraculously discovered this, also works for the largest beneficiary of its discovery. Embrace, extend, extinguish...
Read KenThompson's "Reflection on Trusting Trust" next 😁
48:59 In the team where I work, when you do code review and accept the code, you'll be the one fixing the bugs when the original author is on the holiday. That results in automatic "nope" whenever some piece of code cannot be understood.
A code like the crap in this m4-file is clearly either an exploit or totally unmaintainable. Either way, it doesn't get to live in the official master branch.
I'm not sure if I'm just old enough but the bash scripts seemed easy to understand compared to the m4-stuff.
The thing is precisely that it did not live in the master branch, it was *only* present (added by hand) in the source tarballs. And I guess nobody bothered to check if the source tarballs actually matched the source code from the repository.
@@guillaumebrunerie Same happens with npm way too often. The code you get from npm doesn't match the code published in GitHub for many projects!
"The Three Body Problem" is the best sci-fi book I've read in years, hands down. Also, I refused to get the sequels because the first book freaked me out so much, and I know things don't actually get serious until books 2 and 3.
Read them, it's worth it.
I don't think The Primagean is right about compression code being inscrutable. I think it's investigatable in it's raw form, and LLL is right that that's how it should be committed, or at least SHA256 hash addressable back to compilable code.
Shouldnt `binary_blob | manipulation | eval` be a red flag that could maybe be scanned for somewhat automagically? Not sure if I understand everything here though.
Simple proposition: Make things be more human friendly for some crucial items. For eg, at vulnerable/failure points have understandable code, which I prioritised as chip verification engineer. Everything we do has to send data as binary & get it as binary, so its good practice to have understandable code for someone else
man jblow really predicted these
what did he say?
@@smnomad9276 ruclips.net/video/WGekWFxeD6c/видео.htmlsi=zBJA4Sc6Dyk_mwyO
@@smnomad9276ruclips.net/video/ypZ9JvUqaao/видео.htmlsi=jpgqQSxR1oHxHhAd
He described exactly this sort of thing happening. That there’s thousand of nation state threat actors who’s role is to do stuff just like this
I watched his video, some of his assessments in his video are definitely wrong, but I think it all comes down to: you are accepting code from random person on the Internet.
That code possibly needs more attention than some of that in your own company.
But notice how with the XZ hack they played the long game, to become the new main contributor. NSA does the same when they infiltrate a company. The difference is basically 0.
As an appreciator of origins of words, I love that you just randomly mentioned the origin of the word handsome. 😊
This hack makes my production code look poor with all of its robustness and future proofing 🤣
This is why I leave my repos on private most of the time. I've never been totally sold on open source; I have been on the bad end of unreasonable expectations too many times.
“gaslit by the whole community”
it’s literally just one guy bro..
Open source community in general with all open source projects
IIRC there were like coordinated sock puppets trying to get PRs pushed through.
Yeah they were referring to the other accounts that are now suspected of being sock puppets
@@IronicHavoc even if they were individuals themselves, it would still be like 3 at most lol, nothing to cry over. ive seen women get more hate under instagram comments.
@ThePrimeTimeagen @LowLevelLearning Do either of you (or anyone here) know what tool they used to generate those entropy maps?
Whilst not specifically this exploit, this is why I have been so anti binary repositories for years. That does not mean always building from source is safe; clearly this could have been put into source and still have been missed. However, the main takeaway is not the details of the exploit itself but how once again "social engineering / psychological" attacks against one or a limited number of dedicated individuals enables bad-actors to fly under the radar. Small cogs in a big machine are important. We need to support them!
Agree that any binaries should not be included true and if they do, they should not try to take data from it. It should just be: test is OK if it can be decoded. Test fails if it can't be decoded.
But let's be very clear, they played the long game and were the active maintainer. So they set the policy of the project.
Sadly, this should mean changes to the processes of the Linux distributions. They should look at the parent project and point out problems that don't match the policy of the Linux distribution which is to not take in stuff which can't easily be audited/simple aka fit for purpose.
@@autohmae Although I am sure there will be legitimate exceptions, I agree.
I'm so glad you two tag teamed this bad boy. What a delicious bro AF gigchad exploit lol I absolutely love this, it's a work of art.
8:45 He said he noticed it because of high CPU usage, not because the slowdown
I think the key takeaway of this situation should not be "Oh don't commit binary data to a repository" but rather "how do you enforce integrity of the released build to the actual source code + don't do magic in build systems". 1. For the integrity part, this exploit only worked this way by modifying the build file where the archive got depbfuscated, so if the release tarballs would have been exactly the same as the repos, the payload for the exploit would have only been gibberish to begin with. 2. I think they would also get away with it if the build script would have been committed to the source code. Yes, it would be more obvious, but realistically, nobody would have audited a complicated build like this where sed, tr and other magic commands are normal.
They really shouldnt be normal parts of a build though. There's essentially 0 reason to have those as part of a normal build. The only time I can think of is "here is a plaintext/image file that is perfectly legible, this command will compress and corrupt the file" for the bad case instead of submitting a binary blob
@@TurtleKwitty You are right, modifying a good file would be the best if possible. But still only solves one of the holes, because the release tarball still contained changes to the original source and could have made any change to any file, from build scripts to source itself
@@ES-cf4ph I'm on nix; everything gets built from source release tarballs don't make sense anyways honestly
I don't think this is related to Open source specifically. This could happen even in commercial software. Nothing in the source, everything is split between the tests and the build system!
Agreed, I do think maybe the git repo with the normal code should be separate from the repo with the test-code.
And both should not be run in the same environment. Only the result of the build (without test-cases) should be packaged.
@@autohmae Totally agree since the source code is not compromised
@@khalilzakariazemmoura8995 the scary part is, is the real issue: the active maintainer was the compromise. I really hope Linux distributions and package maintainers take a couple of lessons out of this. They are the most important barrier after code review by the people directly involved in the project itself.
This has been happening for some time. There was a case where a group at a university tried to sneak in a backdoor into the Linux Kernel and got dang close before someone found it and Linus then went back and pulled ALL code that came from that University and banned them from any and all commits going forward. It was much more complex than this condition, but interesting it wasn't as popular because it wasn't on the twitters.
There must be more compromised packages.
The best demonstration of human intelligence and creativity I've ever seen.
Others might get scared by this. I, on the other hand, am getting reassured a bit. There's bound to be backdoors, the fact people are finding some, means there's one less backdoor to worry about.
From WarGames:
D1: "You're telling him about our backdoors!"
D2: "Backdoors are not secrets!"
D1: "Yeah! But you're giving away all our best stuff!"
The main scary part for me is that it was barely found and that it was found accidentally. I’m very glad that it did get found, since this will likely lead to developers scanning through tons of OSS libs
1:02:35 these guys vibe so hard LLL can keep up the tempo by telling a story out of no where. Keep up the good work :D
It is really MOSSAD-ish.
I would guess the 5 checks for Linux has something to do with finding where you are in the memory. You land somewhere in the checks, go until you find the last open/close square brackets, and then you know where you are. You could probably find the checks for Linux being passed to the OS for evaluation.
it's the NSA
I don't know the country, but state-actor is an option.
@@autohmae It's literally the NSA, it's not even the first time they infiltrated Linux to push intentional backdoors to the Linux OS.
So this one got caught, but do we really expect this to be the only one? Just imagine how many of these clever backdoors slipped through the cracks because there wasn’t a clever Andres paying attention…
Tan Jia Cheong is a pretty legit name in Singapore
The way he types in his PR is also oddly Singaporean..
This is Dennis Nedry level !
“The only way to find him now is to go through each line of code one by one. About 2 billion.”
I love LLL, but ngl, everytime LLL goes online, I get the biggest imposter syndrome anxiety I ever had after my previous biggest
Like i'm also doing cybersecurity and software development, so it doesnt help
Also, the NSA be punching the sky right now
Tom would have caught it without the need to experience a random slowdown.
xz -V returns 5.4.1.
I love Debian Stable
Do not ask a potentially malicious software what version it is, use your package manager to do so, use common sense on the internet.
`sudo apt list --installed | grep xz`
That's what the above guy meant to say. Run this instead of asking the malicious program its version. Because that means you are executing the program, potentially running it.
@@abbe9641 I do only that command because I know 5.6.1 is only available on testing and sid and I was just curious to know how old my version was. Stable doesn't get updates that fast
The repetition in the bash script (as well as the binary comment) sounds like an attempt to ensure certain attributes in the compressed data. Padding length and possibly targeting a certain CRC, or maybe to set conditions for the tr replacement to work properly.
plot twist - the person who found the backdoor is the person who implemented it. He's just after a bonus
There was something about moving to zstd from xz in this video, but looking at what the xz package is required by on my system, zstd is one of them (along with rustup and the base package)... Kinda goes back to being able to scarily run arbitrary code at build time in stuff like build(dot)rs (which I remember Jon Gjengset talking about), I guess being more readable/auditable than some of the arcane build systems is one step, but yeah, some sandboxing, like even having all the features (like network or filesystem access), but having to turn them on one by one as needed, and having to justify turning them on to maintainers... because otherwise it all comes back to trusting upstream. I mean given that they set the scene for themselves, by patching the fuzzing library and what not, could still potentially be bypassed, but the more steps a bad actor would have to go through (so long as it doesn't add much more steps to normal users), the less likely.
I think it _looks_ complex from the outside - but if you were _creating_ this kind of hack, it's not too hard, I think. You're _setting_ the rules, not trying to figure them out.
agreed, it's really not that complex, which leads me to believe it wasn't even state sponsored
if it was state sponsored it would probably be the tiniest security bugs that allow for a full exploit chain, not an obvious backdoor
For testing of things like decompression, it's hard to get away from binary blobs. I think the remedy is to treat binary blobs with a lot of suspicion and look carefully at exactly how they're used.
For fuzz testing input... Just use a deterministic random number generator and provide the seeds that have caused problems in the past.
Why would anyone want test stuff in production? I'm not deploying my unit/integration/E2E tests onto production servers.
@@magicmulder - In this case, the test code affected the production build through a lot of bash sleight of hand.
@@Omnifarious0 I mean its mere presence in the release tarball should've raised flags. "WTF are these large 'test files' doing in a production release?"
(Though of course it would probably easy to hide nefarious stuff in image files that appear to belong in production.)
Binary blobs are only a problem if you get no info of how they were created. The fact that the bad.xz got in without any receipts of how it was created/getting creation parameters set by a second source is really the big takeaway no-no for me.
Funny thing is, you could probably do similar things with more 'truly' random blobs, but reconstructing those would then be a (de)compression problem