If I understand correctly, this hack wasn't discovered by auditing any source code, right? I heard the Microsoft guy just had a weird feeling about SSH executing his command for a few too many CPU cycles and was curious enough to investigate. Does anyone knows if it could've been easy to spot this by reading the source code or if it's hidden very well?
The problem is this attack was done during the build process.... so you need the source to build it to verify it ... but its tainted ... so .... its valid? yea... this one was a wild one
It's a decent idea but what do you mean in practice? Do you run your build in a way that refuses to give it access to any of the test data? That's not too crazy. Is it enough? Do you verify that others can produce your build without any of the test data? How do you ensure that the build process doesn't have a side channel running in both cases introducing other logic? What i'm saying is this idea has to be mapped to concrete practices and tools.
Jeez. Virtually every part of this is done in a way to be practically undetectable at every stage and effectively roots every bleeding edge version of the biggest distros and would have seeped into stable versions of them with time. These people are diabolical, and it’s practically a miracle it was found this early before it got worse. To think, detection started with someone noticing ssh connection speeds were a hair slower. Major kudos to them for being so prudent. And thanks for the updates on this.
Pretty much. I dont think it could of even been hidden any better. The main reason anything was looked into, is because someone noticed their SSH login taking half a second longer lol.
@@draken5379 Oh but it could be... What if your billion, no trillion $ chip - board makers are embedding hidden algorithms as such but as undetectable hardware roms that the OS, the user, etc. doesn't know about. Then with that only a handful of elite personnel coming from these giant corps as well as limited personnel within some government agencies have direct remote access into these hidden systems. Think of it this way; some of these newer systems you can load the entire Original Doom Source Code and Assets directly into Modern Cache and never have to go back out to RAM or to the Hard Drive. If you go back to the days of the sneakernet with the good ole floppy disks, people used to sneak malicious code into their 512 byte boot sectors. They can literally miniaturize a 6502, and 8086, or even a basic Pentium these days into a chip so small it would look like a little controller on the motherboard or even onto some peripheral card that one would have no idea what it is or was truly designed, intended for. This doesn't even account for modern UEFI systems compared to obsolete CMOS Bios type systems. And now with the advancement of A.I. and designated A.I. chips showing up... The sky's the limit. Well, skynet is the limit according to some... They can always hide it somewhere that is least expected. How about within a button on your polo shirt? You have to realize that we aren't just in the age of digital we are also in the early stages of Quantum Processing, A.I. Systems sure, but we are also on the stage of Nanotechnology. They can literally make programmable machines that are 1,000th the width of one strand of your hair. You can't even see them with your eyes. So if they truly want to hide something they surely can! And they can hide it just about almost anywhere they want. Even in the foods you eat and the liquids you drink! And this is not science fiction either!
...that feels like a paranoia syndrome waiting to happen because their fears have been proven so much more horrifically right than they thought. That's going to mark them, considering you already have to be pretty paranoid to check code over half a second. @@draken5379
Not enough people aware that maintainer/og auther (Lesse) is victim of social engineering, I read those messages in mailing list and it hurts me. Jia basically took advantage of his health condition.
@@franzlyonheart4362 Probably some asian/russian threat actor. My guesses are North Korea, China or Russia. Maybe Lazarous Group? Killnet is out of question since they do mostly DDoS.
8:56 Agreed that there are certainly vulnerabilities with open-source, but this also highlights the strengths of open-source: having a huge cohort of noble, intelligent, and vigilant coders to sniff-out this type of espionage.
Imagine this happening with a closed source company. Somebody applies, silently checks this in with the binary files, and there's nobody outside of the company that even has a chance at catching these backdoors. How many of these backdoors already exist in commercial software? How many does the company owning the code know of?
@@dascandy THIS. Every time i see somebody talking about how this is "an issue with open source", i think this exact thing. What if this happened in a closed source software? AT LEAST with open-source you're able to find out. With closed-source there's a good possibility that piece of code will never be looked at again because "it just works".
@@justacat3639Thinking back to some part of Windows NT4 shipping with a thing called "NSAKEY" (en.wikipedia.org/wiki/NSAKEY) and nobody knowing what it does or how it got there.
we definitely need to see the investigation behind the contributor who created the backdoor. definitely very good at social engineering, being subtle, etc. if it weren't for Andres Freund we would have never been talking about this.
If anyone's takeaway here is "I don't happen to use those versions of of that library, I'm good", you did not understand the issue. That maintainer is still struggeling. Other maintainers are still struggeling. Other libraries are still compromised. The Perp and their Patron are still out there doing this exact thing and more for the next 30 years together with lots of direct and indirect colleagues. The people introducing vulns are payed exorbitant amounts of money, because buying vulns is more expensive, less exclusive, while just as bad for security. Getting the maintainers of our critical infradtructure payed and having them audited should be our priority world-wide. Every company in every country uses FOSS, but vanishingly few pay for it. Ask your manager whether your company pays for the software they are using!
While I agree with you in spirit, if you're running RHEL and paying for it, IBM isn't giving the maintainers of xz a cut of it, so just paying for FOSS in a corporate setting doesn't exactly solve the problem either.
I totally agree. Corps generate hundreds of dollars per tool used, but they don't even donate a dollar. Then we get a core-js or faker.js situation and people lose their minds
I'm sure I'm not the only one who was reminded of Clifford Stoll's "The Cuckoo's Egg" (1989), where a 75-cent computer usage accounting error ultimately revealed a remote attacker with superuser access.
to be fair the attacker had to disable and ignore several address sanitization features and test checks. Hence why stuff like valgrind was already setup to catch stuff out for this particular project.
I still can't fathom how incredibly lucky we are this got caught as quickly as it did. I still haven't found a system on stable that actually shipped the compromised versions. We don't always get a warning shot like this.
just curious.... why would large organizations be using operating systems that get updates from open source git repositories? maybe im understanding this attack wrong just curious if you dont mind educating me
@@Noname23489 A sizable portion of servers on the internet run some version of Linux. There's no good data, but I've seen stats that are well over 50%. And everything based on Linux is at least partially relying on open-source code.
It does some really complicated/shady things like altering the Global offset table and character recognition, I wonder if we ever get to find out who was behind this
As much as I love the technical aspects of this, its not what we should be focusing on. The "nature of [this vulnerability] of open source software" was primarily and overwhelmingly social. The project had a solo maintainer that was hopelessly overwhelmed. He was conned by one or more anonymous individuals into accepting a malicious co-maintainer by both attacking his weaknesses as the project owner, and offering him the answer he was desperately looking for to save his labor of love project. He was pressured into trusting code and a developer without sufficiently scrutinizing either.
My takeaway is that open source is only as strong as its weakest link. So many projects go to the graveyard because of the lack of maintainers, and there's a lack of maintainers because people need to eat, need a roof over their heads, so they work their 9-5 as software engineers, which is draining in of itself. So when a crucial piece of software has a solo maintainer, it's a prime target for attacks. If there was more compensations for open source maintainers, maybe some of them could afford to maintain the software full time, and as a result would not burn out and be less prone to social engineering attacks. Or we could just create a foundation for those crucial projects, so that there are always developers able to maintain the software, and need to reach a consensus before anything gets merged.
@@Dogeek I think all the companies that used open source software in their commercial projects on a massive scale that command massive profits should be obligated to divert a fraction of a percentage towards maintaining these things. Or better yet when there's a massive fork and that fork goes on to make billions of dollars, leaving the original project in the dust and penniless there should be something that allows maintainers to continue doing what they love to do. I love open source but the (un)ethics of business undermines such a great ethos it really becomes disgusting. Sucks to see such openness taken advantage of again and again, and even more now since technology companies rule as the market leaders in today's world.
Not even that, he knew the attacker for YEARS before he let him into the project. nearly 4 years. And the guy was nothing but nice and helpful until a month ago. Nobody, not a single person on this planet, would have not trusted him by that point.
I would not call this a vulnerability in open source, rather it's a strength. This could, and probably has, happen in closed source software too. They can just get some person to apply to a job and do the same thing. The only difference is that in closed source it's far fewer people that could potentially find it. I think one of the reasons it was found relatively quickly is precisely because it was open source.
Agree 100% with this, the fact that open source code is evaluated and run by so many independent people is a strength that leads to things like this being found. If this change was to a closed-source tool it could sit in a repo for a decade and no one would notice :/
It was more luck and that the new version caused half a second delay that a person was doing performance tests at the time most other people probably wouldn't have noticed unless they was doing performance testing, this was pure luck as it wasn't that Mutiple people picked up on this it was just 1 person if they fixed the delay part it wouldn't have been noticed until a couple of years later on when some high profile targets got compromised and don't know how
What's interesting is the timing. We see the attacker working hard to make Ubuntu include the compromised versions in their 24.04 beta, but also trying to convince fedora to include it in fedora 40. This makes me believe they might've been constrained by time at the end be cause these two releases are very important: The next Ubuntu is LTS and the next Fedora is likely what the next RHEL is going to be based on a year from now
I feel really bad for Lasse. Jia probably looked like an angel sent from heaven to help with project maintenance at a time when they weren't doing so great. Then all this comes out, and they have to confront this nightmare with suspicion cast even on them. It's like a dad finally handing his keys to his daughter for the first time to drive on their own and within 15 minutes she drives through a crowd of pedestrians.
Yeah even going through their commits the majority of them are benign improvements... And then this, with the seeds having been planted in the test files ages ago
@@nuke_parasitine nah, the contributer, before the backdoor, was only ever helpful and nice. If you knew someone for years and they were only ever lovely to you - and it turned out they were a serial killer - it would be a horrible thing for people to imply it was only ever obvious (they were a snake, you let them in) when it wasn't.
cs undergrad here. A lot of this is still over my head, but it is so interesting. Honestly just inspires me even more to learn. I appreciate your breakdowns that are very clearly presented and approachable.
Oh? You mean how this backdoor didn't affect any LTS distributions used in servers or production environments but only affected rolling release and testing distributions? That scale? In other words, only desktop users who are stupid and naive were affected.
@@adibemaxwell6111 It targeted all Debian based distributions. The only reason only rolling distro's were affected is because that Freund dude just so happen to be benchmarking his PostgreSQL databases and therefore end up noticing the backdoor. If he didn't it could have been planted in LTS distro's as well after some time. Also, the bad actor has been committing to the repository for a few years and it's still being investigated if there's more malicious or at least sketchy code in there, which if there is it could mean that even older versions of the libraries could be compromised in some way, which would also mean compromised LTS releases. You have to be short sighted to think these sort of attacks is solely an issue for people running rolling release distro's...
@@adibemaxwell6111 If it hadn't been discovered, it would be on like 80% of servers within 6 months. The only reason it didn't is because some Microsoft employee noticed high CPU usage from sshd and looked into it.
This is a variation on the theme. Open Source projects may only have one or two maintainers. We had a wakeup call with OpenSSL a decade ago. This is just a sneaky supply chain attack via compromise of liblzma and oss-fuzzer. Trust was assumed, but trust was broken. How will single maintainer OSS projects audit all commits to their project if they are hobbyists or part-timers? The payload was to be included in deb or rpm packages during build via checks. That was another way to ensure it got into the main distros, but was low key.
@@smnomad9276you should read it. It’s about “how can you trust your own trust chain” and what can happen if that trust chain is broken. “You can not trust code you did not compile yourself”.
@@smnomad9276 A famous speech/presentation he gave as part of an award acceptance speech. It's basic premise is "what if I maliciously modified the compiler included with the UNIX system to do two things - 1) if compiling a system component, insert a backdoor that allows me full access, and 2) if compiling a compiler, modify the output compiler to perform these two steps." Basically saying, "how far do you trust what's been given to you?"
Trusting Trust is a much deeper concept, and the reaction to it was the practice of Diverse Double Compiling. This trust gap is way easier to deal with, and more analogous to the use of nothing-up-my-sleeve numbers in cryptography. Distributing binary blobs alongside source code should be treated with great suspicion. Let test data be programmatically constructed, and therefore auditable the same as code, or by some other means be provably incapable of hiding machine code (e.g. pi).
one way you could prevent something like this would be disallowing binary blobs, and if you need it for testing, etc, it should be generated by code that is readable. For example a shell script that generates a large amount of the letter a compressed with xz and then intentionally corrupts it. Though this doesn't prevent people from doing bad stuff like this, but it makes it harder to hide. (people can still intentionally introduce bugs in the code that can lead to something like heartbleed for example)
You could also mandate that test data must live in a separate repo, and must only be used while running the tests (in an isolated sandbox). Also the build system should be less flexible so that any weird stuff really stands out. But all of that is moot if it's the maintainer doing bad things and no one else actually reviews any of it.
the main issue is the main issue that has plagued open source recently: business expects services and support from the open source community, but is unwilling to pay for them. in this case, we have a security critical library being maintained by a single person. in no sane world would i expect that to be secure. system critical libraries either need to be fully audited and then quarantined with no more updates possible, or they need well paid, dedicated, redundant staff and regular security audits. the new layoff culture and over investment in ai (and underinvestment in mental healthcare) is going to do massive, yet untold damage to computer security.
Wow this guy is some sort of evil genius. Appreciate the constant video updates, it's great to have someone summarise these really complex topics but still keep the essence of whats happening.
@franzlyonheart4362 why is everyone in these comment sections assuming it's got to be an American agency? My money would be on China, Russia, or Mossad. Hell, Mossad's bread and butter is this kind of diabolical social-engineering into unnoticeable attack vector.
@@franzlyonheart4362 No reason to assume it has to be the US government. Could be state-sponsored hackers from China, Russia, or North Korea, too! But with so little information at this point in time, I don't think it's worth baselessly speculating. It could totally be non-state sponsored, as well-there's a lot of money to be made from having this sort of backdoor access.
Really thankful for Andres Freund's agile mind; finding a needle in the 'stack is hard enough; he found a bd even though he wasn't looking for one, anticipating one.
Not sure how OpenTTD would ever be affected, even if using xz with a backdoor, since it replaces RSA decryption function, and gets the payload from RSA key supplied to SSH.
@@DVSoftware OpenTTD may not be vulnerable to xz itself, but it certainly is vulnerable to GitHub taking down xz's repo and users getting pissed about potential malware being included with the game.
So, as I understand it, if the compromised version made it to the server distributions, the only way to guard against it would be to limit who can connect to ssh using iptables (or some other firewall), since the attacker would still need to connect to the ssh port to send the commands.
To me, the scariest part of software these days are all of those 3rd party libraries. You ever install something with "sudo apt-get install X", and you'll see a stream of things being downloaded. Even just one part of one of those libraries could be infected and who would know? MD5 hashes or whatever would do nothing here because this was compiled into the release itself. Frightening as hell
Dependency graphs give us the tools to fix problems once instead of over and over and over. They also allow us to be more effective and efficient in our work. They also let risks and failures cascade through our ecosystems in astonishing ways. Not all of the problems are even malicious. left-pad in nodejs was just someone unpublishing his work and it took down tons of sites. What's the answer? i have no idea. But if we focus on practical situations we can perhaps come up with practical mitigations that reduce harm, and improve.
@@_devilfish303 The point is Linux in its current state doesn't provide better protection than Windows. The slogan "you can always recompile everything locally to make sure the binary matches the source code" doesn't work anymore. Because every single Linux library is dependent on dozens of others. It's physically impossible to recompile everything on your own. And even if it was possible it would still give no guarantee as this attack was built directly with a hook to inject the binary malware during the compilation process without altering the source files. Yes Microsoft is not better. But it's not worse either. And this is the true issue here. Why bother with all the hassle of securing your linux if anyone can trash all your efforts with a single commit in some obscure source repo you never heard of?
curl | sudo is the one that terrifies me. Literally random code from internet run as root. Multiple attack vectors there from ip / DNS hijacking through to just straight up blobs in the install script... Plus potentially no evidence of what you just downloaded
@@christianbarnay2499microsoft is not worse? If such a backdoor was installed in windows, it wouldnt get discovered EVER. Even a simple babckdoor wouldnt ever get discovered, which wouldnt even get merged if it was open source. Security isnt black and white. No system is ever completely secure. Its about reducing the attack vector. And the attack vector in open source or linux is extremely small compared to proprietary.
@7DuRd3n geniuses that have specific domain knowledge sharing ideas with others in their domain knowledge. this most likely was a state operation. you think this is wild, imagine the concepts and ideas in modern mathematics where you are not constrained by physical limitation. everything is abstracted and abstracted.
Can we agree that whoever made this backdoor was pretty skilled Also the guy who found it is a hero for being diligent and trying to optimize and finding it
It's cool that this vuln has been found, but the scariest thing to me personally is the very real possibility of similar backdoors that may already be present in other widely used OSS projects. The fact that even this one was found by total accident is genuinely terrifying.
Small OSS projects like this usually have a very high bar for performance to meet, and they usually meet them no problem, so it makes sense that the valgrind performance anomalies were the reason why this one got discovered. Now imagine trying to discover "performance anomalies" in commonly used proprietary software...
That's because everyone is under the impression that these kinds of things are rigurously checked by someone else, therefore no one checks them. This should be a wake up call.
Who would win: A multibillion dollar government agency, or one giganerd noticing his ssh takes 0.5 sec longer to log in? That's a W for the nerds, boys.
@@TheOzumat Obsessive optimisation is the root of all evil - and that is why it also naturally finds it. Dude wanted to optimise performance and probably spent days untangling the mess he ran into - just because he wanted a near unnoticeable improvemet.
I've been following this as it was unfolding over the weekend. Just watched the first 3 minutes of your video. The fact that this backdoor is very effectively secured against being used by anyone else than the attacker is a super strong signal for me that this is not some random cyber criminal but a state actor. The sophistication and attention to detail in this whole exploit reminds me a lot of stuxnet. Honestly, it's a whole notch up from stuxnet in a lot of ways.
One key correction here is that the hosted git repo does not contain the necessary code to build the backdoor. Only the private repo for the library maintainers contains the necessary code. Your highest upvoted comment already starts off on the wrong track. I spent a lot of time sorting through this confusion today.
Isn’t the fact that this was caught proof that more people are watching what is going on in open source? This was the best case scenario for a malicious attack and the openness of the code allowed having all the evidence available. I agree that this should be the start of regular audits of crucial open source software. But the log4j exploit a few years ago might have had some influence on the Microsoft employee’s idea to dig deeper. I see this as a large opportunity for more secure code practices in open source.
this was found by chance more than anything, it's only because someone tried to get a good baseline for their system that they noticed that sshd was doing more work than it should and dug into it If that didn't happen then this would never have been found
Well yeah. But a lot of bad actors are now fear mongering so that people get scared of open source projects. We should doubt every project, for sure, but we have the ability to look at code and the whole build process and catch such exploits. But of course, as this example showed, there are simply too many projects for people to realistically screen with every commit. But we have to start being more cautious, starting with the most critical packages would be a good beginning.
@@ratchet1freakBut on the other hand, if the library hadn't been open source, he wouldn't have been about to look into it as deeply as he did. He likely would've had to conclude that the developer introduced a bug, and at most reach out to them, which would've only served to tip the bad actor off that their backdoor was still too detectable. So overall, open source did "win" here.
@@helipad_huck1543as it happens, someone else actually did exactly that from my understanding, and that was the patches that the second version of the backdoored version of xz fixed, some valgrind issues that were found by a maintainer of a different distro, so that's xz 5.6.1 :P
A couple thoughts on prevention. Move the test environment to a temp folder, completely separate from the source, and only run the tests on a stripped binary, so that the actual source cannot be found by the tests.
Plot twist. Andres has the private key. He did this all to rise to stardom in the security world. Being discovered only strengthens his reputation of supreme social engineer. He’s brilliant either way.
Binary blobs are basically immune from standard audit, I don't see much of a way to prevent this kind of attack except to ban blobs from being included in source control. Which is obviously impractical in certain scenarios.
I have a feeling that it's not just a simple uber haxor kid, this is too big and the setup is too perfect in the way it was being set up slowly. I don't want to sound like a conspiracy theorist but I just have a feeling that the guy who we all saw as a co-maintainer is not the sole brain behind this attack. It had to be a group who would greatly benefit from having this backdoor available.
@@Cavi587I think most people consider a state-level actor (like some intelligence agency) the most likely behind this. It's not just the amount of time/work spent on it, it's also the likelihood of failing which was too high for the average cyber criminal without government funding to risk so many resources on it. What if systemd drops its xz dependency 2 years into the project? Or another contributor shows up who likes to do his own improvements to the tests or build system (that touches your manipulated files, so you have to turn down all of their PRs for made-up reasons)? Or the original maintainer is just stubborn and says "no binary blobs of unknown source as test files, period"? Or some person simply finds the backdoor before it makes its way to stable distros? There are so many ways for such an operation to go wrong. You either have to be able to scale it, working on a bunch of these in parallel and hoping for one to succeed - or you have to just be ok with the operation failing and the only thing gained from it being the insight that it's quite a hard thing to pull off successfully.
I don't think the brilliance of this evil attack can be understated. Not only is it a backdoor, it is a backdoor where only one person or group has the key to open!
Great demo. From a cybersecurity point of view, this should teach you to think bigger, implement zero trust and secure your platforms beyond just the front door. Treat every server/service as if it's already been compromised, and limit access from that perspective.
Just so impractical :( I just want to use my system and write things that jave some power to alter things. A big stack of sandboxes and passwords is so frustrating
@@hellowill yeah, honestly. But this is deeper, it's that you can't trust your own machine or things running on it. People put backdoors in everything. We need well funded open source hardware and software but we are moving away from that
An issue with this entire thing is that we keep trying to invoke a technological “Web of Trust” to let us bind dozens of dependencies into a single binary, but instead we should perhaps be encouraging larger teams to form. Many projects have one or two or three primary authors who carry the entire load alone, which is intrinsically broken over the long term.
But how would the attacker know which server got infected? Does it phone home in any way, or was the attacker hoping to just attempt connecting to random open ssh servers accessible through the internet by scanning IP ranges?
Attacker prob patiently waiting for their code to be published with stable release then do crazy stuff. Remember it got exposed just because of 1 dude noticed ~500ms delay on his ssh connection.
@@gie4830 I don't know. I'm pretty sure actual big actors don't have SSH servers directly addressable through the public internet, you most likely need to jump through some enterprise VPNs to get access. So the targets were mid-sized. At least now he's vulnerable to getting honeypot'd. Wonder if someone will attempt to catch him like that. Though it would have been easier to not alert everyone if he really was targeting only a few actors.
So, are you using a virtual machine setup on an air gap? I usually use a set of virtualbox machines in their own virtual network on an old notebook for stuff like this. Would love to see how your setup looks for cases like these.
I don‘t understand how people can frame this as „a problem of open source / linux“. It is only BECAUSE the whole thing is open source, that this backdoor could be found AT ALL. Think about how it would be infinitely easier to do this in a closed source software!
not necessarily. Many OSS projects have only single maintainers, which makes this kind of attack easier. Large software developers have extensive code reviews with many eyes looking over changes before commit, meaning that a malicious actor (other than the company itself) would have to fool or corrupt many developers to sneak in the backdoor. I am not saying that Closed-Source is in any ways better, what I am saying is please don't try and make it seem that OSS is always better able to withstand these attackers. OSS is much better on average than closed source, but lets not get complacent, but try and figure out processes to prevent something like this in OSS in the future.
@@MrSmokinDragon I agree, that this wakeup call should encourage everyone to be more on the lookout for such things. My point is, that you can ONLY do that in open source software. With closed source software, you never know. Sure, large companies also do code review, but not all widely used software is written by large companies with good practices in place. And good practice sometimes get thrown under the bus when time or money is short. If you can't see the git repo, the build scripts, the test files, then you have no way find out why your your program is suddenly slower by a factor of 4, or if some malicious employee committed some questionable binary blobs some time in the past. *shrugs*
@@MrSmokinDragon Yes, but the makers of closed software can put these things in on purpose with no one knowing. We know governments are the biggest cartels in the world and can get your cooperation.
Heard about the nist recommend crypto standards.... Quite a few have mysterious issues. Check out en.m.wikipedia.org/wiki/Nothing-up-my-sleeve_number (the counter examples section). Why subvert the code when you can subvert the committee defining the crypto algorithms everyone has to implement. Espionage predates computers.
No lol ? How exactly are you going to get your code, anywhere near, lets say, RUclips or Photoshop`s source code ? I mean sure, you could try get a job at said place, work your way up, and etc etc. But that is vastly more work/time and secure than letting people who created a github account a week ago to contribute to key aspects of the worlds infrastructure. Like jesus.
wow I was thinking it still would have ran as an unprivileged user since I figured they were just getting login but it makes sense it would be able to execute as root since they are using the daemon process to execute. This was a really informative video.
In cryptography, there is this concept of using a "nothing-up-my-sleeve-number" for (technically artbitrary) seeds / initial values within a cryptographic algorithm. For example, the Blowfish cipher uses digits of Pi as initial values in its key scheduling algorithm. This is done to prove that whoever invented the algorithm did not backdoor it by picking constants that are vulnerable to some sort of cryptographic attack only known to him. Maybe the same concept should be established for binary test files. Use known, publicly available images that rarely change (such as the Wikipedia logo). Or, even better, use a well-known, deterministic pseudo-random number generator seeded with a fix "nothing-up-my-sleeve-number" to generate your test files. If your test file needs a specific structure (that is e. g. known to not suit your compression algorithm well), understand what is needed for the test file to behave the way it should, formalize that and then again generate files of the needed structure in an accountable way. This would make it practically impossible to hide stuff in these files. (And additionally make the repo smaller in most cases.)
You can find any arbitrary sequence of digits inside of Pi if you search long enough. Meaning you can find a sequence you like because it's vulnerable, then pretend like you rolled a dice.
@@alexnoman1498 Indeed, which is why you would want to use digits starting from the very beginning. (I think the Wikipedia article on "nothing-up-my-sleeve-numbers" also notes this in one of the first paragraphs.) I just left that part out in the initial comment because it was already long enough. Anyway, my point is: Binary (test) files of which you cannot explain how you created them are inherently suspicious. They may contain a rickroll link stored in low order bits of some pixel values, the mighty flag if you're playing a CTF challenge, or a secret backdoor to-be-extracted by the build script, as in this case.
This is crazy. But I think this just shows the power of the open source. This would be completely undetectable in closed source software. Also, mass audit needs to be conducted, the level of sophistication of this attack makes it almost certain this wasn't the first time they did this. God only knows what has already been compromised.
All thanks to a bored and curious Microsoft engineer who was running a timing test on the SSH login process. The time difference between a normal and infected login was less than 500 milliseconds , and that's what tipped him off. It's pretty remarkable.
Maybe the good approach to catch things like this would be a kind of "enhanced" coverage test. Mark every accessed address in the binary, what was left not marked is either not tested or not needed at best or malicious at worst.
2:45 the public key is used to ENcrypt, the private key is used to DEcrypt. It would essentially be pointless to encrypt something if your decryption key was publicly available.
He's talking about a digital signature, which is achieved by encrypting the message hash with the private key. The idea is anyone with the public key can decrypt the hash, and if it matches their own hash of the message, then it proves two things: 1. Whoever/whatever encrypted the hash (signed the message) has the private key, which serves as authentication, and 2. The message has not been altered since it was signed.
Everyone will forgett about this in a month and we will be back to pressuring maintainers for open source projects to work harder and faster in no time.
The unix philosophy of “many things which each do one thing really well” could make attacks like these harder. SSHD does a lot - and it’s inevitably network facing. Breaking it down to have a separate (unprivileged) process which handles everything up and unto authentication and setuid… wouldn’t necessarily have *prevented* this, but would have made it a lot harder to pull off. I’ve always been a little surprised SSHD has been this way for so long - given most other services make every effort to run unprivileged, then offload switching user to some other process.
I mean sshd can already hand off to libpam for authorization iirc, and realistically, authenticating requests and launching a shell is kinda the whole point of sshd.
I mentioned this over on Sam's gist..., but it looks to me like there was also an attack planned for things built with CMAKE, not just the GNU gnulib build tools... Urk. (Oh, and please don't spend too much time reading M4 if you don't have to, but also consider learning it because this attack leveraged people not understanding it.)
The perfect, most complexe and most elaborate hack was avoided because one random guy found that one random program was a few milliseconds slower than usual 😂😂😂 Remember kids, always track the bloat lmao
he didn't investigate because of slow down, "FWIW, I didn't actually start looking due to the 500ms - I started looking when I saw failing ssh logins (by the usual automated attempts trying random user/password combinations) using a substantial amount of CPU. Only after that I noticed the slower logins."
We should value the simplicity of software a lot more to reduce the attack surface. I would like to appreciate simple software even if they are a little bit underperforming and feature poor.
This is absolutely wild social engineering attack, jia tan enters fix a non problematic bug which decreases efficiency of system (which has negligible contribution before and after)and then jigar kumar (which has no activity before and after) pressurized developer Lasse Collins to maintian the code , they found the weak exploitable link of the chain Lasse Collins, then Dennis Ens enters and pressurizes even more led Lasse collins to forcefully pass the baton to jia tan and the actors played well and jigar kumar forces jia tan now and then they collectively triggered a hard to trace a backdoor but putting a corrupted code, this is insane because i studied cryptography and algorithms to encrypt which are generally very hard to detect i fear that they can be successfull in what they want to do ,they are really smart guys targeted and 15 year old weak link ,imagine how much backdoors does open source and god knows how much in close source software there is a urgent need for "V for Vendetta "for developers, we need to protect developers like Lasse Collins these guys are pillars of human evolution literally.
Where else has this already happened? How many libraries out there are already compromised with this or something like it? Who is behind it and why? How do we go about finding out?
Signature algorithm does not always equal an encryption algorithm. What's been said in this video applies to RSA, but doesn't apply to ECDSA, for example.
It would be a good idea to purge unused and or until needed files from repository along with verifying the uploaded files were built with the repository code and nothing else
Why do you say it like this is just a problem with open source software? I would say the bigger problem is that we don't know how much closed source software does this, and we probably never will because of the nature of closed source.
Yeah, I don't understand this 'argument' ... You have absolutely no way to verify anything about closed code other than nebulous claims made by whoever released the software ... and blackbox testing. It's not like code review is any different for closed-source code than for open -- a trusted contributor who violates that trust could just as well have sneaked this backdoor into a closed project ... and it would be nearly impossible to detect beyond seeing some sus network traffic.
But what was detected by Andres ? His benchmarking showed ssh connections were 4x slower when sshd was backdoored. What was taking so long ? Was it just the decryption of the signing key ?
This slipped through the cracks because it utilized binary files used as "known good" (or bad) values in tests. A fairly straightforward way of reducing/eliminating this attack surface is to instead use a hash of the generated data instead of comparing it to a "known good" binary.
With the level of sophistication of this and the amount of time and resources spent on it. I'm dead convinced that it's a state actor or 3 letter agency behind it...
ARE YOU KIDDING ME?! This was discovered more or less by accident. How unlucky is the perp. He would have had a golden ticket to some serious wealth if he got away with it. And how lucky are we?! This vuln blows my mind.
Yeah... we need something like KYC for important open-source projects. Because, there is this one country to the east of Eastern Europe which will keep doing stuff like that.
Yeah.... In this way you just kill all the appeal of being Open Source. You may as well just rollback to being a closed source walled garden. ReactOS has done this for years (yes, they require KYC for commit access) and check how well goes to them. At least I would never contribute anything or support in any way any project which forces on me any sort of KYC scheme. Embrace, extend, extinguish.
@hyoenmadan No, the entire issue was only discovered thanks to open source. Closed source projects are much easier to infiltrate for organized adversarial actors. @@eight-double-three Even for a whataboutism, this one is kind of lazy.
@@hyoenmadanI think that just increases the chances of stuff like this being missed. This was caught early enough damage potential is low, solar winds from a couple of years back had much bigger impact
![Felix "Unfair" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/Oswujxm2Ag0/mqdefault.jpg)
THANKS FOR WATCHING THIS WEEK HAS BEEN A ROLLERCOASTER ( come learn C at lowlevel.academy 🥺)
tycoon
If I understand correctly, this hack wasn't discovered by auditing any source code, right? I heard the Microsoft guy just had a weird feeling about SSH executing his command for a few too many CPU cycles and was curious enough to investigate. Does anyone knows if it could've been easy to spot this by reading the source code or if it's hidden very well?
@LowLevelLearning Please revisit 2:36
Are there more videos to come on the topic?
is there a specific version of liblzma.so.5 or liblzma5.4.1 ?
'Binaries must be verifiably created using only the code in source control' would be a great place to start.
The problem is this attack was done during the build process.... so you need the source to build it to verify it ... but its tainted ... so .... its valid?
yea... this one was a wild one
@@mitchellmnr This is what the comment is talking about. The source of the build script must be part of the git repo.
@@khai96x they are ....
The way this attack works is when it runs tests on the source...
at that point, you should generate the binary on the fly. not that that would help when the build process itself is infected
It's a decent idea but what do you mean in practice?
Do you run your build in a way that refuses to give it access to any of the test data? That's not too crazy. Is it enough?
Do you verify that others can produce your build without any of the test data?
How do you ensure that the build process doesn't have a side channel running in both cases introducing other logic?
What i'm saying is this idea has to be mapped to concrete practices and tools.
Jeez. Virtually every part of this is done in a way to be practically undetectable at every stage and effectively roots every bleeding edge version of the biggest distros and would have seeped into stable versions of them with time. These people are diabolical, and it’s practically a miracle it was found this early before it got worse. To think, detection started with someone noticing ssh connection speeds were a hair slower. Major kudos to them for being so prudent.
And thanks for the updates on this.
Pretty much. I dont think it could of even been hidden any better.
The main reason anything was looked into, is because someone noticed their SSH login taking half a second longer lol.
Except arch (due to how arch doesn’t patch sshd to connect to systemd-notificationd.)
@@draken5379 Oh but it could be... What if your billion, no trillion $ chip - board makers are embedding hidden algorithms as such but as undetectable hardware roms that the OS, the user, etc. doesn't know about. Then with that only a handful of elite personnel coming from these giant corps as well as limited personnel within some government agencies have direct remote access into these hidden systems.
Think of it this way; some of these newer systems you can load the entire Original Doom Source Code and Assets directly into Modern Cache and never have to go back out to RAM or to the Hard Drive. If you go back to the days of the sneakernet with the good ole floppy disks, people used to sneak malicious code into their 512 byte boot sectors.
They can literally miniaturize a 6502, and 8086, or even a basic Pentium these days into a chip so small it would look like a little controller on the motherboard or even onto some peripheral card that one would have no idea what it is or was truly designed, intended for. This doesn't even account for modern UEFI systems compared to obsolete CMOS Bios type systems.
And now with the advancement of A.I. and designated A.I. chips showing up... The sky's the limit. Well, skynet is the limit according to some... They can always hide it somewhere that is least expected. How about within a button on your polo shirt? You have to realize that we aren't just in the age of digital we are also in the early stages of Quantum Processing, A.I. Systems sure, but we are also on the stage of Nanotechnology. They can literally make programmable machines that are 1,000th the width of one strand of your hair. You can't even see them with your eyes.
So if they truly want to hide something they surely can! And they can hide it just about almost anywhere they want. Even in the foods you eat and the liquids you drink!
And this is not science fiction either!
...that feels like a paranoia syndrome waiting to happen because their fears have been proven so much more horrifically right than they thought. That's going to mark them, considering you already have to be pretty paranoid to check code over half a second. @@draken5379
This time… what about the others?
Not enough people aware that maintainer/og auther (Lesse) is victim of social engineering,
I read those messages in mailing list and it hurts me. Jia basically took advantage of his health condition.
yeah its really sad :(
@@mayshackwe don't know that Jia is one person or an identity being used by a group of people coordinating the attack
@@channelgogrvk I'd guess, it's the latter...
@@eight-double-three NSA, Google, CIA … ?
@@franzlyonheart4362 Probably some asian/russian threat actor. My guesses are North Korea, China or Russia. Maybe Lazarous Group? Killnet is out of question since they do mostly DDoS.
8:56 Agreed that there are certainly vulnerabilities with open-source, but this also highlights the strengths of open-source: having a huge cohort of noble, intelligent, and vigilant coders to sniff-out this type of espionage.
It's frustrating because I deal with people every day who don't like Open Source, but have no problems using SolarWinds!
Imagine this happening with a closed source company. Somebody applies, silently checks this in with the binary files, and there's nobody outside of the company that even has a chance at catching these backdoors.
How many of these backdoors already exist in commercial software? How many does the company owning the code know of?
@@dascandy THIS.
Every time i see somebody talking about how this is "an issue with open source", i think this exact thing. What if this happened in a closed source software? AT LEAST with open-source you're able to find out. With closed-source there's a good possibility that piece of code will never be looked at again because "it just works".
@@justacat3639Thinking back to some part of Windows NT4 shipping with a thing called "NSAKEY" (en.wikipedia.org/wiki/NSAKEY) and nobody knowing what it does or how it got there.
@@justacat3639It is the well
known selection bias.
we definitely need to see the investigation behind the contributor who created the backdoor. definitely very good at social engineering, being subtle, etc. if it weren't for Andres Freund we would have never been talking about this.
We all know who it is, we just can't say who it is.
@@zirize Three letter agency funded by a big brother?
@@zirizeit's not the NSA they have Intel management engine and AMD PSP
@@saddish2816 Why would they stop at a hardware backdoor? Ideally you want to control every step of the computer's stack from bare metal to the user
@@Superchunk-k2hNSA isn't gonna do that
FIB or IAA might tho
If anyone's takeaway here is "I don't happen to use those versions of of that library, I'm good", you did not understand the issue.
That maintainer is still struggeling.
Other maintainers are still struggeling.
Other libraries are still compromised.
The Perp and their Patron are still out there doing this exact thing and more for the next 30 years together with lots of direct and indirect colleagues.
The people introducing vulns are payed exorbitant amounts of money, because buying vulns is more expensive, less exclusive, while just as bad for security.
Getting the maintainers of our critical infradtructure payed and having them audited should be our priority world-wide. Every company in every country uses FOSS, but vanishingly few pay for it. Ask your manager whether your company pays for the software they are using!
While I agree with you in spirit, if you're running RHEL and paying for it, IBM isn't giving the maintainers of xz a cut of it, so just paying for FOSS in a corporate setting doesn't exactly solve the problem either.
I totally agree. Corps generate hundreds of dollars per tool used, but they don't even donate a dollar. Then we get a core-js or faker.js situation and people lose their minds
whats scary is that this was only discovered, because the backdoor was slowing down ssh as well as generating a lot of errors in valgrind.
I'm sure I'm not the only one who was reminded of Clifford Stoll's "The Cuckoo's Egg" (1989), where a 75-cent computer usage accounting error ultimately revealed a remote attacker with superuser access.
to be fair the attacker had to disable and ignore several address sanitization features and test checks. Hence why stuff like valgrind was already setup to catch stuff out for this particular project.
so it does make me wonder how many of these backdoors are already out there, just not discovered?
And the attacker was working with one distro contributor to help fix the valgrind errors. And the said contributor, AFAIK, was helping in good faith.
@@AeduoI don’t really know jack about software dev and I was still able to use valgrind to find a memory leak in some important software and report it
I still can't fathom how incredibly lucky we are this got caught as quickly as it did. I still haven't found a system on stable that actually shipped the compromised versions. We don't always get a warning shot like this.
The real question now is: how many times have we not gotten so lucky?
just curious.... why would large organizations be using operating systems that get updates from open source git repositories? maybe im understanding this attack wrong just curious if you dont mind educating me
@@Noname23489 A sizable portion of servers on the internet run some version of Linux. There's no good data, but I've seen stats that are well over 50%.
And everything based on Linux is at least partially relying on open-source code.
/cries/ I use arch BTW :(
@@Noname23489 that's basically asking "why would large organizations run linux", and there's lots of reasons to do so.
The program being built being able hook back to the build process and change the linker behavior, worse -during the test phase-, is also super crazy.
It does some really complicated/shady things like altering the Global offset table and character recognition, I wonder if we ever get to find out who was behind this
As much as I love the technical aspects of this, its not what we should be focusing on. The "nature of [this vulnerability] of open source software" was primarily and overwhelmingly social. The project had a solo maintainer that was hopelessly overwhelmed. He was conned by one or more anonymous individuals into accepting a malicious co-maintainer by both attacking his weaknesses as the project owner, and offering him the answer he was desperately looking for to save his labor of love project. He was pressured into trusting code and a developer without sufficiently scrutinizing either.
My takeaway is that open source is only as strong as its weakest link. So many projects go to the graveyard because of the lack of maintainers, and there's a lack of maintainers because people need to eat, need a roof over their heads, so they work their 9-5 as software engineers, which is draining in of itself. So when a crucial piece of software has a solo maintainer, it's a prime target for attacks. If there was more compensations for open source maintainers, maybe some of them could afford to maintain the software full time, and as a result would not burn out and be less prone to social engineering attacks.
Or we could just create a foundation for those crucial projects, so that there are always developers able to maintain the software, and need to reach a consensus before anything gets merged.
@@Dogeek I think all the companies that used open source software in their commercial projects on a massive scale that command massive profits should be obligated to divert a fraction of a percentage towards maintaining these things. Or better yet when there's a massive fork and that fork goes on to make billions of dollars, leaving the original project in the dust and penniless there should be something that allows maintainers to continue doing what they love to do.
I love open source but the (un)ethics of business undermines such a great ethos it really becomes disgusting. Sucks to see such openness taken advantage of again and again, and even more now since technology companies rule as the market leaders in today's world.
Not even that, he knew the attacker for YEARS before he let him into the project. nearly 4 years. And the guy was nothing but nice and helpful until a month ago.
Nobody, not a single person on this planet, would have not trusted him by that point.
@@瑠ちゃんI wouldn't trust him until I saw how he handled LE or drugs/alcohol. Being internet based, I don't trust nobody.
Many projects do not need active contributors. Just stop adding more features.
I would not call this a vulnerability in open source, rather it's a strength. This could, and probably has, happen in closed source software too. They can just get some person to apply to a job and do the same thing. The only difference is that in closed source it's far fewer people that could potentially find it. I think one of the reasons it was found relatively quickly is precisely because it was open source.
Agree 100% with this, the fact that open source code is evaluated and run by so many independent people is a strength that leads to things like this being found. If this change was to a closed-source tool it could sit in a repo for a decade and no one would notice :/
It was more luck and that the new version caused half a second delay that a person was doing performance tests at the time most other people probably wouldn't have noticed unless they was doing performance testing, this was pure luck as it wasn't that Mutiple people picked up on this it was just 1 person
if they fixed the delay part it wouldn't have been noticed until a couple of years later on when some high profile targets got compromised and don't know how
What's interesting is the timing. We see the attacker working hard to make Ubuntu include the compromised versions in their 24.04 beta, but also trying to convince fedora to include it in fedora 40. This makes me believe they might've been constrained by time at the end be cause these two releases are very important: The next Ubuntu is LTS and the next Fedora is likely what the next RHEL is going to be based on a year from now
systemd was transitioning to dlopen
I feel really bad for Lasse. Jia probably looked like an angel sent from heaven to help with project maintenance at a time when they weren't doing so great. Then all this comes out, and they have to confront this nightmare with suspicion cast even on them. It's like a dad finally handing his keys to his daughter for the first time to drive on their own and within 15 minutes she drives through a crowd of pedestrians.
Yeah even going through their commits the majority of them are benign improvements... And then this, with the seeds having been planted in the test files ages ago
More like a snake you sheltered finally showing its true color and biting you to the ground
@@nuke_parasitine nah, the contributer, before the backdoor, was only ever helpful and nice.
If you knew someone for years and they were only ever lovely to you - and it turned out they were a serial killer - it would be a horrible thing for people to imply it was only ever obvious (they were a snake, you let them in) when it wasn't.
@@瑠ちゃんthe shit you nerds say lol.
@@atticus2274 Nerd calling a Nerd a Nerd is peak irony.
cs undergrad here. A lot of this is still over my head, but it is so interesting. Honestly just inspires me even more to learn. I appreciate your breakdowns that are very clearly presented and approachable.
the scale of this attack is absolutely insane
Oh? You mean how this backdoor didn't affect any LTS distributions used in servers or production environments but only affected rolling release and testing distributions? That scale? In other words, only desktop users who are stupid and naive were affected.
@@adibemaxwell6111 My dude, what do you think 90% < of the internet runs on?
@@adibemaxwell6111 It targeted all Debian based distributions. The only reason only rolling distro's were affected is because that Freund dude just so happen to be benchmarking his PostgreSQL databases and therefore end up noticing the backdoor. If he didn't it could have been planted in LTS distro's as well after some time. Also, the bad actor has been committing to the repository for a few years and it's still being investigated if there's more malicious or at least sketchy code in there, which if there is it could mean that even older versions of the libraries could be compromised in some way, which would also mean compromised LTS releases. You have to be short sighted to think these sort of attacks is solely an issue for people running rolling release distro's...
@@adibemaxwell6111 If it hadn't been discovered, it would be on like 80% of servers within 6 months. The only reason it didn't is because some Microsoft employee noticed high CPU usage from sshd and looked into it.
@@adibemaxwell6111 nice comment to generate engagement, go ahead dude.
This is a variation on the theme. Open Source projects may only have one or two maintainers. We had a wakeup call with OpenSSL a decade ago. This is just a sneaky supply chain attack via compromise of liblzma and oss-fuzzer. Trust was assumed, but trust was broken. How will single maintainer OSS projects audit all commits to their project if they are hobbyists or part-timers? The payload was to be included in deb or rpm packages during build via checks. That was another way to ensure it got into the main distros, but was low key.
Large developers that depends on smaller OSS projects needs to contribute to maintain all the projects they depend on. Either man-power or money.
I think this is the modern variant of "Reflections on Trusting Trust" by Ken Thompson.
What's that and how is it relevant to the topic?
@@smnomad9276you should read it. It’s about “how can you trust your own trust chain” and what can happen if that trust chain is broken. “You can not trust code you did not compile yourself”.
@@smnomad9276 go read about it, you have the entirety of the internet at your hands.
@@smnomad9276 A famous speech/presentation he gave as part of an award acceptance speech.
It's basic premise is "what if I maliciously modified the compiler included with the UNIX system to do two things - 1) if compiling a system component, insert a backdoor that allows me full access, and 2) if compiling a compiler, modify the output compiler to perform these two steps."
Basically saying, "how far do you trust what's been given to you?"
Trusting Trust is a much deeper concept, and the reaction to it was the practice of Diverse Double Compiling. This trust gap is way easier to deal with, and more analogous to the use of nothing-up-my-sleeve numbers in cryptography. Distributing binary blobs alongside source code should be treated with great suspicion. Let test data be programmatically constructed, and therefore auditable the same as code, or by some other means be provably incapable of hiding machine code (e.g. pi).
That level of obfuscation though, the backdoor glows very bright
phosphorescent, even
Stop being antisemitic
@@GelatinousSSnake i thought glowing is a fed reference, what does it have to do with semitism
@@GelatinousSSnake Glow = government agent, not anything relating to ethnicity.
@@GelatinousSSnake wtf 😂
one way you could prevent something like this would be disallowing binary blobs, and if you need it for testing, etc, it should be generated by code that is readable. For example a shell script that generates a large amount of the letter a compressed with xz and then intentionally corrupts it.
Though this doesn't prevent people from doing bad stuff like this, but it makes it harder to hide. (people can still intentionally introduce bugs in the code that can lead to something like heartbleed for example)
You could also mandate that test data must live in a separate repo, and must only be used while running the tests (in an isolated sandbox). Also the build system should be less flexible so that any weird stuff really stands out. But all of that is moot if it's the maintainer doing bad things and no one else actually reviews any of it.
@@ville_syrjalaAlso inspect binary blobs before they are uploaded to a repo.
the main issue is the main issue that has plagued open source recently: business expects services and support from the open source community, but is unwilling to pay for them. in this case, we have a security critical library being maintained by a single person. in no sane world would i expect that to be secure. system critical libraries either need to be fully audited and then quarantined with no more updates possible, or they need well paid, dedicated, redundant staff and regular security audits. the new layoff culture and over investment in ai (and underinvestment in mental healthcare) is going to do massive, yet untold damage to computer security.
Wow this guy is some sort of evil genius. Appreciate the constant video updates, it's great to have someone summarise these really complex topics but still keep the essence of whats happening.
Guy? Or guys, plural? Or … state sponsored? NSA, CIA, ((Washington)), etc.?
@@franzlyonheart4362 its always a government conspiracy, eh?
@@MaRBL23563 It does make sense in this case though
@franzlyonheart4362 why is everyone in these comment sections assuming it's got to be an American agency? My money would be on China, Russia, or Mossad. Hell, Mossad's bread and butter is this kind of diabolical social-engineering into unnoticeable attack vector.
@@franzlyonheart4362 No reason to assume it has to be the US government. Could be state-sponsored hackers from China, Russia, or North Korea, too! But with so little information at this point in time, I don't think it's worth baselessly speculating. It could totally be non-state sponsored, as well-there's a lot of money to be made from having this sort of backdoor access.
Really thankful for Andres Freund's agile mind; finding a needle in the 'stack is hard enough; he found a bd even though he wasn't looking for one, anticipating one.
Remember kids: writing a backdoor is no excuse to write poorly performant code that wastes CPU cycles so much it alerts the authorities.
THATS IT! im done
im building my own O.S. and all the libraries it need myself, and running on a custom board with a custom riscv processor
Little do you know that riscv could also have a backdoor. mwahahahaha. You must create a completely custom architecture and custom silicon to be sure.
and also blackjack and hookers
@@mineton1293oh but then there's backdoors built into your circuit design tools as well, so have fun arranging logic gates entirely by hand
@@mineton1293 Easy AF, I'm taking my soldering iron and going to work :P
one of the games i play (openttd) uses xz for savegame compression, and now the latest release is delayed.. because of the backdoor
I LOVE openttd. didnt realize it uses xz D:
S**t just got serious - no-one is touching my OpenTTD! 🤪
That's it!! I freaking knew I recognised xz from some ancient memory!
Not sure how OpenTTD would ever be affected, even if using xz with a backdoor, since it replaces RSA decryption function, and gets the payload from RSA key supplied to SSH.
@@DVSoftware OpenTTD may not be vulnerable to xz itself, but it certainly is vulnerable to GitHub taking down xz's repo and users getting pissed about potential malware being included with the game.
So, as I understand it, if the compromised version made it to the server distributions, the only way to guard against it would be to limit who can connect to ssh using iptables (or some other firewall), since the attacker would still need to connect to the ssh port to send the commands.
That, or install an uncompromised xz on that server instead.
To me, the scariest part of software these days are all of those 3rd party libraries. You ever install something with "sudo apt-get install X", and you'll see a stream of things being downloaded. Even just one part of one of those libraries could be infected and who would know? MD5 hashes or whatever would do nothing here because this was compiled into the release itself. Frightening as hell
Dependency graphs give us the tools to fix problems once instead of over and over and over. They also allow us to be more effective and efficient in our work.
They also let risks and failures cascade through our ecosystems in astonishing ways. Not all of the problems are even malicious. left-pad in nodejs was just someone unpublishing his work and it took down tons of sites.
What's the answer? i have no idea.
But if we focus on practical situations we can perhaps come up with practical mitigations that reduce harm, and improve.
@@_devilfish303 The point is Linux in its current state doesn't provide better protection than Windows. The slogan "you can always recompile everything locally to make sure the binary matches the source code" doesn't work anymore. Because every single Linux library is dependent on dozens of others. It's physically impossible to recompile everything on your own. And even if it was possible it would still give no guarantee as this attack was built directly with a hook to inject the binary malware during the compilation process without altering the source files.
Yes Microsoft is not better.
But it's not worse either. And this is the true issue here. Why bother with all the hassle of securing your linux if anyone can trash all your efforts with a single commit in some obscure source repo you never heard of?
curl | sudo is the one that terrifies me. Literally random code from internet run as root. Multiple attack vectors there from ip / DNS hijacking through to just straight up blobs in the install script... Plus potentially no evidence of what you just downloaded
@@christianbarnay2499microsoft is not worse? If such a backdoor was installed in windows, it wouldnt get discovered EVER. Even a simple babckdoor wouldnt ever get discovered, which wouldnt even get merged if it was open source. Security isnt black and white. No system is ever completely secure. Its about reducing the attack vector. And the attack vector in open source or linux is extremely small compared to proprietary.
@@christianbarnay2499 Gentoo is from what I understand a distro where yes, recompiling everything is practical.
I've been struggling with this topic, but your video cleared it up for me. Thanks a ton!
this backdoor is just... brilliant
@7DuRd3n In the shower
I've heard something like that, I thought it was common
@7DuRd3n geniuses that have specific domain knowledge sharing ideas with others in their domain knowledge. this most likely was a state operation.
you think this is wild, imagine the concepts and ideas in modern mathematics where you are not constrained by physical limitation. everything is abstracted and abstracted.
Can we agree that whoever made this backdoor was pretty skilled
Also the guy who found it is a hero for being diligent and trying to optimize and finding it
I'd love a video that summorizes the entire thing in one, to be able to send that to people and inform them
It's cool that this vuln has been found, but the scariest thing to me personally is the very real possibility of similar backdoors that may already be present in other widely used OSS projects. The fact that even this one was found by total accident is genuinely terrifying.
Small OSS projects like this usually have a very high bar for performance to meet, and they usually meet them no problem, so it makes sense that the valgrind performance anomalies were the reason why this one got discovered.
Now imagine trying to discover "performance anomalies" in commonly used proprietary software...
what's really terrifying to me was how it was legitimately almost not noticed in the first place
That's because everyone is under the impression that these kinds of things are rigurously checked by someone else, therefore no one checks them. This should be a wake up call.
Who would win: A multibillion dollar government agency, or one giganerd noticing his ssh takes 0.5 sec longer to log in?
That's a W for the nerds, boys.
@@TheOzumat Obsessive optimisation is the root of all evil - and that is why it also naturally finds it.
Dude wanted to optimise performance and probably spent days untangling the mess he ran into - just because he wanted a near unnoticeable improvemet.
Everyone's calling it a W, but imagine all the projects that aren't being autistically benchmarked by someone with a unique problem.
I've been following this as it was unfolding over the weekend. Just watched the first 3 minutes of your video. The fact that this backdoor is very effectively secured against being used by anyone else than the attacker is a super strong signal for me that this is not some random cyber criminal but a state actor. The sophistication and attention to detail in this whole exploit reminds me a lot of stuxnet. Honestly, it's a whole notch up from stuxnet in a lot of ways.
Without question. This was a well-funded state-level attack, likely one of the 1st-world governments.
One key correction here is that the hosted git repo does not contain the necessary code to build the backdoor. Only the private repo for the library maintainers contains the necessary code. Your highest upvoted comment already starts off on the wrong track. I spent a lot of time sorting through this confusion today.
a great father who spend time with his children AND keep up with youtube, keep up the great work!
Isn’t the fact that this was caught proof that more people are watching what is going on in open source? This was the best case scenario for a malicious attack and the openness of the code allowed having all the evidence available. I agree that this should be the start of regular audits of crucial open source software. But the log4j exploit a few years ago might have had some influence on the Microsoft employee’s idea to dig deeper.
I see this as a large opportunity for more secure code practices in open source.
this was found by chance more than anything, it's only because someone tried to get a good baseline for their system that they noticed that sshd was doing more work than it should and dug into it
If that didn't happen then this would never have been found
Well yeah. But a lot of bad actors are now fear mongering so that people get scared of open source projects. We should doubt every project, for sure, but we have the ability to look at code and the whole build process and catch such exploits. But of course, as this example showed, there are simply too many projects for people to realistically screen with every commit. But we have to start being more cautious, starting with the most critical packages would be a good beginning.
@@ratchet1freakBut on the other hand, if the library hadn't been open source, he wouldn't have been about to look into it as deeply as he did. He likely would've had to conclude that the developer introduced a bug, and at most reach out to them, which would've only served to tip the bad actor off that their backdoor was still too detectable. So overall, open source did "win" here.
@@helipad_huck1543as it happens, someone else actually did exactly that from my understanding, and that was the patches that the second version of the backdoored version of xz fixed, some valgrind issues that were found by a maintainer of a different distro, so that's xz 5.6.1 :P
A couple thoughts on prevention. Move the test environment to a temp folder, completely separate from the source, and only run the tests on a stripped binary, so that the actual source cannot be found by the tests.
Plot twist. Andres has the private key. He did this all to rise to stardom in the security world. Being discovered only strengthens his reputation of supreme social engineer. He’s brilliant either way.
This is so wild, excited to see what else is found!
Binary blobs are basically immune from standard audit, I don't see much of a way to prevent this kind of attack except to ban blobs from being included in source control. Which is obviously impractical in certain scenarios.
I saw the twitch livestream where he followed the command cmd with a shutdown... and then his vm went dark. That was so awesome.
Imagine how pissed off the exploiter(s) must be right now... "Noooo, it was perfect!"
They spent years setting all this up, too.
And yet they managed to calmly disappear once it was discovered, instead of yelling insults at the xz mailing list. True professionals!
@@Uerdue As opposed to when they were setting this up where they (presumably this was them) were quite rude.
I have a feeling that it's not just a simple uber haxor kid, this is too big and the setup is too perfect in the way it was being set up slowly. I don't want to sound like a conspiracy theorist but I just have a feeling that the guy who we all saw as a co-maintainer is not the sole brain behind this attack. It had to be a group who would greatly benefit from having this backdoor available.
@@Cavi587I think most people consider a state-level actor (like some intelligence agency) the most likely behind this.
It's not just the amount of time/work spent on it, it's also the likelihood of failing which was too high for the average cyber criminal without government funding to risk so many resources on it. What if systemd drops its xz dependency 2 years into the project? Or another contributor shows up who likes to do his own improvements to the tests or build system (that touches your manipulated files, so you have to turn down all of their PRs for made-up reasons)? Or the original maintainer is just stubborn and says "no binary blobs of unknown source as test files, period"? Or some person simply finds the backdoor before it makes its way to stable distros?
There are so many ways for such an operation to go wrong. You either have to be able to scale it, working on a bunch of these in parallel and hoping for one to succeed - or you have to just be ok with the operation failing and the only thing gained from it being the insight that it's quite a hard thing to pull off successfully.
I don't think the brilliance of this evil attack can be understated. Not only is it a backdoor, it is a backdoor where only one person or group has the key to open!
Great demo. From a cybersecurity point of view, this should teach you to think bigger, implement zero trust and secure your platforms beyond just the front door. Treat every server/service as if it's already been compromised, and limit access from that perspective.
Just so impractical :( I just want to use my system and write things that jave some power to alter things. A big stack of sandboxes and passwords is so frustrating
@@kapytanhook then we need to make that easier. I hated the idea of 100s of passwords (for example), but a password manager has made that easy.
@@hellowill yeah, honestly. But this is deeper, it's that you can't trust your own machine or things running on it. People put backdoors in everything. We need well funded open source hardware and software but we are moving away from that
An issue with this entire thing is that we keep trying to invoke a technological “Web of Trust” to let us bind dozens of dependencies into a single binary, but instead we should perhaps be encouraging larger teams to form. Many projects have one or two or three primary authors who carry the entire load alone, which is intrinsically broken over the long term.
But how would the attacker know which server got infected? Does it phone home in any way, or was the attacker hoping to just attempt connecting to random open ssh servers accessible through the internet by scanning IP ranges?
the latter is very viable, given that there are botnets pinging every single public IP address right now in hopes of hitting something
Attacker prob patiently waiting for their code to be published with stable release then do crazy stuff. Remember it got exposed just because of 1 dude noticed ~500ms delay on his ssh connection.
After the hard part is done, the attacker just hopes the biggest targets are infected.
@@gie4830 500ms is crazy delay, no way this backdoor will come to stable
@@gie4830
I don't know. I'm pretty sure actual big actors don't have SSH servers directly addressable through the public internet, you most likely need to jump through some enterprise VPNs to get access. So the targets were mid-sized.
At least now he's vulnerable to getting honeypot'd. Wonder if someone will attempt to catch him like that. Though it would have been easier to not alert everyone if he really was targeting only a few actors.
This whole thing glows brighter than the sun
So, are you using a virtual machine setup on an air gap?
I usually use a set of virtualbox machines in their own virtual network on an old notebook for stuff like this. Would love to see how your setup looks for cases like these.
I don‘t understand how people can frame this as „a problem of open source / linux“. It is only BECAUSE the whole thing is open source, that this backdoor could be found AT ALL. Think about how it would be infinitely easier to do this in a closed source software!
not necessarily. Many OSS projects have only single maintainers, which makes this kind of attack easier. Large software developers have extensive code reviews with many eyes looking over changes before commit, meaning that a malicious actor (other than the company itself) would have to fool or corrupt many developers to sneak in the backdoor. I am not saying that Closed-Source is in any ways better, what I am saying is please don't try and make it seem that OSS is always better able to withstand these attackers. OSS is much better on average than closed source, but lets not get complacent, but try and figure out processes to prevent something like this in OSS in the future.
@@MrSmokinDragon I agree, that this wakeup call should encourage everyone to be more on the lookout for such things. My point is, that you can ONLY do that in open source software. With closed source software, you never know. Sure, large companies also do code review, but not all widely used software is written by large companies with good practices in place. And good practice sometimes get thrown under the bus when time or money is short. If you can't see the git repo, the build scripts, the test files, then you have no way find out why your your program is suddenly slower by a factor of 4, or if some malicious employee committed some questionable binary blobs some time in the past. *shrugs*
@@MrSmokinDragon Yes, but the makers of closed software can put these things in on purpose with no one knowing. We know governments are the biggest cartels in the world and can get your cooperation.
Heard about the nist recommend crypto standards.... Quite a few have mysterious issues. Check out en.m.wikipedia.org/wiki/Nothing-up-my-sleeve_number (the counter examples section). Why subvert the code when you can subvert the committee defining the crypto algorithms everyone has to implement.
Espionage predates computers.
No lol ?
How exactly are you going to get your code, anywhere near, lets say, RUclips or Photoshop`s source code ?
I mean sure, you could try get a job at said place, work your way up, and etc etc.
But that is vastly more work/time and secure than letting people who created a github account a week ago to contribute to key aspects of the worlds infrastructure. Like jesus.
wow I was thinking it still would have ran as an unprivileged user since I figured they were just getting login but it makes sense it would be able to execute as root since they are using the daemon process to execute. This was a really informative video.
In cryptography, there is this concept of using a "nothing-up-my-sleeve-number" for (technically artbitrary) seeds / initial values within a cryptographic algorithm. For example, the Blowfish cipher uses digits of Pi as initial values in its key scheduling algorithm. This is done to prove that whoever invented the algorithm did not backdoor it by picking constants that are vulnerable to some sort of cryptographic attack only known to him.
Maybe the same concept should be established for binary test files. Use known, publicly available images that rarely change (such as the Wikipedia logo). Or, even better, use a well-known, deterministic pseudo-random number generator seeded with a fix "nothing-up-my-sleeve-number" to generate your test files. If your test file needs a specific structure (that is e. g. known to not suit your compression algorithm well), understand what is needed for the test file to behave the way it should, formalize that and then again generate files of the needed structure in an accountable way.
This would make it practically impossible to hide stuff in these files. (And additionally make the repo smaller in most cases.)
You can find any arbitrary sequence of digits inside of Pi if you search long enough. Meaning you can find a sequence you like because it's vulnerable, then pretend like you rolled a dice.
That's just silly. They're test files.
@@alexnoman1498 Indeed, which is why you would want to use digits starting from the very beginning. (I think the Wikipedia article on "nothing-up-my-sleeve-numbers" also notes this in one of the first paragraphs.) I just left that part out in the initial comment because it was already long enough.
Anyway, my point is: Binary (test) files of which you cannot explain how you created them are inherently suspicious. They may contain a rickroll link stored in low order bits of some pixel values, the mighty flag if you're playing a CTF challenge, or a secret backdoor to-be-extracted by the build script, as in this case.
The starting point would be its own magic number which should be a 'nothing-up-my-sleeve-number'@@alexnoman1498
That concept exists because of stuff like Dual_EC_DRBG (although Blowfish predates it).
Terrifyingly well executed 🥶
This is crazy. But I think this just shows the power of the open source. This would be completely undetectable in closed source software. Also, mass audit needs to be conducted, the level of sophistication of this attack makes it almost certain this wasn't the first time they did this. God only knows what has already been compromised.
All thanks to a bored and curious Microsoft engineer who was running a timing test on the SSH login process. The time difference between a normal and infected login was less than 500 milliseconds , and that's what tipped him off. It's pretty remarkable.
@@BillAnt That's a story of human evolution haha.
Maybe the good approach to catch things like this would be a kind of "enhanced" coverage test. Mark every accessed address in the binary, what was left not marked is either not tested or not needed at best or malicious at worst.
People should be sifting through all kinds of archived conversations involving project maintainers looking for signs of social engineering.
2:45 the public key is used to ENcrypt, the private key is used to DEcrypt. It would essentially be pointless to encrypt something if your decryption key was publicly available.
He's talking about a digital signature, which is achieved by encrypting the message hash with the private key. The idea is anyone with the public key can decrypt the hash, and if it matches their own hash of the message, then it proves two things: 1. Whoever/whatever encrypted the hash (signed the message) has the private key, which serves as authentication, and 2. The message has not been altered since it was signed.
@@galacticminx Alright, thanks for clearing that up :)
Everyone will forgett about this in a month and we will be back to pressuring maintainers for open source projects to work harder and faster in no time.
Thank you for the video 👍
The unix philosophy of “many things which each do one thing really well” could make attacks like these harder. SSHD does a lot - and it’s inevitably network facing. Breaking it down to have a separate (unprivileged) process which handles everything up and unto authentication and setuid… wouldn’t necessarily have *prevented* this, but would have made it a lot harder to pull off.
I’ve always been a little surprised SSHD has been this way for so long - given most other services make every effort to run unprivileged, then offload switching user to some other process.
I mean sshd can already hand off to libpam for authorization iirc, and realistically, authenticating requests and launching a shell is kinda the whole point of sshd.
sshd _does_ support privilege separation though. I assume the backdoor activates on the secure side of it though (or disables it entirely).
I finally understood his with your video, thank you!
well anticipated video
I mentioned this over on Sam's gist..., but it looks to me like there was also an attack planned for things built with CMAKE, not just the GNU gnulib build tools... Urk.
(Oh, and please don't spend too much time reading M4 if you don't have to, but also consider learning it because this attack leveraged people not understanding it.)
The perfect, most complexe and most elaborate hack was avoided because one random guy found that one random program was a few milliseconds slower than usual 😂😂😂
Remember kids, always track the bloat lmao
It wasn’t a few milliseconds, iirc it was 500ms
he didn't investigate because of slow down, "FWIW, I didn't actually start looking due to the 500ms - I started looking when I saw failing ssh logins (by the usual automated attempts trying random user/password combinations) using a substantial amount of CPU. Only after that I noticed the slower logins."
Had libzma not been open source we ought to forget about ever discovering such implement!
We should value the simplicity of software a lot more to reduce the attack surface.
I would like to appreciate simple software even if they are a little bit underperforming and feature poor.
Also, code freezing or "don't fix what isn't broken" was not so bad idea after all.
Perhaps the popularity of BSD is about to increase!
This has nothing to do with "simplicity of software" but whatever
@@DaveBucklin
I hope, BSD is such a great system
Really great breakdown!
the NSA must be pissed that they lost their best backdoor
more likely the chinese
As if that was the only attempt - it's the only one that failed, but numerous others have succeeded
Yeah, this isn't a lone actor. It shows you how much more complicated it is to compromise an open source project.
@@outtakontroll3334 both
Would the NSA really be silly enough to try and backdoor an open-source project?
i don't know enough about the actual impact
but this feels like the most of the internet could've gone dark due to this
"turn yourself in" xD
This backdoor is really moving my career forward
1:54 "thank god the researcher at Microsoft found it"
Andreas Freund: 🙂
What's concerning is what other software could have been compromised using this method and if it already has been.
7:20 too late buddy. I will WinNuke95 you! 😎
LOIC INCOMING
Enjoy your new cupholder *ejects CDROM drive*
Great video! Thanks for information.
This is absolutely wild social engineering attack, jia tan enters fix a non problematic bug which decreases efficiency of system (which has negligible contribution before and after)and then jigar kumar (which has no activity before and after) pressurized developer Lasse Collins to maintian the code , they found the weak exploitable link of the chain Lasse Collins, then Dennis Ens enters and pressurizes even more led Lasse collins to forcefully pass the baton to jia tan and the actors played well and jigar kumar forces jia tan now and then they collectively triggered a hard to trace a backdoor but putting a corrupted code,
this is insane because i studied cryptography and algorithms to encrypt which are generally very hard to detect
i fear that they can be successfull in what they want to do ,they are really smart guys targeted and 15 year old weak link ,imagine how much backdoors does open source and god knows how much in close source software
there is a urgent need for "V for Vendetta "for developers, we need to protect developers like Lasse Collins these guys are pillars of human evolution literally.
Great video! Thank you.
You are welcome!
andres freund is legit a hero
You could say he is a freund to us all
Where else has this already happened? How many libraries out there are already compromised with this or something like it? Who is behind it and why? How do we go about finding out?
Sheer luck it didn't get into the next Ubuntu, attacker missed the change freeze by hours, this could have been massive 😳
Great content 🎉 thanks for sharing
Thanks for watching
Signature algorithm does not always equal an encryption algorithm. What's been said in this video applies to RSA, but doesn't apply to ECDSA, for example.
It would be a good idea to purge unused and or until needed files from repository along with verifying the uploaded files were built with the repository code and nothing else
Why do you say it like this is just a problem with open source software? I would say the bigger problem is that we don't know how much closed source software does this, and we probably never will because of the nature of closed source.
Yeah, I don't understand this 'argument' ... You have absolutely no way to verify anything about closed code other than nebulous claims made by whoever released the software ... and blackbox testing. It's not like code review is any different for closed-source code than for open -- a trusted contributor who violates that trust could just as well have sneaked this backdoor into a closed project ... and it would be nearly impossible to detect beyond seeing some sus network traffic.
But what was detected by Andres ? His benchmarking showed ssh connections were 4x slower when sshd was backdoored.
What was taking so long ? Was it just the decryption of the signing key ?
Finally i just finished your video with Prime
Any courses/books recommentations where someone can learn all this stuff to better understand what is being said in the video?
There's an Encryption course from UC San Diego on Coursera that gets into the implementation details of RSA.
Starting to sound like a government 3 letter agency at work in the night 😮
*GCHQ hackers sighing with relief*
The public key is used to ENCRYPT and the private to DECRYPT.
He was talking about signing not regular encryption. He got it the right way round!
Not for signatures, he is right.
This is MojoJojo level Evil.
This slipped through the cracks because it utilized binary files used as "known good" (or bad) values in tests. A fairly straightforward way of reducing/eliminating this attack surface is to instead use a hash of the generated data instead of comparing it to a "known good" binary.
And now you have to wonder what is out there that has not been discovered by 500ms noticers
With the level of sophistication of this and the amount of time and resources spent on it. I'm dead convinced that it's a state actor or 3 letter agency behind it...
ARE YOU KIDDING ME?! This was discovered more or less by accident. How unlucky is the perp. He would have had a golden ticket to some serious wealth if he got away with it. And how lucky are we?! This vuln blows my mind.
Wow, this is nuts! 😮
Can you rickroll everyone with this?
the acariest bit to me is that this was noticed, but how many other things havent been
Yeah... we need something like KYC for important open-source projects.
Because, there is this one country to the east of Eastern Europe which will keep doing stuff like that.
Yeah.... In this way you just kill all the appeal of being Open Source. You may as well just rollback to being a closed source walled garden. ReactOS has done this for years (yes, they require KYC for commit access) and check how well goes to them.
At least I would never contribute anything or support in any way any project which forces on me any sort of KYC scheme. Embrace, extend, extinguish.
Historically many countries have done many unethical things many times...
@hyoenmadan No, the entire issue was only discovered thanks to open source. Closed source projects are much easier to infiltrate for organized adversarial actors.
@@eight-double-three Even for a whataboutism, this one is kind of lazy.
@@hyoenmadanI think that just increases the chances of stuff like this being missed. This was caught early enough damage potential is low, solar winds from a couple of years back had much bigger impact
easy solve is to just bit-shift the local register, cast to double and then spin it thru tornado cash