OpenSSH backdoor'ed via XZ & Systemd on RedHat & Debian systemd: multi-year effort by state actor?
HTML-код
- Опубликовано: 5 сен 2024
- #SSH #Bacdoor #XZ #Debian #RedHat #t2sde #Ad: laptops & more @Amazon: services.exact... You can support my work at: / renerebe github.com/spo...
exactcode.com t2sde.org rene.rebe.de
Dependency and build system nightmare, complexity in the software industry is reaching a critical point. This is systemic. Anyone stupid can make something complex. Bad actors will always be there.
time to start over with openBSD. they never allowed binary blobs.
well, there arelso no binaries in the regular sources. This was re-using test-suite binaries. Sure, if OpenBSD has probably no test suite, ... ¯\_(ツ)_/¯ The only substitute for test suite is more test suite test, ...
@@MoreReneRebe yeah, maybe openBSD is also susceptible to this kind of attack, but at this point, i'm assuming the all the closed source drivers on linux are backdoored.
@@happygomonkey you don't need to.. learn to separate stuff and have diverse systems and you're mostly good. I've never had a vulnerability I didn't mitigate mostly through that easy concept. It's no guarrantee, but it makes a huge difference.
Also, why not mix all OSes? :D
Currently, I use: OpenBSD, FreeBSD, various linux distros and even solaris variants, amongst MS Windows and in private Haiku (which is really awesome, actually).
@@xxxblackvenomxxx you're misunderstanding my point. i'm saying that the openBSD philosophy was right all along.
when i was a kid, i installed 3 different linux distros, openBSD, and hackintosh all on one hard drive just for fun. yes it was fun, but its not a real solution to the closed source problem. you have no way to verify if you actually mitigated your vulnerabilities.
Makes you think what else is compromised and we have no clue 😮
It has been 30 years that I ask myself the same question ..
They got to lucky charms and DMT
If anyone inspected my backdoor they’d find a bunch of farts.
Good job to Andreas FR. Saved our lives
Who is andreas freund? sounds like a made up name. he's probably the one who is behind the whole ordeal.
@@jazzochannel Andres Freund works at Microsoft as a Principal Software Engineer. Probably not behind it.
The problem is described well in xkcd 2347: All modern digital infrastructure rests on a project some random person in Nebraska has been thanklessly maintaining since 2003.
code reviews are only effective if theyre done critically.
it should not be taken personally.
its the code thats being criticized, not the author.
Of course !
Ln:0 "Theyre"-> "They're"
Ln1: keep consistency and use abbreviations everywhere, or not. "Should not" -> "Shouldn't"
Ln2: ditto "its-> it's", "thats->that's"
My cheeky comment aside. I agree that code reviews shouldn't be taken personally. I have worked at places that have a bully culture, and some of the code reviews come across has an extension to this bullying. Some ways I've been thinking about removing this perception of bullying or criticism in code reviews is by getting AI/Linters to auto review code, and for the parts humans need to comment on then it's all about "bedside manner" = Logos + Pathos + Ethos to be convincing which is a skill in-of-it-self. Any tips on a giving awesome code reviews are welcome
Reminds me of Kim Dotcom - who discovered the NZ govt had tapped his internet because his ping increased while playing Call of Duty.
how? tapping doesn't increase latency
@@alexisfrjp Good point. I believe the way they implemented it was by rerouting his traffic at the exchange, so it was ISP level access, not a direct wire tap. Regardless, Dotcom is on record saying this is how he discovered it.
Reflections on Trusting Trust is just relevant now as it ever was
Great paper, well worth the read.
Well, always. The is a good reason why last semester I presented it as a research paper during a course. (Not plagiarism)
I bet it's a team behind that persona. Makes me wonder what else has successfully flown under the radar.
I wonder if Microsoft has anything to do with this. They acquired github in 2018. Need I remind you that the CIA had Bill Gates by the balls back in the day with their anti-trust-anti-monopoly lawsuit. It's hard to say just how deep the tentacles go.
The pilot episode of Lone Gunman speculated that there is a backdoor inside every modern cpu. The layers of complexity goes on and on.
thanks for such an amazing update about all this stuff
From what I've seen some rolling release distros were also affected or at least downgraded packages as a precaution.
They had the version, but either the direct source download and in any case the backdoor did not trigger to build as not matching debian or redhat
Code is still accessible from the tukaani git
We hat once an (performance) issue with Chrome in between a major version. It's not possible to review or get the idea why this issue happened. It went "away" with a more recent version...
Thank you very much for educating me on that matter.
Im pretty sure the last minute rush was because a change was proposed to systemd that would make this backdoor unable to happen
This is pure example of why you must write program that just does stuff without any tricks and with minimal dependencies. Everything is horrible: openssh linked with libsystemd, libsystemd that by the look of it the only feature that's not in it is playing mp4 and emulating x11 server, and build system in xz project that so sophisticated nobody even bothered to look at what the hell there is any other scripts except Makefile or build.sh.
Exactly, 100% agree.
What do you do exactly? Like develop your own linux distro?
Yep T2sde.org exactcode.com
Reproducible builds and better threat intelligence for OSS
gut das man dich hat!! wenn ich mir das erklären lassen will, dann von dir! normalerweise sind mir deine videos zu komplex und damit irgendwie auch ein wenig anstrengend, aber das ist mal ein thema was wirklich wichtig ist.
I use a monitor on my NAS cause I'm new and afraid to get pwned due to misconfiguration lol
This can't be detected by any sca tools due to so much obfuscation..i wonder what can prevent this you know other than patching.
Been on Slackware for 25 years now. No RedHat style S***D here, so no backdoor from zx at least. Our BDFL seems to have lost confidence in the zx git repo enough to not even wanting to go back to an older version for 'current' branch.
"Everybody" jumped on the git train which made it a huge vector for attacks.
I suspect we're going to see more of this.
I'm quite naive in this topic, but should the linker be throwing possible injection warnings when objects are being provided in less usual ways? (where you can configure for what is "usual" for your project)
That's not really practical, and usually not the source of backdoors anyway.
Does it create an authentication key that requires another key that only the creator of this back door could have? What if the actor was just trying to bring to light how easy it is to exploit via maintainers being overloaded. Or what if the maintainers are compromised? Do you have an alternative system for code review to prevent this?
One can always pretend to just wanting to test code review. The most lame excuse for any state actor. What do you mean with maintainers compromised? like blackmailed? family imprisoned like I speculated? What do you suggest in that case? I don't have better system fore code review except code and systems need to be slimier, more minimal and less complex. Case in point, GNU Autoconf is probably way to hard to read to fully comprehend all the possible exploitation points. But then again, even more sophisticated backdoors might be hidden in pure code, even Rust anyway. I would summarize code is simply too little reviewed.
Any patch against this. I saw i already had xz-utils or to say xz 1.0.1 on my wsl ubuntu 22.04.4 LTS. I uninstalled that but still worried whether I am compromised or not. i believe I am though any tips? or should I do a fresh install of my wsl
Ubuntu 22.04.4 LTS is too old to have this version of the package and mostly rolling releases are affected. Ubuntu package listing on their site shows they have xz 5.2.5-2 which is way before this backdoor. Also as far as we know this backdoor allows ssh login bypass so as long as your system isn't exposed to the internet through port forwarding you should be fine. Although this backdoor possibly might have more stuff compromized than just openssh but that is not clear yet since all of this is still being investigated.
@@VitisCZ I don't know mine xz-utils was 1.0.1 something not even close to this. But yeah this was something that could have been major.
1.0.1 does not sound particularly recent
You can still git clone the repo and review the commits etc.
Where? On archive.org?
You say "thanks Github for taking this down" at least four times. This causes unnecessary parsing overhead inefficiencies in a complex topic.
"we can just pay Canada to sit on logs"
thanks - great upload.
so which versions are safe before this Jans Hans Tans?
1) use web archive 2) github does not allow malware 3) they probably don't want spam of comments
1) is slow and cumbersome and not eveything is archived and all web links don't just instantly work 2) it's not malware but a popular open source project that someone just introduced a bug that need to be urgently fixed and rolled out technically not different from other cve in Linux, openssl or Firefox 3) some comment load should be their least of their concern especially when vital technical information where discussed there.
Honestly I doubt that was a state actor. With the resource of a state, it’s unlikely that the person/team would introduce a backdoor that causes obvious cpu usage spike. I mean, there are backdoors found in Linux that even gone 5-10 years unnoticed.
Who else would put in a backdoor just for fun ?
Broooo, they are clearly targeting ssh which is likely linked to uncovering dark web markets and much more... They all of a sudden discover a way in and get access to servers which are not apparently vulnerable to anything ...
@@MoreReneRebe Mr robot
@@user-in2cs1vp6ofor fun or for profit?
For me it has all the hallmarks of less technical people managing technical work. Who else has such resources available for over 3 years.
Great work!
Build systems, huge amount of dependencies, unnecessary complex code and abstraction hell makes this way too easy to do. Systems programmers always joke about the JS/Web ecosystem, but it seems this issue is a lot more fundamental to the current state of software development.
Indeed very interesting - when it comes to the question of who stands behind all this, I am not convinced that it is a state actor. Although this attack was executed on a very high level in a very smart way, the entire execution was still not 'smooth' enough.. One would assume that at a state level, there would have been multiple preliminary tests conducted to eliminate any anomalies that could lead to detection. Furthermore, it's worth considering that in such scenarios, people often gravitate towards more sensational stories. The allure of attributing sophisticated cyber attacks to state actors or large organizations is strong.. -> yet it can sometimes overshadow the reality that highly skilled individuals or smaller groups ..or maybe just one very lonely someone can also orchestrate complex operations....just my two cents..
some government state actor are good, and some are not. Bad luck etc.
I am just wondering if i am wrong please correct me!
Is this only in Debian based systems or in the Linux kernel itself? I mean if it's in GNU/Linux and all of the kernel Android phones also run Linux kernel! So does that mean our phones are vulnerable too? If that happens that will be scary on the computer you can fix the issue. But in your phone you can't do anything about it unless your OEM sends a security patch update!
the backdoor specifically targeted Debian and RPM based x86-64 distributions like RedHat and SuSE. Due to the x86-64 binary injected nature this specific execution of the attack could only work on such Intel, AMD (& clones, like VIA) based x86 systems.
I feel like a lot of EDRs would catch the process hook pretty immediately
Not necessarily
YOUR likely suspects list seems to line up really well with the list of official enemies. I got my eye on you.
while I intentionally named and called out about everyone, from the USA over EU to Russia and China, what is your supposedly "list of official enemies"?
russia, china.... could also be the US, israel, north korea
yes, I literally said this. Thanks for quoting me, … ;-)
thank you so much
0:54 what makes you think its not the US? Plenty of asians living there and nsa has a great track record in not exploiting everything in site,right?
Your comment is straight up discriminatory and you are not even conscious of that.
I literally said at that time in the video "could be …, USA, or stuff who knows. Right?" or any of the other 150+ states of the planet. The us would more likely backdoor redhay directly with fisa court order. Probably has.
Apologies i was listening to the video at 2x speed. Closed captions confirm you are correct.
If you say "Jia Tan" fast its like jitan, is that a play on titan?
I could not care less about the name. As I tried to point out in the video the name is likely fake. Could be Foo Bar or Jane Doe. We need to stop paying attention to any random name. Even Linus Torvalds or Satoshi Nakamoto can be faked. The only thing that should always count is the actually Code. Review.
Good work
Looks like an honest mistake lol.
😂
We need a IA Security inspector of repos for Open Source code.
IA? ai will help little for well hidden and previously unseen undercover changes.
Looking at the time of activity, it is either Russian or Israeli
I said you can make any time or name up!
A great example of why binaries should not be committed to the source tree
So, how do you unit test then ?
I would try to construct within the test code valid inputs to the functions I'd want to test. If you'd need a compressed file to test, you could probably generate it as part of the test setup stage (and if a corrupted file is required, you could always generate a valid file with the compressor logic and corrupt it explicitly by changing some bytes on the file after it was generated). This way the steps would be well documented and you'd have no opaque blobs committed to the source tree.
That said, I'll be the first to admit that, for convenience, I've committed binary blobs to tests in projects I've worked on 😉 I do try to avoid doing that tho. And the actors that out together this backdoor were brilliant in hiding and obfuscating it, so I'm not surprised it did not raise any red flags. I guess my point is, no binary blobs == less surface for this sort of malware to hide
@@MoreReneRebe Maybe generate them. If I remember right the LLVM project uses declarative scripts to generate object files for testing its lld linker, rather than submitting bogus binaries into the repo. This also has the advantage of providing one blob per test and letting you follow how the changes rather than having you trust changes in a binary blob. Although I realize that XZ is a much smaller project than LLVM and probably don't have the means to make custom solutions for testing its XZ file format.
What if this is a false flag attack that is supposed to distract from the actual backdoor?
unlikely, if anything it has served as a reminder how carefully everything needs the reviewed, ...
You cannot review everything even if it simple. Just accept that your system is not uncompromisable, but hard to compromise.
As complexity grows there will be less and less uncompromisable systems. And more and more backdoors.
P.S. This can be, but does not require a team. I alone could come up with such a backdoor. I could even obfuscate it better without breaking sanitizers and without creating valgrind errors. The way those commits went through is clever. A script hidden inside some test files. If it were in the source it would have been found earlier.
P.S.2 Github including externals in generated zips would be in some cases impossible and in some unwanted. In some of my repos this would be unwanted because I would have duplicate files in the downloads, since I copy the externals to specific places on build.
I would agree every change needs to be reviewed. At least twice.
Thanks github that didn't share the ip address of that actor :)
Of the VPN datacenter ?
@@MoreReneRebe Everyone make a mistake, maybe that guy once connected the github without VPN. For investigation each piece of puzzle is valuable.
"unpaid hobby project" is the most important part. Start paying and you'll get better quality. I've always seen this open-source thing as slavery. Working sometimes long hours, for free, and when you slower the pace, people complain. "what? you don't want to work for free anymore? how come?!" just so stupid.
Wayback
Still convenient if it even has all.
heheheheh
?
Everyone is affected. The state sponsored rogue developer contributed to a bunch of projects throughout the last two years. All his commits need to be combed through, and likely will result in more CVEs being discovered.
Yet another linux / open source myth is dying :D
Which? If all it confirmed that unlike in closed source projects security vulnerbabilieh and back doors can be hiding and are found and fixed quickly?
@@MoreReneRebe I am mostly referring to the common urban myths that have existed for quite some time now
"You can't get a virus on linux"
"open source is safe because everyone can have a look on the source code"
It was pure luck that this backdoor was caught so early.
@@Th1200 Open Source is orders of magnitudes safer than the closed low-quality Redneck engineering that is Microsoft Windows and Apple's macOS. Have you seen their code, bugs and reoccurring security advisories? The backdoor was caught because so many eyes are on OpenSource. There are litarly nearly no external eyes on closed sources, and yet so many more vulnerabilities are discovered there by pen testing thru fuzzing and reverse engineering. God knows how many security agencies have their fingers in Apple's and Microsoft's code. Just look at the Snowden revelations from a decade ago, ...!
@@MoreReneRebe I'm 100% on your side, however Linux had this "invincible" status in the past which it has never deserved.
Seing the trend I believe that the clusterfuck which is linux will get worse.
Why does the internet and multi billion $$$ companies have to rely on a burnt out Developer and his "hobby project"?
If its a state actor whos trying to hide why use a Chinese name?
If you are Russian ? North Korean ? USA ?
@@MoreReneRebe yes, i think blame is trying to be placed towards China but its another country
I think it's a lot easier to take a "western name" and link that back to some real world info that either proves a real person is behind the name or that there is no trace of a person with that name that is involved in that area. Western developers go to conferences, have a linkedin etc etc. So you use a name that looks like a Chinese university student or someone working at a Chinese company and not being able to find a paper trail doesn't seem so bad as those people even if legit have very little paper trail that anyone outside of China would have access to.
I think any state actor that thought about this enough to get embedded in a community and work up to getting enough rights to add backdoors would have noticed the presence of untraceable Chinese committers to OSS projects in the last few years and seen that as a good identity to emulate.
There is a simple explanation:
Your non-native- english speaking state sponsored actor, will with a "clean" western sounding name make certain misstakes that damage his background story, and just having a sane explanation for the misstakes is a technique of camouflage in the open.
Also, misstakes are not the only giveaway, certain choice of words is favoured by native english speakers (US vs. UK) and also within non-native speakers certain groups will use english words somewhat different and very repetitive. (more than one word for a thing) / STTNG-Ref: remember what Lore said to Data to stress his linguistic superiority.
I'm very sure that the big three letter agencies are actively scanning all public information and might also search for postings by "Jake Taylor, Senior Software Developer at Malevolent LLC" this will hurt much: "who CANNOT even talk basic english in short and pregnant sentences" - there is a backdoored joke in that sentence.
Where is the POC? Is this exploitable somehow by anyone? Or only a group has access to this exploit, meaning, not yet known and/or not accessible to exploit by regular sec researchers like other vulnerabilities? Just a noob here. Sorry if it was a stupid question, but if I did not ask, I would still be ignorant, thus at least trying to understand
This isn't a traditional exploit in that sense. this is a backdoor that overrides the RSA key validation to approve likely a few specific public keys. Meaning a security researcher would need the hackers keys.
x.com/amlweems/status/1774819428208689241
State actor, China, Russia but not US, Germany or one 🕜 of the five eyes right? What a Hypocrite.
Not true: I specifically named them all, including the USA, EU. And literally said it could be any of the 150+ states and even said the USA would more likely go bug RedHat directly. Did you even watch the video?
@@MoreReneRebe yes I watch the videos, sorry I didn't hear you saying USA in the beginning.