Update: The xz downgrade PR is now in nixos-unstable: nixpk.gs/pr-tracker.html?pr=300028 You should be able to update a nixos-unstable system now to be fully-free of xz-5.6.x!
Hi thanks for the update, can you please explain how this was possible if you said in the video it would take weeks for such a thing to happen? I didnt understand the point you were trying to make about "build servers being busy for weeks"
Thanks for this! I was reading many conflicting things when I had time, around kids and allergies, so I did not get to the part where it is ssh dependent. I think that will be my temp fix for this, SSH disabled!🤞
SSH is not a cause, it is a victim. You need to downgrade xz package, right now, preferably trough another system directly disk, and after that force package manager to know that package is downgraded.
Would it be possible for Nix to start pulling code from release tags instead of from release tarballs? It would mean that no code would end up being built which wasn't present in the code, compared to the tarballs potentially containing code injected during the test/release pipeline...?
Most nix packages are built from source,and you can use not just release tags, but any commit hash as well. The few that aren't compiled from source on your machine are either cached by the NixOS cache (majority of them for mainline architectures ), non-FOSS, or whoever wrote the derivation did it the quick and dirty way of grabbing the tarball.
@@XxZeldaxXXxLinkxXIMO Nix really should avoid ever using tarballs. It's just a completely unnecessary security risk, especially for a distro all about repeatability and crap.
@@felixjohnson3874 Almost every single package doesn't use tarballs already. But the ability to use tarballs / binary distributions is a necessary escape hatch because not every software someone might use is open source (e.g. Nvidia, VirtualBox, etc.) The case where a package uses uses a binary distribution under the hood is rare enough, thankfully, and is generally limited to niche packages. In addition, writing your own derivation to build it from source is very accessible, albeit difficult if you don't know Nix or the build process for the application
@@XxZeldaxXXxLinkxX There might be a benifit of going the route Guix did and have a pure source being the default and needing to be more explicit for using non source builds
@@seppeljordan Did you consider that people should try watching a video before asking questions that are covered in the video they're literally in the comment section for? Don't make me out to be the villain for telling someone to watch the video they're commenting on. FFS go outside.
@@HirschyKissto specify, i know. what i meant was use unstable (as i need it) but then grab the xz package from the stable branch using flakes. i was just unsure if it would be better to use the xz package from the very latest release which was downgraded to a version still a bit higher than the one on stable. i set it up the former way but that wasn't the recommendation of phoenix yk
@@HirschyKiss No I did not consider that. Did I hurt your poor ego and now you feel like a villain just because I remarked on the rudeness of your comment? I would like you to tell me more about it.
the pre-vuln version is now back in unstable. as per that issue. Side note, if you don't have port forwarding set up, it's not an issue to user machines anyway. Unless you have a server than can jump to your user subnet, I guess.
Why bother? The backdoor was never compiled in on NixOS even if you have the affected version. The backdoor was only compiled in on systems with deb and rpm packages.
@@rogo7330 wrong. The code is in the tarball, but it only compiles and will affect you if you are building rpm or deb packages. Please read up more about the backdoor! On NixOS you aren't affected even if you have the 5.6.0 or 5.6.1 installed. Because the xz nix package doesn't build a deb or rpm package. Because the binary path wouldn't match the check of the backdoor. It is obvious that the authors of this backdoor narrowed their target to avoid being detected. They failed.
@interru_io You’re right, but you’re wrong. And both of those points are covered in the first 3 minutes of the video. You’re right that non deb or rpm systems do not seem to be the primary targets, but that doesn’t inherently exclude them from being targets. Just to let you know, your argument of “why bother?” comes off as snarky and/or haughty, which could be sending people the wrong message, but where you’re wrong is what answers that “why bother?” question. It’s basic security practice, that in the event of an exploit, to assume your system is compromised before declaring yourself all clear. Because the exploit code is still being dissected, we don’t fully know yet if other systems are compromised in the same way, compromised in a different way, or not compromised. So taking precautions is a reasonable response. If you don’t wish to take precautions, that is your prerogative, but (alluding back to my statement about sending the wrong message) belittling others for taking or recommending reasonable precautions until we have all the information is unnecessary, if not potentially harmful.
@@ThatGuy-ru8ms > You’re right that non deb or rpm systems do not seem to be the primary targets, but that doesn’t inherently exclude them from being targets. The backdoored m4 file checks if the "RPM_ARCH" env var is set to "x86_64" during the build, and only inserts the malicious code if that is the case. Since nixpkgs doesn't build the package as a deb/rpm, this environment variable will not be set and so the backdoor is not included. Some users on the NixOS discourse forums analysed the current xz derivation and confirmed that it doesn't have the known indicators of this CVE (such as the _get_cpuid symbol). Additionally, even if the nixpkgs build of xz did include the malicious code, it still wouldn't work, because sshd in nixpkgs doesn't have the systemd notification patches that result in xz being pulled as a transitive dependency. To put it simply, no version of nixpkgs is affected by CVE-2024-3094. You don't need to downgrade/upgrade/patch anything in order to mitigate this vulnerability. Now, hypothetically, there might still be some other, currently unknown, completely unrelated vulnerabilities, but since we are not aware of those vulnerabilities, mitigating them is borderline impossible. Some people have suggested that you should use a version of xz that was released before Jian Tan joined the project, however those versions are over 2.5 years old and therefore might not be API compatible with all modern projects and might even have old security vulnerabilities that were patched in later versions.
@@ThatGuy-ru8ms It's already proven that the backdoor isn't in the compiled binary cache in the nix package. So regarding the backdoor, you don't gain anything by reverting to previous versions in NixOS. It would make sense if you assume that there are additional backdoors. But then it would be advisable to revert a couple of version earlier. And if you assume your system is compromised, it isn't advisable to just revert to a previous version. If you are compromised, you have to understand what went wrong and also what was affected. And then you rebuild your system so that it doesn't happen again. If I found a RCE backdoor that's affecting me, I would nuke __everything__. Reverting a version and calling it a day wouldn't be enough for me personally. But we currently have enough evidence that we can say: It doesn't affect NixOS users, as long as they didn't use a compromised xz deb or rpm package through various means. So from my perspective, it is contra - productive to put in extra effort beyond keeping your system up to date to mitigate something that isn't even there. I don't belittle anyone in any way. That's a misrepresentation on your part.
replaceRuntimeDependencies does replace the runtime *dependencies* meaning that “all other packages built against xz” will also magically (well, by the magic of being impure) use the replacement version. In the issue someone did a demonstration using a Rust program without rebuilding the program
Yep, I'm mainly just using it as a bootloader for docker-compose, since its easy to declaratively manage things like sshd and the host firewall using nixos
Nix is amazing for virtualization. I have it running both in VMs and Linux containers in proximox. I've switched to Nix as my main OS, once you get your configuration down you can easily deploy multiple VMs and have them configured just by pulling from GitHub. If you miss anything up it's also really easy to roll back and you have the ability to temporarily set stuff up with nix-shell if you only need to run something once.
how i can see if this is installed as dependency of anything in nixos?? i constantly use devshells, so a lot of packages are not in the path but are cache in system, is there anyway to search through all of it...???
@@lobotomy-victimI feel like you may have misread the question. Pretty sure OP was asking to find all the versions of xz on their system (as one package may require an older/newer version than another), not the version of xz in their path by default.
the stdenv depends on xz, so if your derivations are made using nixpkgs unstable they will depend on the trojaned version. if you have no store paths made with nixpkgs before 5.6.0 existed, then "ls /nix/store/" should get you all the packages you want
Unless you did some manual overrides of xz or installed xz from somewhere other than nixpkgs, then your system is not affected by CVE-2024-3094. This is because the "backdoored" m4 build script checks for environment variables that aren't set during the nixpkgs build. So even if you have xz version 5.6.x somewhere in your nix store, those binaries are still almost certainly safe (or at least safe from CVE-2024-3094 specifically). There might be some other (yet unknown) vulnerabilities that were added to xz by Jia Tan over the last 2.5 years, however we don't currently know of any such vulnerabilities, and so we don't really know how to mitigate them.
Update: The xz downgrade PR is now in nixos-unstable: nixpk.gs/pr-tracker.html?pr=300028
You should be able to update a nixos-unstable system now to be fully-free of xz-5.6.x!
Hi thanks for the update, can you please explain how this was possible if you said in the video it would take weeks for such a thing to happen? I didnt understand the point you were trying to make about "build servers being busy for weeks"
Thanks for this! I was reading many conflicting things when I had time, around kids and allergies, so I did not get to the part where it is ssh dependent. I think that will be my temp fix for this, SSH disabled!🤞
SSH is not a cause, it is a victim. You need to downgrade xz package, right now, preferably trough another system directly disk, and after that force package manager to know that package is downgraded.
*directly on the disk
Did the same, disabled ssh for now
@@rogo7330THIS!
Disabling might not be enough. The code is still being looked into right now.
Would it be possible for Nix to start pulling code from release tags instead of from release tarballs? It would mean that no code would end up being built which wasn't present in the code, compared to the tarballs potentially containing code injected during the test/release pipeline...?
Most nix packages are built from source,and you can use not just release tags, but any commit hash as well.
The few that aren't compiled from source on your machine are either cached by the NixOS cache (majority of them for mainline architectures ), non-FOSS, or whoever wrote the derivation did it the quick and dirty way of grabbing the tarball.
@@XxZeldaxXXxLinkxXIMO Nix really should avoid ever using tarballs. It's just a completely unnecessary security risk, especially for a distro all about repeatability and crap.
@@felixjohnson3874 Almost every single package doesn't use tarballs already. But the ability to use tarballs / binary distributions is a necessary escape hatch because not every software someone might use is open source (e.g. Nvidia, VirtualBox, etc.) The case where a package uses uses a binary distribution under the hood is rare enough, thankfully, and is generally limited to niche packages.
In addition, writing your own derivation to build it from source is very accessible, albeit difficult if you don't know Nix or the build process for the application
@@XxZeldaxXXxLinkxX There might be a benifit of going the route Guix did and have a pure source being the default and needing to be more explicit for using non source builds
Glowie commit
Good info, good explanation
Is it a good idea to use the stable version (5.4.4) on an unstable system?
Did you watch the video?
The answer is use 23.11 if it's feasible for your system. Obviously
@@HirschyKissAhhh. The nix community. Why don't you just add "RTFM" to your reply for good measure. It would make your point event better. Obviously.
@@seppeljordan Did you consider that people should try watching a video before asking questions that are covered in the video they're literally in the comment section for?
Don't make me out to be the villain for telling someone to watch the video they're commenting on. FFS go outside.
@@HirschyKissto specify, i know. what i meant was use unstable (as i need it) but then grab the xz package from the stable branch using flakes. i was just unsure if it would be better to use the xz package from the very latest release which was downgraded to a version still a bit higher than the one on stable. i set it up the former way but that wasn't the recommendation of phoenix yk
@@HirschyKiss No I did not consider that. Did I hurt your poor ego and now you feel like a villain just because I remarked on the rudeness of your comment? I would like you to tell me more about it.
the pre-vuln version is now back in unstable. as per that issue.
Side note, if you don't have port forwarding set up, it's not an issue to user machines anyway. Unless you have a server than can jump to your user subnet, I guess.
"switch to stable" i'm running on the master branch lmaoo
i'm relying on my firewall disallowing access from outside my tailnet and also my obscurity / no target value at this point.
Why bother? The backdoor was never compiled in on NixOS even if you have the affected version. The backdoor was only compiled in on systems with deb and rpm packages.
Malicious build code is in release source tarball. Everyone, who did not edited source tarball build, are affected.
@@rogo7330 wrong. The code is in the tarball, but it only compiles and will affect you if you are building rpm or deb packages. Please read up more about the backdoor!
On NixOS you aren't affected even if you have the 5.6.0 or 5.6.1 installed.
Because the xz nix package doesn't build a deb or rpm package.
Because the binary path wouldn't match the check of the backdoor.
It is obvious that the authors of this backdoor narrowed their target to avoid being detected. They failed.
@interru_io You’re right, but you’re wrong. And both of those points are covered in the first 3 minutes of the video.
You’re right that non deb or rpm systems do not seem to be the primary targets, but that doesn’t inherently exclude them from being targets.
Just to let you know, your argument of “why bother?” comes off as snarky and/or haughty, which could be sending people the wrong message, but where you’re wrong is what answers that “why bother?” question.
It’s basic security practice, that in the event of an exploit, to assume your system is compromised before declaring yourself all clear.
Because the exploit code is still being dissected, we don’t fully know yet if other systems are compromised in the same way, compromised in a different way, or not compromised. So taking precautions is a reasonable response.
If you don’t wish to take precautions, that is your prerogative, but (alluding back to my statement about sending the wrong message) belittling others for taking or recommending reasonable precautions until we have all the information is unnecessary, if not potentially harmful.
@@ThatGuy-ru8ms
> You’re right that non deb or rpm systems do not seem to be the primary targets, but that doesn’t inherently exclude them from being targets.
The backdoored m4 file checks if the "RPM_ARCH" env var is set to "x86_64" during the build, and only inserts the malicious code if that is the case. Since nixpkgs doesn't build the package as a deb/rpm, this environment variable will not be set and so the backdoor is not included. Some users on the NixOS discourse forums analysed the current xz derivation and confirmed that it doesn't have the known indicators of this CVE (such as the _get_cpuid symbol).
Additionally, even if the nixpkgs build of xz did include the malicious code, it still wouldn't work, because sshd in nixpkgs doesn't have the systemd notification patches that result in xz being pulled as a transitive dependency.
To put it simply, no version of nixpkgs is affected by CVE-2024-3094. You don't need to downgrade/upgrade/patch anything in order to mitigate this vulnerability.
Now, hypothetically, there might still be some other, currently unknown, completely unrelated vulnerabilities, but since we are not aware of those vulnerabilities, mitigating them is borderline impossible. Some people have suggested that you should use a version of xz that was released before Jian Tan joined the project, however those versions are over 2.5 years old and therefore might not be API compatible with all modern projects and might even have old security vulnerabilities that were patched in later versions.
@@ThatGuy-ru8ms It's already proven that the backdoor isn't in the compiled binary cache in the nix package. So regarding the backdoor, you don't gain anything by reverting to previous versions in NixOS.
It would make sense if you assume that there are additional backdoors. But then it would be advisable to revert a couple of version earlier.
And if you assume your system is compromised, it isn't advisable to just revert to a previous version. If you are compromised, you have to understand what went wrong and also what was affected. And then you rebuild your system so that it doesn't happen again.
If I found a RCE backdoor that's affecting me, I would nuke __everything__. Reverting a version and calling it a day wouldn't be enough for me personally.
But we currently have enough evidence that we can say: It doesn't affect NixOS users, as long as they didn't use a compromised xz deb or rpm package through various means.
So from my perspective, it is contra - productive to put in extra effort beyond keeping your system up to date to mitigate something that isn't even there.
I don't belittle anyone in any way. That's a misrepresentation on your part.
All Linux YT channels and podcasts are dissecting this to death lol 😂.
These are golden chances to make content. These opportunities come only once or twice a year.
replaceRuntimeDependencies does replace the runtime *dependencies* meaning that “all other packages built against xz” will also magically (well, by the magic of being impure) use the replacement version. In the issue someone did a demonstration using a Rust program without rebuilding the program
Nice summary!
You run nix on your homelab? How do you like it im considering trying it. Any vms or just containers
Yep, I'm mainly just using it as a bootloader for docker-compose, since its easy to declaratively manage things like sshd and the host firewall using nixos
Nix is amazing for virtualization. I have it running both in VMs and Linux containers in proximox.
I've switched to Nix as my main OS, once you get your configuration down you can easily deploy multiple VMs and have them configured just by pulling from GitHub. If you miss anything up it's also really easy to roll back and you have the ability to temporarily set stuff up with nix-shell if you only need to run something once.
how i can see if this is installed as dependency of anything in nixos??
i constantly use devshells, so a lot of packages are not in the path but are cache in system, is there anyway to search through all of it...???
xz -version, if you’re on unstable branch you have the 5.6.1 but it doesn’t have the exploit in it
@@lobotomy-victimI feel like you may have misread the question. Pretty sure OP was asking to find all the versions of xz on their system (as one package may require an older/newer version than another), not the version of xz in their path by default.
@@james-cucumber ah well, the back door is not on nixos anyways
the stdenv depends on xz, so if your derivations are made using nixpkgs unstable they will depend on the trojaned version. if you have no store paths made with nixpkgs before 5.6.0 existed, then "ls /nix/store/" should get you all the packages you want
Unless you did some manual overrides of xz or installed xz from somewhere other than nixpkgs, then your system is not affected by CVE-2024-3094. This is because the "backdoored" m4 build script checks for environment variables that aren't set during the nixpkgs build. So even if you have xz version 5.6.x somewhere in your nix store, those binaries are still almost certainly safe (or at least safe from CVE-2024-3094 specifically).
There might be some other (yet unknown) vulnerabilities that were added to xz by Jia Tan over the last 2.5 years, however we don't currently know of any such vulnerabilities, and so we don't really know how to mitigate them.