Yes first rule of a totally borked bootup is "Don't Panic!". This morning I had both my primary data and backup data drives for my home server both lose their partitions - the TestDisk utility did an amazingly easy job of recreating the partitions (after a cup of coffee).
hey Derek, something you forgot to mention is that this solution works only for legacy bios setup, if you have uefi you must also mount the EFI partition. And not to mention, inside the usb drive you don't need sudo privileges
@@michaelnugent8181 exactly, that's pretty much what it's like whenever a Windows user throws or gives away an older machine because it doesn't run the latest version of Windows; more succulent machines for me
On my EndeavourOS live CD, I did need either sudo (with no password) to do root operations. But you're right on mounting the EFI partition for those systems.
Definitely good to know which is your main system partition! Mount main partition to /mnt. Then look at your fstab "/mnt/etc/fstab" to see which drive/partition is for EFI. Also worth mentioning to mount other stuff if you have separated other directories (*check fstab*): _Very few distros does this to this extent_ - "/home" - "/var" - "/lib" - "/boot" | "/boot/efi" All other mounts should be in your fstab. You shouldn't need to mount your game's partition for instance, but anything that is for your root folder "/" or boot folder "/boot/*".
This is a great way to rescue a Linux install. Also I heard that you have multiple USB drives with ISO's on them. Something you can try to consolidate them is Ventoy, it's a bootloader for running multiple ISO's from a USB disk. For example I have a Ubuntu (server and desktop), fedora, rocky, Debian, arch, Garuda, endeavour OS and windows on the same USB drive
I see where this can be useful, but this results in a SPOF (Sngle Point Of Failure) where if the USB gets corrupted or dies for any reason, then you become in a bad position. Having an extra thumb is always good. 👍🏻👍🏻👍🏻
@@Gurj101 in the more than a year I've used it I've came across 1, yes 1 iso that didn't play nice with it. That was proxmox VE 7.4... everything else worked...
This is why I always recommend keeping a current latest kernel and an LTS kernel on Arch. If something happens with a latest kernel update, either something like what happened here or compatibility issues with some changes on the new kernel, you can simply boot the LTS kernel and reinstall or roll back the latest without having to boot a live environment.
@@Clobercow1 Sure. Any method of rolling back or booting the system before an issue happens without having to have recovery media is preferable to having to write a USB drive and boot a live environment to recover.
Thank God you're talking about this. I had this problem where Pacman crashed in the middle of installation and it deleted my vmlinuz image. Fortunately I knew how to counter this and had a spare bootable usb with me.
You can upgrade your system in virtual console in order to avoid system failures because there is a slew of reasons for the graphical environment to crash especially during a major update/upgrade. Hope it helps.
Since I installed openSUSE Tumbleween in one of my machines I've tasted the divinity of the BTRFS file system paired with Snapper in order to have snapshots created for me and available in a GRUB menu so I don't have to rely on USB pendrives that can be misplaced or easily corrupted. I think that heavenly match can be installed in any distro. Arch would benefit tremendously from this.
If you cannot load after installing new kernel, just reboot and press Esc many times, while PC is loading, and then, in the grub menu, you can select previous kernel.
interesting tip: - create new partition - just copy live cd iso file content inside this partition - if something brokes choose to boot from this partition. wuala you always have live cd with you
This is one thing that's great about Linux. You can fix problems. Can't tell you how many times I had Windows break back when I still used it and have the only solution be to reinstall from scratch (reset wouldn't even work!). On Linux, usually just gotta chroot or look for a solution online. Someone has probably solved your problem before. No need to reinstall from scratch like on Windumb
I’m using a custom GRML for this - with ZFS support and the keyboard layout set to Dvorak. You can work around a lot of these boot issues from the boot loader command prompt and at least get your system to boot, but it is a lot more tedious than the process you described. And the boot loader command prompt also typically only supports QWERTY.
Just another little niche tip i had to figure out the hard way: When you need to fix the Linux install on an old or exotic peace of hardware, some BIOSs or UEFIs only have drivers for USB2.0 - meaning it won't let you boot off of a newer drive. I have a micro-sd to usb adapter which uses usb2.0 which helped me out ( just a little fyi sidenote here ^^ )
Thanks for the great video. I am in a similar problem with manjaro. I had the kernel not install properly on an update a few months ago. I was able to rescue it with a manjaro usb stick and using my timeshift backup, but I've been afraid to update the system since. I am rather behind in the updates to my system.
I'm confused. After the borked update and failed reboot, did you not have a GRUB screen where you can toggle to a different or earlier kernel? (i.e: Do a forced shut-down, power-up and quickly enter the GRUB menu before it tries to load the system. That wasn't an option?)
That would require you keep the old kernel, and some people don't do that. Personally, I only bother to update the kernel for security updates and I'll definitely keep the most recent previous kernel, but I also have space, something that yet again not everyone has.
@@anon_y_mousse thing is DT always mentions how he has both the linux and linux lts kernels installed, he even broyght it up in this video. So by all accounts the kernel that wasn't being updated should still've been fully functional
@Anony Mousse 1. Deleting old kernels is not really the best method of saving space on your drive for now obvious reasons. 2. The Arch package system should remove old kernels as the LAST step, again for now obvious reasons. If not I would suggest a bug report . 3. I am not sure of the exact problem here, but generally the easiest fix is boot-repair. It was made in the Ubuntu ecosystem but according to the Arch AUR fans should be available on Arch. But then the Arch kernels are now configured to not work with most distros so again another example of why I no longer use arch as although I can do all this chroot configuration it is not really how I wish to spend my time.
If you don't have access to the arch-chroot command I'd recommend pulling up the gentoo handbook, under the chapter for installing the base system they have a section going over what needs to be mounted into the new root to get a functional system. That way it can be used normally to fix whatever issue you may have :)
I had a similar issue once. But i solved it a bit different. I started a liveUSB, chrooted and just installed the previous Kernel before the update, which was archived on my machine. Pacman -S was not an option since the new kernel was not compatible with my network drivers of my Wireless devices, which i rely on since the router is on the other side of the building. I had to do it once with my arch system and later with my Debian. Though the Debian case was a bit more difficult to fix. I borked my machine by accidentally overwriting my GLIBC with a newly, freshly compiled GLIBC that i needed to recompile software i developed for my arch system. (I forgot that the folder in which i compiled the GLIBC to was already used by my system native GLIBC) At that point i didn't reinstall the old GLIBC but just used a snapshot iso i had from that system since that can be less trouble as fiddling with that library. But for most Kernel issues, if a Arch update trashes your system with a new kernel and you have no internet connection, a previously installed Kernel package should do the trick.
Protip: Always update the system inside tmux (or screen) so that if the terminal emulator, desktop enviroment or window managar dies on you the update process won't be killed and can be easily resumed on a new terminal emulator process or even on the tty.
Oh man! So many times I have had to do this throughout my Linux career... But for me, the issue that causes this is often Systemd fighting ZFS and I need to set one or the other straight, quite often it is systemd needing a boot to the ass though. Because if a ZFS on root system breaks during an update, you cannot just rescue it with a different kernel boot, you have to do a few extra steps.
I use Garuda and they make it so easy to chroot. But in the past, I had significant issues when I was on Linux Mint dealing with the grub. I made a thread on the Linux Mint forums and a guy that posted said that's an easy fix. Gave me about 6 or 7 commands, almost like what you did. We went back and forth because I didn't get it since almost on every command I put in, I was getting an error. He said, just go on and finish the commands and reboot. And it worked. I had a similar issue with the bootloader when I was on Linux. Since it failed so many times with the grub, I had several bootloaders (probably about 5 or so), that wouldn't get past the grub part. It would hang up. So, again I had to post on the bootloader error. But this time I was on Manjaro, so I posted on their forums. And that was a BIG help. It was simple as deleting those pesky bootloader entries. Then I restarted and installed Manjaro with no problem.
I've only bricked my arch install once, and it was because of an outdated zfs kernel module. Switched that over to dkms and ive never had another issue.
Great video. I did a couple of full re-installs before I discovered manjaro-chroot or arch-chroot. Chroot on its own can be quite daunting until you know about the process you just described.
how did you discover it late ? if your re-installs included base arch re-install then you would atleast be aware of it. welp can't say it will like click in your mind that you can do that but you will be aware since arch is installed that way. you pacstrap basic packages and chroot and install some more necessary ones later to install arch if i remember correctly.
I recently added a sata m.2 drive to my system. I wasn't 100% sure which drive to install /home and /root on. After seeing DT's partition setup it looks like having /home on the slower drive is perfectly fine. After looking through the manual for my motherboard it states the sata m.2 can also use an nvme 2.0 but I've never seen one of those before.
Even though I understand the why, I think it's really cool you explain why you're doing the things you're doing. I know when I was first learning Linux it was so overwhelming how often people and guides would just tell you to do something, kind of expecting you to just know why.
Don't you need to mount /proc /dev /sys or arch-chroot takes care of it? I remember chrooting for rescuing and couldn't get stuff done until I mounted dev (which makes sense - can't grub install into /dev/hda1 if /dev/hda1 doesn't exist) ETA: I read the docs, yes, arch-chroot takes care of it.
I have never had a kernel update blow out on me, but I have hosed the display manager, trying to tweak a setting. As you have shown before, always back up a config file BEFORE you make changes! I think that I was on Ubuntu 7.10 back then, so it was easy enough to just tell the installer to act as an upgrade, and let it fix the settings. I'm sure that there was a much better way to do it, but I wasn't in a mood to look it up at the time, when I knew running an upgrade would only take a few minutes.
In other words, the update didn't kill your pc, its your wm crash that killed your updating at the "best" timing. Had a similar exp a few years back and circling the web was not fun. Thank you for putting out your solution dt. Mine was a bit different, i had to go and change boot parameter as well
I had a similar issue recently. Updated Thursday then didn't boot my system until Friday. Root partition couldn't be found for some reason. System wouldn't boot. Fix was booting to the live usb and running `mkinitcpio -p linux`. Rebooted and everything worked like a charm. Not sure what happened, if the initramfs was corrupted, or never ran after the kernel upgrade.
I actually had this same issue on endeavour OS the other day. The power went out in the middle of a kernel update. I did this method and I couldn't get through. I had ZSH as my default shell when the kernel was messed up and this method kept complaining about bash directory or whatever wasn't there. I reinstalled, I wasn't in the mood to search for a couple of hours. Thankfully, I separate my / and home partitions.
Dang it! I got this in my feed a week ago but didn't watch it then! Just today after updating (successfully, I thought) my efi systemd entries completely disappeared along with my latest and lts kernel images, barring the intel-ucode. I kept a hard drive with EndeavourOS on it just in case, and googled for a while until learning about chroot (the first time I found the arch wiki useful!)... then as I was so close, followed some random internet user's advice to install a conflicting package and uninstalled a system package... then promptly forgot which package I uninstalled. That package is what the kernel used to rebuild itself, so reinstalling didn't help. Eventually I realized that their solution was needlessly complex - I could have just reinstalled the kernel - and facepalmed. I fiddled for a bit with systemd to try to manually build & edit the images & entries but eventually decided to just reinstall the system (I keep an installer script and could access the /home folder through the live usb, so I didn't have to go back to a backup). Most important thing: I learned from my failure. Next time this happens, I won't make the same mistake, but rather just reinstall the kernel through chroot!
Had similar issues every time the linux-nvidia firmware was updating. I would get stuck in the boot screen with no TTY or a window manager. To fix that issue I booted into an LTS kernel on my system which allowed me to reinstall the nvidia drivers, booted back to the regular kernel and it fixed the issue. That's why I like to keep a regular kernel and an LTS version on my system at all times. Just makes sense for these kinds of situations.
never liked how Arch handles kernel and its updates, somehow gentoo is always the most maintenance free, that lets me focus on doing work on computer and not worry about os
Will this still work if you use a distribution with a different package manager? So for example, can you fix a broken update performed using pacman by installing a new Linux kernel using apt?
Please tell me you fixed this by installing a non-rolling release distribution cause that's the only way you're going to solve the problem of sh*tty distributions constantly breaking. There is a reason decent distributions aren't full rolling. It's one thing where a distribution is partially rolling. There are good reasons for distributions to release updates- like to support newer hardware, web browsers, and even instant messaging clients. But NOTHING should be released without some amount of basic QA and testing done. THIS is what makes Arch and Arch based distributions like Manjaro unusable. Some of us do more than just play our with computers. They're not just toys. They're utilized for actual work.
I just made this exact mistake. I'm so used to forcing a rapid shutdown or restart all the time, I downloaded a kernel update, didn't tell me by the way, looked just like a normal app update. It looked like it was done installing, I did a sudo reboot now, now the new kernel won't boot, because it wasn't finished installing. Now I see why there are several kernels to boot from, it's not a big deal, just kind of a pain to remove the partially installed kernel and try it again, perhaps with a little more patience this time.
For me, a similar situation occurs whenever I update any systemd component. So I exclude all systemd components before an update, and I update the systemd components separately in a tty and run mkinitcpio afterwards
I think this was just a bug, not something you did wrong. It seemed like the kernel didn't work with nvme very briefly last week. Had to do the same thing
and this is why people should drop all distros and use nixos and guix, because you can easily rollback to the previous stable version by a simple restart, no need for a USB. Just the functional design (as in functional programming) of NixOS
Im on gentoo running the latest kernel. almost everyday my cpu goes overtime when compiling it. Of course, I’m on a distribution Kernel, meaning it’s a bit more streamlined when updating.
Just today, my secondary Windows drive that I only use for games that don't work on Linux yet messed with my grub. And so I was in class, pulling out my arch usb, reinstalling grub and creating the config. This is exactly why I keep an arch usb in my backpack at all times.
In my case it was completely different. But this video still helped me in narrowing it down by getting rid of the obvious first steps. I'm leaving this as a future reference to both myself and other people who come across the same problem. TL;DR: If you have other drives mounted onto the main Linux drive, this solution may be for you. Since I'm dual booting Windows and Arch Linux I often tend to share files between them, and thus I always have Windows's main drive (C:\) mounted onto the Linux partition. This caused Arch to not boot after I updated Windows 10 to Windows 11. Turns out the solution wasn't too advanced. If I did "cat /etc/fstab" I could see the Windows drive had one UUID which it expected. But upon running "sudo blkid" I would notice the Windows drive UUID had changed, probably due to Windows 11 changing things. I simply had to copy the new UUID from "sudo blkid" and replace the UUID in "nvim /etc/fstab" with the new one. Funnily enough. When I was troubleshooting on my laptop, the desktop PC which was failed to boot still got into a terminal after 90 seconds. In doing so I could pretty much do all this solution work without a flashdrive probably assuming I have the patience to wait 90 seconds for the daemon to time out.
Not really. It just has to be the same architecture, but it could be any distro. You just have to know how to actually set up a chroot environment yourself. It has basically three steps: 1. Mount any relevant normal partitions under the target mountpoint, e.g., your normal boot partition under /mnt/boot 2. Mount your special kernel filesystems into the target mountpoint (/proc, /dev, /sys) as well as your temporary filesystems (/tmp, /run, /var/run, whatever) 3. actually chroot in, with any shell you like arch-chroot is just marginally more convenient
Hey DT, as you are an Arco Linux user and myself too. Can you explain to us how to create a personalized ISO with calamares ? I know Erik Dubois made many and many videos about that, and I watch them all, but I'm a little bit lost about all that information and repeated ones, so I don't know where to begin and the amount of information are huge on his website and to be honest a little bit confusing. I'd really preferred a detailed PDF with step to step but if you can make a video on how to create a personalized ISO that resume all I'll be eternally thankfull.
Don't misunderstand, I don't want to discredit the great and huge work of Erik Dubois witch made a lot for understanding Linux, but I think that another way of explaining could be very useful and should make a great video :)
A lot of things can go wrong and a user may end up locked out of his system. I do not use Arch, but couldn't you boot to a previous kernel? Also what every linux user can do is use timeshift and keep daily, weekly and monthly snapshots of the system, preferably in a different ssd/hdd than the root. That way, you can anytime revert your system to a previous working state.
Something I noticed right away. You're just using a terminal here. Well, when you boot up a USB stick, you don't need sudo privileges. You're automatically the super user just booting up that USB stick. Then the next command after you mounted the root file system to /mnt, you would then arch-chroot /mnt. Not just arch-chroot. You unmounted it correctly but you forgot to actually mount it in your example. Also, if you have a weird bootloader like I need to have (rEFInd) there's a couple of other things you need to do. But yeah, you have the basics down. You just missed a couple of key elements there. One other thing I like to do on my systems, you mentioned something about video drivers going bonkers. I use the Awesome WM but I also have a basic DE installed like Cinnamon or MATE. This way, if for whatever reason, I can't get into my Awesome environment, I may be able to get into a simpler environment that was automagically configured for me on install or maybe I had to tweak a couple of things in that DE. But I got it to a point where I could use it if I really had too. I had to do this on my streaming machine the other night. I was able to log into Cinnamon, fix the issue with Awesome, and reboot (probably didn't need the reboot but I did one anyways) and Awesome came right up. So, there's an extra pointer for you new Tiling Window Manager Linux users... add a simpler interface to do any fixes you may have to do if your Tiling Window Manager won't come up properly.
I have been using btrfs and timeshift for a few months now on a new thinkpad. It absolutely blows my mind how easy it is to fix problems caused by botched updates, I don't even have to remember to take a snapshot. Timeshift creates a snapshot after installaing/upgrading a package and all I have to do is pick one from GRUB. Fingers crossed I don't do anything to break my system so bad that timeshift fails
Now that I use Gentoo I keep the installation medium on a USB if I ever happen to need it, though it may be a good idea to also have something with a graphical interface just in case, in which case I should try out Ventoy I guess.
I have had to do this in the past. Even with windows! I feel like a lot of people don't understand that they need more than basic skills if they don't have a huge IT support system at work/home.
So this is nice, but I had an update a few weeks ago and now it won't let me change my time without changing my time zone, and every time zone is off by about half a day. I installed Manjaro on a backup drive I had, vs and same issue on a fresh install as I have on my main OS. What do I do here?
I’m not a big fan of updates unless something isn’t working. Furthermore, I do whole system backups so I can just go back in time to when my system worked.
Why UEFI dbx updates for Linux Distros? I'm skeptical of allowing this firmware to update my POP__OS on an i5 12600K Z690 build? Any help is appreciated.
Man can you do episodes on MESA explanation for home user, Zink & its usage , How to pick file system for the machine, internal storage, external storage(usb, eSata, eSSD, eNVME) ? I got confused on what file system to pick for internal storage SSD drive, then decided on exFAT Thanks man
Looking at the title i immediately knew that it was an Arch or arch based distro😅 I have tried running Manjaro multiple times, but the longevity was about 6 months, after which my system would be broke completely (and I'm not a power user who fiddles with system files a lot)
why don't you have a backup? i use timeshft, but never needed to restore to-date. my next upgrade will be 23.04 which i will be upgrading about 6weeks after first release giving hopefully my programmes time to update to the new operating system. but thanks i did learn some new commands in terminal,
Hi guys ... great video, my system won`t boot and i would like to try this, but i also have a dual boot with windows ... does the grub loader get affected ? I want the grub menu stay exactly the same ...
I had grub fail on me during a kernel update on Garuda and I had solved it the same way using the Garuda live usb I had, I probably wouldn't have ever thought about using chroot if I hadn't tried installing vanilla arch before.
I've still got my live arch USB in the pile with a bunch of other USB sticks and I don't remember which one it is. Guess I should have put a Dymo Label on it so I would know which one it was.
it just happened to me. The updated deleted everything on my root partition on my arch linux. goodbye backups as well since it's all on the same partition as well (bad habit, I know). As much as I want to recover it, having no files to work on the root partition, the only option left for me was to re-install. The good thing is that my home partition is its own separate ssd so nothing much was lost
One time for some reason both my Linux laptop and PC made my OS unable to start after update. I needed to install missing cryptography package which for some reason disappeared after my updates. Also pacman after chroot wasn't working so i needed to use pacstrap.
I've got a multi boot system with Arch linux being one of the options, after a recent update I get the following error when booting from grub; Error: kernel doesn't support EFI handover Error: you need to load the kernel first. I do have a bootable arch system on an external drive. Although by error differs from yours I shall give it a try.
I use arch distro endeavour os with xfce and I had an issue with some corruption on my drive and wouldn't boot up. I use rescuezilla iso on an usb. It has tools like gparted on there. I used it to run a recovery on the drive and voila it booted up. all good .... :)
this issue happened to me before so luckily i had snapshots to restore my system how it was before the update (using btrfs file system) but after that every time i update the kernel for some reason won't work so i switched to fedora ... btw why won't you give fedora or void linux a try as daily driver ?
If you have both linux and linux-lts, why didn't you just boot to the other one, the one that wasn't in the middle of being installed/updated? And an absolute must-have is 100% a boot stick :-) I use ventoy to have all my images on one stick.
It happens so infrequently (thankfully) that you forget how to fix it.
Yes first rule of a totally borked bootup is "Don't Panic!". This morning I had both my primary data and backup data drives for my home server both lose their partitions - the TestDisk utility did an amazingly easy job of recreating the partitions (after a cup of coffee).
hey Derek, something you forgot to mention is that this solution works only for legacy bios setup, if you have uefi you must also mount the EFI partition. And not to mention, inside the usb drive you don't need sudo privileges
I think it's about time legacy boot goes the way of the dodo, imo
@@majoraxehole alright let me just throw all my old machines into the trash
@@michaelnugent8181 exactly, that's pretty much what it's like whenever a Windows user throws or gives away an older machine because it doesn't run the latest version of Windows; more succulent machines for me
On my EndeavourOS live CD, I did need either sudo (with no password) to do root operations.
But you're right on mounting the EFI partition for those systems.
Definitely good to know which is your main system partition! Mount main partition to /mnt. Then look at your fstab "/mnt/etc/fstab" to see which drive/partition is for EFI.
Also worth mentioning to mount other stuff if you have separated other directories (*check fstab*):
_Very few distros does this to this extent_
- "/home"
- "/var"
- "/lib"
- "/boot" | "/boot/efi"
All other mounts should be in your fstab. You shouldn't need to mount your game's partition for instance, but anything that is for your root folder "/" or boot folder "/boot/*".
This is a great way to rescue a Linux install.
Also I heard that you have multiple USB drives with ISO's on them. Something you can try to consolidate them is Ventoy, it's a bootloader for running multiple ISO's from a USB disk.
For example I have a Ubuntu (server and desktop), fedora, rocky, Debian, arch, Garuda, endeavour OS and windows on the same USB drive
I see where this can be useful, but this results in a SPOF (Sngle Point Of Failure) where if the USB gets corrupted or dies for any reason, then you become in a bad position.
Having an extra thumb is always good. 👍🏻👍🏻👍🏻
@@alrshdn well you can use vendoy on both
@@glidersuzuki5572 You sure can.
I don't think ventoy is a good idea.
it is a little buggy on whether an iso will work or not with it .
@@Gurj101 in the more than a year I've used it I've came across 1, yes 1 iso that didn't play nice with it.
That was proxmox VE 7.4... everything else worked...
For those new to Linux.
Another, knowledge share gem.
Thank you chap!
This is why I always recommend keeping a current latest kernel and an LTS kernel on Arch. If something happens with a latest kernel update, either something like what happened here or compatibility issues with some changes on the new kernel, you can simply boot the LTS kernel and reinstall or roll back the latest without having to boot a live environment.
Or just have snapshots.
@@Clobercow1 Sure. Any method of rolling back or booting the system before an issue happens without having to have recovery media is preferable to having to write a USB drive and boot a live environment to recover.
Gentoo users can do this without an update 😎
Thank God you're talking about this. I had this problem where Pacman crashed in the middle of installation and it deleted my vmlinuz image. Fortunately I knew how to counter this and had a spare bootable usb with me.
You can upgrade your system in virtual console in order to avoid system failures because there is a slew of reasons for the graphical environment to crash especially during a major update/upgrade. Hope it helps.
Since I installed openSUSE Tumbleween in one of my machines I've tasted the divinity of the BTRFS file system paired with Snapper in order to have snapshots created for me and available in a GRUB menu so I don't have to rely on USB pendrives that can be misplaced or easily corrupted. I think that heavenly match can be installed in any distro. Arch would benefit tremendously from this.
"I use arch based distros." Oh yeah, where's your neckbeard then?
If you cannot load after installing new kernel, just reboot and press Esc many times, while PC is loading, and then, in the grub menu, you can select previous kernel.
My thoughts exactly. Just try to boot the previous kernel.
interesting tip:
- create new partition
- just copy live cd iso file content inside this partition
- if something brokes choose to boot from this partition.
wuala you always have live cd with you
I think fedora and for sure Pop!OS does this out of the box. It's a pretty cool system. Just have to keep the iso there up to date.
Linux people will say "this year is the year of linux" and make tutorials on what to do when you accidentally nuked your kernel
Great instructional video and very important information. Thank you, DT!
I just got a notification for a new kernel update in Mint 💀
This is one thing that's great about Linux. You can fix problems.
Can't tell you how many times I had Windows break back when I still used it and have the only solution be to reinstall from scratch (reset wouldn't even work!).
On Linux, usually just gotta chroot or look for a solution online. Someone has probably solved your problem before.
No need to reinstall from scratch like on Windumb
I’m using a custom GRML for this - with ZFS support and the keyboard layout set to Dvorak.
You can work around a lot of these boot issues from the boot loader command prompt and at least get your system to boot, but it is a lot more tedious than the process you described. And the boot loader command prompt also typically only supports QWERTY.
Just another little niche tip i had to figure out the hard way: When you need to fix the Linux install on an old or exotic peace of hardware, some BIOSs or UEFIs only have drivers for USB2.0 - meaning it won't let you boot off of a newer drive. I have a micro-sd to usb adapter which uses usb2.0 which helped me out ( just a little fyi sidenote here ^^ )
Thanks for the great video. I am in a similar problem with manjaro. I had the kernel not install properly on an update a few months ago. I was able to rescue it with a manjaro usb stick and using my timeshift backup, but I've been afraid to update the system since. I am rather behind in the updates to my system.
You should probably update or reinstall...
I'm confused. After the borked update and failed reboot, did you not have a GRUB screen where you can toggle to a different or earlier kernel? (i.e: Do a forced shut-down, power-up and quickly enter the GRUB menu before it tries to load the system. That wasn't an option?)
My thoughts exactly
That would require you keep the old kernel, and some people don't do that. Personally, I only bother to update the kernel for security updates and I'll definitely keep the most recent previous kernel, but I also have space, something that yet again not everyone has.
@@anon_y_mousse thing is DT always mentions how he has both the linux and linux lts kernels installed, he even broyght it up in this video. So by all accounts the kernel that wasn't being updated should still've been fully functional
@@marck0060 So then the explanation is clear why he didn't do it, to show others who didn't have his backups how to do it.
@Anony Mousse
1. Deleting old kernels is not really the best method of saving space on your drive for now obvious reasons.
2. The Arch package system should remove old kernels as the LAST step, again for now obvious reasons. If not I would suggest a bug report .
3. I am not sure of the exact problem here, but generally the easiest fix is boot-repair. It was made in the Ubuntu ecosystem but according to the Arch AUR fans should be available on Arch. But then the Arch kernels are now configured to not work with most distros so again another example of why I no longer use arch as although I can do all this chroot configuration it is not really how I wish to spend my time.
Sir!!! You are a life saver!!! Had this issues with Manjaro! If happens again i will use this solution. Thanks a lot!
If you don't have access to the arch-chroot command I'd recommend pulling up the gentoo handbook, under the chapter for installing the base system they have a section going over what needs to be mounted into the new root to get a functional system. That way it can be used normally to fix whatever issue you may have :)
I had a similar issue once. But i solved it a bit different. I started a liveUSB, chrooted and just installed the previous Kernel before the update, which was archived on my machine. Pacman -S was not an option since the new kernel was not compatible with my network drivers of my Wireless devices, which i rely on since the router is on the other side of the building.
I had to do it once with my arch system and later with my Debian. Though the Debian case was a bit more difficult to fix. I borked my machine by accidentally overwriting my GLIBC with a newly, freshly compiled GLIBC that i needed to recompile software i developed for my arch system. (I forgot that the folder in which i compiled the GLIBC to was already used by my system native GLIBC)
At that point i didn't reinstall the old GLIBC but just used a snapshot iso i had from that system since that can be less trouble as fiddling with that library.
But for most Kernel issues, if a Arch update trashes your system with a new kernel and you have no internet connection, a previously installed Kernel package should do the trick.
Protip: Always update the system inside tmux (or screen) so that if the terminal emulator, desktop enviroment or window managar dies on you the update process won't be killed and can be easily resumed on a new terminal emulator process or even on the tty.
Debian GNU/Linux ... you'd just reboot into the previous Kernel. I'm not sure, though, how you'd get this particular error on a Debian system anyway.
Exactly why I use Pop_OS!
Oh man! So many times I have had to do this throughout my Linux career...
But for me, the issue that causes this is often Systemd fighting ZFS and I need to set one or the other straight, quite often it is systemd needing a boot to the ass though.
Because if a ZFS on root system breaks during an update, you cannot just rescue it with a different kernel boot, you have to do a few extra steps.
Saved my install's countless times with this trick 👌🏻
I use Garuda and they make it so easy to chroot. But in the past, I had significant issues when I was on Linux Mint dealing with the grub. I made a thread on the Linux Mint forums and a guy that posted said that's an easy fix. Gave me about 6 or 7 commands, almost like what you did. We went back and forth because I didn't get it since almost on every command I put in, I was getting an error. He said, just go on and finish the commands and reboot. And it worked. I had a similar issue with the bootloader when I was on Linux. Since it failed so many times with the grub, I had several bootloaders (probably about 5 or so), that wouldn't get past the grub part. It would hang up. So, again I had to post on the bootloader error. But this time I was on Manjaro, so I posted on their forums. And that was a BIG help. It was simple as deleting those pesky bootloader entries. Then I restarted and installed Manjaro with no problem.
I've only bricked my arch install once, and it was because of an outdated zfs kernel module. Switched that over to dkms and ive never had another issue.
And yes, I installed root on ZFS. Sue me.
I've lost track of the number of full Linux installs I have on USB sticks. I have four on a shoestring on me at all times.
Just four? Does it make me a hoarder that I've got 20?
Great video. I did a couple of full re-installs before I discovered manjaro-chroot or arch-chroot. Chroot on its own can be quite daunting until you know about the process you just described.
Yeah, arch-chroot eliminates the need to bind all the correct folders when mounting the broken filesystem. Quite trivial really :)
how did you discover it late ? if your re-installs included base arch re-install then you would atleast be aware of it. welp can't say it will like click in your mind that you can do that but you will be aware since arch is installed that way. you pacstrap basic packages and chroot and install some more necessary ones later to install arch if i remember correctly.
@@Gurj101 Only an absolute imbecile would install Manjaro that way.
Backups are sooooo important
0:45 that nightmare happened with me once with a fork of sway!
never used again, good thing i had a secondary kernel (tks)
I recently added a sata m.2 drive to my system. I wasn't 100% sure which drive to install /home and /root on. After seeing DT's partition setup it looks like having /home on the slower drive is perfectly fine. After looking through the manual for my motherboard it states the sata m.2 can also use an nvme 2.0 but I've never seen one of those before.
They're still really frequent just gotta check the specs. The cheap 20 doller ones usually are
I got a 250g SSD and a 1TB sata platter drive. I install / on the ssd and /home/user on the sata. never had an issue.
@@Dratchev241 I've got the same setup and reinstalls on the SSD work great
Even though I understand the why, I think it's really cool you explain why you're doing the things you're doing. I know when I was first learning Linux it was so overwhelming how often people and guides would just tell you to do something, kind of expecting you to just know why.
1:08 Oof, that brings back some deeply repressed trauma for me
3:55 Make sure you arch-chroot CORRECTLY! That is in this case, "sudo arch-chroot /mnt" You forgot the /mnt part during your video. 😉
Yes!
Don't you need to mount /proc /dev /sys or arch-chroot takes care of it?
I remember chrooting for rescuing and couldn't get stuff done until I mounted dev (which makes sense - can't grub install into /dev/hda1 if /dev/hda1 doesn't exist)
ETA: I read the docs, yes, arch-chroot takes care of it.
I have never had a kernel update blow out on me, but I have hosed the display manager, trying to tweak a setting. As you have shown before, always back up a config file BEFORE you make changes! I think that I was on Ubuntu 7.10 back then, so it was easy enough to just tell the installer to act as an upgrade, and let it fix the settings. I'm sure that there was a much better way to do it, but I wasn't in a mood to look it up at the time, when I knew running an upgrade would only take a few minutes.
Thanks for making this video. Very helpful.
In other words, the update didn't kill your pc, its your wm crash that killed your updating at the "best" timing. Had a similar exp a few years back and circling the web was not fun. Thank you for putting out your solution dt.
Mine was a bit different, i had to go and change boot parameter as well
Full guide here for those who prefer written guide. Thanks DT...
forum.xerolinux.xyz/thread-90.html
I had a similar issue recently. Updated Thursday then didn't boot my system until Friday. Root partition couldn't be found for some reason. System wouldn't boot. Fix was booting to the live usb and running `mkinitcpio -p linux`. Rebooted and everything worked like a charm. Not sure what happened, if the initramfs was corrupted, or never ran after the kernel upgrade.
My file doesn't exist anymore since I took it out when putting the LTS, and then I never got Arch to run. lolol. Thanks though!
I actually had this same issue on endeavour OS the other day. The power went out in the middle of a kernel update. I did this method and I couldn't get through. I had ZSH as my default shell when the kernel was messed up and this method kept complaining about bash directory or whatever wasn't there. I reinstalled, I wasn't in the mood to search for a couple of hours. Thankfully, I separate my / and home partitions.
Dang it! I got this in my feed a week ago but didn't watch it then! Just today after updating (successfully, I thought) my efi systemd entries completely disappeared along with my latest and lts kernel images, barring the intel-ucode. I kept a hard drive with EndeavourOS on it just in case, and googled for a while until learning about chroot (the first time I found the arch wiki useful!)... then as I was so close, followed some random internet user's advice to install a conflicting package and uninstalled a system package... then promptly forgot which package I uninstalled. That package is what the kernel used to rebuild itself, so reinstalling didn't help. Eventually I realized that their solution was needlessly complex - I could have just reinstalled the kernel - and facepalmed. I fiddled for a bit with systemd to try to manually build & edit the images & entries but eventually decided to just reinstall the system (I keep an installer script and could access the /home folder through the live usb, so I didn't have to go back to a backup).
Most important thing: I learned from my failure. Next time this happens, I won't make the same mistake, but rather just reinstall the kernel through chroot!
man that sure looks easy and stress-free 👍
Had similar issues every time the linux-nvidia firmware was updating. I would get stuck in the boot screen with no TTY or a window manager.
To fix that issue I booted into an LTS kernel on my system which allowed me to reinstall the nvidia drivers, booted back to the regular kernel and it fixed the issue.
That's why I like to keep a regular kernel and an LTS version on my system at all times. Just makes sense for these kinds of situations.
never liked how Arch handles kernel and its updates,
somehow gentoo is always the most maintenance free, that lets me focus on doing work on computer and not worry about os
really? I thought using Gentoo requires you to maintain it frequently... tell me more 😊
My favorite distro is Slackware, but I'm probably biased.
You don't have a fallback kernel? Usually booting fallback works fine. Then just re-run update, recompilr kernel and done
Will this still work if you use a distribution with a different package manager? So for example, can you fix a broken update performed using pacman by installing a new Linux kernel using apt?
Please tell me you fixed this by installing a non-rolling release distribution cause that's the only way you're going to solve the problem of sh*tty distributions constantly breaking. There is a reason decent distributions aren't full rolling. It's one thing where a distribution is partially rolling. There are good reasons for distributions to release updates- like to support newer hardware, web browsers, and even instant messaging clients. But NOTHING should be released without some amount of basic QA and testing done. THIS is what makes Arch and Arch based distributions like Manjaro unusable. Some of us do more than just play our with computers. They're not just toys. They're utilized for actual work.
I just made this exact mistake. I'm so used to forcing a rapid shutdown or restart all the time, I downloaded a kernel update, didn't tell me by the way, looked just like a normal app update. It looked like it was done installing, I did a sudo reboot now, now the new kernel won't boot, because it wasn't finished installing. Now I see why there are several kernels to boot from, it's not a big deal, just kind of a pain to remove the partially installed kernel and try it again, perhaps with a little more patience this time.
For me, a similar situation occurs whenever I update any systemd component. So I exclude all systemd components before an update, and I update the systemd components separately in a tty and run mkinitcpio afterwards
Sounds like a reason to not use systemd and makes me happy that I don't.
I think this was just a bug, not something you did wrong. It seemed like the kernel didn't work with nvme very briefly last week. Had to do the same thing
and this is why people should drop all distros and use nixos and guix, because you can easily rollback to the previous stable version by a simple restart, no need for a USB. Just the functional design (as in functional programming) of NixOS
Im on gentoo running the latest kernel. almost everyday my cpu goes overtime when compiling it. Of course, I’m on a distribution Kernel, meaning it’s a bit more streamlined when updating.
After you’ve mounted your internal drive, when you run chroot, do you have to specify the location? (in this case /mnt )
yep
3:52 IT SHOULD BE "sudo arch-chroot /mnt" ;)
Been there too many times :D Tips like these are very good for new users.
Just today, my secondary Windows drive that I only use for games that don't work on Linux yet messed with my grub. And so I was in class, pulling out my arch usb, reinstalling grub and creating the config. This is exactly why I keep an arch usb in my backpack at all times.
What can I do if the update breaks on a VM. I can't use live USB in that case. Don't have a recent snapshot.
Super grub 2 USB ISO fixes any broken grub or system boot issue! Works great!
In my case it was completely different. But this video still helped me in narrowing it down by getting rid of the obvious first steps. I'm leaving this as a future reference to both myself and other people who come across the same problem. TL;DR: If you have other drives mounted onto the main Linux drive, this solution may be for you.
Since I'm dual booting Windows and Arch Linux I often tend to share files between them, and thus I always have Windows's main drive (C:\) mounted onto the Linux partition. This caused Arch to not boot after I updated Windows 10 to Windows 11. Turns out the solution wasn't too advanced. If I did "cat /etc/fstab" I could see the Windows drive had one UUID which it expected. But upon running "sudo blkid" I would notice the Windows drive UUID had changed, probably due to Windows 11 changing things. I simply had to copy the new UUID from "sudo blkid" and replace the UUID in "nvim /etc/fstab" with the new one. Funnily enough. When I was troubleshooting on my laptop, the desktop PC which was failed to boot still got into a terminal after 90 seconds. In doing so I could pretty much do all this solution work without a flashdrive probably assuming I have the patience to wait 90 seconds for the daemon to time out.
Ventoy is your friend :) I always have few isos on my sticks :)
Precision = this has to be an archlinux system (or based) on stick
And have to point to the good mount points
Typically
arch-chroot /mnt
Not really. It just has to be the same architecture, but it could be any distro. You just have to know how to actually set up a chroot environment yourself. It has basically three steps:
1. Mount any relevant normal partitions under the target mountpoint, e.g., your normal boot partition under /mnt/boot
2. Mount your special kernel filesystems into the target mountpoint (/proc, /dev, /sys) as well as your temporary filesystems (/tmp, /run, /var/run, whatever)
3. actually chroot in, with any shell you like
arch-chroot is just marginally more convenient
Hey DT, as you are an Arco Linux user and myself too. Can you explain to us how to create a personalized ISO with calamares ? I know Erik Dubois made many and many videos about that, and I watch them all, but I'm a little bit lost about all that information and repeated ones, so I don't know where to begin and the amount of information are huge on his website and to be honest a little bit confusing. I'd really preferred a detailed PDF with step to step but if you can make a video on how to create a personalized ISO that resume all I'll be eternally thankfull.
I also use Arco with chadwm and I am interested in creating an iso. Erik Dubois is not very good at explaining stuff.
Don't misunderstand, I don't want to discredit the great and huge work of Erik Dubois witch made a lot for understanding Linux, but I think that another way of explaining could be very useful and should make a great video :)
A lot of things can go wrong and a user may end up locked out of his system. I do not use Arch, but couldn't you boot to a previous kernel? Also what every linux user can do is use timeshift and keep daily, weekly and monthly snapshots of the system, preferably in a different ssd/hdd than the root. That way, you can anytime revert your system to a previous working state.
On Arch, all kernel installs for the main Linux kernel are in /boot/vmlinuz-linux. In other words, the old kernel is overwritten by the new kernel.
Something I noticed right away. You're just using a terminal here. Well, when you boot up a USB stick, you don't need sudo privileges. You're automatically the super user just booting up that USB stick. Then the next command after you mounted the root file system to /mnt, you would then arch-chroot /mnt. Not just arch-chroot. You unmounted it correctly but you forgot to actually mount it in your example. Also, if you have a weird bootloader like I need to have (rEFInd) there's a couple of other things you need to do. But yeah, you have the basics down. You just missed a couple of key elements there.
One other thing I like to do on my systems, you mentioned something about video drivers going bonkers. I use the Awesome WM but I also have a basic DE installed like Cinnamon or MATE. This way, if for whatever reason, I can't get into my Awesome environment, I may be able to get into a simpler environment that was automagically configured for me on install or maybe I had to tweak a couple of things in that DE. But I got it to a point where I could use it if I really had too. I had to do this on my streaming machine the other night. I was able to log into Cinnamon, fix the issue with Awesome, and reboot (probably didn't need the reboot but I did one anyways) and Awesome came right up.
So, there's an extra pointer for you new Tiling Window Manager Linux users... add a simpler interface to do any fixes you may have to do if your Tiling Window Manager won't come up properly.
btrfs + timeshift + grub
I have been using btrfs and timeshift for a few months now on a new thinkpad. It absolutely blows my mind how easy it is to fix problems caused by botched updates, I don't even have to remember to take a snapshot. Timeshift creates a snapshot after installaing/upgrading a package and all I have to do is pick one from GRUB. Fingers crossed I don't do anything to break my system so bad that timeshift fails
Now that I use Gentoo I keep the installation medium on a USB if I ever happen to need it, though it may be a good idea to also have something with a graphical interface just in case, in which case I should try out Ventoy I guess.
Hi! It's been ages since I got a corrupt kernel install. I'm not an Arch user. Doesn't Arch have alternative kernels to choose from at boot?
Very useful knowledge!
I have had to do this in the past. Even with windows! I feel like a lot of people don't understand that they need more than basic skills if they don't have a huge IT support system at work/home.
thanks for sharing. Now I hope I don't forget this video so I can come back here just in case something bad happens
So this is nice, but I had an update a few weeks ago and now it won't let me change my time without changing my time zone, and every time zone is off by about half a day. I installed Manjaro on a backup drive I had, vs and same issue on a fresh install as I have on my main OS. What do I do here?
Thank you Tiny Hands
I’m not a big fan of updates unless something isn’t working. Furthermore, I do whole system backups so I can just go back in time to when my system worked.
Why UEFI dbx updates for Linux Distros? I'm skeptical of allowing this firmware to update my POP__OS on an i5 12600K Z690 build? Any help is appreciated.
this exact scenario happened to me like 2 days ago and i did a full reinstall
i wish you released this video earlier lol
Man can you do episodes on MESA explanation for home user, Zink & its usage , How to pick file system for the machine, internal storage, external storage(usb, eSata, eSSD, eNVME) ?
I got confused on what file system to pick for internal storage SSD drive, then decided on exFAT
Thanks man
Looking at the title i immediately knew that it was an Arch or arch based distro😅
I have tried running Manjaro multiple times, but the longevity was about 6 months, after which my system would be broke completely (and I'm not a power user who fiddles with system files a lot)
why don't you have a backup?
i use timeshft, but never needed to restore to-date.
my next upgrade will be 23.04 which i will be upgrading about 6weeks after first release giving hopefully my programmes time to update to the new operating system.
but thanks i did learn some new commands in terminal,
Hi guys ... great video, my system won`t boot and i would like to try this, but i also have a dual boot with windows ... does the grub loader get affected ? I want the grub menu stay exactly the same ...
I had grub fail on me during a kernel update on Garuda and I had solved it the same way using the Garuda live usb I had, I probably wouldn't have ever thought about using chroot if I hadn't tried installing vanilla arch before.
I've still got my live arch USB in the pile with a bunch of other USB sticks and I don't remember which one it is. Guess I should have put a Dymo Label on it so I would know which one it was.
it just happened to me. The updated deleted everything on my root partition on my arch linux. goodbye backups as well since it's all on the same partition as well (bad habit, I know). As much as I want to recover it, having no files to work on the root partition, the only option left for me was to re-install. The good thing is that my home partition is its own separate ssd so nothing much was lost
One time for some reason both my Linux laptop and PC made my OS unable to start after update. I needed to install missing cryptography package which for some reason disappeared after my updates. Also pacman after chroot wasn't working so i needed to use pacstrap.
I've got a multi boot system with Arch linux being one of the options, after a recent update I get the following error when booting from grub;
Error: kernel doesn't support EFI handover
Error: you need to load the kernel first.
I do have a bootable arch system on an external drive. Although by error differs from yours I shall give it a try.
I use arch distro endeavour os with xfce and I had an issue with some corruption on my drive and wouldn't boot up. I use rescuezilla iso on an usb. It has tools like gparted on there. I used it to run a recovery on the drive and voila it booted up. all good .... :)
I use the zen kernel and systemd-boot and I've been doing fine
this issue happened to me before so luckily i had snapshots to restore my system how it was before the update (using btrfs file system) but after that every time i update the kernel for some reason won't work so i switched to fedora ... btw why won't you give fedora or void linux a try as daily driver ?
If you have both linux and linux-lts, why didn't you just boot to the other one, the one that wasn't in the middle of being installed/updated? And an absolute must-have is 100% a boot stick :-) I use ventoy to have all my images on one stick.
i guess we can boot with the live usb with linux mint inside, will be the same process? thanks
Thank you, this already saved me once from re-installing!
All right man. It's time you switch to immutable bulletproof distro. Fedora Silverblue or Fedora Kinoite.
HeyDT: You remind me of a very sober Stone Cold Steve Austin!
All this is well and fine, until you need to chroot to update/reinstall grub on a btrfs LUKS partition with root in a different subvolume than home
i had the same problem but i always get confused bout live boots on arch linux
I do not get a new kernel automatically on PClinuxOS. Kernel updates must be chosen, What kind of broken system do you have?
PCLinuxOS is debian based, not arch