Had a disaster last night with no backups and was able to restore every single vm thanks to your help. Props for not including a life story in the video, seems like most other tech guys are trying to story tell instead of inform. This was of great help. Thank you very much.
i had 5 vm's on an OLD proxmox (v 4 ish...) they ran for many years, and i feared the day the host would give up... at last i had to do it... took the 'backup' method, and all was up and running within 20 min.... proxmox IS FANTASTIC.....
I had a config.db corruption after some risky "live" changes, basically making Proxmox not recognise its own setup. Rebuilding the db was impossible and extracting the entries didn't work either. Adding all vms and containers manually to recreate the .conf was my solution after manually adding local-lvm, which also went missing. Thanks a million! Awful way to learn a lesson, but luckily ended up without any data loss. Back up all the things.
You're delivering excellent content ! Your videos are a very important and indeed outstanding contribution to the open source community ! Thank you so much !
Thank you, looking at your video made me look other ways and i was able to recover all my VM after a big HA Cluster problem, also will take your advice and from now on PROXMOX on separate disk, so can be rebuild with ease. Thank you again
Hello thanks so much for posting this troubleshooting video. I have a situation currently and I know the answer is in the video but I’m not sure the exact method for my particular issue. I have an issue in proxmox with HA. I had a node move to another host migrate, but the disc volume did not go with it. I shared the local ZFS drives as you should. These are also the ZFS Storage drives that proxmox was originally installed on Each node has the ZFS storage, but doesn’t have the data from the migrated node. Only the information from the VM that was created on that node. So, when I try to migrate it back to the original node, it says no link to original volume. I was in the midst of setting up HA but didn’t quite get to the storage configuration yet. I’m surprise the node even moved I don’t think it actually did complete because there was errors and errored out so I had to disable it. I tried to migrate it back, but could not same issue missing disk vol plus USB external connection. All data is intact. The disk is still on the original node, but the VM appears to be on another note. No back up or snapshots yet. I’m still working on that and i’m learning and quite new at this. I was in the process of setting up a three new cluster. Hi availability with cef storage. Thanks so much for any advice or assistance I would Much appreciate it. I’m stuck!
Thanks for this very detailed and still unboring comprehensive video about Desaster Recovery. I was wondering if restoring the pve host using the proxmox backup client is easier or even possible - or maybe not advisable.
Really wish I had found this video a week ago. I was trying to restore a couple VMs after moving my proxmox installation to an nvme but only a couple of my VMs would boot. Luckily, I didn't fully commit to the new boot drive and just had to plug the original drives back in.
Going to try this in VMware workstation. don't understand having to have a ds to get your backups, on esxi you run a backup solution you can backup and then restore into the hypervisor. Will try work it out
This is the whole reason why I'm moving away from using OMV+docker. Love both but backups and restore suck. After dealing with a full OS failure following later with a ZFS failure I figured this is my best bet. Thank God for my borg backup.
currently not using PBS but have a schedule for VZ Dump Backups on a weekly basis. Saved me a lot of time when something needs to be restored. I use the VM on a network share (unraid) and from time to time upon rebooting a vm or shutting it down, it wont get back online statying that the disk/image is missing. Restore always worked 100% but its definitly something weird on this behaviour.
Another way to eliminate the guesswork is to just restore the config file from backup and use the disk files from the old system. This way you may not have to restore terrabytes of data and a older version of the VM. You probably have to edit the config file but its easier than creating a empty VM.
The setup I have is 2 x 1tb ssd mirrored in a micro PC with 5TB ext drive for VM backups. First question is for the failures you have in the video if I lost one drive does the proxmox continue on the other and just highlight one as faulty or does it need reboot or config to use the 2nd of the mirrored drives. Now on the VMs restore. You show the VMs in the local store but dont use them and just recover. Does this mean the VM files in the local need to be deleted first ? Last question. Is there a way to backup the config so I just reinstall proxmox, restore the proxmox config and create the mirror again and restore the VMs from backup ?
Great video - After I remap my containers conf files to the correct zfs paths ie subvol-100 etc. If I try to lxc-attach I get permission errors like "bash: /root/.bashrc: Permission denied". I also have X2Go setup on a few of these containers and for some reason I can't connect to them I suspect it's a permissions thing. In my case I solved by setting unprivileged to 0. Another tip though if you don't know which ct was which ie 100,101,105 etc you can just cat "VMStorage/subvol-105-disk-0/etc/hostname" so you know you are remapping the correct container to the correct name
Hmmm I followed your steps for VMs on ZFS but it's not so much an issue that I'm putting in a new OS and mounting everything as much as my ZFS01/VMs location disappeared in Proxmox and when I re-add it is not finding the VMs like your system has done as they are missing in the directory entirely. The individuals ZFS01/isos and the /ZFS01/backups directories in Proxmox are fine. The ZFS system is fine it's just the virtual machine directory was gone in Proxmox, not on the server itself. So I re-added the pathway of /ZFS01/VMs and nothing. I am on the server via shell and looking for the mountpoint and ZFS01/VMs is there. I go inside and I do not see the disks at all. when I use zfs list it says there are 3 VM disks with no mountpoint. Each titled ZFS01/VMs/vm-101-disk-0 and iterations with the different 3 numbers. The ZFS01/VMs directory has a mount point so I'm expecting the VMs inside the directory do not need their own mountpoints.
There is no good way to backup a pve host. The proxmox suggest solution if a host fails is to do a clean install and restore backups. This makes sense with a large cluster but with a single node a full system restore could save a good amount of time.
Thanks for this. 🙂 I have periodic VM backups via Datacenter-->Backup. The occasional restore has so far always worked. Maybe I missed it, but if your hardware dies how do you restore the Proxmox hypervisor itself on new hardware? What do you use for backup, maybe Veeam? Or just start the new HW with a fresh Proxmox install?
Any decent solution relies on at least a second independent backup server, and the easiest, most reliable solution for Proxmox is to install Proxmox Backup Server on that 2nd host so all your VM/CTs are backed up to separate hardware. If the worst happens, then reinstall Proxmox PVE on the original or new hardware and copy the VM/CTs from the PBS backup machine back to your freshly installed PVE machine. The ideal solution is 7 machines, 5 in a PVE cluster with Ceph storage and 2 running PBS, with the 2nd PBS machine remotely syncing from the first PBS machine.
It is not necessary or "better/safer" to have a separate boot disk for Proxmox. If you install Proxmox on a ZFS Raid+ then if you do lose a drive you can still boot to the raid array, albeit in degraded mode, and examine the problem from within the same OS and hardware, and with the local VM/CTs still intact. Having to wipe and reinstall the OS and then also reinstall VM/CTs is far more of a burden than relying on the kernel and OS installed on each drive in a ZFS raid. I'm not sure of the situation when using LVM, as I've never used it. Try it. Install Proxmox on two drives in a mirror, then disconnect/destroy one of them and see if you can still boot up on the other one.
Hi, thanks for the info. Quick question: I can't log in to proxmox on my laptop even if I type in properly my login and password. Is there an option to remove proxmox from the mini pc server and restore windows?
You can install windows from the installer to have a fresh copy of windows. If you have a backup of the Windows install you can also restore that. Does it open the web ui for Proxmox in the browser?
cool video but, how to do it when you have no zfs? i am just trying to get the darng files off that old drive, and i dont wand to study it engineer to do so
How is the drive formatted? If its LVM you should be able to see the virtual disk images as block devices and copy them to a file on a new system. If there files on a drive, you can mount the drive and copy over the files.
@@ElectronicsWizardrywow, thanks for the quick answer, it was LVM, but when the mounting stuff starts I have no clue what to do. The principle is very understandable, but what are the commands? Asking google didn't help me much either. After hours of frustration, I just decided to start over, giving me days of work for free. Thank god I use unraid, so I just had to import the drives / USB stick and everything was there, the other vms are binned tho I can definitely not recommend proxmox to anyone besides RUclipsrs and infrastructure engineers it is just too bare bones and unuserfriendly to actually request founding.
Good day sir. My proxmox server died., Well the boot disk died. I have rebuilt and reinstalled proxmox os onto a new raid z1. I'm now trying to import the zfs raid z1 where I had all my VMs and such stored. I have imported the zfs disk but have not made it any further. Is there anyone who can help me out? Thanks
Once the drive was imported it shows up under zpool list right? If that’s the case you probably have to add the storage to proxmox and use the same zfs path that you used when the zpool was used before.
@@ElectronicsWizardry So I made it as far as adding the storage and shows up in proxmox as oldszfs. I then used the command: the cd /dev/zvol/vme to see the different virtual disks I have. I'm stuck at the part of creating a new virtual machine and adding existing vmdisk to boot from. I did not ssh into my server, instead I used the shell interface through the proxmox web GUI
I'm a bit confused about the lack of ability to back up the host itself (the system image and/or configuration). The official position of Proxmox team is "just reinstall and copy over some folders" is not inspiring. I wish someone come up with a robust and an easy solution to back up the host.
@@Darkk6969 Why do they plan to add it PBS only? At the end of the day, how to restore quickly the PBS itself, if suddenly it goes down? All kinds of computer appliances offer, at least, minimal config backup from the GUI to a fixed or arbitrary location.
my proxmox is on btrfs file system and running several vms and containers on it but current the host is not booting up and sometime it works vi tahe terminal but slow and cant start the vms,how can i backup when it comes up via terminal
I'm nost sure if you have anouther system or other drives, but in this situation, I'd fire up a new install on different drives, then plug your BTRFS drives in. Then you can probalby copy off the vm image files to the new system, and create vms that use those image files.
Ah yes. One of the reasons I STILL don’t use Proxmox for PRODUCTION! Say what you must about ESXi, but still rules when it comes to easy, quick recovery!
Thank you for this great video! It would appear then, that the VZbackups will ALSO store the config information along with it, vs. the disk files/images alone, which will NOT have the config info in it. Would that be correct? Sounds to me like that if I am going to be manually backing up the disk images, that I would need to back up the /etc/pve/qemu-server/*.conf files and/or /etc/pve/lxc/*.conf files along with it. (Or have an speadsheet file that tracks the VM/CT ID along with the associated details so that if I need to recreate it and/or restore it from the disk images, I would now know config I had when I originally created said VM/CTs.) What do you do for an UEFI restore? Is it basically the same process where you would create a dummy UEFI VM and then edit the config file to point it to the UEFI boot disk and the actual data/OS disk? Thanks.
Taking a look at that service, I don't see a reason for it to stop the VMs from running. Do you have any logs that would show how the vms were shutdown and what may have caused that? Taking a look at the files for the service I don't see a reason why vms would be stopped. Does running /usr/bin/pveupdate also stop the vms? The pve-daily-update.timer runs that script, so that may be able to tell you if its the update script or systemd.
Excellent video!! Love how you walked through each of the various recovery options.
Had a disaster last night with no backups and was able to restore every single vm thanks to your help. Props for not including a life story in the video, seems like most other tech guys are trying to story tell instead of inform. This was of great help. Thank you very much.
Glad I was able to help get your VMs up and running again.
Thank you so much. So much value in one video I was able to recover a very important LVM VM, thanks to you
Man, I love your videos because take a real life common issues, and also mixing many hardware old school scenarios. No bs, no silly stuff
i had 5 vm's on an OLD proxmox (v 4 ish...) they ran for many years, and i feared the day the host would give up... at last i had to do it... took the 'backup' method, and all was up and running within 20 min.... proxmox IS FANTASTIC.....
I had a config.db corruption after some risky "live" changes, basically making Proxmox not recognise its own setup. Rebuilding the db was impossible and extracting the entries didn't work either. Adding all vms and containers manually to recreate the .conf was my solution after manually adding local-lvm, which also went missing.
Thanks a million!
Awful way to learn a lesson, but luckily ended up without any data loss.
Back up all the things.
You're delivering excellent content !
Your videos are a very important and indeed outstanding contribution to the open source community ! Thank you so much !
You my friend, have saved me 5 massive times already. Thank you soo soo much for this video.
I always try to backup the etc folder of my Proxmox hosts as well as the VMs.
This greatly simplifies the restore process...
OMG THANK YOU SO MUCH, i accidently bricked after a Node remove the whole System... But for now its working! THAAAANK You verrrrry much!
Thank you, looking at your video made me look other ways and i was able to recover all my VM after a big HA Cluster problem, also will take your advice and from now on PROXMOX on separate disk, so can be rebuild with ease. Thank you again
Hello thanks so much for posting this troubleshooting video. I have a situation currently and I know the answer is in the video but I’m not sure the exact method for my particular issue.
I have an issue in proxmox with HA. I had a node move to another host migrate, but the disc volume did not go with it. I shared the local ZFS drives as you should. These are also the ZFS Storage drives that proxmox was originally installed on Each node has the ZFS storage, but doesn’t have the data from the migrated node. Only the information from the VM that was created on that node. So, when I try to migrate it back to the original node, it says no link to original volume. I was in the midst of setting up HA but didn’t quite get to the storage configuration yet. I’m surprise the node even moved I don’t think it actually did complete because there was errors and errored out so I had to disable it. I tried to migrate it back, but could not same issue missing disk vol plus USB external connection. All data is intact. The disk is still on the original node, but the VM appears to be on another note. No back up or snapshots yet. I’m still working on that and i’m learning and quite new at this. I was in the process of setting up a three new cluster. Hi availability with cef storage. Thanks so much for any advice or assistance I would Much appreciate it. I’m stuck!
man, I am just newbie to Proxmox but thanks to your enormous work and tutorias on YT I feel much more confident - thank you bro! ;-)
A video like this is ridiculously helpful for the community! @17:34, leave it in!
Oops I didn’t notice that mistake. I’m glad this video is helpful for others.
I haven't needed this -yet-, but what a great resource to have available if needed!
Thanks for this very detailed and still unboring comprehensive video about Desaster Recovery.
I was wondering if restoring the pve host using the proxmox backup client is easier or even possible - or maybe not advisable.
You need to be paid more my good man.
Thank you!!!!
Really wish I had found this video a week ago. I was trying to restore a couple VMs after moving my proxmox installation to an nvme but only a couple of my VMs would boot. Luckily, I didn't fully commit to the new boot drive and just had to plug the original drives back in.
Going to try this in VMware workstation. don't understand having to have a ds to get your backups, on esxi you run a backup solution you can backup and then restore into the hypervisor. Will try work it out
This is the whole reason why I'm moving away from using OMV+docker. Love both but backups and restore suck. After dealing with a full OS failure following later with a ZFS failure I figured this is my best bet. Thank God for my borg backup.
currently not using PBS but have a schedule for VZ Dump Backups on a weekly basis. Saved me a lot of time when something needs to be restored.
I use the VM on a network share (unraid) and from time to time upon rebooting a vm or shutting it down, it wont get back online statying that the disk/image is missing. Restore always worked 100% but its definitly something weird on this behaviour.
Another way to eliminate the guesswork is to just restore the config file from backup and use the disk files from the old system. This way you may not have to restore terrabytes of data and a older version of the VM. You probably have to edit the config file but its easier than creating a empty VM.
Hi, any video guide or article guide to do this? thank you so much.
The setup I have is 2 x 1tb ssd mirrored in a micro PC with 5TB ext drive for VM backups. First question is for the failures you have in the video if I lost one drive does the proxmox continue on the other and just highlight one as faulty or does it need reboot or config to use the 2nd of the mirrored drives. Now on the VMs restore. You show the VMs in the local store but dont use them and just recover. Does this mean the VM files in the local need to be deleted first ? Last question. Is there a way to backup the config so I just reinstall proxmox, restore the proxmox config and create the mirror again and restore the VMs from backup ?
Great video - After I remap my containers conf files to the correct zfs paths ie subvol-100 etc. If I try to lxc-attach I get permission errors like "bash: /root/.bashrc: Permission denied". I also have X2Go setup on a few of these containers and for some reason I can't connect to them I suspect it's a permissions thing. In my case I solved by setting unprivileged to 0.
Another tip though if you don't know which ct was which ie 100,101,105 etc you can just cat "VMStorage/subvol-105-disk-0/etc/hostname" so you know you are remapping the correct container to the correct name
Hmmm I followed your steps for VMs on ZFS but it's not so much an issue that I'm putting in a new OS and mounting everything as much as my ZFS01/VMs location disappeared in Proxmox and when I re-add it is not finding the VMs like your system has done as they are missing in the directory entirely.
The individuals ZFS01/isos and the /ZFS01/backups directories in Proxmox are fine. The ZFS system is fine it's just the virtual machine directory was gone in Proxmox, not on the server itself. So I re-added the pathway of /ZFS01/VMs and nothing. I am on the server via shell and looking for the mountpoint and ZFS01/VMs is there. I go inside and I do not see the disks at all. when I use zfs list it says there are 3 VM disks with no mountpoint. Each titled ZFS01/VMs/vm-101-disk-0 and iterations with the different 3 numbers. The ZFS01/VMs directory has a mount point so I'm expecting the VMs inside the directory do not need their own mountpoints.
I made another VM and the disk was placed inside the /ZFS01/VMs/images directory.
Awesome video. Haven't researched this but i wonder if you can backup the pve host itself.. Ie all the disk and vm configs
There is no good way to backup a pve host. The proxmox suggest solution if a host fails is to do a clean install and restore backups. This makes sense with a large cluster but with a single node a full system restore could save a good amount of time.
Thanks for this. 🙂
I have periodic VM backups via Datacenter-->Backup. The occasional restore has so far always worked.
Maybe I missed it, but if your hardware dies how do you restore the Proxmox hypervisor itself on new hardware? What do you use for backup, maybe Veeam? Or just start the new HW with a fresh Proxmox install?
Any decent solution relies on at least a second independent backup server, and the easiest, most reliable solution for Proxmox is to install Proxmox Backup Server on that 2nd host so all your VM/CTs are backed up to separate hardware. If the worst happens, then reinstall Proxmox PVE on the original or new hardware and copy the VM/CTs from the PBS backup machine back to your freshly installed PVE machine. The ideal solution is 7 machines, 5 in a PVE cluster with Ceph storage and 2 running PBS, with the 2nd PBS machine remotely syncing from the first PBS machine.
It is not necessary or "better/safer" to have a separate boot disk for Proxmox. If you install Proxmox on a ZFS Raid+ then if you do lose a drive you can still boot to the raid array, albeit in degraded mode, and examine the problem from within the same OS and hardware, and with the local VM/CTs still intact. Having to wipe and reinstall the OS and then also reinstall VM/CTs is far more of a burden than relying on the kernel and OS installed on each drive in a ZFS raid. I'm not sure of the situation when using LVM, as I've never used it. Try it. Install Proxmox on two drives in a mirror, then disconnect/destroy one of them and see if you can still boot up on the other one.
Hi, thanks for the info. Quick question: I can't log in to proxmox on my laptop even if I type in properly my login and password. Is there an option to remove proxmox from the mini pc server and restore windows?
You can install windows from the installer to have a fresh copy of windows. If you have a backup of the Windows install you can also restore that.
Does it open the web ui for Proxmox in the browser?
I'm interesting in seeing if the development of Proxmox accelerates now that Broadcom is officially raw dogging VMWare in the asshole.
Great video! Thanks for all the examples.
always love ur video, thankss for sharing
cool video but, how to do it when you have no zfs?
i am just trying to get the darng files off that old drive, and i dont wand to study it engineer to do so
How is the drive formatted? If its LVM you should be able to see the virtual disk images as block devices and copy them to a file on a new system. If there files on a drive, you can mount the drive and copy over the files.
@@ElectronicsWizardrywow, thanks for the quick answer, it was LVM, but when the mounting stuff starts I have no clue what to do. The principle is very understandable, but what are the commands? Asking google didn't help me much either. After hours of frustration, I just decided to start over, giving me days of work for free.
Thank god I use unraid, so I just had to import the drives / USB stick and everything was there, the other vms are binned tho
I can definitely not recommend proxmox to anyone besides RUclipsrs and infrastructure engineers
it is just too bare bones and unuserfriendly to actually request founding.
Thanks for this usefull vidéo y saved me
Good day sir. My proxmox server died., Well the boot disk died. I have rebuilt and reinstalled proxmox os onto a new raid z1. I'm now trying to import the zfs raid z1 where I had all my VMs and such stored. I have imported the zfs disk but have not made it any further. Is there anyone who can help me out?
Thanks
Once the drive was imported it shows up under zpool list right? If that’s the case you probably have to add the storage to proxmox and use the same zfs path that you used when the zpool was used before.
@@ElectronicsWizardry So I made it as far as adding the storage and shows up in proxmox as oldszfs. I then used the command: the cd /dev/zvol/vme to see the different virtual disks I have. I'm stuck at the part of creating a new virtual machine and adding existing vmdisk to boot from. I did not ssh into my server, instead I used the shell interface through the proxmox web GUI
I'm a bit confused about the lack of ability to back up the host itself (the system image and/or configuration). The official position of Proxmox team is "just reinstall and copy over some folders" is not inspiring. I wish someone come up with a robust and an easy solution to back up the host.
ProxMox team did say they're looking into adding this feature into the PBS. For now there are scripts that you can use to back up the host.
@@Darkk6969 Why do they plan to add it PBS only? At the end of the day, how to restore quickly the PBS itself, if suddenly it goes down?
All kinds of computer appliances offer, at least, minimal config backup from the GUI to a fixed or arbitrary location.
do you have blog Proxmox VMs shutting down unexpectedly
I do see this happen with Windows 11 and not sure why.
Thanks!
life saver!!!!!
my proxmox is on btrfs file system and running several vms and containers on it but current the host is not booting up and sometime it works vi tahe terminal but slow and cant start the vms,how can i backup when it comes up via terminal
I'm nost sure if you have anouther system or other drives, but in this situation, I'd fire up a new install on different drives, then plug your BTRFS drives in. Then you can probalby copy off the vm image files to the new system, and create vms that use those image files.
Love you videos, but this one has so distorted audio - like your mic's muffled and is clipping like crazy..
Hey,
Please don't create so many skips in your video. I wanted to see how you set up your Proxmox backup server.
Ah yes.
One of the reasons I STILL don’t use Proxmox for PRODUCTION!
Say what you must about ESXi, but still rules when it comes to easy, quick recovery!
Thank you for this great video!
It would appear then, that the VZbackups will ALSO store the config information along with it, vs. the disk files/images alone, which will NOT have the config info in it.
Would that be correct?
Sounds to me like that if I am going to be manually backing up the disk images, that I would need to back up the /etc/pve/qemu-server/*.conf files and/or /etc/pve/lxc/*.conf files along with it.
(Or have an speadsheet file that tracks the VM/CT ID along with the associated details so that if I need to recreate it and/or restore it from the disk images, I would now know config I had when I originally created said VM/CTs.)
What do you do for an UEFI restore?
Is it basically the same process where you would create a dummy UEFI VM and then edit the config file to point it to the UEFI boot disk and the actual data/OS disk?
Thanks.
I have problem with proxmox, after typing "systemctl disable pve-daily-update.timer" my VMs stop after minutes, how can I fix this
Taking a look at that service, I don't see a reason for it to stop the VMs from running. Do you have any logs that would show how the vms were shutdown and what may have caused that? Taking a look at the files for the service I don't see a reason why vms would be stopped. Does running /usr/bin/pveupdate also stop the vms? The pve-daily-update.timer runs that script, so that may be able to tell you if its the update script or systemd.