As sys admins we often do not have the time to play with settings and experiment; hence, why we often end up with over-spec-ed machines just to be sure to be sure. Thank you for your work and analysis, and more so sharing it. Thanks you.
Also try to starve the VM of RAM as much as possible to be able to use the disk cache on the hypervisor. This will help performance a lot as the cache is shared between all the VMs
I think the item you sort of glossed over was i/o and implementation of the pci-e based ssd - those units from intel are absolute units and can significantly boost performance thereby mitigating having to go to swap in the first place, also caching can be a big performance boost either with zfs or bcache or a ram drive, also just using bigger arrays/raid10 and stuff like that could end up helping - if you did another video and looked at effects of disk subsytems and the ultimate setup for max perf that would be helpful and put some more metrics on the board - swap is just one component
Yea probably skimmed the 900p being nearly the best swap device out there. I also had a quick graph with a 850 evo as well, and while it was worse, it wasn't that much worse. The effect of ram on caching is interesting and I have some ideas for scripts to test different caching systems, but its hard to test caching is all about trying to predict user workloads, and it can be hard to get consistent numbers that are applicable to the real world.
Some experience from work: Don't ever overcommit, not good you'll have a bad time and if you have to do it, use KSM, the lowest swapiness and try to have as much of the same OSes running in the same version as possible. Then you might be able to do 1.1x -1.3x overcommit safely. But if you don't have to disable swap and never think of it again. It will help you a lot especially on slower drives when live migrating...
I have never been able to understand swap in Proxmox. I have a cluster of 3 identical Intel NUCs with 64GB RAM and and they are all below 50% mem usage, but still one node uses 30% swap for some reason - even when it is reporting only using 27% RAM and no VMs are reporting more than 70% mem usage either. Swapiness is 60 on all. No ZFS. Very insightful video - as always.
KSM what are your hints to use it. I have Ryzen 3900, 48GB of ram with KSM active from 50% of ram usage and it seems to be going fantastic job. Please share your view on the topic.
8:11 As it's the kernel that does OOM kills - it's not even guaranteed that the Proxmox web service survives, right? I think your zRAM tests are flawed measuring ffmpegs speed only - try swapping windows/programs in a Windows VM and see how the latency feels..
As far as the Proxmox services getting killed by OOM, It looks like linux has a score that the OOM uses to determine what to kill first, and from a quick look at a system the vms would get killed before all the proxmox services from my quick look. I agree my zram tests were flawed, but the issue is snappyness is hard to measure well. Let me know if you know a good way to measure snappyness and I'll give it a run, but since I couldn't really think of a better test I stuck with the ffmpeg encode. I'm sure zram has its uses as its the default on many distros.
what would you recommend for such issue that appeared suddenly after I rebooted Home Assistant. The same issue i get when I reinstall Home Assistant and I am getting this running when booting for even few hours doing nothing... "A start job is running for HAOS swap" My proxmox is on wyse5070 with 250gb disk ssd, 20gb ram . for home assistant i used 1x4 cpu, 10 gb ram, 32 gb disk... and i also at some point changed swap from 8gb to 12gb as i thought i need more but no change...
Would it be possible to set up zRam for installing operating systems on it. . I read in proxmox forum that it's possible but I don't understand how to make it work
Like as a boot drive? Sure you can use that for the boot drive for the VM with a bit of configuration. In general RAM drives aren't worth it for VM boot, but I can make a video looking into it if your interested.
11:42 I would not recommend a low swappiness setting and waiting for swap being used, keep it higher, because you are reducing performance because you loose things like file system cache.
My mindset was that a low swappyness will be best if you normally have enough RAM, and want to make sure there aren't performance hits if a part of a VM gets swapped out and then needed later on. The swap would then keep the OOM killer from taking out VMs, and be an indicator if more RAM is needed. If you plan on using swap, then a higher swappiness value would be better as its proactively swapping out unused ram instead of waiting for ram usage to be high. This really depends on the usecase of the system and VMs. I often seen Proxmox servers where RAM usage isn't that high, so my thought was avoid performance hits by having parts of VMs in swap and the smaller disk cache shoudn't hurt performance that much, but again really depends on the use case. Hope this explains some of my thoughts in the video.
@@ElectronicsWizardry "If you plan on using swap, then a lower swappiness value would be better" OK, agree. I guess I should have thought about the title of the video again. 🙂
Its not just on Proxmox this doesn't get talked about.... the subject isn't really talked about in all Linux based systems... and probably not even in other systems either. All to often you see "just set it at the same amount or double you real RAM".
So...the thing with swap REALLY depends on what you're doing. On my main Proxmox server at home (which has 256 GB of RAM), I think that I am running something like 17 LXC containers (with almost all of them being Ubuntu), 1 WinXP VM, 1 Win7 VM, 2 Win10 VMs, and 2 Ubuntu VMs. It uses somewhere between 150-180 GB of RAM nominally. That SHOULD leave me with ~56 GB of RAM free, right? And yet, despite this, my Proxmox server WILL swap out before using the remaining ~56 GB of RAM. Always. Without fail. swappiness is set to 500, which means that it SHOULDN'T swap out to disk unless it ABSOLUTELY has to. Between ZFS ARC and virtio-fs, some combination of things is causing the system where it will NEVER be allowed to use the remaining 56 GB of free (and available) RAM. Moral of the story: RAM, for VMs and CTs ended up being the most important hardware choice when picking a virtualisation server.
I'd check that 500 swappiness value. I think swappiness should max out at 200. And larger values typically mean swap will be used more. I'm curious how big you have your ARC set to.
@@ElectronicsWizardry *edit* I stand corrected. I just checked my system (couldn't do it earlier whilst I was at work). Swappiness is set to 10 on my main Proxmox "do it all" server. It was vm.vfs_cache_pressure that I have set to 500. "vfs_cache_pressure This percentage value controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a “fair” rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will never reclaim dentries and inodes due to memory pressure and this can easily lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes. Increasing vfs_cache_pressure significantly beyond 100 may have negative performance impact. Reclaim code needs to take various locks to find freeable directory and inode objects. With vfs_cache_pressure=1000, it will look for ten times more freeable objects than there are." (Source: www.kernel.org/doc/html/v5.4/admin-guide/sysctl/vm.html) Either way, even with a vm.swappiness=10, my system still swaps out like crazy despite the fact that I have 56 GB of RAM free.
As sys admins we often do not have the time to play with settings and experiment; hence, why we often end up with over-spec-ed machines just to be sure to be sure. Thank you for your work and analysis, and more so sharing it. Thanks you.
Love your videos. You deep dive on technical details that no one else seems to.
Loving all your PVE and PBS videos! Great Detail, great information. Thanks so much.
Excellent analysis! I was surprised by the results.
Thanks!
Very interesting, especially your findings on swap set to 0 or 1.
Thanks for the video!
It was definitely interesting vid. Nice to get some info about how things affect.
👍 One of the best videos I've seen in a while anywhere. Amazing work.
Wow, I've never heard of zram. Thank you.
Also try to starve the VM of RAM as much as possible to be able to use the disk cache on the hypervisor. This will help performance a lot as the cache is shared between all the VMs
08:11 ..good to know. Thanks
I think the item you sort of glossed over was i/o and implementation of the pci-e based ssd - those units from intel are absolute units and can significantly boost performance thereby mitigating having to go to swap in the first place, also caching can be a big performance boost either with zfs or bcache or a ram drive, also just using bigger arrays/raid10 and stuff like that could end up helping - if you did another video and looked at effects of disk subsytems and the ultimate setup for max perf that would be helpful and put some more metrics on the board - swap is just one component
Yea probably skimmed the 900p being nearly the best swap device out there. I also had a quick graph with a 850 evo as well, and while it was worse, it wasn't that much worse.
The effect of ram on caching is interesting and I have some ideas for scripts to test different caching systems, but its hard to test caching is all about trying to predict user workloads, and it can be hard to get consistent numbers that are applicable to the real world.
Some experience from work: Don't ever overcommit, not good you'll have a bad time and if you have to do it, use KSM, the lowest swapiness and try to have as much of the same OSes running in the same version as possible. Then you might be able to do 1.1x -1.3x overcommit safely. But if you don't have to disable swap and never think of it again. It will help you a lot especially on slower drives when live migrating...
I think mongo won't start in lxc without overcommit sysctl
I have never been able to understand swap in Proxmox. I have a cluster of 3 identical Intel NUCs with 64GB RAM and and they are all below 50% mem usage, but still one node uses 30% swap for some reason - even when it is reporting only using 27% RAM and no VMs are reporting more than 70% mem usage either. Swapiness is 60 on all. No ZFS. Very insightful video - as always.
KSM what are your hints to use it. I have Ryzen 3900, 48GB of ram with KSM active from 50% of ram usage and it seems to be going fantastic job. Please share your view on the topic.
Thanks for the idea. I should look more into KSM sharing and when/how it works.
8:11 As it's the kernel that does OOM kills - it's not even guaranteed that the Proxmox web service survives, right? I think your zRAM tests are flawed measuring ffmpegs speed only - try swapping windows/programs in a Windows VM and see how the latency feels..
As far as the Proxmox services getting killed by OOM, It looks like linux has a score that the OOM uses to determine what to kill first, and from a quick look at a system the vms would get killed before all the proxmox services from my quick look.
I agree my zram tests were flawed, but the issue is snappyness is hard to measure well. Let me know if you know a good way to measure snappyness and I'll give it a run, but since I couldn't really think of a better test I stuck with the ffmpeg encode. I'm sure zram has its uses as its the default on many distros.
what would you recommend for such issue that appeared suddenly after I rebooted Home Assistant. The same issue i get when I reinstall Home Assistant and I am getting this running when booting for even few hours doing nothing...
"A start job is running for HAOS swap"
My proxmox is on wyse5070 with 250gb disk ssd, 20gb ram . for home assistant i used 1x4 cpu, 10 gb ram, 32 gb disk... and i also at some point changed swap from 8gb to 12gb as i thought i need more but no change...
cool
Would it be possible to set up zRam for installing operating systems on it. .
I read in proxmox forum that it's possible but I don't understand how to make it work
Like as a boot drive? Sure you can use that for the boot drive for the VM with a bit of configuration. In general RAM drives aren't worth it for VM boot, but I can make a video looking into it if your interested.
8:20 it's not proxmox which kills it, it's the Linux kernel.
11:42 I would not recommend a low swappiness setting and waiting for swap being used, keep it higher, because you are reducing performance because you loose things like file system cache.
My mindset was that a low swappyness will be best if you normally have enough RAM, and want to make sure there aren't performance hits if a part of a VM gets swapped out and then needed later on. The swap would then keep the OOM killer from taking out VMs, and be an indicator if more RAM is needed. If you plan on using swap, then a higher swappiness value would be better as its proactively swapping out unused ram instead of waiting for ram usage to be high. This really depends on the usecase of the system and VMs. I often seen Proxmox servers where RAM usage isn't that high, so my thought was avoid performance hits by having parts of VMs in swap and the smaller disk cache shoudn't hurt performance that much, but again really depends on the use case. Hope this explains some of my thoughts in the video.
@@ElectronicsWizardry "If you plan on using swap, then a lower swappiness value would be better" OK, agree. I guess I should have thought about the title of the video again. 🙂
Oops that should have been higher value to proactively swap out ram. I edited the earlier comment.
Its not just on Proxmox this doesn't get talked about.... the subject isn't really talked about in all Linux based systems... and probably not even in other systems either.
All to often you see "just set it at the same amount or double you real RAM".
So...the thing with swap REALLY depends on what you're doing.
On my main Proxmox server at home (which has 256 GB of RAM), I think that I am running something like 17 LXC containers (with almost all of them being Ubuntu), 1 WinXP VM, 1 Win7 VM, 2 Win10 VMs, and 2 Ubuntu VMs.
It uses somewhere between 150-180 GB of RAM nominally.
That SHOULD leave me with ~56 GB of RAM free, right?
And yet, despite this, my Proxmox server WILL swap out before using the remaining ~56 GB of RAM.
Always.
Without fail.
swappiness is set to 500, which means that it SHOULDN'T swap out to disk unless it ABSOLUTELY has to.
Between ZFS ARC and virtio-fs, some combination of things is causing the system where it will NEVER be allowed to use the remaining 56 GB of free (and available) RAM.
Moral of the story: RAM, for VMs and CTs ended up being the most important hardware choice when picking a virtualisation server.
I'd check that 500 swappiness value. I think swappiness should max out at 200. And larger values typically mean swap will be used more.
I'm curious how big you have your ARC set to.
@@ElectronicsWizardry
*edit*
I stand corrected.
I just checked my system (couldn't do it earlier whilst I was at work).
Swappiness is set to 10 on my main Proxmox "do it all" server.
It was vm.vfs_cache_pressure that I have set to 500.
"vfs_cache_pressure
This percentage value controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects.
At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a “fair” rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will never reclaim dentries and inodes due to memory pressure and this can easily lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.
Increasing vfs_cache_pressure significantly beyond 100 may have negative performance impact. Reclaim code needs to take various locks to find freeable directory and inode objects. With vfs_cache_pressure=1000, it will look for ten times more freeable objects than there are."
(Source: www.kernel.org/doc/html/v5.4/admin-guide/sysctl/vm.html)
Either way, even with a vm.swappiness=10, my system still swaps out like crazy despite the fact that I have 56 GB of RAM free.
swappiness