I'm a sysadmin at a decently sized European university. While our large faculty server is stuck with ESXi for the near future (which I'm hoping to change), our smaller servers all run PVE. This includes our storage/backup/mass-deployment server, a cluster of several machines that deal with high compute power for AI and machine learning, a cluster that runs *several hundred* concurrent VMs for networking and white hat hacking exercises, and smaller roles too numerous to mention. I've never had a single "I wish PVE had x..." moment. The only thing that's a bit difficult to work with is the user and permission management, but it's not a deal-breaker. As for LXCs - I'd rather have them than not. My home server has very little memory, and having my services run inside Debian or Alpine LXCs that each consume ~20 megabytes of RAM is pretty baller.
Also LXC could support the same almost-everything live migration if someone put in the work, because LXD supports it using CRIU and it works and uses the same APIs.
I'm basically the same at my company (which is ~100 employees but growing decently mostly at a main location but we have two satellite locations with 10-20 employees each). I put vcenter/esxi at the main office but when we expanded to the other locations and it looked like vmware might get sold I gambled on proxmox at those locations (I was using the free version at home in the home lab) and I haven't looked back. By the end of this year the main area will be transferred off esxi/vcenter to proxmox as well. Having the free version only minorly annoying reminded me at the right time and has turned into a few thousand a year in licence fees for them from my day job so it's basically the best advertising program they can have. The only thing I miss is the multi location management through a single pane you can do with vcenter and I'm hoping sometime in 2025 they release that feature, there is talk of them working on it on the forum on roughly that timeline.
I am a teacher at high school in Norway. My students that were new to virtualization, they prefer Proxmox over VMware when it comes to user friendliness. And LXC containers are very useful, when you're low on resources.
Hello, @Wampa842 , @barfnelson5967 , @2GuysTek I'm going to start building my minilab (especially for AI) and was looking at PVE. My first challange is to port a "custom" vmdk to PVE (to free up a server). The thing is that this one has: 1.bitlocker enabled (windows 10) 2.(virtual)TPM device present (Virtual Machine Settings->Hardware->Trusted Platform Module=Present) 3.VM is fully encrypted (Virtual Machine Settings->Options->Access Control=Encrypted) My question(s): a)Can I migrate it as is / how ? b) if not,do I really need to remove/disable all 3,2,1 ? Thanks!
8:05 "all management needs to be done via web gui" completely incorrect. Proxmox makes virtually everything and more available via the command line. "pct" program and "pve" prefixed commands, tons of them. You can enter hosts via the "pct enter hostID" etc in /etc/pve/ all machines are available for config edit as well
Agreed, it's not as neat as a menu where any ape can just navigate and press enter, but the functionality exists for sure. There are loads of scripts that you run in the terminal that does entire VM or CT setups.
@@watvannou Yea you can tell he didn't do much research and just spat out his video to get views in this high traffic time. The docs etc are filled with command line stuff to use, yet he claims it "NEEDS" to be done via GUI lol. Unfortunately (for those apes) there are still many things not even available in the GUI so sometimes you have to get into the cli and files. One day it'll probably all be in GUI though.
Sure you *can* go into the shell and manage things directly from there, but his perspective is talking about for general use and deployment. Would I trust a L1 or L2 tech to go down to the machine and navigate through the menu of XCP-ng to start a VM? Sure, absolutely. Would I trust one of them to go log into a shell and start running commands that directly manage the entire VM structure and config of the house without prior training? Probably not, no.
@@1armbikerI mean....how is, "qm start (insert VM machine number here)" something one of your tech's wouldn't be able to handle? Train them on the dark arts of --help. The comment section of this video has suddenly made a lot of past interactions make sense...lol
"host" copies the motherboard CPU model. You have to be careful if you have different CPU models in a Proxmox Cluster and then do live migration. Therefore, the default version assigned is intended to secure against issues. If every CPU in the cluster is the same, then you can keep it at Host and in some cases (newest CPU's) you'll experience better performance 😊
What proxmox devs are doing is little different, it is a place where all the opensource virtualization technology meets. Yes, it might not be general user intiuitive, but it provides well on core virtualization. And as a linux nerd I love it.
tbh proxmox's interface seems far *_far_* better to me. Yes, there is a lot, but it's all very clearly formatted and hierarchically displayed. Being able to see everything you can do means you'll be able to sumble across features that solve your problems, it keeps things organized, and it keeps things simple. The Windows route of submenus that open other submenus that do the job of that other submenu but are actually different and can also open this other submenu which is technically deprecated by that submenu over there but actually still lets you do this thing a lot easier, just isn't good. The only reason why people don't complain is they're used to it, but being used to something doesn't mean it isn't bad UX. FFS I literally had to google how to even empty my trash on windows because my windows partition was low on space and there isn't a clear option to do it anywhere. It takes several submenus deep in the system settings window to empty your trash. If your UI is so convoluted that I need to google how to do basic tasks, don't bother making a UI at all, it's more work than if you just made me google the commands to do the same thing. I haven't done much virtualization, (I've only really used virt-manager for local-virtualization, though I've always kept an eye on higher scale virtualization since I've been planning to build a bigger homelab long term) but without any experience, I'm picking the proxmox UI without question. I don't need to look for anything, it's just there, I can see it, it's there. If I have something I need to do I can know where it is without having to memorize it because there is a simple and consistent design philosophy.
LXC containers are one of the main reasons I use Proxmox. They are extremely lightweight and fulfill all my requirements in my homelab. Compared to traditional VMs (Virtual Machines), their resource footprint is significantly lighter. I have only 16 threads available on my CPU, which means I need to be particularly cautious with thread reservation. LXC containers, on the other hand, offer great flexibility in terms of resource usage. I really love Proxmox and plan to continue using it in the future.
Re proxmox management from the console: The gui available via the web browser is only a think overlay over the command line tools, available from the command prompt. All VM and container related functionality is available from the CLI, and in some specialized cases, use of the CLI is still required to complete the task. The big advantage of the GUI is that it is much easier to use. The "short" version of the help text listing only available command line options for "qm" (the VM management tool) is several hundred lines long, spanning many pages. Super intimidating for a beginner, but a life-saver for an advanced user, who wants to automate routine tasks.
I mentioned in the video something to the effect of how you can do everything you need to do from the command line, but the point is, that makes Proxmox less of an option for a homelabber or business that doesn't have that deep knowledge in linux command line. I think that when a company is considering their options in terms of their future replacement for VMware, they're going to have to take into account their IT staff and engineers and how much retraining/learning they're going to have to do, and I think for Proxmox, that's a concern.
@@2GuysTek8:04 You’re saying all management from the host has to happen from the GUI, that’s not at all true. Anything you could do from the GUI is possible from that prompt. You then followed it by saying that kind of access is useful when you have no network connectivity, but again, you have far more access to modify the HV config than you’re letting on. I’m fine with giving ESXi the win for ease of use, but the phrasing is bad dude. Completely blowing past the amount of power in that console because admins don’t know their tools is a cop out.
@@bigjaydogg3384 I don't think the phrasing is bad. He has a point. Perhaps what Proxmox can implement is a CLI menu similar to TrueNAS whether you can drop into Bash
@@zpariharThats the thing, you don’t need a menu option because you’re already there. Thats why the video is so wrong, its not that the prompt it drops you into is incapable of anything, its that the prompt is so open ended simple tasks require at least some level of Linux knowledge.
@@bigjaydogg3384missing the point. "can" do it is different to "how" do it. if i run a small business and nobody in it knows linux nor proxmox, it doesn't matter how much you tout that it can be done if we don't know how to do it, especially if it's an emergency and we need to do something from the console. if i want a linux expert, i need to also pay their salary. i've personally helped several small businesses without any in-house expert team to do these tasks and having a more layman useable interface could save them big bucks hiring a technician to do emergency maintenance on an hourly fee. how much work would it really be to have a TUI where you can change hostname and network options? for example, they could use NetworkManager and simply call nmtui for the user, which is a massive life saver.
I'm still on the fence as to which hypervisor to migrate to, so I appreciate the videos. However, I wanted to say that I appreciate the presentation style of these comparison videos. I've watched both, and they are fantastic examples of how to do an X vs Y comparison. Nice work!
Tom Lawrence (Lawrence Systems on RUclips) has a new video on XCP-ng and Xenorchestra out today. Might take a look to see the Proxmox vs. XCP-ng differences (even if his video doesn't really demo that per se). I use Proxmox on "small systems," and XCP-ng on the big iron.
Been swinging between Proxmox and XCP-ng on my test system. XCP-ng feels more of an 'Enterprise-ready' but Proxmox is pretty strong and powerful out the box without over complicating things. I think for me the built-in LXC is going to make Proxmox the homelab solution. I'd probably want to do XCP-ng if we had to drop ESX at work though.
I can understand the the immediate thought of Docker and kubernetes when it comes to containers, but remember that the container support in Proxmox actually started with support for OpenVZ containers, and moved to LXC even before docker even existed. Today I'm not sure that LXC makes that much sense anymore, but go back 10 years, and LXC provided significantly reduced overhead in contrast to a full VM. I've been using Proxmox since version 1.0, and with 3GB of ram on a server, shaving the memory usage by a few 100mb actually made a difference. As for the upgrade, it's actually not that hard to do. In fact I find it more unnerving to upgrade a vcenter, especially if NSX is involved.... Keeping my hands off the hypervisor and don't changing anything I've never had any issues upgrading a Proxmox host. That's not something I can say about upgrades of ESXi hosts...
I still do not fully understand the upgrade process on standalone ESXi, like why do I need do download some sort of file to start the update? Can't I just push a button or run a command?
Thank you for making these videos! My organization was just talking about this very thing, and this will help me explain options to my management team.
thank for the video. As a linux nerd I say yes to a full featured gui with all options but a fallback to a shell for scripting helpfull tools is a great thing. Converting VM to proxmox works with simple scripts but every Vm should be testet after a transfer.
Good video my dude. I would like to see a follow up video on clustering, vim networking, vm storage, virtual disks, migration, fault tolerence and "DRS".
I'm interested in this as well with Fault Tolerance considering he didn't list it, only that it was feature parity was complete between the two and I do not think that is correct.
would love comparisons with Proxmox and XPG-ng vs Virtual Center. Our VMware bill went from $132k/year to 1.225 mil/year and we need to find some alternative.
LXC have a lot of uses. It's basically for services that would best be run on bare metal, but you don't want to pollute host OS with it (samba server, nfs, ..)
I started using proxmox last year and I have to say that it is clearly very good I agree concerning the anoying repository I must also add that to have the possibility to make snapshots of your vm you have to pay attention to the dis format of your VMs however up to now it is working like a charm
I haven´t made any experience on EXI because as a non professional user I don´t want to pay monthly for a Virtualization. I am running a mac mini but sometimes I have to use a Windows machine and I love Linux. Parallels for mac is so expensive and so a bought a mini PC for about 200 € and can now virtualize Windows and several Linux on that machine by using proxmox. Although I am 71 years old I was able to manage all this because there is a great community supporting proxmox users. And you will find big help on RUclips also. I think proxmox is a good software and free to use.
You know that you can do basically anything in pvr from the console via command line once you're logged in? If you want you can avoid the gui entirely and perform everything the gui can plus stuff the gui cannot do in the terminal? Yeah yeah you know it. But esxi loses. You can't do anything in esxi there even if you want. Let's say network fails, you're lost. You can't even check vm status... Sorry you're absolutely wrong
I moved from Esx to HyperV then to Proxmox a few years ago. VM migration was a nightmare. However I’ve not regretted moving to Proxmox. It works well. Particularly good when you set up a Ceph cluster. There are easy ways to disable the nagging. I have a script that removes the nagging that executes on every boot. Not seen the nag in a very long time. The LXC containers don’t live migrate but they do migrate. They just shut down and start again on the destination host. They’re very useful for lightweight tasks such as running ddclient, letsencrypt, cloudflared etc. Generally, I’m glad I moved to Proxmox. Free HA, good for Linux and Windows VMs, an acceptable built-in VM backup system, Ceph for cluster storage. What’s not to like?
Should have really talked more about the ability to GPU and other pass through capabilities as well without paying an arm an a leg for Nvidia licensing. It's really nice for Plex or Jellyfin transcoding and running cheaper LLM/AI solutions. Also the Proxmox backup solution is truly amazing and probably needs to be talked about more. It doesn't just give you VM backups but it also does file backups as well. For me this was such a better and easier solution than going with Commvault or Veeam.
Ick. CommVault. What a pile of steaming crap that thing is. Thank God we got rid of that a few years ago. We went to Veeam, but someone else manages that, so I don't know how good or bad that is. But, yeah, backup capabilities being built into the platform vs 3rd party is so practical.
I'm sure you will hear linux ops folks talking about how great the cmd line is for admin'ing... and I don't disagree... IF you're doing it frequently enough to remember how to do everything. For the rest of us who don't admin linux machines, we waste a LOT of time looking up commands and that goes away with a basic (even text based like with esxi) ui. I just started playing with Proxmox as a potential migration path from vsphere for my roughly 500 core cluster, the new vGPU non-prod release gives me hope that by the time I need to migrate there will be production gpu support which would make proxmox fully capable of replacing my vsphere e+ environment. Good overview but again I'd like a deeper dive in to advanced features like backup... upgrade workflow, software defined network and so on. oh and yea I DID assume containers meant app containers too.
@@brodriguez11000 Hosts need more than an api... they need a capable diagnostics interface (that should allow admins to manage the local machine). An api only host would be a problem in certain disaster recovery scenarios. Could be console based, ssh or web... but should not rely on external custom software to work. (everything needed on the box).
As you just stated, the reason Proxmox CLI is minimal is the fact the CLI is debian with KVM enabled. I have Ubunutu servers with KVM installed making them TYPE1 hyperivosors and now I just manager them with their gui tools like Cockpit or Virt-Manager. yes I can get in to the hosts via ssh but I don't run additional services on the local host at this point and just manage VM's runing said services. KVM was designed to be open source and full customizable to your liking.
Ubuntu microcloud, I think you are referring to this one the latest version the install fails, microovn doesn’t install and microceph bootstrap fails and you need to install all manually. Also the lxd project leader just left canonical and forked lxd into incus. Microstack bootstrap process fails more than it installs successfully.
I migrated ESXi VMs to Proxmox using the OVFtool from VMware. It was quite easy and where are guides out there. Having used VMware (ESX) since it was running on top of Linux (ya, I'm old) professionally and personally, I've been quite happy with the switch to Proxmox. I switched when ESXi 7 came out and made it harder to run on the same hardware as 6.5 did. Learn Linux. You'll be glad you did.
Yeah, I upgraded via scripts and my machine ran fine. Then did a new install and was told my CPU had aged out. I shut it off and have used a KVM based machine ever since but am interested in ProxMox :) Large customers I work with are moving away from VMware as FAST as they can and the renewal costs I'm hearing about are eye watering.
I believe the reason Proxmox has the VMware features, the pvscsi, vmxnet3, etc is because if installed Vmware Player on a Linux host, it has to compile kernel modules. So they may have the modules precompiled and ready, not to mention there is Open-VM-Tools, which is an open source version of VMware Guest Additions for Linux. When the Open VM Tools became standard, VMware pretty much dropped support for their own proprietary guest additions for linux, especially for newer kernels. Maybe having these for guests is for compatibility should you wish to migrate from ESXi or any other VMware product.
funny - someone running a linux based virtualization server (cluster) is not familiar with linux, hmm. noone needs lxc containers? ok, we are not on the same page here. 😂
There are many use cases. Not all require expertise in everything used across them. Don't assume someone that's an expert at Salesforce, will know D365. Just because it's a CRM. Or an expert on Debian, but gosh darn they just can't get RHEL rock'n the same. Deep Linux knowledge is not a given for DevOps/SysAdmins, or any other IT staff.
I just wiped my drives last night from Vmware ESXI 7u03 after using it since 2022. I ended up going with proxmox for now because due to having way more support in terms of DevOps tools such as Ansible and Terraform as well as installing nested hypervisors. I tested xcp-ng but the support base is too small esp for DevOps and API automation stuff at the moment. Also the RAM usage is slightly higher.
LXC containers are a lot more light weight than full vm's, they start up in ms and use less than 100 mb of ram. If your usecase does not require extreme security then they're great.
I absolutely agree with all of the comments on the lxc containers in proxmox. The first time I started my proxmox virtual server, I started with VMs. However, I realized that it takes more memory and resources to run the virtual machines, so I changed to creating lxc containers in proxmox and it uses very little resources when loading a lxc container. I only use VMs if it calls for using them in proxmox. I am now getting ready to run a second proxmox VM because I love that they added the backup server to backup virtual machines. however, I do want to try other VMs, such as XCP-NG with Xen orchestra since I am very familiar with citrix, and also SUSE harvester. I want try VMware ESXI, but I am glad I did not go further since they got rid of the free edition.
very useful video, thx. never used vmware, trying out pve now. i like to learn the under the covers troubleshooting, but agree they coulda shoulda provide a mgmt interface on host. i struggled to get the connection working to get the web ui, which required navigating local conf files to even get going. i'm wiser for it, but in a production setting i'd like a ui pointing me in the right direction to potentially save hours of downtime.
Video is ok, BUT you are missing so many things in this comparison. First: the feature set, you only say esxi can this, ok proxmox to. BUT you dont say that proxmox can do mush more the esxi even in higher tiers. (With vcenter) there is a complete backup solution already in there, for free. Than there is absolutely no hardware logdown to "certified" shity and expensive hardware, the supportcost + hanrdware are cheaper than the "certified" hardware alone and so on. There are so many point you have ignored wish is ready disapointing
You can still use ESXi, assuming you have kept the ISO's and your keys, which i have. I decided to give Proxmox a go last month an after being a VMware admin for 9 years i found it fairly easy to get to grips with it. There were a few things i had to google such as the ISO storage !. Personally i have found it very good and on comparable hardware im noticing lower CPU use and lower RAM use compared to ESXi running exactly the same VM's.
"But not all support engineers have deep Linux knowledge." No offense, but you can't really call yourself a support engineer in the 2020s if you don't even know (or are too dumb to google) how to set up networking in Linux. It's the OS that runs more than 95% of the top million websites and most modern technologies (especially everything regarding containerization) have it as a foundation. The point about the easier configuration on VMWare still holds though, that's a great convenience feature. But please people, learn your Linux basics. Setting up networking is not "deep Linux knowledge".
Proxmox console is poor?! No… the first requirement to adopt Proxmox is to have hard and strong skills in Linux/Unix environment. Other way… go to click based OS.
Not sure if this has been mentioned but you have multiple CLIs availabla: qm, pct, pvecm, etc, to launch, stop, enter, reboot, etc., vms, lcxs and managing the cluster. Also all the networking and virtual bridges (switches) can be managed from the linux terminal.
"not all support engineers have deeper linux knowledge" - lol so why you are calling them engineers? call them support technicians or better support specialists, as engineers/technicians should understand what is and how it is working under the hood of the used solution :D proxmox is just a gui on top of qemu/kvm and lxc (btw. that's a bit odd, as lxc is quite archaic right now and most recent solutions are based on OCI or directly on containerd)
lol you have very good points but come on it is free. Some of your complaints are petty though. By the way every IT department needs a linux nerd. Two is even better so that he or she does not get bored.
Another thing I'd like to say: You're comparing the ESXi GUI to the Proxmox GUI which is unfair in my opinion. You should have compared vCenter's GUI to the Proxmox GUI. And then, the complexity of both GUIs become much closer and therefore your point is invalid to a certain extent, at least in my opinion. If you wanted to truly compare the ESXi GUI to Proxmox, you should have at most only compared the Proxmox single host GUI to it, but that wouldn't make much sense because in Proxmox the single host GUI options are based on the Datacenter options. So the only valid comparison is vCenter vs Proxmox.
@@pepeshopping See some of don’t have time to spend weeks understanding some broken bullshit platform like Docker when they just want a CRM and small business applications on a small tower server
Well.. on the one hand you say you cannot expect everyone administering an Proxmox host to fix it using the command line, but on the other hand you say that editing the hosts file through the WebUI wasn't needed xD
Great video. Btw, LXC live migration is coming. Docker comes from LXC. LXC (and the previous "original" container technology OpenVZ was around before Docker was a concept
One thing I noticed in your XCP-ng comparison was the mention of the limitation of VM pausing during snapshot creation if memory is included in the snapshot. However I don't remember hearing about Proxmox snapshots in this video. Does it have the same VM pausing issue as XCP-ng for memory-inclusive snapshots? It looks like the snapshot capability is dependent on a number of factors such as the virtual disk format or the file system used to hold VM images (so you need to use QCow virtual disk files if your VM storage is networked using NFS or Samba), and that would have been good to mention.
I can't give a comprehensive answer but the short answer for taking a snapshot with memory. Yes, the VM state is paused for the duration of taking the snapshot.
You can both pause or stop to get all the memory in snapshots. Those limitations are the differences between block and file storage. Not due to proxmox.
thanks for the comparison, i noticed that with proxmox unlike esxi, you can't install a promox host to a proxmox cluster if that host has vm's on it.. even if they are shutdown.. the promox most MUST be clean.
I totally agree! If proxmox made the GUI a little easier, made the upgrade more seamless, and made it possible to migrate from vmware to proxmox smoothly, it would be a no brainer. But these points remains and a couple more you mentioned in the Video. Really hope to see these changes! Thanks for the video, good summary!
Just to add to this, Proxmox recently released a conversion tool to make migrating from VMware to Proxmox easier: news.itsfoss.com/proxmox-vmware-migration/
I´m not exactly sure how user an group management is in esxi, but have you noticed that you can also add other realms in proxmox like from LDAP/AD or even OIDC?
After 24 years with VMware, it’s time to move on. I started with VMware Workstation for Linux in 2000, before GSX and ESX 1. Broadcom screwed up everything that was great about VMware. I have a lot of clients concerned about what is going on with VMware, including corporations that are paying over $300k on licensing and support.I know for sure that some of them will migrate to something else before the end of the year.
LXC is vastly more useful for Linux than a full VM. Almost all things that would run in VM on ESX can be run in a super lightweight LXC container in Proxmox. You can run probably 10x as many LXC containers on the same hardware as full VMs. And they are managed almost identically to a VM. If you are not a Linux expert, and only have VMWare experience you probably have more to learn about LXC and how useful they are. LXC is what is used for most cloud 'shared' instances you get from hosting providers because they are vastly more efficient for Linux systems than hardware emulating VMs.
Hey man, i have to say your X-CPNG and ProxMox vs ESXi has been great and super informative. I appreciate you taking the time to do this.. I run ESXi 6.7 in home lab and, given the state of the company going forward, i am looking to change that. Have you done any videos on running ProxMox in mini PC's? I know it is pretty compatible on desktops and would run fine on my R630's but looking to downsize the power usage footprint a bit. have you done video's on Proxmox Backup Server?? Wondering if that is a worthy addition to the PVE environment ... Either way, great video and i am looking forward to catching some more of your content - Liked & Sub'ed!! keep up the great work!
I'm going to be putting a lot more effort into videos around both Proxmox and XCP-ng in the near future, and they will definitely take into account their ability to run on a variety of different hardware! Thanks for the comment!
I managed Esxi for years at my company (5 hosts, 35 VM) last summer we moved to Proxmox: 100 % satisfied, just a few small issues (solved). The only problem I still have is when I want to download and deploy a virtual appliance but it is available only to vmware. It works on proxmox after conversion but it's not ideal.
reli nice video, I helped my company to migrate all VMware VM to Azure last year and removed all the on-prem hardware. I think the trend will go on that VMWare is not in a priority in the coming future. Personally, i will learn some Proxmox for the on-prem alterative. Just an interest, will talk a bit more about the Proxmox Backup Server? (as for the VMWare, I am familiar with using Veeam Backup and Restore. How about in Proxmox?) Thanks and cheers.
I've been running a PVE cluster for a number of years now. I like it, and I'm not planning to move away from it, though I'm also playing with xcp-ng just a bit too. But I think a number of the comments here trying to defend PVE kind of miss the point: * Sure, a system admin should be able to configure a network interface from the shell (and I've had to do it on a variety of occasions). But a console menu like xcp-ng's would still be a welcome addition. And yes, you could get around this by just installing a full graphical desktop environment onto it, but that really isn't the way the product is intended to be used. * Yes, the subscription nag screen is obnoxious. It's easy enough to disable, but still obnoxious. I get it, devs gotta eat, but... * Agree on the enterprise repo being enabled by default. Like with the nag screen, it's easy enough to fix--easier now that repository management is included in the GUI--but it doesn't make sense as a default.
If you have not tested running multiple Proxmox nodes in a cluster with ZFS snapshot replication running between them, then you have not seen a huge part of the usefulness of Proxmox for small environments. Using that simple setup, you get the ability of almost instant HA or manual failover of VMs/LXCs between Proxmox nodes without needing any shared storage required. This is huge for small business or home users, to be able to have this VM migration capability without requiring any san or external storage.
Proxmox now features easy import from Vmware. Don't know when it was available, but the newest version is capable of that and it works very well (as long as you dont have something like a Server 2003 you need to migrate)
Vmware's workstation can connect to a VM running on a ESXi server and pass through a USB device. That and the way USB passthrough works on the Vmware's workstation keeps me a sobbing vmware client.
I completely hear the point on the benefit of a menu on the console. However, more and more system admins are so reliant on menus and GUIs, that they are not taking the time to learn the CLI. I have been in technology for 25 years and some of the best platforms that I have supported have been on Linux. Getting to "know" the OS and how to maneuver within it is a necessity that the next generation of administrators need to embrace. Just my 2 pennies... :)
i can't agree more! having a simple cli menu for basic tasks would be nice, but knowing linux gives you so much more power on the console. as example: we are moving from vmware to proxmox in my company and i do all migrations on the console because i can convert/move vmdk files directly to proxmox ceph storage. the GUI can not do this.
yep, the whole world pretty much runs on linux now, so if you're a "serious" sysadmin and the best you can do is navigate an ncurses menu or a webui, then it's time upskill
The VM CPU type will be for clusters of hosts that have different CPUs. To maintain the ability to migrate you need the cpu features to be the same. in VMware this is the EVC settings.
Sooo, re: containers not being useful except for shaving ever so slightly more resources off of a virtualized system. Basically everything you said was where I was at ~10 years ago with containers. I thought their benefits didn't outweigh the burden of being tied to the host's kernel. But all of that was based on a misunderstanding of what problems containers solve, and I think you're in basically the same spot, but on a slightly different subject. For me it was containers in general. For you, it's LXC containers versus "application containers" like Docker. Put simply, there is effectively no difference in container tech between Docker and LXC. Docker gives you an API to work with, and it is expected that you're building your containerized applications in Docker fashion so that they can be spun up on any host running docker. Essentially the same with LXC. I don't know that LXC offers a REST API to query, but that's where Proxmox's API comes into play. But as far as containerized applications? LXC and Docker are effectively one and the same. In Proxmox, hit up one of your storage locations that is configured for holding container templates, and download one of the Turnkey containers. Hopefully that will help things click for you. No shade being thrown here. Like I said, I was very much foot-in-mouth about containers, myself.
Yes, mainly through LXD though. But there is remote management possible. Also, live migration for containers doesn’t exist anywhere… only machines can be live migrated (or as far as I know atleast).
@@lawrivanbuel5221that's true, only VMs can be live migrated. I think that has to do with the use case for containers, though. Live migration is a complex thing, and the reason containers are used is to basically be throw-away compute. If your application can't handle running between multiple containers, the app shouldn't be running on containers in the first place. The benefit is we don't care if a container goes down, because if the physical machine it's running on, or another physical machine in a cluster of machines is still up, you just spin up another container and move on. It's just a completely different way of looking at things from how VMs and physical machines have been admin'd in the past. Where VMs and physical machines are admin'd as if they are delicate and precious things, containers exist with the idea that shit will happen, and the container will die.
@@praecorloth And for intents and purposes, docker containers are LXC’s. (Or any other OCI) Containers, especially non-root containers are a better and more scalable way or running workloads. There still are some that require a VM, but most of the time… a container is better.
I'm a little shocked you did not mention that it's possible to import vmdk files from vmware into proxmox and basically migrate your vm's as-is into proxmox with one command in the terminal.
Since when can Proxmox automatically shift VMs to other hosts in the cluster to balance out workloads? It has HA so that in the event a host goes down, the VMs should power up on a host that's online.
Just spent the last year of my life migrating from nutanix ahv to vmware. Looks like im gonna be spending the next year moving elsewhere. Really wish i would have purchased 5 years of support to push this out
@@bouzidoussama496 Our hardware was EOL, so we started pricing out upgrading or replacing it. For the price, we were able to get 3x more in terms of performance, storage, compute, and networking by going back to VMware non-hyperconverged. AHV did work pretty well; however, there were times when it was frustrating. Some vendor appliances only support VMware, and we ended up having to run a small cluster alongside the entire time due to this. Simple things like pulling a report of guest OS were nonexistent, simple but a lot of small things that are easy in VMware do not exist. Going to AHV is pretty easy; their migration tool works great. Moving back to VMware was not as easy but doable with some tools.
As a big fan of infrastructure as code I must sy, I prefer XCP-NG the XenOrchestra Provider for terraform is just superior to the Proxmox one. At least last time I check the Proxmox one a couple years back it was pretty bare bones and was lacking a lot of features.
I've been using mail gateway for a few years and the confusing mess of an interface is one of the main reasons I'm a bit on the fence sbout VE. Also the nagging screen comes up for users clicking release or whitelist in the report emails. That's just irritating and I'm sorry but I'm not going to pay gor the software for myself snd ten family members when the Community Edition is s whopping €175 per year. Some kind of middle ground here would have been nice.
With VMware screwing the pooch I'm betting lots more dev is going to be pushed into KVM based stuff and OEMs of storage solutions are going to want to support it too. Hopefully it comes along!
I love Proxmox and XCP-ng as ESXi replacements but with no support for NVIDIA vGPUs for which my clients have spent years developping using NVIDIA architecture and libraries these paltforms are dead in the water. it would be too costly to change to AMD and would require recertification of their platforms from various government bodies and customers from around the world. For 2D and non-vGPU VDI/VMs then it would work but not for NVIDIA dependant architecture.
I'm old enough to consider containers as full lxc containers, not application containers... I'll leave that to the kiddos and wanna be devops who can neither dev nor ops correcty 🤣 #BOFHinside More seriously, LXC containers are what they are, initscript leveled containers, they can start sub second so the lack of live migration, depending on haw your application is scaled, is actually not a problem... I run one of the biggest french-speaking tech news outlet infrastructure on a 5 host proxmox cluster+cephRBD+CephFS running 98% of containers (and a few VMs for virtualized network equipement) and it's mostly a breeze (they do use docker for dev, but that's in isolated VMs/VLANs)
Proxmox is great, but it is still NOT ready as an “appliance”! It does not have a ONE click, ONE file config backup and restore. Does not come with UPS support. Yes, you can install NUT every time you install Proxmox, but still no GUI for it. The GUI is very non intuitive and often you have to go to the command line to get stuff done: Having to edit the boot loader to add support for passthrough is just NOT acceptable in a high caliber “solution”!
21:31 again with just few commands you can install whole desktop environment in proxmox, I have been using proxmox installed on my primary Laptop and with Debian 12 Desktop environment Best of Both world, greater OS and Greatest Virtualization solution in on pack
I don't think the complaining about ProxMox not having a 'real console interface' is relevant, especially comparing it to the ESXi limited interface where you could literally just reboot, view underlying hardware and other simple stuff. I mean, what is the difference between typing 'reboot' in the console and 'lshw' for viewing hardware info and using the ESMi menu ?
Hey, would you consider doing a video on an in place migration from ESXi to Proxmox. I like many installed ESXi to get the Guest OS off the hardware, making migration to new hardware a little easier when the time came, now I have a number of boxes that need to go from ESXi to Proxmox VE. How about it?
I'd rather have LXC containers than not. They're essentially a poor man's VM, in that they definitely lack a couple of the features a fully fledged VM would have, but in return they are much more lightweight. If HA and live migration aren't important, they can allow you to run a lot more VM like machines on one physical host. Though for enterprise use I think Kubernetes would serve most of the same use cases, but better.
Any news on what they plan to do with VM Workstation Pro? I use Workstation Pro for testing and for keeping older OS's available so I can offer support for older OS's I have over 100 OS's from desktop to server to Appliances like pfsense Any word on those licenses or some sort of alternative?
Since when? Last time I checked LXC could not be live migrated, you had to stop it on one host, migrate and then restart on the other host. VMs you can migrate without even losing a single ping
They don't live migrate, they are stopped, then migrated and started. It happens so quick you might think they live migrate, but they do not. It's important to understand this. VM's can live migrate, but LXC aren't.
Where I work will probably just pay the cost difference and stay on VMWare; but I’m watching these type videos just in case. One issue I have is we have a set of host with Nvidia L40 cards in them for AutoDeskand other graphic intensive software. Linux is notoriously horrible or horrendously complicated with PCIe pass through. What I’d like to see covered in one of the type of videos is how these alternatives deal with hardware like video cards.
In version 8, they made PCIE pass through very easy, tons of instructions online. I passed through a couple of ports on a 4 port nic with a few clicks.
You can use those repos, because they exist and are needed for pve updates. Instead of disabling, you need to change the apt sources with the community repo.
I'm a sysadmin at a decently sized European university. While our large faculty server is stuck with ESXi for the near future (which I'm hoping to change), our smaller servers all run PVE. This includes our storage/backup/mass-deployment server, a cluster of several machines that deal with high compute power for AI and machine learning, a cluster that runs *several hundred* concurrent VMs for networking and white hat hacking exercises, and smaller roles too numerous to mention. I've never had a single "I wish PVE had x..." moment. The only thing that's a bit difficult to work with is the user and permission management, but it's not a deal-breaker.
As for LXCs - I'd rather have them than not. My home server has very little memory, and having my services run inside Debian or Alpine LXCs that each consume ~20 megabytes of RAM is pretty baller.
Also LXC could support the same almost-everything live migration if someone put in the work, because LXD supports it using CRIU and it works and uses the same APIs.
I'm basically the same at my company (which is ~100 employees but growing decently mostly at a main location but we have two satellite locations with 10-20 employees each). I put vcenter/esxi at the main office but when we expanded to the other locations and it looked like vmware might get sold I gambled on proxmox at those locations (I was using the free version at home in the home lab) and I haven't looked back. By the end of this year the main area will be transferred off esxi/vcenter to proxmox as well. Having the free version only minorly annoying reminded me at the right time and has turned into a few thousand a year in licence fees for them from my day job so it's basically the best advertising program they can have. The only thing I miss is the multi location management through a single pane you can do with vcenter and I'm hoping sometime in 2025 they release that feature, there is talk of them working on it on the forum on roughly that timeline.
I am a teacher at high school in Norway. My students that were new to virtualization, they prefer Proxmox over VMware when it comes to user friendliness. And LXC containers are very useful, when you're low on resources.
Hello,
@Wampa842 , @barfnelson5967 , @2GuysTek
I'm going to start building my minilab (especially for AI) and was looking at PVE.
My first challange is to port a "custom" vmdk to PVE (to free up a server).
The thing is that this one has:
1.bitlocker enabled (windows 10)
2.(virtual)TPM device present (Virtual Machine Settings->Hardware->Trusted Platform Module=Present)
3.VM is fully encrypted (Virtual Machine Settings->Options->Access Control=Encrypted)
My question(s):
a)Can I migrate it as is / how ?
b) if not,do I really need to remove/disable all 3,2,1 ?
Thanks!
Lxc containers are better designed and more secure than the mess than docker is, I used them a lot
8:05 "all management needs to be done via web gui"
completely incorrect. Proxmox makes virtually everything and more available via the command line. "pct" program and "pve" prefixed commands, tons of them. You can enter hosts via the "pct enter hostID" etc
in /etc/pve/ all machines are available for config edit as well
Agreed, it's not as neat as a menu where any ape can just navigate and press enter, but the functionality exists for sure.
There are loads of scripts that you run in the terminal that does entire VM or CT setups.
@@watvannou Yea you can tell he didn't do much research and just spat out his video to get views in this high traffic time.
The docs etc are filled with command line stuff to use, yet he claims it "NEEDS" to be done via GUI lol. Unfortunately (for those apes) there are still many things not even available in the GUI so sometimes you have to get into the cli and files. One day it'll probably all be in GUI though.
Sure you *can* go into the shell and manage things directly from there, but his perspective is talking about for general use and deployment. Would I trust a L1 or L2 tech to go down to the machine and navigate through the menu of XCP-ng to start a VM? Sure, absolutely. Would I trust one of them to go log into a shell and start running commands that directly manage the entire VM structure and config of the house without prior training? Probably not, no.
@@1armbiker no it isn't, otherwise the word "needs" wouldn't have been used
@@1armbikerI mean....how is, "qm start (insert VM machine number here)" something one of your tech's wouldn't be able to handle? Train them on the dark arts of --help. The comment section of this video has suddenly made a lot of past interactions make sense...lol
regarding to the cpu types, if you choose "x86-64-v2-AES" you can live migrate VMs between Intel and AMD physical CPUs. :)
Agreed, I'm assuming that's why it's the default selection!
Thank for letting me know that. I am new to proxmox and I was thinking about the diffrence between host and x86-64-v2-AES.
@@2GuysTek ultra charged EVC. :)
@@desiengineerplays3544 "host" replicate host cpu model.
"host" copies the motherboard CPU model. You have to be careful if you have different CPU models in a Proxmox Cluster and then do live migration. Therefore, the default version assigned is intended to secure against issues. If every CPU in the cluster is the same, then you can keep it at Host and in some cases (newest CPU's) you'll experience better performance 😊
What proxmox devs are doing is little different, it is a place where all the opensource virtualization technology meets.
Yes, it might not be general user intiuitive, but it provides well on core virtualization.
And as a linux nerd I love it.
tbh proxmox's interface seems far *_far_* better to me. Yes, there is a lot, but it's all very clearly formatted and hierarchically displayed. Being able to see everything you can do means you'll be able to sumble across features that solve your problems, it keeps things organized, and it keeps things simple. The Windows route of submenus that open other submenus that do the job of that other submenu but are actually different and can also open this other submenu which is technically deprecated by that submenu over there but actually still lets you do this thing a lot easier, just isn't good. The only reason why people don't complain is they're used to it, but being used to something doesn't mean it isn't bad UX.
FFS I literally had to google how to even empty my trash on windows because my windows partition was low on space and there isn't a clear option to do it anywhere. It takes several submenus deep in the system settings window to empty your trash. If your UI is so convoluted that I need to google how to do basic tasks, don't bother making a UI at all, it's more work than if you just made me google the commands to do the same thing.
I haven't done much virtualization, (I've only really used virt-manager for local-virtualization, though I've always kept an eye on higher scale virtualization since I've been planning to build a bigger homelab long term) but without any experience, I'm picking the proxmox UI without question. I don't need to look for anything, it's just there, I can see it, it's there. If I have something I need to do I can know where it is without having to memorize it because there is a simple and consistent design philosophy.
Good video, fair comparison, and good points, especially around the major version upgrades, such as going from 7 to 8.
Life after VMware 🤣 perfect playlist for me
LXC containers are one of the main reasons I use Proxmox. They are extremely lightweight and fulfill all my requirements in my homelab. Compared to traditional VMs (Virtual Machines), their resource footprint is significantly lighter. I have only 16 threads available on my CPU, which means I need to be particularly cautious with thread reservation. LXC containers, on the other hand, offer great flexibility in terms of resource usage. I really love Proxmox and plan to continue using it in the future.
The LXC containers are amazing, fast, easy to configure and use less of those valuable recourses.
Re proxmox management from the console: The gui available via the web browser is only a think overlay over the command line tools, available from the command prompt. All VM and container related functionality is available from the CLI, and in some specialized cases, use of the CLI is still required to complete the task.
The big advantage of the GUI is that it is much easier to use. The "short" version of the help text listing only available command line options for "qm" (the VM management tool) is several hundred lines long, spanning many pages. Super intimidating for a beginner, but a life-saver for an advanced user, who wants to automate routine tasks.
I mentioned in the video something to the effect of how you can do everything you need to do from the command line, but the point is, that makes Proxmox less of an option for a homelabber or business that doesn't have that deep knowledge in linux command line. I think that when a company is considering their options in terms of their future replacement for VMware, they're going to have to take into account their IT staff and engineers and how much retraining/learning they're going to have to do, and I think for Proxmox, that's a concern.
@@2GuysTek8:04 You’re saying all management from the host has to happen from the GUI, that’s not at all true. Anything you could do from the GUI is possible from that prompt.
You then followed it by saying that kind of access is useful when you have no network connectivity, but again, you have far more access to modify the HV config than you’re letting on.
I’m fine with giving ESXi the win for ease of use, but the phrasing is bad dude. Completely blowing past the amount of power in that console because admins don’t know their tools is a cop out.
@@bigjaydogg3384 I don't think the phrasing is bad. He has a point. Perhaps what Proxmox can implement is a CLI menu similar to TrueNAS whether you can drop into Bash
@@zpariharThats the thing, you don’t need a menu option because you’re already there. Thats why the video is so wrong, its not that the prompt it drops you into is incapable of anything, its that the prompt is so open ended simple tasks require at least some level of Linux knowledge.
@@bigjaydogg3384missing the point.
"can" do it is different to "how" do it.
if i run a small business and nobody in it knows linux nor proxmox, it doesn't matter how much you tout that it can be done if we don't know how to do it, especially if it's an emergency and we need to do something from the console.
if i want a linux expert, i need to also pay their salary. i've personally helped several small businesses without any in-house expert team to do these tasks and having a more layman useable interface could save them big bucks hiring a technician to do emergency maintenance on an hourly fee.
how much work would it really be to have a TUI where you can change hostname and network options? for example, they could use NetworkManager and simply call nmtui for the user, which is a massive life saver.
Hi, it's important to emphasize that Proxmox leverages KVM for VM functionality. Great video, keep on home-labing!
I would love to see Nutanix
I'm still on the fence as to which hypervisor to migrate to, so I appreciate the videos. However, I wanted to say that I appreciate the presentation style of these comparison videos. I've watched both, and they are fantastic examples of how to do an X vs Y comparison. Nice work!
Tom Lawrence (Lawrence Systems on RUclips) has a new video on XCP-ng and Xenorchestra out today. Might take a look to see the Proxmox vs. XCP-ng differences (even if his video doesn't really demo that per se). I use Proxmox on "small systems," and XCP-ng on the big iron.
Been swinging between Proxmox and XCP-ng on my test system. XCP-ng feels more of an 'Enterprise-ready' but Proxmox is pretty strong and powerful out the box without over complicating things. I think for me the built-in LXC is going to make Proxmox the homelab solution. I'd probably want to do XCP-ng if we had to drop ESX at work though.
I can understand the the immediate thought of Docker and kubernetes when it comes to containers, but remember that the container support in Proxmox actually started with support for OpenVZ containers, and moved to LXC even before docker even existed. Today I'm not sure that LXC makes that much sense anymore, but go back 10 years, and LXC provided significantly reduced overhead in contrast to a full VM. I've been using Proxmox since version 1.0, and with 3GB of ram on a server, shaving the memory usage by a few 100mb actually made a difference. As for the upgrade, it's actually not that hard to do. In fact I find it more unnerving to upgrade a vcenter, especially if NSX is involved.... Keeping my hands off the hypervisor and don't changing anything I've never had any issues upgrading a Proxmox host. That's not something I can say about upgrades of ESXi hosts...
I still do not fully understand the upgrade process on standalone ESXi, like why do I need do download some sort of file to start the update? Can't I just push a button or run a command?
15 lxc containers in my homelab and one full VM for docker. :) Love the lxc contaiers as they are lightweight.
Thank you for making these videos! My organization was just talking about this very thing, and this will help me explain options to my management team.
This is exactly why I’m making them!
thank for the video.
As a linux nerd I say yes to a full featured gui with all options but a fallback to a shell for scripting helpfull tools is a great thing.
Converting VM to proxmox works with simple scripts but every Vm should be testet after a transfer.
Good video my dude. I would like to see a follow up video on clustering, vim networking, vm storage, virtual disks, migration, fault tolerence and "DRS".
Thanks! I think that's my target after getting through these initial comparison videos.
I'm interested in this as well with Fault Tolerance considering he didn't list it, only that it was feature parity was complete between the two and I do not think that is correct.
would love comparisons with Proxmox and XPG-ng vs Virtual Center. Our VMware bill went from $132k/year to 1.225 mil/year and we need to find some alternative.
LXC have a lot of uses. It's basically for services that would best be run on bare metal, but you don't want to pollute host OS with it (samba server, nfs, ..)
I started using proxmox last year and I have to say that it is clearly very good
I agree concerning the anoying repository
I must also add that to have the possibility to make snapshots of your vm you have to pay attention to the dis format of your VMs
however up to now it is working like a charm
I haven´t made any experience on EXI because as a non professional user I don´t want to pay monthly for a Virtualization. I am running a mac mini but sometimes I have to use a Windows machine and I love Linux. Parallels for mac is so expensive and so a bought a mini PC for about 200 € and can now virtualize Windows and several Linux on that machine by using proxmox. Although I am 71 years old I was able to manage all this because there is a great community supporting proxmox users. And you will find big help on RUclips also. I think proxmox is a good software and free to use.
You know that you can do basically anything in pvr from the console via command line once you're logged in? If you want you can avoid the gui entirely and perform everything the gui can plus stuff the gui cannot do in the terminal?
Yeah yeah you know it. But esxi loses. You can't do anything in esxi there even if you want. Let's say network fails, you're lost. You can't even check vm status...
Sorry you're absolutely wrong
I moved from Esx to HyperV then to Proxmox a few years ago. VM migration was a nightmare. However I’ve not regretted moving to Proxmox. It works well. Particularly good when you set up a Ceph cluster. There are easy ways to disable the nagging. I have a script that removes the nagging that executes on every boot. Not seen the nag in a very long time. The LXC containers don’t live migrate but they do migrate. They just shut down and start again on the destination host. They’re very useful for lightweight tasks such as running ddclient, letsencrypt, cloudflared etc. Generally, I’m glad I moved to Proxmox. Free HA, good for Linux and Windows VMs, an acceptable built-in VM backup system, Ceph for cluster storage. What’s not to like?
LXC is really nice to have, I wouldn't complain about it being there. If you need Docker or k8s, throw a VM with them in there .
You can throw Docker or/and k8s in an LXC container! Proxmox rocks!
Should have really talked more about the ability to GPU and other pass through capabilities as well without paying an arm an a leg for Nvidia licensing. It's really nice for Plex or Jellyfin transcoding and running cheaper LLM/AI solutions. Also the Proxmox backup solution is truly amazing and probably needs to be talked about more. It doesn't just give you VM backups but it also does file backups as well. For me this was such a better and easier solution than going with Commvault or Veeam.
Ick. CommVault. What a pile of steaming crap that thing is. Thank God we got rid of that a few years ago. We went to Veeam, but someone else manages that, so I don't know how good or bad that is. But, yeah, backup capabilities being built into the platform vs 3rd party is so practical.
I'm sure you will hear linux ops folks talking about how great the cmd line is for admin'ing... and I don't disagree... IF you're doing it frequently enough to remember how to do everything. For the rest of us who don't admin linux machines, we waste a LOT of time looking up commands and that goes away with a basic (even text based like with esxi) ui.
I just started playing with Proxmox as a potential migration path from vsphere for my roughly 500 core cluster, the new vGPU non-prod release gives me hope that by the time I need to migrate there will be production gpu support which would make proxmox fully capable of replacing my vsphere e+ environment.
Good overview but again I'd like a deeper dive in to advanced features like backup... upgrade workflow, software defined network and so on.
oh and yea I DID assume containers meant app containers too.
Host with API access only. Let the community write the front-ends they need. Makes the host smaller as well.
@@brodriguez11000 Hosts need more than an api... they need a capable diagnostics interface (that should allow admins to manage the local machine). An api only host would be a problem in certain disaster recovery scenarios.
Could be console based, ssh or web... but should not rely on external custom software to work. (everything needed on the box).
As you just stated, the reason Proxmox CLI is minimal is the fact the CLI is debian with KVM enabled. I have Ubunutu servers with KVM installed making them TYPE1 hyperivosors and now I just manager them with their gui tools like Cockpit or Virt-Manager. yes I can get in to the hosts via ssh but I don't run additional services on the local host at this point and just manage VM's runing said services. KVM was designed to be open source and full customizable to your liking.
What about openstack / Ubuntu micro stack?
Ubuntu microcloud, I think you are referring to this one the latest version the install fails, microovn doesn’t install and microceph bootstrap fails and you need to install all manually. Also the lxd project leader just left canonical and forked lxd into incus. Microstack bootstrap process fails more than it installs successfully.
The repo nag is the price of free, no big deal.
Yep, only takes a second to clear it. No biggie. There is a way to remove it till the next update but I don't bother since it's minor.
I’d love to see a harvester and hyper-v videos
I migrated ESXi VMs to Proxmox using the OVFtool from VMware. It was quite easy and where are guides out there. Having used VMware (ESX) since it was running on top of Linux (ya, I'm old) professionally and personally, I've been quite happy with the switch to Proxmox. I switched when ESXi 7 came out and made it harder to run on the same hardware as 6.5 did. Learn Linux. You'll be glad you did.
Yeah, I upgraded via scripts and my machine ran fine. Then did a new install and was told my CPU had aged out. I shut it off and have used a KVM based machine ever since but am interested in ProxMox :) Large customers I work with are moving away from VMware as FAST as they can and the renewal costs I'm hearing about are eye watering.
I believe the reason Proxmox has the VMware features, the pvscsi, vmxnet3, etc is because if installed Vmware Player on a Linux host, it has to compile kernel modules. So they may have the modules precompiled and ready, not to mention there is Open-VM-Tools, which is an open source version of VMware Guest Additions for Linux. When the Open VM Tools became standard, VMware pretty much dropped support for their own proprietary guest additions for linux, especially for newer kernels. Maybe having these for guests is for compatibility should you wish to migrate from ESXi or any other VMware product.
funny - someone running a linux based virtualization server (cluster) is not familiar with linux, hmm. noone needs lxc containers? ok, we are not on the same page here. 😂
Theoretically you could be an esxi admin and never run a Linux VM 🤷♂️
There are many use cases. Not all require expertise in everything used across them. Don't assume someone that's an expert at Salesforce, will know D365. Just because it's a CRM. Or an expert on Debian, but gosh darn they just can't get RHEL rock'n the same. Deep Linux knowledge is not a given for DevOps/SysAdmins, or any other IT staff.
I just wiped my drives last night from Vmware ESXI 7u03 after using it since 2022. I ended up going with proxmox for now because due to having way more support in terms of DevOps tools such as Ansible and Terraform as well as installing nested hypervisors. I tested xcp-ng but the support base is too small esp for DevOps and API automation stuff at the moment. Also the RAM usage is slightly higher.
Maybe cover Nix Shell.
LXC containers are a lot more light weight than full vm's, they start up in ms and use less than 100 mb of ram. If your usecase does not require extreme security then they're great.
I absolutely agree with all of the comments on the lxc containers in proxmox. The first time I started my proxmox virtual server, I started with VMs. However, I realized that it takes more memory and resources to run the virtual machines, so I changed to creating lxc containers in proxmox and it uses very little resources when loading a lxc container. I only use VMs if it calls for using them in proxmox. I am now getting ready to run a second proxmox VM because I love that they added the backup server to backup virtual machines. however, I do want to try other VMs, such as XCP-NG with Xen orchestra since I am very familiar with citrix, and also SUSE harvester. I want try VMware ESXI, but I am glad I did not go further since they got rid of the free edition.
very useful video, thx. never used vmware, trying out pve now. i like to learn the under the covers troubleshooting, but agree they coulda shoulda provide a mgmt interface on host. i struggled to get the connection working to get the web ui, which required navigating local conf files to even get going. i'm wiser for it, but in a production setting i'd like a ui pointing me in the right direction to potentially save hours of downtime.
And killed the VMWare nonprofit program. Great video, thanks for putting together
Video is ok, BUT you are missing so many things in this comparison. First: the feature set, you only say esxi can this, ok proxmox to. BUT you dont say that proxmox can do mush more the esxi even in higher tiers. (With vcenter) there is a complete backup solution already in there, for free. Than there is absolutely no hardware logdown to "certified" shity and expensive hardware, the supportcost + hanrdware are cheaper than the "certified" hardware alone and so on. There are so many point you have ignored wish is ready disapointing
That's because he isn't familiar with proxmox and took a day or two to look into it and made a video to get views while this is a hot topic
You can still use ESXi, assuming you have kept the ISO's and your keys, which i have. I decided to give Proxmox a go last month an after being a VMware admin for 9 years i found it fairly easy to get to grips with it. There were a few things i had to google such as the ISO storage !. Personally i have found it very good and on comparable hardware im noticing lower CPU use and lower RAM use compared to ESXi running exactly the same VM's.
8:00 Missing bit is qm command. Proxmox can be run from ssh terminal.
Very bad review of Proxmox. Look elsewhere for more professional video.
"But not all support engineers have deep Linux knowledge."
No offense, but you can't really call yourself a support engineer in the 2020s if you don't even know (or are too dumb to google) how to set up networking in Linux. It's the OS that runs more than 95% of the top million websites and most modern technologies (especially everything regarding containerization) have it as a foundation. The point about the easier configuration on VMWare still holds though, that's a great convenience feature. But please people, learn your Linux basics. Setting up networking is not "deep Linux knowledge".
You should checkout Rancher Harvester! Power of k8s applied to VMs with strong defaults to make it just work OTB.
Proxmox console is poor?! No… the first requirement to adopt Proxmox is to have hard and strong skills in Linux/Unix environment. Other way… go to click based OS.
Dayum, seems like I will have to move back to Proxmox due to ESXi not being free anymore.
Not sure if this has been mentioned but you have multiple CLIs availabla: qm, pct, pvecm, etc, to launch, stop, enter, reboot, etc., vms, lcxs and managing the cluster. Also all the networking and virtual bridges (switches) can be managed from the linux terminal.
"not all support engineers have deeper linux knowledge" - lol
so why you are calling them engineers? call them support technicians or better support specialists, as engineers/technicians should understand what is and how it is working under the hood of the used solution :D
proxmox is just a gui on top of qemu/kvm and lxc (btw. that's a bit odd, as lxc is quite archaic right now and most recent solutions are based on OCI or directly on containerd)
tbh the major upgrade from 6to7 and 7to8 was so painless and easy. just a small check-script and the go for it. no much hassle
VMWare or Proxmox?
@@TechBench I´m about Proxmox. Unfortunately I never got the chance to major upgrade a vmware...
lol you have very good points but come on it is free. Some of your complaints are petty though. By the way every IT department needs a linux nerd. Two is even better so that he or she does not get bored.
Another thing I'd like to say: You're comparing the ESXi GUI to the Proxmox GUI which is unfair in my opinion. You should have compared vCenter's GUI to the Proxmox GUI. And then, the complexity of both GUIs become much closer and therefore your point is invalid to a certain extent, at least in my opinion. If you wanted to truly compare the ESXi GUI to Proxmox, you should have at most only compared the Proxmox single host GUI to it, but that wouldn't make much sense because in Proxmox the single host GUI options are based on the Datacenter options. So the only valid comparison is vCenter vs Proxmox.
I would rather use Proxmox than Docker, because Proxmox doesnt give me migraines
If you really understand Docker there is no such newbie issues.
@@pepeshopping See some of don’t have time to spend weeks understanding some broken bullshit platform like Docker when they just want a CRM and small business applications on a small tower server
The main question is, is PVE reliable, i mean rock solid so you would run your essential SAP Environment on it, 24/7 ?
Similar to esxi really. Build a 3 or 5 node PVE cluster with Cepth.
Great comparison. Thank you.
Well.. on the one hand you say you cannot expect everyone administering an Proxmox host to fix it using the command line, but on the other hand you say that editing the hosts file through the WebUI wasn't needed xD
Great video. Btw, LXC live migration is coming.
Docker comes from LXC. LXC (and the previous "original" container technology OpenVZ was around before Docker was a concept
One thing I noticed in your XCP-ng comparison was the mention of the limitation of VM pausing during snapshot creation if memory is included in the snapshot. However I don't remember hearing about Proxmox snapshots in this video. Does it have the same VM pausing issue as XCP-ng for memory-inclusive snapshots? It looks like the snapshot capability is dependent on a number of factors such as the virtual disk format or the file system used to hold VM images (so you need to use QCow virtual disk files if your VM storage is networked using NFS or Samba), and that would have been good to mention.
I can't give a comprehensive answer but the short answer for taking a snapshot with memory. Yes, the VM state is paused for the duration of taking the snapshot.
You can both pause or stop to get all the memory in snapshots. Those limitations are the differences between block and file storage. Not due to proxmox.
Remember, the linux command line is very user friendly. It's just picky about its friends... ;)
you wanna buy some VIM?
Haha… been using both emacs and vi since around 1994, and neovim is my preferred editor today. 😂
Thank you for this and your other companions. How about a comparison with Hyper-V? My company is thinking of going with it as we have server licenses
It’s in the works!
thanks for the comparison, i noticed that with proxmox unlike esxi, you can't install a promox host to a proxmox cluster if that host has vm's on it.. even if they are shutdown.. the promox most MUST be clean.
It depends on the host's storage. If the VMs are on a separate storage then installing ProxMox on the host's boot drive is not an issue.
I totally agree! If proxmox made the GUI a little easier, made the upgrade more seamless, and made it possible to migrate from vmware to proxmox smoothly, it would be a no brainer. But these points remains and a couple more you mentioned in the Video. Really hope to see these changes! Thanks for the video, good summary!
Just to add to this, Proxmox recently released a conversion tool to make migrating from VMware to Proxmox easier: news.itsfoss.com/proxmox-vmware-migration/
I´m not exactly sure how user an group management is in esxi, but have you noticed that you can also add other realms in proxmox like from LDAP/AD or even OIDC?
Yep, that’s also available in ESXi and vCenter.
After 24 years with VMware, it’s time to move on. I started with VMware Workstation for Linux in 2000, before GSX and ESX 1. Broadcom screwed up everything that was great about VMware. I have a lot of clients concerned about what is going on with VMware, including corporations that are paying over $300k on licensing and support.I know for sure that some of them will migrate to something else before the end of the year.
LXC is vastly more useful for Linux than a full VM. Almost all things that would run in VM on ESX can be run in a super lightweight LXC container in Proxmox. You can run probably 10x as many LXC containers on the same hardware as full VMs. And they are managed almost identically to a VM. If you are not a Linux expert, and only have VMWare experience you probably have more to learn about LXC and how useful they are. LXC is what is used for most cloud 'shared' instances you get from hosting providers because they are vastly more efficient for Linux systems than hardware emulating VMs.
Hey man, i have to say your X-CPNG and ProxMox vs ESXi has been great and super informative. I appreciate you taking the time to do this..
I run ESXi 6.7 in home lab and, given the state of the company going forward, i am looking to change that. Have you done any videos on running ProxMox in mini PC's? I know it is pretty compatible on desktops and would run fine on my R630's but looking to downsize the power usage footprint a bit. have you done video's on Proxmox Backup Server?? Wondering if that is a worthy addition to the PVE environment
...
Either way, great video and i am looking forward to catching some more of your content - Liked & Sub'ed!! keep up the great work!
I'm going to be putting a lot more effort into videos around both Proxmox and XCP-ng in the near future, and they will definitely take into account their ability to run on a variety of different hardware! Thanks for the comment!
as a data point for you: my home lab is a mini pc (minisforum HX90) running proxmox. It started life as proxmox 7.x and is now 8.x.
I managed Esxi for years at my company (5 hosts, 35 VM) last summer we moved to Proxmox: 100 % satisfied, just a few small issues (solved). The only problem I still have is when I want to download and deploy a virtual appliance but it is available only to vmware. It works on proxmox after conversion but it's not ideal.
reli nice video, I helped my company to migrate all VMware VM to Azure last year and removed all the on-prem hardware. I think the trend will go on that VMWare is not in a priority in the coming future. Personally, i will learn some Proxmox for the on-prem alterative.
Just an interest, will talk a bit more about the Proxmox Backup Server? (as for the VMWare, I am familiar with using Veeam Backup and Restore. How about in Proxmox?)
Thanks and cheers.
I've been running a PVE cluster for a number of years now. I like it, and I'm not planning to move away from it, though I'm also playing with xcp-ng just a bit too. But I think a number of the comments here trying to defend PVE kind of miss the point:
* Sure, a system admin should be able to configure a network interface from the shell (and I've had to do it on a variety of occasions). But a console menu like xcp-ng's would still be a welcome addition. And yes, you could get around this by just installing a full graphical desktop environment onto it, but that really isn't the way the product is intended to be used.
* Yes, the subscription nag screen is obnoxious. It's easy enough to disable, but still obnoxious. I get it, devs gotta eat, but...
* Agree on the enterprise repo being enabled by default. Like with the nag screen, it's easy enough to fix--easier now that repository management is included in the GUI--but it doesn't make sense as a default.
If you have not tested running multiple Proxmox nodes in a cluster with ZFS snapshot replication running between them, then you have not seen a huge part of the usefulness of Proxmox for small environments. Using that simple setup, you get the ability of almost instant HA or manual failover of VMs/LXCs between Proxmox nodes without needing any shared storage required. This is huge for small business or home users, to be able to have this VM migration capability without requiring any san or external storage.
Proxmox now features easy import from Vmware. Don't know when it was available, but the newest version is capable of that and it works very well (as long as you dont have something like a Server 2003 you need to migrate)
Vmware's workstation can connect to a VM running on a ESXi server and pass through a USB device.
That and the way USB passthrough works on the Vmware's workstation keeps me a sobbing vmware client.
Great videos! Thank you. 😊
Proxmox LXC system containers I find super useful in a homelab situation, for a production environment I use Incus (LXD fork by LXD devs).
Great video! I'd love to see a comparison of xcp-ng and proxmox now!
since it is debian based, couldn't you install any GUI of choice to interface with the host on the same machine?
I completely hear the point on the benefit of a menu on the console. However, more and more system admins are so reliant on menus and GUIs, that they are not taking the time to learn the CLI. I have been in technology for 25 years and some of the best platforms that I have supported have been on Linux. Getting to "know" the OS and how to maneuver within it is a necessity that the next generation of administrators need to embrace. Just my 2 pennies... :)
i can't agree more! having a simple cli menu for basic tasks would be nice, but knowing linux gives you so much more power on the console. as example: we are moving from vmware to proxmox in my company and i do all migrations on the console because i can convert/move vmdk files directly to proxmox ceph storage. the GUI can not do this.
yep, the whole world pretty much runs on linux now, so if you're a "serious" sysadmin and the best you can do is navigate an ncurses menu or a webui, then it's time upskill
The VM CPU type will be for clusters of hosts that have different CPUs. To maintain the ability to migrate you need the cpu features to be the same. in VMware this is the EVC settings.
Sooo, re: containers not being useful except for shaving ever so slightly more resources off of a virtualized system.
Basically everything you said was where I was at ~10 years ago with containers. I thought their benefits didn't outweigh the burden of being tied to the host's kernel. But all of that was based on a misunderstanding of what problems containers solve, and I think you're in basically the same spot, but on a slightly different subject.
For me it was containers in general. For you, it's LXC containers versus "application containers" like Docker. Put simply, there is effectively no difference in container tech between Docker and LXC. Docker gives you an API to work with, and it is expected that you're building your containerized applications in Docker fashion so that they can be spun up on any host running docker.
Essentially the same with LXC. I don't know that LXC offers a REST API to query, but that's where Proxmox's API comes into play. But as far as containerized applications? LXC and Docker are effectively one and the same.
In Proxmox, hit up one of your storage locations that is configured for holding container templates, and download one of the Turnkey containers. Hopefully that will help things click for you. No shade being thrown here. Like I said, I was very much foot-in-mouth about containers, myself.
Yes, mainly through LXD though. But there is remote management possible.
Also, live migration for containers doesn’t exist anywhere… only machines can be live migrated (or as far as I know atleast).
@@lawrivanbuel5221that's true, only VMs can be live migrated. I think that has to do with the use case for containers, though. Live migration is a complex thing, and the reason containers are used is to basically be throw-away compute. If your application can't handle running between multiple containers, the app shouldn't be running on containers in the first place. The benefit is we don't care if a container goes down, because if the physical machine it's running on, or another physical machine in a cluster of machines is still up, you just spin up another container and move on.
It's just a completely different way of looking at things from how VMs and physical machines have been admin'd in the past. Where VMs and physical machines are admin'd as if they are delicate and precious things, containers exist with the idea that shit will happen, and the container will die.
@@praecorloth
And for intents and purposes, docker containers are LXC’s. (Or any other OCI)
Containers, especially non-root containers are a better and more scalable way or running workloads. There still are some that require a VM, but most of the time… a container is better.
I'm a little shocked you did not mention that it's possible to import vmdk files from vmware into proxmox and basically migrate your vm's as-is into proxmox with one command in the terminal.
Since when can Proxmox automatically shift VMs to other hosts in the cluster to balance out workloads?
It has HA so that in the event a host goes down, the VMs should power up on a host that's online.
How many of you are still willing to pay for ESXi? Wondering out of curiosity.
Just spent the last year of my life migrating from nutanix ahv to vmware. Looks like im gonna be spending the next year moving elsewhere. Really wish i would have purchased 5 years of support to push this out
Why have you decided to migrate away from Nutanix AHV, and what factors are preventing you from considering it as an option?
@@bouzidoussama496 Our hardware was EOL, so we started pricing out upgrading or replacing it. For the price, we were able to get 3x more in terms of performance, storage, compute, and networking by going back to VMware non-hyperconverged. AHV did work pretty well; however, there were times when it was frustrating. Some vendor appliances only support VMware, and we ended up having to run a small cluster alongside the entire time due to this. Simple things like pulling a report of guest OS were nonexistent, simple but a lot of small things that are easy in VMware do not exist. Going to AHV is pretty easy; their migration tool works great. Moving back to VMware was not as easy but doable with some tools.
As a big fan of infrastructure as code I must sy, I prefer XCP-NG the XenOrchestra Provider for terraform is just superior to the Proxmox one. At least last time I check the Proxmox one a couple years back it was pretty bare bones and was lacking a lot of features.
I've been using mail gateway for a few years and the confusing mess of an interface is one of the main reasons I'm a bit on the fence sbout VE. Also the nagging screen comes up for users clicking release or whitelist in the report emails. That's just irritating and I'm sorry but I'm not going to pay gor the software for myself snd ten family members when the Community Edition is s whopping €175 per year. Some kind of middle ground here would have been nice.
Thanks, the right video at the right time 🙂
one of the critical feature linux based hypervisor (kvm) lack is storage offload API like VAAI
With VMware screwing the pooch I'm betting lots more dev is going to be pushed into KVM based stuff and OEMs of storage solutions are going to want to support it too. Hopefully it comes along!
Good Company 😊
I love Proxmox and XCP-ng as ESXi replacements but with no support for NVIDIA vGPUs for which my clients have spent years developping using NVIDIA architecture and libraries these paltforms are dead in the water. it would be too costly to change to AMD and would require recertification of their platforms from various government bodies and customers from around the world.
For 2D and non-vGPU VDI/VMs then it would work but not for NVIDIA dependant architecture.
I'm old enough to consider containers as full lxc containers, not application containers... I'll leave that to the kiddos and wanna be devops who can neither dev nor ops correcty 🤣 #BOFHinside
More seriously, LXC containers are what they are, initscript leveled containers, they can start sub second so the lack of live migration, depending on haw your application is scaled, is actually not a problem... I run one of the biggest french-speaking tech news outlet infrastructure on a 5 host proxmox cluster+cephRBD+CephFS running 98% of containers (and a few VMs for virtualized network equipement) and it's mostly a breeze (they do use docker for dev, but that's in isolated VMs/VLANs)
Proxmox is great, but it is still NOT ready as an “appliance”!
It does not have a ONE click, ONE file config backup and restore.
Does not come with UPS support. Yes, you can install NUT every time you install Proxmox, but still no GUI for it.
The GUI is very non intuitive and often you have to go to the command line to get stuff done:
Having to edit the boot loader to add support for passthrough is just NOT acceptable in a high caliber “solution”!
are you gonna make a video about harvester? it's a hypervisor from and meant to be integrated with suse's rancher platform
21:31 again with just few commands you can install whole desktop environment in proxmox, I have been using proxmox installed on my primary Laptop and with Debian 12 Desktop environment
Best of Both world, greater OS and Greatest Virtualization solution in on pack
I don't think the complaining about ProxMox not having a 'real console interface' is relevant, especially comparing it to the ESXi limited interface where you could literally just reboot, view underlying hardware and other simple stuff. I mean, what is the difference between typing 'reboot' in the console and 'lshw' for viewing hardware info and using the ESMi menu ?
Hey, would you consider doing a video on an in place migration from ESXi to Proxmox. I like many installed ESXi to get the Guest OS off the hardware, making migration to new hardware a little easier when the time came, now I have a number of boxes that need to go from ESXi to Proxmox VE. How about it?
I'd rather have LXC containers than not. They're essentially a poor man's VM, in that they definitely lack a couple of the features a fully fledged VM would have, but in return they are much more lightweight. If HA and live migration aren't important, they can allow you to run a lot more VM like machines on one physical host.
Though for enterprise use I think Kubernetes would serve most of the same use cases, but better.
Any news on what they plan to do with VM Workstation Pro?
I use Workstation Pro for testing and for keeping older OS's available so I can offer support for older OS's
I have over 100 OS's from desktop to server to Appliances like pfsense
Any word on those licenses or some sort of alternative?
Didn't the part of VMware that handles Workstation just get sold off a couple days ago? I think changes are likely.
@@thegrossmeyerHmmm this is an amazingly strange move.
proxmox raspberry pi ruclips.net/video/oe1_JVl63a0/видео.html Installing Proxmox 8.1 on Raspberry Pi 5
You can live migrate lxc containers! And it works great taking just a couple of seconds.
Since when? Last time I checked LXC could not be live migrated, you had to stop it on one host, migrate and then restart on the other host. VMs you can migrate without even losing a single ping
They don't live migrate, they are stopped, then migrated and started. It happens so quick you might think they live migrate, but they do not. It's important to understand this.
VM's can live migrate, but LXC aren't.
@@mistakek This is correct, just tried this on an LXC and it doesn't keep its uptime.
LXC Live migration only works on UBUNTU LXD. This will come up Proxmox soon
Where I work will probably just pay the cost difference and stay on VMWare; but I’m watching these type videos just in case.
One issue I have is we have a set of host with Nvidia L40 cards in them for AutoDeskand other graphic intensive software. Linux is notoriously horrible or horrendously complicated with PCIe pass through. What I’d like to see covered in one of the type of videos is how these alternatives deal with hardware like video cards.
In version 8, they made PCIE pass through very easy, tons of instructions online. I passed through a couple of ports on a 4 port nic with a few clicks.
@@eilanbarak thank you for this input. Seems I should get one of my dinosaurs out of the garage and install v8 or later and do some testing.
You can use those repos, because they exist and are needed for pve updates.
Instead of disabling, you need to change the apt sources with the community repo.
Is there a "Distributed Switch" like functionality built in, or does it require 3rd party SDN?
If you are setting up a hosting cluster and don't have enough knowledge to use simple cli commands you shouldn't be doing that
Can you do Incus? It looks very promising.