@@NomadsBoyTv I've got it running on a refurbished Dell PowerEdge R710 with dual 3.33GHz Xeon 5680 CPUs and it runs great! I think those machines date from about the 2012 time frame. Now I'm not doing anything extreme on the VMs, but they are always fairly responsive.
@@obfuscateidentity2329 Don't know what you consider 'old hardware' but I got it running on an IBM x3650 M4 for about two years now with no issues at all.
I would "settle" for a live stream of Wendell simply toying around with the software and hardware for this - I'm sure that would cut down the time to make the video... We all just want to see those IBM model M's hitting the terminal while an absurd amount of orange soda is consumed...
I'm pretty sure that would be rather dull. When I tinker with stuff I haven't done before, I'm very introvert, not ready to explain anything, because I haven't quite figured it out yet, so all you see is me being silent, scratching my head from time to time, sigh'ing or saying "hmm?!?" once in a while - basically pretty boring. Once I've figured stuff out and have tried it a few times, I'm ready to explain to others what's what. I'm pretty certain that when Wendell is diving deep into something new and unchartered, trying to figure stuff out, all you're going to see is a guy focusing on stuff in silence, and my bet is that gets pretty dull to watch.
A few things that may help others with context. Please feel free to correct me if I got anything wrong. (Wendell, this is the kind of stuff that would make some great videos, presenting some background on the concepts, terminology, and differentiating features between the various platforms. I think this is what newcomers struggle with when they consider diving in to "all things virtualization". At least that was my experience.) Xen is considered a "Type 1" or "bare metal" hypervisor (see en.wikipedia.org/wiki/Hypervisor). So is VMWare ESXi. Things like Virtualbox, Hyper-V and VMware Workstation are considered "Type 2". Physically, a Xen kernel is what you get when compiling a Linux kernel from the standard sources, having enabled the Xen hypervisor in the configuration file. The kernel is configured to be either the supervisor domain ("domain 0" or "dom0") or a user domain ("domU"), which is where all of the VM's are instanced. (Thus, the hypervisor is not bound intrinsically to a particular Linux distro.) Once a system is booted with a Xen-based kernel configured as "Dom0", it is now capable of being a "Xen Server". Then it is possible to create and run VM's. That's where the tools and such come in. The Xen Project Hypervisor is an official Linux Foundation project. Xen Project Hypervisor originated ca. 2003 and is considered very mature and robust. It is used at very large scale by the likes of AWS, Alibaba, and many others. xenproject.org/users/why-xen/ wiki.xenproject.org/wiki/Xen_Project_Software_Overview XCP-NG (as well as Citrix) is based on the Xen Project Hypervisor. XCP-NG is a "Xen Project Incubation Project" (they said it, not me). Because of their common heritage, it is possible for commercial Citrix customers to do clean migrations from Citrix Server to XCP-NG, ditching the Citrix licensing model. The folks at XCP-NG seem focused on providing more of a turnkey approach to provisioning a Xen-based host and providing strong management tools, support services along with partner and technology integration offerings. XCP-NG is also used at scale by many large customers, as Tom over at Lawrence Systems can attest.
Woooo! Xc-png baby! I've watched a lot of the Lawrence systems videos on it, he's pretty thorough....but I would like to hear more about the different use cases of each of the hyper visors, like a high level view of why you would choose one over the other. This video helped a lot tho,thanks wendell.
We've been using XCP-ng for six months now, on three Dell PE 730 servers, with about 40 VMs in production (with a EMC VNX storage and a couple of Cisco SAN switches). So far it's been working great! We're planning to migrate seven older servers (Dell PE 720) that are running XenServer in the next couple of months.
Another good follow up video would be GPU and PCIe passthrough on XCP-ng, it handles it pretty gracefully compared to alot of other solutions, and even allows for running in a true "headless" mode with the system GPU being made available to VM's after system boot
Woo Hoo!!!!! Great to see Wendell finally covering XCP-ng!!! It has been a go to hypervisor for me for many years now, both for OP lab and enterprise deployment particularly when used with XCP-ng. Been a great way to wheen certain infrastructure away from Citrix and older VMWare infrastructure
Just FYI, stand-alone "Free" Hyper-V is no longer a thing as of Windows Server 2022. Server 2019 is the last version, supported till 2029. Microsoft is pushing Azure Stack HCI instead. All the more reason to use XCP-NG instead, which is way better IMHO anyway. I wouldn't be surprised to see VMWare do the same with ESXi in the future.
That free Hyper-V thing always seemed like a problem waiting to happen, IMO... They were always going to kill it off, leaving you to either pay a small fortune for the not free version to keep things going as it was, or having to painstakingly migrate every VM to another platform. Who was this problem going to hurt most? Small businesses who probably can't afford either solution.
hyper-v sucked anyways.. but in all reality, the FREE version of esxi, is WAY better than xcp-ng.. i've ran xcp-ng in large scale, and i'm telling you.. RUN AWAY.. dont look back, just RUN. horrible.. if you want to just play with vm's, then its fine.. but if you want to run a department, or a company on it.. get ready for a pink slip.. because its gonna fail.
Its always been better off just to run Server than the weird cut down Hyper-V thing, of course theres cost there that for those of us without access to cheap licenses for personal use is an issue however.
@@danratsnapnames i ran xcp-ng in my home lab and no thanks it seemed horrible for that, honestly i switched everything to containers nowdays so i just run ubuntu and kubernetes cluster on top of it and for those 3 VMs that i need i use libvirt
As an organization trying to migrate away from VMware due to insane costs, having tried Hyper-V, Proxmox and Citrix Xen, I still haven't found something nearly as easy to use and stable as VMware, even in the basic "free hypervisor" version without vCenter. Perhaps I should give xcp-ng and Orchestra a try.
Yes, it has the building blocks to possibly be that. Though it is close to Citrix Xen since it is the open part of it. VMWare is still best in class -- don't get me wrong -- but you may be able to get by with something lesser.
@@Level1Techs I fully agree that VMware is the golden standard, however their yearly pricing for vsphere/vcenter is too high, especially on the Epyc platform. I'll give xcp-ng a try in the lab. Thanks for replying!
I run 3 Proxmox nodes with ZFS replication and HA. Live migration is awesome since we only need to transfer RAM data, and a lost node never sets us back more than the replication rule. FreeNAS with with two optanes for zil etc used to be enough but local storage seems to be the way forward. Dont really like our ESX Cluster or HyperV nodes for diffrent reasons. Ovirt never worked right with all our VLANs. Will have to try this solution.
Really? You can do live migrations with local ZFS and replication? Can you elaborate on this? I've never tried this, but as far as the docs goes: "Guests with replication enabled can currently only be migrated offline." But if its possible than that would be really interesting.
@@lebleb8731 Works great. We found it in a change log and where suprised too since the documentation wasnt updated. We have an old Oracle license that only supports two CPUs "in the cluster", including SAN so we made a two single CPU nodes with ZFS replication for that one two. Only need to add a Pi to audit HA to add some security (avoid split brain). Then we built our other cluster to start leaving ESX. 4 SSD in each. Created our ZFS volume with the same name on all nodes and then enabled it on all. Enabled replication and live migration works without problems. Usually less then 50ms stop on hand over.
This is why I no longer call myself a full-stack web developer. Ten years ago it was a resume enhancement - now it's a liability. There's no way I could design a UI and UX, write code for front end and back end, design a database schema, and then deploy through a clusterphuck of CI scripts and tooling to a k8s cluster, and be able to fully master the entire process. I can wrap my head around running Docker through Hyper-V (I prefer working on Windows platform but have to deploy on Linux platforms), running VirtualBox when needed, setting up some basic pipelines via Bitbucket, AWS, Azure, etc. but I lost it the first time I had to setup a k8s cluster. I just can't keep up with all the clustering and virtualization stuff and do the rest of my job effectively at the same time. I know how to use some of the virtualization tools but don't have a phucking clue how they work. 😫
I would be so mad if full-stack dev starts include configuring whole systems and infrastructure Like, I love k8s, but if you are being forced to set up a k8s cluster to start a web server, I feel like something is very backwards. Also, to be honest microservices are supposed to mean, at least to me, less full-stack and more specialized teams of devs.
There are a lot of businesses that seem to hate the fact that they still have to hire tech folk to do things. I've seen waaaaayyy to many job listings that basically summarize to: we need someone to do everything, all at once. Mind you, I don't look at the dev stuff, I look more towards the sysadmin/* engineer stuff. I don't want to learn Golang, Ruby, Python, etc when I know that the software devs will be a million times more proficient with those languages compared to me. I am sure the reverse is true for the software devs who have to stare at a bunch of useless requirements about public cloud certifications and sysadmin stuff, even though they should solely be focused on writing software! Stop with this DevSecAdminDoEverythingOPS and just hire people who are really good at doing what they're supposed to do.
Great video. I do believe XCP hits the mark on lightweight performance and cost. It does provide more flexibility vs. vmware and has all of the performance benefits. Have been running a 3 host cluster 72vcpu / 300 GB RAM for a few years now with NFS/iSCSI and local NVME storage pools. I would love to send some diagrams and throughput tests to various storages to you and lawrence if you need some comparisons.
In terms of the UI on the XCP-NG host machine's _physical_ console, it is a very basic text-based UI (no GUI) and is really meant to be run in a headless configuration. The main management UI is the Xen Orchestra interface, which is typically hosted on a VM running either on the same host, or on some other machine (separate from the XCP-NG host machine). One XO instance can manage very large numbers of XCP-NG hosts and their running VM's. This last bit about the console UI seems to be a major selection criteria for those looking at small or home-lab setups where there is limited gear to run everything on. This seems to be where some may prefer things like Proxmox or others for single-machine setups.
I used the forerunner of XCP-ng , Xen in about 2005 when it still was owned by Cambridge. I used it as a test bed for different projects, but it performed so well we took it into production. Citrix bought it, as we used Citrix anyway, it was a nice add-on. Citrix really didn't do anything with it and when deciding to refresh Servers from IBM to HP Blades and San we looked at VMware. Never had any problems with Xen but did have some issues with IBM, Murphy's law I suppose.
Should do a story time about when I switched over to XCP-ng and implemented XO for the VM backups, and then someone turned the old backup appliance back on. The hilarity that ensued would make for some great content
Fibre channel can do way more than 10s of thousands of IOPS. Maybe you were meaning iSCSI but FC has much less overhead than Ethernet and modern adapters can do 32gbps etc. Then it is up to the SAN itself on disk type etc.
ESXi - not as customizable as full blown linux, but you can install "vib" packages to add additional functionality. These are typically vendor binaries, yet stilll similar to .deb or .rpm packages.
Is the hyperconverged solution for XCP-ng tied to the XOA license or can it work on the community version as well? Also does anyone have any feedback on how well the continuous replication works in a production environment?
I’ve been using hyper-v for years and have been looking very closely at Xcp-np recently, so this video was very interesting. It seems like Microsoft has given up on local servers and wants to push everyone to use there Azure stack instead. I was hoping the latest version of windows server 2022 would have had more new features but the release has been quite underwhelming. I think it’s time to migrate to Xcp-ng.
I've use HCIBench for vSAN performance testing ever since that came out. Nutanix have xRay for their performance testing. I suppose for Ceph, ZFS, or others you can use vdbench or fio? As for changing platforms completely to test and deep dive, not sure if there's an more automated way of doing so. I imagine is hard for hyper-v and vsphere but at least there are books and deep dive for how vsan and s2d works.
Same here. It still seems more powerful option and most configurable, but also a bare to install. Even installing into a K8s cluster doesn't seem that straight forward
I looked at XCP-ng a while back, but ended up moving my home lab to oVirt with hyperconverged Gluster for storage because oVirt basically gives you a wizard for a standard three node setup, and - my impression was at the time - XCP locks XOSAN behind a subscription. This makes me want to revisit XCP again to see if DIY HCI is more accessible than I had thought.
I started to experiment with this about a year ago but wound up going with UnRaid for my home server needs which includes a gaming VM, Dockers, and multiple shares.
UnRaid - Only booting option is usb key. A bad idea. Can not think of using in production environment. Truenas scale ( even in beta ) is much better product than that of UnRaid.
@@mithubopensourcelab482 maybe. You can make a backup USB key and then the OS is running in memory if I understand correctly. For someone like me that is less advanced with Linux and doesn’t have time to learn it works great for my needs.
I built and maintained a vsan cluster myself utilizing cisco high capacity servers. 2 cpu boards and 24 3.5 discs per machine.. a 6 machine cluster ran without issue for 5 years and to my knowledge is still running. The hybrid approach makes managing compute a breeze. Machines hit the local disk most of the time but replicates to the other nodes. Not centralized storage node to maintain.
I've looked into using vSAN for small office scenarios to replace 2 or 3 ESXi hosts with local storage or with a SAN like an MSA or EMC VNXe and migrating to just 3 nodes with local storage and vSAN, but the licensing is off the charts compared to just running a traditional SAN. They should make an essentials plus kit with vSAN that is affordable. But this looks like an interesting alternative.
XCP-ng is great, but one cannot fear the terminal. A lot of things are broken in the Windows GUI application. Pretty sure I'm looking at over 200 days uptime on my home hypervisor running xcp :)
If only you were limited by a, cough, user interface... Most of these solutions you can setup easily by hand, sure you wont have any support but for learning purposes knowing how to setup a hypervisor with pve/kvm manually will safe your ass in the long run!
XCP-NG is really nice, Xen Orchestra however totally not for cheap private/homelab users. Yeah you can build XO yourself, but a lot of features are still not accessible without a subscription.
A 400Gbit card will more than max out an x16 PCIe 4.0 link, which has a total throughput of 252Gb/s. The current 400Gbit cards are actually PCIe 5.0 draft compliant, which isn't super helpful now. AND you can get DUAL PORT 400Gbit cards, which would take a full PCIe 6.0 slot to handle. To put these speeds into perspective, a DDR4 3200 memory channel has 200Gb/s throughput. And it does all this at several 10s of millions of iop/s and single digit nanosecond port-to-port latencies. Those 2u servers are SUPER cool if you only need 1-2. But a SAN starts to make sense again when you fill a rack.
It should be noted that VMware vsan uses minimum 7 gigs per node! In my case with 3x 32GB servers it used an average 11 GB per node. Performance was great but it means less VM can be run.
Well, it is probably going to have pretty good drivers. How good it is a doing KVM style things such as (with editable ACPI Tables abilities) SR-IOV and VMDq, say if you wanted to run a game servers of multiple games (just a few), one game per VM so that you can share a NIC (or a few NICs overlapping VMs) on bunch of VMs (games)? So an example NIC is a Dell i350-T4. So a person might do that so as to have game-servers be online at specific times to cater for people with different timezones (USA then Great Britain, then Australia, whatever) so people (gamers) are not asleep. You could achieve zero downtime whilst the VMs _(like a overlapping staircase of books, acting as a relay race baton handover)_ share a NIC as that NIC is passed to another VM at a sacrifice to bandwidth _(which is totally worthwhile in the pay-offs and trade offs of having zero-downtime)._ A "normal" (non VMDq, SR-IOV) PC cannot do that. It gets a tiny downtime because it has to disconnect a VM from a NIC (or vice versa) as a new VM is switched on (and an old one is switched off) in the chain of events (that relay race baton handover). A normal PC is not seamless like that. It is like a housing sale "buyer chain" of game VMs and NICs. You don't want to get gazumped. These do not need to be games that demand a 3D graphics videocard, btw. Not worried about sharing that. So software renderer games are fine. It is not about the game, but rather the PC VM system sharing VMs on a NIC. Then it can be scaled up later with many a VM and NIC. It could do things other than games (office software, databases, media, etc).
@asdrubale bisanzio Cheers for info. The reason for doing it the way I mentoned above is so that some humble cisco emulation can be done too. I reckon (considering your comment) you immediately fathom the sort of thing I mean for KVM. You can mess with security flaws in some NICs doing that, however, it is mainly for learning/experimenting. I could have added that in my OP comment but it might have made it a bit long. That would either result in people finding it too confusing to read or YT hiding (censoring) my comment (maybe by some algorithm thing). If you type a comment too large in wordcount, the skittish YT seems to see it as infinite monkeys with infinite typewriters so that YT assumes there is either the entire works of Shakespeare in there or something "edgy". It flips a coin at that moment to decide if it is worth a ban. Sort of like quoting Winston Churchill gets you banned. The GrammarNazily App they keep plugging strips out such comments _"for the greater good"._
@asdrubale bisanzio The first one you said is more like what I mean, like virtual routers. Although it is not the full router experience and you can do some extra things though _(like the gaming example is an extra thing you can do, then test against such as even if only using a 1gbps Delli350T4 and then adding infiniband, neither of which pertain to the security flaws but a HP card you are using for ESXi might have such flaws... not that it is about ESXi)._ It is prototypical and very cobbled together, so, for example it is not intended to be power efficent. However it is quieter than a normal rackmount server and cheaper. Also you can pick it up and carry it like a normal PC. Unlike a "real" cisco router, as it is a PC, you can install say MSWindows Server2012 (or swap out a HDD/SSD) temporarily for a day. That is useful if you have 2 or 3 of such PCs, because then you can use different combinations _(of said PC set-ups, like one is cisco and two are server2012, or whatever like say 2 are cisco and one is server2012, or all three are cisco)._ It is for learning mostly (and some functionality) and not intended for replacing a full business environment cisco router. On account of the fact that you can use such a PC for multiple things _(KVM, gaming, server2012 such as some machine learning, or modest video rendering, or GNS3),_ even though it is not especially fast, you can test and try out new setups and then find pragmatic solutions (or things that don't work very well) without needing to invest in final expensive hardware such as a room full or servers and rackmount gear (which would probably also require a building). As it doe snot have all features, you do not need those features and as such do not need to buy a license for them (because you do not have them).
Dear sir, im using ibm server X3620,and i'm using citrix xenserver 7.6, my server for shutdown due to the power issue, after i try to boot the server its booting was successfully complied, but it show the error no network configured and i try to configure it show the error no interface present. and all my VM is not show and its show there is no VM. kindly help to fix this issue without data loss.
RDMAoIB is your iops solution, cheaper than ethernet and accesstimes measured in ns and it scales much nicer with multi tier storage, well that's my two cents at least.
Specialised NICs and vender locked NICs is my only, and I mean only because IB is cool, problem with infiniband. Well that and security over the wire is still a little bit iffy to me :p
what would you recommend for vm for a single 16 core ryzen system with 128gb ram? like a personal vm system. proxmox or xng? I suppose storage or snapshots could be backed up to another machine or nas so it doesnt need like constant replication.
I wish I could run something like this Tyan machine at work. I am just begging for a new Dell server. I think something like this would freak out my IT people.
hi. i cant connect to our xcp hosts either with center or XO. the only thing i can do is to ssh to the hosts, thats it=( anyone ever experience this? (i can ping the internet and any other machine in thr network from the hosts)
I have been looking to build another vm server. ESXI can be had easy on ebay for cheep but I fear compatibility. My current vm system is running windows server with hyper-v. I was using an esxi hp 2u server that was ewaste from work but it died. This may be a nice option for a custom vm server build.
Yea he just made us all look like we playing with cap guns and he sitting there with a Plasma Rail gun from alpha centauri. My HPDL380P looking real weak right about now
Love the product but sadly twice across two different versions and two different pieces of hardware I've had servers go into maintenance mode with seemingly no way to recover them try as I might.
Great video and overview of the XCP-NG design philosophy. Looking forward to future videos on the topic.
You two should definitely do a few collab videos on this topic
definitely would be a blast to watch
You are 'the' authority to talk on XCP-NG.
I love love love xcp-ng. I'm glad it's getting some attention.
Will it run on old hardware?
@@obfuscateidentity2329 my experience yes as long as your processor can support virtualization.
It's a really popular system we have a lot of clients using it.
@@NomadsBoyTv I've got it running on a refurbished Dell PowerEdge R710 with dual 3.33GHz Xeon 5680 CPUs and it runs great! I think those machines date from about the 2012 time frame. Now I'm not doing anything extreme on the VMs, but they are always fairly responsive.
@@obfuscateidentity2329 Don't know what you consider 'old hardware' but I got it running on an IBM x3650 M4 for about two years now with no issues at all.
Man the comment section on Level1Linux videos are just as good as the videos!
I would "settle" for a live stream of Wendell simply toying around with the software and hardware for this - I'm sure that would cut down the time to make the video...
We all just want to see those IBM model M's hitting the terminal while an absurd amount of orange soda is consumed...
I'm pretty sure that would be rather dull. When I tinker with stuff I haven't done before, I'm very introvert, not ready to explain anything, because I haven't quite figured it out yet, so all you see is me being silent, scratching my head from time to time, sigh'ing or saying "hmm?!?" once in a while - basically pretty boring. Once I've figured stuff out and have tried it a few times, I'm ready to explain to others what's what. I'm pretty certain that when Wendell is diving deep into something new and unchartered, trying to figure stuff out, all you're going to see is a guy focusing on stuff in silence, and my bet is that gets pretty dull to watch.
There is a wendell live stream of setting up Vfio on Fedora. Watched it quite a few times. His brain is very interesting
Nice shout-out to Tom he does provide ALOT info in the form of tutorials and knowledgeable about home server lab setups!
We try to 😀
A few things that may help others with context. Please feel free to correct me if I got anything wrong.
(Wendell, this is the kind of stuff that would make some great videos, presenting some background on the concepts, terminology, and differentiating features between the various platforms. I think this is what newcomers struggle with when they consider diving in to "all things virtualization". At least that was my experience.)
Xen is considered a "Type 1" or "bare metal" hypervisor (see en.wikipedia.org/wiki/Hypervisor).
So is VMWare ESXi. Things like Virtualbox, Hyper-V and VMware Workstation are considered "Type 2".
Physically, a Xen kernel is what you get when compiling a Linux kernel from the standard sources, having enabled the Xen hypervisor in the configuration file. The kernel is configured to be either the supervisor domain ("domain 0" or "dom0") or a user domain ("domU"), which is where all of the VM's are instanced. (Thus, the hypervisor is not bound intrinsically to a particular Linux distro.)
Once a system is booted with a Xen-based kernel configured as "Dom0", it is now capable of being a "Xen Server". Then it is possible to create and run VM's. That's where the tools and such come in.
The Xen Project Hypervisor is an official Linux Foundation project. Xen Project Hypervisor originated ca. 2003 and is considered very mature and robust. It is used at very large scale by the likes of AWS, Alibaba, and many others.
xenproject.org/users/why-xen/
wiki.xenproject.org/wiki/Xen_Project_Software_Overview
XCP-NG (as well as Citrix) is based on the Xen Project Hypervisor.
XCP-NG is a "Xen Project Incubation Project" (they said it, not me).
Because of their common heritage, it is possible for commercial Citrix customers to do clean migrations from Citrix Server to XCP-NG, ditching the Citrix licensing model.
The folks at XCP-NG seem focused on providing more of a turnkey approach to provisioning a Xen-based host and providing strong management tools, support services along with partner and technology integration offerings. XCP-NG is also used at scale by many large customers, as Tom over at Lawrence Systems can attest.
The proxmox ceph solution is very cost effective. You don’t even need to pay for that part.
Woooo! Xc-png baby! I've watched a lot of the Lawrence systems videos on it, he's pretty thorough....but I would like to hear more about the different use cases of each of the hyper visors, like a high level view of why you would choose one over the other. This video helped a lot tho,thanks wendell.
I think it would be really cool to see a L1 video about ceph/rook/longhorn
We've been using XCP-ng for six months now, on three Dell PE 730 servers, with about 40 VMs in production (with a EMC VNX storage and a couple of Cisco SAN switches). So far it's been working great! We're planning to migrate seven older servers (Dell PE 720) that are running XenServer in the next couple of months.
Another good follow up video would be GPU and PCIe passthrough on XCP-ng, it handles it pretty gracefully compared to alot of other solutions, and even allows for running in a true "headless" mode with the system GPU being made available to VM's after system boot
Where are good docs on this passthrough technology?
Woo Hoo!!!!! Great to see Wendell finally covering XCP-ng!!! It has been a go to hypervisor for me for many years now, both for OP lab and enterprise deployment particularly when used with XCP-ng. Been a great way to wheen certain infrastructure away from Citrix and older VMWare infrastructure
Just FYI, stand-alone "Free" Hyper-V is no longer a thing as of Windows Server 2022. Server 2019 is the last version, supported till 2029. Microsoft is pushing Azure Stack HCI instead. All the more reason to use XCP-NG instead, which is way better IMHO anyway. I wouldn't be surprised to see VMWare do the same with ESXi in the future.
That free Hyper-V thing always seemed like a problem waiting to happen, IMO... They were always going to kill it off, leaving you to either pay a small fortune for the not free version to keep things going as it was, or having to painstakingly migrate every VM to another platform.
Who was this problem going to hurt most? Small businesses who probably can't afford either solution.
@@tin2001 microsoft would suggest "just go to the cloud"
hyper-v sucked anyways.. but in all reality, the FREE version of esxi, is WAY better than xcp-ng.. i've ran xcp-ng in large scale, and i'm telling you.. RUN AWAY.. dont look back, just RUN. horrible.. if you want to just play with vm's, then its fine.. but if you want to run a department, or a company on it.. get ready for a pink slip.. because its gonna fail.
Its always been better off just to run Server than the weird cut down Hyper-V thing, of course theres cost there that for those of us without access to cheap licenses for personal use is an issue however.
@@danratsnapnames i ran xcp-ng in my home lab and no thanks it seemed horrible for that, honestly i switched everything to containers nowdays so i just run ubuntu and kubernetes cluster on top of it and for those 3 VMs that i need i use libvirt
As an organization trying to migrate away from VMware due to insane costs, having tried Hyper-V, Proxmox and Citrix Xen, I still haven't found something nearly as easy to use and stable as VMware, even in the basic "free hypervisor" version without vCenter. Perhaps I should give xcp-ng and Orchestra a try.
Yes, it has the building blocks to possibly be that. Though it is close to Citrix Xen since it is the open part of it. VMWare is still best in class -- don't get me wrong -- but you may be able to get by with something lesser.
@@Level1Techs I fully agree that VMware is the golden standard, however their yearly pricing for vsphere/vcenter is too high, especially on the Epyc platform. I'll give xcp-ng a try in the lab. Thanks for replying!
Used VMware and Proxmox. Proxmox is very much easy than that of VMWare.
I find that Orchestra is sloppy and on par with proxmox the XCP-ng center is older but way better
I run 3 Proxmox nodes with ZFS replication and HA. Live migration is awesome since we only need to transfer RAM data, and a lost node never sets us back more than the replication rule. FreeNAS with with two optanes for zil etc used to be enough but local storage seems to be the way forward.
Dont really like our ESX Cluster or HyperV nodes for diffrent reasons. Ovirt never worked right with all our VLANs.
Will have to try this solution.
Really? You can do live migrations with local ZFS and replication? Can you elaborate on this? I've never tried this, but as far as the docs goes: "Guests with replication enabled can currently only be migrated offline." But if its possible than that would be really interesting.
@@lebleb8731 Works great. We found it in a change log and where suprised too since the documentation wasnt updated.
We have an old Oracle license that only supports two CPUs "in the cluster", including SAN so we made a two single CPU nodes with ZFS replication for that one two. Only need to add a Pi to audit HA to add some security (avoid split brain).
Then we built our other cluster to start leaving ESX. 4 SSD in each. Created our ZFS volume with the same name on all nodes and then enabled it on all. Enabled replication and live migration works without problems. Usually less then 50ms stop on hand over.
_engagement as always_
I'm excited. Will have to play with this for my demo center.
This is why I no longer call myself a full-stack web developer. Ten years ago it was a resume enhancement - now it's a liability. There's no way I could design a UI and UX, write code for front end and back end, design a database schema, and then deploy through a clusterphuck of CI scripts and tooling to a k8s cluster, and be able to fully master the entire process. I can wrap my head around running Docker through Hyper-V (I prefer working on Windows platform but have to deploy on Linux platforms), running VirtualBox when needed, setting up some basic pipelines via Bitbucket, AWS, Azure, etc. but I lost it the first time I had to setup a k8s cluster. I just can't keep up with all the clustering and virtualization stuff and do the rest of my job effectively at the same time. I know how to use some of the virtualization tools but don't have a phucking clue how they work. 😫
I would be so mad if full-stack dev starts include configuring whole systems and infrastructure
Like, I love k8s, but if you are being forced to set up a k8s cluster to start a web server, I feel like something is very backwards.
Also, to be honest microservices are supposed to mean, at least to me, less full-stack and more specialized teams of devs.
There are a lot of businesses that seem to hate the fact that they still have to hire tech folk to do things. I've seen waaaaayyy to many job listings that basically summarize to: we need someone to do everything, all at once.
Mind you, I don't look at the dev stuff, I look more towards the sysadmin/* engineer stuff. I don't want to learn Golang, Ruby, Python, etc when I know that the software devs will be a million times more proficient with those languages compared to me.
I am sure the reverse is true for the software devs who have to stare at a bunch of useless requirements about public cloud certifications and sysadmin stuff, even though they should solely be focused on writing software!
Stop with this DevSecAdminDoEverythingOPS and just hire people who are really good at doing what they're supposed to do.
I would love for you to do a video on MaaS. This really reminded me of a less automation-focused version of that.
I don’t get anything but Wendell is so cool
Great video. I do believe XCP hits the mark on lightweight performance and cost. It does provide more flexibility vs. vmware and has all of the performance benefits. Have been running a 3 host cluster 72vcpu / 300 GB RAM for a few years now with NFS/iSCSI and local NVME storage pools. I would love to send some diagrams and throughput tests to various storages to you and lawrence if you need some comparisons.
In terms of the UI on the XCP-NG host machine's _physical_ console, it is a very basic text-based UI (no GUI) and is really meant to be run in a headless configuration. The main management UI is the Xen Orchestra interface, which is typically hosted on a VM running either on the same host, or on some other machine (separate from the XCP-NG host machine). One XO instance can manage very large numbers of XCP-NG hosts and their running VM's.
This last bit about the console UI seems to be a major selection criteria for those looking at small or home-lab setups where there is limited gear to run everything on. This seems to be where some may prefer things like Proxmox or others for single-machine setups.
I used the forerunner of XCP-ng , Xen in about 2005 when it still was owned by Cambridge. I used it as a test bed for different projects, but it performed so well we took it into production. Citrix bought it, as we used Citrix anyway, it was a nice add-on. Citrix really didn't do anything with it and when deciding to refresh Servers from IBM to HP Blades and San we looked at VMware. Never had any problems with Xen but did have some issues with IBM, Murphy's law I suppose.
Good stuff. Learning as much as I can about xcp-ng for a homelab build and this clarifies some things. Thanks
What does Level1 use for their Virtualization for daily use?
Great content Wendell, I appreciate any videos coming
KVM kinda works great for me..
sooo..
And i don't have any server grade hardware.
Should do a story time about when I switched over to XCP-ng and implemented XO for the VM backups, and then someone turned the old backup appliance back on. The hilarity that ensued would make for some great content
Fibre channel can do way more than 10s of thousands of IOPS. Maybe you were meaning iSCSI but FC has much less overhead than Ethernet and modern adapters can do 32gbps etc. Then it is up to the SAN itself on disk type etc.
ESXi - not as customizable as full blown linux, but you can install "vib" packages to add additional functionality. These are typically vendor binaries, yet stilll similar to .deb or .rpm packages.
Is the hyperconverged solution for XCP-ng tied to the XOA license or can it work on the community version as well? Also does anyone have any feedback on how well the continuous replication works in a production environment?
I would really like to see how you benchmark iops in your systems. Storage performance seams to be one of the most ignored performance metrics.
Woo Linux content!
This was very interesting and really useful. This hasn't popped up on my radar yet and I will definitely be looking into this.
I’ve been using hyper-v for years and have been looking very closely at Xcp-np recently, so this video was very interesting. It seems like Microsoft has given up on local servers and wants to push everyone to use there Azure stack instead. I was hoping the latest version of windows server 2022 would have had more new features but the release has been quite underwhelming. I think it’s time to migrate to Xcp-ng.
I've use HCIBench for vSAN performance testing ever since that came out. Nutanix have xRay for their performance testing. I suppose for Ceph, ZFS, or others you can use vdbench or fio? As for changing platforms completely to test and deep dive, not sure if there's an more automated way of doing so. I imagine is hard for hyper-v and vsphere but at least there are books and deep dive for how vsan and s2d works.
Would be interesting to hear about OpenStack and if it's even viable solution for home lab
In my opinion out of all the platforms Openstack takes the most effort to get up and running.
Same here.
It still seems more powerful option and most configurable, but also a bare to install.
Even installing into a K8s cluster doesn't seem that straight forward
I looked at XCP-ng a while back, but ended up moving my home lab to oVirt with hyperconverged Gluster for storage because oVirt basically gives you a wizard for a standard three node setup, and - my impression was at the time - XCP locks XOSAN behind a subscription. This makes me want to revisit XCP again to see if DIY HCI is more accessible than I had thought.
Expensive server room gear is always really good to see and to drool for.
This seems like a DIY Nutanix blade server. I want one of these in my server cab so bad.
Awesome Wendell. Love the content.
Eagerly awaiting more videos!
I started to experiment with this about a year ago but wound up going with UnRaid for my home server needs which includes a gaming VM, Dockers, and multiple shares.
UnRaid - Only booting option is usb key. A bad idea. Can not think of using in production environment. Truenas scale ( even in beta ) is much better product than that of UnRaid.
@@mithubopensourcelab482 maybe. You can make a backup USB key and then the OS is running in memory if I understand correctly. For someone like me that is less advanced with Linux and doesn’t have time to learn it works great for my needs.
I built and maintained a vsan cluster myself utilizing cisco high capacity servers. 2 cpu boards and 24 3.5 discs per machine.. a 6 machine cluster ran without issue for 5 years and to my knowledge is still running. The hybrid approach makes managing compute a breeze. Machines hit the local disk most of the time but replicates to the other nodes. Not centralized storage node to maintain.
Xen can be used to load VM into RAM and remove local storage. Very very useful when combined with zfs/snapshots.
Per node we are getting about 1-2M IOps per node (depending on the type) with vSAN on 40G. Have been for about 5 years.
I wanna like this viddio five times. I am so happy!
Would love a video for bcache running with one of those Optanes.
How does this compare to RHV/Ovirt? looks to be a similar idea
I've looked into using vSAN for small office scenarios to replace 2 or 3 ESXi hosts with local storage or with a SAN like an MSA or EMC VNXe and migrating to just 3 nodes with local storage and vSAN, but the licensing is off the charts compared to just running a traditional SAN.
They should make an essentials plus kit with vSAN that is affordable. But this looks like an interesting alternative.
I wish Beowulf clusters were still a thing. That's the only kind of virtualization/clustering that I find interesting.
they were abandoned by their originators because modern cpus and systems made them useless
XCP-ng is great, but one cannot fear the terminal. A lot of things are broken in the Windows GUI application. Pretty sure I'm looking at over 200 days uptime on my home hypervisor running xcp :)
If only you were limited by a, cough, user interface...
Most of these solutions you can setup easily by hand, sure you wont have any support but for learning purposes knowing how to setup a hypervisor with pve/kvm manually will safe your ass in the long run!
Especially with Xen :)
Now that vmware was bought out by broadcom and they openly stated that licencing will change, this video is more relevant.
keen to see more!
Been using xcp-ng for a while now.
Where do I find the "unlocked, build your own version" documentation?
It's called XenOrchestraInstallerUpdater. Lawrence Systems has a video on it: ruclips.net/video/bq1iKO-0jWs/видео.html
XCP-NG is really nice, Xen Orchestra however totally not for cheap private/homelab users. Yeah you can build XO yourself, but a lot of features are still not accessible without a subscription.
A 400Gbit card will more than max out an x16 PCIe 4.0 link, which has a total throughput of 252Gb/s. The current 400Gbit cards are actually PCIe 5.0 draft compliant, which isn't super helpful now. AND you can get DUAL PORT 400Gbit cards, which would take a full PCIe 6.0 slot to handle.
To put these speeds into perspective, a DDR4 3200 memory channel has 200Gb/s throughput.
And it does all this at several 10s of millions of iop/s and single digit nanosecond port-to-port latencies.
Those 2u servers are SUPER cool if you only need 1-2. But a SAN starts to make sense again when you fill a rack.
Feeding the algorithm with like/sub/bell and comment.
PSA: make a hotkey to help👌
It should be noted that VMware vsan uses minimum 7 gigs per node! In my case with 3x 32GB servers it used an average 11 GB per node. Performance was great but it means less VM can be run.
Well, it is probably going to have pretty good drivers.
How good it is a doing KVM style things such as (with editable ACPI Tables abilities) SR-IOV and VMDq, say if you wanted to run a game servers of multiple games (just a few), one game per VM so that you can share a NIC (or a few NICs overlapping VMs) on bunch of VMs (games)? So an example NIC is a Dell i350-T4.
So a person might do that so as to have game-servers be online at specific times to cater for people with different timezones (USA then Great Britain, then Australia, whatever) so people (gamers) are not asleep. You could achieve zero downtime whilst the VMs _(like a overlapping staircase of books, acting as a relay race baton handover)_ share a NIC as that NIC is passed to another VM at a sacrifice to bandwidth _(which is totally worthwhile in the pay-offs and trade offs of having zero-downtime)._
A "normal" (non VMDq, SR-IOV) PC cannot do that. It gets a tiny downtime because it has to disconnect a VM from a NIC (or vice versa) as a new VM is switched on (and an old one is switched off) in the chain of events (that relay race baton handover). A normal PC is not seamless like that. It is like a housing sale "buyer chain" of game VMs and NICs. You don't want to get gazumped.
These do not need to be games that demand a 3D graphics videocard, btw. Not worried about sharing that. So software renderer games are fine. It is not about the game, but rather the PC VM system sharing VMs on a NIC. Then it can be scaled up later with many a VM and NIC. It could do things other than games (office software, databases, media, etc).
@asdrubale bisanzio Cheers for info. The reason for doing it the way I mentoned above is so that some humble cisco emulation can be done too. I reckon (considering your comment) you immediately fathom the sort of thing I mean for KVM. You can mess with security flaws in some NICs doing that, however, it is mainly for learning/experimenting.
I could have added that in my OP comment but it might have made it a bit long. That would either result in people finding it too confusing to read or YT hiding (censoring) my comment (maybe by some algorithm thing).
If you type a comment too large in wordcount, the skittish YT seems to see it as infinite monkeys with infinite typewriters so that YT assumes there is either the entire works of Shakespeare in there or something "edgy". It flips a coin at that moment to decide if it is worth a ban. Sort of like quoting Winston Churchill gets you banned. The GrammarNazily App they keep plugging strips out such comments _"for the greater good"._
@asdrubale bisanzio The first one you said is more like what I mean, like virtual routers. Although it is not the full router experience and you can do some extra things though _(like the gaming example is an extra thing you can do, then test against such as even if only using a 1gbps Delli350T4 and then adding infiniband, neither of which pertain to the security flaws but a HP card you are using for ESXi might have such flaws... not that it is about ESXi)._ It is prototypical and very cobbled together, so, for example it is not intended to be power efficent. However it is quieter than a normal rackmount server and cheaper. Also you can pick it up and carry it like a normal PC. Unlike a "real" cisco router, as it is a PC, you can install say MSWindows Server2012 (or swap out a HDD/SSD) temporarily for a day. That is useful if you have 2 or 3 of such PCs, because then you can use different combinations _(of said PC set-ups, like one is cisco and two are server2012, or whatever like say 2 are cisco and one is server2012, or all three are cisco)._
It is for learning mostly (and some functionality) and not intended for replacing a full business environment cisco router.
On account of the fact that you can use such a PC for multiple things _(KVM, gaming, server2012 such as some machine learning, or modest video rendering, or GNS3),_ even though it is not especially fast, you can test and try out new setups and then find pragmatic solutions (or things that don't work very well) without needing to invest in final expensive hardware such as a room full or servers and rackmount gear (which would probably also require a building). As it doe snot have all features, you do not need those features and as such do not need to buy a license for them (because you do not have them).
Nice I have some hardware and need to upgrade my all in one box since I'm running out of space on my super old freenas box.
If, my homelab, only consist of an old PC, with only 8GB of RAM, two cheap SSD, and used 1TB laptop hdd, should I use xcp-ng?
Looks really nice.
Wait is that een dutch book in the background? I didn't know you could read in dutch :O
Another HCI that is up and coming is Rancher's Harvester
Great video!!
hmm...I wonder how this compares to TrueNAS Scale ... looks like kindof the same.
You're referring to storage spaces direct, correct? I've heard you can get stupid IOPS through it.
Fuck me, I wish I had money for that tyan server, that looks awesome!
There are older models that aren't too crazy. Ryan, Supermicro, Dell to name a few.
IOps over 1000000, going super SAN!!
Dear sir,
im using ibm server X3620,and i'm using citrix xenserver 7.6, my server for shutdown due to the power issue, after i try to boot the server its booting was successfully complied, but it show the error no network configured and i try to configure it show the error no interface present. and all my VM is not show and its show there is no VM. kindly help to fix this issue without data loss.
OMG.. That EPYC cluster .. drool
8:04 "The whole enchilada"
you should do a video on how to activate fsr on all proton supported games 😉😉
What Tyan Server is this? Where can I get one?
Ha! I just noticed I/O ops/s would be ioops if one of the os didn't go away.
I skimmed over the title quickly and read it as 'XKCD' and I...had questions...
Hey Do you know how to install MacOS on XCP-ng
RDMAoIB is your iops solution, cheaper than ethernet and accesstimes measured in ns and it scales much nicer with multi tier storage, well that's my two cents at least.
Specialised NICs and vender locked NICs is my only, and I mean only because IB is cool, problem with infiniband. Well that and security over the wire is still a little bit iffy to me :p
what would you recommend for vm for a single 16 core ryzen system with 128gb ram? like a personal vm system. proxmox or xng? I suppose storage or snapshots could be backed up to another machine or nas so it doesnt need like constant replication.
proxmox please run away from XCP-NG its horrible
@@bigpod please back up your opinion and experiences
@@bigpod why ?
A video on HCI and no mention of Nutanix.
I wish I could run something like this Tyan machine at work. I am just begging for a new Dell server. I think something like this would freak out my IT people.
Dell makes one also, don't remember the model but STH did a video on it. Up to 4 nodes with dual 64 core Epyc.
All that talk about fancy SAN systems and no word about what Ceph does?
Ceph was mentioned at east once.
@@HiltonT69 That's why I just edited to clarify, what exactly I meant.
If vSAN provided a way to do a three node cluster without a switch or witness my life would be dramatically easier
Someone said it. I say again. XCP .. does it scale?
hi. i cant connect to our xcp hosts either with center or XO. the only thing i can do is to ssh to the hosts, thats it=( anyone ever experience this? (i can ping the internet and any other machine in thr network from the hosts)
Oh boy now I feel like I play with only the best human toys. ;)
I have been looking to build another vm server. ESXI can be had easy on ebay for cheep but I fear compatibility. My current vm system is running windows server with hyper-v. I was using an esxi hp 2u server that was ewaste from work but it died. This may be a nice option for a custom vm server build.
Yea he just made us all look like we playing with cap guns and he sitting there with a Plasma Rail gun from alpha centauri. My HPDL380P looking real weak right about now
Testing and ... and that kind of thing.
For the kids: (shh: the ellipses hide the interesting parts.)
Love the product but sadly twice across two different versions and two different pieces of hardware I've had servers go into maintenance mode with seemingly no way to recover them try as I might.
P45 hundred (epik mulan joke)
i just found this wtf have i been doing all this time
With vmware being bought by Broadcom time to look at other solutions
Are you wearing a dog collar?
Still believing.
more linux channel vid's
Miller Margaret Jackson Laura Johnson Frank
OT: RUclips advertising is stupid and infantile.