As an option, in order not to disable SYNC in NFS, you can enable the cache on virtual disk (for example, Proxmox allows this). This gives better read and write performance.
I've always used NFS for hypervisor storage sharing when the option has been available. It's simple and has thin provisioning out of the box. I've also never seen any performance issues with the production workloads. The convenience and supportability of NFS has over time been more beneficial than any performance gains iSCSI or other methods have provided, in my experience.
Recently i was helping my friend to set up truenas enterprise that they company bought. It is connected to two switches that running mlag between them over 2x40G qsfp (mikrotik CRS326-24S+2Q+RM) yes mikrotik and works excellent for holding l2 over mlag. On truenas was bond 2X10G to each of switch and create 4 vlan tags over that: -mgmt vlan -nfs vlan proxmox for vm storage -nfs vlan vm nsf shares for db puroposes (vm is only boot drive and nfs mount is for db data and logging) with optimized block sizes -smb vlan And have restrictions for each segment per ip, and fw isolation between vlan so nothing cannot talk with each other. Was do not noticed any slowdowns regarding additional tagging. Running all on 1500 MTU but will try jumbo frame scenario also. Truenas backup server for replication is ongoing process , it will be truenas scale vm with HBA passthrough. Was setup this after inspirations from your truenas video series, thank you for all of that You are true truenas ambasador :)
Great Timing! i was just about to do something like that. I deployed a Trunas and I want to create 2 dataset: 1 for xcpng vm storage to set HA and one for the file share of the AD to manage with the file server role on windows server. Do you have anything to suggest for the latter? When adding the NFS share to xcpng though it returns a "SR_BACKEND_FAILURE_73(, NFS mount error [opterr=mount failed with return code 32], )" error, i think i gotta dig deep. EDIT: I thought it was going to be harder, for TrueNAS core apparently the "All dirs" checkbox in the NFS share settings has to be selected because xcpng like to mount subdirs.
Lawrence, what about minIO in the mix.. please do a 3 box setup talk buddy (xcp-ng + truenas + minIO ). I think this is a solid enterprise level core setup.
@@LAWRENCESYSTEMS will look at it buddy, but can you work on something to explain how it works in terms of convergence and backup solutions. But also the fact that lots of tech (databases included) are now reading straight to s3.
I figured this issue out. My backup server was running the Dragonfish beta. my main server was on the normal. When I upgraded my main server to Dragonfish RC it started working. Interesting
if you use compress on the dataset if=/dev/zero will make the writes almost non existent (probably better would be /dev/random)- my dd shows 3GB/s on SSD pool (dev/random shows 260MB/s) and 1GB/s on HDD pool (dev/random shows 250MB/s), but what do I know ;-)
MAC or IP filtering are but a mechanism to prevent automated systems from connecting to resources you don't want them to. It's so trivial to circumvent, and you shouldn't consider it to be any form of security feature, at all. It's not even worth considering. If you want security, you need authentication, which you simply can't do with NFS. NFS is not secure, and you should never use it over a hostile network. It's best to think of NFS and iSCSI like direct attached storage, but over a network. It has zero protection from attack. Anybody can easily spoof a MAC, IP or User ID. Especially, when it's all transmitted in the plaintext. It's like hiding your SSID. These things are to prevent machines from talking to things you don't want them to, unless they are purposely programmed to do so. They are not secure.
@@LAWRENCESYSTEMS TrueNAS is Debian 12 after all! But vanilla Debian and TrueNAS each have their own purpose. This question is like asking Hammer vs Drill, different tools for different jobs.
This is an awesome video. I know it's aimed at XCP-ng users, but the TrueNAS side of things is a great starting point as well. In particular, TrueNAS Scale, as of Wednesday, June 26, 2024, doesn't make it obvious in the UI that the generic template is the best dataset template for NFS use. Since NFS is meant for *nix filesystems and the generic permissions are generic unix permissions, it's implied, but if you don't know that, it's not super-clear. One of those little details that makes the learning curve a bit steeper than it needs to be, but that will hopefully be fixed with a minor adjustment to the UI someday. :) Though, I'm curious about one point: was there a reason you avoided using ACLs for the NFS share?
@@LAWRENCESYSTEMS This is what I get with Sync disable on 10G link: 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 4.537 s, 947 MB/s , seems you are getting only around 500 MB/s , I think something not quite right with your XCP-ng or you are running slow pool.
@@LAWRENCESYSTEMS I get wirespeed over NFS from Proxmox to Truenas with sync disabled on a 10G network using your script. Any reason XCP-ng is only around 500 MB/s. Are these your nested XCP-ng hosts , maybe that impacts performance slightly...
Hi Tom! Thanks for a new great video and your contribution to the popularity of XCP-ng. When you showed your network adapters attached to the storage network, I noticed that the MTU is set to 1500 bytes. Did you try to set it to 9000 bytes? Was there some performance advantage? Sorry if you already answered to this question in another video.
Ya, 9000 Jumbo Frames on the LAN can boost NFS a lot since it means NFS traffic fits in a single packet now (8192 bytes). It greatly boosts the NFS efficiency.
NFS can either have synchronous or asynchronous writes via the NFS client. It seems not all softwares/ vendors have the same defaults so there's a chance whatever NFS client is writing to an NFS share can also have sync or a sync enabled or disabled. Do you know what XCPNG does by default? If it does synchronous by default then it may not even be necessary on the NAS
So because I am hosting Truenas Scale Xen Orchestra web client on my server which has the same IP as my nfs share it wont work. Any work arounds? I tried this on my backup sever and it worked right away.
As an option, in order not to disable SYNC in NFS, you can enable the cache on virtual disk (for example, Proxmox allows this). This gives better read and write performance.
Thanks for making these TrueNAS Scale tutorials ❤
Tutorials Time. Nice.
I've always used NFS for hypervisor storage sharing when the option has been available. It's simple and has thin provisioning out of the box. I've also never seen any performance issues with the production workloads. The convenience and supportability of NFS has over time been more beneficial than any performance gains iSCSI or other methods have provided, in my experience.
Super helpful video Tom. Separate storage network is always high on my list. Thanks for the pros and cons of the NFS syncing.
Thank you for creating this!
Recently i was helping my friend to set up truenas enterprise that they company bought.
It is connected to two switches that running mlag between them over 2x40G qsfp (mikrotik CRS326-24S+2Q+RM) yes mikrotik and works excellent for holding l2 over mlag.
On truenas was bond 2X10G to each of switch and create 4 vlan tags over that:
-mgmt vlan
-nfs vlan proxmox for vm storage
-nfs vlan vm nsf shares for db puroposes (vm is only boot drive and nfs mount is for db data and logging) with optimized block sizes
-smb vlan
And have restrictions for each segment per ip, and fw isolation between vlan so nothing cannot talk with each other.
Was do not noticed any slowdowns regarding additional tagging.
Running all on 1500 MTU but will try jumbo frame scenario also.
Truenas backup server for replication is ongoing process , it will be truenas scale vm with HBA
passthrough.
Was setup this after inspirations from your truenas video series, thank you for all of that
You are true truenas ambasador :)
Thank you for this!
You are the best thank you!
Exactly what I needed! I just asked you this question yesterday... WOW!
Thanks so much!
Great Timing! i was just about to do something like that. I deployed a Trunas and I want to create 2 dataset: 1 for xcpng vm storage to set HA and one for the file share of the AD to manage with the file server role on windows server.
Do you have anything to suggest for the latter?
When adding the NFS share to xcpng though it returns a "SR_BACKEND_FAILURE_73(, NFS mount error [opterr=mount failed with return code 32], )" error, i think i gotta dig deep.
EDIT: I thought it was going to be harder, for TrueNAS core apparently the "All dirs" checkbox in the NFS share settings has to be selected because xcpng like to mount subdirs.
Lawrence, what about minIO in the mix.. please do a 3 box setup talk buddy (xcp-ng + truenas + minIO ). I think this is a solid enterprise level core setup.
I have a Minio video ruclips.net/video/uIm41PhGEgQ/видео.html
@@LAWRENCESYSTEMS will look at it buddy, but can you work on something to explain how it works in terms of convergence and backup solutions. But also the fact that lots of tech (databases included) are now reading straight to s3.
Running Xen Orchestra on Truenas Scale app, has the same IP as truenas scale and I can't connect NFS Share because of same IP
I figured this issue out. My backup server was running the Dragonfish beta. my main server was on the normal. When I upgraded my main server to Dragonfish RC it started working. Interesting
Interesting, i never considered this 🤔
if you use compress on the dataset if=/dev/zero will make the writes almost non existent (probably better would be /dev/random)- my dd shows 3GB/s on SSD pool (dev/random shows 260MB/s) and 1GB/s on HDD pool (dev/random shows 250MB/s),
but what do I know ;-)
MAC or IP filtering are but a mechanism to prevent automated systems from connecting to resources you don't want them to. It's so trivial to circumvent, and you shouldn't consider it to be any form of security feature, at all. It's not even worth considering. If you want security, you need authentication, which you simply can't do with NFS. NFS is not secure, and you should never use it over a hostile network. It's best to think of NFS and iSCSI like direct attached storage, but over a network. It has zero protection from attack. Anybody can easily spoof a MAC, IP or User ID. Especially, when it's all transmitted in the plaintext. It's like hiding your SSID. These things are to prevent machines from talking to things you don't want them to, unless they are purposely programmed to do so. They are not secure.
Well, as I suggested, NFS should be on it's own network.
@@LAWRENCESYSTEMS Very true.
TrueNAS to Debian. What your opinion?
I like TrueNAS and I like Debian.
@@LAWRENCESYSTEMS TrueNAS is Debian 12 after all! But vanilla Debian and TrueNAS each have their own purpose.
This question is like asking Hammer vs Drill, different tools for different jobs.
This is an awesome video. I know it's aimed at XCP-ng users, but the TrueNAS side of things is a great starting point as well.
In particular, TrueNAS Scale, as of Wednesday, June 26, 2024, doesn't make it obvious in the UI that the generic template is the best dataset template for NFS use. Since NFS is meant for *nix filesystems and the generic permissions are generic unix permissions, it's implied, but if you don't know that, it's not super-clear. One of those little details that makes the learning curve a bit steeper than it needs to be, but that will hopefully be fixed with a minor adjustment to the UI someday. :)
Though, I'm curious about one point: was there a reason you avoided using ACLs for the NFS share?
Hey Tom, why so slow with NFS Sync disabled, shouldnt you be getting close to wire speed ?
Sync writes are not about wire speed as much about how fast the data can be written and validated
@@LAWRENCESYSTEMS This is what I get with Sync disable on 10G link: 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 4.537 s, 947 MB/s , seems you are getting only around 500 MB/s , I think something not quite right with your XCP-ng or you are running slow pool.
@@LAWRENCESYSTEMS I get wirespeed over NFS from Proxmox to Truenas with sync disabled on a 10G network using your script. Any reason XCP-ng is only around 500 MB/s. Are these your nested XCP-ng hosts , maybe that impacts performance slightly...
Hi Tom! Thanks for a new great video and your contribution to the popularity of XCP-ng. When you showed your network adapters attached to the storage network, I noticed that the MTU is set to 1500 bytes. Did you try to set it to 9000 bytes? Was there some performance advantage? Sorry if you already answered to this question in another video.
Ya, 9000 Jumbo Frames on the LAN can boost NFS a lot since it means NFS traffic fits in a single packet now (8192 bytes). It greatly boosts the NFS efficiency.
It's not a substantial performance boost on modern systems, so I leave it at 1500.
NFS can either have synchronous or asynchronous writes via the NFS client.
It seems not all softwares/ vendors have the same defaults so there's a chance whatever NFS client is writing to an NFS share can also have sync or a sync enabled or disabled.
Do you know what XCPNG does by default? If it does synchronous by default then it may not even be necessary on the NAS
As far as I know XCP-ng is SYNC writes by default.
Is there any way to build an active-active-HA setup for VM Storage with TrueNAS?
Yes, with IXSystems hardware.
Thanks I was wondering how to do this. I'll try this when I get home from work
i enter my server ip and click the search and I get an error Server Detecton Cannot read properties of undefined (reading "Export")
So because I am hosting Truenas Scale Xen Orchestra web client on my server which has the same IP as my nfs share it wont work. Any work arounds? I tried this on my backup sever and it worked right away.
Worth noting that XO should also have a NIC on the storage network.
Which is why I pointed that out in the video.
must have missed it lol =]@@LAWRENCESYSTEMS
definitely missed the last 2 min =].....tom always covers all the bases@@LAWRENCESYSTEMS