How To Secure TrueNAS Core ruclips.net/video/psVNn-JVT9Q/видео.html TrueNAS Document on "Hardened Backup Repository for Veeam" www.truenas.com/docs/scale/communityrecommends/hardened-backup-repository-for-veeam/ MINIO Object Lock and Immutablity Guide docs.min.io/docs/minio-bucket-object-lock-guide.html How S3 Object Lock works docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html The Best Diagramming Tool "Diagrams.net" ruclips.net/video/mpF1i9sfEJ0/видео.html ⏱ Timestamps ⏱ 00:00 TrueNAS Hardening Backups 02:14 Backups and Protecting the Transport Layer 05:44 Snapshots and Replication 09:00 Cloud Backups 10:26 Hardening TrueNAS 12:18 MINIO S3 Object Locking
Backups are a good start but you must also periodically test and verify the restore process. Any suggestions on how to implement a practical test procedure to verify that the backup/restore will work when needed?
I achieve immutable and air-gapped backups that can be stored offsite by using tape. With the right backup technology and processes you can make this pretty secure and able to recover from any number of different scenarios.
Long-term cloud archival is so cheap that is not a big issue having 2 or more full copy and snap shots in 2 or more Differents locations. Many insurance (especially in nowadays with CCPA or GDPR Hiden rules to backup private data) require you save your backup encrypted in at least 3 location separated by at leas 100miles. In 2023 I highly suggest for any bussines that handle valuable data to have at least one employee with the ISO 27001. certification.
I use Borge backup and an Storage Box from Hetzner. Borge has the permission to delete his own backups but the Hetzner Storage Box is based on ZFS and makes Daily Snapshots which only can be deleted via web Interface. And my backup user has no permission to read or whatever the backups. So I think this is a pretty good approach. What are you guys thinking?
I have a 5TB pool to backup, my home network is only gigabit ethernet so direct usb should be faster? I think it's a replication task source > destination but I'll let an expert confirm that
I use 2 NAS servers for backup. The first one is on the AD where only admins and the backup software have RW. The second NAS is read only for ANY AD user, including admins and the only way it mirrors NAS No.1 is by doing a pull replication/rsync. Any deletion can only be done by logging on to the machine directly with a different user and password, with a local cron job or with something like winscp but only from specific machines and again with a different account that is not part of the AD. Not sure if its an ideal solution or I have missed something, but at the time of setting it up to my mind it was. 😁 Once again, good video and as always informative.👍
@Lawrense Systems, Tom, thanks for the video.. Very clarifying... btw.. your link to TrueNAS Document on "Hardened Backup Repository for Veeam" does not appear to be valid anymore...
Hi Tom, How do you move the snapshots to an off-site Location? Do i just push the ".zfs" folder? In case of a disaster, would restoring the said ".zfs" folder along with the backed up data, give me the snapshots back?
@@marcogenovesi8570 a tape library from someone like HP coupled with A backup suite like Veeam is 100% automated…. And the amount of hours the drive can do is not dissimilar to the amount of hours a spinning hard drive can do before failure rates grow exponentially
@@NickF1227 if it's automated then it is just cold storage with extra steps, if it's not isolated correctly then you can delete the backups, just like with a normal cold storage appliance. I have some horror stories about tape libraries. The amount of hours and/or writes is so different you can't even compare it to hard drives, tapes aren't meant to be written all the time, again, plenty of horror stories about people that kept reusing the same tapes for backups well past their end of life and were hosed when tried to restore backups
I think Tom covered that in the external hard drive storage. You need good people processes, so the drives or tapes are swapped. And tested on a regular basis.
In the example provided, why not have the cloud backup performed by the replicated truenas server, and have it access the first server via SSH using a key pair, that way even if the first truenas server is compromised that they then would not have access to the cloud provider nor would they have an avenue to retrieve the credentials to the second server. You could even add a cron script to check for a canary file with a known MD5 hash to prevent replications.
Immutable means the data only can be written once and never changed. It can be deleted, but never changed. Think of the old WORM systems, Write Once Read Many. Folks do mix up immutability and retention.
Are you all using TrueNAS’ snapshot replication (zettarepl) successfully? I spent months trying to get it to work and just kept hitting bug after bug while trying to replicate ~40TB. I eventually gave up and switched to zrepl.
How To Secure TrueNAS Core
ruclips.net/video/psVNn-JVT9Q/видео.html
TrueNAS Document on "Hardened Backup Repository for Veeam"
www.truenas.com/docs/scale/communityrecommends/hardened-backup-repository-for-veeam/
MINIO Object Lock and Immutablity Guide
docs.min.io/docs/minio-bucket-object-lock-guide.html
How S3 Object Lock works
docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html
The Best Diagramming Tool "Diagrams.net"
ruclips.net/video/mpF1i9sfEJ0/видео.html
⏱ Timestamps ⏱
00:00 TrueNAS Hardening Backups
02:14 Backups and Protecting the Transport Layer
05:44 Snapshots and Replication
09:00 Cloud Backups
10:26 Hardening TrueNAS
12:18 MINIO S3 Object Locking
Backups are a good start but you must also periodically test and verify the restore process. Any suggestions on how to implement a practical test procedure to verify that the backup/restore will work when needed?
I achieve immutable and air-gapped backups that can be stored offsite by using tape. With the right backup technology and processes you can make this pretty secure and able to recover from any number of different scenarios.
Long-term cloud archival is so cheap that is not a big issue having 2 or more full copy and snap shots in 2 or more Differents locations. Many insurance (especially in nowadays with CCPA or GDPR Hiden rules to backup private data) require you save your backup encrypted in at least 3 location separated by at leas 100miles. In 2023 I highly suggest for any bussines that handle valuable data to have at least one employee with the ISO 27001. certification.
I use Borge backup and an Storage Box from Hetzner. Borge has the permission to delete his own backups but the Hetzner Storage Box is based on ZFS and makes Daily Snapshots which only can be deleted via web Interface. And my backup user has no permission to read or whatever the backups. So I think this is a pretty good approach. What are you guys thinking?
Thanks for the idea, I didn't know about Hetzner storage boxes. Looks really nice!
14:37 do you connect it directly to your TrueNas server or connect the usb drive to your pc and copy the data over ssh?
I have a 5TB pool to backup, my home network is only gigabit ethernet so direct usb should be faster? I think it's a replication task source > destination but I'll let an expert confirm that
I use 2 NAS servers for backup. The first one is on the AD where only admins and the backup software have RW. The second NAS is read only for ANY AD user, including admins and the only way it mirrors NAS No.1 is by doing a pull replication/rsync. Any deletion can only be done by logging on to the machine directly with a different user and password, with a local cron job or with something like winscp but only from specific machines and again with a different account that is not part of the AD.
Not sure if its an ideal solution or I have missed something, but at the time of setting it up to my mind it was. 😁
Once again, good video and as always informative.👍
@Lawrense Systems,
Tom, thanks for the video.. Very clarifying... btw.. your link to TrueNAS Document on "Hardened Backup Repository for Veeam" does not appear to be valid anymore...
Updated
@@LAWRENCESYSTEMS Thank you!! Keep on rocking ! Your videos are invaluable!
Hi Tom,
How do you move the snapshots to an off-site Location?
Do i just push the ".zfs" folder?
In case of a disaster, would restoring the said ".zfs" folder along with the backed up data, give me the snapshots back?
I cover that in this video ruclips.net/video/XIj0iHtZvOg/видео.htmlsi=rUfKy8JUNoDWCv0P
Be great to see borgbase or something setup on Scale.
Great information thank you tom
If you want immutable backups...tapes have been around for decades? :P
Correct! There's even support for WORM tapes. Very odd he didn't mention any of this.
Tape backups aren't automated (unless you have the fancy bot stations) so now it's down to human error. They also wear out
@@marcogenovesi8570 a tape library from someone like HP coupled with A backup suite like Veeam is 100% automated…. And the amount of hours the drive can do is not dissimilar to the amount of hours a spinning hard drive can do before failure rates grow exponentially
@@NickF1227 if it's automated then it is just cold storage with extra steps, if it's not isolated correctly then you can delete the backups, just like with a normal cold storage appliance. I have some horror stories about tape libraries.
The amount of hours and/or writes is so different you can't even compare it to hard drives, tapes aren't meant to be written all the time, again, plenty of horror stories about people that kept reusing the same tapes for backups well past their end of life and were hosed when tried to restore backups
I think Tom covered that in the external hard drive storage. You need good people processes, so the drives or tapes are swapped. And tested on a regular basis.
root smb share is broken for anyone who set it up in version 11?
In the example provided, why not have the cloud backup performed by the replicated truenas server, and have it access the first server via SSH using a key pair, that way even if the first truenas server is compromised that they then would not have access to the cloud provider nor would they have an avenue to retrieve the credentials to the second server. You could even add a cron script to check for a canary file with a known MD5 hash to prevent replications.
That could work as well.
Compartmentalization is a good thing.
Immutable means the data only can be written once and never changed. It can be deleted, but never changed. Think of the old WORM systems, Write Once Read Many.
Folks do mix up immutability and retention.
I have just set up TrueNAS 12.0-U3 and trying to set up a one off backup to external USB drive on the same system, I think it's a replication task?
The UI does not offer setting up with USB backup devices, you will have to do some custom scripts to get that working.
@@LAWRENCESYSTEMS thank you
This is very helpful Tom, thank you.
What do you use to encrypt your external sandisk?
LUKS
What is the diff between TrueNas and TruenNAS?
Consultants who know what they are doing configure only full-automated backup solutions, no user involvement required.
Yup
cron job to print the data on paper every now and then
Thank you Tom!
Thank you good sir ..
second! and TrueNAS is awesome!
What a good April fools video 🤣
First!
First
Are you all using TrueNAS’ snapshot replication (zettarepl) successfully? I spent months trying to get it to work and just kept hitting bug after bug while trying to replicate ~40TB. I eventually gave up and switched to zrepl.
I use replication via the web interface and it works well.