Thanks for this video. I had the same AX failure and my rsync turned all my files into 777. I also had owner issues restoring to a new system due to the UID difference like you said. This video helped understand the context of those issues a bit, thanks.
I know I am a bit late in asking this question but here I go. I am trying to create linux (ubuntu) root / directory (ext4 file system) backup on a zfs file system (TrueNAS) via rsync and ssh. I use the options -aAXHvz but the preserve access control list option "A" fails by throwing "protocol incompatibility" error. I can't seem to find a way around this. Without using the A option the file/directories are saved with invalid ACLs and are messed up on restore. Is there anyway to solve this problem?
The "archive" option automatically includes recursion and progress. I use "rysnc -av -H --delete" because I also want to preserve hard-links. With rsync I now avoid EXT4 to NTFS (because every time I find files that failed to transfer due to naming issues). EXT4 to NTFS is possible but really not worth it. Thanks for this video, I just subscribed after seeing your previous TAR video. I am going to try the --exclude-from option tonight.
I also like to use a --progress in a nohuped run of an rsync that way I can go back after a long rsync run and see where it is at by tailing the nohup file. even if the session is dropped. screen can also work too.
I know I am a bit late in asking this question but here I go. I am trying to create linux (ubuntu) root / directory (ext4 file system) backup on a zfs file system (TrueNAS) via rsync and ssh. I use the options -aAXHvz but the preserve access control list option "A" fails by throwing "protocol incompatibility" error. I can't seem to find a way around this. Without using the A option the file/directories are saved with invalid ACLs and are messed up on restore. Is there anyway to solve this problem?
This is the one I use, might not be what you are looking for but works just fine for me. Its not bootable, but I can restore it fairly easy :rsync -avP --numeric-ids --exclude-from excl / USER@HOSTNAME:/ZFSPOOL/ZFSDIRk/$(hostname) replace USER, HOSTNAME, ZFSPOOL and ZFSDIR with yours...I am using ssh to reach the remote server
Okay, you only describe how to _mirror_ files, but how to _backup_ files with rsync? Is this possible? Backup: having versions, ideally incremental versions. Deleted and changed files are saved in versions.
Could you please explain the following case. I have virtual server Ubuntu on AWS, I need to backup data from it to my local FreeNAS which is behind the NAT. So I need to pull data from remote rsync to local rsync. Is it possible to do it? Will it work behind the NAT? I need a bit more information about network interaction. Which part will be an initiator in this scheme?
If you can ssh to the virtual machine, you should be able to pull the backup from the virtual machine using an rsync executed on the machine behind the NAT. Do a google search of rsync over ssh for the command syntax to follow
I've been using rsync for some time to back up my data from a zfs pool on an ubuntu server, and have always been happy with it for that. Due to hardware failure I had to rebuild my system, the pool came back no problem.. but the server install and set up left me looking for a good way to back that up... I am looking for guidance. I don't necessarily need to back up the entire OS.. or maybe I do if it's just as easy that way.... I don't have a problem reinstalling, what I'd like to be able to restore are all the programs and settings I add afterwards.. such as ufw, fail2ban and plexmediaserver, as well as user and group changes. I back up to external WD easystore usb drives. One might be EXT4..others are NFS... sounds like I should convert them all to EXT4? I rotate them, always keeping one off site. Is rsync.. or tar the better solution? If rysync... is there a targeted set of directories that would contain my desired programs/settings? Thanks for any insight.
Thanks for the question, I use rsync for backup, I backup everything on the ARM machines since they require a complete install to rebuild them, on Intel I do a partial backup (/etc, /usr/local, /opt, /home and /var I can restore those faster using the iso and then overylay the rsync on top. There are lots of ways to backup, and so my answer has to be it depends...if you have a database, then you will want to do a dump first on that since copying a database back will result in it not starting (time and date timestamp mismatch), those get written out again when the database is closed. So that's what i do. For me rsync over ssh is way faster than using NFS. For ZFS I use the inbuilt backup will be doing a video on that soon, so look for that after the first of the year. Hope this helps.
what you do on your LAN is up to you as for me I will continue to recommend using rsync inside of SSH. relying on auth hosts and unencrypted data traffic is an idea long past as a sound practice.
@@MarkConstable Enterprise standards are: Encrypt data at rest, Encrypt Data in Transit. Recommending the use of 'r' apps in the open is not recommended LAN, WAN or DMZ. I background my rsync so I really don't care how long they take. and Im not seeing a 20% difference more like 10%.
@@CyberGizmo 20%+ on low end hardware like a PI. I use raw rsyncd for the initial backup to a new NAS then ZFS send/receive via SSH after that. I would never use unencrypted anything on the public internet.
FYI: I am working on an Open Source Web Application for centralized Backup RSync Monitoring of multiple Rsync Jobs / Servers. Check it out: www.marcogriep.com/blog/central-rsync-backup-monitoring-with-golang-and-lumen-web-api/
This specific video shows how silly the Linux community is currently. Rsync has its own GUI, called "Lucky Backup". Lucky backup" is a neglected and badly created GUI application, without the CLI favored by keyboard obsessives of Linux. GUI uses a mouse, favored by the Xerox invented WIMP, as used by Apple and Microsoft. Linux like to currently pretend that only idiots use WIMP and GUI. Like Gkrellm desktop indicator, Lucky Backup is a poorly created GUI. To use the GUI backup properly, choose Free File Sync. Free File Sync is available, open source, in Apple, Microsoft and Linux. Linux people don't understand GUI, so Free File Sync doesn't yet use rsync properly yet. Because Free File Sync is updated regularly over there years of its existence, it might eventually be allowed to use rsync. Free File Sync is available in Linux, only on flatpak. Not yet in any compiled form anywhere on Linux. Not in appimage, rpm, Debian, appimage nor snap.
great video ! still relevant and great way to copy files and make backups.
RUclips algorithm recommending gold!
Nice! I keep it simple and just use grsync to keep my home folder backed up on an external drive
Thanks for this video. I had the same AX failure and my rsync turned all my files into 777. I also had owner issues restoring to a new system due to the UID difference like you said. This video helped understand the context of those issues a bit, thanks.
I know I am a bit late in asking this question but here I go. I am trying to create linux (ubuntu) root / directory (ext4 file system) backup on a zfs file system (TrueNAS) via rsync and ssh. I use the options -aAXHvz but the preserve access control list option "A" fails by throwing "protocol incompatibility" error. I can't seem to find a way around this. Without using the A option the file/directories are saved with invalid ACLs and are messed up on restore. Is there anyway to solve this problem?
I discovered rsync through grsync, and it is lovely. :-)
Yest it is I use it for my little arm machines when backing up, I should transfer it over to Syncthing, but...I'll get aroiund to it :)
Thank you, exactly what I was looking for.
The "archive" option automatically includes recursion and progress. I use "rysnc -av -H --delete" because I also want to preserve hard-links. With rsync I now avoid EXT4 to NTFS (because every time I find files that failed to transfer due to naming issues). EXT4 to NTFS is possible but really not worth it. Thanks for this video, I just subscribed after seeing your previous TAR video. I am going to try the --exclude-from option tonight.
have had similar issues trying to copy files using rsync to NFS too, so now I just use the ssh tunnel method.
I also like to use a --progress in a nohuped run of an rsync that way I can go back after a long rsync run and see where it is at by tailing the nohup file. even if the session is dropped. screen can also work too.
Nice!
I know I am a bit late in asking this question but here I go. I am trying to create linux (ubuntu) root / directory (ext4 file system) backup on a zfs file system (TrueNAS) via rsync and ssh. I use the options -aAXHvz but the preserve access control list option "A" fails by throwing "protocol incompatibility" error. I can't seem to find a way around this. Without using the A option the file/directories are saved with invalid ACLs and are messed up on restore. Is there anyway to solve this problem?
This is the one I use, might not be what you are looking for but works just fine for me. Its not bootable, but I can restore it fairly easy :rsync -avP --numeric-ids --exclude-from excl / USER@HOSTNAME:/ZFSPOOL/ZFSDIRk/$(hostname)
replace USER, HOSTNAME, ZFSPOOL and ZFSDIR with yours...I am using ssh to reach the remote server
@@CyberGizmo makes sense, will give it a try. Thanks
Okay, you only describe how to _mirror_ files, but how to _backup_ files with rsync? Is this possible? Backup: having versions, ideally incremental versions. Deleted and changed files are saved in versions.
Michael if you need versioned backup for Linux you might want to check out Timeshift, it uses rsync as one of its options.
Could you please explain the following case. I have virtual server Ubuntu on AWS, I need to backup data from it to my local FreeNAS which is behind the NAT. So I need to pull data from remote rsync to local rsync. Is it possible to do it? Will it work behind the NAT? I need a bit more information about network interaction. Which part will be an initiator in this scheme?
If you can ssh to the virtual machine, you should be able to pull the backup from the virtual machine using an rsync executed on the machine behind the NAT. Do a google search of rsync over ssh for the command syntax to follow
I've been using rsync for some time to back up my data from a zfs pool on an ubuntu server, and have always been happy with it for that. Due to hardware failure I had to rebuild my system, the pool came back no problem.. but the server install and set up left me looking for a good way to back that up... I am looking for guidance. I don't necessarily need to back up the entire OS.. or maybe I do if it's just as easy that way.... I don't have a problem reinstalling, what I'd like to be able to restore are all the programs and settings I add afterwards.. such as ufw, fail2ban and plexmediaserver, as well as user and group changes. I back up to external WD easystore usb drives. One might be EXT4..others are NFS... sounds like I should convert them all to EXT4? I rotate them, always keeping one off site. Is rsync.. or tar the better solution? If rysync... is there a targeted set of directories that would contain my desired programs/settings? Thanks for any insight.
Thanks for the question, I use rsync for backup, I backup everything on the ARM machines since they require a complete install to rebuild them, on Intel I do a partial backup (/etc, /usr/local, /opt, /home and /var I can restore those faster using the iso and then overylay the rsync on top. There are lots of ways to backup, and so my answer has to be it depends...if you have a database, then you will want to do a dump first on that since copying a database back will result in it not starting (time and date timestamp mismatch), those get written out again when the database is closed. So that's what i do. For me rsync over ssh is way faster than using NFS. For ZFS I use the inbuilt backup will be doing a video on that soon, so look for that after the first of the year. Hope this helps.
Maybe look int Ansible Playbooks to code in the apps and their settings?
Using rsyncd direct is at least 20% faster than via SSH. It's fine for local LAN transfers.
what you do on your LAN is up to you as for me I will continue to recommend using rsync inside of SSH. relying on auth hosts and unencrypted data traffic is an idea long past as a sound practice.
@@CyberGizmo Sure, unless that extra 20% speed bump is more important on a LAN where it's unlikely anyone else is sniffing your unencrypted traffic.
@@MarkConstable Enterprise standards are: Encrypt data at rest, Encrypt Data in Transit. Recommending the use of 'r' apps in the open is not recommended LAN, WAN or DMZ. I background my rsync so I really don't care how long they take. and Im not seeing a 20% difference more like 10%.
@@CyberGizmo 20%+ on low end hardware like a PI. I use raw rsyncd for the initial backup to a new NAS then ZFS send/receive via SSH after that. I would never use unencrypted anything on the public internet.
FYI: I am working on an Open Source Web Application for centralized Backup RSync Monitoring of multiple Rsync Jobs / Servers. Check it out:
www.marcogriep.com/blog/central-rsync-backup-monitoring-with-golang-and-lumen-web-api/
Wonderful, Marco and I sure will thanks for the link!
first!
This specific video shows how silly the Linux community is currently. Rsync has its own GUI, called "Lucky Backup".
Lucky backup" is a neglected and badly created GUI application, without the CLI favored by keyboard obsessives of Linux. GUI uses a mouse, favored by the Xerox invented WIMP, as used by Apple and Microsoft. Linux like to currently pretend that only idiots use WIMP and GUI.
Like Gkrellm desktop indicator, Lucky Backup is a poorly created GUI. To use the GUI backup properly, choose Free File Sync. Free File Sync is available, open source, in Apple, Microsoft and Linux.
Linux people don't understand GUI, so Free File Sync doesn't yet use rsync properly yet. Because Free File Sync is updated regularly over there years of its existence, it might eventually be allowed to use rsync.
Free File Sync is available in Linux, only on flatpak. Not yet in any compiled form anywhere on Linux. Not in appimage, rpm, Debian, appimage nor snap.