I have a server in LA, and when I am at school working on my coding, I can upload my work, go home download it, work on and upload it and have fresh code available all the time. Also prevents me from having to use the schools horrifically slow vpn. Also while the other kids are struggling to get there homework open in eclipse, I am already rocking my code in Vim on a remote server in LA!
@@fullstackwithsantoshgit is good for code but if you have many blob files like images of notes / graphs from textbooks then it quickly becomes inefficient space wise to use .git for tracking. Can sometimes nearly double the dir size
i would like to let you know something very useful to your guide. When you using the -a archive option you must be aware that using this include and preserve all those options (-r recursively, -p permissions, -t modifcation time, -l symlinks, -g group, -D devices) so you don't need to use the -r option as in your example when using the -a option. Hope this make the rysnc -a option more clear... thank you
Thanks so much for your linux series. I especially like the way you provide links in the ...more under the video so that we can have a TOC to help us find the desired content. Blessings on you and yours from the banks of the Suwannee River in North Florida.
Thanks for this tutorial, it will become very useful I think ... I am ditching Wordpress and going back to static websites, and have decided I will simply update the files locally and rsync them to the website. Though having said that there are a few people that argue the ssh vs ftp and reliability issues, so I am still working some things out
Rsync will resume tranfers if the link drops, with scp you'll have to start the transfer over in that case. Also, the OpenSSH folks declared scp deprecated a few years ago. Rsync -avP is how I use it.
Thanks for the tutorial. I am working with linux for a long time now, but still I watch your basic tutorials there is always some little bit that surprise me. Little comment on the tutorial - I think 'command' is bash specific. Better use 'which' in my opinion.
Hello Jay, Very good video thank you for this. I have a question How do I only sync the files that are different is that possible with a command syntax?
In a practical sense RSYNC is sometimes over-used by some sysadmins - I remember it being used for large-group transfers (entire database filesets across 100mbit, we're talking instances of 40gig) and they were creating a huge overhead on the disk packs on both sides in one chunk. The database files were a mixture of some which changed heavily a a lot which didn't but every time you ran RSYNC with the checksum calculation the overhead on the disk was kinda wild. A later admin saw this and changed to a process where he created an MD5 checksum separately and worked out the files which had actually changed that needed to be replicated to the target; he then used RSYNC on only those files. To boot he had a checksum file which he could also transfer to the target to do a validation offline without involving the source system. This was on a shared SAN so disk utilisation spikes had to be considered, this method reduced peak demand quite a bit; the way they were using RSYNC before it was initially creating its internal block checksums on every single file every time (I think they didn't use the change date/time metric). This guy got me into checksum files as a standard practice and I've done them for everything - I archived data to several thousand DVD-R's and produced a checksum for each file set (burnt with the target files to disc) so when retrieved we could work out if there'd been any degradation or loss of files on the optical media. It's a bit of due diligence which can really save your arse especially if your processes are audited.
Man Jay i am forever grateful and thanful for all the knlowledge and tuts you are dropping every week, i learned so much from you when it comes to Linux, i am now looking forward to buy your books :)🙏💯
Thanks for the video. it is very informative. One question, how do you add a password to the destination address? Many of my systems have a password for access and don't have ssh keys. Thanks.
Gentoo's Portage package manager uses rsync to synchronize your package list to the current in the repo. It also uses it when issuing commands with emerge to install packages. Rsync is an intragal part of the initial tool kit required for the installation of that distro. Rsync is really cool.
I've a question related with linux (not this vid): How to install fingerprint reader: U.ARE.U 4500 fingerprint reader Digital Persona, inc (crossmatch patented) on Linux mint? 😢 Any way that works and is safe?
I have studied a lot about rsync recently! My simple use case is to maintain backups of files to an external attached drive. I was disappointed to see that in backing up from macOS to exFat, the metadata/resource fork stuff created by Mac (or some other cause) causes MANY unchanged files to be transferred again unnecessarily. I just saw a Reddit post that said, now, in macOS Ventura, inclusion of the standard “--modify-window=1” flag fails to be useful due to a kernel flaw. This topic is probably not substantial enough for a separate video, but I look forward to your comment.
If you do PXE Diskless-Boot, you have to transfer the entire Root-Directory to a remote system. What's the best way, to do this? So that no files go lost, because of access rights and other stuff. The only reliable way I found, is: Create a complete partitionimage "rootpartition.img" , and then mount it on the remote system. That's the only way that works for me.
Oh cool- I'm glad you covered rsync. I'm an intermediate user, but you touched on an 'issue' that I just experienced on 16TB of backup data. I was using the --delete flag (and -av) for backing up my NAS drives... at some point the NAS went down, but my backup server kept diligently running its daily rsync. GASP, when I found all backup folders were EMPTY. --delete did what it does, and... poof. Any solution here? My local NAS mounts weren't present and I lost backup data. Is there a solution you might suggest while still using rsync? Second; I've been looking at borg-backup since that issue... I don't love it. Any other suggestions? Thanks!!
You have to be careful with the design, it sounds like what you were doing wasn't really a backup at all. Any accidentally deleted file would have had at most one day of protection. There's no way around keeping multiple copies. For example you could rsync to a different destination directory each time. If bandwidth is limited, first make a copy of the destination, then rsync. Those are the safest ways, consuming a lot of redundant storage. Unfortunately doing anything too clever to save space is a risk. Filesystem-level deduplication or snapshotting are probably the lowest risk, or you could triage the data and reduce the retention accordingly. The risks are a bug in the filesystem or putting valuable data in the wrong retention level. Define some requirements and how you're going to evaluate them, choose your level of risk, keep some offline copies and test thoroughly
Weird, everything I've read about backing up says to use rsync. But Syncthing seems to work so much better. I use it as a live backup or clone of my work files and do an off site backup periodically. It's mostly to protect against system or drive failures. If something goes wrong, a full up to date copy is always ready. What do you think?
It's because in reality rsync has old architecture, operates on a signle thread etc. It's a very popular tool and it's been there since... Always? When it comes down to syncing large amount of files or big ones, syncthing is going to be better in a lot of cases. There is a modernized version of rsync called rclone that is much more efficient and ready for nowadays workloads. PS: I use both syncthing and rclone (and rsync in the past). People tend to not write all over the Internet about their problems with rsync. They rather move to something else instead, like rclone which is very similar. This is how the legend of mighty rsync goes on and on :-)
Another rsync problem with multi hosts: I CANNOT use auto completion when I type the full path of destination! Each time I have to ssh to the destination host and pwd the full path, then copy and paste.
I used rsync to clone my Arch-install from one SSD to another (faster). It was easy, just looking up the exact parameters on the wiki (I needed an extra one for VM's) and everything went well. All I had to do after that is install the bootloader again and I could boot on it.
the -a option includes the -r so when you added it, you can remove the -r as no longer necessary. The -avh --delete is so ingrained in my fingers (muscle memory) I have to pause when I want anything else and think about it.
helpful :) what if you have a file of a different user(other than joy) and having 700 permission. it will give permission denied even you run rsync using root. any solution for that?
Rclone is so much better than rsync in any way... So many times I've been having problems with rsync's low transfer rates due to unknown reasons. It's an old software, guys. Rsync on the other hand does the same, is breeze to use, and didn't fail me even once! I imagine people still use rsync due to the habit mostly (like I did). An example of functional superiority of rclone: it has a buil-in mechanism for hashing ssh passwords of the hosts you want to sync with. Very handy.
For backup, yeah. But if you are migrating data from OS version 6 to OS version 8, you may want to preserve some of the new files generated on version 8.
Also for the purpose of explaining rsync, using it on the terminal it becomes completely independent on what distribution you are running, the commands are always the same, and the output looks the same.
Hey this is a great video. rsync is a great tool and with argument -a -a, --archive This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and want to preserve almost everything. :)
I have a server in LA, and when I am at school working on my coding, I can upload my work, go home download it, work on and upload it and have fresh code available all the time. Also prevents me from having to use the schools horrifically slow vpn. Also while the other kids are struggling to get there homework open in eclipse, I am already rocking my code in Vim on a remote server in LA!
Wondering why are not already using git.
@@fullstackwithsantoshgit is good for code but if you have many blob files like images of notes / graphs from textbooks then it quickly becomes inefficient space wise to use .git for tracking. Can sometimes nearly double the dir size
i would like to let you know something very useful to your guide. When you using the -a archive option you must be aware that using this include and preserve all those options (-r recursively, -p permissions, -t modifcation time, -l symlinks, -g group, -D devices) so you don't need to use the -r option as in your example when using the -a option. Hope this make the rysnc -a option more clear... thank you
Thanks so much for your linux series. I especially like the way you provide links in the ...more under the video so that we can have a TOC to help us find the desired content. Blessings on you and yours from the banks of the Suwannee River in North Florida.
¡Gracias!
I always use:
Rsync -Pavh --dry-run blah blah
Progress
Archive
Verbose
Human readable
Super useful
rsync is an excellent tool and worth your while spending some time learning it's options.
Easy to understand and very helpful. Thx!
Thank you very much for this explanation. Concise and with examples. This is a great way to teach.
Thanks for this tutorial, it will become very useful I think ... I am ditching Wordpress and going back to static websites, and have decided I will simply update the files locally and rsync them to the website. Though having said that there are a few people that argue the ssh vs ftp and reliability issues, so I am still working some things out
Really loving your videos. I would like to see you do one on rclone as well.
Rsync will resume tranfers if the link drops, with scp you'll have to start the transfer over in that case. Also, the OpenSSH folks declared scp deprecated a few years ago. Rsync -avP is how I use it.
Loving the content here!! Thx!!!
16:50 Hey Jay, and how we can do a real sync? Is there a command for that?
GREAT VIDEO! YOUR THE BEST! FROM DETROIT MI!
Thanks for the tutorial. I am working with linux for a long time now, but still I watch your basic tutorials there is always some little bit that surprise me. Little comment on the tutorial - I think 'command' is bash specific. Better use 'which' in my opinion.
Hello Jay, Very good video thank you for this. I have a question How do I only sync the files that are different is that possible with a command syntax?
In a practical sense RSYNC is sometimes over-used by some sysadmins - I remember it being used for large-group transfers (entire database filesets across 100mbit, we're talking instances of 40gig) and they were creating a huge overhead on the disk packs on both sides in one chunk. The database files were a mixture of some which changed heavily a a lot which didn't but every time you ran RSYNC with the checksum calculation the overhead on the disk was kinda wild. A later admin saw this and changed to a process where he created an MD5 checksum separately and worked out the files which had actually changed that needed to be replicated to the target; he then used RSYNC on only those files. To boot he had a checksum file which he could also transfer to the target to do a validation offline without involving the source system. This was on a shared SAN so disk utilisation spikes had to be considered, this method reduced peak demand quite a bit; the way they were using RSYNC before it was initially creating its internal block checksums on every single file every time (I think they didn't use the change date/time metric).
This guy got me into checksum files as a standard practice and I've done them for everything - I archived data to several thousand DVD-R's and produced a checksum for each file set (burnt with the target files to disc) so when retrieved we could work out if there'd been any degradation or loss of files on the optical media. It's a bit of due diligence which can really save your arse especially if your processes are audited.
Excellent explanation. Thanks
Man Jay i am forever grateful and thanful for all the knlowledge and tuts you are dropping every week, i learned so much from you when it comes to Linux, i am now looking forward to buy your books :)🙏💯
Very nice explanations. Thank you Jay. Do you use a program in particular for your markdown notes, or just a text editor ?
Awesome video, thanks man!
Thanks for the video. it is very informative. One question, how do you add a password to the destination address? Many of my systems have a password for access and don't have ssh keys. Thanks.
Hello, can you also compare rsync with unison for file sync? Thank you.
Awesome video! I'm going to try and use this to back up my Ubuntu Minecraft server to my WSL
Gentoo's Portage package manager uses rsync to synchronize your package list to the current in the repo. It also uses it when issuing commands with emerge to install packages. Rsync is an intragal part of the initial tool kit required for the installation of that distro. Rsync is really cool.
Another Gentoo enjoyer I see.
FYI When using the -a argument you no longer need -r as it includes it (and some others).
Thanks for another great one
I've a question related with linux (not this vid): How to install fingerprint reader: U.ARE.U 4500 fingerprint reader Digital Persona, inc (crossmatch patented) on Linux mint? 😢
Any way that works and is safe?
I have studied a lot about rsync recently!
My simple use case is to maintain backups of files to an external attached drive.
I was disappointed to see that in backing up from macOS to exFat, the metadata/resource fork stuff created by Mac (or some other cause) causes MANY unchanged files to be transferred again unnecessarily.
I just saw a Reddit post that said, now, in macOS Ventura, inclusion of the standard “--modify-window=1” flag fails to be useful due to a kernel flaw.
This topic is probably not substantial enough for a separate video, but I look forward to your comment.
Isn't the -a option (archive) the same as doing -rlptgoD ? if so the -r (recursive) option is not needed?
8:54 What do you mean with "and DNS is often a problem" ???
If you do PXE Diskless-Boot, you have to transfer the entire Root-Directory to a remote system.
What's the best way, to do this? So that no files go lost, because of access rights and other stuff.
The only reliable way I found, is: Create a complete partitionimage "rootpartition.img" , and then mount it on the remote system.
That's the only way that works for me.
Oh cool- I'm glad you covered rsync. I'm an intermediate user, but you touched on an 'issue' that I just experienced on 16TB of backup data. I was using the --delete flag (and -av) for backing up my NAS drives... at some point the NAS went down, but my backup server kept diligently running its daily rsync. GASP, when I found all backup folders were EMPTY. --delete did what it does, and... poof. Any solution here? My local NAS mounts weren't present and I lost backup data. Is there a solution you might suggest while still using rsync?
Second; I've been looking at borg-backup since that issue... I don't love it. Any other suggestions?
Thanks!!
You have to be careful with the design, it sounds like what you were doing wasn't really a backup at all. Any accidentally deleted file would have had at most one day of protection. There's no way around keeping multiple copies.
For example you could rsync to a different destination directory each time. If bandwidth is limited, first make a copy of the destination, then rsync. Those are the safest ways, consuming a lot of redundant storage. Unfortunately doing anything too clever to save space is a risk. Filesystem-level deduplication or snapshotting are probably the lowest risk, or you could triage the data and reduce the retention accordingly. The risks are a bug in the filesystem or putting valuable data in the wrong retention level. Define some requirements and how you're going to evaluate them, choose your level of risk, keep some offline copies and test thoroughly
@@shanent5793 I don't understand what yer saying.
Is there a way to transfer files from an open media vault server to a Synology server?
I got my RSYNC working. I am thinking. the cache file. I dont know if deleting it is good. But when RSYNC is done. Can it leave the cache alone.
Weird, everything I've read about backing up says to use rsync. But Syncthing seems to work so much better.
I use it as a live backup or clone of my work files and do an off site backup periodically. It's mostly to protect against system or drive failures. If something goes wrong, a full up to date copy is always ready.
What do you think?
It's because in reality rsync has old architecture, operates on a signle thread etc. It's a very popular tool and it's been there since... Always? When it comes down to syncing large amount of files or big ones, syncthing is going to be better in a lot of cases. There is a modernized version of rsync called rclone that is much more efficient and ready for nowadays workloads. PS: I use both syncthing and rclone (and rsync in the past).
People tend to not write all over the Internet about their problems with rsync. They rather move to something else instead, like rclone which is very similar. This is how the legend of mighty rsync goes on and on :-)
How and where do you set that ssh banner message for every ssh connection to the server?
thank you for this!
I learned that I can *ls* the subdirectory using: ls/backup/notes. Never knew that.
One limitation with (-a) archive option is that it won't preserve ownership/group unless you're syncing to target as user "root".
or super user (SU/SUDO). -a is a group of flags that has the -o option that means: preserve owner (super-user only)
Thanks for the video! I gotta ask, where'd you get that shirt! I want one for my daughter and I! :) Much appreciated!
Another rsync problem with multi hosts: I CANNOT use auto completion when I type the full path of destination! Each time I have to ssh to the destination host and pwd the full path, then copy and paste.
Can you share your .md note files? Or is there somewhere you've already published them so we can look at them?
I used rsync to clone my Arch-install from one SSD to another (faster). It was easy, just looking up the exact parameters on the wiki (I needed an extra one for VM's) and everything went well. All I had to do after that is install the bootloader again and I could boot on it.
the -a option includes the -r so when you added it, you can remove the -r as no longer necessary. The -avh --delete is so ingrained in my fingers (muscle memory) I have to pause when I want anything else and think about it.
Is it possible to use rsync to transfer files to Google Drive if I am signed in on Ubuntu?
May you share what Linux distro you’re using and the theme? Maybe not
I'm guessing Debian.
helpful :) what if you have a file of a different user(other than joy) and having 700 permission. it will give permission denied even you run rsync using root. any solution for that?
Really need a simple program that mirrors 2 directories so I can have a backup and separate backup of the samething on my external drive! Please help
great video
I assume SAMBA is installed on both the source and target machines?
smb client always is - at most distros while the server part has to be installed, that is correct.
谢谢！Thank you for your tutorial. Now I use Rsync to do replication between two NAS. The only pity is that Rsync has no windows version.
maybe you could use Windows Subsystem for Linux (WSL) but it might be a pain
@@ironfist7789wsl has extremely poor disk performance iirc
@@-aexc- ah ok, that sucks
Rclone is so much better than rsync in any way... So many times I've been having problems with rsync's low transfer rates due to unknown reasons. It's an old software, guys. Rsync on the other hand does the same, is breeze to use, and didn't fail me even once!
I imagine people still use rsync due to the habit mostly (like I did).
An example of functional superiority of rclone: it has a buil-in mechanism for hashing ssh passwords of the hosts you want to sync with. Very handy.
waiting for an episode about kill command 😁
Cheers!
The link for Akamai is missing
$ command -v rsync -->> what is command in this??
$ which rsync --->> usually this was used to check if it is installed.
Thank you.
Thanks.
You can skip to 2:14 😅
Is that a rhel or Debian server?
sorry, I didn’t catch that the first 20 times…😂
Great tool, great video.
CONTENT starts at 2:30
Concerning the single-directional nature of rsync (without ‘delete’ flag), the *best* wording I saw to describe this is, “Rsync is ADDITIVE”
Yes, seems an important distinction if one is looking for a .photoslibrary backup, for instance.
For backup, yeah. But if you are migrating data from OS version 6 to OS version 8, you may want to preserve some of the new files generated on version 8.
😊
😊
BB❤ VP❤❤b❤❤@@jameswhite1910
are you sharing that notes folder somewhere? interesting
I hate it whenever I forget to put a slash "/" at the end of the folder name.
rsyncd.conf should be covered next time you return to this topic
Wow, a dry run, I can't even imagine what a "wet" would do, but it certainly gonna be a liquidfall?
Are your notes published to the public?
I use scp command for the same including the -r flag. Works same way. Don't see any difference Jay
Kisuka dil jeet ka tum tumra hak adaya karna joga❤❤
Eyy Im fast af boi.
Schimmel Expressway
Schowalter Road
I haven't seen the difference between rsync and scp.
6492 Misty Square
Willard Shoals
First viewer
Hudson Street
Demetrius Corner
7911 Osinski Fall
Simonis Lock
Keven Highway
Rogelio Lodge
Tumhumra pas sa kaya sikhsa
Mila tumko etni jara bat ka liya
Tum gio sisa ka umra gana sunna abhisa etna lachar aga baro dakho kaya hota hai❤❤
Prosacco Estates
Why do you need command lines in linux. That is so old days. Like using dos when you have windows
Because command lines are super leet
If you are allergic to using the terminal there are GUI front-ends. It's linux - so you can choose!
Also... even Windows have a tool like this one! (robocopy)
Thanks thanks for explaining
Also for the purpose of explaining rsync, using it on the terminal it becomes completely independent on what distribution you are running, the commands are always the same, and the output looks the same.
😅😅9m
😅9😅😅
Nn
KN
nHistory HiKstory
oion0lpm9
9nnn
899mm
Nn
Nnnnn😅.lmko
Hey this is a great video. rsync is a great tool and with argument -a
-a, --archive
This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and want to preserve almost everything. :)
Which emulator do you use?