How to copy large amounts of files in Windows
HTML-код
- Опубликовано: 21 янв 2022
- Robocopy is one of those commands that is often known about, but mis-used. Let's go over all the common options and do a little speed test.
Website Guide: www.christitus.com/robocopy .
►► Digital Downloads ➜ www.cttstore.com
►► Reddit ➜ / christitustech
►► Titus Tech Talk ➜ / titustechtalk
►► Twitch ➜ / christitustech 
Наука
It should be noted and I missed it in the video. If you are copying TONs of very small 1-100kb files you can completely turn off logging with /NFL /NDL /NJH /NJS - This will 10-100x the speed, but at the cost of logging.
NFL = No File List
NDL = No Directory List
NJH = No Job Header
NJS = No Job Summary.
You can copy 100,000s to millions of small files extremely fast, but will have no summary or log.
is syncToy using robocopy?
IS galaxies still a game???
@@TinS0lder SWG Legends EMU is alive and well they just released city in the clouds
@@ChrisTitusTech can we robocopy it to a local drive and play it there? 😁
hey Chris, how's the performance of this vs Rsync?
Been using this tool, especially the mirror + copy all attributes to make backups and matching with logfiles to verify all the file copying ever since it was a downloadable tool for Windows XP. A really great commandline tool that really should have a GUI wrapper in the modern age for the less tech savvy.
You can double click the icon in the title bar to close a window.
Just thought I'd throw that out there since Chris was right clicking the title bar of a window with the controls off-screen, and choosing close.
I love this channel. I have learnt so many things from this channel ❤
This was so helpful. Unfortunately 2 weeks late for me. I ran into issue doing a multi terrabyte copy for my NAS migration. I got around it by zipin gup the folders and unziping them to the new destination. Robocopy would have been much smarter and faster. This was awesome! Thanks so much for sharing.
With that much data you might want to look at something like imaging with Macrium Reflect free. That is a lot of data and if you don't have the switches right can be a difficult copy. Using Macrium Reflect might be a better option.
😂
Been using Windows for decades and never knew about robocopy, thanks.
Thank you!!! I had a NAS hard drive die - despite having multiple backups, the VMDK file on the drive was the ONLY corrupted spot on the disk! I loathed COPY and PASTE to another computer/drive, and this has saved my bacon!! Thank you!!!! 👏👏
Sounds like you also could have benefited from GRC Spin Rite, Roadkil's Unstoppable Copier, maybe even Test Disk/Photorec.
This is so very much appreciated. I just want to copy everything on an older smaller drive to a new larger one... but NOT lose any of the dates in any way. Sounds simple... been hard. I will test my method first before I go "BIG".
YO keep these vids coming Chris!!
The one thing I miss in Robocopy is what rsync have, the --backup-dir option so that "deleted" files are actually moved to another location.
This just reminds me of how we used to copy files from drive to drive back in the day using 3.11 in DOS :)
xcopy? By any chance.
No such problems. RS232 to another computer. Then reset connection, restart machines (for the IBM), copy or cp depending on the DOS or Unix machine you access. What I miss though are the cassetes in Amstrad. And the time to go out to flirt chicks.
I switched to robocopy long ago. xcopy is good, but inferior
Ty for this helpful information; I always learn something new from your videos.
Awesome tutorial - I am sure many will benefit, i will for sure.
Nicely explained. Many thanks for sharing!
Great video. Makes it a lot clearer!
I was using shared drives to access my NAS and copy files around. I recently installed Synology Drive to synchronize my laptop to my NAS. So far, so good. The only thing missing is a way to synchronize my Android phone to my NAS. I currently have to dedicate space on my laptop and use Syncthing to sync my phone to the laptop, then the laptop to the NAS.
My "big" file structure is about 300 GB (so far) of music albums stored as FLAC. There are all sorts of challenges with long filenames for classical music. Some people have overloaded the filename field with too much information. I shorten filenames when I rip the CDs. There are alternate character sets used for some albums, such as Blue Oyster Cult. These played havoc with my old Debian-based NAS.
I wish there was a way to put all my music in a database AND be able to play it back. The file system Music > Artist > Albums > Tracks is cumbersome at best. I actually added two folders to the structure to keep software from failing. I use Music > (A-M and N-Z) > Artist > Albums > Tracks. I may have to spit things again.
Thank you for explaining this tool. I will use it now. 👍
good vid. Thanks. Fan of your deep dive presentations vs the glossy overview.
Dude, this is insanely faster than normal method. Thank you so much :D
I've been using this since Server 2003R2 days. Once small caveat to the video is that there are times when you may want to turn off security all together. Like if you're copying user home folders from an old file server to a new one on a different domain prior to a user migration. I also might add that using robocopy in a scheduled task with a service account works well if you want a copy of something to go somewhere else to be backed up. No not as a backup itself, but a copy to be backed up there too. It's a powerful tool, and one that so many admins don't use or forget about using. DFS has gotten better, but it still has quirks (child/forest file copies are garbage), where robocopy will still succeed every time.
Indeed. Just remember it cannot copy any open locked files. Not geared for the same purpose but TSCOPY is an excellent tool that can copy any locked file including the hive files.
Very informative! Thanks.
This was excellent. Thank you.
Thankx for breaking it down.. great tutorial for beginners
I dunno I would rather stick to Teracopy since it does all of what you say but wit a gui. Most probably would want to avoid command line for something like this and anything more important there is backup software better suited.
Teracopy has always failed for me on large copy jobs. I've had it simply skip files and not tell me and has caused me problems. Either that, or it simply crashes. It's a POS.
@@justsaynotoboomers I dunno I've copied hundereds of gigabytes of files across multiple computers and not once did it fail and warned me on corruption even resumed after accidently disconnecting an ethernet once. Sounds to me nothing more than user error on your part
If anyone uses lazy slang like dunno and wit when conversing with others, especially when already writing more than a few words, then anything they have to say is immediately invalidated. Have you even looked at r/DataHoarder? Its chock full of people complaining about how badly Teracopy has been failing on them, only within the last few years, even after years of flawless use. Perhaps you should not be so quick to accuse of others of 'user error' when you obviously don't know the situation, and can't even communicate intelligently.
@@othername2428hey man, people can talk however they want, it doesn’t invalidate their point. I agree with you on Teracopy being issue prone, but you can be right without putting people down and judging their choices. I’m on your side I’m just trying to get you to communicate empathetically. r/DataHoarders is always welcome at my house 😂
@@pcmv6832whew! That’s good, maybe have a backup plan or some other tool to verify it worked as well as it said it did if you continue to use it in the future. A tool is only as good/bad as our expectations of the work we assume it did.
Great video!
cool, informative, educational, thank you kind sir
I feel like for normal day to day file copying tasks this would be kinda overkill. In the example you gave of that 5GB file you saved roughly 10-20 seconds. It would take that long just to type everything out for the command thusly not saving anytime at all. Now I can clearly see the huge professional level applications for such a thing for sure. Just not sure if this is worth it for the average user. Thank you for teaching me something about windows I didn't know though! Really enjoyed the video nonetheless
Yes he said it's for large files, and mostly for big servers and corporation not the average user
Let's say you copied 500GB of data to another drive and it was going to take 1 hour to complete. You copy/paste and walk away and go grab some lunch. You come back only to find that it did not finish copying because it quit for one of the many reasons Chris spoke of. So many different things can stop the Windows UI copying. With Robocopy and using the right switches that is never going to happen. It will always complete the copy process.
@@surfingsub5854 ok yeah I can see that being really useful. Peace of mind is always a good thing.
I work in migration projects and use robocopy a lot for copying more than 500g of data.
Ex: for copying 2gb folder which contains 2m files in it.
U don't know the pain to wait for copying 2gb of data without robocopy..
Robocopy saves the day!!
It has nothing to do with "the average user". if the aver user has 5gb of normal data, then duh, not worth it. if LOTS of data or like kiran said "2gb folder which contains 2m files", then duh, it is worth it.
Thank you very helpful!
Sorry i might have missed it but how did you solve the max character limit in the path issue? Had this many times and only solved with mapped drives which as you mentioned is not a great solution.
Thanks, appreciate the info.
I like your tutorials Very much
Hi Chris. I need to simply copy, or migrate data from an old file server(2008 R2) to a new one(2019). I just need to move, or mirror the data from the old to new, retaining all NTFS permissions. The old partition is called "DATA", and I just need to dump all files, folders, sub-folders into the new server's "DATA" partition. Seems like there's so many options. What switches should I use for that? Again, basically just a mirror. Thanks!
Nice video.. Thank you I learn new everyday from you..💕💕
Thanks for the great video! I didnt find a switch that would skip over unmodified files already in the destination folder. Or does Robocopy skips over by default?
Very good. Thank you
Hi, what is the difference between xcopy and Robocopy? I use xcopy a lot, it's faster than in the GUI, and it controls the files?
Chris, Would this be faster and safer for copying 100’s of GBs of video from a Camera SD card to my SSD?
This works great. Small issue: when the transfer was complete, the only way I can access the folder is to type in the path in explorer, the folder I transfer everything to disappears for some reason once robocopy has completed the transfer. and it’s not hidden. I just created a shortcut to get to it. Just curious if there’s a way around this?
Great video! Does robocopy work with Linux server and Windows client? Is this essentially scp/rsync but for windows?
If your Linux server has samba, then yes. Synology is basically a linux server and that's what he used in this example.
Yes, rsync has some methods that robocopy does not and if you're copying from one network to another and can use rsync that would be better
thanks alot for your decent effort,
Is there a GUI overlay for RoboCopy? That might be more comfortable for many Windows users... Also, how does TeraCopy, and other third-party software to replace Windows default coping service stack up to Window default service, or against RoboCopy?
ChoEazyCopy exists its a gui
thank you for this lesson on robocopy. I did a test, copying files from my C: drive (all of which have dates and times in the past - relative to 4/17/23) to an external USB drive that has bitlocker on it. INITIALLY, the dates and times on the destination drive matched the source drive, but after a minute or so, bitlocker must have done something to them because the dates and times on the destination drive all changed to today's date and time. Are you aware of this occurring with bitlocker? Is there a way around what bitlocker is doing? thanks
Thank you!
Hi Chris, thanks for this video. I will be helping a relative transfer basically everything on her old laptop to a newer one. Is connecting them with a crossover Ethernet cable and then using robocopy the best way to approach this?
Her old laptop is win 7 and the new one is win 10.
this is awesome 👍
How does this compare with teracopy? Anything I’m missing as the average user?
Ah yesm the wizard again. Thanks, your videos are Goodstuff!
Windows Explorer copy and paste on my PC seems to work faster than RoboCopy utility somehow? For instance, I am getting about 113MB/s with the standard explorer copy, and RoboCopy gets roughly the same, but sometimes lower. Another example, between my 2 NVME SSD's, Explorer copy gets 1.2GB/s, and RoboCopy gets ~800MB/s so about 1/3 slower than just normal copy and paste. Any ideas why this might be?
Amazing! Tnx
Not gonna lie, I tought that I would never use this when I saw your video, until I had to backup my sister's external hdd.
This is a god sent, the only thing that I miss is a general progress bar to the whole process...
It sounds like a small thing, but I'm copying a bunch of stuff right now. It's been going on for hours and I wish I had at least a rough idea of where it's at in the process.
The GUI copy will not preserve security settings most of the time, usually just replacing the source ACL's with the target folders ACL instead. Very bad if you need those ACL's preserved intact.
I used robocopy extensively to migrate a file server with over 2 TB of roaming profiles years ago. Did one pass to get the bulk data mirrored to the new system (ran overnight, took hours), command saved for later use. Later we just ran it again and again to get the modified files (took minutes each time). The ability to save logs each run helped us sort out a number of access issues reliably in the end too (use /NDL and /NFL to only log errors and the summary).
You can also use it (in rare cases) to salvage data from unreliable systems or drives as it can resume or at least skip files already copied. Had to do this ONE TIME to get data off a system with failing raid drives (lesson here is to actually check your raid and hardware logs regularly and not wait to the alarms to sound, yellow and red drive lights don't always get noticed). We managed to save all but two files if I recall correctly but it was touch and go for sure.
What program do you use for screen casting and especially how do you insert yourself in the screencast with cutting everything behind you?
OBS for recording with a Green Screen w/ Alpha Layer
@@ChrisTitusTech Ok, thanks
I am going to try this, and you may have changed my life.
there is any way to exclude some format? Let's say I wanna copy the contents of a folder, EXCEPT .mp3 files. How should I do it?
A nice use case is moving game save files to dropbox for those games that Steam does not sync the saves for you. I've got a task set up for that.
Liked the deeper dive into windows powershell
This video came in the right time
very very very usefull !!!!
You are best ;) Thank You !!!!!!!!!!!!!!!!
Hello chris. Is There a way to use this method to copy files from my android phone to pc. I tried the method in vid but it cannot get the android folder path.
Robocopy rocks. Have been using it since it was a part of the Windows 2000 Resource Kit. Was pleased to see it promoted into default install of Windows. One useful switch is /XO (exclude older files). Prevents an older file from overwriting a newer one.
Also, if you don't use the in-built logging function and just pipe the output into a file yourself with >, switch off the progress (NP). If you don't, you'll get loads of percentage counts in your log file and it looks a horrific mess!!
Very good video. Thank you. Can I also access my internal phone storage and copy videos from tis location usin robocopy. It shows up as MTP-device and doesn't have any drive letter assigned? Thanks very much in advance!
if you copy to one hard disk what's the point of using multiple threads? also in large files will it cause file fragmentation?
Which one of the following commands is preferable for copying shared files from an AD serv to another (Win servers)
Robocopy Source destination /DCOPY:T /COPYALL /E /R:0
or
Robocopy Source destination /E /ZB /copy:DATSOU /R:3 /W:3 /V
Sir, i have been facing an issue, I have an SSD which has around 70 gigs free space, quite recently I was trying to install God of War on it which is just 60 gigs, the problem is somehow while installing or copying from my other drive my entire memory gets full and after restart I think it flushs some archive file and that time I again retain the space.
A similar thing happened during the installation of Cyberpunk 2077 (at that time I had 80gigs free & CP2077 was around 58), for that I had to transfer files in segments, I transferred 1 archive at a time from my HDD which tend to occupy double the memory while transferring and after restart it gets fixed.
So can robocopy help in this or is there any other way.
Thanks 👍
There is a RSYNC that runs in the cmd window WITHOUT needing the Linux subsystem installed.
Thanks a lot
53.05 seconds vs. 52.07 seconds.
1.84% faster.
Interesting.
It would be interesting to see how much faster robocopy would be if you are saying using 10 GbE and passing data through a system in order to sync up two servers (or migrate the data) say from one NAS server to a bigger, newer, NAS server.
Exactly, the time is almost equal... The copy over the network is limited by the network speed and in both cases it was used to the max...
I can't find the source but someone was speculating that Win10 was using robocopy with default 8 or 16 threads so the performance is similar.
But so often I have one file failing in the end of the Windows way and then all I can do is click cancel and then you go to the folder and it moved everything back with that one file.
@@nlx78
I guess that the question is if you are using robocopy, and the same file fails -- would you know/would it tell you that the copy failed?
I would be surprised if it failed using the Windows File Explorer vs. but then it would be successful using robocopy. It wouldn't be unheard of, but I would be surprised (given that it's still Windows).
@Mister Robato
I don't think that rsync would really be all that much faster, especially since I don't think that it is native to Windows.
(Like you'd have to install something else (e.g. cygwin) to be able to get rsync. Or at least that's one of the ways to do it.)
Owman Chris, you can't stop yourself hu? In the end you give (even though you leave a lot out) so much options, I still don't know what to do.
Thanks anyway.
👍🏻Greetings from Netherland✌🏻
Moving an entire 3TB drive to another local drive.
Needing to keep ACLs, time stamps, etc
Would I need /ZB at all? In case my permissions are not sufficient?
Also, do I need /COPYALL and /DCOPY:DAT ?
Excellent subject…… one week short lol. That explained all the duplicate files and why nothing could be found. Even when I located it that did not transfer to the very next file in same folder. Still going through all 7 ssd’s and finding whole directories 3 or 4x deep under user/name/etc. dupchecker helpful as long as you tell it where to look. I wish it would scan everything everywhere and show which are complete and working and which can be trashed. So far though I was able to get back 6Gb of space. All I wanted was keep some important program files and empty each drive start with reformatted and clean pc. That didn’t go as expected. Lol.
well i had that file name limit error like 10 days ago when i was backing up my wsl files from the backup folder in windows to a hard drive
and if you are wondering why i used the copy paste as everyone does ? - cause i just testing with it
i had another tar.gz file backup as well
i do have credentials for my (\\) folder , which syntax to be used ?
I use fastcopy and it has a GUI and it syncs files.
Fastcopy rocks!
I also use Fastcopy. Most people have no idea what it's capable of. Can even shut down the pc when done, do wipe deletions, etc etc the list is long.
I use Teracopy because I can verify the files with different Hash functions. I usually pick SHA-256 or SHA-512
@@apreviousseagle836 Can almost do the same with Fastcopy except Terracopy has a couple more algorithms
*"...can use all 16 cores.."*
?? Okay, but where's the BOTTLENECK usually?
I would assume the bottleneck is usually the read or write drive. And if you're reading from, or writing to an HDD wouldn't trying to access multiple files at the same time actually HURT performance because the read/write head is jumping around?
And if you're reading from or writing to an SSD which is RAM (as in Random Access Memory) then what's the point of trying to access files simultaneously?
nice tutorial video...thanks chris. just share my experience, i use "fastcopy" written by japanese, it do able to handle long file name as well. hence, it have a log in notepad for each file copied.... and lots of great features...
I was going to suggest Fastcopy. Thanks.
does it do checksumming like rsync?
Will this fix the issue of copying files to an sd card?
I'm copying a photo directory from my hdd to a microsd via a usb cable plugged into my phone. I've copied a large amount of files but noticed the amount of files did not match the hdd. I selected all the files on the hdd and pasted onto the correct directory in the microsd, a conflict error pops up. If you choose "don't copy" and "do this for all conflicts" the copy process will just end. It won't copy the missing files.
Loads of topics on microsoft forums but they have all been closed with the only solution being to run a sfc scan. Very frustrating and I'm sure I've lost a lot of photos over the years because of it.
edit: nvm, can't even connect my phone to win10 to assign it a drive letter, sigh. Are we living in the future or the past?
So... like _rsync_ for Windows? Is _rsync_ available via WSL/WSL2?
AWESOME!! Well done. I said this is in 2020 long ago wish MS would do such things so we do not have to do this ourselves.... Anyhoo... GREAT JOB AS USUAL bud!!!
Checking who made robocopy....
Is it MS who made robocopy? Anyways does not matter who made it as this is the way it should be. What about large number of small sized files to handle I/O close/open overhead and if/how it is handled. Also there might be room for improvement where you do it raw writing to disk etc. So many things to consider...
Saw the title and instantly knew you were talking about robocopy. Use it all the time mainly for the MT option and especially with mapped drives at my enterprise. You mentioned UNC would be faster (we use SMB shares). How? UNC via file explorer is still single threaded mostly and via command line I just get UNC paths not supported.
So the reason my copy n paste keeps crashing my pc or aborts midway is because i go over some limit and it cant handle that? I have been frustrated as hell doing 500 latge files one by one because it keeps crashing and blue screen and what have you...
Can you simultaneously copy files from say an SD card to both an SSD for fast video editing file storage inside the workstation & a NAS for long-term file storage with Robocopy?
I'm not Chris, but as far as I know, no. You can have multiple robocopy scripts running at the same time, or you can script them to copy one after the other. Say, copy to your SSD, then from there to your NAS.
Chris - do you have a recommendation on dealing with large amounts files in a single directory? Often, photographers have a large amount of files in a single directory that are both RAW, JPG, sometimes sidecar files and sometimes video content too. I find my Windows system super bogs down even though I have 128GB of system ram and a pretty high end gaming MB. I am wondering if there are any Apps that specialize in dealing with large amounts of files. Thanks.
Robocopy will do great with this. At a previous employer my boss used Robocopy to do an "offsite" backup (to a hard drive in a caddy that was swapped out daily) we had multiple TB of data that copied just fine.
Hi Chris! Nice video! I have a bit of an issue, when i tested robocopy and did some real life scenario like interrupting the process (cancelled in the middle of copying, hahah!) .Now, when i tried to delete the interrupted copy file, i can not delete it completely anymore, it says "This is no longer located in C:\. Verify the item's location and try again". I am thinking there is something wrong with my system or what. Hope you can give me some advice how to address this. Thank you.
Using RUclips for that type of support sucks as YT limits what you can post. Better to visit Chris's site and use forums or other support options.
Not sure if you've solved it yet but my guess is that robocopy copied file entries first before actually copying and writing the file into your desired directory. What you did was probably trying to delete the entry while the actual file is not on the disk yet, this causes Windows to get confused.
Looking around in the internet, some suggested to open command prompt (cmd), preferably as the Administrator, and use the DEL command. Make sure to use its "Force" option as well. If that is still not possible and this bug file happens to be the only one or you don't mind losing the folder that contains this, delete the whole folder with RMDIR, also with "Force" and "Recursive" (most commands in Windows just need you to append "/?" to show its help page.
If that still doesn't work, some also suggested to use shredders. If you happen to have a 3rd party antivirus software installed, try seeing if it has a shredder feature. If not, just get a small and simple shredder put there in the internet. (not sure which one to suggest, I use HeX Editor a lot and it has a shredder feature but I don't recommend getting a whole other program just to use one of its small feature). Shredder programs typically lets you customize the shredding method. I recommend a simple "overwrite with 0-bits" or similar shredding method. (when you delete a file, it only deletes the entry of that file, but not its actual physical existence on the disk. This is why a file may still be recoverable, at least until it is overwritten by something else. An overwriting method using only one kind of bit is the quickest)
@@rrinnlonginus thank you for your reply. I solved it by just a simple cmd command. Thanks so much for your suggestion, really appreciate it!
If system crashes during robocopy is in progress, some time source folder gets corrupted. I guess it's due to robocopy using mft directly.
one thing i don't like is, the /MT flag is only for file copyings. when ever i do server migrations there are multi millions of files and lots and lots of folders. It would be really cool if the multithread could handle directory transversal as well. it never fails the initial population to the destination takes weeks then one needs to run a delta to catch new/modified files. Robocopy is a huge waste of resources when its only 1 thread to traverse the files/folders.
Thanks. Very helpful. I wish there was an easy way to detect hackers that have logged on.
Great tut! Don't need anything fancier.
Is there a way to verify file integrity to make sure everything copied correctly and without corruption?
where r those backup maintainance robocopy scripts at?
Any ideas to copy deduplicated refs? I have about 30tb to move to another synology, and it is painfully slow... I am open for ideas :-)
This looks more like a Linux way of doing things! It may be faster, but typing in all that stuff would actually take me a lot longer than just a couple of mouse clicks here and there, plus I'd mess up somewhere. That's why I can't use Linux. Give a GUI any day.
SUPER CONFUSED: I am trying to do:
robocopy "D:\BACKUP FOR HDD REPAIR\DASHCAM\" "F:\DASHCAM\" /e /r:0 /w:0 /COPY:DAT /DCOPY:DAT /mt:12
but it keeps saying there is no destination specified????
This is amazing I just had to sit and listen to
This video enough I wanted to take over 20,000 Files off my external hardrive and put it on my pc very fast if u sit down and follow these steps u will be fine use your note pad to write the codes first so u can copy in post in command prompt without feeling discouraged lol I didn’t need to put my network Ip and it worked
I want to copy file from downloads to desktop can I do it through robocopy?
I need some assistance if you can help me. So I'm trying to copy USB to USB using the robocopy method but I'm having a problem. I don't know if this is correct or not. but if you can tell me how to do it correctly? robocop E:\Karaoke CD (02) D:\Karaoke CD Part (02) CD Drv 01 E:\Karaoke CD (02) USB 02 D:\Karaoke CD Part (02)
SMB Multichannel copying is amazing. my nas has 80gbit connection and I have 20gbit. Rocks. You need a NAS that's 10g, qnap has some that are decently affordable now and 10g switches. qnap 10g nic works really well in linux and windows (very plug in play) for $100. But 10gig is SOOO much better than old school gigabit.
Whoa! You got Star Wars Galaxies? I played that religiously back in 2005-06. I still dream about it.
Check out SWG Legends. Fan run EMU and they have even entire zones like cloud city to the game.
@@ChrisTitusTech Thanks! I'll have a look.
I mean I am giving you a like before I even finish for Star Wars Galaxies alone!
What Teracopy does compred to this?
Teracopy is better just on the fact that it can verify your copy with a Hash function.