I've been writing bash scripts for years now and never bothered to thoroughly read the documentation. I have always used an inferior work around for this thank you for showing me the formal solution.
Just a tip, if you are trying to create a bunch of sequentially numerated files with touch, you can use bash ranges {} syntax, like "touch {1..1000}".. It's really convenient, and it works with other types of sequences (a..z), and you can even pass a step in which next item will be incremented :) Nice video, btw! :)
Great structured intro into xargs, thanks DT! Here is an approximate ToC for this video: 00:48 basic xargs usage + echo 02:03 xargs -t flag = show expanded output 02:55 xargs + ls = recursive ls 03:36 example: /etc/passwd 04:00 passwd + cut + sort + xargs 04:41 xargs -I (capital i) flag = alias for arguments 06:19 example: make 1000 txt files: seq + xargs + touch 07:21 example: mv *.txt to *.text with ls + cut + xargs 08:35 xargs -n flag (number of args) = multiline output with 09:39 example: seq + xargs + bash + sleep 11:34 example: xargs + find = remove all *.text 13:50 performance: find --exec vs xargs rm 15:18 summary This table of content was created using "Smart Bookmarks for RUclips" chrome extension.
xargs is a true lifesaver when it's needed. One key example is when you have a tool that needs to run on every file in a directory. Using ls and xargs with -n 1 can really blast out the results compared to trying to bash for loop it or something. The problem is people don't even know about it to consider it as an option.
Yeah I've always struggled to write bash for-loops because 1) They look kinda noisy when written inline and 2) I don't usually write my code in bash. With 'ls -1 | xargs' (or with -n 1 like you said) the command looks so much cleaner and is more declarative
Hey DT, your comparison of "-exec" and "xargs" wasn't entirely fair. Using -exec with the semicolon is more similar to calling xargs with the "-L 1" option. A fair comparison would be to use "find ... -exec rm {} +". The "+" tries to fit as many arguments to one command as it can just like xargs by default.
@@sorrowfulevening8201 except the find results are not necesarily always in the same folder, nor will you know in what folders the results will pop up.
This is really useful! I made a script that processed several million files, output from the find command using -print0 and xargs with the -0 argument (for using paths / filenames with spaces) and launching another script, 16 of them in parallel with -n1 and -P16 arguments to xargs. The performance was phenomenal: the Dual Xeon E5-2660v2 (2 x 8 cores) usage was up to 95%, averaging about 80%. This is multi-threading bash scripting!
Rather than xargs, I've recently been learning how to use GNU parallel, and I love it. Its syntax is more cumbersome to work with than xargs, but it's also much more powerful. You can interleave multiple inputs (e.g. from stdin and files at the same time), position the incoming strings in any order you want in your command, and modify them before use (e.g. to remove file paths). You can even distribute jobs over multiple machines via ssh. I've updated a few of my scripts to use it and the results have been exceptional. For example I have one script for converting flac files to vorbis, ensuring that the filename, tags, and art are all kept intact, and the time to process one album of tracks dropped from about 3 minutes to under 20 seconds. Sweet.
From Chile, thanks for your dedication and your team also, you are doing fantastic work every day, like the format of your videos and your english is very good, thanks for not talking fast!. I'm addicted to linux.
XARGS is one of those insanely useful tools. It's something in my toolbox that I use EVERY day when I manage any one of the hundreds of servers I could be dealing with each day. Just for clarification, XARGS doesn't make FIND any more or less efficient. FIND will execute the RM statement (Or whatever it's internal equivalent is) for each result it returns. XARGS will pass as many parameters to RM (or whatever command) as it can if you don't set a limit on the number of parameters to pass in. That means that the RM command may be run multiple times against a large number of files/parameters. RM is intelligent with file transactions, just the same as relational databases are intelligent with their ACID transactions. As an example to really put this to the test: for c in `seq 100` do echo "----" echo "${c} step" echo "Creating Files" time seq 1000 | xargs -i touch {}.txt echo "Running RM with ${c} parameters" time find . -type f -name '*.txt' | xargs -n ${c} rm done This will start a loop. In each loop, it'll create 1000 files, then promptly delete them passing a number of files/parameters to be deleted each loop. So on loop one, a single file/parameter will be passed to RM for it to be deleted. On the second pass, two files will be passed to RM, etc. Note that `seq 100` is wrapped in the BACK-TICKS (Unshifted-Tilde key on standard US keyboards, at least), it's not a standard single quote.
Great video! One thing I don't think you mentioned which is great about xargs is how it chunks the input into "shell sized" pieces. The shell has a limited length of a command line or a limit on the number of command line arguments, and if you supply more than these, the shell will refuse to execute your command (example: if you have a find command that produces a list of 1 million files, like `ls $(find / -iname "*.txt")`, the shell will complain the command line is too longl). However, if you use xargs, it will automatically chunk the inputs into sizes that let the shell execute the xargs "command" successfully. This is related to the "-n" flag of xargs, and I guess is the default value for the number of args per chunk if you dont' supply "-n". This chunking alone makes xargs worth it to use for me. Also, the '-0" flag to xargs is quite useful: it parses null-delimited input into distinct command line args. Sometimes useful with `locate -0 '*.txt' | xargs -0 ls -lrt`
very informative video, it literally answered all my questions. meaning all important points were considered and answered, like different use cases and comparisons with other similar tools, which is preferable in which scenario and so on. very good work indeed
It's still amazing that decades after shell and all these little tools were created for Unix that they're still insanely awesome. Still. That's just mindblowing. They were designed as the best operating system/logic building blocks of all time. Just a wonder of human ability to have nailed this.
Great timing DT. Was experimenting yesterday with fzf -m in combination with xargs and couldn't get the multiple inputs working. Them damn curly braces. Thank you.
Nice video, but recommending against using `find -exec` doesn't make sense. The reason it took so long is because you instructed it to run 1,000 individual rm commands by putting a semicolon at the end; replace that with a plus sign (or just use the -delete option in this case) and find will bulk delete those files in no time with no need for xargs.
Thanks again for these command line tutorials. I might honestly start scripting soon for really simple actions. Your videos are definitely where I look for information on all that. I might honestly learn posix shell script or bash or something haha
I use bash and expect scripting exclusively to manage Cisco devices in an enterprise environment. Don't have to worry about libraries from other programming languages being installed on whatever system I am using at the time.
I am learning a lot of command line tips and tricks from you. Thanks. By the way, you frequently use 'clear' to clear screen. You could use ctrl-l (el, not ai) to do the same, which saves a few finger movements.
So should Zshell. Escaping variables is kind of required to be a good shell. Dont forget to use the newline delimiter. The reason piping find is faster is because find has to find all the files, in a single thread, then run the command. xargs can group those files, run multiple processes, run while find is still working too.
Another fun tip for xargs subshells is you can export a function with export -f functionname and call it from the subshell. Makes it easy to write a parallel executed pipeline of bash functions.
Good video! I have one more command in my arsenal. But for the files creations, you can easily do that with bash brace expansion. The syntax looks like this: touch {1..1000}.txt
Correct me if I'm wrong, but isn't the -exec functionality of find slower because it's executing the command for each individual find result, much like running `xargs -n 1`? I tested the find command using `-exec rm {} \+"`, which appends the results to a single command, and got the same time results as using xarg. So if you're using find, it's pretty much comparable to use -exec or xargs interchangeably, for basic tasks, though xargs is obviously more versatile for tasks that aren't related exclusively to the results of find.
@Terminalforlife (LL) That's where I went to confirm the use of -exec vs xargs, and it's exactly as I described. (The man file even makes the comparison to xargs for `+`.) I tested the 4 commands myself. find using `;` and `xargs -n 1` have nearly the same execution time. find using `+` and `xargs` also have nearly the same execution time (in fact find is just slightly faster.) I'm not trying to say that one method should always be used over the other, as it really depends on the situation. The point is that -exec is not slower than xargs when making an equivalent comparison.
Nice, learned a few new things. Regarding 'printf': it's not the quotes wich where missing, but that the first argument of printf should be a format string (e.g. "%s " to get the same behaviour as 'echo'). Regarding 'find' --exec: 'rm' isn't the best example here, since 'find' has a '-delete' flag which does the same but as fast (or faster) as your xargs solution. But I can see how this trick can be useful in general. Another thing which might have been nice to show: I often use xargs without any input pipe, so it just takes input from stdin which I find useful if I just want to process a list I've in my clipboard.
I use -a option to read from a file of hosts, and xargs can ping every host in the file. Using the -P option lets me ping multiple hosts at a time, and I can find the status of hundreds of hosts in a matter of seconds using ping. Good for identifying network problems.
find -exec ... {} + makes it faster and safer than piping to xargs.. As opposed to find -exec ... {} \; which loops through each arg instead of processing multiple args at a time.
Your first example dosen't need "xargs" echo will print out the output of the command, that is enclosed with ` Example: "echo `seq 5`" will output the same as "seq 5 | xargs"
I'm sure 'time' measures the execution time of only the find command, this includes the file deletion with -exec, but not with |xargs, so is this a fair comparison?
Hey! This was very helpful, lucid and comprehensive. I wanted to ask .. how do we set the time taken to be displayed once a process is executed - Like it showed " took 5s" for you. Please do make a video for the same. Thanks in advance!
I always used subshells, like (ls -l -a -h) = (ls `echo "-l -a -h"`) = (ls $(echo "-l -a -h")) = etc. At 1:08 that would be (command2 `command1`) or (command2 $(command1)) But yeah xargs is way nicer, because otherwise you would end up in bracket hell at some point XD
I've been writing bash scripts for years now and never bothered to thoroughly read the documentation. I have always used an inferior work around for this thank you for showing me the formal solution.
Yay uwu
yes i always would just set whatever i needed as a variable and use that but sometimes it just wouldnt work
Just a tip, if you are trying to create a bunch of sequentially numerated files with touch, you can use bash ranges {} syntax, like "touch {1..1000}".. It's really convenient, and it works with other types of sequences (a..z), and you can even pass a step in which next item will be incremented :)
Nice video, btw! :)
Thank you DT for all what you do.
Thanks for watching!
All what you do?
Yes
@@eli1882 whetever
Great structured intro into xargs, thanks DT!
Here is an approximate ToC for this video:
00:48 basic xargs usage + echo
02:03 xargs -t flag = show expanded output
02:55 xargs + ls = recursive ls
03:36 example: /etc/passwd
04:00 passwd + cut + sort + xargs
04:41 xargs -I (capital i) flag = alias for arguments
06:19 example: make 1000 txt files: seq + xargs + touch
07:21 example: mv *.txt to *.text with ls + cut + xargs
08:35 xargs -n flag (number of args) = multiline output with
09:39 example: seq + xargs + bash + sleep
11:34 example: xargs + find = remove all *.text
13:50 performance: find --exec vs xargs rm
15:18 summary
This table of content was created using "Smart Bookmarks for RUclips" chrome extension.
xargs is a true lifesaver when it's needed. One key example is when you have a tool that needs to run on every file in a directory. Using ls and xargs with -n 1 can really blast out the results compared to trying to bash for loop it or something. The problem is people don't even know about it to consider it as an option.
Yeah I've always struggled to write bash for-loops because 1) They look kinda noisy when written inline and 2) I don't usually write my code in bash. With 'ls -1 | xargs' (or with -n 1 like you said) the command looks so much cleaner and is more declarative
Hands down the best rundown of xargs I've ever seen, you rock my dude!
You're content is getting way better DT, congrats! I love this vid, really nice info.
Yes.
Hey DT, your comparison of "-exec" and "xargs" wasn't entirely fair.
Using -exec with the semicolon is more similar to calling xargs with the "-L 1" option. A fair comparison would be to use "find ... -exec rm {} +". The "+" tries to fit as many arguments to one command as it can just like xargs by default.
@Terminalforlife (LL) rm -f *.txt will do the job, no need for find.
@@sorrowfulevening8201 except the find results are not necesarily always in the same folder, nor will you know in what folders the results will pop up.
@@amx2311 agree.
@@amx2311 ... or if you have too many files to fit to arguments length.
This is really useful! I made a script that processed several million files, output from the find command using -print0 and xargs with the -0 argument (for using paths / filenames with spaces) and launching another script, 16 of them in parallel with -n1 and -P16 arguments to xargs. The performance was phenomenal: the Dual Xeon E5-2660v2 (2 x 8 cores) usage was up to 95%, averaging about 80%. This is multi-threading bash scripting!
Never knew about xargs! Neither did my spellchecker. Now we both do! Thanks DT
Rather than xargs, I've recently been learning how to use GNU parallel, and I love it. Its syntax is more cumbersome to work with than xargs, but it's also much more powerful. You can interleave multiple inputs (e.g. from stdin and files at the same time), position the incoming strings in any order you want in your command, and modify them before use (e.g. to remove file paths). You can even distribute jobs over multiple machines via ssh.
I've updated a few of my scripts to use it and the results have been exceptional. For example I have one script for converting flac files to vorbis, ensuring that the filename, tags, and art are all kept intact, and the time to process one album of tracks dropped from about 3 minutes to under 20 seconds. Sweet.
I love this videos explaining those basic CLI tools. Because if you don't know they exist, its hard to find them. ESpecially for new users.
From Chile, thanks for your dedication and your team also, you are doing fantastic work every day, like the format of your videos and your english is very good, thanks for not talking fast!. I'm addicted to linux.
XARGS is one of those insanely useful tools. It's something in my toolbox that I use EVERY day when I manage any one of the hundreds of servers I could be dealing with each day.
Just for clarification, XARGS doesn't make FIND any more or less efficient. FIND will execute the RM statement (Or whatever it's internal equivalent is) for each result it returns. XARGS will pass as many parameters to RM (or whatever command) as it can if you don't set a limit on the number of parameters to pass in. That means that the RM command may be run multiple times against a large number of files/parameters. RM is intelligent with file transactions, just the same as relational databases are intelligent with their ACID transactions.
As an example to really put this to the test:
for c in `seq 100`
do
echo "----"
echo "${c} step"
echo "Creating Files"
time seq 1000 | xargs -i touch {}.txt
echo "Running RM with ${c} parameters"
time find . -type f -name '*.txt' | xargs -n ${c} rm
done
This will start a loop. In each loop, it'll create 1000 files, then promptly delete them passing a number of files/parameters to be deleted each loop. So on loop one, a single file/parameter will be passed to RM for it to be deleted. On the second pass, two files will be passed to RM, etc.
Note that `seq 100` is wrapped in the BACK-TICKS (Unshifted-Tilde key on standard US keyboards, at least), it's not a standard single quote.
This is truly much better than reading the manual pages! loved the accent btw :)
These command line utility videos are amazing. Thank you for the content.
This was an incredibly well written and produced video. Really spectacular job running down this command.
Thank you, very clear explanation and tested and documented. After 17 years, finally I will use xargs
Love the new bash tuts! Looking forward to more
Great video! One thing I don't think you mentioned which is great about xargs is how it chunks the input into "shell sized" pieces. The shell has a limited length of a command line or a limit on the number of command line arguments, and if you supply more than these, the shell will refuse to execute your command (example: if you have a find command that produces a list of 1 million files, like `ls $(find / -iname "*.txt")`, the shell will complain the command line is too longl). However, if you use xargs, it will automatically chunk the inputs into sizes that let the shell execute the xargs "command" successfully. This is related to the "-n" flag of xargs, and I guess is the default value for the number of args per chunk if you dont' supply "-n".
This chunking alone makes xargs worth it to use for me.
Also, the '-0" flag to xargs is quite useful: it parses null-delimited input into distinct command line args. Sometimes useful with `locate -0 '*.txt' | xargs -0 ls -lrt`
very informative video, it literally answered all my questions. meaning all important points were considered and answered, like different use cases and comparisons with other similar tools, which is preferable in which scenario and so on. very good work indeed
Excepcional!!! Estuvo grandiosa la explicación y ejemplos. Ahora, a ponerlo en práctica.
Awesome!
This is one of my favorite channels!
Congrats!
It's still amazing that decades after shell and all these little tools were created for Unix that they're still insanely awesome. Still. That's just mindblowing. They were designed as the best operating system/logic building blocks of all time. Just a wonder of human ability to have nailed this.
Great timing DT. Was experimenting yesterday with fzf -m in combination with xargs and couldn't get the multiple inputs working. Them damn curly braces. Thank you.
Nice explanation DT. Just found out about this channel what a gem.
Concise and informative as always!
Good stuff my dude. I had to open up a terminal and try this out myself.
Nice video, but recommending against using `find -exec` doesn't make sense. The reason it took so long is because you instructed it to run 1,000 individual rm commands by putting a semicolon at the end; replace that with a plus sign (or just use the -delete option in this case) and find will bulk delete those files in no time with no need for xargs.
Thanks again for these command line tutorials. I might honestly start scripting soon for really simple actions. Your videos are definitely where I look for information on all that. I might honestly learn posix shell script or bash or something haha
I use bash and expect scripting exclusively to manage Cisco devices in an enterprise environment. Don't have to worry about libraries from other programming languages being installed on whatever system I am using at the time.
AWESOME SIRE! Much respect.
I am learning a lot of command line tips and tricks from you. Thanks. By the way, you frequently use 'clear' to clear screen. You could use ctrl-l (el, not ai) to do the same, which saves a few finger movements.
That's such a cool command. I like it.
Thanks. This will be very time saving
So should Zshell. Escaping variables is kind of required to be a good shell.
Dont forget to use the newline delimiter.
The reason piping find is faster is because find has to find all the files, in a single thread, then run the command. xargs can group those files, run multiple processes, run while find is still working too.
Another fun tip for xargs subshells is you can export a function with export -f functionname and call it from the subshell. Makes it easy to write a parallel executed pipeline of bash functions.
I'm interested in this, can you elaborate? Maybe a quick example, thanks!!
Lovely explanation. Thanks a ton DT
Another great video from DT. Many thanks.
Good video!
I have one more command in my arsenal.
But for the files creations, you can easily do that with bash brace expansion. The syntax looks like this:
touch {1..1000}.txt
Correct me if I'm wrong, but isn't the -exec functionality of find slower because it's executing the command for each individual find result, much like running `xargs -n 1`? I tested the find command using `-exec rm {} \+"`, which appends the results to a single command, and got the same time results as using xarg.
So if you're using find, it's pretty much comparable to use -exec or xargs interchangeably, for basic tasks, though xargs is obviously more versatile for tasks that aren't related exclusively to the results of find.
@Terminalforlife (LL) That's where I went to confirm the use of -exec vs xargs, and it's exactly as I described. (The man file even makes the comparison to xargs for `+`.) I tested the 4 commands myself. find using `;` and `xargs -n 1` have nearly the same execution time. find using `+` and `xargs` also have nearly the same execution time (in fact find is just slightly faster.)
I'm not trying to say that one method should always be used over the other, as it really depends on the situation. The point is that -exec is not slower than xargs when making an equivalent comparison.
Thanks for another briliant tutorial
Very Very awesome video. So much to learn.
thank you DT, love your content.
Thank you DT for this amazing video!!!
The lighting in this video is flattering for your blue or green eyes.
Very good explanation!!!!!!
So useful, and so well explained!
This is very very cool.
Nicely done DT.
12:13 What about using the -delete flag with find command?: find . -type f -name "*.text" -delete Is there any advantage over -exec flag?
Great video!
Love ya DT!! thank you!
Just discovered your channel, love it
Thank you very much for this video
I'm also impressed by perf of xargs
Wow, cool explanation. Thank you
very useful and perfectly demonstrated. 👍🏻
A pearl as usual. Keep it up dt
Nice, learned a few new things. Regarding 'printf': it's not the quotes wich where missing, but that the first argument of printf should be a format string (e.g. "%s " to get the same behaviour as 'echo'). Regarding 'find' --exec: 'rm' isn't the best example here, since 'find' has a '-delete' flag which does the same but as fast (or faster) as your xargs solution. But I can see how this trick can be useful in general.
Another thing which might have been nice to show: I often use xargs without any input pipe, so it just takes input from stdin which I find useful if I just want to process a list I've in my clipboard.
Thanks! Learned several things here.
Drove my Alexa totally mad! I thought having the wake up word as Echo was a good idea 🥺
It's so cool xargs. Thank you!
Thank you, very much, my friend!
Thank you good sir, your videos are very helpful.
I use -a option to read from a file of hosts, and xargs can ping every host in the file. Using the -P option lets me ping multiple hosts at a time, and I can find the status of hundreds of hosts in a matter of seconds using ping. Good for identifying network problems.
Good video, thank you!
Thank you. Very helpful.
Awesome video, I learned something new, thank you
Great video DT thanks
Thanks I have been using exec forever. As you said it works but the timing shows xargs is more efficient.
find -exec ... {} + makes it faster and safer than piping to xargs.. As opposed to find -exec ... {} \; which loops through each arg instead of processing multiple args at a time.
That's a nice tutorial! thanks a lot for it
XXXX is a brand of beer in my country. People use it as standard input ALL THE TIME.
Very useful video! Thank you!
Excelente video, felicidades.
Greate Share! cool expanation, -exec is always a huge mess.
a count of processes in "-P" would keep the system responsive to other commands and functions. usually a quarter of the "ulimit -u"
i love your time cmd very much
I use xargs to push to all git remotes:
alias gpsa="git remote | xargs -I remotes git push remotes master"
beautiful video, thank you sir
Toinen hieno video DT
Your first example dosen't need "xargs"
echo will print out the output of the command, that is enclosed with `
Example: "echo `seq 5`" will output the same as "seq 5 | xargs"
Beautiful
Comment to bring this video to the tops
Thank you!
Thanks for this
I'm sure 'time' measures the execution time of only the find command, this includes the file deletion with -exec, but not with |xargs, so is this a fair comparison?
please do a tutorial on shell expansion
thank you sir
Although I know a bash substitutes for some of the things shown in video there is no harm learning new methods of doing things.
Hey! This was very helpful, lucid and comprehensive.
I wanted to ask .. how do we set the time taken to be displayed once a process is executed - Like it showed " took 5s" for you. Please do make a video for the same.
Thanks in advance!
Amazing content!
Hey DT use + instead of ; . + appends
Hey DT! How do you get " took 5s" in the terminal prompt ? What do I need to add to the command prompt to get the time it took to execute a command ?
It's part of the starship prompt. I did a video on starship some time ago. It's a great shell prompt that works on bash, fish and zsh.
@@DistroTube I see, thank you :) I take a look at it. thanks for a quick answer! cheers!
Found something similar on github. It's called "fish-command-timer". just tested it and works.
Superb!
7:22 Oh my poor inode hehe
¡THANK YOU!
BTW, with ChatGPT I found a couple of interesting commands:
find "$(pwd)" -type f -iname "*.md" -print0 | xargs -0 grep -l -i "keyword"
find "$(pwd)" -type f -iname "*.md" -print0 | xargs -0 grep -zi "keyword" --files-with-matches
and
find "$(pwd)" -type f -iname "*.md" -print0 | xargs -0 grep -n -H -i "keyword"
Really nice.
Hey DT, please share your colorscript which comes when you do a clear command
Check my dotfiles repo on my GitLab (in show description). The clear color thingy is function in my config.fish.
I have very strong "essing" on your videos is there any way you could turn down the upper frequencies in an equalizer for us all?
Or probably a de-esser would work
Yea same
or even the compressor setting
I always used subshells, like (ls -l -a -h) = (ls `echo "-l -a -h"`) = (ls $(echo "-l -a -h")) = etc.
At 1:08 that would be (command2 `command1`) or (command2 $(command1))
But yeah xargs is way nicer, because otherwise you would end up in bracket hell at some point XD