How The 'awk' Command Made Me A 10x Engineer
HTML-код
- Опубликовано: 7 сен 2024
- Learning to use the 'awk' command increased my productivity significantly. awk allows me to quickly manipulate and inspect data in text files. This is especially true for column-based data, such as .csv files. Since awk is a fully functional programming language, I can quickly use awk to solve problems that are too complicated for programs like grep or sed.
A text-based version of this video with all commands can be found here:
blog.roberteld...
SOCIALS
----------------
Patreon: www.patreon.co...
Tiktok: / roberteldersoftware
Linkedin: / robert-elder-software
Blog: blog.roberteld...
Twitter: / roberteldersoft
Twitch: / roberteldersoftware
Github: github.com/Rob...
Facebook: / roberteldersoftware
Instagram: / roberteldersoftware
Merch: store.robertel...
the narrative that lead to the tutorial is so cute as it never happened.
I sure hope so, or someone needs to learn about spreadsheets.
it never happened indeed and a maximum of 4 lines of python can do this, that's why I avoid learning these tools
spreadsheet? is that like lasagna or vegemite or nutella,@@Bjawu ?
@@jazzochannel There *are* some similarities with vegemite, now that I thing about it. You don't want to use too much.
I feel like people just don’t get this guy’s sense of humor
I used to use awk every day about 30 years ago. A nice walk down the memory lane with this video.
memory lane? why did you stop?
@@jazzochannelmy work now doesn’t involve coding and data manipulation any more.
The awk command is now my favorite Linux command
I avoided Awk for 15 years, but then I saw the light.
@@catsupchutney I have a year left of avoiding it like the plague, maybe. Then I'll give it a shot.
Many moons ago I helped maintain what we thought might be the largest awk script ever created - it converted between the proprietary FORTRAN code for compilers by our competitors into our own proprietary FORTRAN code. Of course, monthly, if not weekly, we'd find something it did not convert properly, and it would have to be changed. Not only was it possibly the largest awk script, but it was the most modified awk script...
Fascinating! Do you remember how many lines (roughly) it was?
@@artemiasalina1860 It was well over a thousand lines.
Sounds awk ward to work with…. I’m sorry
@@bide7603It's not funny, it's a way of life...
I once took a little boy under my wing who thought that text manipulation equated to social skills. He was a little awk ward
Thanks for sharing.
Good luck with your 1% raise!
Thank you!
@@RobertElderSoftware so... did you get the raise?
I like how there was a narrative style to the video. That's really engaging. You've earned a sub!
Great pace too.
Utilities like awk are one of the reasons I love Unix. Everything is in a text file and the OS gives you the tools to parse and manipulate the contents.
Wow.. I know awk well, but what a presentation. This is an underrated video. Keep it up dude.
Thanks, will do!
I enjoy using GREP and SED and I've been really wanting to learn how to use AWK. This video motivated me to do so, I'm gonna!! Also LOL, the end of the video made me laugh a lot. I love this!
Bro went from a 0x developer to a 0.5x dev
10x0 is still 0
what a hater, the video was great
It is but the title is wayy over the top
Dude........ Your content is brilliant. You must become more known! We desperately need more Unix content on youtube.
The solution to the original problem was a bit of overkill, but this was a good example for some teaching of base awk functionality. This is a more direct and shorter solution, only changing the F lines and printing all lines:
$ awk '$2=="F" {$1=sprintf("%.1f",($1-32)/1.8); $2="C";} 1' temps.csv
(the single "1" at the end is the condition, which evaluates as "true", evoking the default action: print -- nice shortcut if you mean only to change some lines but print all)
Excellent. Glad to see someone else recognized and posted the solution I thought was more elegant
And taught me something new with that “1” trick!
Thank you for describing the pattern-matching and default assumptions so thoroughly and illustrating each point with example code. Until today have never seen any awk tutorial that could get me up to speed so quickly as your video. Kudos!
Showing how awk can replace both the grep and sed commands was icing on your cake.
And your last sentence was a big flaming torch sitting atop the cake!
Get real- once you learn regex, and in my own world, specifically Perl regex, that stuff becomes fairly trivial. "Analysing log files from umpty-hundred jobs" used to be a chore but these days I can script a working solution in an hour or so. In Perl. Because that's the tool I'm familiar with.
'awk', 'sed', and 'Perl'. All I ever need for scraping from log-files!
head and tail is also useful for a redneck like me :D
If you know Perl, do you still have a use for Awk? (I'm wondering if I should learn it, I already know Perl. From the looks of this video, I can do the same with Perl in maybe 20-30% more characters - not enough more to trigger learning it)
@@obiwankenobe3962 Sure. Back in the old Unix days you needed sed and awk to to text munging. Now I just use Perl for all of it. Perhaps more characters to type in, but much simpler syntax and easier to learn, IMO.
@@BytebroUK thx for confirming my suspicion
@@obiwankenobe3962 old comment but still, Perl was devised in part to be awk/sed killer, it has special flags for "read a line at a time and do stuff" and syntax for "write-only" one-liners.
Perl is a Swiss Army Chainsaw, and that includes awk and sed functionality. It borrows quite a bit from sed and awk syntax.
This is video is great and demystified awk in a very easy to understand way
I get that awk is a useful tool overall, but with your given example, why exactly would writing a program be time consuming?
i think it was just a poor attempt at a motivating example and a joke at the same time
the whole premise was quite funny
It would take as long to write in python as it would take learning awk. At least pre AI, now AI could generate the command in moments.
@@jeanhaley3051 i just tried writing that python script and measured how much time it took me (without trying to rush it), it took 1 minute. it might be faster with awk in the long run? but 1 minute is good enough for me
Even less code with Perl. And this example is not recommended at all. What happens when the data is changed. It all fails.
so much content in just 10mins. amazing!
I love how simple programs developed 50 years ago are still extremely useful.
That is because the core of Computing hasn't changed much at all over that time, and UNIX did an excellent job of capturing that core.
I pity the fools who know nothing outside of a GUI. Because with a GUI, you get a single tool that can do only what its designers allow you to do. But the UNIX concept is to have a load of little tools that one can combine to do exactly that what you want it to do.
Perhaps I should write a GUI overlay for the shell, allowing the old UNIX tools to be configured and linked together graphically. That would be fun, and might help people discover the tools that are available at the CLI.
Finally and after so many years, I understand how 'awk' works. Thank you, Legend
I always thought awk was super hard to learn. 😭 Thanks for explaining it so well! 🙏
Great video! So, it's 3 years later, I hope you got your juicy 1% raise!
@4:10 "As you see, awk is not quite this smart. And here you can see how awk isn't quite as good as a CSV parser. "
In GNU awk set the built-in FPAT variable. In your example: FPAT = "([^,]+)|(\"[^\"]+\")"
"I was so unproductive!" → Proceeds to open the Gnome terminal
Alacritty FTW 🙂
This is why people hate the Linux community.
using a better terminal does t make you more productive, just pipe the massive prints so you dont have to worry about print speed on compilation or something. (i say this as someone who uses a fast terminal (foot))
The temperature data format is a good one for exercising awk. This is a data driven world.
Amazing presentation! I've been avoiding Awk for close to a decade, so much so I default to Sed for this kind of tasks! After watching your video, I feel more confident to try Awk next time :) Thank you!
Excellent presentation of how AWK works and how it could benefit you.
I haven't used it personally (I would use Python: slightly more complex to use, but MUCH more powerful), but I know people who use it often, e.g. to scan log files for certain situations. It is great for iteratively improving the filters, browsing through the data.
I've never done more with awk than printing columns from a text file, like the 'cut' command. And for other things, use a perl one-liner. Loved the average calculation example in awk. Good to have a new tool in the belt!
At my past job where we used Linux exclusively, I used awk extensively along with bash scripts, sed, tr, od, etc. There are so many handy Linux commands.
This is an amazingly instructive video. Thank you.
BEST INTRO TO AWK. PERIOD!
This was an exceptionally good tutorial. Thank you.
This is such a great explainer of a tool i rarely used, but saw the potential, although i never dove in deep enough to learn it. Thank you!
Great presentation style. I just subscribed, hoping to hear if you got that 1% raise.
I didn't get any raise this year, but I've got a good feeling that if I work overtime for just a couple more years, someone will finally notice my hard work :)
Humorous but informative, I'm no longer afraid of awk, nice video
Don't even ask how many python scripts i wrote to do this kind of stuff, grappling with pandas csv reader and such
Before awk I was living on the street and had no significant other.
With awk I'm now living on a mansion and have two wives 🎉
The awk command is my favorite linux command. 😂😂😂 Great content! Keep going!
It's been 3 years since you made this video. I really hope you got your raise.
Not yet, but I'm still hopeful :)
At a time when Excel had only 65000 lines, I used awk to condense big text data files to a manageable size used the associative array feature of awk.
If you’re working with same files over again, I suggest to include one sample line for each pattern because you may not recognise the line from the pattern after 2 months.
The Fahrenheit example I’d done in libreOffice. Import and so on.
There was a time where you had difficulties getting an awk that works on windows.
I used awk and sed to convert entire flight manuals into our dataformat that won us the deal with pilatus as we could convert it to our software suit.
Nice tips and examples, man.
Greetings from Brazil.
I piped awk into sed once so I could graph certain lines of a csv dataset in Excel because Excel couldn't open the full file for me to filter it. Awk hey that worked, yeah that's what I sed.
It's been 3 years since you posted the video. I hope you got that raise. Gotta offset inflation somehow.
I challenged a guy who believes in awk, I let him define the problem. He could use awk, sed, … and I would use cpp, yacc, and lex. The challenge was on two fronts, dev time - first to start, and run time. He insisted that he may run slower than me, but he would clearly win dev time.
Well, I ended up not using yacc, just cpp and lex. Beat him in dev time by nearly an hour. Mine ran in less than a second and his took a few minutes.
To me awk is awkful with a silent K. It is a very useful tool, but it is just so easy to create your own specific, and much better tool quicker than using awkful.
I really think we need better parsing libraries and at some point the most effective way to dev will be an editor that is easy to add features to which essentially will be coding from the command line. Even between awk and sed there are editing cases where I cannot cleanly implement my operation/macro.
Neovim?
Ah the classic 10x engineer,
Making 10x the work for everyone else. I too have earned this little.
1.5% raise for the gentleman please.
As a data engineer I did a lot in VIM. But mainly just for small stuff like formatting sample files that I would test transform. And finding anomalies in huge files. Not as powerful as awk, but also a different interface.
AWK is very cool. I wore a small program using it and Bourne Shell that was about 300 lines long, but it would have been a lot longer if I was using something else. I think people probably overlook it nowadays because it's old and it's syntax is a bit archaic, but it's very powerful!
Real world examples IME look something like this:
Some poorly documented program just spat out a noisy status message and produced a bunch of poorly named files. You want to parse the message to associate metadata with the output files, then rename the files with metadata as part of the name.
First pipe message into awk to separate into fields, then do simple transform in awk or pipe into sed or grep, then sort file names for association with the metadata output. Then pipe both into awk again to generate a shell script to rename the files.
"I was immediately able to complete this task in 1 second with the following command" (after spending an hour learning Awk, followed by another 15 mins writing and testing the command). On a related note did you know it is possible to write an entire text processing program in 10 characters using this little known concept called "regular expressions"? That's right, you only have to type 10 characters, that's ALL there is to it, it's just so quick and easy!
Nice Video, this is a great explanation of awk, thank you for sharing.
I like your shorts "this is my favorite command", but the longer are more explanatory and better pauseable aswell. Thanx
Awk is a very powerful program indeed. The only thing that keeps it from completely replacing grep for me is the lack of a Perl mode for regex.
agreed - awk is awesome and i use it all the time - the most extreme example was a ~400 line script to analyze network files extracted from big integrated circuits but it’s great (and fast) at processing big text files - and it’s highly portable too since it’s installed on pretty much every unix/linux system dating back to the 1960s
You are the great teacher
In VSCode:
Step1: Find and select all °C.
Step 2: Copy them in a new tab
Step 3: Modify the text to have the conversion formula.
Step 4: Use Math Calculate extension to get the result
Step 5: Select, copy, and paste back in original file
Mmm... Demasiados pasos y todavía tengo que instalar VSCode para poder hacer una tarea tan trivial.
Its now 3 years later, he better have gotten that 1% raise
Here we work to live, we don't live to work. Social life is more important than work.
I remember when I used to torture myself with arcane bash commands and construct a towering awk command for complex file parsing. The cryptic awk errors were chef's kiss.
Honestly I just prefer any other language at that point.
0:30 dedicated Python code for that task is 6 lines.
``python
with open('temps.txt', 'r') as input_file:
with open('temps_all_c.txt', 'w') as output_file:
temp, scale = line.strip().split()
if scale == 'F':
temp = (float(temp) - 32) * 5/9
out_file.write(f'{temp} C
')
```
I use awk over grep daily because of it's usefulness in creating conversions or finding ways of pattern matching smartly. It's grep and cut in one. The only time i use cut is for specific columns due field separators being a pain sometimes.
Writing a script in C#, F# or Python to do this takes 1 minute and the code will be more understandable and can be copied into your project. Awk is Awesome in niche situations, but I'd only use it when other languages are not available.
yeah given this is just a backstory case for a awk lesson.
It still really depends the tech the dev is more used to. If the peson uses shell commands a lot everyday, it's easy to just awk, great to use what it offers. Otherwise is just no-brainer to write a small standalone app/script in your fav language (or any tech), hell it's most of the times just easier to google spreadseet or prompt into ai assistant (that appeared after this video being posted)
The awk command is my favorite command.
1% raise 😂
Thanks for the tutorial!
Awk is great, but not the only hammer for every nail. Sed, cut, bc, (node maybe), vim macro, bash, lots of ways to solve this with one line or so
Writing a dedicated program would have taken about 7 lines of code and 5 min, not forever. But, I digress. Us Nix folks live by grep, sed, awk, cut, xargs, etc.
Today AWK isn't only a text-processing utility, it is a complete general purpose programming language! You can use loops, variables, regex, conditionals, functions, array and string manipulation, math lib, include files...
If you're working in a Windows environment, you can use PowerShell in much the same way for data processing tasks like this.
That said, I really need to start using awk in Linux instead of just contorting sed into doing things it wasn't made to do.
Awesome explanation. Thank you
Very cool. I hope you don't mind, but I've just placed a link to this video in the awk chapter of the new shell scripting book that I'm currently writing. (It will be out sometime in the spring.)
Many thanks!
Thanks for the link, I feel honored :)
For reference, here is a PowerShell solution for that problem:
Import-Csv .\temps.csv -Delimiter "`t" | Select-Object @{Name="temp";Expression={if($_.unit -eq "F") {"$([math]::Round(($_.temp-32)/1.8,1))"} else {$_.temp}}}, @{Name="unit";Expression={"C"}} | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | %{$_.toString().Replace("`"", "")}
However normally you wouldn't do that as one line in PowerShell. You can just save the whole table as a variable and work on that instead:
$temps = Import-Csv .\temps.csv -Delimiter "`t"
foreach($entry in $temps) {if($entry.unit -eq "F") { $entry.unit="C"; $entry.temp="$([math]::Round(($entry.temp-32)/1.8,1))" }}
$temps | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | %{$_.toString().Replace("`"", "")}
Assuming you need it in exactly the same format again. I'd argue that using commas in a csv file and quoting the values would be better though.
The Awk command is my favorite Linux command
This is my favorite comment.
I'll never use IDEs and Vscode anymore to process texts!
Thanks ❤
It's 3 years later, did you get that raise bro?
Awk is great. Been using it for 20+ years
in 2024 it's counter-productive to rely on awk as it's missing libraries to work with common formats: multiline rows, binary (like parquet), csv with quotes, headers, footers, excel
i.e. pandas.read_csv in jupyter/ipython would do the same (could have a one-liner if you wish) but it's compatible with binary formats and anything you could possibly think of.
another option is excel if there is no need to repeat the transformation.
Okay, so this was three years ago. Did you get that raise? 🤣
I'm still waiting, but hopefully just another few years of overtime and someone will finally notice my hard work :)
Hmm, first time I watched this video, sound was behind the image, so you looked like Bruce Lee speaking in movies 😁 I thought it was intentional at first, then realized it was a problem in my browser. AWKward.
the awk command is my favorite linux command.
I agree that awk is cool and powerful, especially if you are working exclusively in a shell script, but, as you mentioned, it's basically a language of it's own, so instead of learning a new one, I'd rather just make a quick Python ascript that does exactly what I want in a more legible way
Super Useful, and entertaining
Writing a dedicated program wasn't an option, because that would take forever
1) I could write a simple Python script to solve this in minutes, maybe even under a minute.
2) awk commands are essentially little programs, except you are probably not very proficient with it and crafting the command will take more than writing the code in any other language you are familiar
and this task was never going to be repeated again
3) I cannot count the number of times I thought I will not repeat a task, didn't save the solution and then solved the thing all over again later.
writing a dedicated program takes forever? I agree awk does faster, but writing a python script will take at most half a minute...
When I look at the command you ended up with I think if it was me it would have taken me less time to write it in python that I already know than to learn all that in awk.
Dude, this video was sick! Looovely stuff. You're awesome: 👑
I love this video. Awk is one of the basic UNIX tools I always skipped over.
Bad call, lol.
I started to use perl for most of those kinda things years ago.
But indeed AWK is a usefull tool.
Beside stuff like AWK, SED or PERL differs you from the dotNot kiddies.
0:10 before I read about the emacs command I was a very productive employee. My boss would assign me a task and I would just do it.
Then I read about emacs and now my boss assigns me a task and I spent half a day configuring emacs and another half to write something in emacs that gets the job done
Perl was written because Larry Wall was frustrated with the Awk+Sed combo. Awk+Sed didn't do what he wanted it to do. Perl is still much better then the Awk+Sed combo, however very few people learn Perl but have reverted to learning Awk+Sed instead for text processing. Its a real shame because Perl is still hands down the best for text processing, whether it be on the command line or processing text files.
Awk was pretty much the first scripting language; it was the ancestor of Perl. It's largely taken over shell scripting on Stack Overflow these days; the kids there love it because they're so paranoid about only calling a single process, and doing everything in a single line. I think its' syntax is a bit tricky, personally; but it does make it extremely easy to create mathematical filters that are hard to do with virtually anything else.
Hello, thanks for sharing this knowledge. Interesting way of presentation and very concise and informative.
"writing a dedicated program to accomplish this wasn't an option, because that would take forever" ... i wrote a PHP script in about 2 mins to accomplish this and i was barely trying, am high AF, and watching an episode of TV at the same time. you could also use sheets or excel or numbers to do this pretty easily.
yep, awk changed my habits either, thanks for sharing!
There is also jq for json manipulation, but I've never used it in practice.
When you don't know how to name a function you made just call it by your name:
Awk is abbreviated from the names of the developers - Aho, Weinberger, and Kernighan (geeksforgeeks).
I'll be honest, I would have written a Perl script that took about 20 lines, and if we were co-workers, after seeing your one-line command... I would have returned to data entry.
awk is great! So many simple tools like grep, awk, and even more advanced-ish tools like jq are so often overlooked!
Though for me, once I know that I'm going to need something like an average temperature, I'd probably end up just running the python command and using the python repl which is already quite fast to work with.
Better yet, if they ever ask me to get the median or standard deviation, I can do so without the hassle of writing the equivalent awk script.
jq is insanely powerful 🙂
@@GreyDeathVaccine for real
I was absolutely convinced this was satirical. This would be imminently more readable, expandable and manageable as a python script which, unless you already completely mastered awk, would take about the same time to write. Kudos on you for showing what mastery of a command line tool can accomplish though.
>Write in python
No, that will take too long
>Learn a new tool
I got time for that
Writing a simple script to do that task wouldn't take forever. I'm pretty sure I could write one in Autohotkey (using the StrSplit() function) or in Python (using the split() method) in under 5 minutes, which is probably about the same time it took you to figure out that awk command.