Utilities like awk are one of the reasons I love Unix. Everything is in a text file and the OS gives you the tools to parse and manipulate the contents.
Many moons ago I helped maintain what we thought might be the largest awk script ever created - it converted between the proprietary FORTRAN code for compilers by our competitors into our own proprietary FORTRAN code. Of course, monthly, if not weekly, we'd find something it did not convert properly, and it would have to be changed. Not only was it possibly the largest awk script, but it was the most modified awk script...
The solution to the original problem was a bit of overkill, but this was a good example for some teaching of base awk functionality. This is a more direct and shorter solution, only changing the F lines and printing all lines: $ awk '$2=="F" {$1=sprintf("%.1f",($1-32)/1.8); $2="C";} 1' temps.csv (the single "1" at the end is the condition, which evaluates as "true", evoking the default action: print -- nice shortcut if you mean only to change some lines but print all)
I enjoy using GREP and SED and I've been really wanting to learn how to use AWK. This video motivated me to do so, I'm gonna!! Also LOL, the end of the video made me laugh a lot. I love this!
Thank you for describing the pattern-matching and default assumptions so thoroughly and illustrating each point with example code. Until today have never seen any awk tutorial that could get me up to speed so quickly as your video. Kudos! Showing how awk can replace both the grep and sed commands was icing on your cake. And your last sentence was a big flaming torch sitting atop the cake!
Get real- once you learn regex, and in my own world, specifically Perl regex, that stuff becomes fairly trivial. "Analysing log files from umpty-hundred jobs" used to be a chore but these days I can script a working solution in an hour or so. In Perl. Because that's the tool I'm familiar with.
@@jeanhaley3051 i just tried writing that python script and measured how much time it took me (without trying to rush it), it took 1 minute. it might be faster with awk in the long run? but 1 minute is good enough for me
@4:10 "As you see, awk is not quite this smart. And here you can see how awk isn't quite as good as a CSV parser. " In GNU awk set the built-in FPAT variable. In your example: FPAT = "([^,]+)|(\"[^\"]+\")"
If you know Perl, do you still have a use for Awk? (I'm wondering if I should learn it, I already know Perl. From the looks of this video, I can do the same with Perl in maybe 20-30% more characters - not enough more to trigger learning it)
@@obiwankenobe3962 Sure. Back in the old Unix days you needed sed and awk to to text munging. Now I just use Perl for all of it. Perhaps more characters to type in, but much simpler syntax and easier to learn, IMO.
@@obiwankenobe3962 old comment but still, Perl was devised in part to be awk/sed killer, it has special flags for "read a line at a time and do stuff" and syntax for "write-only" one-liners. Perl is a Swiss Army Chainsaw, and that includes awk and sed functionality. It borrows quite a bit from sed and awk syntax.
At my past job where we used Linux exclusively, I used awk extensively along with bash scripts, sed, tr, od, etc. There are so many handy Linux commands.
Amazing presentation! I've been avoiding Awk for close to a decade, so much so I default to Sed for this kind of tasks! After watching your video, I feel more confident to try Awk next time :) Thank you!
I've never done more with awk than printing columns from a text file, like the 'cut' command. And for other things, use a perl one-liner. Loved the average calculation example in awk. Good to have a new tool in the belt!
Excellent presentation of how AWK works and how it could benefit you. I haven't used it personally (I would use Python: slightly more complex to use, but MUCH more powerful), but I know people who use it often, e.g. to scan log files for certain situations. It is great for iteratively improving the filters, browsing through the data.
I love how simple programs developed 50 years ago are still extremely useful. That is because the core of Computing hasn't changed much at all over that time, and UNIX did an excellent job of capturing that core. I pity the fools who know nothing outside of a GUI. Because with a GUI, you get a single tool that can do only what its designers allow you to do. But the UNIX concept is to have a load of little tools that one can combine to do exactly that what you want it to do. Perhaps I should write a GUI overlay for the shell, allowing the old UNIX tools to be configured and linked together graphically. That would be fun, and might help people discover the tools that are available at the CLI.
agreed - awk is awesome and i use it all the time - the most extreme example was a ~400 line script to analyze network files extracted from big integrated circuits but it’s great (and fast) at processing big text files - and it’s highly portable too since it’s installed on pretty much every unix/linux system dating back to the 1960s
Real world examples IME look something like this: Some poorly documented program just spat out a noisy status message and produced a bunch of poorly named files. You want to parse the message to associate metadata with the output files, then rename the files with metadata as part of the name. First pipe message into awk to separate into fields, then do simple transform in awk or pipe into sed or grep, then sort file names for association with the metadata output. Then pipe both into awk again to generate a shell script to rename the files.
At a time when Excel had only 65000 lines, I used awk to condense big text data files to a manageable size used the associative array feature of awk. If you’re working with same files over again, I suggest to include one sample line for each pattern because you may not recognise the line from the pattern after 2 months. The Fahrenheit example I’d done in libreOffice. Import and so on. There was a time where you had difficulties getting an awk that works on windows.
I really think we need better parsing libraries and at some point the most effective way to dev will be an editor that is easy to add features to which essentially will be coding from the command line. Even between awk and sed there are editing cases where I cannot cleanly implement my operation/macro.
I remember when I used to torture myself with arcane bash commands and construct a towering awk command for complex file parsing. The cryptic awk errors were chef's kiss. Honestly I just prefer any other language at that point.
These elegant tools focused on practicality and performance are what made me fall in love with Unix (and then OSX). Windows feels like a cludgy toy OS in comparison.
using a better terminal does t make you more productive, just pipe the massive prints so you dont have to worry about print speed on compilation or something. (i say this as someone who uses a fast terminal (foot))
Humorous but informative, I'm no longer afraid of awk, nice video Don't even ask how many python scripts i wrote to do this kind of stuff, grappling with pandas csv reader and such
AWK is very cool. I wore a small program using it and Bourne Shell that was about 300 lines long, but it would have been a lot longer if I was using something else. I think people probably overlook it nowadays because it's old and it's syntax is a bit archaic, but it's very powerful!
I use awk over grep daily because of it's usefulness in creating conversions or finding ways of pattern matching smartly. It's grep and cut in one. The only time i use cut is for specific columns due field separators being a pain sometimes.
I didn't get any raise this year, but I've got a good feeling that if I work overtime for just a couple more years, someone will finally notice my hard work :)
As a data engineer I did a lot in VIM. But mainly just for small stuff like formatting sample files that I would test transform. And finding anomalies in huge files. Not as powerful as awk, but also a different interface.
When I look at the command you ended up with I think if it was me it would have taken me less time to write it in python that I already know than to learn all that in awk.
I challenged a guy who believes in awk, I let him define the problem. He could use awk, sed, … and I would use cpp, yacc, and lex. The challenge was on two fronts, dev time - first to start, and run time. He insisted that he may run slower than me, but he would clearly win dev time. Well, I ended up not using yacc, just cpp and lex. Beat him in dev time by nearly an hour. Mine ran in less than a second and his took a few minutes. To me awk is awkful with a silent K. It is a very useful tool, but it is just so easy to create your own specific, and much better tool quicker than using awkful.
Writing a dedicated program would have taken about 7 lines of code and 5 min, not forever. But, I digress. Us Nix folks live by grep, sed, awk, cut, xargs, etc.
0:30 dedicated Python code for that task is 6 lines. ``python with open('temps.txt', 'r') as input_file: with open('temps_all_c.txt', 'w') as output_file: temp, scale = line.strip().split() if scale == 'F': temp = (float(temp) - 32) * 5/9 out_file.write(f'{temp} C ') ```
I piped awk into sed once so I could graph certain lines of a csv dataset in Excel because Excel couldn't open the full file for me to filter it. Awk hey that worked, yeah that's what I sed.
If you're working in a Windows environment, you can use PowerShell in much the same way for data processing tasks like this. That said, I really need to start using awk in Linux instead of just contorting sed into doing things it wasn't made to do.
Very cool. I hope you don't mind, but I've just placed a link to this video in the awk chapter of the new shell scripting book that I'm currently writing. (It will be out sometime in the spring.) Many thanks!
in 2024 it's counter-productive to rely on awk as it's missing libraries to work with common formats: multiline rows, binary (like parquet), csv with quotes, headers, footers, excel i.e. pandas.read_csv in jupyter/ipython would do the same (could have a one-liner if you wish) but it's compatible with binary formats and anything you could possibly think of. another option is excel if there is no need to repeat the transformation.
Hmm, first time I watched this video, sound was behind the image, so you looked like Bruce Lee speaking in movies 😁 I thought it was intentional at first, then realized it was a problem in my browser. AWKward.
Awk was pretty much the first scripting language; it was the ancestor of Perl. It's largely taken over shell scripting on Stack Overflow these days; the kids there love it because they're so paranoid about only calling a single process, and doing everything in a single line. I think its' syntax is a bit tricky, personally; but it does make it extremely easy to create mathematical filters that are hard to do with virtually anything else.
Today AWK isn't only a text-processing utility, it is a complete general purpose programming language! You can use loops, variables, regex, conditionals, functions, array and string manipulation, math lib, include files...
For reference, here is a PowerShell solution for that problem: Import-Csv .\temps.csv -Delimiter "`t" | Select-Object @{Name="temp";Expression={if($_.unit -eq "F") {"$([math]::Round(($_.temp-32)/1.8,1))"} else {$_.temp}}}, @{Name="unit";Expression={"C"}} | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | %{$_.toString().Replace("`"", "")} However normally you wouldn't do that as one line in PowerShell. You can just save the whole table as a variable and work on that instead: $temps = Import-Csv .\temps.csv -Delimiter "`t" foreach($entry in $temps) {if($entry.unit -eq "F") { $entry.unit="C"; $entry.temp="$([math]::Round(($entry.temp-32)/1.8,1))" }} $temps | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | %{$_.toString().Replace("`"", "")} Assuming you need it in exactly the same format again. I'd argue that using commas in a csv file and quoting the values would be better though.
Writing a script in C#, F# or Python to do this takes 1 minute and the code will be more understandable and can be copied into your project. Awk is Awesome in niche situations, but I'd only use it when other languages are not available.
yeah given this is just a backstory case for a awk lesson. It still really depends the tech the dev is more used to. If the peson uses shell commands a lot everyday, it's easy to just awk, great to use what it offers. Otherwise is just no-brainer to write a small standalone app/script in your fav language (or any tech), hell it's most of the times just easier to google spreadseet or prompt into ai assistant (that appeared after this video being posted)
I started to use perl for most of those kinda things years ago. But indeed AWK is a usefull tool. Beside stuff like AWK, SED or PERL differs you from the dotNot kiddies.
awk is great! So many simple tools like grep, awk, and even more advanced-ish tools like jq are so often overlooked! Though for me, once I know that I'm going to need something like an average temperature, I'd probably end up just running the python command and using the python repl which is already quite fast to work with. Better yet, if they ever ask me to get the median or standard deviation, I can do so without the hassle of writing the equivalent awk script.
Writing a simple script to do that task wouldn't take forever. I'm pretty sure I could write one in Autohotkey (using the StrSplit() function) or in Python (using the split() method) in under 5 minutes, which is probably about the same time it took you to figure out that awk command.
I agree that awk is cool and powerful, especially if you are working exclusively in a shell script, but, as you mentioned, it's basically a language of it's own, so instead of learning a new one, I'd rather just make a quick Python ascript that does exactly what I want in a more legible way
Perl was written because Larry Wall was frustrated with the Awk+Sed combo. Awk+Sed didn't do what he wanted it to do. Perl is still much better then the Awk+Sed combo, however very few people learn Perl but have reverted to learning Awk+Sed instead for text processing. Its a real shame because Perl is still hands down the best for text processing, whether it be on the command line or processing text files.
Writing a dedicated program wasn't an option, because that would take forever 1) I could write a simple Python script to solve this in minutes, maybe even under a minute. 2) awk commands are essentially little programs, except you are probably not very proficient with it and crafting the command will take more than writing the code in any other language you are familiar and this task was never going to be repeated again 3) I cannot count the number of times I thought I will not repeat a task, didn't save the solution and then solved the thing all over again later.
0:10 before I read about the emacs command I was a very productive employee. My boss would assign me a task and I would just do it. Then I read about emacs and now my boss assigns me a task and I spent half a day configuring emacs and another half to write something in emacs that gets the job done
I wanted to try this in Python since that's my goto for this kind of thing. Here's a simple, clear solution: print(next(sys.stdin), end="") for line in sys.stdin: value, unit = line.split() value = float(value) if unit == 'F': value -= 32 value *= 5/9 unit = 'C' print(f"{value:>.1f}\t{unit}") And here's an attempt to single line it: print(next(sys.stdin) + " ".join(f"{5*(value - 32)/9 if unit == 'F' else value:>.1f}\tC" for value, unit in starmap(lambda v, u: (float(v), u), map(str.split, sys.stdin)))) Not pretty or practical, but fun!
Here's a shorter solution: temps = [] with open('temperatures.txt', 'r') as file: convert_temp = lambda line: int(line.strip(' F')) if 'F' in line else int(line.strip(' C')) temps = [convert_temp(line) if any(unit in line for unit in ['C', 'F']) else int(line) for line in file] print(temps)
In VSCode: Step1: Find and select all °C. Step 2: Copy them in a new tab Step 3: Modify the text to have the conversion formula. Step 4: Use Math Calculate extension to get the result Step 5: Select, copy, and paste back in original file
Awk is cool but it's just another scripting language. I don't really get the premise, I would have immediately just written a simple bash script to do it or used Python or lisp depending on what machine I was working on. Awk is useful for sure but it's literally the same thing.
I really need to practice with awk more. My problem is that I don't use it enough and then I forget everything. Plus at my age (40) it requires more effort to get into a new flow (possible though!!)
the narrative that lead to the tutorial is so cute as it never happened.
I sure hope so, or someone needs to learn about spreadsheets.
it never happened indeed and a maximum of 4 lines of python can do this, that's why I avoid learning these tools
spreadsheet? is that like lasagna or vegemite or nutella,@@Bjawu ?
@@jazzochannel There *are* some similarities with vegemite, now that I thing about it. You don't want to use too much.
I feel like people just don’t get this guy’s sense of humor
Utilities like awk are one of the reasons I love Unix. Everything is in a text file and the OS gives you the tools to parse and manipulate the contents.
Many moons ago I helped maintain what we thought might be the largest awk script ever created - it converted between the proprietary FORTRAN code for compilers by our competitors into our own proprietary FORTRAN code. Of course, monthly, if not weekly, we'd find something it did not convert properly, and it would have to be changed. Not only was it possibly the largest awk script, but it was the most modified awk script...
Fascinating! Do you remember how many lines (roughly) it was?
@@artemiasalina1860 It was well over a thousand lines.
Sounds awk ward to work with…. I’m sorry
@@bide7603It's not funny, it's a way of life...
I once took a little boy under my wing who thought that text manipulation equated to social skills. He was a little awk ward
I used to use awk every day about 30 years ago. A nice walk down the memory lane with this video.
memory lane? why did you stop?
@@jazzochannelmy work now doesn’t involve coding and data manipulation any more.
The awk command is now my favorite Linux command
I avoided Awk for 15 years, but then I saw the light.
@@catsupchutney I have a year left of avoiding it like the plague, maybe. Then I'll give it a shot.
The solution to the original problem was a bit of overkill, but this was a good example for some teaching of base awk functionality. This is a more direct and shorter solution, only changing the F lines and printing all lines:
$ awk '$2=="F" {$1=sprintf("%.1f",($1-32)/1.8); $2="C";} 1' temps.csv
(the single "1" at the end is the condition, which evaluates as "true", evoking the default action: print -- nice shortcut if you mean only to change some lines but print all)
Excellent. Glad to see someone else recognized and posted the solution I thought was more elegant
And taught me something new with that “1” trick!
I like how there was a narrative style to the video. That's really engaging. You've earned a sub!
Great pace too.
I enjoy using GREP and SED and I've been really wanting to learn how to use AWK. This video motivated me to do so, I'm gonna!! Also LOL, the end of the video made me laugh a lot. I love this!
Thank you for describing the pattern-matching and default assumptions so thoroughly and illustrating each point with example code. Until today have never seen any awk tutorial that could get me up to speed so quickly as your video. Kudos!
Showing how awk can replace both the grep and sed commands was icing on your cake.
And your last sentence was a big flaming torch sitting atop the cake!
Get real- once you learn regex, and in my own world, specifically Perl regex, that stuff becomes fairly trivial. "Analysing log files from umpty-hundred jobs" used to be a chore but these days I can script a working solution in an hour or so. In Perl. Because that's the tool I'm familiar with.
Wow.. I know awk well, but what a presentation. This is an underrated video. Keep it up dude.
Thanks, will do!
I get that awk is a useful tool overall, but with your given example, why exactly would writing a program be time consuming?
i think it was just a poor attempt at a motivating example and a joke at the same time
the whole premise was quite funny
It would take as long to write in python as it would take learning awk. At least pre AI, now AI could generate the command in moments.
@@jeanhaley3051 i just tried writing that python script and measured how much time it took me (without trying to rush it), it took 1 minute. it might be faster with awk in the long run? but 1 minute is good enough for me
Even less code with Perl. And this example is not recommended at all. What happens when the data is changed. It all fails.
@4:10 "As you see, awk is not quite this smart. And here you can see how awk isn't quite as good as a CSV parser. "
In GNU awk set the built-in FPAT variable. In your example: FPAT = "([^,]+)|(\"[^\"]+\")"
This is video is great and demystified awk in a very easy to understand way
'awk', 'sed', and 'Perl'. All I ever need for scraping from log-files!
head and tail is also useful for a redneck like me :D
If you know Perl, do you still have a use for Awk? (I'm wondering if I should learn it, I already know Perl. From the looks of this video, I can do the same with Perl in maybe 20-30% more characters - not enough more to trigger learning it)
@@obiwankenobe3962 Sure. Back in the old Unix days you needed sed and awk to to text munging. Now I just use Perl for all of it. Perhaps more characters to type in, but much simpler syntax and easier to learn, IMO.
@@BytebroUK thx for confirming my suspicion
@@obiwankenobe3962 old comment but still, Perl was devised in part to be awk/sed killer, it has special flags for "read a line at a time and do stuff" and syntax for "write-only" one-liners.
Perl is a Swiss Army Chainsaw, and that includes awk and sed functionality. It borrows quite a bit from sed and awk syntax.
At my past job where we used Linux exclusively, I used awk extensively along with bash scripts, sed, tr, od, etc. There are so many handy Linux commands.
Amazing presentation! I've been avoiding Awk for close to a decade, so much so I default to Sed for this kind of tasks! After watching your video, I feel more confident to try Awk next time :) Thank you!
I've never done more with awk than printing columns from a text file, like the 'cut' command. And for other things, use a perl one-liner. Loved the average calculation example in awk. Good to have a new tool in the belt!
This is such a great explainer of a tool i rarely used, but saw the potential, although i never dove in deep enough to learn it. Thank you!
Finally and after so many years, I understand how 'awk' works. Thank you, Legend
Great video! So, it's 3 years later, I hope you got your juicy 1% raise!
The temperature data format is a good one for exercising awk. This is a data driven world.
I always thought awk was super hard to learn. 😭 Thanks for explaining it so well! 🙏
I used awk and sed to convert entire flight manuals into our dataformat that won us the deal with pilatus as we could convert it to our software suit.
Excellent presentation of how AWK works and how it could benefit you.
I haven't used it personally (I would use Python: slightly more complex to use, but MUCH more powerful), but I know people who use it often, e.g. to scan log files for certain situations. It is great for iteratively improving the filters, browsing through the data.
so much content in just 10mins. amazing!
I love how simple programs developed 50 years ago are still extremely useful.
That is because the core of Computing hasn't changed much at all over that time, and UNIX did an excellent job of capturing that core.
I pity the fools who know nothing outside of a GUI. Because with a GUI, you get a single tool that can do only what its designers allow you to do. But the UNIX concept is to have a load of little tools that one can combine to do exactly that what you want it to do.
Perhaps I should write a GUI overlay for the shell, allowing the old UNIX tools to be configured and linked together graphically. That would be fun, and might help people discover the tools that are available at the CLI.
Dude........ Your content is brilliant. You must become more known! We desperately need more Unix content on youtube.
agreed - awk is awesome and i use it all the time - the most extreme example was a ~400 line script to analyze network files extracted from big integrated circuits but it’s great (and fast) at processing big text files - and it’s highly portable too since it’s installed on pretty much every unix/linux system dating back to the 1960s
Real world examples IME look something like this:
Some poorly documented program just spat out a noisy status message and produced a bunch of poorly named files. You want to parse the message to associate metadata with the output files, then rename the files with metadata as part of the name.
First pipe message into awk to separate into fields, then do simple transform in awk or pipe into sed or grep, then sort file names for association with the metadata output. Then pipe both into awk again to generate a shell script to rename the files.
At a time when Excel had only 65000 lines, I used awk to condense big text data files to a manageable size used the associative array feature of awk.
If you’re working with same files over again, I suggest to include one sample line for each pattern because you may not recognise the line from the pattern after 2 months.
The Fahrenheit example I’d done in libreOffice. Import and so on.
There was a time where you had difficulties getting an awk that works on windows.
Bro went from a 0x developer to a 0.5x dev
10x0 is still 0
what a hater, the video was great
It is but the title is wayy over the top
Awk is a very powerful program indeed. The only thing that keeps it from completely replacing grep for me is the lack of a Perl mode for regex.
I really think we need better parsing libraries and at some point the most effective way to dev will be an editor that is easy to add features to which essentially will be coding from the command line. Even between awk and sed there are editing cases where I cannot cleanly implement my operation/macro.
Neovim?
This was an exceptionally good tutorial. Thank you.
This is an amazingly instructive video. Thank you.
I like your shorts "this is my favorite command", but the longer are more explanatory and better pauseable aswell. Thanx
I remember when I used to torture myself with arcane bash commands and construct a towering awk command for complex file parsing. The cryptic awk errors were chef's kiss.
Honestly I just prefer any other language at that point.
Awk is great, but not the only hammer for every nail. Sed, cut, bc, (node maybe), vim macro, bash, lots of ways to solve this with one line or so
These elegant tools focused on practicality and performance are what made me fall in love with Unix (and then OSX).
Windows feels like a cludgy toy OS in comparison.
"I was so unproductive!" → Proceeds to open the Gnome terminal
Alacritty FTW 🙂
This is why people hate the Linux community.
using a better terminal does t make you more productive, just pipe the massive prints so you dont have to worry about print speed on compilation or something. (i say this as someone who uses a fast terminal (foot))
Humorous but informative, I'm no longer afraid of awk, nice video
Don't even ask how many python scripts i wrote to do this kind of stuff, grappling with pandas csv reader and such
AWK is very cool. I wore a small program using it and Bourne Shell that was about 300 lines long, but it would have been a lot longer if I was using something else. I think people probably overlook it nowadays because it's old and it's syntax is a bit archaic, but it's very powerful!
BEST INTRO TO AWK. PERIOD!
I use awk over grep daily because of it's usefulness in creating conversions or finding ways of pattern matching smartly. It's grep and cut in one. The only time i use cut is for specific columns due field separators being a pain sometimes.
It's been 3 years since you posted the video. I hope you got that raise. Gotta offset inflation somehow.
Great presentation style. I just subscribed, hoping to hear if you got that 1% raise.
I didn't get any raise this year, but I've got a good feeling that if I work overtime for just a couple more years, someone will finally notice my hard work :)
As a data engineer I did a lot in VIM. But mainly just for small stuff like formatting sample files that I would test transform. And finding anomalies in huge files. Not as powerful as awk, but also a different interface.
Thanks for sharing.
Good luck with your 1% raise!
Thank you!
@@RobertElderSoftware so... did you get the raise?
When I look at the command you ended up with I think if it was me it would have taken me less time to write it in python that I already know than to learn all that in awk.
I'll never use IDEs and Vscode anymore to process texts!
Thanks ❤
I challenged a guy who believes in awk, I let him define the problem. He could use awk, sed, … and I would use cpp, yacc, and lex. The challenge was on two fronts, dev time - first to start, and run time. He insisted that he may run slower than me, but he would clearly win dev time.
Well, I ended up not using yacc, just cpp and lex. Beat him in dev time by nearly an hour. Mine ran in less than a second and his took a few minutes.
To me awk is awkful with a silent K. It is a very useful tool, but it is just so easy to create your own specific, and much better tool quicker than using awkful.
I love this video. Awk is one of the basic UNIX tools I always skipped over.
Bad call, lol.
Writing a dedicated program would have taken about 7 lines of code and 5 min, not forever. But, I digress. Us Nix folks live by grep, sed, awk, cut, xargs, etc.
0:30 dedicated Python code for that task is 6 lines.
``python
with open('temps.txt', 'r') as input_file:
with open('temps_all_c.txt', 'w') as output_file:
temp, scale = line.strip().split()
if scale == 'F':
temp = (float(temp) - 32) * 5/9
out_file.write(f'{temp} C
')
```
I piped awk into sed once so I could graph certain lines of a csv dataset in Excel because Excel couldn't open the full file for me to filter it. Awk hey that worked, yeah that's what I sed.
Nice Video, this is a great explanation of awk, thank you for sharing.
The awk command is my favorite command.
If you're working in a Windows environment, you can use PowerShell in much the same way for data processing tasks like this.
That said, I really need to start using awk in Linux instead of just contorting sed into doing things it wasn't made to do.
Ah the classic 10x engineer,
Making 10x the work for everyone else. I too have earned this little.
You are the great teacher
The Awk command is my favorite Linux command
This is my favorite comment.
Very cool. I hope you don't mind, but I've just placed a link to this video in the awk chapter of the new shell scripting book that I'm currently writing. (It will be out sometime in the spring.)
Many thanks!
Thanks for the link, I feel honored :)
Nice tips and examples, man.
Greetings from Brazil.
in 2024 it's counter-productive to rely on awk as it's missing libraries to work with common formats: multiline rows, binary (like parquet), csv with quotes, headers, footers, excel
i.e. pandas.read_csv in jupyter/ipython would do the same (could have a one-liner if you wish) but it's compatible with binary formats and anything you could possibly think of.
another option is excel if there is no need to repeat the transformation.
Hmm, first time I watched this video, sound was behind the image, so you looked like Bruce Lee speaking in movies 😁 I thought it was intentional at first, then realized it was a problem in my browser. AWKward.
Awk was pretty much the first scripting language; it was the ancestor of Perl. It's largely taken over shell scripting on Stack Overflow these days; the kids there love it because they're so paranoid about only calling a single process, and doing everything in a single line. I think its' syntax is a bit tricky, personally; but it does make it extremely easy to create mathematical filters that are hard to do with virtually anything else.
Today AWK isn't only a text-processing utility, it is a complete general purpose programming language! You can use loops, variables, regex, conditionals, functions, array and string manipulation, math lib, include files...
You can use libraries to extend AWK.
yep, awk changed my habits either, thanks for sharing!
There is also jq for json manipulation, but I've never used it in practice.
Awesome explanation. Thank you
For reference, here is a PowerShell solution for that problem:
Import-Csv .\temps.csv -Delimiter "`t" | Select-Object @{Name="temp";Expression={if($_.unit -eq "F") {"$([math]::Round(($_.temp-32)/1.8,1))"} else {$_.temp}}}, @{Name="unit";Expression={"C"}} | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | %{$_.toString().Replace("`"", "")}
However normally you wouldn't do that as one line in PowerShell. You can just save the whole table as a variable and work on that instead:
$temps = Import-Csv .\temps.csv -Delimiter "`t"
foreach($entry in $temps) {if($entry.unit -eq "F") { $entry.unit="C"; $entry.temp="$([math]::Round(($entry.temp-32)/1.8,1))" }}
$temps | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | %{$_.toString().Replace("`"", "")}
Assuming you need it in exactly the same format again. I'd argue that using commas in a csv file and quoting the values would be better though.
Its now 3 years later, he better have gotten that 1% raise
Writing a script in C#, F# or Python to do this takes 1 minute and the code will be more understandable and can be copied into your project. Awk is Awesome in niche situations, but I'd only use it when other languages are not available.
yeah given this is just a backstory case for a awk lesson.
It still really depends the tech the dev is more used to. If the peson uses shell commands a lot everyday, it's easy to just awk, great to use what it offers. Otherwise is just no-brainer to write a small standalone app/script in your fav language (or any tech), hell it's most of the times just easier to google spreadseet or prompt into ai assistant (that appeared after this video being posted)
I started to use perl for most of those kinda things years ago.
But indeed AWK is a usefull tool.
Beside stuff like AWK, SED or PERL differs you from the dotNot kiddies.
Awk is great. Been using it for 20+ years
awk is amazing , I love it !
the awk command is my favorite linux command.
Here we work to live, we don't live to work. Social life is more important than work.
awk is great! So many simple tools like grep, awk, and even more advanced-ish tools like jq are so often overlooked!
Though for me, once I know that I'm going to need something like an average temperature, I'd probably end up just running the python command and using the python repl which is already quite fast to work with.
Better yet, if they ever ask me to get the median or standard deviation, I can do so without the hassle of writing the equivalent awk script.
jq is insanely powerful 🙂
@@GreyDeathVaccine for real
Writing a simple script to do that task wouldn't take forever. I'm pretty sure I could write one in Autohotkey (using the StrSplit() function) or in Python (using the split() method) in under 5 minutes, which is probably about the same time it took you to figure out that awk command.
I agree that awk is cool and powerful, especially if you are working exclusively in a shell script, but, as you mentioned, it's basically a language of it's own, so instead of learning a new one, I'd rather just make a quick Python ascript that does exactly what I want in a more legible way
Perl was written because Larry Wall was frustrated with the Awk+Sed combo. Awk+Sed didn't do what he wanted it to do. Perl is still much better then the Awk+Sed combo, however very few people learn Perl but have reverted to learning Awk+Sed instead for text processing. Its a real shame because Perl is still hands down the best for text processing, whether it be on the command line or processing text files.
1% raise 😂
Thanks for the tutorial!
1.5% raise for the gentleman please.
The awk command is my favorite linux command. 😂😂😂 Great content! Keep going!
Writing a dedicated program wasn't an option, because that would take forever
1) I could write a simple Python script to solve this in minutes, maybe even under a minute.
2) awk commands are essentially little programs, except you are probably not very proficient with it and crafting the command will take more than writing the code in any other language you are familiar
and this task was never going to be repeated again
3) I cannot count the number of times I thought I will not repeat a task, didn't save the solution and then solved the thing all over again later.
Hello, thanks for sharing this knowledge. Interesting way of presentation and very concise and informative.
0:10 before I read about the emacs command I was a very productive employee. My boss would assign me a task and I would just do it.
Then I read about emacs and now my boss assigns me a task and I spent half a day configuring emacs and another half to write something in emacs that gets the job done
Super Useful, and entertaining
Excellent explanation.
I wanted to try this in Python since that's my goto for this kind of thing. Here's a simple, clear solution:
print(next(sys.stdin), end="")
for line in sys.stdin:
value, unit = line.split()
value = float(value)
if unit == 'F':
value -= 32
value *= 5/9
unit = 'C'
print(f"{value:>.1f}\t{unit}")
And here's an attempt to single line it:
print(next(sys.stdin) + "
".join(f"{5*(value - 32)/9 if unit == 'F' else value:>.1f}\tC" for value, unit in starmap(lambda v, u: (float(v), u), map(str.split, sys.stdin))))
Not pretty or practical, but fun!
Here's a shorter solution:
temps = []
with open('temperatures.txt', 'r') as file:
convert_temp = lambda line: int(line.strip(' F')) if 'F' in line else int(line.strip(' C'))
temps = [convert_temp(line) if any(unit in line for unit in ['C', 'F']) else int(line) for line in file]
print(temps)
It's 3 years later, did you get that raise bro?
In VSCode:
Step1: Find and select all °C.
Step 2: Copy them in a new tab
Step 3: Modify the text to have the conversion formula.
Step 4: Use Math Calculate extension to get the result
Step 5: Select, copy, and paste back in original file
Mmm... Demasiados pasos y todavía tengo que instalar VSCode para poder hacer una tarea tan trivial.
Awk command is my favourite linux command.
writing a dedicated program takes forever? I agree awk does faster, but writing a python script will take at most half a minute...
Thanks -- You have a new Sub!! Take care
someone has earned themselves a pizza party
Awk is cool but it's just another scripting language. I don't really get the premise, I would have immediately just written a simple bash script to do it or used Python or lisp depending on what machine I was working on. Awk is useful for sure but it's literally the same thing.
I really need to practice with awk more. My problem is that I don't use it enough and then I forget everything. Plus at my age (40) it requires more effort to get into a new flow (possible though!!)