Unix Pipeline (Brian Kernighan) - Computerphile
HTML-код
- Опубликовано: 12 ноя 2015
- Just what is a pipeline in the computer science sense? We asked Computer Science guru Professor Brian Kernighan
Why Asimov's Laws of Robotics Don't Work: • Why Asimov's Laws of R...
Brian Kerninghan on Bell Labs: • The Factory of Ideas: ...
Don Knuth on Email: • Why Don Knuth Doesn't ...
Computer That Changed Everything: • Computer That Changed ...
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com
Brian Kernighan should have his own RUclips channel, I'm sure he has many interesting UNIX and general computer history stories to share.
That is a really excellent idea!
Anyone that has used Unix before pipes existed deserved an award and I would love to see this
he does
Pipes are so elegant that they seem obvious and trivial. But that's often the way with great ideas!
They are neither obvious nor trivial, but they are elegant. If they seem obvious or trivial to you is because you have used them for a very long time. But most of humanity still likes something to click with mouse or finger and a progress bar. That is their expectation of computing.
Which is very different from shell and pipes.
@@axlslak Totally agree!
It is the way of Ken Thompson's programs
@@axlslak Maybe it wasn't obvious from the way I phrased it but that was the point I was making. The best ideas *seem* obvious in hindsight but they're really not, they actually took a lot of careful thought and genius to come up with.
I'm NOT actually saying they ARE obvious or ARE trivial.
I just love hearing Brian Kernighan talk about this stuff. There is an awesome video on youtube called "The UNIX Operating System" on AT&T Tech Channel from the 70's where a younger Brian Kernighan descripes the exact same principles of UNIX, back when they were still new and groundbreaking - I recommend that video to anyone.
+trylleklovn Great find and thanks for sharing!
thanks for the tip 👍
+trylleklovn Wow great find, that video was awesome!
+trylleklovn Nice. This looks cool. Syns navnet ditt er ustyrtelig morsomt av en eller annen grunn :D
+PropaneWP Haha :D
My favorite use of pipeline in my own experience was when I when I piped arecord (to create a pcm data stream from microphone input) into oggenc (to compress the pcm into a vorbis audio stream) into gpg (to encrypt the audio stream(probably should have used ssl but it worked albeit with a delay for entering the passphrase)) into netcat (to send the encrypted audio stream over a network) and on the other machine the reverse ( netcat | gpg | oggdec | aplay ) and effectively created a one-way encrypted voice-over-ip system with just a handful of popular commands.
+John Euell You just made a lower level Skype
That's honestly wild 😂
Beautiful....
Does anyone know how to achieve the same but with a video output from an integrated webcam on a computer?
@@MrJegerjeg I think you can replace arecord with ffmpeg and aplay with ffplay
When I saw pipes, I fully understood the magic of unix.
Its the genius of writing simple elegant programs that do one thing really well
As a system admin I use pipelines all the time. It's so useful to take existing programs and simply string them together. It also makes developing the command / program easier since you can test each step before tacking the next operation on.
It's the simple things like this in Unix that make the system great as a whole.
+gskema Seriously, the utility of the terminal is astounding!
a living legend
Pipes are amazing BTW. Most people who use them take them totally for granted, including me.
+Charles Miller
The problem is that they are text-based. So you either need to do a lot of regex and parse some arbitrary garbage the other program outputs or you agree on some format like XML to pass data (which rarely happens).
pipes work with all sorts of data. All the common tools assume text because most of the stuff is you want to use it with are text.
+Charles Miller Problems arise when they aren't available...
Pipes, like files, don't have the concept of "type of data."
It's just a stream of bytes. No more, no less.
+Charles Miller So would using "program.exe < input.in > output.out" be an extremely simple example of pipelining?
Great vid.
One fundamental piece of information missing in the video is that all those programs you glue together are in fact working at the same time in parallel. So if some of those are CPU hungry, it helps spread the load. And the time needed for the complete chain to complete is basically the time needed for the longest program to complete (except when you are using programs that need the whole data set to do their job, like "sort").
+Guillaume Laurent In theory true. In practice, though, your pipeline is normally limited by a single program. So if 'A | B | C' is slow due to B being slow, then you can often speed it up by 'A | parallel --pipe B | C'
I just realized why the | symbol is called the "pipe" o_0 Unix is awesome.
It's a Unix system, I know this!
It's Computer Science .at higher level
Brian -- Hello. It's Ruth Klein. I worked in Berk Tague's Dept. at Murray Hill. I may have some insight as to how Doug McIlroy's idea of streaming together programs actually got implemented. I believe it came about because of a unique characteristic of the PDP-11 machine language. Before the PDP-11, computer machine languages had specialized instructions to deal with each type of I/O device. For example, there were specialized registers and a specific machine language instruction to send output to a printer. Another set of registers and another machine language instruction would have been used if you wanted to send the same data to a disk file, etc. The PDP-11 got rid of these device dependent machine language instructions. Instead, the PDP-11 replaced these with a single READ and a single WRITE machine language instruction. The actual address of the device would have been put into a machine register used by the READ or WRITE instruction. Bottom line, this meant you could write a device independent program and, when you went to run the program, you could provide parameters as to which input and output devices to use. When DEC announced the PDP-11 computer, they hosted a presentation in NYC. I was the one sent to attend that presentation and found their new machine language method of dealing with I/O devices to be a simple and elegant design. Back at the Labs, I wrote a short memo describing DEC's new mini-computer and this simplified method of dealing with I/O devices. Two days after this memo was distributed, Doug McIlroy stopped by my office, waving the memo, and saying "Tell me more about this". I believe it was this feature of the PDP-11 design that provided the straightforward method of implementing standard input and standard output. And, linking the standard output of one program to the standard input of another program implemented the concept of the pipe.
Amazing story Ruth! Must have been exciting times! Did you have any idea at the time how durable the paradigm would be, and how we'd be taking it for granted many decades later?
You're redirecting output with a system call pipe(), which is used in the shell you're using. Your shell is the parent that instantiates the child, which is the program your executing (grep, etc). So for all you curious out there, you can do this from within your C programs, it's not just for fun on the shell.
By playing around with pids basically?
The first ever programming book that got into my hands was written by Kernighan and Rithcie. Its a pleasure to learn from the pros.
Always a pleasure to hear BK speak about the beauty of the Unix system.
Very subtly placed copy of "The C Programming Language", maybe a salute to its co-writer.
This is in my opinion the most compelling reason to script in bash or sh. I love the simplicity of which I can test each element individually.
I'd love to take a class from this guy. What a collection of history!
Wow, thank you so much! I've always wondered how these "pipelines" work but I couldn't find a consise explanation video.
Now I just need to find another video, which goes deep into the specifics of 3D pipelines, because I can't find such a thing either...
more of this delightful man, please.
Unix is just bloody genious.
Funny how they misinterpreted his mention of using grep, wc, and sort. He meant those are programs that you would frequently pipe data into, not that you would ever use them in a three-stage pipe as was shown in the animation, as that would make no sense at all, wc only outputs a count of lines, words or characters, none of which would make any sense to pipe into sort.
+John Euell
How big are the files that have 'fish' in:
grep fish **/* -l | xargs wc | sort -r
Near enough? :)
+goyabee I only came down to read the comments because I didn't understand why anyone would sort the output of wc. I thought I must be wrong because I couldn't believe I knew better than Brian Kernighan. Thanks for explaining it wasn't his error that I spotted (I should have listened more closely to what he said instead of watching the animation).
i put it in to terminal and it didnt work. not a great first experience.
Thanks a lot for the lesson! i'll love more stuff unix related.
Please more of these principles of how operating systems work today, had to learn this piping myself.
Fascinating. I am getting a picture of how the systems work, and relates to me. That is to say what runs the machines and systems I use in my day to day life.
Do one thing and do it well.
Write Programs that work together
Write programs to handle text streams, because that is a universal interface
One of the living deities of the field, describing one of the fundamental ideas. Anything Kernighan's said or written is worth studying. If he published "Proctology for Amateurs" I'd pick it up without bothering to read the blurb.
wow. a true legend. well done computerphile.
This video has the perfect timing. I've got an exam on linux commands on the near future, pipelines included. Understanding the basics and the origins makes it so much easier.
Thanks Computerphile!
Really? I mean, how shitty do your teachers have to be to not get the idea o pipelines across?
***** I know but pipelines? That's something so trivial that when you know it exist you pretty much know everything there is to know (unless you want to know how they're implemented in the OS of course). It's hard to imagine someone knowing pipelines exist but not knowing how to use them unless they received their explanation from a blabbering idiot who couldn't put a decent sentence together.
What I'm trying to say is that in this instance, the level of fail is so high that it's very hard to believe it could be the student's fault alone.
+hellterminator I agree. How much effort does it take to go to a command line and play around with the commands to solidify your understanding of these basics?
3:12 also it saves time especially with multithreaded systems as you do multiple processing at the same time while reading in the first input rather than doing everything sequential. Process1+2+3 run on the data at the same time, so ideally you can save 2/3 of the processing time with proper multithreading.
+Realraven2000 In most real-world scenarios where a pipe takes any significant amount of time, a single process is *the* bottleneck and takes orders of magnitude more time than the others, so you end up not making much use of the multiple cores or processors.
Try to find mutithreaded versions of the bottleneck program, like pigz instead of gzip, or use something like GNU parallel to run multiple iterations of the same job at a time.
+Major Gnuisance also piping to sort, the example in the video is a bit clunky since this requires buffering the entire input. But pipes are especially useful when one of the steps is slow, because the OS will handle scheduling and the speedy parts will block (i.e. pause) until the slow parts clear the buffer.
man! it must be a privilege to talk to this man.
Hey, I'm currently learning some UNIX basics for school, thanks for motivation (and procrastination).
howz it going
mf is watching vtubers now i see 😼😼😼
I'm only a few seconds in, but damn, he explains things so, so well!
thank you so much. you explained it way easier than the book im studying from.
Perhaps the most important thing he says here is:
"Having small programs is better than big programs."
I was using all the Unix utils when I was in college and when I graduated from college in my first job out of school 30 years ago and when I look at Linux today I think they forgot that "Small programs are better than big programs."
The version of Awk in Debian today is orders of magnitude more complex than the version I was using 20 years ago. And that starts to defeat the purpose.
I think this is perhaps one if the biggest flaws of "batteries include" programming languages of today. They try so hard to do everything, that they forgot to "just do one thing well."
I remember using this feature in DOS, but of course, UNIX did it before there was DOS.
therealEmpyre also the DOS version of the pipe worked in exactly the method he says the Unix pipe avoids: it wrote the output of the first program into a temporary file and did this until the first program finished running, then it connected this temporary file to stdin of the second program and if that piped its output into program 3 the output would be connected to another temporary file and program 2 runs to completion before deleting its temporary stdin file and connecting its temporary stdout file to stdin of program 3, etc.
Is there a correlation between pipe and unix domain socket?
"Why is it better to use numerous small programs rather than one massive program"?
"Because, for one thing, using small programs is better than using one big program..."
Ok, to be fair, he did answer the question later in the video.
This is my favourite part of Unix systems. Alongside > and >>.
So I was doing pipeline in bash without even knowing what I was doing, but now I grasp more easily compared to the first time when I watch the video.
Pipes are one of the most useful things in computer which I use on a day-to-day basis. Pretty much not a day goes by without me using them in some way or another.
I believe this was filmed at the same event as his previous video. The background is the same, literally
Does this assume each program has one and only one input and one and only one output?
If so how does this relate to nesting single parameter functions/methods inside each other?
e.g.
double square (double x) {
return x^2;
}
double cube (double x) {
return x^3;
}
And elsewhere you put
System.out.print(cube(square(2)));
and it outputs 64
+Primus Productions You should try your program before you say what it produces. It should output 64.
The lesser-known "pv" command deserved to be mentioned somewhere in this thread...
Pipe viewer is not a standard package, but it is awesome for sure!
Please someone make time travel possible so that I can witness these legends creating UNIX
I learnt C from the K&R book :) Also pipes are extremely useful.
is it possible to "pipe" to several instances of stdout at the same time and have different programs configured to receive those outputs?
+ytdlamPerhaps 'tee' is what you are looking for.
echo 123 | tee >(tr 1 a) | tr 1 b
works (from Stackoverflow)
can't live without it (queue scorpions music) ... the concept is just awesome ... (well, ran into some buffer issues, but anyway)
"wc | sort"
wat
Oh, I remember pipelines from my Amiga days...
But there are programs like sorting (of any kind) that in themselves can still be small, but that need the whole output of the previous program before they can really sort it, so there you really need a lot of memory if you have a large catalog of things that need sorting. Then the only advantage of the pipeline mechanism is the simple syntax, not any resource saving.
My teacher actually recommended this video. Well done.
Same here.
***** Do you happen to live in Slovenia?
Yes...
Well, we better get working on the task then, it's till tomorrow evening ;)
pipelines are so nice. Life is much easier with them than without them! :D
I wonder. If the output of 1 command were too big to store in a file, where is it stored temporarily, until it is passed to another command as input?
+Shirsh Zibbu The pipe is not stored to a temporary file. What is normally happening in 'prg_a | prg_b' is that prg_a generates some output. When prg_a has generated a full block (typically 4 KB) this block is passed to prg_b, which reads the block. If prg_b is not ready to read the block yet, prg_a will be blocked and not run until prg_b has read the block. Likewise prg_b can be waiting for a new block from prg_a and in that situation prg_b will be blocked and not use any CPU-time.
HE IS LEGEND!!!
I remember being taught this subject at school, not the easiest thing to understand .... this was back in the early 90s.
What happens when program 1 can output data faster than program 2 can process it? Does the data get buffered in RAM (which is what happens in another OS's pipeline that I know of), or does program 1 halt at the output command until program 2 asks for more data? If the data gets buffered, I don't really see the benefit in terms of required storage space mentioned at 2:22. RAM has always been more expensive than storage, so if the cost/availability of storage space is a limiting factor, the cost/availability of RAM is even worse.
Program 1 will be blocked until program 2 used up the data, but in some case there's also a small buffer.
you also have to remember the TIME COST, and when this was first created, you wanted to use as little time on the system as possible, as there was a line waiting. You can also consider the fact that RAM is accessed much more quickly than a drive, unless you happen to be working with something like Intel's OPTANE products.
I tried explaining the importance of the shell to someone and even broke it down to a sandwich assembly line or what have you. I just had these confused eyes looking at me.
Why can't I share important scientific (though computerized) breakthroughs with anyone!? It's not fair!
Signed -- a GNU user.
A better animation would have been to have water flowing through the pipes and change color each time it went through a new little program. Ideally enough water for some of it to still be going into program 1 while some of it is already coming out of program 2.
+joelproko Colored dots would be more appropriate, I think. But you're right, the animation fails to convey the important point that, conceptually, all programs in the pipeline will be running concurrently.
Otherwise, the system would necessarily have to hold the full output of each program in the pipeline before starting the next one, which would return us to the problem of having to store those intermediate results somewhere.
+Major Gnuisance I think MS-DOS used to run the programs one at a time and store the intermediate results in temporary files.
+TaiFerret MSDOS 1.0 was based on CP/M. MSDOS 2.0 copied many features from Unix (path names, redirection, pipes, asciiz strings) but as there was no multitasking the pipes are really pseudo pipes that use temporary files. In practice they can do most of the stuff that Unix command pipes. There are exceptions, for example the first program cannot be interactive as prompts would be written just to the file and never seen. Also in MS DOS programs of the bypassed the OS in IO so they would not be affected by redirection or pipes.
In Unix the pipe is an integral part of the actual OS. In MSDOS it is just a command line element. Windos 8.1 works like MSDOS. One can test this with dir /s /b | find "\" at the root. With real pipes the output would start immediately. Windows has real pipes also but they are separate from the standard IO-system.
TaiFerret As ***** explained, MS-DOS pipes are naught but a bastardization of the pipes of Unixlike OSes.
Let's not bring the dabblings of such a lowly OS family into the conversation.
This reminds me of Douglas Crockford who says that JavaScript is the single computer language with most installations on earth because virtually every web browser on any OS on any computer, mobile device etc implements it. There are other instances where ubiquity doesn't correlate with usability, e.g. machine code or non-coding DNA.
Also, I would dial your »million instances« to maybe a dozen, because Unix is a tad older than DOS, and every larger network with some servers it has components running shells with redirection capabilities in some form or other. Think of all those routers, switches, firewalls and other appliances with built-in busybox or proper Unix/Linux installation.
Additionally, for the recent ten years, iOS and Android mobile devices have become so numerous, each single one of them with FreeBSD or Linux on board, each of them (including watchOS) with a fully functioning bash. In my mind, the Unix way easily is the predominant way of accessing digital technology. It has even been standardized (POSIX), which cannot be said about GUIs (tiles or ribbons, anybody?).
Conversely, Microsoft was forced by the market to put some semblence of POSIX conformity into WinNT and make the DOS shell more capable. But it's still a nightmare of incompatibilities to automate something with it. My latest example with the DOS idiocy: try to put an ISO-8601-style date+time stamp into the name of a file to be written into. Yeah.
I just use pipes every day :) wonderful concept
So grep is monad?
Nice.
This is interesting but I was very distracted by the weirdly completely empty book shelves.
Professor Kernighan *almost* touches on PowerShell in the video
Wait, who's piping the output of wc in to sort? It's output (when it's input is from a pipe and not command line arguments) is just three numbers on one line :-/
Who sorts the output of wc?
Indeed, that flashed in my mind also, if I remember correctly wc is word count, so it just outputs a number.
MS-DOS has had Unix-style input/output redirection in its shell since version 1.0 (1981) - that's four years before the first version of "GUI" Windows (Windows 1.0, 1985), and 12 years before the first version of Windows as an OS (Windows NT 3.1, 1993). Thus, redirection wasn't "retrofitted" into Windows - it was built-in from the start. MS-DOS and Windows is more Unix-like than most people realize (which is reasonable, since Microsoft was a vendor of Unix back in the 80s - they made and sold Xenix for the IBM PC).
That's input/output redirection to/from files you are referring to. MS-DOS did not support anonymous pipes which Brian Kerninghan explained in the video. That's, you could not make a one-liner to use output of one program as input for another (|). There was a workaround to reroute the output (>) into a tmp file and then use this file for input in a separate line (
Pipeline is one of the most powerful uses of the properties of operating systems and it is just one tiny thing
This video really could do with some examples. Unix pipes are very easy to demonstrate and you can appreciate results almost instantly.
I think the people who don't already understand what pipes are, will not understand from this video the importance of the subject. Not only pipes are absolutely necessary to efficiently use shell, but they also embody one of the core Unix principles: building complex functional systems from simple and easy to use blocks. I could go on for quite some time about pipes, so I'll stop here.
Just to put things into perspective, I checked my shell history on several Unix machines, and on average about 15 to 20% of my commands contain pipes.
Isn't this guy the "K" in awk?
Yep! It sure is!
And the K in K&R
@@FandCCD not in this one, the k stands for Ken thompson!
@@charlesarmstrelli1688 No, it's short for Kernighan & Ritchie, the authors of the book The C Programming Language.
The greatest idea in language design. Too bad so few modern programming languages has it.
If only MVS/CMS Pipelines was part of Unix, what a wonderful world it would be.
Nice explanation. However, piping the output of "wc" into sort .... well ... one line. The other way around is more common. Still nicely done.
I used this today to pipe the output of neofetch through to lolcat to make the text a rainbow of colours! ">_ neofetch | lolcat" 😁
"big word for store it" hahahahahahaha
but ? users should be able to tune the 'wait time' and/or 'chunk bytes count' aspects of this ? no ?
And when you combine pipes with xargs, amazing things happen
Sorting the output of wc lol! Cause he gave the examples in that order.
It's a minor pity that you took his mention (at 3:46 and 4:35) of "grep, wc, and sort" as the basis for the animation... piping the output of wc into sort is... a bit pointless. Now if wc was replaced by awk (3:51), that'd make a lot more sense. :) Ah well! Still, a cool video on a very very very useful paradigm.
Do one thing , do it well . I love Unix
I don't use pipes often- I mostly use my computer like a typical consumer, and the aren't many use cases for them in this context. When I do use them , though, every alternative would be a giant pain in the ass. Whether it's a more complicated swiss army knife tool I'd need to learn, messing with files, or coding up a custom solution, it would just be a giant pain.
grep | wc | sort?
Sounds like one awk.
so this is kinda like APIs yes?
ok, I guess I don't get the complexity in this. An output of one program is being fed as input to another program. I mean what's the big deal about coming up with the command-line "|" (pipe) symbol that would be interpreted to do that? Why would it even require discussion among multiple people and be such an historic thing to be implemented? Isn't it just a trivial and obvious thing to do when writing a parser? What the non-obvious / non-trivial part about it?
+subh1 The non-obvious at the time idea of pipes had to be thought of by someone comfortable with poking around in the Unix kernel , then the code to make pipes work had to be invented and integrated into the operating system. I guess inventing code that hadn't previously existed just isn't that impressive to you.
"cat random_binary_file.dat | hd" I love this to death. So usefull!
+John Kapri It looks like "hd < random_binary_file.dat" will work too.
+John Kapri I just realised I made an alias in my .bashrc for that to work. Full command is "cat random_binary_file.dat | hexdump -C". Shows you a nice hexdump including an ASCII readout of whatever file you pipe in. "hexdump -C file.dat" would work as well, but you know... PIPES!!!
Suppose you downloaded all computerphile videos in a single folder, and you want to list all files containing unix:
ls -l | grep "unix"
ls -l lists all files in the folder, with each filename per line.
grep "unix" retains all lines containing "unix"
And voila: you have all filenames which contain "unix" :)
Also, if you just want to know how many files there are containing "unix" in the filename, just throw in "wc -l" to count lines:
ls -l | grep "unix" | wc -l
+Daan Wilmer
And then there were a bunch of files with newlines in their names and your count went over. Tsk, tsk.
find -iname '*unix*' -type f -printf 1 | wc -c
The 1 is arbitrary, any single-byte character would do.
+Daan Wilmer Or just
ls -l | grep -c "unix"
And to make sure it doesn't care about capitalization:
ls -l | grep -c -i "unix"
+Daan Wilmer But this you can achieve without pipes: ls -ld '*unix*'.
+Bjorn P. Munch you missed the -i for ignore case in the grep command. I don't think that ls -ld ignores the case. It certainly doesn't on the linux system I use every day.
My .bashrc depends heavily on pipes.
This made me think of lazy evaluation. Please cover lazy evaluation!
I wish that Brian Kernighan was my grandpa.
how to make a linux/unix pro mad in one command:
cat something | grep something
he needs more books.... for credibility
andrew campbell He's the one that writes the books
I got a laugh out of seeing wc in the middle of a pipeline there. grep, wc, and sort are all things you'd use in a pipeline, but you would never arrange them like that.
IT makes no sense to pipe wc to sort. wc outputs 1 line, and there is no sense to sort that.
it only outputs a single line if running against a single file...
reminds me of the laughing pain meme
why would you pipe the output of wc into sort? XD
+1classified1 Oh. I was not the only one being puzzled by that.
Maybe you want to get a word count from a number of documents, then sort them by the word count.
Pipes are incredibly useful but a lot of people also don't use here strings which bothers me. Here strings push a string into the STDIN of another command. A lot of people instead will just echo the string and pipe the result of the echo command into the command to process the string, but to me I've always found that to be ugly because you can just pass the string directly into it with here strings.
agree, +1 for here strings. but echo is more efficient, as here strings usually create a file in tmp, in order to get an fd...
+Amelia Hartman I don't think it really matters, and the here string may actually be slower (if that even matters which it probably doesn't). Personally I prefer to avoid writing to the filesystem, but just do it whichever way you prefer IMO.
Is the pipeline the same as or similar to object orientated programming ?
+Andrewsarcus Not really. It does tackle some of the concerns behind the idea of OOP, but in a completely different way.
+Andrewsarcus Not really, it's more like functional programming.
ls -l | grep -i "unix"
would be something like
grep(ls(LSOPT_LIST),"unix",GREPOPT_IGNORECASE)
if grep and ls were PHP functions.
+Andrewsarcus It's not the same, but pipelining doesn't have to be stringbased like it's the case with the UNIX shell. The windows powershell for example is object oriented and pipes .NET objects from program to program.
"a mechanism for screwing..." ;-)
Piping stuff like this is functional programmen.