19

I have a shell pipeline for generating 10 characters password at random:

cat /dev/urandom | base64 | head -c 10

My question is cat /dev/urandom | base64 is an infinite output stream which will not exit by itself. But why does appending head -c 10 make the whole pipeline exit? I assume cat, base64 and head are 3 separated processes, how can head make the cat process exit?

Dagang
  • 24,586
  • 26
  • 88
  • 133

3 Answers3

23

head closes the input file after reading the required amount. when a pipe is closed from one side, the other side gets write errors; this causes base64 to close, which in turn causes cat to close.

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820
Woodrow Douglass
  • 2,605
  • 3
  • 25
  • 41
  • 7
    It's probably worth mentioning that the only reason `head` gets any input at all is that `base64` writes output after it gets a certain amount of input, i.e. when its buffer is full. If it were to read until EOF, it would be reading forever, and `head` would never get a crack at any of it. So a similar pipeline, like `cat /dev/urandom | sum | head -c 10` would behave differently, since `sum` waits for EOF. – Rob Davis Apr 05 '12 at 16:23
  • 1
    Rob's comment is *extremely* relevant. If the process inherits a SIGPIPE handler or ignores SIGPIPE (eg if it is run under older python interpreters subprocess module) and does not check write errors, it will not terminate. There is a *huge* difference between a write error and receiving a SIGPIPE, and programs that ignore both issues are prone to running indefinitely. – William Pursell Apr 05 '12 at 19:55
  • @WilliamPursell: A process cannot inherit a SIGPIPE handler. It can inherit the SIG_IGN non-handler, but any signal handler is reset to the default when a process uses `exec()` to replace itself. It's not hard to see why: the pointer to a function in the old process is almost certainly not a good choice for the new process. But you're right that a process that ignores write errors and is also ignoring SIGPIPE will continue working far too long. – Jonathan Leffler Apr 06 '12 at 03:46
13

After base64 outputs 10 bytes, head gets enough inputs and exits. When the former attempts to output more bytes, it will receive SIGPIPE signal and hence exit too。For the same reason, cat will exit in turn.

Hui Zheng
  • 10,084
  • 2
  • 35
  • 40
4

Piping works by connection the output of one process A to the input of B. The connection can be broken, when

  • A closes its output. B will get EOF.
  • B closes its input. A will get an error that the output is no longer available when it tries to write the next byte.

Since these two cases are so common, the handling has been moved into the C standard lib.

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820
  • Thanks, but what does "the handling has been moved into the C standard lib" mean? is A and B terminated by shell instead of they detects the closing of input/output and stop themselves? – Dagang Apr 05 '12 at 15:31
  • When B closes its side of the pipe, A will receive a signal. The code in the I/O routines of the standard library c.lib (`fprintf()`, `open()`, `read()`, ...) handle the signal and the function call will return return the errno EPIPE = "Broken pipe". – Aaron Digulla Apr 05 '12 at 15:36
  • 1
    @AaronDigulla, I disagree. I am not aware of any C library that installs a SIGPIPE handler, nor do my experiments suggest it. In fact, SIGPIPE is delivered which causes any program that hasn't installed a handler (that is, 99% of all programs) to exit. – Robᵩ Apr 05 '12 at 18:57
  • correct, Rob, it's SIGPIPE. thanks! could you create an answer? i will accept it. currently, the 2 answers are not perfectly correct. – Dagang Apr 06 '12 at 04:36
  • @Robᵩ: What I mean is that you don't see "Broken Pipe" error in the console, so someone must handle this signal. – Aaron Digulla Apr 06 '12 at 11:12