0

Using techniques mentioned here (Pipe output to two different commands) we can split a stdout into multiple processes.

expensive_command | tee >(proc_1) >(proc_2) | proc_3

my problem is this interlaces the output.

Is there a way to copy the stdout but force proc_2 to block until proc_1 finishes?

I'm thinking something like

expensive_command | tee >(proc_1) | wait for EOF | tee >(proc_2) ...
Jon
  • 1,785
  • 2
  • 19
  • 33

2 Answers2

2

You can create a buffer holder that would release the output once data from input reaches eof like

expensive_command | awk '{ a[i++] = $0 }END{for (i = 0; i in a; ++i) { print a[i] | "tee temp.txt" } }'

Only that awk does not support process substitution.

In bash you can do:

readarray -t lines <(expressive_command | tee >(proc_1))
printf '%s\n' "${lines[@]}" | tee >(proc_2)

Depending on the peak data size of your output from expressive_command or version of your Bash, the command may require adjustments. You can also consider using another language.

Add: You can also use stdbuf. It runs command with modified buffering operations for its standard streams.

konsolebox
  • 72,135
  • 12
  • 99
  • 105
2

You can use a fifo as a cheap lock. Have proc1 write to it after it completes, and wait until a read from the fifo succeeds before running proc2.

mkfifo cheap_lock
expensive_command | tee >(proc1; echo foo > cheap_lock) \
                        >(read < cheap_lock; proc2 ) | proc3

(Of course, it's your responsibility to ensure that no other processes try to read from or write to cheap_lock.)

chepner
  • 497,756
  • 71
  • 530
  • 681
  • hard to choose an answer between konsolebox and chepner since they both are well thought out but in the end I am using chepner's solution so he gets the vote. – Jon Jul 18 '14 at 13:17
  • While you can only select one answer, you can still upvote both. – chepner Jul 18 '14 at 13:39