Split up text and process in parallel

Question

I have a program that generates lots (terabytes) of output and sends it to stdout.

I want to split that output and process it in parallel with a bunch of instances of another program. It can be distributed in any way, as long as the lines are left intact.

Parallel can do this, but it takes a fixed number of lines and restartes the filter process after this:

./relgen | parallel -l 100000 -j 32 --spreadstdin ./filter

Is there a way to keep a constant number of processes running and distribute data among them?

Ole Tange · Accepted Answer · 2016-11-21T19:25:01.587

2

-l is no good for performance. Use --block instead if possible.

You can have the data distributed roundrobin with: --roundrobin.

./relgen | parallel --block 3M --round-robin -j 32 --pipe ./filter

edited Nov 21 '16 at 19:25

answered Nov 21 '16 at 19:11

Ole Tange

31,768
5
86
104

Using this command the filter-processes are spawned, but only one is running at a time. The others are waiting for input. – Craden Nov 22 '16 at 08:41
Then relgen generates too slowly: Lower 3M to 0.3M – Ole Tange Nov 22 '16 at 22:28
relgen is generating fast enough to keep all 32 filter instances running. Still the same behavior with 0.3M. – Craden Nov 23 '16 at 19:56
Does this run 32 active 'wc's? seq 1000000000| parallel -j32 --pipe --round-robin wc – Ole Tange Nov 23 '16 at 20:03
It spawns 32 wc's but only one of them is running at a time. – Craden Nov 25 '16 at 07:51
Upgrade it newest release. If you are running newest release: File a bug report as per: https://www.gnu.org/software/parallel/man.html#REPORTING-BUGS – Ole Tange Nov 25 '16 at 22:59
1

Thank you very much. Works like a charm now. – Craden Nov 25 '16 at 23:13

Split up text and process in parallel

1 Answers1