0

I've written a C program that works when I pipe data into my program using stdin like:

gunzip -c IN.gz|./a.out

If I want to run my program on a list of files I can do something like:

for i `cat list.txt`
do
  gunzip -c $i |./a.out
done

But this will start my program 'number of files' times. I'm interested in piping all the files into the same process run.

Like doing

for i `cat list.txt`
do
  gunzip -c $i >>tmp
done
cat tmp |./a.out

How can I do this?

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
monkeyking
  • 6,670
  • 24
  • 61
  • 81

5 Answers5

2

You should be able get one gunzip process unzip multiple files.

zcat $(cat list.txt) | ./a.out

(zcat is another way of calling gunzip -c on many systems and shows the parallel with cat; but check for gzcat if your system's zcat is actually uncompress.)

Alternatively you can use a sub shell.

(
  for i in $(cat list.txt)
  do
    gunzip -c "$i"
  done
) | ./a.out
CB Bailey
  • 755,051
  • 104
  • 632
  • 656
  • Beware: zcat was originally associated with 'pack' and 'unpack' which produced '.z' files. These days, they are, mercifully, relics from the past - they were not very efficient (even less so than 'compress' and 'uncompress', which generated '.Z' files. Nominally, the command you are looking for is 'gzcat'; however, it is not uncommon for 'zcat' to be equivalent to 'gzcat'. – Jonathan Leffler Mar 20 '10 at 22:57
  • @Jonathan Leffler: Good point, looks like I've been using modern linux a bit too much. – CB Bailey Mar 20 '10 at 23:06
  • Parentheses around the for cycle are optional, bash is smart enough. – Giuseppe Guerrini Mar 20 '10 at 23:10
  • @Giuseppe Guerrini: Yes, in this case they're not necessary, as the `for` loop is a single command. A sub-shell can be useful in more general cases. – CB Bailey Mar 20 '10 at 23:20
  • I mis-spake: it was pack, unpack, and pcat; it was compress, uncompress, and zcat. Gzip certainly handles the decompression of .Z files; these days, you'd be hard-pressed to find .z files generated by pack (though the very earliest versions of gzip used that extension, until the confusion was deemed unacceptable). (I just found some files on my machine, gzipped in 1990, with the '.z' suffix.) – Jonathan Leffler Mar 21 '10 at 01:49
2

There is no need for a shell loop:

gzip -cd $(<list.txt) | ./a.out

With the '-cd' option, gzip will uncompress a list of files to standard output (or you can use 'gunzip -c'). The $(<file) notation expands the contents of the named file as a list of arguments without launching a sub-process. It is equivalent to $(cat list.txt) otherwise.

However, if you feel you must use a loop, then simply pipe the output from the loop into a single instance of your program:

for i in `cat list.txt`
do
    gunzip -c $i
done |
./a.out

If the contents of the loop are more complex (than simply gunzipping a single file), this might be necessary. You can also use '{ ... }' I/O redirection:

{
cat /etc/passwd /etc/group
for i in `cat list.txt`
do
    gunzip -c $i
done
} |
./a.out

Or:

{
cat /etc/passwd /etc/group
for i in `cat list.txt`
do
    gunzip -c $i
done; } |
./a.out

Note the semi-colon; it is necessary with braces. In this example, it is essentially the same as using a formal sub-shell with parentheses:

(
cat /etc/passwd /etc/group
for i in `cat list.txt`
do
    gunzip -c $i
done
) |
./a.out

Or:

( cat /etc/passwd /etc/group
  for i in `cat list.txt`
  do
      gunzip -c $i
  done) |
./a.out

Note the absence of a semi-colon here; it is not needed. The shell is wonderfully devious on occasion. The braces I/O redirection can be useful when you need to group commands after the pipe symbol:

some_command arg1 arg2 |
{
first sub-command
second command
for i in $some_list
do
    ...something with $i...
done
} >$outfile 2>$errfile
Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
1

This is rather a shell question. But AFAIK you can do:

cat file* | your_program

or

for i in file*; do gunzip -c $i; done | your_program
jpalecek
  • 47,058
  • 7
  • 102
  • 144
  • This is what I was about to type, though would have added brackets for clarity: `( for i in file*; ...; done ) | ./a.out` – crazyscot Mar 20 '10 at 22:47
0

If your program doesn't need to know when a particular input ends and another one begins, you can do this:

for i `cat list.txt`
do
  gunzip -c $i
done |./a.out

I hope it will help you Regards

Dennis Williamson
  • 346,391
  • 90
  • 374
  • 439
Giuseppe Guerrini
  • 4,274
  • 17
  • 32
0

xargs is your friend

% cat list.txt | xargs gunzip -c | ./a.out

if the files in list.txt have spaces in them then you need to go through some extra hoops.

margaret
  • 1