6

I have a simple bash script to run:

cat full_path.csv | parallel --progress -j +0 'echo -n {},; pdfgrep -c [^_] {};' > path_count.csv

Parallel's progress indicator "--progress", writes into the file path_count.csv. I only want echo {} and pdfgrep {} to write to the file, while showing --progress output to screen.

If I do : 

cat full_path.csv | parallel --progress -j +0  'echo -n {},>>path_count.csv; pdfgrep -c [^_] {}>>path_count.csv;'

the file path_count is still garbled with progress.

Any help is appreciated. Thanks Alvin

Ole Tange
  • 31,768
  • 5
  • 86
  • 104
Alvin Das
  • 95
  • 2
  • 2
  • 6

2 Answers2

6

The behaviour you see is not what GNU Parallel is designed to do: --progress is normally sent to STDERR and not to STDOUT for exactly that reason:

$ seq 3 | bin/parallel --progress echo {} >/tmp/out

Computers / CPU cores / Max jobs to run
1:local / 8 / 3

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
local:0/3/100%/0.0s
$ cat /tmp/out
1
2
3

Has there been local modifications of GNU Parallel? Can you reproduce the issue on other systems?

PS: instead of 'echo -n' why to try: --tag

Ole Tange
  • 31,768
  • 5
  • 86
  • 104
  • Thanks, I am really not sure how tag replaces echo -n {}. Maybe an example with my script? – Alvin Das Sep 17 '13 at 18:57
  • 1
    --tag will add the sequence number to the start of the output line delimited by a tab, so in your case, you might see in your target file (escaped characters because no newlines in comments): 1 \t output,goes,here \n 2 \t more,output,here – Andrej Aug 17 '16 at 13:59
0

Try to redirect it completely within the subshell using exec:

: > path_count.csv  ## truncate file
cat full_path.csv | parallel --progress -j +0 'exec >>path_count.csv; echo -n {},; pdfgrep -c [^_] {};'
konsolebox
  • 72,135
  • 12
  • 99
  • 105
  • 1
    Nice, although I have to deal with the formatting in the file "path_count". I have multiple echo output and then the pdfgrep output. parallel -k and parallel --group did not work, in various combination with 'echo -n' and/or just 'echo'. Thanks. – Alvin Das Sep 17 '13 at 18:56
  • @AlvinDas Do you want each instance to be redirected to another file? – konsolebox Sep 17 '13 at 19:52