Write stdout to two different files (one overwrite, one append) in bash

Question

I have a bunch of experiments that are organized by group and by number. I have 3 different groups and for number, I wish to run 2 different experiments. In other words, I have the following experiments to run:

group A, 1
group A, 2
group B, 1
group B, 2
group C, 1
group C, 2

Every time I run a program that prints the results of the experiment to stdout, I want those results to be put into a text file. I want a separate text file for the results of each group and number combination, and also a separate text file for the results of each group containing all the number runs.

So, here is my bash script that runs all these experiments:

#!/bin/bash

groups="A B C"
numbers="1 2"

rm *.txt

for g in $groups; do

    # Set the group settings based on the value of $g

    for n in $numbers; do

        # Set the number settings based on the value of $n

        ./myprogram >> $g-results.txt

    done

done

Using the above code, I end up with these text files:

A-results.txt
B-results.txt
C-results.txt

But I also want to have the text files:

A-1-results.txt
A-2-results.txt
B-1-results.txt    
B-2-results.txt
C-1-results.txt
C-2-results.txt

How can I change my ./myprogram... command so that the output is concatenated (>>) to one text file (like I'm already doing) and overwritten (>) to another (like I want to be doing)?

score 3 · Accepted Answer · answered Jul 31 '14 at 19:42

Use the tee command to "split" standard output to multiple destinations.

./myprogram | tee "$g-$number-results.txt" >> $g-results.txt

tee writes its standard input to one (or more) named files as well as to standard output, so the above pipeline writes the output of each instance of myprogram to a unique per-run output file, as well as aggregating output for all $g runs to one file.

You can also aggregate the output of the inner for loop instead of appending to the file.

for g in $groups; do
    # Set the group settings based on the value of $g
    for n in $numbers; do
        # Set the number settings based on the value of $n
        ./myprogram | tee "$g-$number-results.txt"
    done > "$g-results.txt"
done

Great! Didn't know about `for` loop aggregation. Also wasn't aware that tee did multiple writes. — synaptik, Jul 31 '14 at 19:44

score 2 · Answer 2 · answered Jul 31 '14 at 19:42

2

Since you already list the tee command:

./myprogram | tee $g-$n-results.txt >> $g-results.txt

answered Jul 31 '14 at 19:42

chrisaycock

36,470
14
88
125

score 1 · Answer 3 · answered Jul 31 '14 at 19:42

1

As a simple way instead of:

./myprogram >> $g-results.txt

You can capture the output once and write it twice:

$out=$(./myprogram)
echo "$out" >> "$g-results.txt"
echo "$out" > "$g-$n-results.txt"

answered Jul 31 '14 at 19:42

anubhava

761,203
64
569
643

Thanks, but I need to scan and monitor the text file during runtime. (I didn't mention this, but it's a requirement for me.) – synaptik Jul 31 '14 at 19:43
oh ok in that case `tee` based answer may work better for you I think. – anubhava Jul 31 '14 at 19:44

score 1 · Answer 4 · answered Jul 31 '14 at 19:44

1

Use tee twice.

myprog | tee -a appendFile.txt | tee overwriteFile.txt

Just like this it will also print to stdout. You could add something on the end to pipe it to something else if you wanted.

IF you needed to any manipulation in between sed is your friend.

answered Jul 31 '14 at 19:44

Threadicide

126
7

True, but I believe you need to do it like this if you want one tee to append (-a) and the other to overwrite. – Threadicide Jul 31 '14 at 19:48

Write stdout to two different files (one overwrite, one append) in bash

4 Answers4

Linked