Linux shell: Pipe stderr and stdout to one file AND to individual files at the same time (using tee?)

Question

I would like to form a pipe-stream that captures stdout to one file and stderr to another file AND both in a third file (so also the complete terminal output is in one file)

somehow like this:

process ----+-- <stdout> --- + --- stdlog.txt 
            |                |
            |                }---- <terminal> --- alllog.txt
            |                |
            \-- <stderr> --- +---- errlog.txt

I know I have to use tee but I can't figure it out. You people have a solution? Bash/sh preferred...

Thanx for any help, I really appreciate it !!!

cg_foreau · Answer 1 · 2021-01-05T16:25:52.467

First: It's not something I'd recommend you do, prepending logged messages with timestamps and labels such as [INFO] and [ERROR] in a single file or just timestamps in your two files would probably be a better and more reliable idea in the long run. while still allowing you to build the other file you want by using grep or sorting.

To your issue: I've never had to do it, but you probable need to use named pipes. Untested bash example:

# Log one output type
function logger() {
    unique_file="$1"
    common_file="$2"
    # Tee reads from the function's standard input, writes to unique_file, appends to common_file, then outputs what was written
    tee "$unique_file" | tee -a "$common_file"
}

# Call your code's main function to start processing
main > >(logger info_log_file common_log_file) 2> >(logger error_log_file common_log_file >&2)

You might be tempted to use exec >"$info_pipe" 2>"$error_pipe". Do not, or you will generate a loop that will make your log file fill all available disk space

Note that if your bash does not support implicit named pipes (it should, but I've seen environments that didn't), you might want to use them explicitly by calling mkfifo, however, this requires more plumbing :

# Log one output type
function logger() {
    unique_file="$1"
    common_file="$2"
    # Tee reads from the function's standard input, writes to unique_file, appends to common_file, then outputs what was written
    tee "$unique_file" | tee -a "$common_file"
}

# Setup logging
function prepare() {
    tmp_pipe_dir="$(mktemp -d)"
    error_pipe="$(mkfifo "$tmp_pipe_dir/error")"
    info_pipe="$(mkfifo "$tmp_pipe_dir/info")"
}

function cleanup() {
    rm -rf "$tmp_pipe_dir"
}

prepare
# Start logging, note that logger won't stop on its own. Removing the pipe files should do the trick
logger info_log_file common_log_file <"$info_pipe" &
logger error_log_file common_log_file <"$error_pipe" >&2 &
# Call your code's main function to start processing
main >"$info_pipe" 2>"$error_pipe"
cleanup

Linux shell: Pipe stderr and stdout to one file AND to individual files at the same time (using tee?)

1 Answers1