bash: How to prepend a string to stderr lines and combine both stdout and stderr in exact order and store in one variable in bash?

Question

What I have done so far is :

#!/bin/bash

exec 2> >(sed 's/^/ERROR= /')

var=$(
        sleep 1 ; 
        hostname ; 
        ifconfig | wc -l ; 
        ls /sfsd; 
        ls hasdh;
        mkdir /tmp/asdasasd/asdasd/asdasd;
        ls /tmp ;
) 

echo "$var"

This does prepend ERROR= at the start of each error lines, but displays all errors first and then stdout, (not in order in which it was executed).

If we skip storing the output in variable and execute the commands directly, the output comes in desired order.

Any expert opinion would be appreciated.

Answer 1

The primary problem with your script is that the command substitution $(...) only captures the subshell's standard output; the subshell's standard error still just flows through to the parent shell's standard error. As it happens, you've redirected the parent shell's standard error in a way that ends up populating the parent shell's standard output; but that completely circumvents the $(...), which is only capturing the subshell's standard output.

Do you see what I mean?

So, you can fix that by redirecting the subshell's standard error in a way that ends up populating its standard output, which is what gets captured:

var=$(
    exec 2> >(sed 's/^/ERROR= /')
    sleep 1
    hostname
    ifconfig | wc -l
    ls /sfsd
    ls hasdh
    mkdir /tmp/asdasasd/asdasd/asdasd
    ls /tmp
)

echo "$var"

Even so, this does not guarantee proper ordering of lines. The problem is that sed is running in parallel with everything else in the subshell, so while it's just received an error-line and is busy planning to write to standard output, one of the later commands in the subshell can be plowing ahead and already writing more things to standard output!

You can improve that by launching sed separately for each command, so that the shell will wait for sed to complete before proceeding to the next command:

var=$(
    sleep 1 2> >(sed 's/^/ERROR= /')
    hostname 2> >(sed 's/^/ERROR= /')
    { ifconfig | wc -l ; } 2> >(sed 's/^/ERROR= /')
    ls /sfsd 2> >(sed 's/^/ERROR= /')
    ls hasdh 2> >(sed 's/^/ERROR= /')
    mkdir /tmp/asdasasd/asdasd/asdasd 2> >(sed 's/^/ERROR= /')
    ls /tmp 2> >(sed 's/^/ERROR= /')
)

echo "$var"

Even so, sed will be running concurrently with each command, so if any of those commands is a complicated command that writes both to standard output and to standard error, then the order that that command's output is captured in may not match the order in which the command actually wrote it. But this should probably be good enough for your purposes.

You can improve the readability a bit by creating a wrapper function for the simple-command (non-pipeline) case:

var=$(
    function fix-stderr () {
       "$@" 2> >(sed 's/^/ERROR= /')
    }

    fix-stderr sleep 1
    fix-stderr hostname
    fix-stderr eval 'ifconfig | wc -l'   # using eval to get a simple command
    fix-stderr ls /sfsd
    fix-stderr ls hasdh
    fix-stderr mkdir /tmp/asdasasd/asdasd/asdasd
    fix-stderr ls /tmp
)

echo "$var"

Answer 2

The sed command runs asynchronously from the rest of the shell; its output goes to standard error as soon as it processes its input from the commands in the command substitution. The standard output of those commands, however, are captured in $var and not displayed until the echo command runs.

Even if you weren't capturing the output, there is a chance the standard error and standard output of those commands wouldn't appear as you expect, because the sed command that ultimately produces the error messages might not be scheduled by the OS when you expect it to be, delaying the appearance of the error messages.

When you run a command in the usual way from the terminal, that command's standard error and standard output point to the same file: the terminal itself. As such, writes to the file maintain the order in which they occur in the program. As soon as you pipe one or the other to another process, you lose all control over how the two are spliced back together, if ever. In your case, you are redirecting standard error to sed, which writes modified lines back to standard output. But you have no control over when the the OS schedules sed to run and when your shell runs, so you can't control the order in which lines are written.

It helps to redirect standard error separately for each command:

tag_error () { sed 's/^/ERROR= /'; }

hostname 2> >(tag_error)
{ ifconfig | wc -l ; } 2> >(tag_error)
# etc

but this still doesn't guarantee that writes from within the same program are ordered as if they were all writing to the same file.

(ruakh has covered how to combine this with capturing standard output, so I won't bother adding it now. See his answer.)

Answer 3

One possible solution would be putting the commands in an array then execute them within a loop:

declare -a cmds=('sleep 1' 'hostname' 'eval ifconfig | wc -l' 'ls /sfsd' 'ls /tmp' 'ls hasdh')

for i in "${cmds[@]}"; do
    $i 2> >(sed -E 's/^/ERROR=/')
done

When an error occurs it should print in the same order that it occurred in the execution. Using a command such as sh script.sh within the array should also reveal any stdout or stderr from the resulting external script. For piped command an eval will likely be needed as well.