Redirection operator is not working as expected

Question

I have 2 files in my current directory : /home/tushar/Desktop

file1.txt
file4.txt

So when I hit command:

1. ls file{1..4}.txt

Output:

ls: cannot access 'file2.txt': No such file or directory
ls: cannot access 'file3.txt': No such file or directory
file1.txt  file4.txt

Now I redirect both stderr and stdout to a file res.txt using :

2. ls file{1..4}.txt 1>res.txt 2>res.txt

res.txt :

file1.txt
file4.txt
ile2.txt': No such file or directory
ls: cannot access 'file3.txt': No such file or directory

Above, we can see that the some content of the stderr got omitted in the file res.txt.

Now I changed my command to:

3. ls file{1..4}.txt 1>res.txt 2>&1

res.txt :

ls: cannot access 'file2.txt': No such file or directory
ls: cannot access 'file3.txt': No such file or directory
file1.txt
file4.txt

Both above commands are exactly same except in 2nd I redirected stderr where stdout is redirecting using file descriptor(&1).

Now I know I should use a command which opens res.txt in appending mode as given below :

4. ls file{1..4}.txt 1>>res.txt 2>&1
OR
5. ls file{1..4}.txt &> res.txt

but my concerns are:

Why the results of cmd 2 and cmd 3 are different ?

Why cmd 3 derived correct output even without using >> symbol but not cmd 2 ?

Answer 1

For the second question:

Because 2>&1 means that fd 2 is a duplicate of fd (file descriptor) 1. Then the > or >> open 1 (stdin) in append or in new/truncate mode. So they describe two different operation, using the same > symbol, just because: 1- they are working on file descriptors (> could remember that, instead of new symbol, e.g. involving $, 2- It was an invalid syntax, before adding such extension, so the meaning is unique.

Question 1 is more tricky. You are opening the same file twice, independently, and the program see them independently, so they may have different cache and priorities. In general: do not do 2, but if you do it, you must flush output at every line (and keep line possibly short). You may see something like 2, in logging, but so, sometime you also see output of two programs that are mixed, and partially merged.

I'm also not sure that 2. is defined, and available in all POSIX compatible shells.

Answer 2

In cmd 3 you are redirecting the stderr to res.txt. You will find the error message in res.txt.

Answer 3

Why the results of cmd 2 and cmd 3 are different ?

Because both stdout and stderr override res.txt content

Why cmd 3 derived correct output even without using >> symbol but not cmd 2 ?

cmd 3 won't always work, see "How to redirect and append both stdout and stderr to a file with Bash?"

 ls file{1..4}.txt >>res.txt 2>&1

That would work better, because the file is opened in append mode, and the file descriptor '2' for stderr will use the same mode.

Answer 4

To answer the first question in a bit more detail: when you run ls file{1..4}.txt 1>res.txt 2>res.txt, it opens res.txt twice, independently. This means that each file descriptor writing to it has a different idea of where it is (its offset, or position) in the file.

When the command starts running, you have an empty file, with two file descriptors, both of which are ready to write to the beginning of the file. So far so good.

ls then prints its error messages to stdout, and they get stored in the file, starting at the beginning. We're still ok at this point; the file looks like this:

ls: cannot access 'file2.txt': No such file or directory
ls: cannot access 'file3.txt': No such file or directory

After that, ls starts writing the files it found to stdout. And here we have a problem: the stdout file descriptor is still pointing to the beginning of the file, so it starts overwriting the file from the beginning. It writes the first filename, giving:

file1.txtt access 'file2.txt': No such file or directory
ls: cannot access 'file3.txt': No such file or directory

Then it writes a newline, which overwrites the "t" from "cannot", giving:

file1.txt
 access 'file2.txt': No such file or directory
ls: cannot access 'file3.txt': No such file or directory

Then it writes the next filename, giving:

file1.txt
file4.txtfile2.txt': No such file or directory
ls: cannot access 'file3.txt': No such file or directory

And then a final newline, this time over the "f" in "file2":

file1.txt
file4.txt
ile2.txt': No such file or directory
ls: cannot access 'file3.txt': No such file or directory

...and that's the final result you see.

Basically, the two descriptors are both trying to modify the same file at the same time, but aren't properly coordinated (as they would be if one was a copy of the other), so they trip over each other.