0

let's assume I have a text file abc.txt and I want to do some operations on it and then save the result to itself like this:

sort abc | head -10 | tail -3 | cut -f3 > abc

or more simple:

sort abc > abc

I understand that the contents of the file will be deleted because I have used the > operator. My question is: why is this happening?

Mikołaj Głodziak
  • 4,775
  • 7
  • 28
  • https://unix.stackexchange.com/a/185600/358681 – acvill Jun 02 '22 at 13:48
  • 1
    **All parts of a pipeline run in parallel.** That means the final output destination has to be open from the very beginning of the process. If you're opening that output file with the `O_TRUNC` flag set instead of appending, that means the output destination is empty the moment the pipeline finishes invocation; since that file is also the input file, that means it's very likely to be empty from the moment `sort` finishes startup and tries to start reading input. – Charles Duffy Jun 02 '22 at 13:50
  • Something like `sort file > tmp && mv tmp file` should work. – James Brown Jun 02 '22 at 13:50
  • 1
    ("Very likely" instead of "certain" because there are race conditions here; if your pipeline is long enough, it's possible -- albeit very unlikely -- for `sort` to start reading input _before_ the shell starts setting up the `cut` process at the end; the safe thing is to consider invocation ordering undefined and avoid relying on it in any way). – Charles Duffy Jun 02 '22 at 13:54
  • 2
    ...note that the "setting up" referenced above is `open("abc", O_WRONLY|O_TRUNC)` done before the `dup2()` copying that FD to stdout of the process-that-will-later-be-`cut`, which is done before the `execve()` to transform that process _into_ a copy of `/usr/bin/cut`. So having `/usr/bin/cut` be slow to start up makes no difference -- the `open()` causing the truncation is done _by the shell_ before `cut`'s invocation starts. – Charles Duffy Jun 02 '22 at 13:56
  • @CharlesDuffy, yes, you have totally right. My commands were only as a example. Thanks for your useful comments! – Mikołaj Głodziak Jun 02 '22 at 14:03

0 Answers0