1

Here is a short example of the problem

Short script that writes every 10 seconds test to its stdout

> cat write.sh #!/bin/sh while [ 1 ];do echo test sleep 10 done

And here is how we execute it, where we redirect the stdout to file named output.txt

./write.sh > output.txt

Now, the script should run "forever", but for meantime it fills up the output.txt

We are looking for a way to backup this output.txt without to restart the script. Moreover, we are looking for an implicit solution, so even the script won't be aware that the file was backed up.

Of course that once we remove this file (e.g. compress it) the process (script) lose the track of this file and stop writing. Even if we backup the file (compress) and then return the file back using touch the script no long able to reattach to this file.

For example:

mv test_file.txt test_file111.txt

rm test_file111.txt

> lsof | grep write write.sh 2644 ronnyr 1w REG 253,1 36 106059124 /home/ronnyr/test_file111.txt (deleted)

> touch test_file111.txt write.sh 2644 ronnyr 1w REG 253,1 36 106059124 /home/ronnyr/test_file111.txt (deleted)

chepner
  • 497,756
  • 71
  • 530
  • 681
YanivN
  • 391
  • 1
  • 3
  • 9
  • See the `logrotate` command. – chepner Mar 24 '14 at 13:54
  • That is a nice solution, but it seems that I need to use root for this in configuration and/or execution part (will be sure more after I will investigate it more..) anyway, I do look for a way to backup log as I described and not likely "the best way to rotate files". please note that the solution needs to work also on Solaris – YanivN Mar 24 '14 at 14:37

1 Answers1

2

If you want to alter or truncate the output file use >> like this:

./write.sh >> output.txt
# take backup
cp -p output.txt /path/backup/
# truncate stdout file
> output.txt

Due to use of >> (append) ./write.sh will keep writing the output at the end of file and will remain unaffected when you truncate the output file.

chepner
  • 497,756
  • 71
  • 530
  • 681
anubhava
  • 761,203
  • 64
  • 569
  • 643
  • There is a race condition here: write.sh could write more output to output.txt after you copy it to backup.txt, but before you truncate it. You would lose those in-between updates. – chepner Mar 24 '14 at 13:53
  • Yes there will be bit of race condition here and I'm not sure how to avoid it. Probably `sed -i` will be a better option instead of copy & truncate. – anubhava Mar 24 '14 at 13:59
  • Ok, so I did what you suggested, and changed the redirection from `>` to `>>`, and then I copied the output file. but that doesn't solve my problem, because the script still directs its output to `output.txt`, how can I "truncate" it so write.sh will continue its writing to a "new" & "empty" file. b.t.w. I don't mind to lose some outputs during that "backup maintenance" – YanivN Mar 24 '14 at 14:02
  • From shell, I think the only thing you can do is suspend `write.sh` with a STOP signal, flush its output to ensure everything has actually been written to `output.txt` (even possible from shell?), *then* copy and truncate, and finally restart `write.sh` with a CONT signal. `sed -i` still doesn't atomically rename the input file, so the race condition would still be there. – chepner Mar 24 '14 at 14:07
  • @chepner: As per OP `I don't mind to lose some outputs during that "backup maintenance"` so I think this approach will be fine for this case. – anubhava Mar 24 '14 at 14:11
  • Maybe I am missing something here, but I don't understand in your solution the following line: `> output.txt` what do you mean by that? on both cases `./write.sh >> output.txt` and `./write.sh > output.txt` the output is lost according to lsof – YanivN Mar 24 '14 at 14:24
  • `> output.txt` actually truncates the `output.txt` making it a 0 byte file. With `>` it will not work as shell will write that many null bytes. It can only work with `>>` – anubhava Mar 24 '14 at 14:55
  • 1
    wow - amazing, 10x!!! I now understand why it didn't work for me - I used `tcsh` as my shell rather than `sh`, therefore I received the error `Invalid null command.`, but when I executed as follow: `sh -c '> output.txt'` it worked like magic :-) 10x u all for the help and especially to @anubhava :-) – YanivN Mar 24 '14 at 15:12
  • Ah I didn't know you were using tcsh otherwise I would have alerted you. Glad to know it worked out well. – anubhava Mar 24 '14 at 15:13
  • @anubhava - Is there a way to make `> test.txt` return immediately, without using `CTRL+C` or perhaps I should stick with `echo -n > test.txt`? Another thing wouldn't it make more sense, just doing `./write.sh >> /path/backup/output.txt` and skip the copy instruction? Perhaps adding a timestamp to the filename would be nice as well. – Cyclonecode Sep 18 '19 at 05:50
  • `> test.txt` returns immediately on my shell. – anubhava Sep 18 '19 at 05:59