1

I have a command that takes a long time that I like to run in the background like this:

python3 script.py -f input.key -o output >> logs/script.log 2>&1 &

This works perfectly in the sense that the command is indeed in the background and I can check the output and potential errors later.

The main problem is the output is only appended after the command is completely finished, whereas I would like to have up-to-date log messages so I check the progress.

So currently the log would be empty and than suddenly at 08:30 two lines would appear:

[08:00] Script starting...
[08:30] Script finished!

Instead, I would like to have output saved to file before the command is completely finished.

l'L'l
  • 44,951
  • 10
  • 95
  • 146
Roberto
  • 958
  • 13
  • 33
  • 1
    See [this](https://stackoverflow.com/questions/24621790/realtime-output-redirection) and [this](https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe) post. I guess in mac pipe doesn’t do buffering because I am getting realtime output. So can’t test it. – Mihir Luthra Jul 20 '19 at 10:07
  • 1
    Can you show us how you print from python? If I remember some calls are buffered and you might have to call `sys.stdout.flush()` – urban Jul 20 '19 at 10:50
  • I use regular `print()` statements. Buffering was indeed the problem, and your solution might work too. – Roberto Jul 20 '19 at 14:12

1 Answers1

2

Since you are calling a Python script you would want to use the -u option, which forces the stdout and stderr streams to be unbuffered.

$ python3 -u script.py -f input.key -o output >> logs/script.log 2>&1 &

You can check the log periodically using cat or realtime in combination with watch:

$ watch cat logs/script.log

↳ https://docs.python.org/3.7/using/cmdline.html#cmdoption-u

l'L'l
  • 44,951
  • 10
  • 95
  • 146