Python is not working in unbuffered mode

Question

I'm stuggeling with an issue with python. I used Python 2.7.13 and Python 3.6.0 on Red Hat Enterprise Linux Server release 7.1 (Maipo). In order to monitor the proccesses output, I want to use tail -f to take a live look into the STDOUT and STDERR. The Keyword here is unbuffered output. Many suggestion on the internet say use python -u ... or the environment variable PYTHONUNBUFFERED like PYTHONUNBUFFERED=1 python ... or stdbuf -e0 -o0 python .... Nevertheless nothing is wokring for the following test script.

import sys
import time
while(True):
   print("Test String")
   time.sleep(1);

For all different commands I always have buffered output. Even though when I want to use STDERR. It's still buffered which really confuses me because STDERR should be unbuffered by default. Using sys.stdout.flush() or sys.stderr.flush() is also not doing the job. When using flush=True inside print() it's working as intended.

I'm looking for a solution which doesn't make it necessary to edit code because I can't edit all programs in order to get unbuffered and immediately flushed out output. How can I achieve this?

Looking forward to your answers!

Best wishes!

So my original script has some output of deep learning and neural network steps I want to monitor. In my test script I narrowed it down to some dummy output to see if the output is unbuffered. @ElisByberi for that I've got to edit code. Some programs are so huge it would end in a mess to edit it all.. I'm looking for something `python -u` or `PYTHONUNBUFFERED` instead that they're working.. I don't even know why they're not working as advertised. — fwillo, Nov 07 '17 at 14:25
Again. What do you want `print("Test String")` to do that it doesn't? no matter if buffered or not, the output will be the same. — Nils Werner, Nov 08 '17 at 18:21
As described in my original post, I want to call a python script in the background which is creating stdout and stderr output in files. From time to time I want to monitor output created by this script via `print` with `tail` to see how the progress is. `print("Test String")` is just a dummy script with which I want to test that. Until now, stdout and stderr as well as `print` create buffered output when redirecting them into a file. Therefore it is hard to monitor my logs with `tailf` because I get new output every 4-6 hours. — fwillo, Nov 09 '17 at 17:25

Elis Byberi · Accepted Answer · 2017-11-08T18:18:23.517

4

You can override print() function in Python 3. This way you will not need to change every print() function in your scripts.

import builtins


def print(*args):
    builtins.print(*args, sep=' ', end='\n', file=None, flush=True)


print(
    'hello', 'world',
    'I have overrode print() function!',
    1,  # integer
    [1, 2],  # list
    {1, 2},  # set
    (1, 2),  # tuple
    {1: 2}  # dict
)

will print:

hello world I have overrode print() function! 1 [1, 2] {1, 2} (1, 2) {1: 2}

edited Nov 08 '17 at 18:18

answered Nov 07 '17 at 14:32

Elis Byberi

1,422
1
11
20

Thanks for your suggestion. Unfortunately the output is `('Test String',)`. I'd be happiest when `python -u`or `PYTHONUNBUFFERED` would work as advertised.. do you know if I missed something regarding those to options? – fwillo Nov 08 '17 at 13:59
1

@TheOrangeman To be honest, I did not test the code. I put it there as an example only. It works now as expected. Thank you for reporting it. – Elis Byberi Nov 08 '17 at 18:20
@TheOrangeman Try this command: `python -e PYTHONUNBUFFERED=0 yourscript.py` – Elis Byberi Nov 08 '17 at 18:35
Hey Elis, thanks for correction. As you said, the builtins function works now as advertised by you. When trying `python -e` both of my python Installations say that the `-e` parameter is an unknown option :( – fwillo Nov 09 '17 at 17:22
@TheOrangeman In `python --help` are listed all options and arguments (and corresponding environment variables). It is capital E. This would work: `python -E PYTHONUNBUFFERED=0 yourscript.py` – Elis Byberi Nov 09 '17 at 19:19
1

Also not working. The Documentation tells me about this `-E Ignore all PYTHON* environment variables, e.g. PYTHONPATH and PYTHONHOME, that might be set.` Weirdly enough `PYTHONUNBUFFERED=1 python [...]` is working now... I triple checked the spelling of the environment variable. I can't explain this and feel really dumb that I took your time. Thanks for your efforts @Elias Byberi. – fwillo Nov 09 '17 at 21:57
@ElisByberi does this function override even prints inside packages? – Leonardo Rick Feb 13 '21 at 05:55
@LeonardoRick You can override a function in a script only, then you can import that script in other scripts. E.g. You override function `print()` in script custom.py and wherever you import custom.py function `print()` will get overridden. – Elis Byberi Feb 13 '21 at 19:30
@ElisByberi The problem is when the print is maded inside a dependecy library of my project. This way I can't override this method print so your solution do not work for me – Leonardo Rick Feb 13 '21 at 20:02
@LeonardoRick It’s not perfectly clear to me what it is you’re trying to say. – Elis Byberi Feb 13 '21 at 20:10
@ElisByberi I use a library (pyrogram) that emits a print itself. This print will not be unbeffered if I just override the function on my `main.py` – Leonardo Rick Feb 13 '21 at 20:39
@LeonardoRick I emphasize that **"You can override a function in a script only."** E.g. If you override function `print()` in script main.py, it will not override that function anywhere else. – Elis Byberi Feb 13 '21 at 20:48

Python is not working in unbuffered mode

1 Answers1

Linked