0

I have the following notebook test.ipynb:

[1]: print("hi")
     !pip install boto3

After running the notebook with this command:

jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --inplace --execute test.ipynb

I get the following output:

[NbConvertApp] Converting notebook nbconvert.ipynb to notebook
[NbConvertApp] Writing 5088 bytes to nbconvert.ipynb

I would like to view cell output in real-time in the command-line along with the NbConvertApp logs, e.g.:

hi
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: boto3 in /home/hadil/.local/lib/python3.10/site-packages (1.26.41)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/hadil/.local/lib/python3.10/site-packages (from boto3) (1.0.1)
Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /home/hadil/.local/lib/python3.10/site-packages (from boto3) (0.6.0)
Requirement already satisfied: botocore<1.30.0,>=1.29.41 in /home/hadil/.local/lib/python3.10/site-packages (from boto3) (1.29.41)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3/dist-packages (from botocore<1.30.0,>=1.29.41->boto3) (2.8.1)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.30.0,>=1.29.41->boto3) (1.26.5)

I have figured that if I change the print statement to a log with a custom console handler, then I could get "hi" output to the command-line in real-time thanks to this answer: https://stackoverflow.com/a/69227590/10695613

[1] import logging 
    import sys
    import io

    def console_handler(stream='stdout'):
        assert stream in {'stdout', 'stderr'}, "stream must be one of 'stdin' or 
       'stdout'"
        fh = getattr(sys, stream)._original_stdstream_copy
        stream = io.TextIOWrapper(io.FileIO(fh, 'w'))
        return logging.StreamHandler(stream)

    logger = logging.getLogger()
    logger.addHandler(console_handler())
    logger.setLevel(logging.INFO)

    logger.info("hi")

    !pip install boto3

Output:

[NbConvertApp] Converting notebook nbconvert.ipynb to notebook
hi
[NbConvertApp] Writing 2716 bytes to nbconvert.ipynb

It remains a mystery, however, to get the output of !pip install boto3 in the command-line.

EDIT: nevermind, I found a way to do it:

import logging 
import sys
import io
import subprocess

def console_handler(stream='stdout'):
    assert stream in {'stdout', 'stderr'}, "stream must be one of 'stdin' or 'stdout'"
    fh = getattr(sys, stream)._original_stdstream_copy
    stream = io.TextIOWrapper(io.FileIO(fh, 'w'))
    return logging.StreamHandler(stream)

logger = logging.getLogger()
logger.addHandler(console_handler())
logger.setLevel(logging.INFO)

logger.info("hi")
result = subprocess.run('pip install boto3', shell=True, stdout=subprocess.PIPE).stdout.decode('utf-8')
logger.info(result)

Let me know if I should delete the question.

BovineScatologist
  • 145
  • 1
  • 7
  • I'm confused why you are converting a notebook to a notebook :( Also its not really clear why you would need to see cell output from a notebook when converting a notebook (to a notebook). I guess if you got it to work that's good - but what are you doing? – topsail Jul 29 '23 at 15:23
  • @topsail we have a cronjob that runs an 'ETL notebook' using nbconvert. The ETL job occasionally fails. Real-time output while running the notebook with nbconvert is immensely helpful for debugging. Why notebook to notebook? Everyday, the notebook has a different output/result. – BovineScatologist Jul 30 '23 at 08:06
  • couldn't you just convert the notebook to a script (permanently) and run the script going forward? Why use a notebook? A python script would be much better than a notebook in a situation like this. It just seems strange to me, that's all. nbconvert is not normally for "running" notebooks, it is for converting them to other formats. So, well, just curious. Don't answer if you are busy. – topsail Jul 30 '23 at 13:11

0 Answers0