Logging while nbconvert execute

Question

I have a Jupyter notebook that needs to run from the command line. For this I have the following command:

jupyter nbconvert --execute my_jupyter_notebook.ipynb --to python

This command creates a python script and then executes it. However, I'm using the logging library in Python to log certain events. When it executes the script from the command above, nothing can be seen on the terminal.

However, when I execute manually the converted jupyter, like below, I can see all the logs on my terminal:

python3 my_jupyter_notebook.py

I've tried adding extra arguments like --debug and --stdout but those just output all the code, not just the logs. Is it possible to output on the terminal the results of logging while doing an nbconvert execute command?

score 2 · Answer 1 · answered Sep 17 '21 at 17:42

Jupyter modifies the sys.stderr and sys.stdout streams so they no longer point to the "standard" stderr and stdout (pun intended). However, they store a copy to the original file handle under _original_stdstream_copy, and you can create a log handler that explicitly writes to the original stream.

def console_handler(stream='stdout'):
    """
    Create a handler for logging to the original console.
    """
    assert stream in {'stdout', 'stderr'}, "stream must be one of 'stdin' or 'stdout'"
    # Get the file handle of the original std stream.
    fh = getattr(sys, stream)._original_stdstream_copy
    # Create a writable IO stream.
    stream = io.TextIOWrapper(io.FileIO(fh, 'w'))
    # Set up a stream handler.
    return logging.StreamHandler(stream)

You can use the function as follows.

import logging

logging.basicConfig(...)  # Or some more sophisticated setup.
logger = logging.getLogger('myLogger')
logger.add_handler(console_handler())
logger.error('something went wrong')  # This will write to stderr.

using python 3.6.9 i get that '_ original_stdstream_copy' does not exist, but there are sys.__stdout__ and sys.__stderr__ attributes that keep the original values and can be used for the same purposes (Edit: formatting) — Iosu S., May 16 '22 at 09:07
Thanks @Till Hoffman in my case running un python 3.8 I just had to change add_handler for addHandler — orellabac, Nov 29 '22 at 13:21
Any idea how to display the output of `!pip install somepackage` to the console as well? — BovineScatologist, Jul 28 '23 at 19:51

score 1 · Answer 2 · answered Jul 31 '18 at 10:23

Here is a code that catch warning and exception produced during execution of nbconvert and pass them to the logger. Jupyter and nbconvert use a different way of handling exceptions.

from logging import getLogger
import sys
import traceback
import warnings
import IPython
import logging

logger = getLogger(name)
logging.basicConfig(stream=sys.stdout, level=logging.WARNING)

# Catch Traceback 
def showtraceback(self):
    traceback_lines = traceback.format_exception(*sys.exc_info())
    del traceback_lines[1]
    message = ''.join(traceback_lines)
    logger.error(traceback_lines[-1] + str(message))
IPython.core.interactiveshell.InteractiveShell.showtraceback = showtraceback

# Catch Warning 
def warning_on_one_line(message, category, filename, lineno, file=None, line=None):
    logger.warning(str(message) + '\n' + str(filename) + ' : ' + str(lineno))
    return '%s:%s: %s:%s\n' % (filename, lineno, category.__name__, message)
warnings.formatwarning = warning_on_one_line

Not sure this actually answers the OP's question: How does one get logger messages produced within the notebook to show in the terminal when running the notebook through nbconvert? — RoyalTS, Jan 18 '19 at 08:44

Logging while nbconvert execute

2 Answers2

Linked