`executenb`: Monitor cell execution stdout stream asynchronously

Notebook nbconvert
1

I am using nbconvert to execute code snippets inside generated notebooks in order to capture the rich outputs that are produced in JupyterNotebook frontend:

import nbformat
from nbconvert.preprocessors.execute import executenb
from jupyter_client.kernelspec import get_kernel_spec

code_block = '''
import pandas as pd
df = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})
df  # produces HTML table
'''

kernel_name = 'python3'
spec = get_kernel_spec(kernel_name)
notebook = nbformat.v4.new_notebook(metadata={
    'kernelspec': {
        'display_name': spec.display_name,
        'language': spec.language,
        'name': kernel_name,
    }})
notebook.cells = [nbformat.v4.new_code_cell(code_block)]
jupyter_execute_kwargs = dict(timeout=-1, allow_errors=True, store_widget_state=True)
executenb(notebook, **jupyter_execute_kwargs)

# Execution output is appended to every cell's `outputs` dict field
for c in notebook.cells:
    print(c.outputs)

This allows me to capture HTML outputs, plots etc… but it also means that all stdout and stderr becomes available when the code block execution has finished. Is there some way to monitor the stdout and stderr streams asynchronously, as they are generated, while the code executes?

The documentation of nbconvert.preprocessors.execute.executenb states: “Execute a notebook’s code, updating outputs within the notebook object”. I was hoping this implies that it is indeed an asynchronous process

2

I know this is an old topic; however, I think I recently came upon a feature of Papermill that would have enabled this, I believe, and it may help others trying to monitor and interact with a notebook as it gets run from the command line. See here where it says Papermill saves a cell’s result after each cell is run, and not just at the end.