Testing notebooks

Take a look at fastai and quantecon’s tooling. IIRC both groups are autotesting notebooks for their courses and websites.

1 Like

This is a fairly long post linking out to various building blocks to illustrate a solution I once built for a customer.

The whole system uses CircleCI to execute several (unexecuted) notebooks with papermill, turns them into HTML, stores that HTML as artefact, and has a bot that will post a link to them in the PR as a comment.

You’d mark the executed notebook (the one you just run ) as an “artefact”. That way it will be available after the CI jobs ends.

The CircleCI config goes something like this: https://gist.github.com/betatim/0a31ea563289afa2ce077d99a64b0948

We used a bot (source here https://github.com/betatim/notebook-bot). It will post (and then update) a comment like the following in your PR:

and if you follow the link it takes you to a HTML rendered notebook:

in your browser. This means that it is easy to inspect your notebooks visually.

The CircleCI config linked above it uses papermill and a conda env on the CI node. However there is also a version that uses something like repo2docker --editable . papermill path/to/notebook.ipynb to run the notebooks inside a repo2docker built image on the CI.

Papermill has the concept of “execution engines”. These are extensions you can write (and contribute to the core package or have as separate package) that take care of executing the notebook. One idea is to build a repo2docker execution engine that takes care of creating the container for you. A BinderHub execution engine that runs your notebook on a BinderHub is a logical next step/alternative. That way you can launch your compute intensive notebooks on a BinderHub from CircleCI/travis/etc.

Your CI command would only change like this: papermill --engine binderhub --some-engine-extra-args-here notebook.ipynb output/notebook.ipynb to run the notebook on mybinder.org.

My attempt to make a kaggle engine kinda worked. It is on hold because we can’t get the rendered notebook back from the Kaggle API :frowning:


1 Like

Parcels uses nbval to ensure all tutorials work.


@willirath Have you by any chance posted a write up of how you went about setting up the nbval notebook testing?

I need to do something similar for some course notebooks and keep putting it off because I’m not sure where to start with things like travis, circle.ci etc.


Am not allowed to put more than two links in a post… So the refs are a bit weird below.
Parcels repo: https://github.com/OceanParcels/parcels/


See environment_py3_linux.yml in the Parcels repo for full env. They get pytest and nbval from PyPi, and py (why though?) from conda-forge.


Full Travis config in .travis.yml of the Parcels repo.

There is some issue with running nbval without a display. So they use

export DISPLAY=:99.0;
sh -e /etc/init.d/xvfb start;
sleep 3;
py.test -v -s --nbval-lax examples/;

in a linux image that seems to have xvfb preinstalled.

(Note that Parcels uses --nbval-lax and hence just checks for execution without error.)

A similar workflow seems to be used by the vtkiorg/vtki repo.

CI logs

This is a sample output: https://travis-ci.org/OceanParcels/parcels/jobs/515320252#L1871

1 Like

You can head to Introduce yourself! to introduce yourself to gain more karma. I’ll see if I can fix up some of the links for you.

1 Like

This second post seems to have lifted me above the initial barrier.

1 Like

Maybe relevant: https://github.com/ReviewNB/treon

1 Like

I’m also using nbval, an X display isn’t needed. It’s not run automatically by Travis yet, but in theory it should be as easy as executing a script.

This is the test script: https://github.com/IDR/idr-notebooks/blob/9b639ca2fdb4cf6b4f64802f7e656f0f67aa9261/docker/test_notebooks.sh

This is all run inside a Docker container: https://github.com/IDR/idr-notebooks/blob/9b639ca2fdb4cf6b4f64802f7e656f0f67aa9261/docker/test.sh

I was going to mention nvbal I pretty much use it will all my notebooks collections/repos. In the case of very lengthy computations you can mark cells that should be skipped during the automated test. Which is useful when doing loads of development in a short period of time.

Maybe relevant: https://github.com/ReviewNB/treon

I built treon recently for testing Notebooks. It runs through every Notebook top to bottom in a fresh kernel and flags execution errors if any. Additionally, you can add unittests, doctests in the Notebook cell and they would get executed as well. More details in Readme.

It’s a command line tool so can be used easily in any CI environment. I intend to build Notebook CI at https://www.reviewnb.com/ with the help of treon.

To pick this up again: I’m playing with papermill in a repo2docker-built docker image run on Github actions here.

All this basically works. But there’s some friction from the way Github actions are set up if things are directly run in the container (uid 1001 for GH Actions vs. 1000 for repo2docker, non-interactive shells not properly activating conda, etc.). I didn’t, for example, succeed in running a notebook that %pip installs stuff along the way.

1 Like

In the mean time @hamel has done some cool stuff with repo2docker and GH actions in:

Maybe worth looking/sharing/reusing from that as well.

(Which reminds me we wanted to move that repository to the JupyterHub organisation on GH… :smiley: )


Yes please do let me know if anyone wants to use that Action or has any feedback, I will gladly make changes or improvements!

1 Like

Also our Google Summer of Code project in nteract had it’s first initial release for testing notebooks: https://test-book.readthedocs.io/en/latest/. It treats ipynb files as py files for testing purposes. Try it out, @rohitsanj is looking for early users to get feedback on the library.

Given a code cell in a Jupyter Notebook:

def func(a, b):
   return a + b

You can write a unit test using testbook in a Python file as follows:

import testbook

@testbook.testbook('/path/to/notebook.ipynb', execute=True)
def test_func(tb):
   func = tb.ref("func")

   assert func(1, 2) == 3

Can I pass/set a relative path as well in the testbook decorator? Or asking my real question: what is the idea for dealing with notebooks and their tests being moved around on the filesystem?


Yes the notebook path can be relative, it just does an open based on the current working directory. Though what you’re maybe getting as is it should be relative to the test file? It doesn’t do that today, but I believe is possible to do and would be a good issue to open if you wanted to chat through options / expectations.


what is the idea for dealing with notebooks and their tests being moved around on the filesystem?

The idea was to follow conventional unit testing norms of having the source code (notebook) and tests as separate entities.

As to tests being moved around… that’s again an issue where the user needs to make sure that all the test modules are organized and can provide a uniform relative path to the notebooks which could be at least one directory up.

Something like this…


And the relative path in the testbook decorators will be ../notebooks/foo.ipynb and so on.


Relative paths sound great. I was mainly thinking about the case where there is a repository of notebooks + tests that you checkout to some different absolute path than I. Or me moving the directory around.

Relative paths sounds like a good solution. I don’t have any experience/inputs, just curious to find out what you were thinking as you are one of the world wide experts on this because you actively work on this :smiley:


Yes I have an example:

This calls a bash script

This is used to refresh notebooks with papermill with CI on a recurring schedule for https://covid19dashboards.com/

cc: @choldgraf