During the Pangeo meeting today with @rabernat, @jhamman, @jsignell, @jwagemann, @robfatland, @amanda-tan, and @mrocklin, the question “How do you keep your notebooks fresh?” came up and there currently isn’t an agreed upon solution for testing notebooks and detecting when they need to be updated. The purpose of this post is to collect working ideas for maintaining notebooks and collaborate on a path forward.
State the problem
“We created an awesome demo notebook to teach people how to use Pangeo, but after 2 months some upstream changes were made and the notebook no longer runs ”
If we can catch these errors in a timely manner, the notebooks are easier to maintain.
A sampling of solutions
There isn’t one common solution to this, so many groups are inventing their own, lets identify the best parts and assemble them!
- nbsmoke: does smoke testing and linking and works for bokeh/holoviews output
- testipynb: treats each notebook as a test case and checks that it runs (see for example notebooks with a recent publication
- nbsphinx: compiles notebooks for documentation, also catches errors. Used for the dask examples
- sphinx-nbexamples: Used in the pangeo documentation
Edge cases to be aware of
Some notebooks will require intensive compute steps or large data sets, so running them on the free tier of available CI systems might not be feasible. We don’t necessarily need to tackle this right now, but ideally a solution could be sufficiently extensible to address these cases.
The test scenario should be able to cover notebooks that will be running on Binder and on JupyterHub deployments. @rabernat uses a single docker image for both to avoid the scenario where the notebook runs on Binder but not JupyterHub or visa versa.
Beyond maintaining examples, testing notebooks gets to ideas around best practices in reproducible science - this is an important problem!
Please post your current solutions or ideas on how to develop a solution for testing notebooks here. We can use these as a starting point to collaborate on a common approach.