Testing notebooks

Binder
21

Yes please do let me know if anyone wants to use that Action or has any feedback, I will gladly make changes or improvements!

1 Like
22

Also our Google Summer of Code project in nteract had it’s first initial release for testing notebooks: https://test-book.readthedocs.io/en/latest/. It treats ipynb files as py files for testing purposes. Try it out, @rohitsanj is looking for early users to get feedback on the library.

Given a code cell in a Jupyter Notebook:

def func(a, b):
   return a + b

You can write a unit test using testbook in a Python file as follows:

import testbook

@testbook.testbook('/path/to/notebook.ipynb', execute=True)
def test_func(tb):
   func = tb.ref("func")

   assert func(1, 2) == 3
5 Likes
23

Can I pass/set a relative path as well in the testbook decorator? Or asking my real question: what is the idea for dealing with notebooks and their tests being moved around on the filesystem?

3 Likes
24

Yes the notebook path can be relative, it just does an open based on the current working directory. Though what you’re maybe getting as is it should be relative to the test file? It doesn’t do that today, but I believe is possible to do and would be a good issue to open if you wanted to chat through options / expectations.

4 Likes
25

what is the idea for dealing with notebooks and their tests being moved around on the filesystem?

The idea was to follow conventional unit testing norms of having the source code (notebook) and tests as separate entities.

As to tests being moved around… that’s again an issue where the user needs to make sure that all the test modules are organized and can provide a uniform relative path to the notebooks which could be at least one directory up.

Something like this…

notebooks/
tests/

And the relative path in the testbook decorators will be ../notebooks/foo.ipynb and so on.

3 Likes
26

Relative paths sound great. I was mainly thinking about the case where there is a repository of notebooks + tests that you checkout to some different absolute path than I. Or me moving the directory around.

Relative paths sounds like a good solution. I don’t have any experience/inputs, just curious to find out what you were thinking as you are one of the world wide experts on this because you actively work on this :smiley:

2 Likes
27

Yes I have an example:

This calls a bash script

This is used to refresh notebooks with papermill with CI on a recurring schedule for https://covid19dashboards.com/

cc: @choldgraf

2 Likes
28

This is used to refresh notebooks with papermill with CI on a recurring schedule for https://covid19dashboards.com/

Those are really neat btw @hamel ! And that’s a good example of some easily combined tools to add real value for folks.

Several jupyter related libraries use a pattern like this: https://github.com/nteract/papermill/blob/2d26912065575e955595e711fee3f4c415e8ea81/tox.ini#L31-L35 to also test notebooks used in documentation or example directories. I’m probably going to transition those to testbook in combination with How to parametrize fixtures and test functions — pytest documentation to make it more visible and have better debugging for failures in the future.

2 Likes
29

I started working on a project to test notebooks with non-deterministic outputs. For example, a cell that prints the accuracy of a model is expected to vary from one run to the other. The idea is to keep a record of the previous cell’s outputs and throw an error when the output deviates too much from the expected range (right now the range is defined as 3 standard deviations from the mean).

Currently, it only works with numeric outputs. I’ve been thinking of adding support for plots.

I’d love to hear what others think!

1 Like