How to Version Control Jupyter Notebooks

Notebook
1

Hello all

I thought a few of you might be interested in this overview, How to Version Control Jupyter Notebooks. I try to strike a balance between writing something detailed but includes a broad number of tools. Feedback and discussion welcome!

/David

.

5 Likes
2

If I intend to publish the notebook ā€˜directlyā€™ (nbviewer) with intact output cells. I tend to use print(ā€¦) and save charts as external PNG and then insert those external images. Both keeps the notebook files more manageable by git, and especially the diffs quite clean.

And ā€œrestart kernel & cleanā€ followed by ā€œrun allā€ before a commit. That adds the insurance that you donā€™t have added any state / order problems, unlike just cleaning the outputs.

3

Great summary. Chiming in wrt my workflow (hundreds of notebooks used for research, collaborative development of university courses, graduate student projects etc.) Iā€™ve found jupytext to be transformative. A couple of comments from my own experience:

  1. I keep the python version and the json version in different folders. This makes diffing, grepping, adding files to the git index etc. easier. The ipynb files reside one level up, but arenā€™t committed to my git repo. My global jupyter_notebook_config.py looks like this:
c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"  # noqa
c.ContentsManager.preferred_jupytext_formats_save = "py:percent" # noqa
c.ContentsManager.default_jupytext_formats = "ipynb,python//py" # noqa
c.ContentsManager.default_notebook_metadata_filter = "all,-language_info"
c.ContentsManager.default_cell_metadata_filter = "all"

This passes almost all metadata and pairs mynotes.ipynb with python/mynotes.py, creating the python folder if it doesnā€™t exist.

  1. I use the pre-commit package to run black, reorder-python-imports and flake8 on every python file, which is obviously a major improvement for informative diffs.

  2. For those of you using nbgrader-- youā€™ll need to disable jupytext when you convert source notebooks to student released versions. You can do that by editing the formats metadata string for that notebook to ā€œipynbā€ only.

  3. Hereā€™s an example of a set of teaching notebooks with their py:percent counterparts: EOSC 213

  4. Bottom line ā€“ jupytext is the missing piece Iā€™ve been looking for since I started working with IPython notebooks

2 Likes
4

For me nbdime (https://nbdime.readthedocs.io/en/stable/) works extremely well especially if you set it up so that git uses it for rendering diffā€™s between notebooks.

The only time people I work with run into trouble with this is when they attempt to put too much code into a notebook. However I consider this a feature not a bug as the advice then is to put that code in a .py file instead. This keeps the notebook focussed on explaining stuff and we get real IDE features for the code.

1 Like
5

Do you just do the reset/run/commit every time or have you written a script to accomplish this? Itā€™s simple enough, but Iā€™m sure I could still somehow mess this up.

6

Since I add -p on principle there is no persistent messing up.

7

And nbdime is preferable to ReviewNB because you want to use git and not GitHub or some additional flexibility? The ReviewNB folks just added GitHub commenting as part of their product, which looks pretty slick.

8

@schmudde, any thoughts on how to manage NBs in pull requests?

Iā€™m reviewing a PR in BitBucket; it doesnā€™t render NBs nicely and therefore I end up scrolling up and down the page. What are the options in the case? Should I review the PR locally and not in BB?

9

I using GitHub, Iā€™d highly suggest ReviewNB. It allows you to see notebook diff for any commit or pull request. But thatā€™s a GitHub plugin and you said youā€™re on BitBucket.

NBDime also offers version control integration, but with Git and Mercurial only.

What did you end up doing? Visual diffing locally seems like a manual process - but perhaps the only one available in this instance.

10

I ended up excluding .ipynb from the diff page, itā€™s still in Beta (BitBucket Labs) at the moment.

11

Hi! Weā€™ve been building automatic version control support for all Jupyter notebooks. Itā€™s basically an addon (that you can download and use for free) that adds a button to your notebook. The button a) runs your experiment on your cloud instance of choice (AWS, GCP, Azure) and b) stores a snapshot of the notebook so you can revert to any experiment you did 5 minutes ago or 5 years ago ā€“ including the notebooks outputs (e.g. if you plot some graphs).

Screenshots, videos, descriptions and instructions here: https://blog.valohai.com/valohai-jupyter-notebook-extension

Please add feedback!

1 Like
12

Sorry for (kind of) marketing plug here but I think it will be interesting to you.

Weā€™ve recently built an extension to jupyter-notebooks and jupyter-lab that lets you version checkpoints by clicking a button and then you can browse versions and diff easily:

nb_compare
nb_compare.gif800Ɨ425 2.72 MB

and I can easily share it with anyone by sending a link like this one.

Weā€™ll be adding new features around this like commenting and stuff and I would love to hear what you think and what you would like to see there.
Anyway, I hope this helps.

13

Finally got to take a look at this. Itā€™s really slick! Happy to see the inclusion of rich media in the diffs. Very cool. Iā€™ll add it to the article.

Iā€™m curious - how does it generate the versions?

14

Glad you liked it @schmudde!

If you install the extension you will get a new upload button in your jupyter lab/notebook.
Then, whenever you decide to upload a snapshot you click that button (you can name the snapshot and add a description too).

If you want to version your machine learning experiment runs inside of a versioned notebook (a lot of versioning I know), then those snapshots can be generated automatically whenever you run a cell with neptune.create_experiment() in it.
You can see an example of model training in notebook here.

15

Hi all!

In the spirit of moving this discussion about diffing/merging, version controlling, etc forward, weā€™re working on a new OPTIONAL file format JEP. Hereā€™s the thread - Proposed-JEP: Investigate alternate, optional file formats

Please come join the discussion!

16

@schmudde,
A thanks for jupytext.
I have a few questions, and I canā€™t find a better place to ask them.
my setup is jupyterlab on a tljh instance.
Iā€™ve got the *.ipynb in my gitignore so that I only save the *.py files.
now, whwn i pull a new py file, the jupytext does not automatically convert it to the notebook version,
and I need to jupytext --to notebook file.py myself.
even though my .jupyter/jupyter_notebook_config.py filr has the

c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager" c.ContentsManager.notebook_extensions = "ipynb,py"

in it.
that seems to only sync from notebook to script.

am i missing something?

if i generate the notebook with the --to command above,
i need to also touch the ā€¦py file.

I must be doing something wrong. because this would all be solved if the autopair worked in both directions.

17

Hereā€™s a docker image that is behaving the way I expect (i.e. when I create and save a new notebook
both a .md and a .ipynb file are created, and when I change one of them in jupyter the other one
is modified).

Hereā€™s the config file that does that for the spawned notebook:

1 Like
18

Is there a way to configure Jupyer that on exit it will clean all outputs of teh file leaving it basically with the code itself?

19

Hi @Royi - this issue might contain the info you need: Suggestion for content: Configuring jupyter to scrub notebook output Ā· Issue #1803 Ā· alan-turing-institute/the-turing-way Ā· GitHub

1 Like
20

Just to say - like others - I have got completely used to Jupytext, configured to save the notebook as .Rmd (RMarkdown), and I ignore the .ipynb files for version control. The .Rmd files (or your preferred text flavor) are so much easier to diff, and they donā€™t contain the output.