One of the issues with using notebooks in a code development workflow (and yes, I know… JupyterLab… but I prefer notebooks for several reasons…).
A common example is the mess you can get in with half-working notebooks used as scratchpads, and code management around code within and derived from the notebooks.
One of my attempts at managing this is to load notebooks as modules into other notebooks. The notebook modules are littered with if __name__=='__main__': guards in some of the code cells; these guards allow me to call on code fragments for testing and previewing the outputs of functions defined within that module notebook when I run / check/ develop code explicitly within the module notebook.
I’ve also started using Jupytext to allow me to write text and code in markdown (.md) files edited and run as notebooks (the code cell outputs aren’t saved to the md file, just the md and code source; a “dual” notebook, with outputs, can also be automatically saved alongside the md file). Checking the markdown file into git makes Github diffs useful again.
Equally, I could use Jupytext to let me author notebooks saved as Python (.py) files.
Now I’m wondering what I’d need to do to let me write code in Python files (.py) rendered as editable notebooks using Jupytext, without lots of if __name__=='__main__': baggage.
I need to check that Jupytext respects RawNBConvert cells (about) but if it does, I wonder if the following workflow would make sense:
- edit the py file as a notebook under Jupytext, with code from previously guarded code cells in the RawNBConvert cells marked as ‘Python’ RawNBConvert cells;
- the py file can be loaded as a module into other py files or notebooks in the normal way (none of the special machinery required to load notebooks into notebooks, etc);
- (the voodoo magic step) when viewing a .py file as a notebook under Jupytext, if an NBConvert format cell is set to ‘Python’, treat it as an executable code cell.
Now I should be able to have a messy .py file with lots of code I can execute when I view it as a notebook under Jupytext, but that isn’t executed if I load the .py file in as a module anywhere?
Or is that all just crazy talk?
1 Like
Hello Tony,
Indeed, raw cells are rendered as inactive comments in the python file. Yet there’s a better answer to your question: you could mark those cells as active only in the ipynb file, with an "active": "ipynb" cell metadata.
This is documented here: https://jupytext.readthedocs.io/en/latest/examples.html#importing-jupyter-notebooks-as-modules
Marc
Hi Marc
Thanks - that’s exactly what I want… (example)
If I also just define my environment as:
c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
and don’t define any pairing (i.e. leave c.ContentsManager.default_jupytext_formats unset, then this means I should be able to:
- edit all my
.py python files in the notebook interface via Jupytext;
- run all the cells (including the metadata’d ones) in that UI;
- only save the document as a .py file (no ipynb);
- save the code in the metadata’d cells commented out.
Similarly for md files etc (in fact, all filetypes recognised by Jupytext).
# ---
# jupyter:
# jupytext:
# text_representation:
# extension: .py
# format_name: light
# format_version: '1.4'
# jupytext_version: 1.1.7
# kernelspec:
# display_name: Python 3
# language: python
# name: python3
# ---
# # Hidden Code
#
# This notebook / py file contains code cells that are marked as executable in the notebook but commented out in the `.py` file.
def hello(msg='Hello'):
print(msg)
# + {"active": "ipynb"}
# #This cell is marked with cell metadata: "active": "ipynb"
# msg='active'
# hello(msg)
# #It should run as code in the notebook but not be loaded in as a py module
# -
thanks
–tony
Marc
Trying to thing this through a bit further…
If I have a metadata’d cell that comments out code in a py file but that I can run in the ntoebook editing view, what would be the best way to automate the execution of that code from the command line if the commented out code was part of a test suite?
eg suppose I have foo.py that I’m using as a module, and foo.py has commented out code cells that are actually tests. I notice from this issue that the following commands now exist (presumably py execution etc as wellas md execution works?):
jupytext --to ipynb --execute *.md # create and execute all md files (in the current python env)
jupytext --set-formats md,ipynb --execute *.md # idem, plus pairs the resulting ipynb notebook to the original md files.
So this means I could run the tests via the first command and then view the test results via the ipynb (can the execution be forced to run all cells even if one cell fails earlier on?), or, using the second command:
- run tests from command line;
- check tests via ipynb output;
- edit in fixes into the ipynb file;
- have those saved via pairing into the
py file.
Can I also use Jupytext to force the .py file to execute as a notebook and run through all errors (eg as supported by nbfconvert)?
I note from the issue mentioned above that Jupytext can also be piped to work with nbconvert, eg as per:
jupytext --to ipynb --pipe-fmt ipynb --pipe 'jupyter nbconvert --stdin --stdout --to notebook --execute' script.py
so I guess we can call nbconvert with the allow_errors=True traitlet via that route (for now) at least?
Sure! I’ve not much to add as everything you stated is correct. Let me just comment on a few points:
- When you use the default configuration of Jupytext’s contents manager, you don’t even need to modify explicitely
c.NotebookApp.contents_manager_class. From version 1.0 on, Jupytext includes a server extension that activates Jupytext’s contents manager automatically. That also means that you can drop the postBuild step in Binder for a few branches of your ouseful-template-repos/jupytext-md repository.
- The new
--execute option in Jupytext CLI will be available in 1.2.0. For now it is only available as a release candidate (1.2.0rc0 and 1.2.0rc1). Jupytext does use nbconvert under the hood for the --execute option. There is no plan to map every nbconvert option, so indeed if you want to ignore errors you will have to pipe into nbconvert directly. Here again you will need the latest rc (in versions <1.2, jupytext ignore the output cells after a pipe command).
- Indeed your use case of documenting a module with sample commands (yielding tables and plots I guess) is interesting! Personally I had considered the other way around: including test functions in the notebook, and running pytest over the
.py representation of the notebook…
Marc
Thanks for clarifications. The fact that Jupytext will support text file editing without the need to explicitly create a settings file entry is interesting (if a little dangerous perhaps, if someone does not expect it?).
That said, I guess you can edit the raw file by setting the root URL path element to edit if you want to edit a parsed text file as the raw text file rather than via the notebook UI (maybe useful for that to be noted in FAQ if it isn’t already?).
Re: the use case: I need to try this out for myself, but I think I now see how just by installing Jupytext, I could write a single .py file that:
- is a standalone module that can be imported as a py file into other py files;
- that can be executed to run tests using pytests configured to look for tests in all
.py files; or
- that can be executed to produce a rich vignette (as in R vignettes) that document / show how to use the package.
I’m not sure how to enable running tests in the notebook form so that you can see the test outputs, as distinct from running non-test documentation code cells in the notebook form? (I guess things like papermill allow parameters to be passed in to a notebook which could do the switching, although that would require guards on test vs documentation code cells? Ah - I see you have already started exploring papermill+jupytext workflows.)
Nor am I sure how I’d directly do something like create an HTML file from a .py file executed as a notebook? (Although I can see how I’d do that by creating an output .ipynb file in one step then running a seconding nbconvert step to transform that to HTML.)
Just an aside on the question of editing cell metadata - I note that notebooks have an undocumented(?) ability to make cells non-editable as well as non-deletable which might also be useful when creating documentation notebooks viewed as notebooks, rather than generated as HTML docs, for example.
That said, I guess you can edit the raw file by setting the root URL path element to edit if you want to edit a parsed text file as the raw text file rather than via the notebook UI (maybe useful for that to be noted in FAQ if it isn’t already?).
That is correct. Or you can select the file, and click on edit:
https://jupytext.readthedocs.io/en/latest/using-server.html#how-to-open-scripts-with-either-the-text-or-notebook-view-in-jupyter
Nor am I sure how I’d directly do something like create an HTML file from a .py file executed as a notebook? (Although I can see how I’d do that by creating an output .ipynb file in one step then running a seconding nbconvert step to transform that to HTML.)
Yes, that works in two steps: 1. create the ipynb file, then 2. execute it and convert it to HTML using nbconvert. You could use pipes with jupytext --to py -o - script.py | ...
Just an aside on the question of editing cell metadata - I note that notebooks have an undocumented(?) ability to make cells non-editable as well as non-deletable which might also be useful when creating documentation notebooks viewed as notebooks, rather than generated as HTML docs, for example.
Interesting. By the way, Jupytext also supports frozen cells, but that’s of less interest to you since these cells are inactive in both the .py and ipynb representation.