How can I run a note book on a remote workstation where I will
type “screen” to enter the screen mode so that I can run the notebook from
the command line and then I detach the screen so that I can shutdown my local
Linux machine. Say the job finishes in two days. Then from my local computer I can
ssh to the remote workstation and to use screen to check the job is indeed finished and
a nice .ipynb with all output in it.
In other words.
I would like to keep all plots stored in ipynb
during run time to be save in the .pynb just like I run
it interactively in a local workstation.
I know I can convert ipynb to .py and run py from command line but then
the results are not stored as nice in the ipynb.
jupytext - The ability to use it to directly execute notebooks that are in normal notebook format previously was more clearly featured in the documentation of the Jupytext command line examples; however, --execute option is noted there now & if you use jupytext --execute notebook.ipynb it runs the notebook.
Papermill - on top of the ability to execute notebooks, Papermill adds the ability to parameterize notebooks.
Ploomber - This note briefly summarizes where and how ploomber is meant to be leaveraged.
Papermill supposedly has one advantage for long running notebooks: I’ve seen it said that it saves on a per cell basis and that offers more possibilities to monitor progress. Whereas, jupyter nbconvert --exectute doesn’t save the output in the executed notebook file until completion. I haven’t tested this myself yet, and I don’t know what Jupytext does along these lines.
Sorry I am too thick to understand this. Is this jupytext newer implementation and it can
do what nbconvert cannot do? If it does not offer anything new then I will just ignore this.
Yes it does. However, you can add the --inplace option to specify it to overwrite the original notebook with the executed one. Type jupyter nbconvert --help-all to get an exhaustive list of options, or type jupyter nbconver alone to get information on the most typical options.
By default, without specifying --output or -o, jupytext will execute in place. Run jupytext -h to see a list of options.
Jupytext is a separate package. You have to install it with Anaconda/conda or pip beyond what comes with Jupyter by default. Same as the other options I mentioned aside from jupyter nbconvert. Jupytext is mainly for converting notebooks to text based form and back again., see here to learn more about it. It does more than jupyter nbconvert does, for example, it can make a paired notebook where it saves a text form of a notebook whenever the .ipynb file gets updated. However, right now you seem satisfied with only needing to run notebooks from the command line and so jupyter nbconvert may be all you need for now.
Thanks for comments. There are just too many things happening simultaneously, so I would rather go slow.
I am more and more convinced that editing a any.py and then run it through a one-line any.ipynb that contains %run any.py is the cure to all my miseries associated with not having my
power tool vi to edit the source (moving through cells in .ipynb drain my energy and time very quickly!)
Now, I would to even explore not having any.ipynb and just run a command line that takes any.py
and generate results in any.ipynb or any.nbconvert.ipynb.
Using text placed inside a Python script to later generate a separate notebook that contains the results as a report-style artifact is something I commonly do.
Examples you can look over to maybe spark more ideas:
here (You can see it in action by going to here, clicking on the top instance of ‘launch bend.it’ and running the notebook that comes up. You’ll see it executes ‘bendIt_analysis.py’ in the course of running and that among the output produced are notebooks made from text in the script.)
here (You can see it in action by going to here, clicking ‘launch binder’ and selecting the notebook ’ Using snakemake to highlight changes in multiple protein-protein interactions via PDBsum data’ from the list to run.)
Besides using jupytext to generate notebook from markdown or python code inside a Python script and also executing that at the same time, another utility to be aware of in this type of work is nbformat. You can make a notebook programmatically from the individual cell code. Here is an example using nbformat to build a notebook when I don’t know before hand exactly how many cells it will have. You can see that in action by going here and clicking launch binder and running all the cells in the notebook that comes up in the session. It will make a notebook with images inside code cells to be used in a slideshow later.
Parallel to this, I’d suggest you are seeking jupytext’s ability to edit text files and have them be paired notebooks if you are looking to use vi to edit the equivalent of the source code that would be your notebook. In particular for quickly throwing together a notebook for development.