Unexpected working directory when running `ipynb` notebook through `nbconvert`

Notebook nbconvert
,
1

I’ve written an ipynb notebook in vs code. It resides in a pwd/scripts/notebook.ipynb folder. Within the notebook I access data in the pwd/data/data.csv folder.

This works within vs code.

When I now try to run the notebook using nbconvert as follows:

jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --inplace --execute scripts/hierarchical_numpyro.ipynb

I get an error

FileNotFoundError: [Errno 2] No such file or directory: 'data/data.csv'

It looks like the working directory of the notebook execution through nbconvert is the directory the notebook is in. This is unfortunate.

Is there a way to change the working directory other than requiring me to move the file to the root of the project?

I’d expect the working directory to be the directory I ran the command in, not the directory of the file.

2

In general, in the Jupyter ecosystem the working directory in relation to the notebook’s namespace defaults to the directory in which the notebook resides. That you can apparently(?) override this in VScode is leading you to confusion now. To illustrate the default directory for a notebook is where it resides, go to any launch from Try Jupyter page and place a notebook in a newly generated subdirectory and open it and put pwd in a cell.

Alas, you can easily add a magics command to your notebooks that sets the current working directory for the notebook to match your VScode settings. One solution is to put a cell at the top of your hierarchical_numpyro.ipynb notebook that has the following:

import os
if os.getcwd() != 'pwd':
   %cd ..

Where pwd is whatever is actually string equivalent to pwd.
Alternatively, you could just put a cell at the top of your hierarchical_numpyro.ipynb notebook with %cd followed by the absolute path of pwd. That way the notebook independent settings and those of VScode would always match.

You can learn more about the %cd magic command here.

(I will point out that the behavior you report was an issue for PyCharm a while back and you can see similar suggestions here.)

Note that you cannot supply --notebook-dir setting when calling jupyter nbconvert like you can with other jupyter commands illustrated here. Also, I didn’t find a setting in the config file that running jupyter nbconvert --generate-config generates to allow a setting for this.

1 Like