Unable to open any notebooks -- File Not Found error (2)

Notebook
1

I’m trying to open existing jupyter notebooks using a freshly installed jupyter from conda. I am doing this on MacOS Catalina (this may be involved).

When I try to open any of the notebooks in the directory from the Jupyter Home Page, I get this error:

Unreadable Notebook: /Users/rpg/projects/xplan/xplan-dev-env/xplan-experiment-analysis/MyNotebook.ipynb FileNotFoundError(2, 'No such file or directory')

The files are definitely there – I can paste the filenames into the terminal and ls them, and they are there.

The answer:

Conda environments, unlike standard virtual environments, do not shadow the user’s native configuration. So when I was in the conda environment that I thought was all set up correctly:

  1. python was loading packages from ~/.local instead of from my conda environment and
  2. jupyter was finding a global configuration that was wildly inappropriate to the virtual environment at hand.

So this is really a conda issue, not a jupyter one. But on the off-chance that anyone else gets bitten by this, the answer is to configure your conda environment so that the environment variable $PYTHONNOUSERSITE is “True” (or really any value the shell sees as true), and JUPYTER_PATH is bound to the configuration in your conda environment, so jupyter isn’t decoyed into looking in ~/.local or goodness only knows where else.

The whole MacOS thing turns out to be a big red herring.

Old stuff, no longer of interest

The logs dumped to the window show this:

[W 17:24:35.865 NotebookApp] 400 GET /api/contents/MyNotebookipynb?type=notebook&_=1610666675658 (127.0.0.1): Unreadable Notebook: /Users/rpg/projects/xplan/xplan-dev-env/xplan-experiment-analysis/notebooks/MyNotebook.ipynb FileNotFoundError(2, 'No such file or directory')
[W 17:24:35.866 NotebookApp] Unreadable Notebook: /Users/rpg/projects/xplan/xplan-dev-env/xplan-experiment-analysis/notebooks/MyNotebook.ipynb FileNotFoundError(2, 'No such file or directory')
[W 17:24:35.866 NotebookApp] 400 GET /api/contents/MyNotebook.ipynb?type=notebook&_=1610666675658 (127.0.0.1) 8.190000ms referer=http://localhost:8888/notebooks/MyNotebook.ipynb

One thing I wondered was whether this could be related to Apple’s new, more aggressive access controls. Is there any chance that this means not that the file isn’t there, but that it isn’t readable by jupyter? And if so, does anyone know how to fix this?

Alternatively, is there any way to get more robust logging or – even better – get myself into pdb so that I can see what is going wrong here?

1 Like
2

If it helps, installing jupyter from pip opens these notebooks just fine. It’s a conda-installed jupyter in a special environment that can’t.

If I use jupyter in my conda base environment, everything is fine.

3

Yet more information: in a different conda environment, I tried to open this file and got this traceback:

Traceback (most recent call last):
  File "/Users/rpg/.local/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute
    result = await result
  File "/Users/rpg/.local/lib/python3.8/site-packages/tornado/gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/Users/rpg/.local/lib/python3.8/site-packages/notebook/services/sessions/handlers.py", line 69, in post
    model = yield maybe_future(
  File "/Users/rpg/.local/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/Users/rpg/.local/lib/python3.8/site-packages/tornado/gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/Users/rpg/.local/lib/python3.8/site-packages/notebook/services/sessions/sessionmanager.py", line 88, in create_session
    kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
  File "/Users/rpg/.local/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/Users/rpg/.local/lib/python3.8/site-packages/tornado/gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/Users/rpg/.local/lib/python3.8/site-packages/notebook/services/sessions/sessionmanager.py", line 100, in start_kernel_for_session
    kernel_id = yield maybe_future(
  File "/Users/rpg/.local/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/Users/rpg/.local/lib/python3.8/site-packages/notebook/services/kernels/kernelmanager.py", line 176, in start_kernel
    kernel_id = await maybe_future(self.pinned_superclass.start_kernel(self, **kwargs))
  File "/Users/rpg/.local/lib/python3.8/site-packages/jupyter_client/multikernelmanager.py", line 185, in start_kernel
    km.start_kernel(**kwargs)
  File "/Users/rpg/.local/lib/python3.8/site-packages/jupyter_client/manager.py", line 313, in start_kernel
    self.kernel = self._launch_kernel(kernel_cmd, **kw)
  File "/Users/rpg/.local/lib/python3.8/site-packages/jupyter_client/manager.py", line 222, in _launch_kernel
    return launch_kernel(kernel_cmd, **kw)
  File "/Users/rpg/.local/lib/python3.8/site-packages/jupyter_client/launcher.py", line 134, in launch_kernel
    proc = Popen(cmd, **kwargs)
  File "/Users/rpg/opt/anaconda3/envs/pymc3-dev-py38/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/rpg/opt/anaconda3/envs/pymc3-dev-py38/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/rpg/opt/anaconda3/envs/xplan-experiment-analysis/bin/python'

I should note that the file not found error for python here, refers to a virtual environment – xplan-experiment-analysis` that no longer exists.

Somehow this is tied into the invocation of /Users/rpg/.local/lib/python3.8/site-packages/tornado/web.py which … why should anything be looking for packages in ~/.local instead of in ~/opt/anaconda3/envs/<my-env>/bin?

Even if so, where is the reference to an obsolete python install being stashed? Somewhere inside jupyter’s environments?

I’m starting to feel like the XKCD cartoon about my python installation being a superfund site…