There seems to be a generic problem with recent versions of jupyter/notebook on Apple Silicon using homebrew python.
TL;DR 1: Under homebrew python on Apple Silicon, pip is installing some files into /opt/homebrew/{share,etc}/jupyter but the apps are not searching for them there.
TL;DR 2: are any Apple Silicon homebrew python users able to use recent versions of jupyter/notebook out of the box?
Less comprehensive versions of this have been posted before: (here, here, here, as well as this notebook issue, this jupyterlab issue, and this PR) but I wanted to consolidate these, add new information, and try to get a better idea of how pervasive the problem might be.
There seems to be a set of several interlinked issues relating to where different jupyter apps both store and search for various data and config files, which show up when the following conditions all hold:
- running homebrew python
- directly, not in a virtual environment
- with jupyter and related packages installed by pip
- on an Apple Silicon mac (M1 or M2)
- with various issues only appearing with notebook versions ≥ 7 and more recent versions of the various jupyterlab packages. (In particular, running notebook v6.x without lab worked fine.)
The symptoms of the problem are:
- In
jupyterlab:- The “Open in…” menu (for notebook or nbclassic) does not appear
- The “Installed” extension list is incomplete and, in particular, there is no sidebar entry for
ipyapralleleven when installed
- In
notebook:- Error messages related to json schema in the jupyter console (originally reported here )
- The “File” menu ends at “Save and Export Notebook As…” (i.e., “Log Out” and “Shut Down” do not appear)
- Ipyparallel inaccessible even when installed
(There are also occasional cache issues when starting jupyter, but it’s not clear if this is related.)
The problems arise because homebrew on Apple Silicon puts its files below /opt/homebrew/ (rather than, say, /usr/local).
Most of these problems can be ameliorated by setting
JUPYTER_PATH=/opt/homebrew/share/jupyter JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter
orJUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter JUPYTER_PLATFORM_DIRS=1
The problem seems to arise mostly because jupyter --paths gives the following:
config:
/Users/jaffe/.jupyter
/Users/jaffe/Library/Python/3.11/etc/jupyter
/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/etc/jupyter
/usr/local/etc/jupyter
/etc/jupyter
data:
/Users/jaffe/Library/Jupyter
/Users/jaffe/Library/Python/3.11/share/jupyter
/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/share/jupyter
/usr/local/share/jupyter
/usr/share/jupyter
runtime:
/Users/jaffe/Library/Jupyter/runtime
In particular, in this setup, /opt/homebrew/etc/jupyter and /opt/homebrew/share/jupyter should probably appear, but do not, which is why the failures happen.
There must be a mis-match somewhere: some files are being installed into /opt/homebrew/{share,etc}/jupyter but not searched for there.
Other notes:
- The “schemas” error messages seem somewhat separate – they are not fixed by setting the environment variables. (See here)
JUPYTER_PLATFORM_DIRS=1(which will effectively become the default IIUC) does not quite fix the problem: it does not search for config files in/opt/homebrew/etc/jupyter.- The problem does not appear when running in a venv; presumably the venv overrides the path settings (although, weirdly, it does seem to appear with
.venv/bin/python -m jupyter notebook)
Is anyone else seeing these issues? (I am pretty sure that answer is yes.) Is anyone using this setup not seeing these issues? Or not seeing all of them?