Notebook 7 + Jupyterlab 4 + Apple Silicon + Homebrew + pip — errors

There seems to be a generic problem with recent versions of jupyter/notebook on Apple Silicon using homebrew python.

TL;DR 1: Under homebrew python on Apple Silicon, pip is installing some files into /opt/homebrew/{share,etc}/jupyter but the apps are not searching for them there.

TL;DR 2: are any Apple Silicon homebrew python users able to use recent versions of jupyter/notebook out of the box?

Less comprehensive versions of this have been posted before: (here, here, here, as well as this notebook issue, this jupyterlab issue, and this PR) but I wanted to consolidate these, add new information, and try to get a better idea of how pervasive the problem might be.

There seems to be a set of several interlinked issues relating to where different jupyter apps both store and search for various data and config files, which show up when the following conditions all hold:

  • running homebrew python
  • directly, not in a virtual environment
  • with jupyter and related packages installed by pip
  • on an Apple Silicon mac (M1 or M2)
  • with various issues only appearing with notebook versions ≥ 7 and more recent versions of the various jupyterlab packages. (In particular, running notebook v6.x without lab worked fine.)

The symptoms of the problem are:

  • In jupyterlab:
    • The “Open in…” menu (for notebook or nbclassic) does not appear
    • The “Installed” extension list is incomplete and, in particular, there is no sidebar entry for ipyaprallel even when installed
  • In notebook:
    • Error messages related to json schema in the jupyter console (originally reported here )
    • The “File” menu ends at “Save and Export Notebook As…” (i.e., “Log Out” and “Shut Down” do not appear)
    • Ipyparallel inaccessible even when installed

(There are also occasional cache issues when starting jupyter, but it’s not clear if this is related.)

The problems arise because homebrew on Apple Silicon puts its files below /opt/homebrew/ (rather than, say, /usr/local).

Most of these problems can be ameliorated by setting

  • JUPYTER_PATH=/opt/homebrew/share/jupyter JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter
  • JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter JUPYTER_PLATFORM_DIRS=1

The problem seems to arise mostly because jupyter --paths gives the following:


In particular, in this setup, /opt/homebrew/etc/jupyter and /opt/homebrew/share/jupyter should probably appear, but do not, which is why the failures happen.

There must be a mis-match somewhere: some files are being installed into /opt/homebrew/{share,etc}/jupyter but not searched for there.

Other notes:

  • The “schemas” error messages seem somewhat separate – they are not fixed by setting the environment variables. (See here)
  • JUPYTER_PLATFORM_DIRS=1 (which will effectively become the default IIUC) does not quite fix the problem: it does not search for config files in /opt/homebrew/etc/jupyter.
  • The problem does not appear when running in a venv; presumably the venv overrides the path settings (although, weirdly, it does seem to appear with .venv/bin/python -m jupyter notebook)

Is anyone else seeing these issues? (I am pretty sure that answer is yes.) Is anyone using this setup not seeing these issues? Or not seeing all of them?

1 Like

Just pinging this topic one (last?) time… I suppose the lack of response implies that this is really not an issue for many other users.

But it would still be useful to check whether there is anyone else using this setup (Apple Silcon + homebrew + pip) not seeing this set of issues. (I.e., is this just a rare use-case, or is there something else specific to those of us who are seeing them?

FYI. I do see the issue in my setup as I have mentioned on GitHub discussion.

FWIW, and as expected, all of these errors and misbehaviours vanish for the framework build (of 3.12, but I assume more generally), without requiring any environment variables or moving/linking files or directories. This still lets me install packages by pip and not be wedded to using a venv, so I think I am moving back to this infrastructure from homebrew, at least for now.