So I came across two recommendations – either set SHELL under /etc/environment, or create a site-wide jupyter_notebook_config.py under /etc/jupyter (this issue pointing to the JupyterHub docs was particularly helpful, thanks!).
The first suggestion – modifying /etc/environment – did not work, but what did work was setting Environment=SHELL=/usr/bin/fish in the systemd service file I’ve set up to launch JupyterHub. So it seems setting the SHELL env var indeed does it, it’s just that it’s not picked up from /etc/environment and has to be set somewhere else.
The second suggestion worked as advertised, so just for reference or anyone else trying to do this, since it’s independent of the way you launch JupyterHub – just put this in your /etc/jupyter/jupyter_notebook_config.py:
c.NotebookApp.terminado_settings = { "shell_command": ["/usr/bin/fish"] }
This variant is also more flexible, since you can provide additional arguments to the shell command, e.g. forcing a login shell.