# Jupyterhub-singleuser and jupyter\_server

**URL:** <https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942>\
**Category:** JupyterHub\
**Created:** [September 23, 2021, 3:00pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942 "2021-09-23T15:00:45Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![albertmichaelj](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@albertmichaelj](https://discourse.jupyter.org/u/albertmichaelj)\
**Post date:** [September 23, 2021, 3:00pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/1 "2021-09-23T15:00:45Z")

</div>

Hi everyone,

I’ve got a question that is more about trying to understand what’s happening instead of asking for support. I have a JupyterHub deployment on K8 using the helm chart (with a significant number of my own modifications), and I’ve been fighting to allow users to delete non-empty directories for a while (it is related to a few long standing github issues, but if you want to go down the rabbit trail you can start [here](https://github.com/jupyterlab/jupyterlab/issues/835) or [here](https://github.com/jupyter/notebook/issues/4916)). Last year with my annual academic cycle deployment, I ended up just patching the `filemanager.py` file from the notebook app to allow for deleting non-empty directories, but it is a bit of a kludge to manually patch the file.

This year, when I was creating my single user docker container, I realized that my old patch didn’t work when running Jupyter Lab locally due to JupyterLab 3.0 having switched to jupyter\_server as the notebook backend instead of the notebook app. Moreover, I discovered that jupyter\_server had two configurables that when set allowed deleting directories (`--FileContentsManager.always_delete_dir=True --FileContentsManager.delete_to_trash=False`)!

However, when I then set up my JupyterHub k8 deployment with my new single user docker image, those configurables didn’t work, and I had to revert to patching filemanager.py for the **notebook app** , not jupyter\_server.

So, that was the long winded way to get to my real question: Why doesn’t JupyterLab when launched from JupyterHub use the jupyter\_server backend and instead uses the notebook app backend? I know that when launching the single user server the “jupyterhub-singleuser” command is ultimately used to launch the server (and the default url needs to be set to “/lab” in order to actually use the Jupyter Lab interface), and it seems that this command makes JupyterLab use the notebook app backend instead of the jupyter\_server backend (which is the official backend for JupyterLab 3.\*).

So, why does jupyterhub-singleuser work this way? I can find no explanation, and I’m having a really hard time tracking down what exactly that command calls even. I don’t know if this is expected behavior, so I haven’t opened an issue with the JupyterHub repo yet, but if this is not intended behavior, I’m happy to do so. It does seem, since JupyterLab is moving to jupyter\_server and (I think) there are plans to sunset the notebook app, at some point JupyterHub will need to also use the jupyter\_server backend.

I’d appreciate any insight anyone has on this!

---

<div class="post-metadata">

**Author:** ![agoose77](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/agoose77/32/3455_2.png) [@agoose77](https://discourse.jupyter.org/u/agoose77)\
**Post date:** [September 23, 2021, 3:18pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/2 "2021-09-23T15:18:08Z")

</div>

The `singleuser` module in `jupyterhub` [allows the user (developer) to override the Application](https://github.com/jupyterhub/jupyterhub/blob/3674ada640e88d0ce3f678e69e6f7b4a0a10ffde/jupyterhub/singleuser/app.py) launched by the single-user image. By default, it is `"notebook.notebookapp.NotebookApp"`

You can change this by setting the environment variable `JUPYTERHUB_SINGLEUSER_APP='jupyter_server.serverapp.ServerApp'`

---

<div class="post-metadata">

**Author:** ![albertmichaelj](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@albertmichaelj](https://discourse.jupyter.org/u/albertmichaelj)\
**Post date:** [September 23, 2021, 3:19pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/3 "2021-09-23T15:19:44Z")

</div>

Ah, gotcha! That’s very helpful. There even looks like the app can be configured using the `JUPYTERHUB_SINGLEUSER_APP` environmental variable (edit: got ninja’d by an edit). Is there any downside to switching the single user app to jupyter\_server?

---

<div class="post-metadata">

**Author:** ![agoose77](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/agoose77/32/3455_2.png) [@agoose77](https://discourse.jupyter.org/u/agoose77)\
**Post date:** [September 23, 2021, 3:23pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/4 "2021-09-23T15:23:55Z")

</div>

There _shouldn’t_ be any issues for most deployments. There is a shim that copies any `NotebookApp` configuration to the `ServerApp`, so if you have Jupyter Notebook installed (via `nbclassic`), that configuration should still be available.

I’ve been using it in deployment for a while, and no issues.

---

<div class="post-metadata">

**Author:** ![albertmichaelj](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@albertmichaelj](https://discourse.jupyter.org/u/albertmichaelj)\
**Post date:** [September 23, 2021, 3:33pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/5 "2021-09-23T15:33:20Z")

</div>

Gotcha. Do you need `defaultURL: "/lab"` anymore if you switch backends but don’t have nbclassic installed? Will it automatically use JupyterLab?

---

<div class="post-metadata">

**Author:** ![agoose77](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/agoose77/32/3455_2.png) [@agoose77](https://discourse.jupyter.org/u/agoose77)\
**Post date:** [September 23, 2021, 3:35pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/6 "2021-09-23T15:35:42Z")

</div>

I can’t remember. I’d expect you probably need to set it, because the `ServerApp` doesn’t prefer any particular URL (as JupyterLab is now a server extension), and I don’t _think_ that JupyterHub sets a default.

---

<div class="post-metadata">

**Author:** ![albertmichaelj](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@albertmichaelj](https://discourse.jupyter.org/u/albertmichaelj)\
**Post date:** [September 23, 2021, 3:49pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/7 "2021-09-23T15:49:44Z")

</div>

Just tested things, and you do need the default url set. Thanks for all of your help!

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [September 23, 2021, 4:39pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/8 "2021-09-23T16:39:25Z")

</div>

FYI JupyterHub will probably switch to jupyter-server/jupyterlab in the 2.0 release: [2.0: jupyterlab by default by minrk · Pull Request #3615 · jupyterhub/jupyterhub · GitHub](https://github.com/jupyterhub/jupyterhub/pull/3615)

I’m surprised you still needed `defaultURL: "/lab"` though!

---

<div class="post-metadata">

**Author:** ![albertmichaelj](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@albertmichaelj](https://discourse.jupyter.org/u/albertmichaelj)\
**Post date:** [September 23, 2021, 4:57pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/9 "2021-09-23T16:57:28Z")

</div>

Yeah, if I didn’t have the “/lab” url, it just said something like “Server is running” when I logged in (until I manually went to “/lab”). I checked and my environment does have nbclassic installed (the image is based on the official Jupyter Docker images), so I can’t say for certain what happens if it’s not installed. However, I don’t think that will have any effect on what is started, though.

Good to know that it’s switching to jupyter-server by default, though.

---

<div class="post-metadata">

**Author:** ![mriedem](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/mriedem/32/1632_2.png) [@mriedem](https://discourse.jupyter.org/u/mriedem)\
**Post date:** [November 17, 2021, 3:57pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/10 "2021-11-17T15:57:11Z")

</div>

Is there a definitive way to tell if you’re using the legacy notebook app at all? We’re using jupyterhub 1.5.0 and z2jh 1.2.0 with a docker-stacks/scipy-notebook parent image for our singleuser image. We’re calling `jupyter-labhub` to start the singleuser app. With kubespawner it’s a bit confusing because the container is hard-coded to be named `notebook` and you see things in the `notebook` container logs saying notebook (it’s got `nbclassic` and `notebook` installed in the image) and we always see this:

> Nov 17 09:22:58 jupyter-6192b1ecab4f10c6670c9e5b notebook INFO INFO 2021-11-17T15:22:58.080Z [SingleUserLabApp serverapp:2577] Jupyter Server 1.11.2 is running at:

We just started setting this yesterday in our z2jh hub config:

> JUPYTERHUB\_SINGLEUSER\_APP: “jupyter\_server.serverapp.ServerApp”

But that’s currently only in one of our testing clusters and I’m trying to compare the notebook logs and REST API between those environments and it’s hard to see the difference, e.g. `GET /user/<userid>/api/` returns the jupyter-server version in both cases, not the notebook app web server version.

I’m thinking the thing that I can use to tell is the _absence_ of this log message:

> Nov 11 11:29:24 jupyter-5e18a4193f4a3f001127f809 notebook [I 2021-11-11 17:29:24.785 SingleUserLabApp notebookapp:2302] Jupyter Notebook 6.4.5 is running at:

Is there something better?

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [November 18, 2021, 8:23am UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/11 "2021-11-18T08:23:09Z")

</div>

When you start with jupyter-server (usually via setting JUPYTERHUB\_SINGLEUSER\_APP or `Spawner.cmd = "jupyter-labhub"`), you will see the log line:

```auto
[I 2021-11-18 09:18:46.022 SingleUserNotebookApp serverapp:2577] Jupyter Server 1.11.2 is running at:

```

When you start the classic notebook server, it will look like:

```auto
[I 2021-11-18 09:19:01.888 SingleUserNotebookApp notebookapp:2329] Jupyter Notebook 6.4.6 is running at:

```

So both log the same `{name of server} {version} running at` line identifying which server implementation is used. If you see that “Jupyter Server” log line, it is using Jupyter server and not the classic notebook server.

JupyterHub 2.0 will log the inheritance of the singeluser app more explicitly:

```auto
[I 2021-11-08 09:17:37.223 SingleUserLabApp mixins:617] Starting jupyterhub single-user server version 2.0.0rc3
[I 2021-11-08 09:17:37.223 SingleUserLabApp mixins:629] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 3.2.1
[I 2021-11-08 09:17:37.223 SingleUserLabApp mixins:629] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.9.0

```

---

<div class="post-metadata">

**Author:** ![mriedem](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/mriedem/32/1632_2.png) [@mriedem](https://discourse.jupyter.org/u/mriedem)\
**Post date:** [November 18, 2021, 3:51pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/12 "2021-11-18T15:51:51Z")

</div>

> [@minrk](#):
>
> When you start with jupyter-server (usually via setting JUPYTERHUB\_SINGLEUSER\_APP or `Spawner.cmd = "jupyter-labhub"` ), you will see the log line:

We have a wrapper script around the `Spawner.cmd` but it eventually calls `jupyter-labhub`. We’re always seeing the `Jupyter Server 1.11.2 is running at` line so that’s why I was confused about what was being used.

Anyway, I _think_ we’re doing all of the correct things now and I’ll assume that if I don’t see `Jupyter Notebook * is running` then we’re OK.

This was all originated from my creating a `jupyter-server` issue and being told I was using `notebook`:

> <https://github.com/jupyter-server/jupyter_server/issues/609>
>
> \## Description
> 
> I'm doing some stress and scale testing on a testing environme…nt jupyterhub deployment using zero-to-jupyterhub-k8s 1.2.0 with jupyterhub 1.5.0, kubespawner 1.1.2 and jupyter-server 1.11.2:
> 
> \`\`\`
> \>\>\> print(jupyter\_server.\_\_version\_\_)
> 1.11.2
> \`\`\`
> 
> We build our own user image based on \`jupyter/scipy-notebook:837c0c870545\` from docker-stacks and then with some additional extensions and updates (like jupyterhub 1.5.0).
> 
> I was running a scale up test using \[hub-stress-test\](https://github.com/IBM/jupyter-tools) to 1000 singleuser-server pods (these are basically micro pods just to stress the hub and proxy \[core pods\], we care less about the actual notebook servers \_doing\_ anything).
> 
> In 1 out of 1000 notebook servers there was a failure to spawn. In the hub log I noticed this:
> 
> \`\`\`
> Nov 10 14:57:09 hub-85fcbd49b8-p52dt hub ERROR ERROR 2021-11-10T20:57:09.759Z \[JupyterHub gen:623\] Exception in Future \<Task finished name='Task-33261' coro=\<BaseHandler.spawn\_single\_user.\<locals\>.finish\_user\_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:900\> exception=TimeoutError("Server at http://172.30.102.60:8888/user/hub-stress-test-979/ didn't respond in 30 seconds")\> after timeout
> Traceback (most recent call last):
> File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error\_callback
> future.result()
> File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 907, in finish\_user\_spawn
> await spawn\_future
> File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 748, in spawn
> await self.\_wait\_up(spawner)
> File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 795, in \_wait\_up
> raise e
> File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 762, in \_wait\_up
> resp = await server.wait\_up(
> File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 236, in wait\_for\_http\_server
> re = await exponential\_backoff(
> File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 184, in exponential\_backoff
> raise TimeoutError(fail\_message)
> TimeoutError: Server at http://172.30.102.60:8888/user/hub-stress-test-979/ didn't respond in 30 seconds
> \`\`\`
> 
> Looking in the notebook server pod logs for that pod I see this:
> 
> \`\`\`
> Nov 10 14:56:58 jupyter-hub-stress-test-979 notebook \[C 2021-11-10 20:56:58.977 SingleUserLabApp notebookapp:1972\] received signal 15, stopping
> Nov 10 14:56:58 jupyter-hub-stress-test-979 notebook \[E 2021-11-10 20:56:58.977 LabApp\] Exception while loading config file jupyter\_lab\_config
> Traceback (most recent call last):
> File "/opt/conda/lib/python3.8/genericpath.py", line 30, in isfile
> st = os.stat(path)
> FileNotFoundError: \[Errno 2\] No such file or directory: '/home/jovyan/jupyter\_lab\_config.json'
>     
> During handling of the above exception, another exception occurred:
>     
> Traceback (most recent call last):
> File "/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py", line 738, in \_load\_config\_files
> config = loader.load\_config()
> File "/opt/conda/lib/python3.8/site-packages/traitlets/config/loader.py", line 560, in load\_config
> self.\_find\_file()
> File "/opt/conda/lib/python3.8/site-packages/traitlets/config/loader.py", line 542, in \_find\_file
> self.full\_filename = filefind(self.filename, self.path)
> File "/opt/conda/lib/python3.8/site-packages/traitlets/utils/\_\_init\_\_.py", line 58, in filefind
> if os.path.isfile(testname):
> File "/opt/conda/lib/python3.8/genericpath.py", line 30, in isfile
> st = os.stat(path)
> File "/opt/conda/lib/python3.8/site-packages/notebook/notebookapp.py", line 1973, in \_signal\_stop
> self.io\_loop.add\_callback\_from\_signal(self.io\_loop.stop)
> AttributeError: 'SingleUserLabApp' object has no attribute 'io\_loop' 
> \`\`\`
> 
> I'm not sure what's going on there, maybe a race condition while the pod is starting up? Or maybe because it was taking a long time to start something started failing in weird ways? Here is a gist of the notebook app logs:
> 
> https://gist.github.com/mriedem/384ff1578aca0163e743fe4ddea176a7
> 
> We can see that it's 34 seconds from the time the app is starting up to the time that error happens. I'm wondering if maybe we hit this and the hub/kubespawner killed the pod?
> 
> https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.http\_timeout
> 
> !\[image\](https://user-images.githubusercontent.com/3237936/141196219-b1271ecb-f71a-4a53-bdb4-12ed9c49eed8.png)
> 
> Actually digging more into the hub logs it looks like yes the hub hits that \`http\_timeout\` and then kills the pod, here are the hub logs scoped for that pod:
> 
> https://gist.github.com/mriedem/8d226d7f934ae27e71a35ef6ba8a13ec
> 
> So I guess the issue is just there is an ugly AttributeError in the notebook app / jupyter-server logs when there is a race on startup where the pod is killed during startup and things aren't all setup. I guess that's probably hard to predict and account for though so probably a low priority issue to resolve.
> 
> \## Reproduce
> 
> Hard to reproduce. I've been running scale tests all day and have hit maybe a couple of the http\_timeouts in the hub logs but only this one AttributeError in the notebook server app logs.
> 
> \## Expected behavior
> 
> Not to see AttributeError red herring-like errors in the app logs.
> 
> \## Context
> 
> \- Operating System and version: ubuntu focal
> \- Browser and version: n/a
> \- Jupyter Server version: 1.11.2

Which, to be fair, looks like we at least were because of the `notebookapp.py` in the stacktrace in that issue.

---

<div class="post-metadata">

**Author:** ![adhyapakh](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/adhyapakh/32/4907_2.png) [@adhyapakh](https://discourse.jupyter.org/u/adhyapakh)\
**Post date:** [April 23, 2022, 1:19pm UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/13 "2022-04-23T13:19:29Z")

</div>

In which file should I pass this parameter to?

---

<div class="post-metadata">

**Author:** ![agoose77](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/agoose77/32/3455_2.png) [@agoose77](https://discourse.jupyter.org/u/agoose77)\
**Post date:** [April 27, 2022, 7:42am UTC](https://discourse.jupyter.org/t/jupyterhub-singleuser-and-jupyter-server/10942/14 "2022-04-27T07:42:42Z")

</div>

If you’re using JupyterHub\>=2.0, you should no longer need to pass this parameter, as `jupyter_server` is the default server.