Jupyterhub fails to spawn, singleuser logs indicate a read permission error

JupyterHub
1

Jupyterhub fails to spawn with the following errors:

Sep 06 06:02:38 python3[553254]: [I 2024-09-06 06:02:38.149 JupyterHub pages:399] app_jupyterhub is pending spawn
Sep 06 06:02:38 python3[553254]: [W 2024-09-06 06:02:38.329 JupyterHub _xsrf_utils:195] Skipping XSRF check for insecure request GET /hub/api/users/app_jupyterhub/server/progress
Sep 06 06:02:47 python3[553254]: [W 2024-09-06 06:02:47.025 JupyterHub base:1190] User app_jupyterhub is slow to start (timeout=10)
Sep 06 06:02:55 python3[553254]: [I 2024-09-06 06:02:55.099 JupyterHub log:192] 200 GET /hub/api/ (cull-idle@127.0.0.1) 17.02ms
Sep 06 06:02:55 python3[553254]: [I 2024-09-06 06:02:55.108 JupyterHub log:192] 200 GET /hub/api/users?state=[secret] (cull-idle@127.0.0.1) 6.15ms
Sep 06 06:03:37 python3[553254]: [W 2024-09-06 06:03:37.147 JupyterHub user:879] app_jupyterhub’s server failed to start in 60 seconds, giving up.
Sep 06 06:03:37 python3[553254]:
Sep 06 06:03:37 python3[553254]: Common causes of this timeout, and debugging tips:
Sep 06 06:03:37 python3[553254]:
Sep 06 06:03:37 python3[553254]: 1. Everything is working, but it took too long.
Sep 06 06:03:37 python3[553254]: To fix: increase Spawner.start_timeout configuration
Sep 06 06:03:37 python3[553254]: to a number of seconds that is enough for spawners to finish starting.
Sep 06 06:03:37 python3[553254]: 2. The server didn’t finish starting,
Sep 06 06:03:37 python3[553254]: or it crashed due to a configuration issue.
Sep 06 06:03:37 python3[553254]: Check the single-user server’s logs for hints at what needs fixing.
Sep 06 06:03:37 python3[553254]:
Sep 06 06:03:37 python3[553254]: [E 2024-09-06 06:03:37.188 JupyterHub gen:629] Exception in Future <Task finished name=‘Task-1269’ coro=<BaseHandler.spawn_single_user..finish_user_spawn() done, defined at /opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/handlers/base.py:1081> exception=TimeoutError(‘Timeout’)> after timeout
Sep 06 06:03:37 python3[553254]: Traceback (most recent call last):
Sep 06 06:03:37 python3[553254]: File “/opt/tljh/hub/lib/python3.8/site-packages/tornado/gen.py”, line 624, in error_callback
Sep 06 06:03:37 python3[553254]: future.result()
Sep 06 06:03:37 python3[553254]: File “/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/handlers/base.py”, line 1088, in finish_user_spawn
Sep 06 06:03:37 python3[553254]: await spawn_future
Sep 06 06:03:37 python3[553254]: File “/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/user.py”, line 905, in spawn
Sep 06 06:03:37 python3[553254]: raise e
Sep 06 06:03:37 python3[553254]: File “/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/user.py”, line 801, in spawn
Sep 06 06:03:37 python3[553254]: url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
Sep 06 06:03:37 python3[553254]: asyncio.exceptions.TimeoutError: Timeout
Sep 06 06:03:37 python3[553254]:

If I look at the singleuser logs, I see the following:

Sep 06 05:13:28 systemd[1]: jupyter-app_jupyterhub.service: Failed to load environment files: No such file or directory
Sep 06 05:13:28 systemd[1]: jupyter-app_jupyterhub.service: Failed to run ‘start’ task: No such file or directory
Sep 06 05:13:28 systemd[1]: jupyter-app_jupyterhub.service: Failed with result ‘resources’.
Sep 06 05:13:28 systemd[1]: Failed to start /opt/tljh/user/bin/jupyterhub-singleuser.
Sep 06 05:13:52 systemd[1]: jupyter-app_jupyterhub.service: Failed to open /run/systemd/transient/jupyter-app_jupyterhub.service: No such >
Sep 06 05:13:52 systemd[1]: Started /opt/tljh/user/bin/jupyterhub-singleuser.
Sep 06 05:13:52 systemd[545647]: jupyter-app_jupyterhub.service: Changing to the requested working directory failed: Permission denied
Sep 06 05:13:52 systemd[545647]: jupyter-app_jupyterhub.service: Failed at step CHDIR spawning /opt/tljh/user/bin/jupyterhub-singleuser: P>
Sep 06 05:13:52 systemd[1]: jupyter-app_jupyterhub.service: Main process exited, code=exited, status=200/CHDIR
Sep 06 05:13:52 systemd[1]: jupyter-app_jupyterhub.service: Failed with result ‘exit-code’.

Blockquote

Not sure what files it is unable to find. I have removed /opt/tljh and reinstalled.

2

Can you give us enough information to reproduce the problem? For example, what operating system are you using, is it a standard OS or have you customised it, how did you install and configure TLJH?

3

I am running Jupyterhub on my own Ubuntu server. I am following the steps described here: Installing on your own server — The Littlest JupyterHub documentation

The only custom change beyond what is described in the link above is to install some additional python modules. Was getting an error about missing jupyter-core earlier, so ran this: (this was from a github issue where someone else was running into a similar problem)
sudo /opt/tljh/user/bin/python3 -m pip install notebook

I get the same error on both Ubuntu 20.04 and 22.04.

Thanks,
Ashutosh

4

Do you get the same error if you install TLJH on a clean system, without your custom changes?

5

Hi @manics - apologies for the delay in responding. On a clean system where I am only running the following commands:

sudo apt install python3 python3-dev git curl
curl -L The Littlest Jupyterhub | sudo -E python3 - --admin app_jupyterhub

I see the following error in the single user journalctl logs:
ModuleNotFoundError: No module named ‘jupyter_core’

Per the github issue ModuleNotFoundError: No module named 'jupyter_core' · Issue #965 · jupyterhub/the-littlest-jupyterhub · GitHub the fix for this is to run
sudo /opt/tljh/user/bin/python3 -m pip install notebook

This is the only custom command I run. After doing this, I am now able to get Jupyterhub to run successfully on one machine. But this not the one that I actually want to run it on
On the intended machine, I get the CHDIR permissions error in my original post.

I have cleaned my install by recursively removing /opt/tljh and rebooting. Are there any other directories that need to be removed for a clean reinstall of tljh?

6

You might have some user home directories left behind.