Hello,
I am experiencing an issue with JupyterHub on my Ubuntu server, which is configured for multiple users.
Initially, the server was managed with a single admin account (server_admin
) that has sudo privileges. I installed Anaconda under this account and set up a Jupyter Notebook server on port 8089
. All users accessed the Jupyter Notebook server through this account, which, for security reasons, was not an ideal configuration. To improve this, I set up JupyterHub to support multi-user access.
My JupyterHub Setup
From the server_admin
account (with the base Conda environment activated), I followed the JupyterHub tutorial:
conda install -c conda-forge jupyterhub
conda install jupyterlab notebook
Here is the configuration file /etc/jupyterhub/jupyterhub_config.py
:
c.JupyterHub.authenticator_class = 'jupyterhub.auth.PAMAuthenticator'
c.JupyterHub.extra_log_file = '/var/log/jupyterhub.log'
c.JupyterHub.ip = '=========' # Hidden for security reasons
c.JupyterHub.port = 8080
c.JupyterHub.spawner_class = 'jupyterhub.spawner.LocalProcessSpawner'
c.Spawner.debug = True
c.Spawner.default_url = '/tree'
c.Spawner.http_timeout = 60
c.Spawner.start_timeout = 60
c.Authenticator.admin_users = set(["server_admin"])
c.Authenticator.allow_all = True
c.Authenticator.allow_existing_users = True
I registered a service to run JupyterHub as follows:
[Unit]
Description=JupyterHub
After=syslog.target network.target
[Service]
User=root
Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/server_admin/anaconda3/bin:/home/server_admin/anaconda3/condabin"
ExecStart=/home/server_admin/anaconda3/bin/jupyterhub -f /etc/jupyterhub/jupyterhub_config.py
[Install]
WantedBy=multi-user.target
This setup was working fine until I moved the /home
directory from an SSD to an HDD due to insufficient disk space on the SSD.
Steps to Move /home
Directory
Here are the commands I used to relocate /home
:
sudo rsync -avh --progress /home/ /mnt/hdd5t/home_backup/
sudo rsync -avh --progress /home/ /mnt/hdd5t/home/
diff -r /home /mnt/hdd5t/home
rsync -avh --progress /home/ /mnt/hdd5t/home/
diff -r --no-dereference /home /mnt/hdd5t/home
sudo du -hsx /* | sort -rh
sudo du -hsx /mnt/hdd5t/* | sort -rh
systemctl isolate multi-user.target
mount --bind /mnt/hdd5t/home /home
echo "/mnt/hdd5t/home /home none bind 0 0" >> /etc/fstab
After this change, JupyterHub started timing out when launching the server_admin
user’s single-user server.
Current Issues
- When Jupyter Notebook is running on port 8089:
- Launching the single-user server from JupyterHub as
server_admin
fails with the following error in the logs:
ERROR: the jupyter server could not be started because port 8089 is not available.
- When Jupyter Notebook is not running:
- JupyterHub still times out when launching the single-user server for
server_admin
. The logs fromsudo journalctl -u jupyterhub
show:
16:54:14 server_admin2 jupyterhub[11857]: [I 2025-01-16 16:54:14.943 JupyterHub log:192] 200 GET /hub/api/users?include_stopped_servers=1&offset=0&limit=50&name_filter=&sort=id&state=[secret]&_xsrf=[secre>
16:54:15 server_admin2 jupyterhub[11857]: [W 2025-01-16 16:54:15.439 JupyterHub user:1055] server_admin's server never showed up at http://127.0.0.1:35707/user/server_admin/ after 60 seconds. Giving up.
16:54:15 server_admin2 jupyterhub[11857]:
16:54:15 server_admin2 jupyterhub[11857]: Common causes of this timeout, and debugging tips:
16:54:15 server_admin2 jupyterhub[11857]:
16:54:15 server_admin2 jupyterhub[11857]: 1. The server didn't finish starting,
16:54:15 server_admin2 jupyterhub[11857]: or it crashed due to a configuration issue.
16:54:15 server_admin2 jupyterhub[11857]: Check the single-user server's logs for hints at what needs fixing.
16:54:15 server_admin2 jupyterhub[11857]: 2. The server started, but is not accessible at the specified URL.
16:54:15 server_admin2 jupyterhub[11857]: This may be a configuration issue specific to your chosen Spawner.
16:54:15 server_admin2 jupyterhub[11857]: Check the single-user server logs and resource to make sure the URL
16:54:15 server_admin2 jupyterhub[11857]: is correct and accessible from the Hub.
16:54:15 server_admin2 jupyterhub[11857]: 3. (unlikely) Everything is working, but the server took too long to respond.
16:54:15 server_admin2 jupyterhub[11857]: To fix: increase `Spawner.http_timeout` configuration
16:54:15 server_admin2 jupyterhub[11857]: to a number of seconds that is enough for servers to become responsive.
16:54:15 server_admin2 jupyterhub[11857]:
16:54:15 server_admin2 jupyterhub[11857]: [E 2025-01-16 16:54:15.488 JupyterHub gen:629] Exception in Future <Task finished name='Task-34' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() >
16:54:15 server_admin2 jupyterhub[11857]: Traceback (most recent call last):
16:54:15 server_admin2 jupyterhub[11857]: File "/home/server_admin/anaconda3/lib/python3.8/site-packages/tornado/gen.py", line 624, in error_callback
16:54:15 server_admin2 jupyterhub[11857]: future.result()
16:54:15 server_admin2 jupyterhub[11857]: File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 1122, in finish_user_spawn
16:54:15 server_admin2 jupyterhub[11857]: await spawn_future
16:54:15 server_admin2 jupyterhub[11857]: File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/user.py", line 1033, in spawn
16:54:15 server_admin2 jupyterhub[11857]: await self._wait_up(spawner)
16:54:15 server_admin2 jupyterhub[11857]: File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/user.py", line 1076, in _wait_up
16:54:15 server_admin2 jupyterhub[11857]: raise e
16:54:15 server_admin2 jupyterhub[11857]: File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/user.py", line 1047, in _wait_up
16:54:15 server_admin2 jupyterhub[11857]: resp = await server.wait_up(
16:54:15 server_admin2 jupyterhub[11857]: File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/utils.py", line 322, in wait_for_http_server
16:54:15 server_admin2 jupyterhub[11857]: re = await exponential_backoff(
16:54:15 server_admin2 jupyterhub[11857]: File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/utils.py", line 265, in exponential_backoff
16:54:15 server_admin2 jupyterhub[11857]: raise asyncio.TimeoutError(fail_message)
16:54:15 server_admin2 jupyterhub[11857]: asyncio.exceptions.TimeoutError: Server at http://127.0.0.1:35707/user/server_admin/api didn't respond in 60 seconds
I suspect the following could be causing the issue:
- A conflict between the Jupyter Notebook server and the single-user server spawned by JupyterHub.
- Potential problems with symbolic links or file paths after moving
/home
.
Despite ensuring that file permissions and symbolic links seem intact, the issue persists.
Questions
- Could moving
/home
to a new drive have broken some critical configuration for JupyterHub? - How can I debug this issue further to identify the root cause?
- How can I make both Jupyter Notebook and JupyterHub work properly as they did before moving the
/home
directory?
Any guidance would be greatly appreciated!
Thank you in advance for your help.