Timeout Issue with JupyterHub After Moving /home Directory to a Different Drive

Hello,

I am experiencing an issue with JupyterHub on my Ubuntu server, which is configured for multiple users.

Initially, the server was managed with a single admin account (server_admin) that has sudo privileges. I installed Anaconda under this account and set up a Jupyter Notebook server on port 8089. All users accessed the Jupyter Notebook server through this account, which, for security reasons, was not an ideal configuration. To improve this, I set up JupyterHub to support multi-user access.

My JupyterHub Setup

From the server_admin account (with the base Conda environment activated), I followed the JupyterHub tutorial:

conda install -c conda-forge jupyterhub  
conda install jupyterlab notebook  

Here is the configuration file /etc/jupyterhub/jupyterhub_config.py:

c.JupyterHub.authenticator_class = 'jupyterhub.auth.PAMAuthenticator'  
c.JupyterHub.extra_log_file = '/var/log/jupyterhub.log'  
c.JupyterHub.ip = '========='  # Hidden for security reasons  
c.JupyterHub.port = 8080  
c.JupyterHub.spawner_class = 'jupyterhub.spawner.LocalProcessSpawner'  
c.Spawner.debug = True  
c.Spawner.default_url = '/tree'  
c.Spawner.http_timeout = 60  
c.Spawner.start_timeout = 60  
c.Authenticator.admin_users = set(["server_admin"])  
c.Authenticator.allow_all = True  
c.Authenticator.allow_existing_users = True  

I registered a service to run JupyterHub as follows:

[Unit]  
Description=JupyterHub  
After=syslog.target network.target  

[Service]  
User=root  
Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/server_admin/anaconda3/bin:/home/server_admin/anaconda3/condabin"  
ExecStart=/home/server_admin/anaconda3/bin/jupyterhub -f /etc/jupyterhub/jupyterhub_config.py  

[Install]  
WantedBy=multi-user.target  

This setup was working fine until I moved the /home directory from an SSD to an HDD due to insufficient disk space on the SSD.

Steps to Move /home Directory

Here are the commands I used to relocate /home:

sudo rsync -avh --progress /home/ /mnt/hdd5t/home_backup/  
sudo rsync -avh --progress /home/ /mnt/hdd5t/home/  
diff -r /home /mnt/hdd5t/home  
rsync -avh --progress /home/ /mnt/hdd5t/home/  
diff -r --no-dereference /home /mnt/hdd5t/home  

sudo du -hsx /* | sort -rh  
sudo du -hsx /mnt/hdd5t/* | sort -rh  

systemctl isolate multi-user.target  
mount --bind /mnt/hdd5t/home /home  
echo "/mnt/hdd5t/home /home none bind 0 0" >> /etc/fstab  

After this change, JupyterHub started timing out when launching the server_admin user’s single-user server.

Current Issues

  1. When Jupyter Notebook is running on port 8089:
  • Launching the single-user server from JupyterHub as server_admin fails with the following error in the logs:
ERROR: the jupyter server could not be started because port 8089 is not available.  
  1. When Jupyter Notebook is not running:
  • JupyterHub still times out when launching the single-user server for server_admin. The logs from sudo journalctl -u jupyterhub show:
16:54:14 server_admin2 jupyterhub[11857]: [I 2025-01-16 16:54:14.943 JupyterHub log:192] 200 GET /hub/api/users?include_stopped_servers=1&offset=0&limit=50&name_filter=&sort=id&state=[secret]&_xsrf=[secre>
16:54:15 server_admin2 jupyterhub[11857]: [W 2025-01-16 16:54:15.439 JupyterHub user:1055] server_admin's server never showed up at http://127.0.0.1:35707/user/server_admin/ after 60 seconds. Giving up.
16:54:15 server_admin2 jupyterhub[11857]:
16:54:15 server_admin2 jupyterhub[11857]:     Common causes of this timeout, and debugging tips:
16:54:15 server_admin2 jupyterhub[11857]:
16:54:15 server_admin2 jupyterhub[11857]:     1. The server didn't finish starting,
16:54:15 server_admin2 jupyterhub[11857]:        or it crashed due to a configuration issue.
16:54:15 server_admin2 jupyterhub[11857]:        Check the single-user server's logs for hints at what needs fixing.
16:54:15 server_admin2 jupyterhub[11857]:     2. The server started, but is not accessible at the specified URL.
16:54:15 server_admin2 jupyterhub[11857]:        This may be a configuration issue specific to your chosen Spawner.
16:54:15 server_admin2 jupyterhub[11857]:        Check the single-user server logs and resource to make sure the URL
16:54:15 server_admin2 jupyterhub[11857]:        is correct and accessible from the Hub.
16:54:15 server_admin2 jupyterhub[11857]:     3. (unlikely) Everything is working, but the server took too long to respond.
16:54:15 server_admin2 jupyterhub[11857]:        To fix: increase `Spawner.http_timeout` configuration
16:54:15 server_admin2 jupyterhub[11857]:        to a number of seconds that is enough for servers to become responsive.
16:54:15 server_admin2 jupyterhub[11857]:
16:54:15 server_admin2 jupyterhub[11857]: [E 2025-01-16 16:54:15.488 JupyterHub gen:629] Exception in Future <Task finished name='Task-34' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() >
16:54:15 server_admin2 jupyterhub[11857]:     Traceback (most recent call last):
16:54:15 server_admin2 jupyterhub[11857]:       File "/home/server_admin/anaconda3/lib/python3.8/site-packages/tornado/gen.py", line 624, in error_callback
16:54:15 server_admin2 jupyterhub[11857]:         future.result()
16:54:15 server_admin2 jupyterhub[11857]:       File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 1122, in finish_user_spawn
16:54:15 server_admin2 jupyterhub[11857]:         await spawn_future
16:54:15 server_admin2 jupyterhub[11857]:       File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/user.py", line 1033, in spawn
16:54:15 server_admin2 jupyterhub[11857]:         await self._wait_up(spawner)
16:54:15 server_admin2 jupyterhub[11857]:       File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/user.py", line 1076, in _wait_up
16:54:15 server_admin2 jupyterhub[11857]:         raise e
16:54:15 server_admin2 jupyterhub[11857]:       File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/user.py", line 1047, in _wait_up
16:54:15 server_admin2 jupyterhub[11857]:         resp = await server.wait_up(
16:54:15 server_admin2 jupyterhub[11857]:       File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/utils.py", line 322, in wait_for_http_server
16:54:15 server_admin2 jupyterhub[11857]:         re = await exponential_backoff(
16:54:15 server_admin2 jupyterhub[11857]:       File "/home/server_admin/anaconda3/lib/python3.8/site-packages/jupyterhub/utils.py", line 265, in exponential_backoff
16:54:15 server_admin2 jupyterhub[11857]:         raise asyncio.TimeoutError(fail_message)
16:54:15 server_admin2 jupyterhub[11857]:     asyncio.exceptions.TimeoutError: Server at http://127.0.0.1:35707/user/server_admin/api didn't respond in 60 seconds

I suspect the following could be causing the issue:

  • A conflict between the Jupyter Notebook server and the single-user server spawned by JupyterHub.
  • Potential problems with symbolic links or file paths after moving /home.

Despite ensuring that file permissions and symbolic links seem intact, the issue persists.

Questions

  1. Could moving /home to a new drive have broken some critical configuration for JupyterHub?
  2. How can I debug this issue further to identify the root cause?
  3. How can I make both Jupyter Notebook and JupyterHub work properly as they did before moving the /home directory?

Any guidance would be greatly appreciated!

Thank you in advance for your help.

Well, there can be lot of issues that can go wrong with your setup. Why did you choose to install stack on home directory of a user? It is better to create a conda env at system locations like /opt or /srv so that all users will have read permissions on the Python packages.

Can you share the complete logs for the single-user server that failed to start? It looks like it might be ports-in-use, so maybe some things are holding ports or something similar. Port 8089 also looks like it’s not using the JupyterHub-assigned port and is defaulting to default jupyter-server behavior, which is suspicious. Getting the single-user server startup logs would help a lot.