Jupyterhub spawn fails - tornado.web.HTTPError: HTTP 500: Internal Server Error

General
1

Bug description

I have installed jupyter hub development install using the following documentation:
https://jupyterhub.readthedocs.io/en/stable/contributing/setup.html

The python3 installed is a python alt install and not a virtual env via conda. When I start the hub in a conda environment, i am able to login, do a notebook server spawn and perform python operations.
In the python3 alt install env, when i start the hub, the spawn fails with the following error:

ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<BaseHandler.spawn_single_user() done, defined at /data/jhub/jupyterhub/jupyterhub/handlers/base.py:741> exception=HTTPError()>
Traceback (most recent call last):
  File "/data/jhub/jupyterhub/jupyterhub/handlers/base.py", line 939, in spawn_single_user
    timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
tornado.util.TimeoutError: Timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/jhub/jupyterhub/jupyterhub/handlers/base.py", line 972, in spawn_single_user
    % (status, spawner._log_name),
tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for admin may contain details.)
[W 2020-08-20 13:06:37.464 JupyterHub user:697] admin's server never showed up at http:// 127.0.0.1:60497/user/admin/ after 30 seconds. Giving up
[E 2020-08-20 13:06:37.530 JupyterHub gen:599] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /data/jhub/jupyterhub/jupyterhub/handlers/base.py:845> exception=TimeoutError("Server at http:// 127.0.0.1:60497/user/admin/ didn't respond in 30 seconds",)> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib64/python3.6/site-packages/tornado/gen.py", line 593, in error_callback
        future.result()
      File "/data/jhub/jupyterhub/jupyterhub/handlers/base.py", line 852, in finish_user_spawn
        await spawn_future
      File "/data/jhub/jupyterhub/jupyterhub/user.py", line 673, in spawn
        await self._wait_up(spawner)
      File "/data/jhub/jupyterhub/jupyterhub/user.py", line 720, in _wait_up
        raise e
      File "/data/jhub/jupyterhub/jupyterhub/user.py", line 688, in _wait_up
        http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
      File "/data/jhub/jupyterhub/jupyterhub/utils.py", line 234, in wait_for_http_server
        timeout=timeout,
      File "/data/jhub/jupyterhub/jupyterhub/utils.py", line 177, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http:// 127.0.0.1:60497/user/admin/ didn't respond in 30 seconds

[I 2020-08-20 13:06:37.539 JupyterHub log:181] 200 GET /hub/api/users/admin/server/progress (admin@::ffff:10.184.95.74) 31249.70ms

But if i spawn a anaconda virtualenv, it works properly there.

2

Hi! Welcome to the forum. Please could you explain what you mean by

Thanks!

3

The system has a default python 2.7 on top of which python 3 was installed. Using the make && make altinstall command.

4

Hello, i seem to have the exact same problem, in the same scenario (venv).

Using python 3.6 on CentOS7 and a venv for installing latest jupterhub and using the default spawner, i always face:
Core problem:

Traceback (most recent call last):
  File "/srv/jupyterhub/venvjupyterhub/lib64/python3.6/site-packages/jupyterhub/handlers/base.py", line 1069, in spawn_single_user
    % (status, spawner._log_name),
tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for mytestuser may contain details.)

as soon as the user successfully logged in.

The context is:

[D 2022-02-17 21:26:47.915 JupyterHub roles:474] Checking token permissions against requested role server
[I 2022-02-17 21:26:47.919 JupyterHub roles:478] Adding role server to token: <APIToken('7d1f...', user='mytestuser', client_id='jupyterhub')>
[I 2022-02-17 21:26:47.932 JupyterHub provider:607] Creating oauth client jupyterhub-user-mytestuser
[W 2022-02-17 21:26:48.247 JupyterHub auth:1107] Failed to open PAM session for mytestuser: [PAM Error 14] Cannot make/remove an entry for the specified session
[W 2022-02-17 21:26:48.247 JupyterHub auth:1108] Disabling PAM sessions from now on.
[D 2022-02-17 21:26:48.256 JupyterHub user:676] Calling Spawner.start for mytestuser
[I 2022-02-17 21:26:48.258 JupyterHub spawner:1526] Spawning jupyterhub-singleuser
[D 2022-02-17 21:26:48.277 JupyterHub spawner:1221] Polling subprocess every 30s
[I 2022-02-17 21:26:48.911 JupyterHub log:189] 302 GET /hub/spawn -> /hub/spawn-pending/mytestuser (mytestuser@10.10.0.5) 1004.75ms
[D 2022-02-17 21:26:48.939 JupyterHub scopes:491] Checking access via scope servers
[D 2022-02-17 21:26:48.939 JupyterHub scopes:402] Argument-based access to /hub/spawn-pending/mytestuser via servers
[I 2022-02-17 21:26:48.939 JupyterHub pages:405] mytestuser is pending spawn
[I 2022-02-17 21:26:48.945 JupyterHub log:189] 200 GET /hub/spawn-pending/mytestuser (mytestuser@10.10.0.5) 10.86ms
Traceback (most recent call last):
  File "/srv/jupyterhub/venvjupyterhub/bin/jupyterhub-singleuser", line 5, in <module>
    from jupyterhub.singleuser import main
  File "/srv/jupyterhub/venvjupyterhub/lib64/python3.6/site-packages/jupyterhub/singleuser/__init__.py", line 5, in <module>
    from .app import main
  File "/srv/jupyterhub/venvjupyterhub/lib64/python3.6/site-packages/jupyterhub/singleuser/app.py", line 38, in <module>
    raise _import_error
TypeError: exceptions must derive from BaseException
[D 2022-02-17 21:26:49.321 JupyterHub scopes:491] Checking access via scope read:servers
[D 2022-02-17 21:26:49.322 JupyterHub scopes:402] Argument-based access to /hub/api/users/mytestuser/server/progress via read:servers



ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<BaseHandler.spawn_single_user() done, defined at /srv/jupyterhub/venvjupyterhub/lib64/python3.6/site-packages/jupyterhub/handlers/base.py:831> exception=HTTPError()>
Traceback (most recent call last):
  File "/srv/jupyterhub/venvjupyterhub/lib64/python3.6/site-packages/jupyterhub/handlers/base.py", line 1033, in spawn_single_user
    timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
tornado.util.TimeoutError: Timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/jupyterhub/venvjupyterhub/lib64/python3.6/site-packages/jupyterhub/handlers/base.py", line 1069, in spawn_single_user
    % (status, spawner._log_name),
tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for mytestuser may contain details.)

It happens upon automatically launching the server after the user logged in successfully, where the launch times out.
When user is logged in he can head over to the page where he can click again for launching the server, which again fails.

May the venv adds the problem?
Although permissions look perfectly fine (or don’t they?):

ls -al /srv/jupyterhub/
total 0
drwxr-xr-x. 3 root root  28 Feb 17 11:59 .
drwxr-xr-x. 3 root root  24 Feb 17 11:37 ..
drwxr-xr-x. 7 root root 181 Feb 17 19:34 venvjupyterhub


ls -al
total 64
drwxr-xr-x. 7 root root   181 Feb 17 19:34 .
drwxr-xr-x. 3 root root    28 Feb 17 11:59 ..
drwxr-xr-x. 2 root root  4096 Feb 17 11:52 bin
drwxr-xr-x. 3 root root    18 Feb 17 11:52 include
-rw-------. 1 root root    65 Feb 17 18:05 jupyterhub_cookie_secret
-rw-r--r--. 1 root root 47104 Feb 17 19:34 jupyterhub.sqlite
drwxr-xr-x. 3 root root    23 Feb 17 11:40 lib
lrwxrwxrwx. 1 root root     3 Feb 17 11:40 lib64 -> lib
-rw-r--r--. 1 root root    61 Feb 17 11:46 pip-selfcheck.json
-rw-r--r--. 1 root root    69 Feb 17 11:45 pyvenv.cfg
drwxr-xr-x. 3 root root    24 Feb 17 11:52 share
drwxr-xr-x. 2 root root    38 Feb 17 14:10 ssl

Would highly appreciate any ideas.
Venv is crucial for me, since server and therefore don’t want to use the OS level system’s python environment.

Best regards

5

You probably forgot to install the notebook. Try running:

pip install notebook