I am trying to run JupyterHub in a Docker container using the DockerSpawner. When I run the container, I see the error that I pasted below. I have provided:
- The contents of the DockerFile
- The setting in jupyterhub_config.py.
- The command that I use to run container.
- The debug output that shows the error.
Dockerfile
FROM jupyterhub/jupyterhub:latest
WORKDIR /root
COPY jupyterhub_config.py /etc/jupyterhub/jupyterhub_config.py
RUN apt-get update
RUN apt-get -y install \
net-tools \
ca-certificates \
curl \
gnupg \
lsb-release
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update
RUN apt-get -y install docker-ce docker-ce-cli containerd.io
RUN python3 -m pip install notebook jupyterlab dockerspawner oauthenticator
RUN useradd -m -s /bin/bash admin -p "$(openssl passwd -1 82481956)"
CMD ["jupyterhub", "--debug", "-f", "/etc/jupyterhub/jupyterhub_config.py"]
The only change to default jupyterhub_config.py
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
This is how I run the container
docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 8000:8000 -t etc_jupyterhub_rtc:latest
Debug output
$ docker logs -f 7e1f5e4ef1100e26148fa23064e0d542c339110e28dbd6df87dd90db6965127d
[D 2022-03-11 14:33:17.203 JupyterHub application:731] Looking for /etc/jupyterhub/jupyterhub_config in /root
[D 2022-03-11 14:33:17.204 JupyterHub application:753] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2022-03-11 14:33:17.211 JupyterHub app:2769] Running JupyterHub version 2.2.0
[I 2022-03-11 14:33:17.211 JupyterHub app:2799] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.2.0
[I 2022-03-11 14:33:17.211 JupyterHub app:2799] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-12.1.0
[I 2022-03-11 14:33:17.212 JupyterHub app:2799] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.2.0
[D 2022-03-11 14:33:17.245 JupyterHub app:1646] Generating new cookie_secret
[I 2022-03-11 14:33:17.245 JupyterHub app:1651] Writing cookie_secret to /root/jupyterhub_cookie_secret
[D 2022-03-11 14:33:17.245 JupyterHub app:1773] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2022-03-11 14:33:17.259 JupyterHub orm:924] Stamping empty database with alembic revision 833da8570507
[I 2022-03-11 14:33:17.262 alembic.runtime.migration migration:201] Context impl SQLiteImpl.
[I 2022-03-11 14:33:17.262 alembic.runtime.migration migration:204] Will assume non-transactional DDL.
[I 2022-03-11 14:33:17.268 alembic.runtime.migration migration:615] Running stamp_revision -> 833da8570507
[D 2022-03-11 14:33:17.268 alembic.runtime.migration migration:815] new branch insert 833da8570507
[I 2022-03-11 14:33:17.358 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2022-03-11 14:33:17.358 JupyterHub app:2022] Loading roles into database
[I 2022-03-11 14:33:17.379 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
[D 2022-03-11 14:33:17.382 JupyterHub app:2281] Purging expired APITokens
[D 2022-03-11 14:33:17.384 JupyterHub app:2281] Purging expired OAuthCodes
[D 2022-03-11 14:33:17.385 JupyterHub app:2114] Loading role assignments from config
[D 2022-03-11 14:33:17.391 JupyterHub app:2427] Initializing spawners
[D 2022-03-11 14:33:17.392 JupyterHub app:2558] Loaded users:
[I 2022-03-11 14:33:17.393 JupyterHub app:2838] Initialized 0 spawners in 0.002 seconds
[W 2022-03-11 14:33:17.394 JupyterHub proxy:687] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2022-03-11 14:33:17.394 JupyterHub proxy:691] Starting proxy @ http://:8000
[D 2022-03-11 14:33:17.394 JupyterHub proxy:692] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error']
[D 2022-03-11 14:33:17.398 JupyterHub proxy:610] Writing proxy pid file: jupyterhub-proxy.pid
14:33:17.809 [ConfigProxy] info: Proxying http://*:8000 to (no default)
14:33:17.813 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2022-03-11 14:33:18.339 JupyterHub proxy:728] Proxy started and appears to be up
[D 2022-03-11 14:33:18.342 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
14:33:18.369 [ConfigProxy] info: 200 GET /api/routes
[I 2022-03-11 14:33:18.369 JupyterHub app:3087] Hub API listening on http://127.0.0.1:8081/hub/
[D 2022-03-11 14:33:18.370 JupyterHub proxy:343] Fetching routes to check
[D 2022-03-11 14:33:18.370 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
14:33:18.372 [ConfigProxy] info: 200 GET /api/routes
[D 2022-03-11 14:33:18.372 JupyterHub proxy:346] Checking routes
[I 2022-03-11 14:33:18.372 JupyterHub proxy:431] Adding route for Hub: / => http://127.0.0.1:8081
[D 2022-03-11 14:33:18.373 JupyterHub proxy:821] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
14:33:18.374 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
14:33:18.374 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
14:33:18.375 [ConfigProxy] info: 201 POST /api/routes/
[I 2022-03-11 14:33:18.375 JupyterHub app:3154] JupyterHub is now running at http://:8000
[D 2022-03-11 14:33:18.375 JupyterHub app:2762] It took 1.176 seconds for the Hub to start
[I 2022-03-11 14:33:48.701 JupyterHub log:189] 302 GET / -> /hub/ (@::ffff:172.18.0.1) 0.96ms
[W 2022-03-11 14:33:48.706 JupyterHub base:393] Invalid or expired cookie token
[I 2022-03-11 14:33:48.707 JupyterHub log:189] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:172.18.0.1) 1.22ms
[I 2022-03-11 14:33:48.744 JupyterHub log:189] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:172.18.0.1) 33.91ms
[D 2022-03-11 14:33:59.053 JupyterHub roles:454] Assigning default role to User admin
[I 2022-03-11 14:33:59.062 JupyterHub roles:366] Adding role user for User: admin
[D 2022-03-11 14:33:59.072 JupyterHub roles:454] Assigning default role to User admin
[D 2022-03-11 14:33:59.077 JupyterHub base:563] Setting cookie for admin: jupyterhub-hub-login
[D 2022-03-11 14:33:59.077 JupyterHub base:559] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2022-03-11 14:33:59.077 JupyterHub base:807] User logged in: admin
[I 2022-03-11 14:33:59.078 JupyterHub log:189] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (admin@::ffff:172.18.0.1) 44.18ms
[D 2022-03-11 14:33:59.083 JupyterHub base:281] Recording first activity for <User(admin 0/1 running)>
[D 2022-03-11 14:33:59.090 JupyterHub user:399] Creating <class 'dockerspawner.dockerspawner.DockerSpawner'> for admin:
[I 2022-03-11 14:33:59.091 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn (admin@::ffff:172.18.0.1) 10.39ms
[D 2022-03-11 14:33:59.096 JupyterHub scopes:491] Checking access via scope servers
[D 2022-03-11 14:33:59.096 JupyterHub scopes:402] Argument-based access to /hub/spawn via servers
[D 2022-03-11 14:33:59.096 JupyterHub pages:215] Triggering spawn with default options for admin
[D 2022-03-11 14:33:59.096 JupyterHub base:925] Initiating spawn for admin
[D 2022-03-11 14:33:59.096 JupyterHub base:929] 0/100 concurrent spawns
[D 2022-03-11 14:33:59.096 JupyterHub base:934] 0 active servers
[D 2022-03-11 14:33:59.101 JupyterHub roles:477] Checking token permissions against requested role server
[I 2022-03-11 14:33:59.103 JupyterHub roles:482] Adding role server to token: <APIToken('8e5f...', user='admin', client_id='jupyterhub')>
[I 2022-03-11 14:33:59.111 JupyterHub provider:607] Creating oauth client jupyterhub-user-admin
[D 2022-03-11 14:33:59.129 JupyterHub user:728] Calling Spawner.start for admin
[D 2022-03-11 14:33:59.141 JupyterHub dockerspawner:982] Getting container 'jupyter-admin'
[I 2022-03-11 14:33:59.143 JupyterHub dockerspawner:988] Container 'jupyter-admin' is gone
[D 2022-03-11 14:33:59.145 JupyterHub dockerspawner:1148] Starting host with config: {'auto_remove': False, 'binds': {}, 'links': {}, 'mounts': [], 'mem_limit': 0, 'cpu_period': 100000, 'cpu_quota': 0, 'port_bindings': {8888: ('127.0.0.1',)}, 'network_mode': 'bridge'}
[I 2022-03-11 14:33:59.198 JupyterHub dockerspawner:1272] Created container jupyter-admin (id: 80539e2) from image jupyterhub/singleuser:2.2
[I 2022-03-11 14:33:59.198 JupyterHub dockerspawner:1296] Starting container jupyter-admin (id: 80539e2)
[D 2022-03-11 14:33:59.444 JupyterHub spawner:1253] Polling subprocess every 30s
[I 2022-03-11 14:34:00.098 JupyterHub log:189] 302 GET /hub/spawn -> /hub/spawn-pending/admin (admin@::ffff:172.18.0.1) 1003.46ms
[D 2022-03-11 14:34:00.103 JupyterHub scopes:491] Checking access via scope servers
[D 2022-03-11 14:34:00.103 JupyterHub scopes:402] Argument-based access to /hub/spawn-pending/admin via servers
[I 2022-03-11 14:34:00.103 JupyterHub pages:401] admin is pending spawn
[I 2022-03-11 14:34:00.106 JupyterHub log:189] 200 GET /hub/spawn-pending/admin (admin@::ffff:172.18.0.1) 5.04ms
[D 2022-03-11 14:34:00.148 JupyterHub scopes:491] Checking access via scope read:servers
[D 2022-03-11 14:34:00.148 JupyterHub scopes:402] Argument-based access to /hub/api/users/admin/server/progress via read:servers
[D 2022-03-11 14:34:09.098 JupyterHub dockerspawner:982] Getting container 'jupyter-admin'
[D 2022-03-11 14:34:09.103 JupyterHub dockerspawner:967] Container 80539e2 status: {'Dead': False,
'Error': '',
'ExitCode': 0,
'FinishedAt': '0001-01-01T00:00:00Z',
'OOMKilled': False,
'Paused': False,
'Pid': 375518,
'Restarting': False,
'Running': True,
'StartedAt': '2022-03-11T14:33:59.433584941Z',
'Status': 'running'}
[W 2022-03-11 14:34:09.103 JupyterHub base:1086] User admin is slow to become responsive (timeout=10)
[D 2022-03-11 14:34:09.103 JupyterHub base:1091] Expecting server for admin at: http://127.0.0.1:49158/user/admin/
[W 2022-03-11 14:34:29.216 JupyterHub user:864] admin's server never showed up at http://127.0.0.1:49158/user/admin/ after 30 seconds. Giving up.
Common causes of this timeout, and debugging tips:
1. The server didn't finish starting,
or it crashed due to a configuration issue.
Check the single-user server's logs for hints at what needs fixing.
2. The server started, but is not accessible at the specified URL.
This may be a configuration issue specific to your chosen Spawner.
Check the single-user server logs and resource to make sure the URL
is correct and accessible from the Hub.
3. (unlikely) Everything is working, but the server took too long to respond.
To fix: increase `Spawner.http_timeout` configuration
to a number of seconds that is enough for servers to become responsive.
[D 2022-03-11 14:34:29.217 JupyterHub user:913] Stopping admin
[D 2022-03-11 14:34:29.217 JupyterHub dockerspawner:982] Getting container 'jupyter-admin'
[D 2022-03-11 14:34:29.223 JupyterHub dockerspawner:967] Container 80539e2 status: {'Dead': False,
'Error': '',
'ExitCode': 0,
'FinishedAt': '0001-01-01T00:00:00Z',
'OOMKilled': False,
'Paused': False,
'Pid': 375518,
'Restarting': False,
'Running': True,
'StartedAt': '2022-03-11T14:33:59.433584941Z',
'Status': 'running'}
[I 2022-03-11 14:34:29.223 JupyterHub dockerspawner:1390] Stopping container jupyter-admin (id: 80539e2)
[D 2022-03-11 14:34:29.584 JupyterHub user:936] Finished stopping admin
[E 2022-03-11 14:34:29.590 JupyterHub gen:623] Exception in Future <Task finished name='Task-30' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:945> exception=TimeoutError("Server at http://127.0.0.1:49158/user/admin/ didn't respond in 30 seconds")> after timeout
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
future.result()
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 952, in finish_user_spawn
await spawn_future
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 845, in spawn
await self._wait_up(spawner)
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 889, in _wait_up
raise e
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 859, in _wait_up
resp = await server.wait_up(
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 241, in wait_for_http_server
re = await exponential_backoff(
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 189, in exponential_backoff
raise asyncio.TimeoutError(fail_message)
asyncio.exceptions.TimeoutError: Server at http://127.0.0.1:49158/user/admin/ didn't respond in 30 seconds
[I 2022-03-11 14:34:29.593 JupyterHub log:189] 200 GET /hub/api/users/admin/server/progress (admin@::ffff:172.18.0.1) 29446.80ms
[D 2022-03-11 14:38:18.376 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
14:38:18.377 [ConfigProxy] info: 200 GET /api/routes
[D 2022-03-11 14:38:18.378 JupyterHub proxy:346] Checking routes