How to run JupyterHub in Docker using DockerSpawner and PAM Authenticator?

I am trying to run JupyterHub in a Docker container using the DockerSpawner. When I run the container, I see the error that I pasted below. I have provided:

  1. The contents of the DockerFile
  2. The setting in
  3. The command that I use to run container.
  4. The debug output that shows the error.


FROM jupyterhub/jupyterhub:latest


COPY /etc/jupyterhub/

RUN apt-get update

RUN apt-get -y install \
    net-tools \
    ca-certificates \
    curl \
    gnupg \

RUN curl -fsSL | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

RUN apt-get update

RUN apt-get -y install docker-ce docker-ce-cli

RUN python3 -m pip install notebook jupyterlab dockerspawner oauthenticator 

RUN useradd -m -s /bin/bash admin -p "$(openssl passwd -1 82481956)"

CMD ["jupyterhub", "--debug", "-f", "/etc/jupyterhub/"]

The only change to default

c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'

This is how I run the container
docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 8000:8000 -t etc_jupyterhub_rtc:latest

Debug output

$ docker logs -f 7e1f5e4ef1100e26148fa23064e0d542c339110e28dbd6df87dd90db6965127d
[D 2022-03-11 14:33:17.203 JupyterHub application:731] Looking for /etc/jupyterhub/jupyterhub_config in /root
[D 2022-03-11 14:33:17.204 JupyterHub application:753] Loaded config file: /etc/jupyterhub/
[I 2022-03-11 14:33:17.211 JupyterHub app:2769] Running JupyterHub version 2.2.0
[I 2022-03-11 14:33:17.211 JupyterHub app:2799] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.2.0
[I 2022-03-11 14:33:17.211 JupyterHub app:2799] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-12.1.0
[I 2022-03-11 14:33:17.212 JupyterHub app:2799] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.2.0
[D 2022-03-11 14:33:17.245 JupyterHub app:1646] Generating new cookie_secret
[I 2022-03-11 14:33:17.245 JupyterHub app:1651] Writing cookie_secret to /root/jupyterhub_cookie_secret
[D 2022-03-11 14:33:17.245 JupyterHub app:1773] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2022-03-11 14:33:17.259 JupyterHub orm:924] Stamping empty database with alembic revision 833da8570507
[I 2022-03-11 14:33:17.262 alembic.runtime.migration migration:201] Context impl SQLiteImpl.
[I 2022-03-11 14:33:17.262 alembic.runtime.migration migration:204] Will assume non-transactional DDL.
[I 2022-03-11 14:33:17.268 alembic.runtime.migration migration:615] Running stamp_revision  -> 833da8570507
[D 2022-03-11 14:33:17.268 alembic.runtime.migration migration:815] new branch insert 833da8570507
[I 2022-03-11 14:33:17.358 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2022-03-11 14:33:17.358 JupyterHub app:2022] Loading roles into database
[I 2022-03-11 14:33:17.379 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
[D 2022-03-11 14:33:17.382 JupyterHub app:2281] Purging expired APITokens
[D 2022-03-11 14:33:17.384 JupyterHub app:2281] Purging expired OAuthCodes
[D 2022-03-11 14:33:17.385 JupyterHub app:2114] Loading role assignments from config
[D 2022-03-11 14:33:17.391 JupyterHub app:2427] Initializing spawners
[D 2022-03-11 14:33:17.392 JupyterHub app:2558] Loaded users:
[I 2022-03-11 14:33:17.393 JupyterHub app:2838] Initialized 0 spawners in 0.002 seconds
[W 2022-03-11 14:33:17.394 JupyterHub proxy:687] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2022-03-11 14:33:17.394 JupyterHub proxy:691] Starting proxy @ http://:8000
[D 2022-03-11 14:33:17.394 JupyterHub proxy:692] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '', '--api-port', '8001', '--error-target', '']
[D 2022-03-11 14:33:17.398 JupyterHub proxy:610] Writing proxy pid file:
14:33:17.809 [ConfigProxy] info: Proxying http://*:8000 to (no default)
14:33:17.813 [ConfigProxy] info: Proxy API at
[D 2022-03-11 14:33:18.339 JupyterHub proxy:728] Proxy started and appears to be up
[D 2022-03-11 14:33:18.342 JupyterHub proxy:821] Proxy: Fetching GET
14:33:18.369 [ConfigProxy] info: 200 GET /api/routes 
[I 2022-03-11 14:33:18.369 JupyterHub app:3087] Hub API listening on
[D 2022-03-11 14:33:18.370 JupyterHub proxy:343] Fetching routes to check
[D 2022-03-11 14:33:18.370 JupyterHub proxy:821] Proxy: Fetching GET
14:33:18.372 [ConfigProxy] info: 200 GET /api/routes 
[D 2022-03-11 14:33:18.372 JupyterHub proxy:346] Checking routes
[I 2022-03-11 14:33:18.372 JupyterHub proxy:431] Adding route for Hub: / =>
[D 2022-03-11 14:33:18.373 JupyterHub proxy:821] Proxy: Fetching POST
14:33:18.374 [ConfigProxy] info: Adding route / ->
14:33:18.374 [ConfigProxy] info: Route added / ->
14:33:18.375 [ConfigProxy] info: 201 POST /api/routes/ 
[I 2022-03-11 14:33:18.375 JupyterHub app:3154] JupyterHub is now running at http://:8000
[D 2022-03-11 14:33:18.375 JupyterHub app:2762] It took 1.176 seconds for the Hub to start
[I 2022-03-11 14:33:48.701 JupyterHub log:189] 302 GET / -> /hub/ (@::ffff: 0.96ms
[W 2022-03-11 14:33:48.706 JupyterHub base:393] Invalid or expired cookie token
[I 2022-03-11 14:33:48.707 JupyterHub log:189] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff: 1.22ms
[I 2022-03-11 14:33:48.744 JupyterHub log:189] 200 GET /hub/login?next=%2Fhub%2F (@::ffff: 33.91ms
[D 2022-03-11 14:33:59.053 JupyterHub roles:454] Assigning default role to User admin
[I 2022-03-11 14:33:59.062 JupyterHub roles:366] Adding role user for User: admin
[D 2022-03-11 14:33:59.072 JupyterHub roles:454] Assigning default role to User admin
[D 2022-03-11 14:33:59.077 JupyterHub base:563] Setting cookie for admin: jupyterhub-hub-login
[D 2022-03-11 14:33:59.077 JupyterHub base:559] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2022-03-11 14:33:59.077 JupyterHub base:807] User logged in: admin
[I 2022-03-11 14:33:59.078 JupyterHub log:189] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (admin@::ffff: 44.18ms
[D 2022-03-11 14:33:59.083 JupyterHub base:281] Recording first activity for <User(admin 0/1 running)>
[D 2022-03-11 14:33:59.090 JupyterHub user:399] Creating <class 'dockerspawner.dockerspawner.DockerSpawner'> for admin:
[I 2022-03-11 14:33:59.091 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn (admin@::ffff: 10.39ms
[D 2022-03-11 14:33:59.096 JupyterHub scopes:491] Checking access via scope servers
[D 2022-03-11 14:33:59.096 JupyterHub scopes:402] Argument-based access to /hub/spawn via servers
[D 2022-03-11 14:33:59.096 JupyterHub pages:215] Triggering spawn with default options for admin
[D 2022-03-11 14:33:59.096 JupyterHub base:925] Initiating spawn for admin
[D 2022-03-11 14:33:59.096 JupyterHub base:929] 0/100 concurrent spawns
[D 2022-03-11 14:33:59.096 JupyterHub base:934] 0 active servers
[D 2022-03-11 14:33:59.101 JupyterHub roles:477] Checking token permissions against requested role server
[I 2022-03-11 14:33:59.103 JupyterHub roles:482] Adding role server to token: <APIToken('8e5f...', user='admin', client_id='jupyterhub')>
[I 2022-03-11 14:33:59.111 JupyterHub provider:607] Creating oauth client jupyterhub-user-admin
[D 2022-03-11 14:33:59.129 JupyterHub user:728] Calling Spawner.start for admin
[D 2022-03-11 14:33:59.141 JupyterHub dockerspawner:982] Getting container 'jupyter-admin'
[I 2022-03-11 14:33:59.143 JupyterHub dockerspawner:988] Container 'jupyter-admin' is gone
[D 2022-03-11 14:33:59.145 JupyterHub dockerspawner:1148] Starting host with config: {'auto_remove': False, 'binds': {}, 'links': {}, 'mounts': [], 'mem_limit': 0, 'cpu_period': 100000, 'cpu_quota': 0, 'port_bindings': {8888: ('',)}, 'network_mode': 'bridge'}
[I 2022-03-11 14:33:59.198 JupyterHub dockerspawner:1272] Created container jupyter-admin (id: 80539e2) from image jupyterhub/singleuser:2.2
[I 2022-03-11 14:33:59.198 JupyterHub dockerspawner:1296] Starting container jupyter-admin (id: 80539e2)
[D 2022-03-11 14:33:59.444 JupyterHub spawner:1253] Polling subprocess every 30s
[I 2022-03-11 14:34:00.098 JupyterHub log:189] 302 GET /hub/spawn -> /hub/spawn-pending/admin (admin@::ffff: 1003.46ms
[D 2022-03-11 14:34:00.103 JupyterHub scopes:491] Checking access via scope servers
[D 2022-03-11 14:34:00.103 JupyterHub scopes:402] Argument-based access to /hub/spawn-pending/admin via servers
[I 2022-03-11 14:34:00.103 JupyterHub pages:401] admin is pending spawn
[I 2022-03-11 14:34:00.106 JupyterHub log:189] 200 GET /hub/spawn-pending/admin (admin@::ffff: 5.04ms
[D 2022-03-11 14:34:00.148 JupyterHub scopes:491] Checking access via scope read:servers
[D 2022-03-11 14:34:00.148 JupyterHub scopes:402] Argument-based access to /hub/api/users/admin/server/progress via read:servers
[D 2022-03-11 14:34:09.098 JupyterHub dockerspawner:982] Getting container 'jupyter-admin'
[D 2022-03-11 14:34:09.103 JupyterHub dockerspawner:967] Container 80539e2 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '0001-01-01T00:00:00Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 375518,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2022-03-11T14:33:59.433584941Z',
     'Status': 'running'}
[W 2022-03-11 14:34:09.103 JupyterHub base:1086] User admin is slow to become responsive (timeout=10)
[D 2022-03-11 14:34:09.103 JupyterHub base:1091] Expecting server for admin at:
[W 2022-03-11 14:34:29.216 JupyterHub user:864] admin's server never showed up at after 30 seconds. Giving up.
    Common causes of this timeout, and debugging tips:
    1. The server didn't finish starting,
       or it crashed due to a configuration issue.
       Check the single-user server's logs for hints at what needs fixing.
    2. The server started, but is not accessible at the specified URL.
       This may be a configuration issue specific to your chosen Spawner.
       Check the single-user server logs and resource to make sure the URL
       is correct and accessible from the Hub.
    3. (unlikely) Everything is working, but the server took too long to respond.
       To fix: increase `Spawner.http_timeout` configuration
       to a number of seconds that is enough for servers to become responsive.
[D 2022-03-11 14:34:29.217 JupyterHub user:913] Stopping admin
[D 2022-03-11 14:34:29.217 JupyterHub dockerspawner:982] Getting container 'jupyter-admin'
[D 2022-03-11 14:34:29.223 JupyterHub dockerspawner:967] Container 80539e2 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '0001-01-01T00:00:00Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 375518,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2022-03-11T14:33:59.433584941Z',
     'Status': 'running'}
[I 2022-03-11 14:34:29.223 JupyterHub dockerspawner:1390] Stopping container jupyter-admin (id: 80539e2)
[D 2022-03-11 14:34:29.584 JupyterHub user:936] Finished stopping admin
[E 2022-03-11 14:34:29.590 JupyterHub gen:623] Exception in Future <Task finished name='Task-30' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/> exception=TimeoutError("Server at didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/", line 618, in error_callback
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/", line 952, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/", line 845, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/", line 889, in _wait_up
        raise e
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/", line 859, in _wait_up
        resp = await server.wait_up(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/", line 241, in wait_for_http_server
        re = await exponential_backoff(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/", line 189, in exponential_backoff
        raise asyncio.TimeoutError(fail_message)
    asyncio.exceptions.TimeoutError: Server at didn't respond in 30 seconds
[I 2022-03-11 14:34:29.593 JupyterHub log:189] 200 GET /hub/api/users/admin/server/progress (admin@::ffff: 29446.80ms
[D 2022-03-11 14:38:18.376 JupyterHub proxy:821] Proxy: Fetching GET
14:38:18.377 [ConfigProxy] info: 200 GET /api/routes 
[D 2022-03-11 14:38:18.378 JupyterHub proxy:346] Checking routes

Is docker definitely working inside the container? If you use docker exec -it <name> bash can you manually run a docker container inside your JupyterHub container? If you can’t it suggests either a configuration or a permissions problem with docker-in-docker. You can try passing additional privileges: --privileged is the easy option, though using --cap-add ... to pass individual capabilities is best practice if you can work out what you need.

1 Like

Thank you. That’s a good idea. I should try to start up a container from within the container and examine the results. Thank you for the guidance. I will try that.