Jupyterhub on WSL-2 cannot spawn Docker

JupyterHub
,
1

Hi to all. I’m having some problems creating a custom docker image and spawning it as a singleUserSpawner.
I’m on window 10 with WSL2 running Ubuntu-20.04. I want to be able to spawn a docker container for every user i let connect to docker hub.
Jupyterhub version → 2.2.0
Dockerspawner version-> 12.1.0

My jupyterhub_configuration.py:

from dockerspawner import SystemUserSpawner
c = get_config()

class CustomSpawner(SystemUserSpawner):
    def start(self, **args):
        self.image = "jupyterhub/singleuser:2.2.2"
        #self.ip = "127.0.0.1"
        return super(CustomSpawner, self).start(**args)
    # without overwriting this methods the container won't even start with error failed
    # to get ip and port info
    async def get_ip_and_port(self):
        return self.ip, self.port

c.JupyterHub.spawner_class = CustomSpawner
c.JupyterHub.authenticator_class = "dummy"

Without overriding get_ip_and_port i have this error:

Stopping container jupyter-mattia (id: 10f090c)
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-16' coro=<BaseHandler.spawn_single_user() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:841> exception=RuntimeError('Failed to get port info for 10f090c1a0c240108e49ba37e5b01f535cb1098ed2b723eca331c5779746fb70')>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 1042, in spawn_single_user
    await gen.with_timeout(
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 952, in finish_user_spawn
    await spawn_future
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 833, in spawn
    raise e
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 732, in spawn
    url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
  File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 1309, in start
    ip, port = await self.get_ip_and_port()
  File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 1357, in get_ip_and_port
    raise RuntimeError("Failed to get port info for %s" % self.container_id)
RuntimeError: Failed to get port info for 10f090c1a0c240108e49ba37e5b01f535cb1098ed2b723eca331c5779746fb70

I tried with different images but none of them are working with Dockerspawner:

  • jupyter/minimal-notebook:latest
  • jupyter/base-notebook:hub-2.2.2
  • jupyter/datascience-notebook
  • jupyter/base-notebook
  • jupyterhub/singleuser:2.2.2
  • jupyter/base-notebook

Defining get_ip_and_port allow the container to be spawn BUT it can’t reach the hub and the final output of the command:

sudo jupyterhub -f jupyterhub_config.py
[I 2022-03-21 16:16:28.591 JupyterHub app:2769] Running JupyterHub version 2.2.0
[I 2022-03-21 16:16:28.592 JupyterHub app:2799] Using Authenticator: jupyterhub.auth.DummyAuthenticator-2.2.0
[I 2022-03-21 16:16:28.592 JupyterHub app:2799] Using Spawner: builtins.CustomSpawner
[I 2022-03-21 16:16:28.592 JupyterHub app:2799] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.2.0
[I 2022-03-21 16:16:28.595 JupyterHub app:1606] Loading cookie_secret from /etc/jupyterhub/jupyterhub_cookie_secret
[I 2022-03-21 16:16:28.663 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2022-03-21 16:16:28.670 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
[I 2022-03-21 16:16:28.687 JupyterHub app:2838] Initialized 0 spawners in 0.002 seconds
[W 2022-03-21 16:16:28.688 JupyterHub app:2588] Deleting OAuth client jupyterhub-user-mattia
[W 2022-03-21 16:16:28.739 JupyterHub proxy:687] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2022-03-21 16:16:28.739 JupyterHub proxy:691] Starting proxy @ http://:8000
16:16:29.114 [ConfigProxy] info: Proxying http://*:8000 to (no default)
16:16:29.116 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
16:16:29.727 [ConfigProxy] info: 200 GET /api/routes
[I 2022-03-21 16:16:29.727 JupyterHub app:3087] Hub API listening on http://127.0.0.1:8081/hub/
16:16:29.729 [ConfigProxy] info: 200 GET /api/routes
[I 2022-03-21 16:16:29.729 JupyterHub proxy:431] Adding route for Hub: / => http://127.0.0.1:8081
16:16:29.731 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
16:16:29.731 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
16:16:29.731 [ConfigProxy] info: 201 POST /api/routes/
[I 2022-03-21 16:16:29.732 JupyterHub app:3154] JupyterHub is now running at http://:8000
[I 2022-03-21 16:16:34.675 JupyterHub log:189] 302 GET / -> /hub/ (@::1) 0.96ms
[I 2022-03-21 16:16:34.698 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn (mattia@::1) 16.78ms
[I 2022-03-21 16:16:34.714 JupyterHub roles:482] Adding role server to token: <APIToken('98a6...', user='mattia', client_id='jupyterhub')>
[I 2022-03-21 16:16:34.726 JupyterHub provider:607] Creating oauth client jupyterhub-user-mattia
[I 2022-03-21 16:16:34.763 JupyterHub dockerspawner:1281] Found existing container jupyter-mattia (id: 10f090c)
[I 2022-03-21 16:16:34.763 JupyterHub dockerspawner:1296] Starting container jupyter-mattia (id: 10f090c)
[I 2022-03-21 16:16:35.713 JupyterHub log:189] 302 GET /hub/spawn -> /hub/spawn-pending/mattia (mattia@::1) 1004.88ms
[I 2022-03-21 16:16:35.727 JupyterHub pages:401] mattia is pending spawn
[I 2022-03-21 16:16:35.754 JupyterHub log:189] 200 GET /hub/spawn-pending/mattia (mattia@::1) 29.72ms
[I 2022-03-21 16:16:37.759 JupyterHub provider:609] Updating oauth client jupyterhub-user-mattia
[W 2022-03-21 16:16:44.727 JupyterHub base:1086] User mattia is slow to become responsive (timeout=10)
[E 2022-03-21 16:17:09.830 JupyterHub gen:623] Exception in Future <Task finished name='Task-17' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:945> exception=TimeoutError("Server at http://127.0.0.1:8888/user/mattia/ didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
        future.result()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 952, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 845, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 889, in _wait_up
        raise e
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 859, in _wait_up
        resp = await server.wait_up(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 241, in wait_for_http_server
        re = await exponential_backoff(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 189, in exponential_backoff
        raise asyncio.TimeoutError(fail_message)
    asyncio.exceptions.TimeoutError: Server at http://127.0.0.1:8888/user/mattia/ didn't respond in 30 seconds

The container spawned cannot comunicate to jupyterhub.
Checking the logs of the container spawned I have:

WARNING: using start-singleuser.sh instead of start-notebook.sh to start a server associated with JupyterHub.
Entered start.sh with args: jupyterhub-singleuser --ip=0.0.0.0
WARNING: container must be started as root to change the desired user's name with NB_USER="mattia"!
Executing the command: jupyterhub-singleuser --ip=0.0.0.0
[I 2022-03-21 15:22:00.325 SingleUserLabApp mixins:614] Starting jupyterhub single-user server version 2.2.2
[I 2022-03-21 15:22:00.326 SingleUserLabApp mixins:628] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 3.3.1
[I 2022-03-21 15:22:00.326 SingleUserLabApp mixins:628] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.13.5
[W 2022-03-21 15:22:00.340 SingleUserLabApp configurable:193] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[W 2022-03-21 15:22:00.358 SingleUserLabApp configurable:193] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2022-03-21 15:22:00.359 SingleUserLabApp manager:345] jupyterlab | extension was successfully linked.
[W 2022-03-21 15:22:00.362 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2022-03-21 15:22:00.362 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2022-03-21 15:22:00.362 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2022-03-21 15:22:00.369 SingleUserLabApp configurable:193] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2022-03-21 15:22:00.369 SingleUserLabApp manager:345] nbclassic | extension was successfully linked.
[I 2022-03-21 15:22:00.516 SingleUserLabApp manager:345] notebook_shim | extension was successfully linked.
[I 2022-03-21 15:22:00.533 SingleUserLabApp manager:367] notebook_shim | extension was successfully loaded.
[I 2022-03-21 15:22:00.534 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[I 2022-03-21 15:22:00.534 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2022-03-21 15:22:00.537 SingleUserLabApp manager:367] jupyterlab | extension was successfully loaded.
[I 2022-03-21 15:22:00.540 SingleUserLabApp manager:367] nbclassic | extension was successfully loaded.
[I 2022-03-21 15:22:00.541 SingleUserLabApp mixins:640] Starting jupyterhub-singleuser server version 2.2.2
[E 2022-03-21 15:22:00.542 SingleUserLabApp mixins:489] Failed to connect to my Hub at http://127.0.0.1:8081/hub/api (attempt 1/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/mixins.py", line 487, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2022-03-21 15:22:02.546 SingleUserLabApp mixins:489] Failed to connect to my Hub at http://127.0.0.1:8081/hub/api (attempt 2/5). Is it running?

Can anyone help me understanding what i’m doing wrong? Or suggestion on how to solve it… Thanks a lot :slight_smile:

2

It looks like your spawned container is trying to connect to JupyterHub on 127.0.0.1 which is obviously incorrect. You’ll need to configure JupyterHub to listen on an interface that other containers can access, and to use an IP that resolves to the JupyterHub host. For example

3

No luck so far…

c.JupyterHub.hub_ip="0.0.0.0"
c.JupyterHub.hub_connect_ip = <ip.add>

The ip address should be localhost. The ip retrived with

cat /etc/resolv.conf

isn’t working. I tried to launch a jupyterhub/singleuser container and make a curl call inside the container with

curl http://host.docker.internal:8081/hub/api

and it works retriving the version of the hub. But if i use “host.docker.internal” as hub_connect_ip I won’t be able to connect througth localhost…
c.JupyterHub.hub_ip=“0.0.0.0”
c.JupyterHub.hub_connect_ip = <ip.add>
seems to have no effect inside the spawned container… the singleuser cmd it’s always try to connect to 127.0.0.1:8081
Must I use a docker network in order to use it? Jupyterhub it’s not running in a container. It’s vanilla pip install in ubuntu and then launch the comand.

4

Update:

# this work
c.JupyterHub.spawner_class = DockerSpawner
## this also work
# c.JupyterHub.spawner_class = SystemUserSpawner

# this WON'T work
c.JupyterHub.spawner_class = CustomSpawner

c.JupyterHub.authenticator_class = "dummy"
c.JupyterHub.hub_ip="192.168.233.166"

This code will spawn a container as expected