JupyterHub + Podman compatibility

[jiri.t]

Hi,
I’m trying to deploy Jupyterhub with Podman instead of Docker (because I’m on RHEL).

I tried clean install of Podman 4.4.1 with Jupyterhub 4.0.2 and Dockerspawner 13.0.0 but I’m getting strange error.

[E 2024-02-13 15:43:13.897 JupyterHub user:894] Failed to cleanup <username>'s server that failed to start
    Traceback (most recent call last):
      File "<installPath>/envs/sys_jupyterhub/lib/python3.11/site-packages/jupyterhub/user.py", line 892, in spawn
        await self.stop(spawner.name)
      File "<installPath>/envs/sys_jupyterhub/lib/python3.11/site-packages/jupyterhub/user.py", line 986, in stop
        status = await spawner.poll()
                 ^^^^^^^^^^^^^^^^^^^^
      File "<installPath>/envs/sys_jupyterhub/lib/python3.11/site-packages/dockerspawner/dockerspawner.py", line 1006, in poll
        container = await self.get_object()
                    ^^^^^^^^^^^^^^^^^^^^^^^
      File "<installPath>/envs/sys_jupyterhub/lib/python3.11/site-packages/dockerspawner/dockerspawner.py", line 1029, in get_object
        obj = await self.docker("inspect_%s" % self.object_type, self.object_name)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<installPath>/envs/sys_jupyterhub/lib/python3.11/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<installPath>/envs/sys_jupyterhub/lib/python3.11/site-packages/dockerspawner/dockerspawner.py", line 992, in _docker
        m = getattr(self.client, method)
                    ^^^^^^^^^^^
      File "<installPath>/envs/sys_jupyterhub/lib/python3.11/site-packages/dockerspawner/dockerspawner.py", line 140, in client
        client = docker.APIClient(**kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<installPath>/envs/sys_jupyterhub/lib/python3.11/site-packages/docker/api/client.py", line 197, in __init__
        self._version = self._retrieve_server_version()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<installPath>/envs/sys_jupyterhub/lib/python3.11/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
        raise DockerException(
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

I redacted path and my username (using <>).

Is this info outdated ?
https://jupyterhub-dockerspawner.readthedocs.io/en/latest/spawner-types.html#using-podman

I’m running podman as service, yet JupyterHub refuse to work.

Can I ask for help ?
Is there any compatibility matrix between podman and jupyterhub ?
Or is my installation/config bad ? I’m using clean installation, with very basic config. If you managed to get Jupyterhub to work with Podman, can you share your config file ?

Thank you very much for any advice. J.T.

[manics]

@jiri.t

docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

This sounds like your podman socket is either missing, in a different location, or there’s some other configuraiton issue. Have you checked it exists?

Dockerspawner can definitely work with Podman, see for example

JupyterHub with Podman and DockerSpawner: Immediate 'JupyterLab Server Unavailable' Error on Login, Resolved by Refresh

Hi guys, I am using Jupyterhub with podman and DockerSpawner. When I log in, I immediately get an error message that the Jupyterlab server is not available: 503 : Service Unavailable Your server appears to be down. Try restarting it from the hub But once I refresh the page I can get access to the server with no problems. It is always the same process regardless of the user: [image] My assumption is that Jupyterhub tries to access the container server before it starts, it does not find it …

[jiri.t]

thank you for your insight.
It seems
export DOCKER_HOST=unix:///run/podman/podman.sock
fixed the problem with connecting to podman api.

Nevertheless I’m still getting some strange errors.

Does this mean, my image is bad and I should download a new one ? I’ve pick just basic all-spark-notebook from Docker

[D 2024-02-14 16:06:32.941 JupyterHub pages:217] Triggering spawn with default options for <userName>
[D 2024-02-14 16:06:32.941 JupyterHub base:961] Initiating spawn for <userName>
[D 2024-02-14 16:06:32.941 JupyterHub base:965] 0/100 concurrent spawns
[D 2024-02-14 16:06:32.941 JupyterHub base:970] 0 active servers
[I 2024-02-14 16:06:32.946 JupyterHub provider:659] Creating oauth client jupyterhub-user-<userName>
[D 2024-02-14 16:06:32.956 JupyterHub user:794] Calling Spawner.start for <userName>
[I 2024-02-14 16:06:32.957 JupyterHub dockerspawner:1275] pulling image quay.io/jupyterhub/singleuser:4.0
[D 2024-02-14 16:06:32.959 JupyterHub dockerspawner:1027] Getting container 'jupyter-<userName>'
[I 2024-02-14 16:06:32.960 JupyterHub dockerspawner:1033] Container 'jupyter-<userName>' is gone
[E 2024-02-14 16:06:32.963 JupyterHub user:884] Unhandled error starting <userName>'s server: 404 Client Error for http+docker://localhost/v1.41/images/quay.io/jupyterhub/singleuser:4.0/json: Not Found ("failed to find image quay.io/jupyterhub/singleuser:4.0: quay.io/jupyterhub/singleuser:4.0: No such image")
    Traceback (most recent call last):
      File "<condaEnvPath>/python3.11/site-packages/jupyterhub/user.py", line 798, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<condaEnvPath>/python3.11/site-packages/dockerspawner/dockerspawner.py", line 1309, in start
        obj = await self.create_object()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<condaEnvPath>/python3.11/site-packages/dockerspawner/dockerspawner.py", line 1164, in create_object
        command=(await self.get_command()),
                 ^^^^^^^^^^^^^^^^^^^^^^^^
      File "<condaEnvPath>/python3.11/site-packages/dockerspawner/dockerspawner.py", line 1058, in get_command
        image_info = await self.docker("inspect_image", self.image)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<condaEnvPath>/python3.11/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<condaEnvPath>/python3.11/site-packages/dockerspawner/dockerspawner.py", line 993, in _docker
        return m(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^
      File "<condaEnvPath>/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
        return f(self, resource_id, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<condaEnvPath>/python3.11/site-packages/docker/api/image.py", line 251, in inspect_image
        return self._result(
               ^^^^^^^^^^^^^
      File "<condaEnvPath>/python3.11/site-packages/docker/api/client.py", line 274, in _result
        self._raise_for_status(response)
      File "<condaEnvPath>/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status
        raise create_api_error_from_http_exception(e) from e
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<condaEnvPath>/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
        raise cls(e, response=response, explanation=explanation) from e
    docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/quay.io/jupyterhub/singleuser:4.0/json: Not Found ("failed to find image quay.io/jupyterhub/singleuser:4.0: quay.io/jupyterhub/singleuser:4.0: No such image")

[D 2024-02-14 16:06:32.964 JupyterHub user:982] Stopping <userName>

[manics]

Does the minimal example in

JupyterHub with Podman and DockerSpawner: Immediate 'JupyterLab Server Unavailable' Error on Login, Resolved by Refresh

This minimal config is working for me: # jupyterhub_config_podman.py c.JupyterHub.spawner_class = "docker" c.JupyterHub.authenticator_class = "dummy" c.Authenticator.admin_users = {'demo'} c.DockerSpawner.image = "quay.io/jupyter/base-notebook:latest" import socket s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.connect(("8.8.8.8", 80)) c.JupyterHub.hub_ip = s.getsockname()[0] s.close() c.ConfigurableHTTPProxy.debug = True Podman command: podman system service -t0 --log-level=debug …

work for you?

[jiri.t]

Thank you very much. This minimal config is able to spawn containers !

Your insight really helped me !