Cannot run docker on another host, separate from JupyterHub

Hello!

I have a JupyerHub installation on a Debian host [1] and a docker installed on a CentOS host [2]. What I would like to do is that JupyterHub loads its containers from the docker host, not from the docker from the JupyterHub instance. My configuration is in [3]. The host name was changed on purpose.

My docker context is properly pointing to the remote host and all related commands work on the remote host [4].

But whenever Jupyter tries to start a container for a logged user, I get the error on [5].

What I am missing? Thank you!


[1]

root@jupyterhub:/etc/jupyterhub# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye

[2]

[root@docker]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

[3]

c.JupyterHub.spawner_class = ‘dockerspawner.SystemUserSpawner’
c.DockerSpawner.extra_host_config = {‘network_mode’: ‘host’}
c.DockerSpawner.use_internal_hostname = True
c.DockerSpawner.network_name = ‘docker.domain.com
c.DockerSpawner.host_ip = ‘10.0.100.238’
c.DockerSpawner.use_internal_ip = False
c.DockerSpawner.allowed_images = {
‘python’: ‘jupyterhub/singleuser’,
‘tensorflow’: ‘jupyter/tensorflow-notebook’
}
c.JupyterHub.ssl_cert = ‘/etc/jupyterhub/JupyterHub.crt’
c.JupyterHub.ssl_key = ‘/etc/jupyterhub/JupyterHub.key’
c.JupyterHub.hub_ip = ‘0.0.0.0’
c.JupyterHub.hub_connect_ip = ‘10.0.100.249’

[4]

root@jupyterhub:/etc/jupyterhub# docker context ls
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default Current DOCKER_HOST based configuration unix:///var/run/docker.sock swarm
remote * ssh://root@docker.domain.com

[5]

Spawn failed: 500 Server Error for http+docker://localhost/v1.41/containers/034449544b763f4bca27fd00a6f436dd529c53e1abfd00a255ee093853303b38/start: Internal Server Error (“driver failed programming external connectivity on endpoint jupyter-rvaladao (adc4915d20c74f69e1b245c18084a877b7d0d2eb3d299e94f9588e55a011d240): Error starting userland proxy: listen tcp4 10.0.100.238:49323: bind: cannot assign requested address”)

DockerSpawner uses the docker-py library. Are you setting DOCKER_HOST? Client — Docker SDK for Python 5.0.3 documentation

According to Docker Context | Docker Documentation docker context is only relevant to the CLI

1 Like

Thank you!

We set it up as you suggested [1]. We tried with both SSH and TCP, but both returned the same error [2]. We also uninstalled docker from our JupyterHub host just to make sure it wasn’t interfering.


[1]

root@jupyterhub:~# echo $DOCKER_HOST
tcp://10.0.100.238:2375

[2]

Aug 26 11:32:07 jupyterhub jupyterhub[469]: [I 2022-08-26 11:32:07.994 JupyterHub provider:607] Creating oauth client jupyterhub-user-rvaladao
Aug 26 11:32:08 jupyterhub jupyterhub[469]: [E 2022-08-26 11:32:08.095 JupyterHub user:815] Unhandled error starting rvaladao’s server: Error while fetching server API version: (‘Connection aborted.’, FileNotFoundError(2, ‘No such file or directory’))
Aug 26 11:32:08 jupyterhub jupyterhub[469]: [E 2022-08-26 11:32:08.179 JupyterHub user:825] Failed to cleanup rvaladao’s server that failed to start
Aug 26 11:32:08 jupyterhub jupyterhub[469]: Traceback (most recent call last):
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/jupyterhub/user.py”, line 823, in spawn
Aug 26 11:32:08 jupyterhub jupyterhub[469]: await self.stop(spawner.name)
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/jupyterhub/user.py”, line 917, in stop
Aug 26 11:32:08 jupyterhub jupyterhub[469]: status = await spawner.poll()
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/dockerspawner/dockerspawner.py”, line 961, in poll
Aug 26 11:32:08 jupyterhub jupyterhub[469]: container = await self.get_object()
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/dockerspawner/dockerspawner.py”, line 984, in get_object
Aug 26 11:32:08 jupyterhub jupyterhub[469]: obj = await self.docker(“inspect_%s” % self.object_type, self.object_name)
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/lib/python3.9/concurrent/futures/thread.py”, line 52, in run
Aug 26 11:32:08 jupyterhub jupyterhub[469]: result = self.fn(*self.args, **self.kwargs)
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/dockerspawner/dockerspawner.py”, line 947, in _docker
Aug 26 11:32:08 jupyterhub jupyterhub[469]: m = getattr(self.client, method)
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/dockerspawner/dockerspawner.py”, line 122, in client
Aug 26 11:32:08 jupyterhub jupyterhub[469]: client = docker.APIClient(**kwargs)
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/docker/api/client.py”, line 197, in init
Aug 26 11:32:08 jupyterhub jupyterhub[469]: self._version = self._retrieve_server_version()
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/docker/api/client.py”, line 221, in _retrieve_server_version
Aug 26 11:32:08 jupyterhub jupyterhub[469]: raise DockerException(
Aug 26 11:32:08 jupyterhub jupyterhub[469]: docker.errors.DockerException: Error while fetching server API version: (‘Connection aborted.’, FileNotFoundError(2, ‘No such file or directory’))

How are you running JupyterHub? Is the environment variable definitely getting passed to JupyterHub?

You’re right, now I’m thinking it’s not being passed to JupyterHub. JupyterHub is running as a service [1].

I edited the code on [2], [3] to debug (I added the file-related lines), but it seems it never enters this part of the code, so I think DOCKER_HOST is never recognized by JupyterHub. Do you have any idea how to change that?

Thank you so much!


[1]

root@jupyterhub:/# systemctl status jupyterhub | grep active
Active: active (running) since Thu 2022-09-01 16:22:55 -03; 17h ago

[2]

root@jupyterhub:/usr/local/lib/python3.9/dist-packages/dockerspawner# ls -lah dockerspawner.py
-rw-r–r-- 1 root root 49K Sep 1 16:23 dockerspawner.py

[3]

image

Systemd passes a very limited set of environment variables to services.
You’ll need to add all other variables in your systemd service file.

1 Like

Thank you, @manics! That solved my problem. I put DOCKER_HOST env variable in my systemd service file. But after that, I also had to create the docker network, as it wasn’t created. Then, things started working correctly.

1 Like