Cannot run docker on another host, separate from JupyterHub

JupyterHub
1

Hello!

I have a JupyerHub installation on a Debian host [1] and a docker installed on a CentOS host [2]. What I would like to do is that JupyterHub loads its containers from the docker host, not from the docker from the JupyterHub instance. My configuration is in [3]. The host name was changed on purpose.

My docker context is properly pointing to the remote host and all related commands work on the remote host [4].

But whenever Jupyter tries to start a container for a logged user, I get the error on [5].

What I am missing? Thank you!

[1]

root@jupyterhub:/etc/jupyterhub# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye

[2]

[root@docker]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

[3]

c.JupyterHub.spawner_class = ‘dockerspawner.SystemUserSpawner’
c.DockerSpawner.extra_host_config = {‘network_mode’: ‘host’}
c.DockerSpawner.use_internal_hostname = True
c.DockerSpawner.network_name = ‘docker.domain.com’
c.DockerSpawner.host_ip = ‘10.0.100.238’
c.DockerSpawner.use_internal_ip = False
c.DockerSpawner.allowed_images = {
‘python’: ‘jupyterhub/singleuser’,
‘tensorflow’: ‘jupyter/tensorflow-notebook’
}
c.JupyterHub.ssl_cert = ‘/etc/jupyterhub/JupyterHub.crt’
c.JupyterHub.ssl_key = ‘/etc/jupyterhub/JupyterHub.key’
c.JupyterHub.hub_ip = ‘0.0.0.0’
c.JupyterHub.hub_connect_ip = ‘10.0.100.249’

[4]

root@jupyterhub:/etc/jupyterhub# docker context ls
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default Current DOCKER_HOST based configuration unix:///var/run/docker.sock swarm
remote * ssh://root@docker.domain.com

[5]

Spawn failed: 500 Server Error for http+docker://localhost/v1.41/containers/034449544b763f4bca27fd00a6f436dd529c53e1abfd00a255ee093853303b38/start: Internal Server Error (“driver failed programming external connectivity on endpoint jupyter-rvaladao (adc4915d20c74f69e1b245c18084a877b7d0d2eb3d299e94f9588e55a011d240): Error starting userland proxy: listen tcp4 10.0.100.238:49323: bind: cannot assign requested address”)

2

DockerSpawner uses the docker-py library. Are you setting DOCKER_HOST? Client — Docker SDK for Python 5.0.3 documentation

According to Docker Context | Docker Documentation docker context is only relevant to the CLI

1 Like
3

Thank you!

We set it up as you suggested [1]. We tried with both SSH and TCP, but both returned the same error [2]. We also uninstalled docker from our JupyterHub host just to make sure it wasn’t interfering.

[1]

root@jupyterhub:~# echo $DOCKER_HOST
tcp://10.0.100.238:2375

[2]

Aug 26 11:32:07 jupyterhub jupyterhub[469]: [I 2022-08-26 11:32:07.994 JupyterHub provider:607] Creating oauth client jupyterhub-user-rvaladao
Aug 26 11:32:08 jupyterhub jupyterhub[469]: [E 2022-08-26 11:32:08.095 JupyterHub user:815] Unhandled error starting rvaladao’s server: Error while fetching server API version: (‘Connection aborted.’, FileNotFoundError(2, ‘No such file or directory’))
Aug 26 11:32:08 jupyterhub jupyterhub[469]: [E 2022-08-26 11:32:08.179 JupyterHub user:825] Failed to cleanup rvaladao’s server that failed to start
Aug 26 11:32:08 jupyterhub jupyterhub[469]: Traceback (most recent call last):
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/jupyterhub/user.py”, line 823, in spawn
Aug 26 11:32:08 jupyterhub jupyterhub[469]: await self.stop(spawner.name)
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/jupyterhub/user.py”, line 917, in stop
Aug 26 11:32:08 jupyterhub jupyterhub[469]: status = await spawner.poll()
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/dockerspawner/dockerspawner.py”, line 961, in poll
Aug 26 11:32:08 jupyterhub jupyterhub[469]: container = await self.get_object()
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/dockerspawner/dockerspawner.py”, line 984, in get_object
Aug 26 11:32:08 jupyterhub jupyterhub[469]: obj = await self.docker(“inspect_%s” % self.object_type, self.object_name)
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/lib/python3.9/concurrent/futures/thread.py”, line 52, in run
Aug 26 11:32:08 jupyterhub jupyterhub[469]: result = self.fn(*self.args, **self.kwargs)
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/dockerspawner/dockerspawner.py”, line 947, in _docker
Aug 26 11:32:08 jupyterhub jupyterhub[469]: m = getattr(self.client, method)
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/dockerspawner/dockerspawner.py”, line 122, in client
Aug 26 11:32:08 jupyterhub jupyterhub[469]: client = docker.APIClient(**kwargs)
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/docker/api/client.py”, line 197, in init
Aug 26 11:32:08 jupyterhub jupyterhub[469]: self._version = self._retrieve_server_version()
Aug 26 11:32:08 jupyterhub jupyterhub[469]: File “/usr/local/lib/python3.9/dist-packages/docker/api/client.py”, line 221, in _retrieve_server_version
Aug 26 11:32:08 jupyterhub jupyterhub[469]: raise DockerException(
Aug 26 11:32:08 jupyterhub jupyterhub[469]: docker.errors.DockerException: Error while fetching server API version: (‘Connection aborted.’, FileNotFoundError(2, ‘No such file or directory’))

4

How are you running JupyterHub? Is the environment variable definitely getting passed to JupyterHub?

5

You’re right, now I’m thinking it’s not being passed to JupyterHub. JupyterHub is running as a service [1].

I edited the code on [2], [3] to debug (I added the file-related lines), but it seems it never enters this part of the code, so I think DOCKER_HOST is never recognized by JupyterHub. Do you have any idea how to change that?

Thank you so much!

[1]

root@jupyterhub:/# systemctl status jupyterhub | grep active
Active: active (running) since Thu 2022-09-01 16:22:55 -03; 17h ago

[2]

root@jupyterhub:/usr/local/lib/python3.9/dist-packages/dockerspawner# ls -lah dockerspawner.py
-rw-r–r-- 1 root root 49K Sep 1 16:23 dockerspawner.py

[3]

image

6

Systemd passes a very limited set of environment variables to services.
You’ll need to add all other variables in your systemd service file.

1 Like
7

Thank you, @manics! That solved my problem. I put DOCKER_HOST env variable in my systemd service file. But after that, I also had to create the docker network, as it wasn’t created. Then, things started working correctly.

1 Like