SwarmSpawner -- JupyterHub can't connect to user servers

JupyterHub
1

I’m trying to setup a jupyterhub using dockerspawner.SwarmSpawner to distribute user servers among multiple compute nodes in a docker swarm. I’m having trouble spawning user servers and I’m not sure whether it’s a bug or a missing configuration option.

The hub is able to spawn the user servers and the user servers are able to reach the hub api, but the hub is not able to access the user servers and eventually shuts them down after they time out. On the web interface users are told that their server has failed to spawn.

The hub’s container can see the open port on the user server container, but doesn’t see any open ports on the user server’s docker swarm service. The hub appears to be searching for the user server at the user’s swarm service address, not at the container address.

I have included my configuration files below.

deployment_minimal.yml

version: "3.5"
services:
  hub:
    image: jupyterhub/jupyterhub
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/etc/jupyterhub:/etc/jupyterhub"
      - "/srv/jupyterhub:/srv/jupyterhub"
    networks:
      - jupyterhub-net
    ports:
      - target: 8443
        published: 8443
        protocol: tcp
        mode: "host"
      - "8081:8081"
    command: bash -c '(pip install dockerspawner) && jupyterhub -f ../../etc/jupyterhub/jupyterhub_minimal_config.py --debug'
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
networks:
  jupyterhub-net:
    driver: overlay

jupyterhub_minimal_config.py

c.JupyterHub.bind_url = 'https://:8443/cds21/'
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.ssl_cert = '/srv/jupyterhub/server.crt'
c.JupyterHub.ssl_key = '/srv/jupyterhub/server.key'
c.JupyterHub.authenticator_class = 'jupyterhub.auth.DummyAuthenticator'
c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'

c.Spawner.default_url = '/lab'
c.Spawner.image = 'jupyter/datascience-notebook:latest'
c.Spawner.pull_policy = 'ifnotpresent'
c.Spawner.start_timeout = 600
c.Spawner.http_timeout = 600
c.Spawner.hub_ip_connect = 'swarm_hub'
c.Spawner.network_name = 'swarm_jupyterhub-net'
c.Spawner.extra_host_config = {'network_mode': 'swarm_jupyterhub-net'}
2

After much further investigation, I finally noticed that the user server containers had no trouble seeing things at service addresses but the hub couldn’t even see open ports on its own service address.

I rebooted the entire swarm and everything worked properly. (I suspect the manager node was the only problem.)