Hello Guys,
i am posting this, because i really need help with deploying jupyterhub in swarm mode.
What I have:
- Three VMs in certain Cloud provider
- Swarm cluster (1x Manager 2x Workers)
My aim:
I want to deploy Jupyterhub on Manager VM in Swarm cluster network. The users log into Jupyterhub. Jupyterhub uses Swarmspawner to start single user Server for users on worker nodes in swarm network. Users will be redirected to single user server after logging in.
What I have tried:
1. Created a Swarm Cluster with 3 VMs.
2. Created a overlay network for Swarm called jupyter_network
3. Files and directories structure:
./
+-- docker-compose.yml
+-- jupyterhub/
| +-- Dockerfile
| +-- jupyter-config.py
+-- traefik.toml
+-- reverse-proxy/
+-- traefik.toml
4. Self configured Jupyterhub image using Dockerfile and jupyterhub_config.py:
Dockerfile for jupyterhub:
FROM quayio/jupyterhub/jupyterhub:latest
COPY jupyterhub_config.py /srv/jupyterhub/jupyterhub_config.py
COPY traefik.toml /srv/jupyterhub/traefik.tomlRUN pip install
dockerspawner
jupyterhub-firstuseauthenticator
jupyterhub-traefik-proxy
jupyterhub-idle-culler
And jupyterhub_config.py
import os
c = get_config()
c.JupyterHub.cleanup_servers = False
c.JupyterHub.proxy_class = “traefik_toml”
c.TraefikTomlProxy.should_start = False
c.TraefikTomlProxy.toml_static_config_file=“/srv/jupyterhub/traefik.toml”
c.TraefikTomlProxy.traefik_api_username = “abc”
c.TraefikTomlProxy.traefik_api_password = "123
c.JupyterHub.hub_ip = ‘0.0.0.0’
c.Authenticator.allowed_users = {“xx”}
c.JupyterHub.admin_access = True
c.JupyterHub.authenticator_class = “dummy”
c.DummyAuthenticator.password = “xxx”
c.JupyterHub.spawner_class = ‘dockerspawner.SwarmSpawner’
c.SwarmSpawner.image = ‘Quay’
network_name = ‘jupyterhub_network’
c.SwarmSpawner.network_name = network_name
5. Reverse proxy using Traefik (toml file looks like this:)
debug = false
logLevel = “ERROR”
defaultEntryPoints = [“https”,“http”]#Redirect HTTP → HTTPS, install certificates
[entryPoints]
[entryPoints.http]
address = “:80”[entryPoints.https]
address = “:443”
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
certFile = “/etc/certs/fullchain.pem”
keyFile = “/etc/certs/privkey.pem”
[http.middlewares.redirect-to-https.redirectScheme]
scheme = “https”[http.routers]
[http.routers.router1]
rule = “Host(xxx.com)”
service = “service1”
entryPoints = [“http”]
middlewares = [“redirect-to-https”]
#Activate docker API
[docker]
domain = “docker.local”
watch = true#Activate Traefik dashboard
[api]
dashboard = true
6. Docker compose file:
version: “3.8”
services:
jupyterhub:
#jupyterhub_image, which in step 4 created
image: jupyterhub_image volumes: - /var/run/docker.sock:/var/run/docker.sock labels: - "traefik.enable=true" - "traefik.frontend.rule=Host:${HOST}" deploy: placement: constraints: - node.id == <manager id> networks: - jupyterhub_networktraefik:
image: traefik:v2.5 command: - "--api.insecure=true" - "--providers.docker=true" - "--providers.docker.exposedbydefault=false" - "--entrypoints.web.address=:80" ports: - "80:80" - "443:443" - "8080:8080" volumes: - /var/run/docker.sock:/var/run/docker.sock - ./reverse-proxy/traefik.toml:/etc/traefik/traefik.toml - /etc/letsencrypt/live/xxxxx/etc/certs networks: - jupyterhub_network
Test deployment with docker compoes build/up
The result i got is like this:
WARN[0000] The “HOST” variable is not set. Defaulting to a blank string.
[+] Running 4/4
Network jupyter-dhbw_jupyterhub_network Created 0.1s 0.4s
Attaching to jupyterhub-1, traefik-1
traefik-1 | 2024/02/28 13:47:17 command traefik error: field not found, node: tls
traefik-1 exited with code 1
jupyterhub-1 | [I 2024-02-28 13:47:19.828 JupyterHub app:2859] Running JupyterHub version 4.0.2
jupyterhub-1 | [I 2024-02-28 13:47:19.828 JupyterHub app:2889] Using Authenticator: jupyterhub.auth.DummyAuthenticator-4.0.2
jupyterhub-1 | [I 2024-02-28 13:47:19.828 JupyterHub app:2889] Using Spawner: dockerspawner.swarmspawner.SwarmSpawner-13.0.0
jupyterhub-1 | [I 2024-02-28 13:47:19.828 JupyterHub app:2889] Using Proxy: jupyterhub_traefik_proxy.toml.TraefikTomlProxy-1.1.0
jupyterhub-1 | [I 2024-02-28 13:47:19.845 JupyterHub app:1709] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
jupyterhub-1 | [I 2024-02-28 13:47:19.889 alembic.runtime.migration migration:213] Context impl SQLiteImpl.
jupyterhub-1 | [I 2024-02-28 13:47:19.889 alembic.runtime.migration migration:216] Will assume non-transactional DDL.
jupyterhub-1 | [I 2024-02-28 13:47:19.911 alembic.runtime.migration migration:621] Running stamp_revision → 0eee8c825d24
jupyterhub-1 | [W 2024-02-28 13:47:20.275 JupyterHub proxy:142] TraefikTomlProxy.toml_static_config_file is deprecated in TraefikTomlProxy 1.0, use TraefikTomlProxy.static_config_file instead
jupyterhub-1 | [W 2024-02-28 13:47:20.276 JupyterHub toml:18] TraefikTomlProxy is deprecated in jupyterhub-traefik-proxy 1.0. Use `c.JupyterHub.proxy_class = ‘traefik_file’
jupyterhub-1 | [I 2024-02-28 13:47:20.476 JupyterHub fileprovider:99] Creating the dynamic configuration file: rules.toml
jupyterhub-1 | [I 2024-02-28 13:47:20.498 JupyterHub app:2928] Initialized 0 spawners in 0.022 seconds
jupyterhub-1 | [I 2024-02-28 13:47:20.504 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
jupyterhub-1 | [I 2024-02-28 13:47:20.505 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.seven_days
jupyterhub-1 | [I 2024-02-28 13:47:20.506 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.thirty_days
jupyterhub-1 | [I 2024-02-28 13:47:20.506 JupyterHub app:3142] Not starting proxy
jupyterhub-1 | [E 2024-02-28 13:47:51.467 JupyterHub app:3382]
jupyterhub-1 | Traceback (most recent call last):
jupyterhub-1 | File “/usr/local/lib/python3.10/dist-packages/jupyterhub/app.py”, line 3380, in launch_instance_async
jupyterhub-1 | await self.start()
jupyterhub-1 | File “/usr/local/lib/python3.10/dist-packages/jupyterhub/app.py”, line 3146, in start
jupyterhub-1 | await self.proxy.get_all_routes()
jupyterhub-1 | File “/usr/local/lib/python3.10/dist-packages/jupyterhub_traefik_proxy/proxy.py”, line 754, in get_all_routes
jupyterhub-1 | await self._start_future
jupyterhub-1 | File “/usr/local/lib/python3.10/dist-packages/jupyterhub_traefik_proxy/proxy.py”, line 587, in _start_external
jupyterhub-1 | await self._wait_for_static_config()
jupyterhub-1 | File “/usr/local/lib/python3.10/dist-packages/jupyterhub_traefik_proxy/proxy.py”, line 427, in _wait_for_static_config
jupyterhub-1 | await exponential_backoff(
jupyterhub-1 | File “/usr/local/lib/python3.10/dist-packages/jupyterhub/utils.py”, line 237, in exponential_backoff
jupyterhub-1 | raise asyncio.TimeoutError(fail_message)
jupyterhub-1 | asyncio.exceptions.TimeoutError: Traefik static configuration not available
jupyterhub-1 |
jupyterhub-1 exited with code 1