JupyterHub redirect loop unknown version using SwarmSpawner

JupyterHub
,
1

I am attempting to configure JupyterHub with a SwarmSpawner, and I would like to get a functioning server that properly connects to the notebook swarm docker instances it launches. Upon accessing the URL of the JupyterHub server at the default port 8081 and spawning the image, the URL alternates between /hub/user/fewf/ and /user/fewf/ until it results in a 500 redirect loop detected error. I have explored other posts on this issue, but they did not seem to be relevant to my problem. This machine is the swarm manager, although I will later add others on the local network. Jupyterhub is running in a docker image with /var/run/docker.sock mounted. The relevant messages from jupyterhub are as follows:

> docker run -v /var/run/docker.sock:/var/run/docker.sock -p 8000:8000 -p 8001:8001 -p 8081:8081 --network swarm_jupyterhub_net --rm jupyterswarmserver

[I 2023-10-20 22:50:34.424 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2023-10-20 22:50:34.424 JupyterHub app:2889] Using Authenticator: jupyterhub.auth.DummyAuthenticator-4.0.2
[I 2023-10-20 22:50:34.424 JupyterHub app:2889] Using Spawner: builtins.KnoxLabSpawner
[I 2023-10-20 22:50:34.424 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[I 2023-10-20 22:50:34.433 JupyterHub app:1709] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
[I 2023-10-20 22:50:34.453 alembic.runtime.migration migration:213] Context impl SQLiteImpl.
[I 2023-10-20 22:50:34.453 alembic.runtime.migration migration:216] Will assume non-transactional DDL.
[I 2023-10-20 22:50:34.462 alembic.runtime.migration migration:621] Running stamp_revision  -> 0eee8c825d24
[I 2023-10-20 22:50:34.602 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2023-10-20 22:50:34.632 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[I 2023-10-20 22:50:34.644 JupyterHub app:2928] Initialized 0 spawners in 0.001 seconds
[I 2023-10-20 22:50:34.647 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2023-10-20 22:50:34.647 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2023-10-20 22:50:34.648 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.thirty_days
[W 2023-10-20 22:50:34.648 JupyterHub proxy:746] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2023-10-20 22:50:34.648 JupyterHub proxy:750] Starting proxy @ http://0.0.0.0:8000/
22:50:34.794 [ConfigProxy] info: Proxying http://0.0.0.0:8000 to (no default)
22:50:34.796 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
22:50:35.538 [ConfigProxy] info: 200 GET /api/routes 
[I 2023-10-20 22:50:35.538 JupyterHub app:3178] Hub API listening on http://0.0.0.0:8081/hub/
[I 2023-10-20 22:50:35.538 JupyterHub app:3180] Private Hub API connect url http://172.17.0.1:8081/hub/
22:50:35.539 [ConfigProxy] info: 200 GET /api/routes 
22:50:35.540 [ConfigProxy] info: Adding route / -> http://172.17.0.1:8081
22:50:35.540 [ConfigProxy] info: Route added / -> http://172.17.0.1:8081
22:50:35.541 [ConfigProxy] info: 201 POST /api/routes/ 
[I 2023-10-20 22:50:35.539 JupyterHub proxy:477] Adding route for Hub: / => http://172.17.0.1:8081
[I 2023-10-20 22:50:35.541 JupyterHub app:3245] JupyterHub is now running at http://0.0.0.0:8000/
[I 2023-10-20 22:50:38.936 JupyterHub log:191] 302 GET / -> /hub/ (@10.5.6.86) 0.70ms
[W 2023-10-20 22:50:38.946 JupyterHub base:415] Invalid or expired cookie token
[I 2023-10-20 22:50:38.946 JupyterHub log:191] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@10.5.6.86) 0.99ms
[I 2023-10-20 22:50:38.984 JupyterHub log:191] 200 GET /hub/login?next=%2Fhub%2F (@10.5.6.86) 29.87ms
[I 2023-10-20 22:50:41.073 JupyterHub roles:238] Adding role user for User: fewf
[I 2023-10-20 22:50:41.104 JupyterHub base:837] User logged in: fewf
[I 2023-10-20 22:50:41.104 JupyterHub log:191] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (fewf@10.5.6.86) 41.87ms
[I 2023-10-20 22:50:41.129 JupyterHub log:191] 302 GET /hub/ -> /hub/spawn (fewf@10.5.6.86) 15.45ms
[I 2023-10-20 22:50:41.176 JupyterHub provider:659] Creating oauth client jupyterhub-user-fewf
[I 2023-10-20 22:50:41.202 JupyterHub dockerspawner:988] Service 'jupyter-fewf' is gone
[I 2023-10-20 22:50:42.142 JupyterHub log:191] 302 GET /hub/spawn -> /hub/spawn-pending/fewf (fewf@10.5.6.86) 1003.30ms
[I 2023-10-20 22:50:42.211 JupyterHub dockerspawner:1272] Created service jupyter-fewf (id: 8qawhl4) from image jupyterhub/singleuser
[I 2023-10-20 22:50:42.211 JupyterHub dockerspawner:1296] Starting service jupyter-fewf (id: 8qawhl4)
[I 2023-10-20 22:50:42.215 JupyterHub pages:398] fewf is pending spawn
[I 2023-10-20 22:50:42.219 JupyterHub log:191] 200 GET /hub/spawn-pending/fewf (fewf@10.5.6.86) 5.93ms
[I 2023-10-20 22:50:43.581 JupyterHub log:191] 200 GET /hub/api (@172.18.0.1) 0.59ms
[I 2023-10-20 22:50:43.620 JupyterHub log:191] 200 POST /hub/api/users/fewf/activity (fewf@172.18.0.1) 24.10ms
22:50:50.389 [ConfigProxy] info: Adding route /user/fewf -> http://jupyter-fewf:8888
22:50:50.389 [ConfigProxy] info: Route added /user/fewf -> http://jupyter-fewf:8888
[W 2023-10-20 22:50:50.387 JupyterHub _version:37] Single-user server has no version header, which means it is likely < 0.8. Expected 4.0.2
[I 2023-10-20 22:50:50.387 JupyterHub base:990] User fewf took 9.246 seconds to start
[I 2023-10-20 22:50:50.387 JupyterHub proxy:330] Adding user fewf to proxy /user/fewf/ => http://jupyter-fewf:8888
22:50:50.390 [ConfigProxy] info: 201 POST /api/routes/user/fewf 
[I 2023-10-20 22:50:50.390 JupyterHub users:768] Server fewf is ready
[I 2023-10-20 22:50:50.390 JupyterHub log:191] 200 GET /hub/api/users/fewf/server/progress?_xsrf=[secret] (fewf@10.5.6.86) 8096.85ms
[I 2023-10-20 22:50:50.420 JupyterHub log:191] 302 GET /hub/spawn-pending/fewf -> /user/fewf/ (fewf@10.5.6.86) 1.94ms
[I 2023-10-20 22:50:50.431 JupyterHub log:191] 302 GET /user/fewf/ -> /hub/user/fewf/ (@10.5.6.86) 0.45ms
[I 2023-10-20 22:50:50.444 JupyterHub log:191] 302 GET /hub/user/fewf/ -> /user/fewf/?redirects=1 (fewf@10.5.6.86) 1.80ms
[I 2023-10-20 22:50:50.454 JupyterHub log:191] 302 GET /user/fewf/?redirects=1 -> /hub/user/fewf/?redirects=1 (@10.5.6.86) 0.43ms
[W 2023-10-20 22:50:50.465 JupyterHub base:1656] Redirect loop detected on /hub/user/fewf/?redirects=1
[I 2023-10-20 22:50:52.466 JupyterHub log:191] 302 GET /hub/user/fewf/?redirects=1 -> /user/fewf/?redirects=2 (fewf@10.5.6.86) 2002.16ms
[I 2023-10-20 22:50:52.554 JupyterHub log:191] 302 GET /user/fewf/?redirects=2 -> /hub/user/fewf/?redirects=2 (@10.5.6.86) 0.64ms
[W 2023-10-20 22:50:52.567 JupyterHub web:1869] 500 GET /hub/user/fewf/?redirects=2 (10.5.6.86): Redirect loop detected. Notebook has jupyterhub version unknown (likely < 0.8), but the Hub expects 4.0.2. Try installing jupyterhub==4.0.2 in the user environment if you continue to have problems.
[E 2023-10-20 22:50:52.597 JupyterHub log:183] {
      "Host": "10.32.5.19:8081",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/118.0",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept-Encoding": "gzip, deflate",
      "Referer": "http://10.32.5.19:8081/hub/login?next=%2Fhub%2F",
      "Dnt": "1",
      "Connection": "keep-alive",
      "Cookie": "_xsrf=[secret]; jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
      "Upgrade-Insecure-Requests": "1"
    }
[E 2023-10-20 22:50:52.597 JupyterHub log:191] 500 GET /hub/user/fewf/?redirects=2 (fewf@10.5.6.86) 32.48ms
22:55:35.544 [ConfigProxy] info: 200 GET /api/routes 
[I 2023-10-20 22:56:12.569 JupyterHub log:191] 200 POST /hub/api/users/fewf/activity (fewf@172.18.0.1) 14.30ms
22:57:31.763 [ConfigProxy] warn: Terminated
// This is printed as the args and kwargs from the subclassed KnoxLabSpawner constructor
KnoxLabSpawner() () {'user': <User(fewf 0/1 running)>, 'orm_spawner': <jupyterhub.orm.Spawner object at 0x7fd401666d70>, 'hub': <Hub 0.0.0.0:8081>, 'authenticator': <jupyterhub.auth.DummyAuthenticator object at 0x7fd4009c6050>, 'config': {'ConfigurableHTTPProxy': {'should_start': True}, 'JupyterHub': {'authenticator_class': <class 'jupyterhub.auth.DummyAuthenticator'>, 'hub_connect_ip': '172.17.0.1', 'ip': '0.0.0.0', 'hub_ip': '0.0.0.0', 'debug': True, 'spawner_class': <class 'KnoxLabSpawner'>}, 'SwarmSpawner': {'http_timeout': 300, 'start_timeout': 300}, 'NotebookApp': {'ip': '0.0.0.0', 'allow_origin': '*'}}, 'proxy_spec': '/user/fewf/', '_deprecated_db_session': <sqlalchemy.orm.session.Session object at 0x7fd40025d570>, 'oauth_client_id': 'jupyterhub-user-fewf', 'cookie_options': {}, 'trusted_alt_names': [], 'user_options': {}}

Note: 172.17.0.1 is my docker0 interface IP.

My docker network is configured as swarm_jupyterhub_net overlay swarm
Jupyterhub version: 4.0.2 (latest)
jupyterhub/singleuser version: latest

My jupyterhub_config.py is as follows:

import jupyterhub
import os
import socket
import sys
import dockerspawner
from jupyterhub.auth import DummyAuthenticator
c = get_config()
c.ConfigurableHTTPProxy.should_start = True
c.JupyterHub.authenticator_class = DummyAuthenticator

class KnoxLabSpawner(dockerspawner.SwarmSpawner):
  def __init__(self, *args, **kwargs):
    super(KnoxLabSpawner, self).__init__(*args, **kwargs)
    print('KnoxLabSpawner()', args, kwargs)
    self.use_internal_ip = True
    self.debug = True
    self.remove = True
    self.remove_containers = self.remove
    self.network_name = "swarm_jupyterhub_net"
    self.extra_host_config = { 'network_mode': "swarm_jupyterhub_net"}
    self.image = 'jupyterhub/singleuser'
    self.cmd = ["jupyterhub-singleuser"]
    #self.hostname = os.environ['HOSTNAME']
    #self.default_url = '/'
    #self.use_internal_hostname = False
    #self.port = 8888
    #self.host_ip='0.0.0.0'
    #self.ip = '0.0.0.0'

c.JupyterHub.spawner_class = KnoxLabSpawner

c.JupyterHub.ip = '0.0.0.0' # Main proxy listening ip
c.JupyterHub.hub_ip = '0.0.0.0'  # Single user server api connection ip
c.JupyterHub.debug = True

c.JupyterHub.ssl_key = "/srv/jupyterhub/certificate/key.pem"
c.JupyterHub.ssl_cert = "/srv/jupyterhub/certificate/cert.pem"

c.SwarmSpawner.http_timeout = 300
c.SwarmSpawner.start_timeout = 300
2

This must be an address that singleuser servers can access the hub on, i.e. a valid IP that isn’t localhost such as the external IP. 0.0.0.0 usually means “listen on all interfaces”, but you can’t connect to it.

3

My understanding was that hub_ip is the ip the the ip address that the hub API binds to, and hub_connect_ip was the ip the containers query to attempt to reach the API on the host. However, I admit I am still a little confused over the roles of each. On the Networking basics page for JupyterHub, it gives the example configuration:

c.JupyterHub.hub_ip = '0.0.0.0'  # listen on all interfaces
c.JupyterHub.hub_connect_ip = '10.0.1.4'  # IP as seen on the docker network. Can also be a hostname.

After setting hub_ip to the LAN address of the server as you suggested, and hub_connect_ip to the ip of the docker0 interface, I get:

18:47:28.790 [ConfigProxy] info: Proxying https://0.0.0.0:8000 to (no default)
18:47:28.792 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
18:47:28.993 [ConfigProxy] info: 200 GET /api/routes 
[E 2023-10-21 18:47:28.993 JupyterHub app:3182] Failed to bind hub to http://172.17.0.1:8081/hub/

As if the inverse of my understanding were true, I also tried setting hub_ip to the LAN address of the server and hub_connect_ip to 0.0.0.0, which resulted in the containers not being able to connect to “0.0.0.0”.

4

Sorry, you’re right about hub_ip, 0.0.0.0 should work. Your original config didn’t have hub_connect_ip though- was it missing, or was that only part of your config?

Have you tried using DockerSpawner on a single node to get things working, before moving to SwarmSpawner? There’s an example in

which is tested in CI.

Also, I just noticed you said

Upon accessing the URL of the JupyterHub server at the default port 8081

Port 8081 is for internal use only. Users should be using port 8000.

5

Sorry, I forgot to include hub_connect_ip in my sample. It was set to the ip of the docker0 interface, 172.17.0.1. I tried switching between DockerSpawner and SwarmSpawner on that node, but still got the same redirect loop error. I have read other posts that say that this problem can be solved by connecting to port 8000 for the interface, and I have left this as default, but connecting to that port yields no data.

> nc -zv 10.32.5.19 8000
Connection to 10.32.5.19 port 8000 [tcp/irdmi] succeeded!
> curl http://10.32.5.19:8000/
curl: (52) Empty reply from server

However, I just tried https://10.32.5.19:8000, and it works correctly! This also means my certificate key is working correctly. I will mark this issue as solved.