JupyterHub 5.0 and batchspawner challenges

JupyterHub
,
1

I’ve been running JupyterHub inside a docker container fronting a SLURM cluster. nginx is sitting in front as a reverse proxy with SSL termination. I’ve actually been running two hubs on the same system, one dev and one production, with different CNAMEs and nginx virtual hosts. (with different hub configs, databases, etc.)

I’m running into trouble trying to upgrade to 4.x and now 5.0.0. I’m also upgrading to batchspawner 1.3.0. After upgrade, the hub was able to launch servers for users, however when the user was redirected to /user/username, they would get a 403 error and the message “user username is not allowed”.

jh5

The user is in the allowed user list, and their server was actually successfully spawned onto the cluster. The browser’s dev console shows 403 errors trying to access static assets like css and js at correct URLs. I checked the user server’s environment and compared it to one running under jupyterhub 3.x, focusing on JUPYTERHUB_ env vars. 3.x had JUPYTERHUB_HOST= while 5.0 had JUPYTERHUB_HOST=https://jupyter-dev.example.org. The server spawned by 5.0 had JUPYTERHUB_SERVICE_URL=http://0.0.0.0:0/user/username/ while the same var was not set in 3.x. Everything else was comparable.

I’m not worried about the port because that should get changed by batchspawner. I’m guessing the address is 0.0.0.0 because c.JupyterHub.hub_ip is set to 0.0.0.0 because the hub is inside a container. I tried setting c.JupyterHub.hub_connect_ip to the real hostname, e.g. jupyter-dev.example.org (as documented at Networking basics — JupyterHub documentation) The hub was able to start, but there were errors in the hub logs:

23:19:03.592 [ConfigProxy] error: 503 GET / connect ECONNREFUSED a.b.c.d:8081
23:19:03.598 [ConfigProxy] error: Failed to get custom error page: Error: connect ECONNREFUSED a.b.c.d:8081
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1605:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: 'a.b.c.d',
  port: 8081
}

where a.b.c.d is the public IP address of the system running the hubs, nginx, etc. This seems like the proxy is trying to connect to the hub at the public address port 8081 after resolving hub_connect_ip, while at this point I need it to be using the docker container’s internal address or docker service name. The spawned servers running on the slurm nodes need to connect to the public ip/hostname.

I’m kind of stumped at this point.

2

Additionally, the notebook app at /user/username/tree also responds with a 403 although with no broken assets.

I am able to view a list of servers at /hub/home.

So the user seems to be authorized to access the hub, but not their singleuser server. I am not modifying roles or scopes in any way.

I logged out, and then logged in as an admin user and tried accessing the initial user’s running server, but was also given a 403.

3

This is resolved. I needed to pass additional JUPYTERHUB_* environment variable to the singleuser environment.

1 Like