Updated JupyterHub 5.2.1 issue: not allowing user

JupyterHub
,
1

Hello,

I am developing a custom spawner, which used to work with JupyterHub 3.1.1. When I updated to 5.2.1 (I am using this container with tag cuda12-hub-5.2.1), then jupyterhub-singleserver started to give me “Not allowing user” errors:


[I 2024-11-20 03:49:26.016 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-11-20 03:49:26.017 ServerApp] http://swm-3522242b-main:8888/user/taras/lab?token=...
[I 2024-11-20 03:49:26.017 ServerApp] http://127.0.0.1:8888/user/taras/lab?token=...
[I 2024-11-20 03:49:26.017 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[D 2024-11-20 03:49:32.766 ServerApp] No user identified
[I 2024-11-20 03:49:32.766 ServerApp] 200 GET /user/taras/api (@[5.77.195.157](http://5.77.195.157)) 1.13ms
[I 2024-11-20 03:49:33.153 ServerApp] 302 GET /user/taras/ -> /user/taras/lab? (@::ffff:172.17.0.1) 0.48ms
[D 2024-11-20 03:49:33.558 ServerApp] No user identified
[D 2024-11-20 03:49:33.558 ServerApp] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-taras&redirect_uri=%2Fuser%2Ftaras%2Foauth_callback&response_type=code
[I 2024-11-20 03:49:33.559 ServerApp] 302 GET /user/taras/lab? -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-taras&redirect_uri=%2Fuser%2Ftaras%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.17.0.1) 1.22ms
[D 2024-11-20 03:49:34.350 ServerApp] HubAuth cache miss: token:5099e94a2ba54afe9bcda58e3aae63b7:34757c071dd14a1268981eeb1c9a3e8f593d2716a7925d758fd193e63f81f472
[D 2024-11-20 03:49:34.861 ServerApp] Received request from Hub user {'name': 'taras', 'admin': True, 'kind': 'user', 'groups': [], 'token_id': 'a393', 'session_id': '5099e94a2ba54afe9bcda58e3aae63b7', 'scopes': ['access:servers!server=taras/', 'read:users:groups!user=taras', 'read:users:name!user=taras']}
[I 2024-11-20 03:49:34.861 ServerApp] Logged-in user taras
[D 2024-11-20 03:49:34.861 ServerApp] User model {'name': 'taras', 'admin': True, 'kind': 'user', 'groups': [], 'token_id': 'a393', 'session_id': '5099e94a2ba54afe9bcda58e3aae63b7', 'scopes': ['access:servers!server=taras/', 'read:users:groups!user=taras', 'read:users:name!user=taras']}
[D 2024-11-20 03:49:34.861 ServerApp] Setting oauth cookie for ::ffff:[172.17.0.1](http://172.17.0.1): jupyterhub-user-taras, {'path': '/user/taras', 'httponly': True}
[I 2024-11-20 03:49:34.861 ServerApp] Setting new xsrf cookie for b'5099e94a2ba54afe9bcda58e3aae63b7:34757c071dd14a1268981eeb1c9a3e8f593d2716a7925d758fd193e63f81f472' {'path': '/user/taras/'}
[D 2024-11-20 03:49:34.862 ServerApp] Allowing Hub user taras (all Hub users and services allowed)
[I 2024-11-20 03:49:34.862 ServerApp] 302 GET /user/taras/oauth_callback?code=[secret]&state=[secret] -> /user/taras/lab? (taras@::ffff:172.17.0.1) 884.23ms
[D 2024-11-20 03:49:35.160 ServerApp] Checking user taras with scopes ['access:servers!server=taras/', 'read:users:groups!user=taras', 'read:users:name!user=taras'] against set()
[W 2024-11-20 03:49:35.160 ServerApp] Not allowing user taras
[D 2024-11-20 03:49:35.160 ServerApp] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-taras&redirect_uri=%2Fuser%2Ftaras%2Foauth_callback&response_type=code
[W 2024-11-20 03:49:35.161 ServerApp] 403 GET /user/taras/lab? (::ffff:172.17.0.1): user taras is not allowed.

I tried multiple recommendations from the internet I found for the new version of JupyterHub 5.x, but nothing helped.

My jupyterhub_config.py:

from swmjupyter.spawner import SwmSpawner

c.JupyterHub.hub_ip = '127.0.0.1'
c.JupyterHub.hub_port = 8081
c.JupyterHub.log_level = 'DEBUG'
c.JupyterHub.authenticator_class = 'dummy'
c.JupyterHub.oauth_token_expires_in = 3600
c.JupyterHub.tornado_settings = {
    "slow_spawn_timeout": 0,
}
c.JupyterHub.spawner_class = 'swmjupyter.spawner.SwmSpawner'
c.Spawner.debug = True

# For JupyterHub 5.2.1:
c.Authenticator.allow_all = True
c.JupyterHub.load_roles = [
    {
        "name": "user",
        "scopes": [
            "self",
            "access:servers!user",
            "read:users:name",
            "read:users:groups",
        ],
    }
]

In the custom spawner class I added the get_env function:

 def get_env(self) -> dict[str, str]:
        env = super().get_env()
        env.update({
            'JUPYTERHUB_CLIENT_ID': f'jupyterhub-user-{self.user.name}',
            'JUPYTERHUB_OAUTH_CALLBACK_URL': f'{self.hub.url}user/{self.user.name}/oauth_callback',
            'JUPYTERHUB_OAUTH_SCOPES': [
                'access:servers!server=' + self.user.name,
                'access:servers!user=' + self.user.name,
                'read:users:name',
                'read:users:groups',
            ]
        })
        return env

Did I miss something else when updated jupyterhub?

2

You shouldn’t need to override get_env(), JupyterHub should already be passing the required environment variables to the parent spawner.

If you use one of the standard spawners do you have any problems?

Are you able to share a (possibly cut down) version of your spawner that exhibits the problem?

3

Are you able to share a (possibly cut down) version of your spawner that exhibits the problem?

Sure, if I don’t override get_env(), then it is the same as the one I use, JupyterHub 3.1.1, and located here. It’s a pretty simple spawner that connects to a so called swm daemon and sends information about running jupyterhub-singleuser to the daemon.

Thanks for the idea of checking some standard spawner in my environment. I tried DockerSpawner, and it worked fine for me with the same container I use for my custom spawner. I run jupyterhub in a docker container with --network=bridge, while DockerSpawner started a new container with --network=host. So it worked in my environment with this config:

import dockerspawner
class DockerSpawnerCustomIps(dockerspawner.DockerSpawner):
    async def get_ip_and_port(self):
        return self.host_ip, self.port
c.JupyterHub.hub_ip = '127.0.0.1'
c.JupyterHub.hub_port = 8081
c.JupyterHub.log_level = 'DEBUG'
c.JupyterHub.authenticator_class = 'dummy'
c.JupyterHub.spawner_class = DockerSpawnerCustomIps
c.JupyterHub.hub_ip = '172.17.0.2'
c.DockerSpawner.host_ip = '172.17.0.1'
c.DockerSpawner.image = 'quay.io/jupyter/pytorch-notebook:cuda12-hub-5.2.1'
c.DockerSpawner.extra_host_config = {'network_mode': 'host'}
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.remove = True
c.DockerSpawner.network_name = "host"
c.Authenticator.allow_all = True

My custom spawner is sort of a mix of BatchSpawner and DockerSpawner: it connects to a separate daemon that, in turn, creates a VM in the cloud, opens an SSH tunnel between the local Jupyterhub container and the VM, and then a single user server connects to Jupyterhub to its localhost IP in the VM. This works fine in Jupyterhub 3.1.1, so I suppose some improvements in the latest version require me to make changes in my custom spawner or in the exported environment variables for jupyterhub-singleuser.

When I start jupyterhub-singleuser I export these jupyter-related environment variables:

JUPYTERHUB_API_TOKEN (generated by jupyterhub)
JUPYTERHUB_CLIENT_ID (generated by jupyterhub)
JUPYTERHUB_API_URL is f"http://{jupyterhub_host}:{jupyterhub_port}/hub/api"
JUPYTER_RUNTIME_DIR=/tmp
JUPYTERHUB_USER=taras
JUPYTERHUB_SERVICE_PREFIX=/user/taras
JUPYTERHUB_SERVICE_URL="http://0.0.0.0:8888"

Do I need to export something else for 5.2.1?

4

I compared environment variables set by DockerSpawner and what my spawner exports and found a set of variables that I don’t export. I tested all of them and found one which absent breaks my spawner: JUPYTERHUB_OAUTH_ACCESS_SCOPES. When I set it to '["access:servers!server=$USER/", "access:servers!user=$USER"]' before jupyterhub-singleuser execution, then everything works. Is this expected?

5

That variable controls what the OAuth token can do

By default JupyterHub sets it to access:servers!server={self.user.name}/{self.name} for spawners, and passes this to the spawner using the JUPYTERHUB_OAUTH_ACCESS_SCOPES variable. Since you were missing that variable the token defaulted to no permissions, i.e. you didn’t have permission to access your singleuser server.

1 Like