Action is not authorized with current scopes; requires any of [read:servers]

Hi all,
I come from here.
As it seems the error is coming from an strange behaviour with Traefik, I have decided to use nginx as proxy.
However after configuring it, after log-in the server is stuck in “Your server is starting up” and the progress bar is not moving. If I reload, the lab is correctly setup and I can use it.
Checking my firefox logs, it seems that I get:

status -> 403 
message -> Action is not authorized with current scopes; requires any of [read:servers]

I have tried to look in the forum, and notice Stuck in “Your server is starting up” after an upgrade and then redirected to /tree, but there is no answers to the question.

I have the following in my jupyterhub_config.py:

c.JupyterHub.load_roles = [
    {
        "name": "jupyterhub-idle-culler-role",
        "scopes": [
            "list:users",
            "read:users:activity",
            "read:servers",
            "delete:servers",
        ],
        # assignment of role's permissions to:
        "services": ["jupyterhub-idle-culler-service"],
    },
]

I have find in the logs that they claim to include other permissions in scope, but I don’t know where :frowning:

  jupyterhub-user-userjlanza  | Running as jovyan: jupyterhub-singleuser --ip=0.0.0.0 --SingleUserNotebookApp.default_url=/lab
  jupyterhub-jlanza             | [I 2022-11-07 11:26:02.242 JupyterHub log:186] 302 GET /hub/spawn -> /hub/spawn-pending/userjlanza (userjlanza@10.10.200.20) 1004.43ms      nginx-jlanza                  | 10.10.200.20 - - [07/Nov/2022:11:26:02 +0000] "GET /hub/spawn HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0"
  jupyterhub-jlanza             | [D 2022-11-07 11:26:02.285 JupyterHub scopes:796] Checking access via scope servers
  jupyterhub-jlanza             | [D 2022-11-07 11:26:02.285 JupyterHub scopes:610] Unrestricted access to /hub/spawn-pending/userjlanza via servers
  jupyterhub-jlanza             | [I 2022-11-07 11:26:02.286 JupyterHub pages:394] userjlanza is pending spawn
  jupyterhub-jlanza             | [I 2022-11-07 11:26:02.291 JupyterHub log:186] 200 GET /hub/spawn-pending/userjlanza (userjlanza@10.10.200.20) 8.86ms
  nginx-jlanza                  | 10.10.200.20 - - [07/Nov/2022:11:26:02 +0000] "GET /hub/spawn-pending/userjlanza HTTP/1.1" 200 7218 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0"
  jupyterhub-jlanza             | [W 2022-11-07 11:26:02.900 JupyterHub base:70] Blocking API request with no referer
  jupyterhub-jlanza             | [D 2022-11-07 11:26:02.901 JupyterHub scopes:796] Checking access via scope read:servers
  jupyterhub-jlanza             | [D 2022-11-07 11:26:02.901 JupyterHub scopes:607] No access to /hub/api/users/userjlanza/server/progress via read:servers
  jupyterhub-jlanza             | [W 2022-11-07 11:26:02.901 JupyterHub scopes:804] Not authorizing access to /hub/api/users/userjlanza/server/progress. Requires any of [read:servers], not derived from scopes []
  jupyterhub-jlanza             | [W 2022-11-07 11:26:02.902 JupyterHub web:1796] 403 GET /hub/api/users/userjlanza/server/progress (10.10.200.20): Action is not authorized with current scopes; requires any of [read:servers]

Any help is more than welcome. I don’t know whatelse I can modify :frowning:

My nginx.conf looks like:

events {
  # configuration of connection processing
   worker_connections 4096;
}

# top-level http config for websocket headers
http {
  upstream backend {
    server jupyterhub-jlanza:8000;
  }
  
  # If Upgrade is defined, Connection = upgrade
  # If Upgrade is empty, Connection = close
  map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
  }

  # HTTP server to redirect all 80 traffic to SSL/HTTPS
  server {
    listen 80;
    server_name myserver.com;

    # Tell all requests to port 80 to be 302 redirected to HTTPS
    return 302 https://$host$request_uri;
  }

  # HTTPS server to handle JupyterHub
  server {
    listen 443 ssl;
    listen [::]:443 ssl http2;
    server_name myserver.com;

    ssl_certificate /etc/ssl/letsencrypt/server.crt.pem;
    ssl_certificate_key /etc/ssl/letsencrypt/server.key.pem;
    ssl_dhparam /etc/ssl/letsencrypt/dhparam4096.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ALL:+HIGH:!ADH:!EXP:!SSLv2:!SSLv3:!MEDIUM:!LOW:!NULL:!aNULL;
    
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_stapling on;
    ssl_stapling_verify on;
    
    # Add headers to serve security related headers    
    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;
    add_header Referrer-Policy "no-referrer";

    # Managing literal requests to the JupyterHub front end
    location / {
      proxy_pass http://backend;
      # proxy_redirect off;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # websocket headers
      #proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_set_header X-Scheme $scheme;

      proxy_buffering off;
    }

    # Managing requests to verify letsencrypt host
    location ~ /.well-known {
      allow all;
    }
  }
}

I have further gone on configuring another proxy, haproxyin this case, and I get the same problem as with nginx.
It seems the only one that “works” is traefik but it is random when connecting to the kernels :frowning:
Any help is more than welcome.

This usually means you proxy isn’t correctly setting or forwarding some HTTP headers.

Can you open your browser’s web/debug/networking console, and check the headers that are being sent/received?

Hi manics,

I have the same problem as jlanza with nginx as reverse proxy (same nginx.conf) except that the error in the hub logs is :

Blocking Cross Origin API request. Referer: http://jupyterhub-rec.xxxx/hub/spawn-pending/to203766, Host: jupyterhub-rec.xxxx, Host URL: https://jupyterhub-rec.xxxx/hub/

I noticed the http/https difference but the referer looks good from the browser pov.

Here are the headers I get from the brownser’s networking console :

Your help would be much appreciated :pray:

As with the above post it looks like the Host or Forwarding headers aren’t being passed to JupyterHub.

Please could you describe your entire JupyterHub setup?

We are using the z2jh for kubernetes helm chart version 2.0.0. The jupyterhub image is so the k8s-hub:2.0.0
Our k8s cluster is a HPE Container Platform, so to get an endpoint we are using this kind of service :

apiVersion: v1
kind: Service
metadata:
  namespace: xxxx-rec
  name: jupyterhub-gw-svc
  labels:
    hpecp.hpe.com/hpecp-internal-gateway: "True"
spec:
  ports:
  - name: http-jupyterhub-gw-svc
    port: 8080
    protocol: TCP
    targetPort: 8000
  selector:
    component: proxy
    release: jupyterhub-atr-rec
  sessionAffinity: None
  type: NodePort

It allows us to get an access point through their gateway but on a random port (10000+).
We don’t want to expose our service on this port so that’s why we are using the reverse proxy.

Note that it works when we use the default url generated by the gateway (on port 10000+)

I know it’s a bit messy so I would totally understand that you can’t help us :smiling_face_with_tear:

You’ll need to work out how to set the fowarding headers.

If you don’t mind testing a dev version of the Helm Chart the latest versions include JupyterHub 4.0.0b1, which changes how the cross site checks are performed, and may solve the problem without too much (or any!) work.

I was having the same problem and in fact just joined up to ask for help.

I ended up solving it with the solution in this thread: Can not get admin UI in JupyterHub 2.2.0

(Set proxy_set_header X-scheme to https instead of $scheme)

2 Likes