Action is not authorized with current scopes; requires any of [read:servers]

Hi all,
I come from here.
As it seems the error is coming from an strange behaviour with Traefik, I have decided to use nginx as proxy.
However after configuring it, after log-in the server is stuck in “Your server is starting up” and the progress bar is not moving. If I reload, the lab is correctly setup and I can use it.
Checking my firefox logs, it seems that I get:

status -> 403 
message -> Action is not authorized with current scopes; requires any of [read:servers]

I have tried to look in the forum, and notice Stuck in “Your server is starting up” after an upgrade and then redirected to /tree, but there is no answers to the question.

I have the following in my jupyterhub_config.py:

c.JupyterHub.load_roles = [
    {
        "name": "jupyterhub-idle-culler-role",
        "scopes": [
            "list:users",
            "read:users:activity",
            "read:servers",
            "delete:servers",
        ],
        # assignment of role's permissions to:
        "services": ["jupyterhub-idle-culler-service"],
    },
]

I have find in the logs that they claim to include other permissions in scope, but I don’t know where :frowning:

  jupyterhub-user-userjlanza  | Running as jovyan: jupyterhub-singleuser --ip=0.0.0.0 --SingleUserNotebookApp.default_url=/lab
  jupyterhub-jlanza             | [I 2022-11-07 11:26:02.242 JupyterHub log:186] 302 GET /hub/spawn -> /hub/spawn-pending/userjlanza (userjlanza@10.10.200.20) 1004.43ms      nginx-jlanza                  | 10.10.200.20 - - [07/Nov/2022:11:26:02 +0000] "GET /hub/spawn HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0"
  jupyterhub-jlanza             | [D 2022-11-07 11:26:02.285 JupyterHub scopes:796] Checking access via scope servers
  jupyterhub-jlanza             | [D 2022-11-07 11:26:02.285 JupyterHub scopes:610] Unrestricted access to /hub/spawn-pending/userjlanza via servers
  jupyterhub-jlanza             | [I 2022-11-07 11:26:02.286 JupyterHub pages:394] userjlanza is pending spawn
  jupyterhub-jlanza             | [I 2022-11-07 11:26:02.291 JupyterHub log:186] 200 GET /hub/spawn-pending/userjlanza (userjlanza@10.10.200.20) 8.86ms
  nginx-jlanza                  | 10.10.200.20 - - [07/Nov/2022:11:26:02 +0000] "GET /hub/spawn-pending/userjlanza HTTP/1.1" 200 7218 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0"
  jupyterhub-jlanza             | [W 2022-11-07 11:26:02.900 JupyterHub base:70] Blocking API request with no referer
  jupyterhub-jlanza             | [D 2022-11-07 11:26:02.901 JupyterHub scopes:796] Checking access via scope read:servers
  jupyterhub-jlanza             | [D 2022-11-07 11:26:02.901 JupyterHub scopes:607] No access to /hub/api/users/userjlanza/server/progress via read:servers
  jupyterhub-jlanza             | [W 2022-11-07 11:26:02.901 JupyterHub scopes:804] Not authorizing access to /hub/api/users/userjlanza/server/progress. Requires any of [read:servers], not derived from scopes []
  jupyterhub-jlanza             | [W 2022-11-07 11:26:02.902 JupyterHub web:1796] 403 GET /hub/api/users/userjlanza/server/progress (10.10.200.20): Action is not authorized with current scopes; requires any of [read:servers]

Any help is more than welcome. I don’t know whatelse I can modify :frowning:

My nginx.conf looks like:

events {
  # configuration of connection processing
   worker_connections 4096;
}

# top-level http config for websocket headers
http {
  upstream backend {
    server jupyterhub-jlanza:8000;
  }
  
  # If Upgrade is defined, Connection = upgrade
  # If Upgrade is empty, Connection = close
  map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
  }

  # HTTP server to redirect all 80 traffic to SSL/HTTPS
  server {
    listen 80;
    server_name myserver.com;

    # Tell all requests to port 80 to be 302 redirected to HTTPS
    return 302 https://$host$request_uri;
  }

  # HTTPS server to handle JupyterHub
  server {
    listen 443 ssl;
    listen [::]:443 ssl http2;
    server_name myserver.com;

    ssl_certificate /etc/ssl/letsencrypt/server.crt.pem;
    ssl_certificate_key /etc/ssl/letsencrypt/server.key.pem;
    ssl_dhparam /etc/ssl/letsencrypt/dhparam4096.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ALL:+HIGH:!ADH:!EXP:!SSLv2:!SSLv3:!MEDIUM:!LOW:!NULL:!aNULL;
    
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_stapling on;
    ssl_stapling_verify on;
    
    # Add headers to serve security related headers    
    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;
    add_header Referrer-Policy "no-referrer";

    # Managing literal requests to the JupyterHub front end
    location / {
      proxy_pass http://backend;
      # proxy_redirect off;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # websocket headers
      #proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_set_header X-Scheme $scheme;

      proxy_buffering off;
    }

    # Managing requests to verify letsencrypt host
    location ~ /.well-known {
      allow all;
    }
  }
}

I have further gone on configuring another proxy, haproxyin this case, and I get the same problem as with nginx.
It seems the only one that “works” is traefik but it is random when connecting to the kernels :frowning:
Any help is more than welcome.

This usually means you proxy isn’t correctly setting or forwarding some HTTP headers.

Can you open your browser’s web/debug/networking console, and check the headers that are being sent/received?