Cross origin issue upgrading from 1.5

JupyterHub
1

(I’m reporting this on discourse rather than github because I suspect it could be a configuration issue and not a bug.)

One of our hubs is configured with slurmspawner to spawn to nodes in a small hpc cluster. It runs with a container, fronted by nginx for ssl termination and for other virtual host configuration. There are actually a couple of hubs – one for dev/test and one for production.

I just tried updating the dev instance from 1.5 to 3.0.0b1 but ran into a cors issue. User servers can be started, and I can access the hub admin UI just fine, but the hub reports a “Content security violation” on the user server progress page. The user-facing spawn status page doesn’t update, and the hub doesn’t redirect the user to their server. If I manually navigate to the /progress page, I see:

{"status": 403, "message": "Action is not authorized with current scopes; requires any of [read:servers]"}

I initially thought this was due to scopes I may have misconfigured, but I removed all scope-related configs and still see the error. There are similar reports of this from around when jupyterhub 2.0 came out, but the fixes/workarounds for those reports don’t work here.

Related:

My sanitized nginx configuration is below, where {our_fqdn} is our hostname reachable by our hpc nodes. jupyterhub-dev is reachable from within the docker network.

server { 
    listen 443 ssl;
    server_name {our_fqdn};
    ...
    location / {
        proxy_pass http://jupyterhub-dev:8002;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # websocket headers
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        # https://discourse.jupyter.org/t/cors-issues-behind-nginx-ingress-after-updating-to-jupyterhub-2-1-1/13308/9
        #proxy_set_header X-Scheme $scheme;
        proxy_set_header X-Scheme https;

        # https://github.com/jupyterhub/jupyterhub/issues/3780
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_buffering off;
    }
}

jupyterhub configuration below. We have to specify the hub urls because otherwise the hub will provide only the randomized name that it sees within the docker container:

c = get_config()
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.hub_port = 9082
c.JupyterHub.port = 8002

c.JupyterHub.spawner_class = OurOptionsSlurmSpawner
c.Spawner.cmd = [
    'jupyterhub-singleuser',
    f'--hub-host=http://{our_fqdn}',
    f'--hub-api-url=http://{our_fqdn}:9082/hub/api'
]
c.Spawner.environment = {
    'JUPYTERHUB_API_URL': f'http://{our_fqdn}:9082/hub/api'
}

user’s jupyter server log. Probably unrelated but it seems like a method in batchspawner needs to be async. Note that I’m using a named server – “p4” this time. hpc1 is the hostname of the hpc node. The server remains up, but the hub never redirects from the status page.

/path/to/lib/python3.8/site-packages/batchspawner/singleuser.py:16: RuntimeWarning: coroutine 'HubAuth._api_request' was never awaited
  hub_auth._api_request(method='POST',
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
[I  SingleUserLabApp mixins:610] Starting jupyterhub single-user server version 3.0.0b1
...
[I  SingleUserLabApp mixins:671] Starting jupyterhub-singleuser server version 3.0.0b1
[I  SingleUserLabApp serverapp:2726] Serving notebooks from local directory: /home/user1
[I  SingleUserLabApp serverapp:2726] Jupyter Server 1.18.1 is running at:
[I  SingleUserLabApp serverapp:2726] http://hpc1:54657/user/user1/p4/lab
[I  SingleUserLabApp serverapp:2726]  or http://127.0.0.1:54657/user/user1/p4/lab
[I  SingleUserLabApp serverapp:2727] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W  SingleUserLabApp mixins:590] Activity events disabled

The progress URL https://{our_fqdn}/hub/api/users/user1/servers/p4/progress

{"status": 403, "message": "Action is not authorized with current scopes; requires any of [read:servers]"}

An excerpt from the hub logs:

[W JupyterHub base:182] Rolling back dirty objects IdentitySet([<Server(:0)>])
[I JupyterHub log:186] 302 POST /hub/spawn/user1/p4 -> /hub/spawn-pending/user1/p4 (user1@76.217.50.229) 1011.45ms
[I JupyterHub pages:394] user1:p4 is pending spawn
[I JupyterHub log:186] 200 GET /hub/spawn-pending/user1/p4 (user1@76.217.50.229) 6.85ms
[W JupyterHub base:89] Blocking Cross Origin API request.  Referer: {our_fqdn}, Host: {our_fqdn}, Host URL: https://{our_fqdn}/hub/
[W JupyterHub scopes:804] Not authorizing access to /hub/api/users/user1/servers/p4/progress. Requires any of [read:servers], not derived from scopes []
[W JupyterHub web:1796] 403 GET /hub/api/users/user1/servers/p4/progress (76.217.50.229): Action is not authorized with current scopes; requires any of [read:servers]
[W JupyterHub log:186] 403 GET /hub/api/users/user1/servers/p4/progress (@76.217.50.229) 4.31ms
[I JupyterHub log:186] 200 GET /hub/api (@hpc1.ip.address) 1.48ms
06:#033[32minfo#033[39m: 200 GET /api/routes
[W JupyterHub base:89] Blocking Cross Origin API request.  Referer: {our_fqdn}, Host: {our_fqdn}, Host URL: https://{our_fqdn}/hub/
[W JupyterHub scopes:804] Not authorizing access to /hub/api/users/user1/servers/p4/progress. Requires any of [read:servers], not derived from scopes []
[W JupyterHub web:1796] 403 GET /hub/api/users/user1/servers/p4/progress (76.217.50.229): Action is not authorized with current scopes; requires any of [read:servers]
[W JupyterHub log:186] 403 GET /hub/api/users/user1/servers/p4/progress (@76.217.50.229) 5.26ms
[W JupyterHub base:1706] Content security violation: {"csp-report":{"document-uri":"https://{our_fqdn}/hub/api/users/user1/servers/p4/progress","referrer":"","violated-directive":"style-src-attr","effective-directive":"style-src-attr","original-policy":"frame-ancestors 'self'; report-uri /hub/security/csp-report; default-src 'none'","disposition":"enforce","blocked-uri":"inline","line-number":1,"source-file":"https://{our_fqdn}/hub/api/users/user1/servers/p4/progress","status-code":403,"script-sample":""}}
[I JupyterHub log:186] 200 POST /hub/security/csp-report (user1@76.217.50.229) 3.76ms
[W JupyterHub base:89] Blocking Cross Origin API request.  Referer: {our_fqdn}, Host: {our_fqdn}, Host URL: https://{our_fqdn}/hub/
[W JupyterHub scopes:804] Not authorizing access to /hub/api/users/user1/servers/p4/progress. Requires any of [read:servers], not derived from scopes []
[W JupyterHub web:1796] 403 GET /hub/api/users/user1/servers/p4/progress (76.217.50.229): Action is not authorized with current scopes; requires any of [read:servers]
[W JupyterHub log:186] 403 GET /hub/api/users/user1/servers/p4/progress (@76.217.50.229) 8.17ms
[W JupyterHub base:1706] Content security violation: {"csp-report":{"document-uri":"https://{our_fqdn}/hub/api/users/user1/servers/p4/progress","referrer":"","violated-directive":"style-src-attr","effective-directive":"style-src-attr","original-policy":"frame-ancestors 'self'; report-uri /hub/security/csp-report; default-src 'none'","disposition":"enforce","blocked-uri":"inline","line-number":1,"source-file":"https://{our_fqdn}/hub/api/users/user1/servers/p4/progress","status-code":403,"script-sample":""}}
[I JupyterHub log:186] 200 POST /hub/security/csp-report (user1@76.217.50.229) 5.83ms
2

Is the referer really just your fqdn? It should be https://{our_fqdn}/hub/spawn/...

I really want to eliminate this check. It’s been the source of so many issues. I’ll need to revisit why we do it in the first place, and see if there’s a better way. Maybe SameSite cookies can be a replacement.

3

In the log lines prefixed by “Blocking Cross Origin API request”, the Referer is http://{our_fqdn}:8002 rather than just {our_fqdn}. (sanitization gone too far)

Do you mean it should also have the path info?

Can you tell from the nginx config what has gone wrong and/or if there are any workarounds?

4

Cross origin checks are intended to stop a malicious script on one website making a change on another site. To count as the “same” origin the scheme (http vs https) domain (our_fqdn) and port in the host and referer should all match, and they should also match what appears in the user’s browser location bar.

Is http://{our_fqdn}:8002 appearing in all those places? If not you may have a misconfigured proxy, e.g. the proxy may override the Host or other header instead of sending the header from the client.

5

That’s very concise, thanks @manics! We were unnecessarily configuring the hub ports – a leftover of the pre-docker deployment. Removing that eliminated the cross-origin messages.

I’m still seeing the /server/progress scope issue though so the hub isn’t redirecting users to their servers. I’ll dig into that a bit more.

1 Like
6

Is it possible to disable cross-origin in JupyterHub 3.1 (for troubleshooting purposes ofc)?