Cross-origin and authorization errors after Z2JH 1.2.0 -> 2.0.0 upgrade

JupyterHub Zero to JupyterHub on Kubernetes
,
1

After upgrading my Z2JH deployment (running on K8s 1.21) from 1.2.0 to 2.0.0, I have the following symptoms:

  • No progress bar when starting a user’s server; the browser freezes here with an empty progress bar, but hitting F5 once the server has started brings up the lab.
  • Attempting to save a notebook results in a “File Save Error for mynb.ipynb: Forbidden” dialog
  • Attempting to stop the server gives “API request failed (403): Action is not authorized with current scopes; requires any of [delete:servers]”
  • Admins cannot view other users’ servers in the Admin panel.

Relevant log messages: on start-up,

[W 2023-01-12 16:30:39.520 JupyterHub base:89] Blocking Cross Origin API
request.  Referer: https://example.com/hub/spawn-pending/myuser, Host:
example.com, Host URL: http://example.com/hub/
[W 2023-01-12 16:30:39.521 JupyterHub scopes:804] Not authorizing access to
/hub/api/users/myuser/server/progress. Requires any of [read:servers], not
derived from scopes []

And similar messages later for [delete:servers], [list:users], and [read:servers] which I assume are correlated with the other operations I attempted.

There’s no mismatch between chart, hub, and single-user server versions.

The problems are reminiscent of those described in Stuck in "Your server is starting up" after an upgrade and then redirected to /tree , which contains the observation “It looks like your proxy/load-balancer is incorrectly forwarding or setting the scheme (http vs https).”. I’m not sure where to start digging into this though – I’ve searched through the available configuration fields under proxy without finding anything likely-looking.

I’d be grateful for any suggestions.

2

I managed to solve the problem. This deployment runs on AWS, and I had forgotten that it wasn’t configured with the usual JupyterHub Let’s Encrypt HTTPS implementation, but for historical reasons instead used an AWS certificate on the load balancer (ID censored in values.yaml snippet below):

 proxy:
   https:
     enabled: true
     type: offload
     hosts:
       - example.com

   service:
     annotations:
       service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:■■■■■■■■"
       service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
       service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
       service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'

I removed this configuration and briefly restarted JupyterHub with HTTPS disabled, then added a standard Let’s Encrypt configuration section instead:

  proxy:
    https:
      enabled: true
      hosts:
        - example.com
      letsencrypt:
        contactEmail: someone@example.com

This resolved all the problems I reported above. I assume that they could also have been fixed by additional load balancer configuration, but in our case Let’s Encrypt was in any case the preferred solution.