I have Jupyterhub deployed on AWS EKS as a k8s deployment with one pod. Jupyterhub is using RDS as a DB. Jupyterhub is v3.0.0.
When I spawn an instance (another k8s pod), I stay on /spawn-pending/ forever. I’m just not getting redirected anywhere:
I see the following error in the logs:
[W 2023-01-18 13:45:56.403 JupyterHub base:89] Blocking Cross Origin API request. Referer: https://REDACTEDemail@example.com, Host: REDACTED, Host URL: http://REDACTED/hub/ [W 2023-01-18 13:45:56.403 JupyterHub scopes:804] Not authorizing access to /firstname.lastname@example.org/server/progress. Requires any of [read:servers], not derived from scopes  [W 2023-01-18 13:45:56.403 JupyterHub web:1796] 403 GET /email@example.com/server/progress (100.64.159.13): Action is not authorized with current scopes; requires any of [read:servers] [W 2023-01-18 13:45:56.404 JupyterHub log:186] 403 GET /firstname.lastname@example.org/server/progress (@100.64.159.13) 4.38ms
However, Jupyterhub is indeed spawning the pod successfully - I can see it via
kubectl get pod and also I can get in a terminal via
kubectl exec -ti pod-name -- bash.
If I go directly to
https://email@example.com/lab, I occasionally can access my notebook. When I can’t, I get HTTP 404 or HTTP 500 (I can’t determine which of both happens when).
I have a k8s
If I go inside the my user’s pod terminal and do
curl -kv http://jupyterhub:8081/hub/api, I get either of the below two responses:
* Rebuilt URL to: http://jupyterhub:8081/ * Trying 172.20.196.220... * TCP_NODELAY set * connect to 172.20.196.220 port 8081 failed: Connection refused * Failed to connect to jupyterhub port 8081: Connection refused * Closing connection 0 curl: (7) Failed to connect to jupyterhub port 8081: Connection refused
* Rebuilt URL to: http://jupyterhub:8081/ * Trying 172.20.196.220... * TCP_NODELAY set * Connected to jupyterhub (172.20.196.220) port 8081 (#0) > GET / HTTP/1.1 > Host: jupyterhub:8081 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 302 Found < Server: TornadoServer/6.2 < Content-Type: text/html < Date: Wed, 18 Jan 2023 13:10:58 GMT < X-Jupyterhub-Version: 3.0.0 < Access-Control-Allow-Headers: accept, content-type, authorization < Content-Security-Policy: frame-ancestors 'self'; report-uri /hub/security/csp-report < Location: /hub/ < Content-Length: 0 < * Connection #0 to host jupyterhub left intact*
It doesn’t matter whether I use IP address or the service’s DNS name - I get successes and failures randomly.
If I go to Juopyterhub’s pod directly and do requests to itself, I get the same results: random failures when I use DNS name (be it “jupyterhub:8081” or “localhost:8081”) or k8s service IP. However, if I
curl -kv http://127.0.0.1:8081/hub/api I always get successful responses.
How should I go about this?