I have Jupyterhub deployed on AWS EKS as a k8s deployment with one pod. Jupyterhub is using RDS as a DB. Jupyterhub is v3.0.0.
When I spawn an instance (another k8s pod), I stay on /spawn-pending/ forever. I’m just not getting redirected anywhere:
I see the following error in the logs:
[W 2023-01-18 13:45:56.403 JupyterHub base:89] Blocking Cross Origin API request. Referer: https://REDACTED/hub/spawn-pending/user@mymail.com, Host: REDACTED, Host URL: http://REDACTED/hub/
[W 2023-01-18 13:45:56.403 JupyterHub scopes:804] Not authorizing access to /hub/api/users/user@mymail.com/server/progress. Requires any of [read:servers], not derived from scopes []
[W 2023-01-18 13:45:56.403 JupyterHub web:1796] 403 GET /hub/api/users/user@mymail.com/server/progress (100.64.159.13): Action is not authorized with current scopes; requires any of [read:servers]
[W 2023-01-18 13:45:56.404 JupyterHub log:186] 403 GET /hub/api/users/user@mymail.com/server/progress (@100.64.159.13) 4.38ms
However, Jupyterhub is indeed spawning the pod successfully - I can see it via kubectl get pod and also I can get in a terminal via kubectl exec -ti pod-name -- bash.
If I go directly to https://redacted/user/user@mymail.com/lab, I occasionally can access my notebook. When I can’t, I get HTTP 404 or HTTP 500 (I can’t determine which of both happens when).
I have a k8s Service called jupyterhub.
If I go inside the my user’s pod terminal and do curl -kv http://jupyterhub:8081/hub/api, I get either of the below two responses:
* Rebuilt URL to: http://jupyterhub:8081/
* Trying 172.20.196.220...
* TCP_NODELAY set
* connect to 172.20.196.220 port 8081 failed: Connection refused
* Failed to connect to jupyterhub port 8081: Connection refused
* Closing connection 0
curl: (7) Failed to connect to jupyterhub port 8081: Connection refused
or
* Rebuilt URL to: http://jupyterhub:8081/
* Trying 172.20.196.220...
* TCP_NODELAY set
* Connected to jupyterhub (172.20.196.220) port 8081 (#0)
> GET / HTTP/1.1
> Host: jupyterhub:8081
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: TornadoServer/6.2
< Content-Type: text/html
< Date: Wed, 18 Jan 2023 13:10:58 GMT
< X-Jupyterhub-Version: 3.0.0
< Access-Control-Allow-Headers: accept, content-type, authorization
< Content-Security-Policy: frame-ancestors 'self'; report-uri /hub/security/csp-report
< Location: /hub/
< Content-Length: 0
<
* Connection #0 to host jupyterhub left intact*
It doesn’t matter whether I use IP address or the service’s DNS name - I get successes and failures randomly.
If I go to Juopyterhub’s pod directly and do requests to itself, I get the same results: random failures when I use DNS name (be it “jupyterhub:8081” or “localhost:8081”) or k8s service IP. However, if I curl -kv http://127.0.0.1:8081/hub/api I always get successful responses.
How should I go about this?
