Jupyterhub sometimes can't access localhost:8081/api/hub

I have Jupyterhub deployed on AWS EKS as a k8s deployment with one pod. Jupyterhub is using RDS as a DB. Jupyterhub is v3.0.0.

When I spawn an instance (another k8s pod), I stay on /spawn-pending/ forever. I’m just not getting redirected anywhere:

I see the following error in the logs:

[W 2023-01-18 13:45:56.403 JupyterHub base:89] Blocking Cross Origin API request.  Referer: https://REDACTED/hub/spawn-pending/user@mymail.com, Host: REDACTED, Host URL: http://REDACTED/hub/
[W 2023-01-18 13:45:56.403 JupyterHub scopes:804] Not authorizing access to /hub/api/users/user@mymail.com/server/progress. Requires any of [read:servers], not derived from scopes []
[W 2023-01-18 13:45:56.403 JupyterHub web:1796] 403 GET /hub/api/users/user@mymail.com/server/progress (100.64.159.13): Action is not authorized with current scopes; requires any of [read:servers]
[W 2023-01-18 13:45:56.404 JupyterHub log:186] 403 GET /hub/api/users/user@mymail.com/server/progress (@100.64.159.13) 4.38ms

However, Jupyterhub is indeed spawning the pod successfully - I can see it via kubectl get pod and also I can get in a terminal via kubectl exec -ti pod-name -- bash.

If I go directly to https://redacted/user/user@mymail.com/lab, I occasionally can access my notebook. When I can’t, I get HTTP 404 or HTTP 500 (I can’t determine which of both happens when).

I have a k8s Service called jupyterhub.
If I go inside the my user’s pod terminal and do curl -kv http://jupyterhub:8081/hub/api, I get either of the below two responses:

* Rebuilt URL to: http://jupyterhub:8081/
*   Trying 172.20.196.220...
* TCP_NODELAY set
* connect to 172.20.196.220 port 8081 failed: Connection refused
* Failed to connect to jupyterhub port 8081: Connection refused
* Closing connection 0
curl: (7) Failed to connect to jupyterhub port 8081: Connection refused

or

* Rebuilt URL to: http://jupyterhub:8081/
*   Trying 172.20.196.220...
* TCP_NODELAY set
* Connected to jupyterhub (172.20.196.220) port 8081 (#0)
> GET / HTTP/1.1
> Host: jupyterhub:8081
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: TornadoServer/6.2
< Content-Type: text/html
< Date: Wed, 18 Jan 2023 13:10:58 GMT
< X-Jupyterhub-Version: 3.0.0
< Access-Control-Allow-Headers: accept, content-type, authorization
< Content-Security-Policy: frame-ancestors 'self'; report-uri /hub/security/csp-report
< Location: /hub/
< Content-Length: 0
<
* Connection #0 to host jupyterhub left intact*

It doesn’t matter whether I use IP address or the service’s DNS name - I get successes and failures randomly.

If I go to Juopyterhub’s pod directly and do requests to itself, I get the same results: random failures when I use DNS name (be it “jupyterhub:8081” or “localhost:8081”) or k8s service IP. However, if I curl -kv http://127.0.0.1:8081/hub/api I always get successful responses.

How should I go about this?

+ trap 'kill -TERM $PID' TERM INT
+ cd /opt/app-root/etc/
+ jupyterhub upgrade-db
### --> namespace: jh-namespace
### --> instance: <kubernetes.client.configuration.Configuration object at 0x7f3782949fa0>
### ---> api_client: <openshift.dynamic.client.DynamicClient object at 0x7f3781746a90>
Exception raised: No matches found for {'api_version': 'image.openshift.io/v1', 'kind': 'ImageStream'}
environ_config_file: /opt/app-root/configs/jupyterhub_config.py
--->Starting custom JupyterHUB configuration
SAMLAuthenticator configured
====================================================================================================
Persistant Volume configured
Custom JupyterHUB configuration done<---
+ cd -
/opt/app-root/src
+ PID=11
+ wait 11
+ start-jupyterhub.sh
### --> namespace: jh-namespace
### --> instance: <kubernetes.client.configuration.Configuration object at 0x7f8bb6268d60>
### ---> api_client: <openshift.dynamic.client.DynamicClient object at 0x7f8bb50998b0>
Exception raised: No matches found for {'api_version': 'image.openshift.io/v1', 'kind': 'ImageStream'}
environ_config_file: /opt/app-root/configs/jupyterhub_config.py
--->Starting custom JupyterHUB configuration
SAMLAuthenticator configured
====================================================================================================
Persistant Volume configured
Custom JupyterHUB configuration done<---
[I 2023-01-18 13:44:09.078 JupyterHub app:2775] Running JupyterHub version 3.0.0
[I 2023-01-18 13:44:09.079 JupyterHub app:2805] Using Authenticator: samlauthenticator.samlauthenticator.SAMLAuthenticator
[I 2023-01-18 13:44:09.079 JupyterHub app:2805] Using Spawner: kubespawner.spawner.KubeSpawner-4.2.0
[I 2023-01-18 13:44:09.079 JupyterHub app:2805] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-3.0.0
[I 2023-01-18 13:44:09.366 JupyterHub proxy:559] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2023-01-18 13:44:09.384 JupyterHub app:1934] Not using allowed_users. Any authenticated user will be allowed.
[I 2023-01-18 13:44:09.473 JupyterHub app:2844] Initialized 0 spawners in 0.003 seconds
[W 2023-01-18 13:44:09.479 JupyterHub proxy:750] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2023-01-18 13:44:09.479 JupyterHub proxy:754] Starting proxy @ http://:8080/
13:44:10.264 [ConfigProxy] ←[32minfo←[39m: Proxying http://*:8080 to (no default)
13:44:10.269 [ConfigProxy] ←[32minfo←[39m: Proxy API at http://127.0.0.1:8082/api/routes
13:44:10.281 [ConfigProxy] ←[32minfo←[39m: 200 GET /api/routes
[I 2023-01-18 13:44:10.282 JupyterHub app:3093] Hub API listening on http://0.0.0.0:8081/hub/
[I 2023-01-18 13:44:10.282 JupyterHub app:3095] Private Hub API connect url http://jupyterhub:8081/hub/
13:44:10.362 [ConfigProxy] ←[32minfo←[39m: 200 GET /api/routes
[I 2023-01-18 13:44:10.362 JupyterHub proxy:480] Adding route for Hub: / => http://jupyterhub:8081
13:44:10.364 [ConfigProxy] ←[32minfo←[39m: Adding route / -> http://127.0.0.1:8081
13:44:10.365 [ConfigProxy] ←[32minfo←[39m: Route added / -> http://127.0.0.1:8081
13:44:10.365 [ConfigProxy] ←[32minfo←[39m: 201 POST /api/routes/
[I 2023-01-18 13:44:10.366 JupyterHub app:3160] JupyterHub is now running at http://:8080/
13:45:36.107 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /user/user@mymail.com/api/kernels to http://127.0.0.1:8081
13:45:36.115 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /user/user@mymail.com/api/terminals to http://127.0.0.1:8081
13:45:36.117 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /user/user@mymail.com/api/metrics/v1 to http://127.0.0.1:8081
13:45:36.118 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /user/user@mymail.com/api/sessions to http://127.0.0.1:8081
13:45:36.119 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /user/user@mymail.com/api/contents to http://127.0.0.1:8081
13:45:36.120 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /user/user@mymail.com/dask/clusters to http://127.0.0.1:8081
[I 2023-01-18 13:45:36.123 JupyterHub log:186] 302 GET /user/user@mymail.com/api/kernels?1674049535817 -> /hub/user/user@mymail.com/api/kernels?1674049535817 (@100.64.159.13) 1.23ms
13:45:36.125 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 302 on /
[I 2023-01-18 13:45:36.127 JupyterHub log:186] 302 GET /user/user@mymail.com/api/terminals?1674049535796 -> /hub/user/user@mymail.com/api/terminals?1674049535796 (@100.64.159.13) 2.26ms
[I 2023-01-18 13:45:36.128 JupyterHub log:186] 302 GET /user/user@mymail.com/api/metrics/v1?1674049535790 -> /hub/user/user@mymail.com/api/metrics/v1?1674049535790 (@100.64.159.13) 1.96ms
[I 2023-01-18 13:45:36.128 JupyterHub log:186] 302 GET /user/user@mymail.com/api/sessions?1674049535803 -> /hub/user/user@mymail.com/api/sessions?1674049535803 (@100.64.159.13) 2.28ms
[I 2023-01-18 13:45:36.129 JupyterHub log:186] 302 GET /user/user@mymail.com/api/contents?content=1&1674049535812 -> /hub/user/user@mymail.com/api/contents?content=1&1674049535812 (@100.64.159.13) 2.50ms
[I 2023-01-18 13:45:36.129 JupyterHub log:186] 302 GET /user/user@mymail.com/dask/clusters?1674049535762 -> /hub/user/user@mymail.com/dask/clusters?1674049535762 (@100.64.159.13) 2.80ms
13:45:36.161 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 302 on /
13:45:36.162 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 302 on /
13:45:36.163 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 302 on /
13:45:36.163 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 302 on /
13:45:36.164 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 302 on /
13:45:36.209 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /hub/user/user@mymail.com/api/kernels to http://127.0.0.1:8081
[W 2023-01-18 13:45:36.242 JupyterHub base:1409] Failing suspected API request to not-running server: /hub/user/user@mymail.com/api/kernels
13:45:36.244 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 424 on /
[W 2023-01-18 13:45:36.261 JupyterHub log:186] 424 GET /hub/user/user@mymail.com/api/kernels?1674049535817 (user@mymail.com@100.64.159.13) 50.73ms
13:45:36.262 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /hub/user/user@mymail.com/api/sessions to http://127.0.0.1:8081
13:45:36.263 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /hub/user/user@mymail.com/api/contents to http://127.0.0.1:8081
13:45:36.264 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /hub/user/user@mymail.com/api/metrics/v1 to http://127.0.0.1:8081
13:45:36.266 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /hub/user/user@mymail.com/dask/clusters to http://127.0.0.1:8081
13:45:36.271 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /hub/user/user@mymail.com/api/terminals to http://127.0.0.1:8081
[W 2023-01-18 13:45:36.288 JupyterHub base:1409] Failing suspected API request to not-running server: /hub/user/user@mymail.com/api/sessions
[W 2023-01-18 13:45:36.289 JupyterHub log:186] 424 GET /hub/user/user@mymail.com/api/sessions?1674049535803 (user@mymail.com@100.64.159.13) 22.33ms
13:45:36.289 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 424 on /
[W 2023-01-18 13:45:36.301 JupyterHub base:1409] Failing suspected API request to not-running server: /hub/user/user@mymail.com/api/contents
[W 2023-01-18 13:45:36.302 JupyterHub log:186] 424 GET /hub/user/user@mymail.com/api/contents?content=1&1674049535812 (user@mymail.com@100.64.159.13) 34.65ms
13:45:36.302 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 424 on /
[W 2023-01-18 13:45:36.363 JupyterHub base:1409] Failing suspected API request to not-running server: /hub/user/user@mymail.com/api/metrics/v1
[W 2023-01-18 13:45:36.364 JupyterHub log:186] 424 GET /hub/user/user@mymail.com/api/metrics/v1?1674049535790 (user@mymail.com@100.64.159.13) 95.89ms
13:45:36.364 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 424 on /
[W 2023-01-18 13:45:36.471 JupyterHub log:186] 424 GET /hub/user/user@mymail.com/dask/clusters?1674049535762 (user@mymail.com@100.64.159.13) 106.34ms
13:45:36.471 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 424 on /
[W 2023-01-18 13:45:36.474 JupyterHub base:1409] Failing suspected API request to not-running server: /hub/user/user@mymail.com/api/terminals
[W 2023-01-18 13:45:36.475 JupyterHub log:186] 424 GET /hub/user/user@mymail.com/api/terminals?1674049535796 (user@mymail.com@100.64.159.13) 109.52ms
13:45:36.475 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 424 on /
[I 2023-01-18 13:45:36.483 JupyterHub reflector:274] watching for pods with label selector='component=singleuser-server' in namespace jh-namespace
[I 2023-01-18 13:45:36.486 JupyterHub reflector:274] watching for events with field selector='involvedObject.kind=Pod' in namespace jh-namespace
13:45:39.444 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /hub/spawn to http://127.0.0.1:8081
[I 2023-01-18 13:45:39.453 JupyterHub log:186] 200 GET /hub/spawn (user@mymail.com@100.64.159.13) 7.18ms
13:45:54.781 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /hub/spawn to http://127.0.0.1:8081
[I 2023-01-18 13:45:54.806 JupyterHub provider:651] Creating oauth client jupyterhub-user-user%40mymail.com
[W 2023-01-18 13:45:54.871 JupyterHub utils:77] 'notebook_container.security_context' current value: '{'runAsUser': 1000, 'allowPrivilegeEscalation': False}' is overridden with '{'runAsUser': 1001, 'runAsGroup': 1001}', which is the value of 'extra_container_config.security_context'.
[I 2023-01-18 13:45:54.872 JupyterHub spawner:2469] Attempting to create pod jupyterhub-nb-user-40mymail-2ecom, with timeout 3
[I 2023-01-18 13:45:55.787 JupyterHub log:186] 302 POST /hub/spawn -> /hub/spawn-pending/user@mymail.com (user@mymail.com@100.64.159.13) 1003.78ms
13:45:55.787 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 302 on /
13:45:56.030 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /hub/spawn-pending/user@mymail.com to http://127.0.0.1:8081
[I 2023-01-18 13:45:56.097 JupyterHub pages:394] user@mymail.com is pending spawn
[I 2023-01-18 13:45:56.101 JupyterHub log:186] 200 GET /hub/spawn-pending/user@mymail.com (user@mymail.com@100.64.159.13) 68.57ms
13:45:56.398 [ConfigProxy] ←[34mdebug←[39m: PROXY WEB /hub/api/users/user@mymail.com/server/progress to http://127.0.0.1:8081
[W 2023-01-18 13:45:56.403 JupyterHub base:89] Blocking Cross Origin API request.  Referer: https://REDACTED/hub/spawn-pending/user@mymail.com, Host: REDACTED, Host URL: http://REDACTED/hub/
[W 2023-01-18 13:45:56.403 JupyterHub scopes:804] Not authorizing access to /hub/api/users/user@mymail.com/server/progress. Requires any of [read:servers], not derived from scopes []
[W 2023-01-18 13:45:56.403 JupyterHub web:1796] 403 GET /hub/api/users/user@mymail.com/server/progress (100.64.159.13): Action is not authorized with current scopes; requires any of [read:servers]
[W 2023-01-18 13:45:56.404 JupyterHub log:186] 403 GET /hub/api/users/user@mymail.com/server/progress (@100.64.159.13) 4.38ms
13:45:56.405 [ConfigProxy] ←[34mdebug←[39m: Not recording activity for status 403 on /
[W 2023-01-18 13:46:04.788 JupyterHub base:1095] User user@mymail.com is slow to become responsive (timeout=10)
[I 2023-01-18 13:46:08.380 JupyterHub log:186] 200 GET /hub/api (@100.64.175.212) 0.71ms
[I 2023-01-18 13:46:10.523 JupyterHub log:186] 200 POST /hub/api/users/user@mymail.com/activity (user@mymail.com@100.64.175.212) 14.50ms
[I 2023-01-18 13:46:11.649 JupyterHub base:963] User user@mymail.com took 16.863 seconds to start
[I 2023-01-18 13:46:11.649 JupyterHub proxy:333] Adding user user@mymail.com to proxy /user/user@mymail.com/ => http://100.64.175.212:8080
13:46:11.651 [ConfigProxy] ←[32minfo←[39m: Adding route /user/user@mymail.com -> http://100.64.175.212:8080
13:46:11.651 [ConfigProxy] ←[32minfo←[39m: Route added /user/user@mymail.com -> http://100.64.175.212:8080
13:46:11.652 [ConfigProxy] ←[32minfo←[39m: 201 POST /api/routes/user/user@mymail.com

This usually means your proxy or loadbalancer isn’t setting the correct headers, so JupyterHub isn’t aware that you’re using a HTTPS connection, which means the host origins don’t match.

Regarding the connection problems, have you tried using the official Helm Chart?

How was your cluster setup? Do you have name resolution problems with other services?

I needed some time to play around.

I don’t have resolution problems with the other services; in general everything looks ok except for Jupyterhub.

I tried to make the suggested change, but have the same error (I’m using ingress nginx controller).
The configuration of the NGINX controller is below:
image

I also tried setting ssl-redirect: "true", but then I totally lost access to Jupyterhub, so I reverted it.

Please let me know if any information is needed. I’m configuring Jupyterhub for corprate environment, so I might not be able to share some things.

P.S. I’m accessing Jupyterhub from the internal corporate network, so there should be no firewall or other issues with the corporate network; I have already tested successfully other applications (e.g. REST APIs) in the same EKS cluster.

Oh, and what about the random failures when curling localhost:8081 from a notebook pod’s terminal?

Not sure, but this looks kind of different than the CORS issue…

It’ll be a different problem. I’m afraid you’re going to have to do some debugging yourself since you’re using a custom deployment and haven’t given much information about it. If you try the official Helm chart instead it’ll be easier for us to help since we’ll be starting from a known good configuration.

Do you have any hub logs corresponding to the curl failures? Is the pod begin forced to restart by your k8s cluster, e.g. due to a failing health check?

If you’re using EKS where does this openshift error come from?