JupyterHub starts, and when a user logs in, it fails to spawn a notebook server on the page

JupyterHub
,
1

JupyterHub starts, and when a user logs in, it fails to spawn a notebook server on the page, and I check the container is started and can be exec in it , but the page show

503 : Service Unavailable
Your server appears to be down. Try restarting it [from the hub](http://127.0.0.1:16667/hub/home)

the error logs below:

01:52:33.185 [ConfigProxy] error: 503 GET /user/qycai/oauth_callback connect ECONNREFUSED 10.0.19.171:8888
[I 2021-12-10 01:52:33.189 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fqycai%2Foauth_callback%3Fcode%3D3cL8Gq5d9kqkoZEEkYzkMHlehq9nMM%26state%3DeyJ1dWlkIjogIjM1NDE2YjExYzNmYjQ3ZTRiYzRiZDg1OTEzZDJkYjg5IiwgIm5leHRfdXJsIjogIi91c2VyL3F5Y2FpL2xhYj8iLCAiY29va2llX25hbWUiOiAianVweXRlcmh1Yi11c2VyLXF5Y2FpLW9hdXRoLXN0YXRlLW55VHd4ZG5hIn0 (@10.0.19.86) 3.21ms
2

I am also having this issue on JupyterHub 3.0.0 running on the jupyterhub/k8s-hub:2.0.0 and associated helm chart 2.0.0. Did you ever find a resolution to this issue?

3

ECONNREFUSED suggests that the proxy pod cannot connect to the single-user server pod. Narrowing it down should mean trying to connect to the

  • check that the ip 10.0.19.171 is actually assigned to the pod
  • check if other pods can connect to that URL (e.g. the hub or a dedicated test pod)
  • check connecting to itself from within the user pod (test localhost and the pod ip)

Different combinations of those may point to different sources of the problem.

One possible culprit is network policies. You might try temporarily disabling the network policy for the singleuser server and/or proxy:

singleuser:
  networkPolicy:
    enabled: false
proxy:
  chp:
    networkPolicy: 
      enabled: false

This will at least tell you if one or both of those is the reason for the failure to connect.

4

Hi minrk,

Thanks for the response on this! I am not experiencing the ECONNREFUSED error, just the second piece about the user getting redirected to the 503 error page when they attempt to start their server.

I dug into this a bit deeper, and I found this issue, which seems to be what was happening to users. It seemed to be an authentication error, so this makes a lot of sense.

Not sure if this is still an issue in the helm chart 3.0.X, which I am currently reading through change logs in order to upgrade to.

Warning:

[W 2023-08-30 00:18:24.705 JupyterHub _version:68] jupyterhub version 3.0.0 != jupyterhub-singleuser version 3.1.1. This could cause failure to authenticate and result in redirect loops!

Warning in context:

[I 2023-08-30 00:16:47.001 JupyterHub base:810] User logged in: XXXXXX0454@example.com
[I 2023-08-30 00:16:47.069 JupyterHub log:186] 200 GET /hub/home (XXXXXX0454@example.com@::ffff:10.244.42.170) 19.11ms
[I 2023-08-30 00:16:52.249 JupyterHub log:186] 200 GET /hub/spawn/XXXXXX0454@example.com (XXXXXX0454@example.com@::ffff:10.244.42.170) 8.23ms
[I 2023-08-30 00:16:52.633 JupyterHub log:186] 302 GET /hub/spawn/XXXXXX0454@example.com -> /hub/login?next=%2Fhub%2Fspawn%2FXXXXXX0454%40example.com (@::ffff:10.244.42.170) 2.04ms
[I 2023-08-30 00:16:52.761 JupyterHub log:186] 200 GET /hub/login?next=%2Fhub%2Fspawn%2FXXXXXX0454%40example.com (@::ffff:10.244.42.170) 2.45ms
[I 2023-08-30 00:18:09.795 JupyterHub provider:651] Creating oauth client jupyterhub-user-XXXXXX0454%40example.com
[I 2023-08-30 00:18:09.812 JupyterHub spawner:2509] Attempting to create pvc claim-XXXXXX0454-40example-2ecom, with timeout 3
[I 2023-08-30 00:18:09.814 JupyterHub log:186] 302 POST /hub/spawn/XXXXXX0454@example.com -> /hub/spawn-pending/XXXXXX0454@example.com (XXXXXX0454@example.com@::ffff:10.244.42.170) 53.95ms
[I 2023-08-30 00:18:09.862 JupyterHub pages:394] XXXXXX0454@example.com is pending spawn
[I 2023-08-30 00:18:09.864 JupyterHub log:186] 200 GET /hub/spawn-pending/XXXXXX0454@example.com (XXXXXX0454@example.com@::ffff:10.244.42.170) 6.70ms
[I 2023-08-30 00:18:09.901 JupyterHub spawner:2525] PVC claim-XXXXXX0454-40example-2ecom already exists, so did not create new pvc.
[I 2023-08-30 00:18:09.914 JupyterHub spawner:2469] Attempting to create pod jupyter-XXXXXX0454-40example-2ecom, with timeout 3
[I 2023-08-30 00:18:10.180 JupyterHub log:186] 302 GET /hub/spawn-pending/XXXXXX0454@example.com -> /hub/login?next=%2Fhub%2Fspawn-pending%2FXXXXXX0454%40example.com (@::ffff:10.244.42.170) 1.44ms
[I 2023-08-30 00:18:10.305 JupyterHub log:186] 200 GET /hub/login?next=%2Fhub%2Fspawn-pending%2FXXXXXX0454%40example.com (@::ffff:10.244.42.170) 4.11ms
[I 2023-08-30 00:18:22.592 JupyterHub log:186] 200 POST /hub/api/users/XXXXXX0454@example.com/activity (XXXXXX0454@example.com@10.244.50.194) 16.20ms
[W 2023-08-30 00:18:24.705 JupyterHub _version:68] jupyterhub version 3.0.0 != jupyterhub-singleuser version 3.1.1. This could cause failure to authenticate and result in redirect loops!
[I 2023-08-30 00:18:24.705 JupyterHub base:963] User XXXXXX0454@example.com took 14.932 seconds to start
[I 2023-08-30 00:18:24.705 JupyterHub proxy:333] Adding user XXXXXX0454@example.com to proxy /user/XXXXXX0454@example.com/ => http://10.244.50.194:8888
[I 2023-08-30 00:18:24.707 JupyterHub users:749] Server XXXXXX0454@example.com is ready
[I 2023-08-30 00:18:24.708 JupyterHub log:186] 200 GET /hub/api/users/XXXXXX0454@example.com/server/progress (XXXXXX0454@example.com@::ffff:10.244.42.170) 14772.69ms
[I 2023-08-30 00:18:29.751 JupyterHub log:186] 200 GET /hub/error/503?url=%2Fhub%2Fspawn-pending%2FXXXXXX0454%40example.com (@10.244.114.205) 3.33ms
[I 2023-08-30 00:18:43.886 JupyterHub log:186] 200 GET /hub/home (XXXXXX0454@example.com@::ffff:10.244.42.170) 7.71ms
[W 2023-08-30 00:18:45.845 JupyterHub auth:298] Service Server at /user/XXXXXX0454@example.com/ requested scopes access:servers!server=XXXXXX0454@example.com/,read:users:name!user,read:users:groups!user,access:servers!user=XXXXXX0454@example.com for user XXXXXX0454@example.com, granting only access:servers!server=XXXXXX0454@example.com/,read:users:name!user,read:users:groups!user.
[I 2023-08-30 00:18:45.853 JupyterHub log:186] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-XXXXXX0454%2540example.com&redirect_uri=%2Fuser%2FXXXXXX0454%40example.com%2Foauth_callback&response_type=code&state=[secret] -> /user/XXXXXX0454@example.com/oauth_callback?code=[secret]&state=[secret] (XXXXXX0454@example.com@::ffff:10.244.42.170) 16.88ms
[I 2023-08-30 00:18:45.917 JupyterHub log:186] 200 POST /hub/api/oauth2/token (XXXXXX0454@example.com@10.244.50.194) 23.46ms
[I 2023-08-30 00:18:45.994 JupyterHub log:186] 200 GET /hub/api/user (XXXXXX0454@example.com@10.244.50.194) 10.90ms
[I 2023-08-30 00:18:46.653 JupyterHub log:186] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-XXXXXX0454%2540example.com&redirect_uri=%2Fuser%2FXXXXXX0454%40example.com%2Foauth_callback&response_type=code&state=[secret] -> /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-XXXXXX0454%252540example.com%26redirect_uri%3D%252Fuser%252FXXXXXX0454%2540example.com%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjVlNDNiYWRkZmNjNDQxZjM5NmY1NDQ0MTg4MTMyNTNkIiwgIm5leHRfdXJsIjogIi91c2VyL3lmYXJhajA0NTRAc2RzdS5lZHUvbGFiPyJ9 (@::ffff:10.244.42.170) 1.76ms
[I 2023-08-30 00:18:48.031 JupyterHub log:186] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-XXXXXX0454%2540example.com&redirect_uri=%2Fuser%2FXXXXXX0454%40example.com%2Foauth_callback&response_type=code&state=[secret] -> /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-XXXXXX0454%252540example.com%26redirect_uri%3D%252Fuser%252FXXXXXX0454%2540example.com%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjk1OWU5YzFiMGM3MTQzMmI4OGFmY2I5NDNiZjYwZTQ1IiwgIm5leHRfdXJsIjogIi91c2VyL3lmYXJhajA0NTRAc2RzdS5lZHUvbGFiIn0 (@::ffff:10.244.42.170) 1.92ms
[I 2023-08-30 00:18:48.166 JupyterHub log:186] 200 GET /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-XXXXXX0454%252540example.com%26redirect_uri%3D%252Fuser%252FXXXXXX0454%2540example.com%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjk1OWU5YzFiMGM3MTQzMmI4OGFmY2I5NDNiZjYwZTQ1IiwgIm5leHRfdXJsIjogIi91c2VyL3lmYXJhajA0NTRAc2RzdS5lZHUvbGFiIn0 (@::ffff:10.244.42.170) 3.57ms
[I 2023-08-30 00:18:51.807 JupyterHub log:186] 200 GET /hub/error/503?url=%2Fhub%2Flogin%3Fnext%3D%252Fhub%252Fapi%252Foauth2%252Fauthorize%253Fclient_id%253Djupyterhub-user-XXXXXX0454%25252540example.com%2526redirect_uri%253D%25252Fuser%25252FXXXXXX0454%252540example.com%25252Foauth_callback%2526response_type%253Dcode%2526state%253DeyJ1dWlkIjogIjVlNDNiYWRkZmNjNDQxZjM5NmY1NDQ0MTg4MTMyNTNkIiwgIm5leHRfdXJsIjogIi91c2VyL3lmYXJhajA0NTRAc2RzdS5lZHUvbGFiPyJ9 (@10.244.114.205) 3.15ms
[I 2023-08-30 00:18:56.844 JupyterHub log:186] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-XXXXXX0454%2540example.com&redirect_uri=%2Fuser%2FXXXXXX0454%40example.com%2Foauth_callback&response_type=code&state=[secret] -> /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-XXXXXX0454%252540example.com%26redirect_uri%3D%252Fuser%252FXXXXXX0454%2540example.com%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogImNmNDg3MDAwMjZiNDQyZDU4Mjc1Yzc5OThkOWNhNjAxIiwgIm5leHRfdXJsIjogIi91c2VyL3lmYXJhajA0NTRAc2RzdS5lZHUvZGVza3RvcCJ9 (@::ffff:10.244.42.170) 2.31ms
[I 2023-08-30 00:18:56.966 JupyterHub log:186] 200 GET /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-XXXXXX0454%252540example.com%26redirect_uri%3D%252Fuser%252FXXXXXX0454%2540example.com%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogImNmNDg3MDAwMjZiNDQyZDU4Mjc1Yzc5OThkOWNhNjAxIiwgIm5leHRfdXJsIjogIi91c2VyL3lmYXJhajA0NTRAc2RzdS5lZHUvZGVza3RvcCJ9 (@::ffff:10.244.42.170) 4.23ms