Oauth flow in jupyterhub on z2jh

koder · June 29, 2022, 10:08pm

Jupyterhub is setup in API only mode. I
First redirect is to → /hub/api/oauth2/authorize
Next redirect → /hub/login which is giving 404
auto_login is set to true in the config.
Is there way to skip hub/login?
The github auth flow in the jhub documentation does show the redirect to /hub/login
https://jupyterhub.readthedocs.io/en/stable/reference/oauth.html

using keycloak as the auth provider. z2jh is deployed using the latest helm chart.
Thanks!

koder · July 1, 2022, 1:16pm

Edited the original:
Jupyterhub is setup in API only mode.
When I try to access a notebook that was spawned using an api call, the flow is like below
user1/notebook → /hub/api/oauth2/authorize
Next redirect → /hub/login which is giving 404
auto_login is set to true in the config.
Is there way to skip hub/login redirect?
The github auth flow in the jhub documentation does not show the redirect to /hub/login page.
https://jupyterhub.readthedocs.io/en/stable/reference/oauth.html

using keycloak as the auth provider. z2jh is deployed using the latest helm chart.
Thanks!

manics · July 1, 2022, 7:11pm

Hi! Please can you show us your Z2JH configuration with secrets redacted, tell us your version of Z2JH, and show us the hub logs with debug logging enabled?

koder · July 1, 2022, 8:00pm

Hi @manics!

chart version is 1.1.3-n674.h4367794e

config

logs

hub [D 2022-07-01 19:55:12.562 JupyterHub log:189] 200 GET /hub/health (@10.0.22.117) 0.78ms
hub [D 2022-07-01 19:55:14.562 JupyterHub log:189] 200 GET /hub/health (@10.0.22.117) 0.92ms
hub [I 2022-07-01 19:55:15.895 JupyterHub log:189] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-kadmin&redirect_uri=%2Fuser%2Fkadmin%2Foauth_callback&response_type=code&state=[secret] -> /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-kadmin%26redirect_uri%3D%252Fuser%252Fkadmin%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjVjMzA4ZjhlNzQyODQyMmU4OGQ0ZTMxMzY1NzY4MzRiIiwgIm5leHRfdXJsIjogIi91c2VyL2thZG1pbi9sYWIiLCAiY29va2llX25hbWUiOiAianVweXRlcmh1Yi11c2VyLWthZG1pbi1vYXV0aC1zdGF0ZS15dHNtcHVyZSJ9 (@::ffff:10.0.22.117) 0.83ms
hub [D 2022-07-01 19:55:16.562 JupyterHub log:189] 200 GET /hub/health (@10.0.22.117) 0.68ms
hub [D 2022-07-01 19:55:18.083 JupyterHub reflector:362] pods watcher timeout
hub [D 2022-07-01 19:55:36.562 JupyterHub log:189] 200 GET /hub/health (@10.0.22.117) 0.83ms
hub [I 2022-07-01 19:55:37.156 JupyterHub log:189] 200 GET /hub/error/404?url=%2Fhub%2Flogin%3Fnext%3D%252Fhub%252Fapi%252Foauth2%252Fauthorize%253Fclient_id%253Djupyterhub-user-kadmin%2526redirect_uri%253D%25252Fuser%25252Fkadmin%25252Foauth_callback%2526response_type%253Dcode%2526state%253DeyJ1dWlkIjogIjVjMzA4ZjhlNzQyODQyMmU4OGQ0ZTMxMzY1NzY4MzRiIiwgIm5leHRfdXJsIjogIi91c2VyL2thZG1pbi9sYWIiLCAiY29va2llX25hbWUiOiAianVweXRlcmh1Yi11c2VyLWthZG1pbi1vYXV0aC1zdGF0ZS15dHNtcHVyZSJ9 (@10.0.42.200) 1.35ms
hub [I 2022-07-01 19:55:37.285 JupyterHub log:189] 200 GET /hub/error/404?url=%2Fhub%2Fstatic%2Fcss%2Fstyle.min.css%3Fv%3Dbff49b4a161afb17ee3b71927ce7d6c4e5b0e4b9ef6f18ca3e356a05f29e69776d3a76aee167060dd2ae2ee62d3cfdcf203b4b0090b1423f7d629ea7daa3f9da (@10.0.42.200) 1.24ms
hub [I 2022-07-01 19:55:37.347 JupyterHub log:189] 200 GET /hub/error/404?url=%2Fhub%2Fstatic%2Fcomponents%2Fjquery%2Fdist%2Fjquery.min.js%3Fv%3Df3de1813a4160f9239f4781938645e1589b876759cd50b7936dbd849a35c38ffaed53f6a61dbdd8a1cf43cf4a28aa9fffbfddeec9a3811a1bb4ee6df58652b31 (@10.0.42.200) 1.61ms
hub [I 2022-07-01 19:55:37.350 JupyterHub log:189] 200 GET /hub/error/404?url=%2Fhub%2Fstatic%2Fcomponents%2Frequirejs%2Frequire.js%3Fv%3Dbd1aa102bdb0b27fbf712b32cfcd29b016c272acf3d864ee8469376eaddd032cadcf827ff17c05a8c8e20061418fe58cf79947049f5c0dff3b4f73fcc8cad8ec (@10.0.42.200) 3.01ms
hub [I 2022-07-01 19:55:37.358 JupyterHub log:189] 200 GET /hub/error/404?url=%2Fhub%2Fstatic%2Fcomponents%2Fbootstrap%2Fdist%2Fjs%2Fbootstrap.min.js%3Fv%3Da014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce (@10.0.42.200) 4.18ms
hub [I 2022-07-01 19:55:37.418 JupyterHub log:189] 200 GET /hub/error/404?url=%2Fhub%2Flogo (@10.0.42.200) 2.66ms
hub [I 2022-07-01 19:55:37.439 JupyterHub log:189] 200 GET /hub/error/404?url=%2Fhub%2Fstatic%2Fcomponents%2Fjquery%2Fdist%2Fjquery.min.js%3Fv%3Df3de1813a4160f9239f4781938645e1589b876759cd50b7936dbd849a35c38ffaed53f6a61dbdd8a1cf43cf4a28aa9fffbfddeec9a3811a1bb4ee6df58652b31 (@10.0.42.200) 1.24ms
hub [I 2022-07-01 19:55:37.535 JupyterHub log:189] 200 GET /hub/error/404?url=%2Fhub%2Fstatic%2Fcomponents%2Fbootstrap%2Fdist%2Fjs%2Fbootstrap.min.js%3Fv%3Da014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce (@10.0.42.200) 2.04ms
hub [I 2022-07-01 19:55:37.644 JupyterHub log:189] 200 GET /hub/error/404?url=%2Fhub%2Flogo (@10.0.42.200) 1.52ms
hub [D 2022-07-01 19:55:38.125 JupyterHub reflector:362] events watcher timeout
hub [D 2022-07-01 19:55:38.126 JupyterHub reflector:281] Connecting events watcher
hub [D 2022-07-01 19:55:38.129 JupyterHub reflector:362] pods watcher timeout

manics · July 2, 2022, 1:30pm

It looks like you’re trying to connect to a notebook but there’s no way for your user to be identified since since you’ve disabled the login endpoints. auto_login means the login page automatically redirects to your OAuth provider.

You’ll need to handle authentication some other way:
https://jupyterhub.readthedocs.io/en/stable/reference/api-only.html#rich-ui-customization-with-rest-api-based-apps

koder · July 3, 2022, 1:35am

I have tried adding login endpoints using extra_routes in config

I see in the logs that these routes are constantly being added and deleted. Is this expected behavior?

hub [D 2022-07-02 05:20:11.044 JupyterHub proxy:821] Proxy: Fetching GET http://proxy-api:8001/api/routes
hub [D 2022-07-02 05:20:11.049 JupyterHub proxy:346] Checking routes
hub [W 2022-07-02 05:20:11.049 JupyterHub proxy:422] Deleting stale route /hub/login/
hub [W 2022-07-02 05:20:11.049 JupyterHub proxy:422] Deleting stale route /hub/oauth_login/
hub [W 2022-07-02 05:20:11.049 JupyterHub proxy:422] Deleting stale route /hub/oauth_callback/
hub [D 2022-07-02 05:20:11.049 JupyterHub proxy:821] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub/login
hub [D 2022-07-02 05:20:11.050 JupyterHub proxy:821] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub/oauth_login
hub [D 2022-07-02 05:20:11.050 JupyterHub proxy:821] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub/oauth_callback
hub [D 2022-07-02 05:20:11.050 JupyterHub proxy:821] Proxy: Fetching DELETE http://proxy-api:8001/api/routes/hub/login
hub [D 2022-07-02 05:20:11.050 JupyterHub proxy:821] Proxy: Fetching DELETE http://proxy-api:8001/api/routes/hub/oauth_login
hub [D 2022-07-02 05:20:11.050 JupyterHub proxy:821] Proxy: Fetching DELETE http://proxy-api:8001/api/routes/hub/oauth_callback
hub [D 2022-07-02 05:21:11.044 JupyterHub proxy:821] Proxy: Fetching GET http://proxy-api:8001/api/routes
hub [D 2022-07-02 05:21:11.048 JupyterHub proxy:346] Checking routes
hub [D 2022-07-02 05:21:11.049 JupyterHub proxy:821] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub/login
hub [D 2022-07-02 05:21:11.049 JupyterHub proxy:821] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub/oauth_login
hub [D 2022-07-02 05:21:11.050 JupyterHub proxy:821] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub/oauth_callback
hub [D 2022-07-02 05:22:11.043 JupyterHub proxy:821] Proxy: Fetching GET http://proxy-api:8001/api/routes
hub [D 2022-07-02 05:22:11.048 JupyterHub proxy:346] Checking routes
hub [W 2022-07-02 05:22:11.048 JupyterHub proxy:422] Deleting stale route /hub/login/
hub [W 2022-07-02 05:22:11.048 JupyterHub proxy:422] Deleting stale route /hub/oauth_login/
hub [W 2022-07-02 05:22:11.048 JupyterHub proxy:422] Deleting stale route /hub/oauth_callback/
hub [D 2022-07-02 05:22:11.049 JupyterHub proxy:821] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub/login
hub [D 2022-07-02 05:22:11.049 JupyterHub proxy:821] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub/oauth_login
hub [D 2022-07-02 05:22:11.049 JupyterHub proxy:821] Proxy: Fetching POST http://proxy-api:8001/api/routes/hub/oauth_callback
hub [D 2022-07-02 05:22:11.049 JupyterHub proxy:821] Proxy: Fetching DELETE http://proxy-api:8001/api/routes/hub/login
hub [D 2022-07-02 05:22:11.049 JupyterHub proxy:821] Proxy: Fetching DELETE http://proxy-api:8001/api/routes/hub/oauth_login
hub [D 2022-07-02 05:22:11.049 JupyterHub proxy:821] Proxy: Fetching DELETE http://proxy-api:8001/api/routes/hub/oauth_callback

manics · July 4, 2022, 9:03pm

Changing the proxy routes won’t help, as it’s the hub that would normally handle user authentication. If you want to use the hub in API-only mode you’ll have to implement your own authentication process.

minrk · July 7, 2022, 8:25pm

I believe this approach will work to add hub routes one by one, but prefix routes must end in /. We should handle normalizing these trailing slashes if they’re missing (we do this almost everywhere url prefixes come up, but apparently not extra_routes, yet). So if you do:

extraConfig:
  Proxy:
    extra_routes:
      "/hub/login/": "http://hub:8081"
      ...

I think it might work.

koder · July 7, 2022, 8:38pm

It works. I figured out the ‘/’ at the end after a little bit of debugging. Thanks for all the help.

minrk · July 8, 2022, 4:45pm

validate proxy.extra_routes by minrk · Pull Request #3967 · jupyterhub/jupyterhub · GitHub will accept the missing slash with a warning instead of misbehaving