Missing or invalid credentials issue with Jupyterhub>4.0.1

JupyterHub Zero to JupyterHub on Kubernetes
, , ,
1

Hello!

For Z2JHv2 I get this error:

On JupyterLab:

[E 2024-01-24 10:00:03.042 JupyterHubSingleUser] Error notifying Hub of activity
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/extension.py", line 425, in notify
        await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[E 2024-01-24 10:00:03.042 JupyterHubSingleUser] Error notifying Hub of activity
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/extension.py", line 450, in keep_activity_updated
        await self.notify_activity()
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/extension.py", line 432, in notify_activity
        await exponential_backoff(
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff
        raise asyncio.TimeoutError(fail_message)
    asyncio.exceptions.TimeoutError: Failed to notify Hub of activity

On Jupyterhub:

[D 2024-01-24 09:58:05.774 JupyterHub user:431] Creating <class 'kubespawner.spawner.KubeSpawner'> for myuser:
[I 2024-01-24 09:58:05.790 JupyterHub log:191] 200 GET /hub/spawn-pending/myuser (myuser@10.2.132.224) 23.15ms
[D 2024-01-24 09:58:06.063 JupyterHub log:191] 200 GET /hub/static/js/not_running.js?v=20240124095657 (@10.2.132.224) 0.94ms
[D 2024-01-24 09:58:07.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.71ms
[D 2024-01-24 09:58:07.250 JupyterHub scopes:877] Checking access to /hub/spawn/myuser via scope servers
[D 2024-01-24 09:58:07.250 JupyterHub scopes:677] Unrestricted access to /hub/spawn/myuser via servers
[D 2024-01-24 09:58:07.966 JupyterHub pages:209] Serving options form for myuser
[I 2024-01-24 09:58:07.973 JupyterHub log:191] 200 GET /hub/spawn/myuser (myuser@10.2.132.224) 726.40ms
[D 2024-01-24 09:58:08.206 JupyterHub reflector:374] pods watcher timeout
[D 2024-01-24 09:58:08.206 JupyterHub reflector:289] Connecting pods watcher
[D 2024-01-24 09:58:09.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.73ms
[D 2024-01-24 09:58:11.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.78ms
[D 2024-01-24 09:58:11.887 JupyterHub scopes:877] Checking access to /hub/spawn/myuser via scope servers
[D 2024-01-24 09:58:11.887 JupyterHub scopes:677] Unrestricted access to /hub/spawn/myuser via servers
[D 2024-01-24 09:58:11.887 JupyterHub pages:257] Triggering spawn with supplied form options for myuser
[D 2024-01-24 09:58:11.888 JupyterHub base:961] Initiating spawn for myuser
[D 2024-01-24 09:58:11.888 JupyterHub base:965] 0/64 concurrent spawns
[D 2024-01-24 09:58:11.888 JupyterHub base:970] 1 active servers
[I 2024-01-24 09:58:11.928 JupyterHub provider:659] Creating oauth client jupyterhub-user-myuser
[I 2024-01-24 09:58:11.947 JupyterHub log:191] 302 POST /hub/spawn/myuser?_xsrf=[secret] -> /hub/spawn-pending/myuser?_xsrf=[secret] (myuser@10.2.132.224) 63.70ms
[D 2024-01-24 09:58:12.404 JupyterHub user:794] Calling Spawner.start for myuser
[D 2024-01-24 09:58:12.405 JupyterHub spawner:3220] Applying KubeSpawner override for profile 'my profile'
[D 2024-01-24 09:58:12.405 JupyterHub spawner:3198] .. overriding KubeSpawner value image=myimage
[D 2024-01-24 09:58:12.414 JupyterHub scopes:877] Checking access to /hub/spawn-pending/myuser via scope servers
[D 2024-01-24 09:58:12.414 JupyterHub scopes:677] Unrestricted access to /hub/spawn-pending/myuser via servers
[I 2024-01-24 09:58:12.415 JupyterHub pages:398] myuser is pending spawn
[I 2024-01-24 09:58:12.419 JupyterHub log:191] 200 GET /hub/spawn-pending/myuser?_xsrf=[secret] (myuser@10.2.132.224) 11.68ms
[I 2024-01-24 09:58:12.420 JupyterHub reflector:282] watching for events with field selector='involvedObject.kind=Pod' in namespace mynamespace
[D 2024-01-24 09:58:12.420 JupyterHub reflector:289] Connecting events watcher
[I 2024-01-24 09:58:12.439 JupyterHub spawner:2720] Pod is being modified via modify_pod_hook
[I 2024-01-24 09:58:19.235 JupyterHub spawner:2511] Attempting to create pod jupyter-myuser, with timeout 3
[D 2024-01-24 09:58:19.247 JupyterHub scopes:877] Checking access to /hub/api/users/myuser/server/progress via scope read:servers
[D 2024-01-24 09:58:19.247 JupyterHub scopes:677] Unrestricted access to /hub/api/users/myuser/server/progress via read:servers
[D 2024-01-24 09:58:19.249 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 6.97ms
[D 2024-01-24 09:58:19.249 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 7.04ms
[D 2024-01-24 09:58:19.250 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 7.09ms
[D 2024-01-24 09:58:19.250 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 7.15ms
[D 2024-01-24 09:58:19.250 JupyterHub reflector:374] pods watcher timeout
[D 2024-01-24 09:58:19.250 JupyterHub reflector:289] Connecting pods watcher
[D 2024-01-24 09:58:19.253 JupyterHub spawner:2303] progress generator: jupyter-myuser
[W 2024-01-24 09:58:20.388 JupyterHub web:1873] 403 GET /hub/metrics (172.16.8.4): Access to metrics requires scope 'read:metrics'
[D 2024-01-24 09:58:20.389 JupyterHub base:1371] No template for 403
[W 2024-01-24 09:58:20.390 JupyterHub log:191] 403 GET /hub/metrics (@172.16.8.4) 1.79ms
[D 2024-01-24 09:58:21.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.73ms
[D 2024-01-24 09:58:23.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.74ms
[D 2024-01-24 09:58:25.030 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 1.60ms
[D 2024-01-24 09:58:27.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.70ms
[D 2024-01-24 09:58:29.030 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.74ms
[D 2024-01-24 09:58:29.245 JupyterHub reflector:374] events watcher timeout
[D 2024-01-24 09:58:29.245 JupyterHub reflector:289] Connecting events watcher
[D 2024-01-24 09:58:29.288 JupyterHub reflector:374] pods watcher timeout
[D 2024-01-24 09:58:29.288 JupyterHub reflector:289] Connecting pods watcher
[D 2024-01-24 09:58:31.030 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.78ms
[D 2024-01-24 09:58:33.030 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.90ms
[D 2024-01-24 09:58:35.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.79ms
[D 2024-01-24 09:58:37.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.67ms
[D 2024-01-24 09:58:39.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.77ms
[D 2024-01-24 09:58:39.261 JupyterHub reflector:374] events watcher timeout
[D 2024-01-24 09:58:39.261 JupyterHub reflector:289] Connecting events watcher
[D 2024-01-24 09:58:39.305 JupyterHub reflector:374] pods watcher timeout
[D 2024-01-24 09:58:39.305 JupyterHub reflector:289] Connecting pods watcher
[D 2024-01-24 09:58:41.030 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.74ms
[D 2024-01-24 09:58:43.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.67ms
[D 2024-01-24 09:58:45.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.78ms
[D 2024-01-24 09:58:47.030 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.71ms
[D 2024-01-24 09:58:49.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.73ms
[D 2024-01-24 09:58:49.293 JupyterHub reflector:374] events watcher timeout
[D 2024-01-24 09:58:49.293 JupyterHub reflector:289] Connecting events watcher
[D 2024-01-24 09:58:49.318 JupyterHub reflector:374] pods watcher timeout
[D 2024-01-24 09:58:49.318 JupyterHub reflector:289] Connecting pods watcher
[D 2024-01-24 09:58:51.030 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.70ms
[D 2024-01-24 09:58:51.166 JupyterHub spawner:2821] pod mynamespace/jupyter-myuser events before launch: 2024-01-24T09:58:19.420066Z [Normal] Successfully assigned
mynamespace/jupyter-myuser to aks-linuxpool-20983041-vmss000fqu
    2024-01-24T09:58:24Z [Normal] Container image "myimage2" already present on machine
    2024-01-24T09:58:24Z [Normal] Created container create-user-folder
    2024-01-24T09:58:24Z [Normal] Started container create-user-folder
    2024-01-24T09:58:30Z [Normal] Pulling image "myimage"
    2024-01-24T09:58:49Z [Normal] Successfully pulled image "myimage" in 19.001188223s
    2024-01-24T09:58:49Z [Normal] Created container notebook
    2024-01-24T09:58:50Z [Normal] Started container notebook
[D 2024-01-24 09:58:51.173 JupyterHub spawner:1384] Polling subprocess every 30s
[D 2024-01-24 09:58:53.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.71ms
[D 2024-01-24 09:58:55.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.68ms
[D 2024-01-24 09:58:57.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.68ms
[D 2024-01-24 09:58:58.110 JupyterHub proxy:880] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2024-01-24 09:58:58.115 JupyterHub utils:278] Server at http://refresher:8081/services/refresher/ responded with 404
[D 2024-01-24 09:58:58.115 JupyterHub app:2494] External service refresher running at http://refresher:8081/
[D 2024-01-24 09:58:58.116 JupyterHub proxy:953] Omitting non-jupyterhub route '/'
[D 2024-01-24 09:58:58.130 JupyterHub proxy:392] Checking routes
[D 2024-01-24 09:58:59.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.71ms
[D 2024-01-24 09:58:59.308 JupyterHub reflector:374] events watcher timeout
[D 2024-01-24 09:58:59.309 JupyterHub reflector:289] Connecting events watcher
[D 2024-01-24 09:58:59.332 JupyterHub reflector:374] pods watcher timeout
[D 2024-01-24 09:58:59.333 JupyterHub reflector:289] Connecting pods watcher
[I 2024-01-24 09:59:00.022 JupyterHub log:191] 200 GET /hub/api (@172.16.8.63) 0.68ms
[W 2024-01-24 09:59:00.209 JupyterHub web:1873] 403 POST /hub/api/users/myuser/activity (172.16.8.63): Missing or invalid credentials.
[W 2024-01-24 09:59:00.209 JupyterHub log:191] 403 POST /hub/api/users/myuser/activity (@172.16.8.63) 0.98ms
[W 2024-01-24 09:59:00.572 JupyterHub web:1873] 403 POST /hub/api/users/myuser/activity (172.16.8.63): Missing or invalid credentials.
[W 2024-01-24 09:59:00.572 JupyterHub log:191] 403 POST /hub/api/users/myuser/activity (@172.16.8.63) 0.97ms
[W 2024-01-24 09:59:00.668 JupyterHub web:1873] 403 POST /hub/api/users/myuser/activity (172.16.8.63): Missing or invalid credentials.
[W 2024-01-24 09:59:00.668 JupyterHub log:191] 403 POST /hub/api/users/myuser/activity (@172.16.8.63) 1.00ms
[D 2024-01-24 09:59:01.030 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.77ms
[D 2024-01-24 09:59:01.162 JupyterHub utils:278] Server at http://172.16.8.63:8888/user/myuser/ responded with 302
[W 2024-01-24 09:59:01.162 JupyterHub _version:37] Single-user server has no version header, which means it is likely < 0.8. Expected 4.0.2
[I 2024-01-24 09:59:01.163 JupyterHub base:990] User myuser took 49.275 seconds to start
[I 2024-01-24 09:59:01.163 JupyterHub proxy:330] Adding user myuser to proxy /user/myuser/ => http://172.16.8.63:8888
[D 2024-01-24 09:59:01.163 JupyterHub proxy:880] Proxy: Fetching POST http://proxy-api:8001/api/routes/user/myuser
[I 2024-01-24 09:59:01.166 JupyterHub users:768] Server myuser is ready
[I 2024-01-24 09:59:01.166 JupyterHub log:191] 200 GET /hub/api/users/myuser/server/progress?_xsrf=[secret] (myuser@10.2.132.224) 41925.09ms
[D 2024-01-24 09:59:01.434 JupyterHub scopes:877] Checking access to /hub/spawn-pending/myuser via scope servers
[D 2024-01-24 09:59:01.434 JupyterHub scopes:677] Unrestricted access to /hub/spawn-pending/myuser via servers
[I 2024-01-24 09:59:01.434 JupyterHub log:191] 302 GET /hub/spawn-pending/myuser?_xsrf=[secret] -> /user/myuser/ (myuser@10.2.132.224) 11.53ms
[D 2024-01-24 09:59:02.197 JupyterHub provider:420] Validating client id jupyterhub-user-myuser
[D 2024-01-24 09:59:02.200 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:362] Validating redirection uri /user/myuser/oauth_callback for cl
ient jupyterhub-user-myuser.
[D 2024-01-24 09:59:02.201 oauthlib.oauth2.rfc6749.grant_types.base base:231] Using provided redirect_uri /user/myuser/oauth_callback
[D 2024-01-24 09:59:02.201 JupyterHub provider:495] validate_redirect_uri: client_id=jupyterhub-user-myuser, redirect_uri=/user/myuser/oauth_callback
[D 2024-01-24 09:59:02.204 oauthlib.oauth2.rfc6749.grant_types.base base:172] Validating access to scopes ['read:users:groups!user', 'read:users:name!user', 'access:servers
!server=myuser/'] for client 'jupyterhub-user-myuser' (<OAuthClient(identifier='jupyterhub-user-myuser')>).
[D 2024-01-24 09:59:02.205 JupyterHub provider:622] Allowing request for scope(s) for jupyterhub-user-myuser:  read:users:groups!user,read:users:name!user,access:servers!s
erver=myuser/
[D 2024-01-24 09:59:02.207 JupyterHub auth:320] Skipping oauth confirmation for <User(myuser 1/1 running)> accessing Server at /user/myuser/
[D 2024-01-24 09:59:02.207 oauthlib.oauth2.rfc6749.endpoints.authorization authorization:98] Dispatching response_type code request to <oauthlib.oauth2.rfc6749.grant_types.
authorization_code.AuthorizationCodeGrant object at 0x7f2f9e42fe80>.
[D 2024-01-24 09:59:02.207 JupyterHub provider:420] Validating client id jupyterhub-user-myuser
[D 2024-01-24 09:59:02.209 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:362] Validating redirection uri /user/myuser/oauth_callback for cl
ient jupyterhub-user-myuser.
[D 2024-01-24 09:59:02.209 oauthlib.oauth2.rfc6749.grant_types.base base:231] Using provided redirect_uri /user/myuser/oauth_callback
[D 2024-01-24 09:59:02.209 JupyterHub provider:495] validate_redirect_uri: client_id=jupyterhub-user-myuser, redirect_uri=/user/myuser/oauth_callback
[D 2024-01-24 09:59:02.210 oauthlib.oauth2.rfc6749.grant_types.base base:172] Validating access to scopes {'read:users:groups!user', 'read:users:name!user', 'access:servers
!server=myuser/'} for client 'jupyterhub-user-myuser' (<OAuthClient(identifier='jupyterhub-user-myuser')>).
[D 2024-01-24 09:59:02.212 JupyterHub provider:622] Allowing request for scope(s) for jupyterhub-user-myuser:  read:users:groups!user,read:users:name!user,access:servers!s
erver=myuser/
[D 2024-01-24 09:59:02.212 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:245] Pre resource owner authorization validation ok for <oauthlib.Reque
st SANITIZED>.
[D 2024-01-24 09:59:02.212 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:170] Created authorization code grant {'code': 'DCpVJmUw2pQ9oiL8BCVj0oH
lR6AafU', 'state': 'eyJ1dWlkIjogIjBmMzRh..cvdXNlci9iYW5zeWFzL2xhYj8ifQ'} for request <oauthlib.Request SANITIZED
>.
[D 2024-01-24 09:59:02.212 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:277] Saving grant {'code': 'DCpVJmUw2pQ9oiL8BCVj0oHlR6AafU', 'state': '
eyJ1dWlkIjogIjBmM..W5zeWFzL2xhYj8ifQ'} for <oauthlib.Request SANITIZED>.
[D 2024-01-24 09:59:02.212 JupyterHub provider:246] Saving authorization code jupyterhub-user-myuser, DCp..., (), {}
[I 2024-01-24 09:59:02.222 JupyterHub log:191] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-myuser&redirect_uri=%2Fuser%2Fmyuser%2Foauth_callback
&response_type=code&state=[secret] -> /user/myuser/oauth_callback?code=[secret]&state=[secret] (myuser@10.2.132.224) 32.55ms
[D 2024-01-24 09:59:02.463 oauthlib.oauth2.rfc6749.endpoints.token token:112] Dispatching grant_type authorization_code request to <oauthlib.oauth2.rfc6749.grant_types.auth
orization_code.AuthorizationCodeGrant object at 0x7f2f9e42fe80>.
[D 2024-01-24 09:59:02.463 JupyterHub provider:57] authenticate_client <oauthlib.Request SANITIZED>
[D 2024-01-24 09:59:02.468 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:463] Client authentication failed, <oauthlib.Request SANITIZED>.
[D 2024-01-24 09:59:02.468 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:303] Client error during validation of <oauthlib.Request SANITIZED>. In
validClientError('(invalid_client)  <oauthlib.Request SANITIZED>').
[W 2024-01-24 09:59:02.469 JupyterHub log:191] 401 POST /hub/api/oauth2/token (@172.16.8.63) 6.60ms
[D 2024-01-24 09:59:03.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.79ms
[W 2024-01-24 09:59:04.168 JupyterHub web:1873] 403 POST /hub/api/users/myuser/activity (172.16.8.63): Missing or invalid credentials.
[W 2024-01-24 09:59:04.168 JupyterHub log:191] 403 POST /hub/api/users/myuser/activity (@172.16.8.63) 0.97ms
[D 2024-01-24 09:59:05.029 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.76ms
[D 2024-01-24 09:59:07.030 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.72ms
[D 2024-01-24 09:59:09.030 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.70ms
[W 2024-01-24 09:59:09.127 JupyterHub web:1873] 403 POST /hub/api/users/myuser/activity (172.16.8.63): Missing or invalid credentials.
[W 2024-01-24 09:59:09.128 JupyterHub log:191] 403 POST /hub/api/users/myuser/activity (@172.16.8.63) 1.03ms

Jupyterhub package versions :

aiohttp==3.9.1
aiosignal==1.3.1
alembic==1.13.1
async-generator==1.10
async-timeout==4.0.3
attrs==23.2.0
azure-core==1.29.7
azure-identity==1.15.0
bcrypt==4.1.2
cachetools==5.3.2
certifi==2023.11.17
certipy==0.1.3
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
cryptography==42.0.0
cx-Oracle==8.3.0
dask==2024.1.0
dask-gateway==2022.11.0
distributed==2024.1.0
docstring-parser==0.15
escapism==1.0.1
frozenlist==1.4.1
fsspec==2023.12.2
future==0.18.3
google-auth==2.26.2
greenlet==3.0.3
idna==3.6
importlib-metadata==7.0.1
Jinja2==3.1.3
JPype1==1.5.0
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
jupyter-telemetry==0.1.0
jupyterhub==4.0.2
jupyterhub-firstuseauthenticator==1.0.0
jupyterhub-hmacauthenticator==1.0
jupyterhub-idle-culler==1.2.1
jupyterhub-kubespawner==6.2.0
jupyterhub-ldapauthenticator==1.3.2
jupyterhub-ltiauthenticator==1.6.2
jupyterhub-nativeauthenticator==1.2.0
jupyterhub-tmpauthenticator==1.0.0
kazoo==2.9.0
kubernetes==29.0.0
kubernetes_asyncio==29.0.0
ldap3==2.9.1
locket==1.0.0
lxml==5.1.0
Mako==1.3.1
MarkupSafe==2.1.4
msal==1.26.0
msal-extensions==1.1.0
msgpack==1.0.7
multidict==6.0.4
mwoauth==0.4.0
NamedAtomicLock==1.1.3
nullauthenticator==1.0.0
numpy==1.26.3
oauthenticator==15.1.0
oauthlib==3.2.2
onetimepass==1.0.1
packaging==23.2
pamela==1.1.0
pandas==2.2.0
partd==1.4.1
portalocker==2.8.2
prometheus-client==0.19.0
psutil==5.9.8
psycopg2-binary==2.9.9
py-spy==0.3.14
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
pycryptodome==3.20.0
pycurl==7.45.2
PyJWT==2.8.0
PyMySQL==1.1.0
pyOpenSSL==24.0.0
pyparsing==3.1.1
pyrsistent==0.20.0
python-dateutil==2.8.2
python-json-logger==2.0.7
python-slugify==8.0.1
pytz==2023.3.post1
PyYAML==6.0.1
referencing==0.32.1
requests==2.31.0
requests-oauthlib==1.3.1
rpds-py==0.17.1
rsa==4.9
ruamel.yaml==0.18.5
ruamel.yaml.clib==0.2.8
scipy==1.12.0
six==1.16.0
sortedcontainers==2.4.0
SQLAlchemy==2.0.25
sqlalchemy-cockroachdb==2.0.2
statsd==4.0.1
tblib==3.0.0
text-unidecode==1.3
toolz==0.12.0
tornado==6.4
traitlets==5.14.1
typing_extensions==4.9.0
tzdata==2023.4
urllib3==2.1.0
websocket-client==1.7.0
yarl==1.9.4
zict==3.0.0
zipp==3.17.0

Jupyterlab packages:

- jupyter=1.0.0
- jupyter-lsp=2.2.1
- jupyter-packaging=0.12.3
- jupyter_console=6.6.3
- jupyter_core=5.7.1
- jupyter_events=0.6.3
- jupyter_server_terminals=0.5.1
- jupyter_telemetry=0.1.0
- jupyterlab_pygments=0.3.0
- jupyter-client==7.4.4
- jupyter-resource-usage==1.0.1
- jupyter-server==2.4.0
- jupyter-server-fileid==0.9.1
- jupyter-server-mathjax==0.2.6
- jupyter-server-proxy==3.2.1
- jupyter-server-ydoc==0.8.0
- jupyter-ydoc==0.3.1
- jupyterhub==4.0.2
- jupyterlab==4.0.11
- jupyterlab-code-formatter==2.2.1
- jupyterlab-git==0.50.0
- jupyterlab-lsp==5.0.2
- jupyterlab-server==2.22.1
- jupyterlab-templates==0.4.0
- jupyterlab-widgets==1.1.5
- oauthlib=3.2.2
- openssl=3.1.4
- requests-oauthlib==1.3.1

Here’s our custom authenticator for Auth0:

import json
import os
import logging
import time

from jupyterhub.auth import LocalAuthenticator
from tornado.httpclient import HTTPRequest
from traitlets import default
from traitlets import Unicode

from oauthenticator.oauth2 import OAuthenticator

MAX_RETRY = 3

class CustomAuth0OAuthenticator(OAuthenticator):

    login_service = "Auth0"

    auth0_subdomain = Unicode(config=True)
    logout_params = Unicode(config=True)

    @default("auth0_subdomain")
    def _auth0_subdomain_default(self):
        subdomain = os.getenv("AUTH0_SUBDOMAIN")
        if not subdomain:
            raise ValueError(
                "Please specify $AUTH0_SUBDOMAIN env or %s.auth0_subdomain config"
                % self.__class__.__name__
            )

    username_key = Unicode(
        os.environ.get("OAUTH2_USERNAME_KEY", "email"),
        config=True,
        help="Userdata username key from returned json with user data login information",
    )

    @default("logout_redirect_url")
    def _logout_redirect_url_default(self):
        return 'https://%s/v2/logout%s'%(self.auth0_subdomain, self.logout_params)

    @default("authorize_url")
    def _authorize_url_default(self):
        return "https://%s/authorize" % self.auth0_subdomain

    @default("token_url")
    def _token_url_default(self):
        return "https://%s/oauth/token" % self.auth0_subdomain

    async def authenticate(self, handler, data=None):
        code = handler.get_argument("code")

        params = {
            'grant_type': 'authorization_code',
            'client_id': self.client_id,
            'client_secret': self.client_secret,
            'code': code,
            'redirect_uri': self.get_callback_url(handler),
        }
        url = self.token_url

        req = HTTPRequest(
            url,
            method="POST",
            headers={"Content-Type": "application/json"},
            body=json.dumps(params),
        )

        retry = 0
        success = False
        while(not success):
            try:
                resp_json = await self.fetch(req)
                success = True
            except Exception as e:
                if retry < 3:
                    retry = retry + 1
                    time.sleep((1 << retry) * 1)
                    logging.info("Retry connection with oauth - %s "%retry)
                else:
                    raise e


        access_token = resp_json['access_token']

        refresh_token = resp_json.get('refresh_token')
        id_token = resp_json.get('id_token')

        # Determine who the logged in user is
        headers = {
            "Accept": "application/json",
            "User-Agent": "JupyterHub",
            "Authorization": "Bearer {}".format(access_token),
        }
        req = HTTPRequest(
            "https://%s/userinfo" % self.auth0_subdomain,
            method="GET",
            headers=headers,
        )
        resp_json = await self.fetch(req)

        name = resp_json.get(self.username_key)
        if not name:
            self.log.error(
                "Auth0 user contains no key %s: %s", self.username_key, resp_json
            )
            return
        
        logging.info("Returning auth_state - %s "%resp_json)

        return {
            'name': name,
            'auth_state': {
                'access_token': access_token,
                'refresh_token': refresh_token,
                'id_token': id_token,
                'auth0_user': resp_json,
            },
        }


class LocalCustomAuth0OAuthenticator(LocalAuthenticator, CustomAuth0OAuthenticator):

    """A version that mixes in local system user creation"""

    pass
2

Update:
For Jupyterhub 4.0.1 & 4.0.0,
Jupyterhub logs :

[I 2024-01-25 07:15:34.874 JupyterHub log:191] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-myuser&redirect_uri=%2Fuser%2Fmyuser%2Foauth_callback
&response_type=code&state=[secret] -> /user/myuser/oauth_callback?code=[secret]&state=[secret] (myuser@10.2.132.224) 37.38ms
[W 2024-01-25 07:15:35.130 JupyterHub web:1873] 403 POST /hub/api/oauth2/token (172.16.8.65): '_xsrf' argument missing from POST
[W 2024-01-25 07:15:35.130 JupyterHub log:191] 403 POST /hub/api/oauth2/token (@172.16.8.65) 1.09ms
[D 2024-01-25 07:15:36.143 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.74ms
[D 2024-01-25 07:15:38.142 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.73ms
[W 2024-01-25 07:15:38.585 JupyterHub web:1873] 403 POST /hub/api/users/myuser/activity (172.16.8.65): '_xsrf' argument missing from POST
[W 2024-01-25 07:15:38.586 JupyterHub log:191] 403 POST /hub/api/users/myuser/activity (@172.16.8.65) 1.12ms
[W 2024-01-25 07:15:39.906 JupyterHub web:1873] 403 POST /hub/api/users/myuser/activity (172.16.8.65): '_xsrf' argument missing from POST
[W 2024-01-25 07:15:39.907 JupyterHub log:191] 403 POST /hub/api/users/myuser/activity (@172.16.8.65) 0.97ms
[D 2024-01-25 07:15:40.143 JupyterHub log:191] 200 GET /hub/health (@172.16.9.1) 0.69ms
[D 2024-01-25 07:15:41.666 JupyterHub reflector:374] events watcher timeout
[D 2024-01-25 07:15:41.666 JupyterHub reflector:289] Connecting events watcher
[D 2024-01-25 07:15:41.678 JupyterHub reflector:374] pods watcher timeout

And for Jupyterlab:

[I 2024-01-25 07:15:33.083 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[E 2024-01-25 07:15:33.099 JupyterHubSingleUser] Error notifying Hub of activity
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/extension.py", line 425, in notify
        await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[E 2024-01-25 07:15:33.966 JupyterHubSingleUser] Error notifying Hub of activity
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/extension.py", line 425, in notify
        await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
......
[E 2024-01-25 07:15:34.594 JupyterHubSingleUser] Error notifying Hub of activity
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/extension.py", line 425, in notify
        await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
......
2024-01-25 07:15:35,135 - ServerApp - ERROR - I don't have permission to check authorization with JupyterHub, my auth token may have expired: [403] Request forbidden -- aut
horization will not help
2024-01-25 07:15:35,135 - ServerApp - ERROR - {"status": 403, "message": "'_xsrf' argument missing from POST"}
.......
[E 2024-01-25 07:15:38.591 JupyterHubSingleUser] Error notifying Hub of activity
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/extension.py", line 425, in notify
        await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden

IMPORTANT: This deployment is hosted on Azure K8s Service behind an Azure Application Gateway

3

Which version of the Z2JH Helm chart are you using? Can you share your full configuration?

Could you try simplifying your deployment to help rule out some problems. For example, use as close to the default Z2JH configuration as possible with the dummy authenticator and check everything is working. If it is then add your customisations back one at a time.

4

I’m also experiencing something similar.

@yashali have you made any progress? It looks like the self.hub_auth.api_token in extension.py is incorrect or not being set.

I know it defaults to JUPYTERHUB_API_TOKEN. Before I explicitly set JUPYTERHUB_API_TOKEN, JupyterLab was passing an empty string.

I then explicitly set it using openssl but haven’t had success yet.

In my case, I’m using Cognito for auth and a custom spawner.

Obviously, I haven’t given enough information here specific to my configuration to properly debug, but wanted to ping this thread to keep it alive.

I’m deploying now with dummy auth to see if I can narrow the problem and will report back.

5

Ok - dummy auth did not help. Still getting the following in JupyterLab:

[E 2024-02-02 01:24:08.760 JupyterHubSingleUser] Error notifying Hub of activity

2024-02-01T19:24:08.760-06:00	Traceback (most recent call last):

2024-02-01T19:24:08.760-06:00	File "/usr/local/lib/python3.10/dist-packages/jupyterhub/singleuser/extension.py", line 425, in notify

2024-02-01T19:24:08.760-06:00	await client.fetch(req)

2024-02-01T19:24:08.760-06:00	tornado.httpclient.HTTPClientError: HTTP 403: Forbidden

2024-02-01T19:24:11.314-06:00	[E 2024-02-02 01:24:11.314 JupyterHubSingleUser] Error notifying Hub of activity

2024-02-01T19:24:11.314-06:00	Traceback (most recent call last):

2024-02-01T19:24:11.314-06:00	File "/usr/local/lib/python3.10/dist-packages/jupyterhub/singleuser/extension.py", line 425, in notify

2024-02-01T19:24:11.314-06:00	await client.fetch(req)

2024-02-01T19:24:11.314-06:00	tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
6

@manics thanks for your reply. We use Z2JH version 2. An older singleuser image works, so it has something to do with the packages that we are trying to upgrade in the image. Will have to upgrade each pkg one by one and get back to you

@Benjamin_Leff , No I havent found anything so far. Trying to zero in on the package upgrade that caused this. :crossed_fingers:

1 Like
7

Thanks so much @yashali. Super interesting that it’s probably a package issue for you guys.

We had a working implementation locally a week ago during development (without the custom spawner, and without pinned package versions, but with our Cognito OAuth) so I figured it had something to do with migration to the custom spawner in production.

Going to downgrade and do some more testing.

FWIW - we’re on AWS and not using k8s. Will post a stripped down version of our implementation shortly.

8

Can you try setting JUPYTERHUB_SINGLEUSER_EXTENSION=0 env var in single user container? In JupyterHub 4, single user implementation has been changed to be a jupyter server extension and this env var will use legacy implementation as in JupyterHub 3.

9

@mahendrapaipuri thanks for the suggestion. Unfortunately, that didn’t work.

@manics is there any reason you know of that can cause the JUPYTERHUB_API_TOKEN to be set to an empty string inside of the JupyterLab container?

What’s interesting is that it’s set in JupyterHub but gets reset in JupyterLab

10

Resolved! For future readers of custom spawners:

if you’ve created a helper class to break up some of the logic in your spawner, do not set the instance of your helper in the custom spawner’s __init__ method if you have environment variables (to be passed into the JupyterLab container) that you need in the helper.

For example, do not do this:

class CustomSpawner(Spawner):
    # Your traitlets...
    def __init__(self) -> None:
        self.helper = CustomSpawnerHelper(self, container_environ_vars=self.get_env())
    
    async def start(self) -> str:
        self.helper.do_work()

instead, do this:

class CustomSpawner(Spawner):
    # Your traitlets...
    
    async def start(self) -> str:
        helper = CustomSpawnerHelper(self, container_environ_vars=self.get_env())
        helper.do_work()

The reason is that the container env vars don’t get set yet in __init__.

Lesson learned. Maybe worth mentioning in the docs (it’s also possible this is documented and I missed it).

Cheers!

1 Like
11

thanks @mahendrapaipuri . Setting this env var helped in my case!

2 Likes