We have a JupyterHub deployment that uses FirecRESTSpawner. Keycloak is used both for JupyterHub authentication and for authentication with the FirecREST service, which handles interactions with HPC clusters.
We run periodic automated tests to verify that notebook servers can be launched successfully. However, since upgrading to JupyterHub 5 a few weeks ago, these tests only work if the user account used by the tests already has an active Keycloak SSO session through the JupyterHub UI.
We have investigated several possible causes, including configuring services and roles, and experimenting with different scope configurations, but so far we have not been able to reproduce the behavior we had with JupyterHub 4, where the tests worked without requiring an active UI session.
Has anyone encountered a similar issue after upgrading to JupyterHub 5, or can suggest what might have changed in the authentication flow?
Are you using API token to spawn a server when doing your automated tests? How do you authenticate against your FirecREST service during the automated tests?
Could you also share the logs, if possible in DEBUG mode of these failed automated tests?
The response it gives when trying something like submitting a job, or checking user info is {"status": 403, "message": "Missing or invalid credentials."}
This is the relevant part of the logs (I redacted the IPs)
[D 2026-06-02 13:50:57.013 JupyterHub base:411] Refreshing auth for user01
[I 2026-06-02 13:50:57.013 JupyterHub oauth2:1383] No auth_state found for user user01 refresh, need full authentication
[W 2026-06-02 13:50:57.013 JupyterHub base:415] User user01 has stale auth info. Login is required to refresh.
[W 2026-06-02 13:50:57.013 JupyterHub web:1964] 403 GET /hub/api/users/user01 (<CLIENT_IP>): Missing or invalid credentials.
[W 2026-06-02 13:50:57.014 JupyterHub log:192] 403 GET /hub/api/users/user01 (@<CLIENT_IP>) 283.51ms
[D 2026-06-02 13:50:57.077 JupyterHub base:411] Refreshing auth for user01
[I 2026-06-02 13:50:57.077 JupyterHub oauth2:1383] No auth_state found for user user01 refresh, need full authentication
[W 2026-06-02 13:50:57.077 JupyterHub base:415] User user01 has stale auth info. Login is required to refresh.
[W 2026-06-02 13:50:57.077 JupyterHub web:1964] 403 POST /hub/api/users/user01/server (<CLIENT_IP>): Missing or invalid credentials.
[W 2026-06-02 13:50:57.077 JupyterHub log:192] 403 POST /hub/api/users/user01/server (@<CLIENT_IP>) 2.19ms
[D 2026-06-02 13:50:57.122 JupyterHub base:411] Refreshing auth for user01
[I 2026-06-02 13:50:57.122 JupyterHub oauth2:1383] No auth_state found for user user01 refresh, need full authentication
[W 2026-06-02 13:50:57.122 JupyterHub base:415] User user01 has stale auth info. Login is required to refresh.
[W 2026-06-02 13:50:57.122 JupyterHub web:1964] 403 DELETE /hub/api/users/user01/server (<CLIENT_IP>): Missing or invalid credentials.
[W 2026-06-02 13:50:57.123 JupyterHub log:192] 403 DELETE /hub/api/users/user01/server (@<CLIENT_IP>) 2.39ms
Well, according to the logs what I see is that access token provided by Keycloak is expired and hence, JupyterHub is asking to restart the auth flow. Looking at the git blame, this part of the code has not changed much since last few years. So, are you sure something has not changed on your Keycloak side? Maybe lifetime of the access tokens have been shortened now?
The short answer is that if you don’t want this, you can set the config:
c.Authenticator.refresh_pre_spawn = False
This behavior would be related to the Authenticator.refresh_pre_spawn setting, When True, auth info is refreshed prior to spawn to ensure it is launching with valid credentials. If it cannot refresh auth (i.e. the refresh token is invalid or revoked), spawn will fail until the user logs in again. OAuthenticator implemented refresh functionality in 17.2, so that’s likely when the behavior changed in practice for you.
If this is a regular spawn via the UI, the experience will be clicking spawn goes through an OAuth redirect sequence (often invisible if the oauth provider remembers the user’s authorization and doesn’t prompt for reauthorization) before. But because spawn via the API cannot trigger a fresh login, the result is a 403 because fresh auth info is required, but cannot be retrieved.
The downside of this setting is that it’s actually impossible to spawn a user’s server if they haven’t logged in recently (i.e. admins cannot spawn user servers because they cannot issue credentials on their behalf).
We could have a setting that’s refresh if possible, but still allow launching with stale or missing auth info for requests that cannot be redirected to login (i.e. are not cookie-authenticated requests from the server-owning user). That could lead to confusing situations if the fresh credentials really are required for the user environment (sometimes they are, sometimes they aren’t).
I have c.Authenticator.refresh_pre_spawn set to the default, which is False. Since our spawner needs the access token from keycloak for it’s own authentication with the firecrest service, it could be that we are doing something that doesn’t play well with the current OAuthenticator. I will have a look with that in mind and report here.
OAuthenticator changes refresh_pre_spawn default to if enable_auth_state, so if you have auth state enabled, refresh_pre_spawn defaults to True.
However, I now see from your logs that this is also affecting GET /hub/api/users/user01, which is not related to refresh_pre_spawn, that’s the regular refresh_user behavior invoked after auth_refresh_age (default: 300s) for any authenticated request.
One thing that’s notable here is that auth_state is empty for this user, which suggests they have never logged in via oauth.
is this a regular user, or a synthetic user that never logs in with a browser, and thus never gets the oauth credentials? If it’s a synthetic user, you’ll need to skip the default refresh_user behavior for that user, or they won’t be able to do ~anything. 2i2c’s infrastructure does exactly this, which you can use as an example:
def refresh_user_hook(authenticator, user, auth_state):
if user.name == "deployment-service-check":
# if this is the user,
# refresh_user doesn't make sense
# consider it always fresh
return True
# for all other users, refresh as usual
return None
c.OAuthenticator.refresh_user_hook = refresh_user_hook
Right! The request to /hub/api/users/user01 is not related to the spawner.
Yes, that’s a synthetic user that has never logged into the UI. Our spawner supports “service accounts” that use the Client Credentials flow in Keycloak. In the JupyterHub configuration, we define a service-account role that does not require a username/password login to launch notebooks.
Nevertheless, the same behavior occurs when I use my own account while not logged into the UI.
Thanks a lot! I’ll try the hook and let you know how it goes.