we’re using JupyterHub to spawn JupyterLab containers via the dockerspawner.SystemUserSpawner class. For authentication we are using a custom class extension of the PAMAuthenticator class which does some additional creation of kerberos tickets if the user is authenticated with the system.
The Kerberos tickets expire after 10 hours and we need to force the users to login with JH in order to create a new kerberos ticket.
We have tried with various approaches to force a redirect to JH from the JupyterLab servers.
Both of which seems to generate a log entry in jupyterhub in the lines of:
[...] JupyterHub base:350] Invalid or expired cookie token
but which doesn’t force the user to log out.
If the user clicks on the “Hub Control Panel” entry in the file menu, they are redirected to a login page on the JH side, but if they use a deep link to their JupyterLab instance (e.g. https://{server_name}/user/{username}/lab? ), they are not prompted for their credentials.
How do I force a redirection to the jupyterhub login page?
The new refresh_user API in 1.0 is meant to specifically enable this kind of thing, so it would be a great use case to test with. This is meant to refresh auth if possible (e.g. refresh tokens), or return None if a new login should be required (e.g. auth revoked or refresh token expired).
Thanks for the answer and pointing me towards the new API feature. It looked promising, and the coroutine is also picked up as advertised. I have created the method refresh_user in my Authenticator class as such:
and in the JH log I see that it is being hit and that returning None triggers the following message from base.py:
JupyterHub base:294] User smhe has stale auth info. Login is required to refresh.
However, when I then in a new tab in the browser use the deeplink https://{servername}/user/{username}/lab?, it opens the JupyterLab without requiring re-authentication from JH. Am I doing something wrong with the implementation?
Essentially a bump, but I would really like to see a working example of how to implement refresh_user to force jupyterhub to redirect the browser to the login screen.
I have tried returning both False and None which, as described above, is picked up to some degree, but not enough to have the JH server do anything about it. Any ideas appreciated.
I have still found no solution to this problem. However, I can see that a jupyterhub-session-id cookie is created upon successful login. This can be deleted without affecting the authentication status of the logged in user. Can someone point me in the direction of the documentation describing how cookies relate to authentication status in jupyterhub?
There aren’t detailed docs of the cookies (there should be). I’m going to write down a sketch here, if anyone would like to make a PR adding a formal description to the docs.
Cookies used by jupyterhub:
jupyterhub-hub-login
This is the login token when visiting hub-served pages (main login, spawn, etc.)
encrypted (resetting cookie secret effectively revokes this cookie)
restricted to path: /hub/ so that only the Hub process receives it.
If this cookie is set, the user is logged in.
jupyterhub-user-username
This is the cookie used for authenticating with a single-user server
encrypted (resetting cookie secret effectively revokes this cookie)
set by the single-user server after OAuth with the Hub
set on /users/name so that only the user’s server receives it
effectively the same as jupyterhub-hub-login, but for the single-user server instead of the Hub
contains an OAuth access token, which is checked with the Hub to authenticate the browser. Each OAuth access token has a session id (see below)
To avoid hitting the Hub on every request, the authentication response is cached. To avoid a stale cache, the cache key is both the token and session id.
jupyterhub-session-id
set on / so all endpoints receive it, can clear it, etc.
this is a random string, meaningless in itself, and the only cookie shared by the Hub and single-user servers
its sole purpose is to to coordinate logout of the multiple oauth cookies.
jupyterhub-user-name-oauth-state
a short-lived cookie, used solely to store and validate oauth state. Only set while oauth between the notebook and the Hub is processing
Logging in
The login process, starting with no cookies and accessing an running server at /user/name (the most complicated case)
user visits /user/name (served by notebook)
jupyterhub-user-name is not set, login process begins. That’s OAuth with the Hub, redirect to /hub/api/oauth2/authorize
/hub/api/oauth2/authorize (served by hub) is an authenticated page. jupyterhub-hub-login is not set, redirect to /hub/login to begin login process
login process may be a form or external oauth, etc.
on successful login, jupyterhub-hub-login and jupyterhub-session-id are set, redirect back to /hub/api/oauth2/authorize
authenticated access to /hub/api/oauth2/authorize completes OAuth process with an oauth code and redirects back to /users/name/oauth_callback (will start here if already authenticated with the Hub at the time of requesting access to /user/name)
/users/name/oauth_callback (served by notebook) completes oauth with the Hub, retrieving an OAuth access token and storing it in jupyterhub-user-name
redirects back to /user/name, checks jupyterhub-user-name cookie
finds token
verifies token with the Hub via /hub/api/authorizations/token
caches response using jupyterhub-session-id and token together as key
allows access based on who is authenticated
Logging out
Clicking a logout button anywhere should:
redirect to /hub/logout (served by Hub)
which calls clear_login_cookie, which clears jupyterhub-hub-login
revokes all OAuth access tokens associated with jupyterhub-session-id
clears jupyterhub-session-id
After this, visiting /user/namewill have jupyterhub-user-name set. This cookie is not cleared by the Hub. However, since jupyterhub-session-idis cleared, the cached authentication response will not be used. The Hub will be consulted again for the validity of the token in jupyterhub-user-name, which was revoked during logout due to the session id, so authentication will fail and the cookie will be cleared and login will be prompted to start again.
We are using jupyterhub + jupyterhub configurable-http-proxy + jupyterlab.
The problem with this is that, once you are in the notebook, the hub never sees any requests from the user’s browser. The only requests are to the notebook server (kernelspecs, sessions, terminals, etc…) and other communication is done over websockets.
So I’m not sure quite sure how a redirect could happen?
force=True is optional and i am not sure it makes any difference.
But yeah, that bit works. (although it doesn’t redirect the page in the browser, but it simply stops the server and throws a Service Unavailable/Server Error default message. Which is ok for my case.).
So far, tested on PAMAuthentication-default installation. I will need to try it with OAuth too.
Hi, I am using Auth0, added your functionality getting this error
“Page isn’t redirected properly - problem might be caused by disabling or refusing to accept cookies”
from oauthenticator.auth0 import Auth0OAuthenticator, LocalAuth0OAuthenticator
Where did you put that snippet? I have it on /usr/local/lib/python3.6/dist-packages/jupyterhub/auth.py which is the basic authentication class which also contains this method (as empty) by itself . (the method refresh_user() i mean.). Then any other Authenticator class will use it. This error you get, also, may not be related to this snippet. I don’t think it is. Maybe something in the configuration file is not correct. (haven’t used LocalAuth0 to be honest). Also, i would try to open the developer console (F12) and select all cookies of localhost and Delete them all .
I have implemented Pingfederate integration by extending class PingfedOAuthenticator(OAuthenticator):…
I have implemented the refresh_user in the same class like below: @gen.coroutine
def refresh_user(self, user, handler=None):
print(‘I am in refresh user’)
return False
I tried to return None.
It does work on staling the session and forcing user to login .
However this introduced the issue that even watcher is getting a 403 and the Culling feature also not working.
I will try to experiment with the approach that this JupyterLab extension takes. Since we’re using an external OAuth IdP, we’ll probably try to check with that (as also alluded to in this post) and do the redirect based on the information we get back, rather than a fixed timeout
@minrk We are also trying the same use case i.e. trying to force re-login a user based on their token expiration. I tried returning False from inside the refresh_user function. But it doesn’t redirect back to the login form inside the jupyter lab container. If we hit the jupyterhub(/hub/login) page again, it asks for login but within the container/ user server, it will not ask for creds within the user server. Don’t know if this is relevant, but on returning False from refresh_user, The user server logs have the following error:
[E 2021-10-06 22:26:59.510 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
await client.fetch(req)
tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
If you can guide as to how to achieve forced re-login without killing the user server would be really helpful
Hey - I’m setting up JupyterHub for our company and it’s a real security problem if people are logged in forever. In particular, this is a separate problem from culling. A user’s notebook can, and should, keep running in the background. But the user should just have to login again to do anything with it.
This extension works for Lab and runs client side code to detect inactivity, but that’s not the core of the problem I see (besides the fact that I need something that works with classic notebooks and lab).
As far as I see, Hub is in charge of user management, that’s one of the main purposes it serves. It has a proxy which also forwards requests to the individual notebook servers. Should that proxy not also handle authentication? Why rely on the separate auth cookies for the user server? I understand that notebook servers have their own cookie management for login, and always have had for traditional use-cases, but working within a Hub is a distinct scenario. This lack of clear ownership of the authentication seems to be the core of the confusion/bugs described in this issue too.
Is there any reason why the Hub and the proxy could not, with some work, handle all authentication including session management and preventing forwarding any requests to a user servers if the user’s session has timed out?
I’d like to jump in on this as well, if you were able to clear the cookies were you redirected to the login page? If i recall correctly the default logout handler will return the login_url if the cookies don’t exist.
We chose OAuth as the mechanism to negotiate auth between the hub and other services authenticated with the hub. We also chose to limit the logic in the proxy, in part to allow alternate implementations of the proxy (e.g. kubernetes ingress, traefik), and in part to limit the amount of JupyterHub-specific logic that needed to be developed in a separate, node.js package (configurable-http-proxy). If we rely on auth implemented in the proxy, we would also need an additional, separate mechanism to authenticate requests to verify that they come directly from the proxy (or hub), e.g. SSL client certificates. That’s not an unreasonable thing to do, it’s just not the choice we made.
To be clear, the Hub is responsible for all authentication. The cookie set by the single-user server is only the OAuth token issued by the hub. Requests are only allowed after validating oauth tokens via an API request to the hub. Single-user servers have no independent auth, they are only oauth clients for the Hub.
So in normal circumstances, auth looks like this, with 3 tokens:
jupyterhub cookie set on /hub/
jupyterhub-session-id set on / (the only shared cookie used by everything in jupyterhub)
oauth token for single-user server, associated with session id and stored on /user/name/. This token is:
issued by the hub
associated with the session id
stored in a cookie, encrypted by the single-user server
When a request is made to a single-user server, it gets the oauth token out of the single-user cookie and session id. It then checks with the Hub to identify the user associated with the token before allowing the request.
When a user requests logout from the single-user server, it makes 2 requests:
/user/name/logout which clears the single-user cookie, which redirects to
/hub/logout which clears the session id and hub cookie and revokes all oauth tokens
If you just visit /hub/logout that should be sufficient, because the token stored in the singleuser cookie has been revoked based on its association with the session id, so while there may still be a token in the single-user cookie, when the single-user server asks the Hub who the token belongs to, it won’t resolve to a user and the request will be rejected.
However a bug in the singleuser part of JupyterHub < 1.5 when used with JupyterLab could result in persisting a different token in the cookie, the one associated with the server itself ($JUPYTERHUB_API_TOKEN), which is not associated with the client’s session-id and thus does not get revoked while the server is running. This is what could cause ‘sticky’ single-user logins, if e.g. a second tab is open while you try to logout.
The session id was added to enable this token revocation, however cookie-authenticated requests without a session id are still allowed (mainly for backward-compatibility as the session id was added in a minor release). We could make things a bit more robust in 2.0, if we made the change in the single-user token check:
require session-id to be set for cookie auth
require session-id to match the one associated with the token
Both of these would have prevented the cookie set prior to 1.5 from being accepted.
I think we should make this change for 2.0, though we are already in the release candidate phase.