How to force re-login for users

JupyterHub
#1

Hi,

we’re using JupyterHub to spawn JupyterLab containers via the dockerspawner.SystemUserSpawner class. For authentication we are using a custom class extension of the PAMAuthenticator class which does some additional creation of kerberos tickets if the user is authenticated with the system.

The Kerberos tickets expire after 10 hours and we need to force the users to login with JH in order to create a new kerberos ticket.

We have tried with various approaches to force a redirect to JH from the JupyterLab servers.

e.g. in the jupyterhub_config.py:

JupyterHub.cookie_max_age_days = 0.4125

as suggested by https://github.com/jupyterhub/jupyterhub/issues/673

and

c.JupyterHub.tornado_settings.cookie_options = dict(expires_days=0.41.25, max_age_days=0.4125)

as suggested by https://github.com/jupyterhub/jupyterhub/issues/2277

Both of which seems to generate a log entry in jupyterhub in the lines of:

[...] JupyterHub base:350] Invalid or expired cookie token

but which doesn’t force the user to log out.

If the user clicks on the “Hub Control Panel” entry in the file menu, they are redirected to a login page on the JH side, but if they use a deep link to their JupyterLab instance (e.g. https://{server_name}/user/{username}/lab? ), they are not prompted for their credentials.

How do I force a redirection to the jupyterhub login page?

Update: this seems tangentially related to Culling JupyterLab and OAuth cookies causing errors when user returns days later to use system

Invalidate JupyterHub user session after defined expiration time
User automatic logout on inactivity
Run a specific script
Question about user authentication with another webapp
How to force user to re-login after logout using JupyterHub and Auth0?
#2

The new refresh_user API in 1.0 is meant to specifically enable this kind of thing, so it would be a great use case to test with. This is meant to refresh auth if possible (e.g. refresh tokens), or return None if a new login should be required (e.g. auth revoked or refresh token expired).

1 Like
#3

Thanks for the answer and pointing me towards the new API feature. It looked promising, and the coroutine is also picked up as advertised. I have created the method refresh_user in my Authenticator class as such:

@gen.coroutine
def refresh_user(self, user, handler=None):
    return None

and in the JH log I see that it is being hit and that returning None triggers the following message from base.py:

JupyterHub base:294] User smhe has stale auth info. Login is required to refresh.

However, when I then in a new tab in the browser use the deeplink https://{servername}/user/{username}/lab?, it opens the JupyterLab without requiring re-authentication from JH. Am I doing something wrong with the implementation?

#4

Essentially a bump, but I would really like to see a working example of how to implement refresh_user to force jupyterhub to redirect the browser to the login screen.

I have tried returning both False and None which, as described above, is picked up to some degree, but not enough to have the JH server do anything about it. Any ideas appreciated.

#5

I have still found no solution to this problem. However, I can see that a jupyterhub-session-id cookie is created upon successful login. This can be deleted without affecting the authentication status of the logged in user. Can someone point me in the direction of the documentation describing how cookies relate to authentication status in jupyterhub?

Thanks in advance

#6

There aren’t detailed docs of the cookies (there should be). I’m going to write down a sketch here, if anyone would like to make a PR adding a formal description to the docs.

Cookies used by jupyterhub:

  • jupyterhub-hub-login
    • This is the login token when visiting hub-served pages (main login, spawn, etc.)
    • encrypted (resetting cookie secret effectively revokes this cookie)
    • restricted to path: /hub/ so that only the Hub process receives it.
    • If this cookie is set, the user is logged in.
  • jupyterhub-user-username
    • This is the cookie used for authenticating with a single-user server
    • encrypted (resetting cookie secret effectively revokes this cookie)
    • set by the single-user server after OAuth with the Hub
    • set on /users/name so that only the user’s server receives it
    • effectively the same as jupyterhub-hub-login, but for the single-user server instead of the Hub
    • contains an OAuth access token, which is checked with the Hub to authenticate the browser. Each OAuth access token has a session id (see below)
    • To avoid hitting the Hub on every request, the authentication response is cached. To avoid a stale cache, the cache key is both the token and session id.
  • jupyterhub-session-id
    • set on / so all endpoints receive it, can clear it, etc.
    • this is a random string, meaningless in itself, and the only cookie shared by the Hub and single-user servers
    • its sole purpose is to to coordinate logout of the multiple oauth cookies.
  • jupyterhub-user-name-oauth-state
    • a short-lived cookie, used solely to store and validate oauth state. Only set while oauth between the notebook and the Hub is processing

Logging in

The login process, starting with no cookies and accessing an running server at /user/name (the most complicated case)

  • user visits /user/name (served by notebook)
  • jupyterhub-user-name is not set, login process begins. That’s OAuth with the Hub, redirect to /hub/api/oauth2/authorize
  • /hub/api/oauth2/authorize (served by hub) is an authenticated page. jupyterhub-hub-login is not set, redirect to /hub/login to begin login process
  • login process may be a form or external oauth, etc.
  • on successful login, jupyterhub-hub-login and jupyterhub-session-id are set, redirect back to /hub/api/oauth2/authorize
  • authenticated access to /hub/api/oauth2/authorize completes OAuth process with an oauth code and redirects back to /users/name/oauth_callback (will start here if already authenticated with the Hub at the time of requesting access to /user/name)
  • /users/name/oauth_callback (served by notebook) completes oauth with the Hub, retrieving an OAuth access token and storing it in jupyterhub-user-name
  • redirects back to /user/name, checks jupyterhub-user-name cookie
    • finds token
    • verifies token with the Hub via /hub/api/authorizations/token
    • caches response using jupyterhub-session-id and token together as key
    • allows access based on who is authenticated

Logging out

Clicking a logout button anywhere should:

  • redirect to /hub/logout (served by Hub)
  • which calls clear_login_cookie, which clears jupyterhub-hub-login
  • revokes all OAuth access tokens associated with jupyterhub-session-id
  • clears jupyterhub-session-id

After this, visiting /user/name will have jupyterhub-user-name set. This cookie is not cleared by the Hub. However, since jupyterhub-session-id is cleared, the cached authentication response will not be used. The Hub will be consulted again for the validity of the token in jupyterhub-user-name, which was revoked during logout due to the session id, so authentication will fail and the cookie will be cleared and login will be prompted to start again.

8 Likes
Cookie secret and state - how does it ensure secure login?
JupyterHub log off a previous session on another browser after a new browser log in
#7

I have marked this post. If I have time, I can have a try.

#8

We are using jupyterhub + jupyterhub configurable-http-proxy + jupyterlab.

The problem with this is that, once you are in the notebook, the hub never sees any requests from the user’s browser. The only requests are to the notebook server (kernelspecs, sessions, terminals, etc…) and other communication is done over websockets.

So I’m not sure quite sure how a redirect could happen?

2 Likes
#9

Any update on the original issue here? I have seen the same behavior using the off the shelf generic Oauth Authenticator (with Auth0)

1 Like
#11

I wanted to achieve exactly the same; I think I did it.

In auth.py:

async def refresh_user(self, user, handler, force=True):
    await handler.stop_single_user(user, user.spawner.name)
    handler.clear_cookie("jupyterhub-hub-login")
    handler.clear_cookie("jupyterhub-session-id")
    handler.redirect('/hub/logout')

force=True is optional and i am not sure it makes any difference.
But yeah, that bit works. (although it doesn’t redirect the page in the browser, but it simply stops the server and throws a Service Unavailable/Server Error default message. Which is ok for my case.).
So far, tested on PAMAuthentication-default installation. I will need to try it with OAuth too.

#12

Hi, I am using Auth0, added your functionality getting this error
“Page isn’t redirected properly - problem might be caused by disabling or refusing to accept cookies”

from oauthenticator.auth0 import Auth0OAuthenticator, LocalAuth0OAuthenticator

c.JupyterHub.authenticator_class = LocalAuth0OAuthenticator

c.LocalAuth0OAuthenticator.auth0_subdomain = ‘domain’

c.LocalAuthenticator.add_user_cmd = [‘adduser’, ‘-q’, ‘–gecos’, ‘’, ‘–disabled-password’, ‘–force-badname’]

c.LocalAuthenticator.create_system_users = True

c.Authenticator.delete_invalid_users = True

c.Auth0OAuthenticator.client_id = ‘id’

c.Auth0OAuthenticator.client_secret = ‘sec’

c.Auth0OAuthenticator.oauth_callback_url = ‘http://localhost:8000/hub/oauth_callback’

c.Auth0OAuthenticator.scope = [‘openid’, ‘email’]

c.JupyterHub.cookie_max_age_days = 0.4125

c.Authenticator.refresh_pre_spawn = True

c.Authenticator.auth_refresh_age = 300

c.Authenticator.admin_users = {‘myemail’}

Any ideas what I could do to fix? My problem is that it keeps auto logging me in - I cannot test the actual signup/login page

#13

Where did you put that snippet? I have it on /usr/local/lib/python3.6/dist-packages/jupyterhub/auth.py which is the basic authentication class which also contains this method (as empty) by itself . (the method refresh_user() i mean.). Then any other Authenticator class will use it. This error you get, also, may not be related to this snippet. I don’t think it is. Maybe something in the configuration file is not correct. (haven’t used LocalAuth0 to be honest). Also, i would try to open the developer console (F12) and select all cookies of localhost and Delete them all .

#14

Thank you for posting your solution.
However, I need a solution without shutting down the user server.

Does anyone has an idea, how the redirect to ‘/hub/logout/’ can be enabled?

#15

I have implemented Pingfederate integration by extending class PingfedOAuthenticator(OAuthenticator):…

I have implemented the refresh_user in the same class like below:
@gen.coroutine
def refresh_user(self, user, handler=None):
print(‘I am in refresh user’)
return False

I tried to return None.
It does work on staling the session and forcing user to login .
However this introduced the issue that even watcher is getting a 403 and the Culling feature also not working.

Can you please suggest/direct.

#16

I will try to experiment with the approach that this JupyterLab extension takes. Since we’re using an external OAuth IdP, we’ll probably try to check with that (as also alluded to in this post) and do the redirect based on the information we get back, rather than a fixed timeout

#17

Can I use this extension with no JupyterLab installed? Currently I am using only JupyterHub and ContainDS.

#18

@minrk We are also trying the same use case i.e. trying to force re-login a user based on their token expiration. I tried returning False from inside the refresh_user function. But it doesn’t redirect back to the login form inside the jupyter lab container. If we hit the jupyterhub(/hub/login) page again, it asks for login but within the container/ user server, it will not ask for creds within the user server. Don’t know if this is relevant, but on returning False from refresh_user, The user server logs have the following error:

      [E 2021-10-06 22:26:59.510 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
        await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden

If you can guide as to how to achieve forced re-login without killing the user server would be really helpful

1 Like
#19

Hey - I’m setting up JupyterHub for our company and it’s a real security problem if people are logged in forever. In particular, this is a separate problem from culling. A user’s notebook can, and should, keep running in the background. But the user should just have to login again to do anything with it.

This extension works for Lab and runs client side code to detect inactivity, but that’s not the core of the problem I see (besides the fact that I need something that works with classic notebooks and lab).

As far as I see, Hub is in charge of user management, that’s one of the main purposes it serves. It has a proxy which also forwards requests to the individual notebook servers. Should that proxy not also handle authentication? Why rely on the separate auth cookies for the user server? I understand that notebook servers have their own cookie management for login, and always have had for traditional use-cases, but working within a Hub is a distinct scenario. This lack of clear ownership of the authentication seems to be the core of the confusion/bugs described in this issue too.

Is there any reason why the Hub and the proxy could not, with some work, handle all authentication including session management and preventing forwarding any requests to a user servers if the user’s session has timed out?

1 Like
#20

I’d like to jump in on this as well, if you were able to clear the cookies were you redirected to the login page? If i recall correctly the default logout handler will return the login_url if the cookies don’t exist.

Thanks!

#21

We chose OAuth as the mechanism to negotiate auth between the hub and other services authenticated with the hub. We also chose to limit the logic in the proxy, in part to allow alternate implementations of the proxy (e.g. kubernetes ingress, traefik), and in part to limit the amount of JupyterHub-specific logic that needed to be developed in a separate, node.js package (configurable-http-proxy). If we rely on auth implemented in the proxy, we would also need an additional, separate mechanism to authenticate requests to verify that they come directly from the proxy (or hub), e.g. SSL client certificates. That’s not an unreasonable thing to do, it’s just not the choice we made.

To be clear, the Hub is responsible for all authentication. The cookie set by the single-user server is only the OAuth token issued by the hub. Requests are only allowed after validating oauth tokens via an API request to the hub. Single-user servers have no independent auth, they are only oauth clients for the Hub.

So in normal circumstances, auth looks like this, with 3 tokens:

  • jupyterhub cookie set on /hub/
  • jupyterhub-session-id set on / (the only shared cookie used by everything in jupyterhub)
  • oauth token for single-user server, associated with session id and stored on /user/name/. This token is:
    • issued by the hub
    • associated with the session id
    • stored in a cookie, encrypted by the single-user server

When a request is made to a single-user server, it gets the oauth token out of the single-user cookie and session id. It then checks with the Hub to identify the user associated with the token before allowing the request.

When a user requests logout from the single-user server, it makes 2 requests:

  1. /user/name/logout which clears the single-user cookie, which redirects to
  2. /hub/logout which clears the session id and hub cookie and revokes all oauth tokens

If you just visit /hub/logout that should be sufficient, because the token stored in the singleuser cookie has been revoked based on its association with the session id, so while there may still be a token in the single-user cookie, when the single-user server asks the Hub who the token belongs to, it won’t resolve to a user and the request will be rejected.

However a bug in the singleuser part of JupyterHub < 1.5 when used with JupyterLab could result in persisting a different token in the cookie, the one associated with the server itself ($JUPYTERHUB_API_TOKEN), which is not associated with the client’s session-id and thus does not get revoked while the server is running. This is what could cause ‘sticky’ single-user logins, if e.g. a second tab is open while you try to logout.

The session id was added to enable this token revocation, however cookie-authenticated requests without a session id are still allowed (mainly for backward-compatibility as the session id was added in a minor release). We could make things a bit more robust in 2.0, if we made the change in the single-user token check:

  1. require session-id to be set for cookie auth
  2. require session-id to match the one associated with the token

Both of these would have prevented the cookie set prior to 1.5 from being accepted.

I think we should make this change for 2.0, though we are already in the release candidate phase.

4 Likes