Launching multiple notebooks in jupyterhub with different jwt tokens loads the first spawned notebook

Bug description :

  • I am using jupyterhub with custom authentication where a userId is extracted from jwt , and a notebook is spawned on that userId . But when I launch multiple notebooks with different userId from the same browser , it loads the first notebook .

  • The new userId has been passed correctly in the logs , but somewhere it changes and the previous values are loaded .

  • This works fine when I launch each unique notebook in different browsers

  • In the logs , I am getting

[I 2023-10-03 18:17:53.016 JupyterHub base:837] User logged in: b372 ->2nd persons userId

  • But in the next line , it is taking first spawned notebook’s userId
    [I 2023-10-03 18:17:53.020 JupyterHub log:191] 302 GET /hub/login?token=[secret] → /hub/spawn?token=[secret] (1563) ->Previous userId
    [I 2023-10-03 18:17:53.615 JupyterHub pages:168] Server 1563 is already active

I am using Jupyterhub 4.0.2 and Jupyterlab 4.0.6

2 Likes

You can’t have multiple users logged in to the same JupyterHub on the same browser since the session cookies identify the user.

If you’re using something that bypasses the standard authentication you’ll probably need to share your custom authenticator, or create an equivalent example that demonstrates the problem, to make it easier for people to help you.

1 Like

Thank you for the reply !

  • Ok, it is fine if I can’t have multiple users on the same browser . But , why am I getting someone else’s notebook when I launch my notebook , even when we are on different devices .

  • I can confirm from the logs that it is taking my userId from jwt token when I launch hub (b372) , but somewhere in between it is taking a different userId .I am not sure if this is due to some kind of caching during oauth or somewhere else.

  • As you mentioned , also attaching my custom authenticator for better understanding . Kindly tell if you need any other details .

Edit :
Leaving my custom authenticator here , as I am not able to edit my post for some reason ,

class CustomLoginHandler(BaseHandler):
    """
    Authenticate users via the CAS protocol.
    """

    async def get(self):
        app_log = logging.getLogger("tornado.application")
        token = self.get_argument("token", None)
        has_service_token = token is not None
        app_log.debug("Has service ticket? {0}".format(has_service_token))
        # Redirect to get ticket if not presenting one
        if not has_service_token:
            url = self.authenticator.c_login_url
            app_log.debug("Redirecting to Custom to get service token: {0}".format(url))
            self.redirect(url)
            return
        user = await self.login_user()
        next_url = self.get_next_url(user)
        self.redirect(next_url)


    def make_service_url(self):
        """
        Make the service URL CAS will use to redirect the browser back to this service.
        """
        cas_service_url = self.authenticator.c_service_url
        if cas_service_url is None:
            cas_service_url = self.request.protocol + "://" + self.request.host + self.request.uri
        return cas_service_url

    """async def validate_jwt_token(self, token):
        
        #Validate a CAS service ticket.
        #Returns (is_valid, user, attribs).
        #is_valid` - boolean
        #`attribs` - set of attribute-value tuples.
        
        app_log = logging.getLogger("tornado.application")
        http_client = AsyncHTTPClient()
        qs_dict = dict(token=token)
        qs = urllib.parse.urlencode(qs_dict)
        validate_url = self.authenticator.c_service_validate_url + "?" + qs
        print("Validating Ticket:",validate_url)
        response = None
        app_log.debug("Validate URL: {0}".format(validate_url))
        try:
            response = await http_client.fetch(
                validate_url,
                method="GET")
            jsondata = json.loads(response.body)
            if jsondata["status"] == "success":
                accessList = jsondata["data"]['accessList']
                name = jsondata["data"]["displayName"]
                uid = jsondata["data"]["userId"]
                notebookConfig = jsondata["Config"]
                return (True, accessList, name, uid, notebookConfig)
            app_log.debug("Response was successful: {0}".format(response))
        except Exception:
            app_log.debug("Response was unsuccessful: {0}".format(response))
            return (False, None, None, None, None)

        return (False, None, None, None, None)
"""

class CustomAuthenticator(Authenticator):
    from tornado import gen
    """
    Validate a CAS service ticket and optionally check for the presence of an
    authorization attribute.
    """
    c_login_url = Unicode(
        config=True,
        help="""The CAS URL to redirect unauthenticated users to.""")

    c_logout_url = Unicode(
        config=True,
        help="""The CAS URL for logging out an authenticated user.""")

    c_client_ca_certs = Unicode(
        allow_none=True,
        default_value=None,
        config=True,
        help="""Path to CA certificates the CAS client will trust when validating a service ticket.""")

    c_service_validate_url = Unicode(
        config=True,
        help="""The CAS endpoint for validating service tickets.""")

    c_required_attribs = Set(
            help="A set of attribute name and value tuples a user must have to be allowed access."
        ).tag(config=True)

    def get_handlers(self, app):
        return [
            (r'/login', CustomLoginHandler),
            (r'/logout', CustomLogoutHandler),
        ]
    async def validate_jwt_token(self, token):
        """
        Validate a CAS service ticket.
        Returns (is_valid, user, attribs).
        `is_valid` - boolean
        `attribs` - set of attribute-value tuples.
        """
        app_log = logging.getLogger("tornado.application")
        http_client = AsyncHTTPClient()
        qs_dict = dict(token=token)
        qs = urllib.parse.urlencode(qs_dict)
        print(f"QueryParams --> {qs}")
        validate_url = self.c_service_validate_url + "?" + qs
        print("Validating Ticket:", validate_url)
        response = None
        jsondata = None
        app_log.debug("Validate URL: {0}".format(validate_url))
        try:
            response = await http_client.fetch(
                validate_url,
                method="GET")
            jsondata = json.loads(response.body)
            print(jsondata)
            if jsondata["status"] == "success":
                accessList = jsondata["data"]['accessList']
                name = jsondata["data"]["displayName"]
                uid = jsondata["data"]["userId"]
                orgid = jsondata["data"]["orgId"]
                email = jsondata["data"]["email"]
                notebookConfig = jsondata["config"]
                return (True, accessList, name, uid, orgid, email, notebookConfig)
            print("Response was successful: {0}".format(response))
        except Exception:
            print("Response was unsuccessful: {0}".format(response))
            return (False, None, None,None,None)

        return (False, None, None, None, None)

    @gen.coroutine
    def pre_spawn_start(self, user, spawner):
        auth_state = yield user.get_auth_state()
        print("User Details: {}".format(user))
        print("User key type: {}".format(type(user)))
        print("Auth Check Pre Spawn", auth_state)
        if not auth_state:
            return
        spawner.environment['CAUTH_TOKEN'] = auth_state["config"][0]["token"]
        print(auth_state["config"])
        spawner.profile_list = auth_state["config"]
        spawner.http_timeout = 300

    async def authenticate(self, handler,data=None):
        token = handler.get_query_argument("token")
        result = await self.validate_jwt_token(token=token, notebook_id=notebook_id)
        print(f"Result ==> {result}")
        is_valid, accessList, name, id, orgId, email, config = result
        if not is_valid:
            return
        user = {
            'name': id,
            'auth_state': {'token': token, "config": config, 'displayName': name, 'orgId': orgId, 'email': email}
        }
        print("User Details to Spawner: {}".format(user))
        return user

Could you share your hub logs covering the events when first user logs in, start server, logout and second user logs in, start server. Somethings you can check:

  • Are user cookies for first user and second user upon logging in are properly set?
  • You havent shared your custom logout handler. Are you clearing the user cookie on logout?

Yes , on logout the user cookies will be cleared . But no one actually logs out when they are done :cry:

For the first person :

  • 302 GET /login?token=[secret] → /hub/login?token=[secret] (@100.123.167.64)

  • Refreshing auth for 1563

  • Assigning default role to User 1563

  • Setting cookie jupyterhub-session-id: {‘httponly’: True, ‘secure’: True, ‘path’: ‘/’}

  • User logged in: 1563

  • Creating <class ‘kubespawner.spawner.KubeSpawner’> for 1563

  • 302 GET /hub/login?token=[secret] → /hub/spawn?token=[secret] (1563@100.123.167.64

Here 1563 is the userId taken from the jwt token .

For the 2nd user :

  • 302 GET /login?token=[secret] → /hub/login?token=[secret]

  • User logged in: a5fe

  • 302 GET /hub/login?token=[secret] → /hub/spawn?token=[secret] (a5fe@100.123.167.64) → UserId is changed here

  • User is running: 9c5da52d-8cb2-4801-9257-e46a25041563 (Not shown always)

I can notice that for the first user cookies are set properly based on the logs , whereas it is not the case for the second users . Is there any way I can set the cookies manually for each user every time they login or could there any bug in the code ?

Really appreciate your help , thank you

Could you replace print in your custom authenticator with self.log.info and share entire logs without any filtering? I mean, you can redact any sensitive info, but dont remove any log lines. If you filter logs, its hard to see what is happening with your custom authenticator.

self.login_user() in your custom login handler will set the user cookie. So, you dont have to do anything manually.

Well, users will access hub from their browsers and not a single browser. So, not logging out is not an issue. Moreover you can set a maximum cookie life using c.JupyterHub.cookie_max_age_days to automatically invalidate cookie after a certain time.

As @manics stated, JupyterHub recognises user from browser cookie. If the browser cookie is not being set for your second user, I assume the first user’s cookie is never invalidated. I guess more correct way of simulating multiple users from same browser is to use incognito sessions.

2 Likes

@mahendrapaipuri Sure, I’m attaching the entire logs below,

I am logging into jupyterhub with the userId b372, on a new browser

  • [I 2023-10-09 19:47:07.714 JupyterHub log:191] 302 GET /login?token=[secret] → /hub/login?token=[secret] (@100.110.124.192) 0.88ms

  • [D 2023-10-09 19:47:08.384 JupyterHub base:344] Refreshing auth for b514 -->This is not my current useId

  • [2023-10-09 19:47:08,692] [INFO] [get] [53] Token → xxxxxx

  • [2023-10-09 19:47:08,692] [INFO] [get] [56] service Token ==> True

  • [D 2023-10-09 19:47:08.692 JupyterHub auth:55] Has service ticket? True

  • [2023-10-09 19:47:08,694] [INFO] [validate_jwt_token] [181] QueryParams → token=xxxxxx

  • [2023-10-09 19:47:08,694] [INFO] [validate_jwt_token] [183] Validating Ticket:, Will be validating jwt here

  • [D 2023-10-09 19:47:08.697 JupyterHub log:191] 200 GET /hub/health (@172.20.42.105) 2.26ms

  • [2023-10-09 19:47:08,800] [INFO] [validate_jwt_token] [192] {‘status’: ‘success’}

  • [2023-10-09 19:47:08,800] [INFO] [authenticate] [235] Result ==> (Details of the notebook to be launched )

  • [2023-10-09 19:47:08,800] [INFO] [authenticate] [256] User Details to Spawner: {Required env values}

  • [D 2023-10-09 19:47:08.860 JupyterHub roles:281] Assigning default role to User b372

  • [D 2023-10-09 19:47:09.101 JupyterHub base:587] Setting cookie jupyterhub-session-id: {‘httponly’: True, ‘secure’: True, ‘path’: ‘/’}

  • [I 2023-10-09 19:47:09.394 JupyterHub base:837] User logged in: b372

  • [D 2023-10-09 19:47:09.394 JupyterHub user:431] Creating <class ‘kubespawner.spawner.KubeSpawner’> for b372:
    [2023-10-09 19:47:09,394] [INFO] [get] [86] Login User ==> <User(b372 0/1 running)>

  • [2023-10-09 19:47:09,397] [INFO] [get] [89] Next URL ==> /hub/spawn?token=[token for the user b372]

  • [I 2023-10-09 19:47:09.398 JupyterHub log:191] 302 GET /hub/login?token=[secret] → /hub/spawn?token=[secret] (b514@100.110.124.192) 1427.08ms

  • [D 2023-10-09 19:47:09.715 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers

  • [D 2023-10-09 19:47:09.715 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers

  • [D 2023-10-09 19:47:09.716 JupyterHub pages:191] Triggering spawn with supplied query arguments for b514

  • [D 2023-10-09 19:47:09.716 JupyterHub base:961] Initiating spawn for b514

  • [D 2023-10-09 19:47:09.716 JupyterHub base:965] 1/64 concurrent spawns

  • [D 2023-10-09 19:47:09.716 JupyterHub base:970] 2 active servers

  • [I 2023-10-09 19:47:11.018 JupyterHub provider:659] Creating oauth client jupyterhub-user-b514

  • [I 2023-10-09 19:47:11.555 JupyterHub log:191] 302 GET /hub/spawn?token=[secret] → /hub/spawn-pending/b514?token=[secret] (b514@100.110.124.192) 1901.01ms

  • [D 2023-10-09 19:47:11.556 JupyterHub log:191] 200 GET /hub/health (@172.20.42.105) 2.42ms

  • [2023-10-09 19:47:11,556] [INFO] [pre_spawn_start] [213] User Details: <User(b514 1/1 running)>

  • [2023-10-09 19:47:11,556] [INFO] [pre_spawn_start] [214] User key type: <class ‘jupyterhub.user.User’>

  • [2023-10-09 19:47:11,557] [INFO] [pre_spawn_start] [215] Auth Check Pre Spawn {Env values for spawner}

  • [D 2023-10-09 19:47:11.971 JupyterHub user:794] Calling Spawner.start for b514

  • [D 2023-10-09 19:47:11.972 JupyterHub spawner:3020] Applying KubeSpawner override for profile ‘CustomAuth Notebook’

  • [D 2023-10-09 19:47:11.972 JupyterHub spawner:3032] … overriding KubeSpawner value image=thickstatdevops/conversight:v0.5.6

  • [D 2023-10-09 19:47:11.972 JupyterHub spawner:3032] … overriding KubeSpawner value image_pull_policy=IfNotPresent

[D 2023-10-09 19:47:11.972 JupyterHub spawner:3032] … overriding KubeSpawner value image_pull_secrets=docker-secret

  • [D 2023-10-09 19:47:11.972 JupyterHub spawner:3032] … overriding KubeSpawner value cpu_limit=2

  • [D 2023-10-09 19:47:11.973 JupyterHub spawner:3032] … overriding KubeSpawner value mem_limit=8G

  • [D 2023-10-09 19:47:11.973 JupyterHub spawner:3032] … overriding KubeSpawner value mem_guarantee=4G

  • [D 2023-10-09 19:47:11.973 JupyterHub spawner:3032] … overriding KubeSpawner value pod_name_template=

  • [D 2023-10-09 19:47:11.973 JupyterHub spawner:3032] … overriding KubeSpawner value extra_container_config={‘envFrom’: [{‘configMapRef’: {‘name’: ‘env-config’}}]}

  • [D 2023-10-09 19:47:11.973 JupyterHub spawner:3032] … overriding KubeSpawner value node_selector={‘notebook’: ‘medium’}

  • [D 2023-10-09 19:47:12.267 JupyterHub scopes:877] Checking access to /hub/spawn-pending/b514 via scope servers

  • [D 2023-10-09 19:47:12.268 JupyterHub scopes:690] Argument-based access to /hub/spawn-pending/b514 via servers

  • [I 2023-10-09 19:47:12.269 JupyterHub spawner:2519] Attempting to create pvc claim-b514, with timeout 3

  • [D 2023-10-09 19:47:12.271 JupyterHub reflector:362] events watcher timeout

  • [D 2023-10-09 19:47:12.271 JupyterHub reflector:281] Connecting events watcher

  • [I 2023-10-09 19:47:12.274 JupyterHub pages:398] b514 is pending spawn

  • [I 2023-10-09 19:47:12.284 JupyterHub log:191] 200 GET /hub/spawn-pending/b514?token=[secret] (b514@100.110.124.192) 309.89ms

  • [I 2023-10-09 19:47:12.291 JupyterHub spawner:2535] PVC claim-b514 already exists, so did not create new pvc.

  • [I 2023-10-09 19:47:12.301 JupyterHub spawner:2479] Attempting to create pod jupyter-b514, with timeout 3

  • [D 2023-10-09 19:47:12.338 JupyterHub log:191] 200 GET /hub/health (@172.20.42.105) 0.58ms

  • [D 2023-10-09 19:47:12.716 JupyterHub scopes:877] Checking access to /hub/api/users/b514/server/progress via scope read:servers

  • [D 2023-10-09 19:47:12.716 JupyterHub scopes:690] Argument-based access to /hub/api/users/b514/server/progress via read:servers

  • [D 2023-10-09 19:47:12.776 JupyterHub spawner:2285] progress generator: jupyter-b514

While launching a notebook for the user b372, ends up displaying the notebook for b514 , even though I am on new browser

Thanks for the logs. I am afraid that I am not able to find any obvious errors here. In your initial post, you said it worked when you launch notebooks for different users from different browsers. So, if it is not working now, you should probably rollback any changes that you made.

I would suggest you to change jupyterhub_cookie_secret in your deployment which will invalidate all the existing cookies in different browsers you are testing. Then test your scenario by using different browsers for different users or different incognito sessions.

Also one more thing to try: could you set _jupyterhub_user attribute once the login_user() returns a valid user. So, get method in your CustomLoginHandler should have:

user = await self.login_user()
if user:
    # register current user for subsequent requests to user (e.g. logging the request)
    self._jupyterhub_user = user
    self.redirect(self.get_next_url(user))

I assume on your “new” browser there is already a cookie exists for user b514 and as you are not setting up _jupyterhub_user attribute, the hub is getting current user from cookie.

@mahendrapaipuri , the issue where I am getting others notebook sometimes , when I launch my own notebook has been present from the starting itself . One workaround I found was to logout that another user’s notebook and launch jupyterhub again . Then I’ll get my notebook .

But I have no idea , why it is bringing another user’s notebook even when I try from incognito tab .

One thing to note is , replicating this scenario is extremely inconsistent . 8/10 this works as expected , user logs in → notebook with their userId is launched . But like I said , sometimes I am getting another user’s notebook even when I launch from incognito .

Anyway , really appreciate you guys helping me out . Will try if setting this works → self._jupyterhub_user = user