Something triggering "Removing user from proxy" after exactly 5 minutes

Hi,

I’m having an issue.
My problems is that after I login after 5 minutes, something happens and the session crashes. If I click the restart button everything start up, then after 5 minutes it crashes again.

I modified jhub_remote_user_authenticator to use a Cookie from another website instead of the REMOTE_USER I also did this to check if the user is still logged in.

class RefreshRemoteUserLocalAuthenticator(RemoteUserLocalAuthenticator):
    async def refresh_user(self, user, handler=None):
        await handler.stop_single_user(user, user.spawner.name)
        handler.clear_cookie("jupyterhub-hub-login")
        handler.clear_cookie("jupyterhub-session-id")
        handler.redirect('/jupyterhub/hub/logout')
        return True

My config

c.JupyterHub.cookie_max_age_days = 1
c.JupyterHub.bind_url = 'http://:8000/jupyterhub/'
c.RefreshRemoteUserLocalAuthenticator.header_name = "Cookie"
c.RefreshRemoteUserLocalAuthenticator.auth_refresh_age = 10
c.JupyterHub.authenticator_class = 'jhub_remote_user_authenticator.remote_user_auth.RefreshRemoteUserLocalAuthenticator'

Jupyterhub is behind nginx

    location /jupyterhub/ {
        proxy_pass http://127.0.0.1:8000/jupyterhub/;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # websocket headers
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection upgrade;
        proxy_set_header X-Scheme $scheme;

        proxy_buffering off;
    }

Crash log

[I 2024-04-05 14:45:56.590 JupyterHub base:964] User admin took 2.819 seconds to start
------
[I 2024-04-05 14:51:00.315 JupyterHub proxy:310] Removing user admin from proxy (/jupyterhub/user/admin/)
14:51:00.316 [ConfigProxy] info: Removing route /jupyterhub/user/admin
14:51:00.316 [ConfigProxy] info: 204 DELETE /api/routes/jupyterhub/user/admin
[E 2024-04-05 14:51:00.317 SingleUserLabApp http1connection:67] Uncaught exception
    Traceback (most recent call last):
      File "/usr/local/lib64/python3.6/site-packages/tornado/http1connection.py", line 273, in _read_message
        delegate.finish()
      File "/usr/local/lib64/python3.6/site-packages/tornado/httpserver.py", line 387, in finish
        self.delegate.finish()
      File "/usr/local/lib64/python3.6/site-packages/tornado/routing.py", line 268, in finish
        self.delegate.finish()
      File "/usr/local/lib64/python3.6/site-packages/tornado/web.py", line 2290, in finish
        self.execute()
      File "/usr/local/lib64/python3.6/site-packages/tornado/web.py", line 2310, in execute
        self.application, self.request, **self.handler_kwargs
      File "/usr/local/lib64/python3.6/site-packages/tornado/web.py", line 227, in __init__
        self.clear()
      File "/usr/local/lib64/python3.6/site-packages/tornado/web.py", line 328, in clear
        self.set_default_headers()
      File "/usr/local/lib/python3.6/site-packages/jupyter_server/base/handlers.py", line 314, in set_default_headers
        elif self.token_authenticated and "Access-Control-Allow-Origin" not in self.settings.get(
      File "/usr/local/lib/python3.6/site-packages/jupyter_server/base/handlers.py", line 159, in token_authenticated
        return self.login_handler.is_token_authenticated(self)
      File "/usr/local/lib/python3.6/site-packages/jupyterhub/singleuser/mixins.py", line 105, in is_token_authenticated
        handler.get_current_user()
      File "/usr/local/lib/python3.6/site-packages/jupyter_server/base/handlers.py", line 630, in get_current_user
        self._user_cache = user = super(APIHandler, self).get_current_user()
      File "/usr/local/lib/python3.6/site-packages/jupyterhub/services/auth.py", line 1046, in get_current_user
        user_model = self.hub_auth.get_user(self)
      File "/usr/local/lib/python3.6/site-packages/jupyterhub/services/auth.py", line 573, in get_user
        user_model = self.user_for_token(token, session_id=session_id)
      File "/usr/local/lib/python3.6/site-packages/jupyterhub/services/auth.py", line 498, in user_for_token
        use_cache=use_cache,
      File "/usr/local/lib/python3.6/site-packages/jupyterhub/services/auth.py", line 390, in _check_hub_authorization
        allow_403=True,
      File "/usr/local/lib/python3.6/site-packages/jupyterhub/services/auth.py", line 411, in _api_request
        r = requests.request(method, url, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 61, in request
        return session.request(method=method, url=url, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 529, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 645, in send
        r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 450, in send
        timeout=timeout
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 467, in _make_request
        six.raise_from(e, None)
      File "<string>", line 3, in raise_from
      File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 462, in _make_request
        httplib_response = conn.getresponse()
      File "/usr/lib64/python3.6/http/client.py", line 1365, in getresponse
        response.begin()
      File "/usr/lib64/python3.6/http/client.py", line 320, in begin
        version, status, reason = self._read_status()
      File "/usr/lib64/python3.6/http/client.py", line 281, in _read_status
        line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
      File "/usr/lib64/python3.6/socket.py", line 586, in readinto
        return self._sock.recv_into(b)
    KeyboardInterrupt
14:51:00.324 [ConfigProxy] error: 503 GET /jupyterhub/user/admin/api/contents socket hang up
[I 2024-04-05 14:51:00.327 JupyterHub log:189] 200 GET /jupyterhub/hub/error/503?url=%2Fjupyterhub%2Fuser%2Fadmin%2Fapi%2Fcontents%3Fcontent%3D1%261712328658000 (@127.0.0.1) 1.84ms
[E 2024-04-05 14:51:03.991 JupyterHub base:101] Failed to get current user
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 96, in prepare
        await self.get_current_user()
      File "/usr/local/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 429, in get_current_user
        user = await self.refresh_auth(user)
      File "/usr/local/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 327, in refresh_auth
        auth_info = await self.authenticator.refresh_user(user, self)
      File "/usr/local/lib/python3.6/site-packages/jhub_remote_user_authenticator/remote_user_auth.py", line 129, in refresh_user
        await handler.stop_single_user(user, user.spawner.name)
      File "/usr/local/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 1149, in stop_single_user
        raise RuntimeError(f"{spawner._log_name} pending {spawner.pending}")
    RuntimeError: admin pending stop

Can you show the surrounding logs that precede Removing user admin from proxy? This means the server stopped or was requested to stop.

Are you running an idle culler service? If so, logs from that would be useful as well.

Nothing out of the ordinary

[I 2024-04-10 10:29:24.962 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/translations/?1712744963966 (admin@****) 30.56ms
[I 2024-04-10 10:29:24.992 SingleUserLabApp log:189] 200 GET /jupyterhub/user/admin/lab/api/workspaces/default?1712744963997 (admin@****) 28.76ms

I figured out from other posts that I’m using refresh_user wrong. RemoteUserLocalAuthenticator is called once when the user logs in… and then I kill the session when auth_refresh_age expires. Which is triggered when I access /hub/ from the browser or when some internal mechanism (/jupyterhub/hub/api/users/admin/activity) is executed.

The bigger picture was to log off the user from jupyterhub after he would logout from the webapp which provided the initial cookie for login. Couldn’t find a way to do it with refresh_user.