Jupyterhub throws 403 error, when I open a notebook without any actions

Hi guys
I am new to Jupyterhub.
Now I want to integrate Jupyterhub into my system, by using custom authentication, because the user info will be updated, so I added the refresh_user function into my custom authentication class to refresh user info using cookie
Through my work, it can start. Still, when I open a notebook without any actions, it throws an error.

I tried to follow the log to find out why but failed.

Can you answer these questions that confused me

  1. Why is this error append?
  2. I found that there were some different handlers for refresh_user. Such as: tornado.web.RequestHandler or None, What’s the difference between these two? When will Handler be None?
  3. According to the specification, should my custom authentication be placed in jupyter_config.py or auth.py?

Here are the info

Env Info:
Python 3.10.0
JupyterHub latest
Jupyter latest

Page Alert Error:

image

Log Error:

[W 2023-11-09 21:14:40.656 JupyterHub base:348] User admin has stale auth info. Login is required to refresh.
[W 2023-11-09 21:14:40.658 JupyterHub log:191] 403 GET /hub/api/user (@127.0.0.1) 14.96ms
[W 2023-11-09 21:14:40.658 ServerApp] No Hub user identified for request
[W 2023-11-09 21:14:40.659 ServerApp] Token stored in cookie may have expired
[W 2023-11-09 21:14:40.661 ServerApp] wrote error: 'Forbidden'
    Traceback (most recent call last):

[D 2023-11-09 21:17:23.249 JupyterHub proxy:392] Checking routes
[W 2023-11-09 21:17:40.687 ServerApp] Token stored in cookie may have expired
[W 2023-11-09 21:17:40.687 ServerApp] Detected unused OAuth state cookies
[W 2023-11-09 21:17:40.689 ServerApp] wrote error: 'Forbidden'
    Traceback (most recent call last):

Config Info

from jupyterhub.auth import Authenticator
from jupyterhub.spawner import SimpleLocalProcessSpawner

c = get_config()  # noqa

# Auth application info
c.JupyterHub.authenticator_class = "MyCustomAuthenticator"
c.Authenticator.admin_users = ['admin', 'test']
c.JupyterHub.admin_access = True
# Auto login
c.Authenticator.auto_login = True
# The max age (in seconds) of authentication info before forcing a refresh of user auth info
c.Authenticator.auth_refresh_age = 60
# Force refresh before start spawn, it needs to work with refresh_user()
c.Authenticator.refresh_pre_spawn = True
# Let authenticator manage the user' group
c.Authenticator.manage_groups = False

# Open auth state to storage the data
c.Authenticator.enable_auth_state = True
# c.Authenticator.allow_all = True
#export JUPYTERHUB_CRYPT_KEY=$(openssl rand -hex 32)

# Spawner
# TODO
c.JupyterHub.spawner_class = SimpleLocalProcessSpawner
# c.JupyterHub.spawner_class = dockerspawner.DockerSpawner
# TODO
c.Spawner.notebook_dir = '/tmp/my-jupyterhub'
c.Spawner.default_url = ''
# Allow spawner run as root
c.Spawner.args = ['--allow-root']

# Only listen on localhost for testing
c.JupyterHub.bind_url = 'http://localhost:8000'

Thank you all!

Can you show us the code for MyCustomAuthenticator? If necessary show us a simplified version.

Thanks for your reply!

Here is my custom authentication code

class CUSTOMAuthenticator(Authenticator):
    """Authenticator process
    1. Get CUSTOM user' cookie who has benn logined
    2. Call CUSTOM api to check this cookie's validity
    3. Check api response, if pass will set user info to jupyterhub, if not will return None
    """

    """Authenticate CUSTOM user' by cookie
    Parameters
        handler (tornado.web.RequestHandler) – the current request handler
        data (dict) – The formdata of the login form. The default form has ‘username’ and ‘password’ fields.
    Returns
        The username of the authenticated user, or None if Authentication failed.
        The Authenticator may return a dict instead, which MUST have a key name holding the username, and MAY have additional keys:
        auth_state, a dictionary of of auth state that will be persisted;
        admin, the admin setting value for the user
        groups, the list of group names the user should be a member of, if Authenticator.manage_groups is True.
    Return type
        user (str or dict or None)
    """

    async def authenticate(self, handler, data):
        # Authenticate CUSTOM user info
        custom_authentication_result = self.custom_authentication(handler)

        if not custom_authentication_result:
            self.log.error("CUSTOM_AUTH_FAILED_COOKIE_NOT_AVAILABLE")
            return None
        custom_username = str(custom_authentication_result.get(self.CUSTOM_API_KEY_USERNAME, ""))
        custom_user_id = custom_authentication_result.get(self.CUSTOM_API_KEY_ID, "")
        if not custom_username or not custom_user_id:
            self.log.error("CUSTOM_AUTH_FAILED_USER_INFO_EMPTY | CUSTOM_USERNAME: %s | CUSTOM_USER_ID: %s",
                           custom_username, custom_user_id)
            return None

        # Jupyter username
        custom_detail_id = str(self.custom_extract_cookie_by_key(handler, [self.CUSTOM_COOKIE_KEY_DETAIL_ID])
                                  .get(self.CUSTOM_COOKIE_KEY_DETAIL_ID, ""))
        self.log.debug("CUSTOM_AUTH_FAILED_DETAIL_INFO: %s", custom_detail_id)

        if not custom_detail_id:
            self.log.error("CUSTOM_AUTH_FAILED_DETAIL_INFO_EMPTY | CUSTOM_USERNAME: %s | CUSTOM_USER_ID: %s",
                           custom_username, custom_user_id)
            return None
        jupyter_username = str.lower(custom_username) + "-" + custom_detail_id
        self.log.debug("CUSTOM_AUTH_JUPYTER_USERNAME: %s", jupyter_username)

        # Set auth state
        authenticator_auth_state_result = {}
        authenticator_auth_state_result[self.CUSTOM_AUTH_STATE_KEY_NAME] = jupyter_username

        # Db connect info
        custom_biz_data_result = self.custom_get_biz_data(handler, custom_user_id)
        if not custom_biz_data_result:
            for auth_state_key in self.CUSTOM_AUTH_STATE_LIST:
                authenticator_auth_state_result[auth_state_key] = custom_biz_data_result.get(auth_state_key, "")

        self.log.debug("CUSTOM_AUTH_AUTHENTICATOR_AUTH_STATE_RESULT %s", str(authenticator_auth_state_result))
        return authenticator_auth_state_result

    """Hook for after authenticate
    Parameters
        handler (tornado.web.RequestHandler) – the current request handler
        authentication (dict) – User authentication data dictionary. Contains the username (‘name’), admin status (‘admin’), and auth state dictionary (‘auth_state’).
    Returns
        The hook must always return the authentication dict
    Return type
        Authentication (dict)
    """

    # Refresh user when spawner starts every time, it needs work with: c.Authenticator.refresh_pre_spawn = True
    """Refresh auth data for a given user. Only override if your authenticator needs to refresh its data about users once in a while.
    Parameters
        user (User) – the user to refresh
        handler (tornado.web.RequestHandler or None) – the current request handler
    Returns
        Return True if auth data for the user is up-to-date and no updates are required.
        Return False if the user’s auth data has expired, and they should be required to login again.
        Return a dict of auth data if some values should be updated. This dict should have the same structure as that returned by authenticate() when it returns a dict. Any fields present will refresh the value for the user. Any fields not present will be left unchanged. This can include updating .admin or .auth_state fields.
    Return type
        auth_data (bool or dict)
    """

    async def refresh_user(self, user, handler=None):
        if handler and isinstance(handler, tornado.web.RequestHandler):
            """Refresh user process
            1. Check the user's authentication is valid. If it's valid will make the user's latest custom data dict, if not will return None
            2. Get custom data from cookie, build and return the data dict for the spawner
            """
            # Authenticate CUSTOM user info
            custom_authentication_result = self.custom_authentication(handler)

            if not custom_authentication_result:
                self.log.error("CUSTOM_AUTH_REFRESH_USER_COOKIE_NOT_AVAILABLE")
                return False
            custom_username = custom_authentication_result.get(self.CUSTOM_API_KEY_USERNAME, "")
            custom_user_id = custom_authentication_result.get(self.CUSTOM_API_KEY_ID, "")
            if not custom_username:
                self.log.error("CUSTOM_AUTH_REFRESH_USER_INFO_EMPTY | CUSTOM_USERNAME: %s | CUSTOM_USER_ID: %s",
                               custom_username, custom_user_id)
                return False

            # Set auth state
            authenticator_auth_state_result = {}
            authenticator_auth_state_result_item = {}
            # Jupyter username
            custom_detail_id = str(
                self.custom_extract_cookie_by_key(handler, [self.CUSTOM_COOKIE_KEY_DETAIL_ID])
                .get(self.CUSTOM_COOKIE_KEY_DETAIL_ID, ""))

            if not custom_detail_id:
                self.log.error("CUSTOM_AUTH_REFRESH_USER_DETAIL_INFO_EMPTY | CUSTOM_USERNAME: %s | CUSTOM_USER_ID: %s",
                               custom_username, custom_user_id)
                return False
            jupyter_username = str.lower(custom_username) + "-" + custom_detail_id
            authenticator_auth_state_result[self.CUSTOM_AUTH_STATE_KEY_NAME] = jupyter_username

            # Db connect info
            custom_biz_data_result = self.custom_get_biz_data(handler, custom_user_id)
            if custom_biz_data_result:
                for auth_state_key in self.CUSTOM_AUTH_STATE_LIST:
                    authenticator_auth_state_result_item[auth_state_key] = custom_biz_data_result.get(auth_state_key,
                                                                                                        "")
                    self.log.debug("CUSTOM_AUTH_REFRESH_USER_AUTH_STATE_RESULT_ITEM %s", auth_state_key)

            authenticator_auth_state_result[self.CUSTOM_AUTH_STATE_KEY_STATE] = authenticator_auth_state_result_item
            self.log.debug("CUSTOM_AUTH_REFRESH_USER_AUTH_STATE_RESULT %s", authenticator_auth_state_result)
            return authenticator_auth_state_result
        else:
            self.log.debug("CUSTOM_AUTH_REFRESH_USER_START_HANDLER_EMPTY: %s", handler)
            # Some action to update user's plugin and file, don't need to update auth_state
            self.log.warning("CUSTOM_AUTH_REFRESH_USER_HANDLER_NOT_REQUEST")
            return True

    """Hook called before spawning a user’s server
    """

    async def pre_spawn_start(self, user, spawner):
        # Pass custom custom data to spawner via environment variable
        auth_state = await user.get_auth_state()
        if auth_state:
            self.log.debug("CUSTOM_AUTH_CUSTOM_DADA: %s", str(auth_state))

            db_host = auth_state.get(self.CUSTOM_API_KEY_DB_HOST, "")
            # !!!Note: Spawner environment parameter's value must be string!!!
            spawner.environment[self.CUSTOM_ENV_KEY_DB_HOST] = str(
                auth_state.get(self.CUSTOM_API_KEY_DB_HOST, ""))
            spawner.environment[self.CUSTOM_ENV_KEY_DB_PORT] = str(
                auth_state.get(self.CUSTOM_API_KEY_DB_PORT, ""))
        else:
            self.log.error("CUSTOM_AUTH_CUSTOM_DADA_EMPTY: %s", str(user))

    """Hook called after stopping a user container
    """

    async def post_spawn_stop(self, user, spawner):
        self.log.debug("CUSTOM_AUTH_POST_SPAWNER_STOP")


    """Authenticate CUSTOM cookie by calling api
    """

    def custom_authentication(self, handler):
        custom_cookie_dict = self.custom_extract_cookie_by_key(handler, [self.CUSTOM_COOKIE_KEY_UCENTER,
                                                                             self.CUSTOM_COOKIE_KEY_SESSION])
        if not custom_cookie_dict:
            self.log.error("CUSTOM_AUTH_COOKIE_DICT_EMPTY")
            return False
        custom_cookie_result_ucenter_session = custom_cookie_dict.get(self.CUSTOM_COOKIE_KEY_UCENTER, "")
        custom_cookie_result_session = custom_cookie_dict.get(self.CUSTOM_COOKIE_KEY_SESSION, "")

        if not custom_cookie_result_ucenter_session or not custom_cookie_result_session:
            self.log.error("CUSTOM_AUTH_AUTH_COOKIE_EMPTY | UCENTER_SESSION: %s | SESSION: %s",
                           custom_cookie_result_ucenter_session, custom_cookie_result_session)
            return False
        try:
            self.log.debug("CUSTOM_AUTH_HTTP_URL: %s | COOKIES: %s", self.CUSTOM_API_URI_AUTHENTICATION,
                           str(custom_cookie_dict))
            http_response = requests.get(self.CUSTOM_API_URI_AUTHENTICATION, cookies=custom_cookie_dict,
                                         timeout=self.CUSTOM_API_TIME_OUT_AUTHENTICATION)
            self.log.debug("CUSTOM_AUTH_HTTP_RESPONSE: %s", str(http_response))
            if not http_response or http_response.status_code != 200:
                self.log.error("CUSTOM_AUTH_FAILED: %s", http_response)
                return False

            http_response_decode = http_response.json()
            http_response_result = http_response_decode.get("value", {})
            self.log.debug("CUSTOM_AUTH_HTTP_RESPONSE_RESULT: %s", str(http_response_result))
            return http_response_result
        except Exception as ex:
            self.log.error("CUSTOM_AUTH_EXCEPTION: %s", ex)
            return False

    """Get CUSTOM Biz Data
    """

    def custom_get_biz_data(self, handler, custom_user_id):
        custom_cookie_dict = self.custom_extract_cookie_by_key(handler, [self.CUSTOM_COOKIE_KEY_DATASET_ID,
                                                                             self.CUSTOM_COOKIE_KEY_DETAIL_ID])
        if not custom_cookie_dict:
            self.log.error("CUSTOM_AUTH_GET_BIZ_DATA_COOKIE_DICT_EMPTY")
            return None
        request_params = {
            self.CUSTOM_API_KEY_USER_ID: custom_user_id,
            self.CUSTOM_API_KEY_DATASET_ID: custom_cookie_dict.get(self.CUSTOM_COOKIE_KEY_DATASET_ID, ""),
            self.CUSTOM_API_KEY_DETAIL_ID: custom_cookie_dict.get(self.CUSTOM_COOKIE_KEY_DETAIL_ID, ""),
        }
        try:
            self.log.debug("CUSTOM_AUTH_HTTP_URL_BIZ_DATA: %s | PRAMAS: %s", self.CUSTOM_API_URI_DB_INFO,
                           str(request_params))
            http_response = requests.get(self.CUSTOM_API_URI_DB_INFO, params=request_params,
                                         cookies=custom_cookie_dict,
                                         timeout=self.CUSTOM_API_TIME_OUT_AUTHENTICATION)
            if not http_response or http_response.status_code != 200:
                self.log.error("CUSTOM_AUTH_GET_BIZ_DATA_FAILED: %s", http_response)
                return False

            http_response_decode = http_response.json()
            http_response_result = http_response_decode.get("value", {})
            self.log.debug("CUSTOM_AUTH_GET_BIZ_DATA_RESULT: %s", str(http_response_result))
            return http_response_result
        except Exception as ex:
            self.log.error("CUSTOM_AUTH_GET_BIZ_DATA_EXCEPTION: %s", ex)
            return False

    """Extract cookie from handler
    """

    def custom_extract_cookie_by_key(self, handler, cookie_keys=[]):
        custom_cookie_dict = {}
        if not handler:
            self.log.error("CUSTOM_AUTH_EXTRACT_COOKIE_HANDLER_EMPTY")
            return custom_cookie_dict
        if not cookie_keys or len(cookie_keys) == 0:
            return custom_cookie_dict

        for cookie_key in cookie_keys:
            custom_cookie_dict[cookie_key] = handler.get_cookie(cookie_key, "")
        return custom_cookie_dict


What does authenticator_auth_state_result look like? Feel free to redact any sensitive info.

Thank you for reply.

Here are the refresh user responses:

1. User authenticate pass and auth_state has results

{
    "name": "username01",
    "auth_state": {
        "key01": "value01",
        "key02": "value02",
        "key03": "value03"
    }
}

2. User authenticate failed

False

3. User authenticate pass

True

Can you turn on debug logging and share your logs? There should be lots of entries corresponding to your log statements in your authenticator.