Error while enabling auth_state persistance for GenericOAuthenticator

JupyterHub The Littlest JupyterHub
,
1

Bug description

Unable to enable auth_state for GenericOAuthenticator by adding c.GenericOAuthenticator.enable_auth_state = True into the
file jupyterhub_config.d/awscognito.py
It throws error traitlets.config.configurable.MultipleInstanceError: An incompatible sibling of 'JupyterHub' is already instantiated as singleton: JupyterHub and fails to start.

How to reproduce

Add the following config for Generic Authenticator
jupyterhub_config.d/awscognito.py

c.GenericOAuthenticator.client_id = "client_id"
c.GenericOAuthenticator.client_secret = "client_secret"
c.GenericOAuthenticator.oauth_callback_url = "https://oauth_callback/link"

c.GenericOAuthenticator.authorize_url = "https://awsname.auth.region.amazoncognito.com/oauth2/authorize"
c.GenericOAuthenticator.token_url = "https://awsname.auth.region.amazoncognito.com//oauth2/token"
c.GenericOAuthenticator.userdata_url = "https://awsname.auth.region.amazoncognito.com/oauth2/userInfo"
c.GenericOAuthenticator.logout_redirect_url = "https://awsname.auth.region.amazoncognito.com/oauth2/logout?client_id=client_id&logout_uri=https://mysite.com/"

c.GenericOAuthenticator.login_service = "AWS Cognito"
c.GenericOAuthenticator.username_key = "username"
c.GenericOAuthenticator.userdata_method = "POST"
c.Authenticator.allow_all = True
c.Authenticator.admin_users = {'uuid1', 'uuid2', 'uuid3'}
c.GenericOAuthenticator.enable_auth_state = True

and reload jupyterhub via tljh-config reload

Expected behaviour

auth_state to be persisted and Jupyterhub be started successfully

Actual behaviour

Jupyter hub refuses to start with the error log

systemd[1]: Started jupyterhub.service.
python3[52364]: [I 2024-08-21 11:39:56.775 JupyterHub app:2885] Running JupyterHub version 4.1.5
python3[52364]: [I 2024-08-21 11:39:56.776 JupyterHub app:2915] Using Authenticator: oauthenticator.generic.GenericOAuthenticator-16.3.1
python3[52364]: [I 2024-08-21 11:39:56.776 JupyterHub app:2915] Using Spawner: tljh.user_creating_spawner.UserCreatingSpawner
python3[52364]: [I 2024-08-21 11:39:56.776 JupyterHub app:2915] Using Proxy: jupyterhub_traefik_proxy.fileprovider.TraefikFileProviderProxy-1.1.0
python3[52364]: [I 2024-08-21 11:39:56.789 JupyterHub app:1683] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
python3[52364]: [W 2024-08-21 11:39:56.873 JupyterHub configurable:214] Config option `userdata_method` not recognized by `GenericOAuthenticator`.  Did you mean one of: `userdata_params, userdata_token_method, userdata_url`?
python3[52364]: [E 2024-08-21 11:39:56.920 JupyterHub app:3408]
python3[52364]:     Traceback (most recent call last):
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/jupyterhub/app.py", line 3405, in launch_instance_async
python3[52364]:         await self.initialize(argv)
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/jupyterhub/app.py", line 2932, in initialize
python3[52364]:         await self.init_users()
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/jupyterhub/app.py", line 1957, in init_users
python3[52364]:         ck = crypto.CryptKeeper.instance()
python3[52364]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/config/configurable.py", line 583, in instance
python3[52364]:         inst = cls(*args, **kwargs)
python3[52364]:                ^^^^^^^^^^^^^^^^^^^^
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/config/configurable.py", line 119, in __init__
python3[52364]:         self._load_config(self.config)
python3[52364]:                           ^^^^^^^^^^^
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/traitlets.py", line 687, in __get__
python3[52364]:         return t.cast(G, self.get(obj, cls))  # the G should encode the Optional
python3[52364]:                          ^^^^^^^^^^^^^^^^^^
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/traitlets.py", line 635, in get
python3[52364]:         default = obj.trait_defaults(self.name)
python3[52364]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/traitlets.py", line 1897, in trait_defaults
python3[52364]:         return t.cast(Sentinel, self._get_trait_default_generator(names[0])(self))
python3[52364]:                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/traitlets.py", line 1241, in __call__
python3[52364]:         return self.func(*args, **kwargs)
python3[52364]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/jupyterhub/crypto.py", line 93, in _config_default
python3[52364]:         return JupyterHub.instance().config
python3[52364]:                ^^^^^^^^^^^^^^^^^^^^^
python3[52364]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/config/configurable.py", line 592, in instance
python3[52364]:         raise MultipleInstanceError(
python3[52364]:     traitlets.config.configurable.MultipleInstanceError: An incompatible sibling of 'JupyterHub' is already instantiated as singleton: JupyterHub
python3[52364]:     
systemd[1]: jupyterhub.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: jupyterhub.service: Failed with result 'exit-code'.
systemd[1]: jupyterhub.service: Consumed 1.341s CPU time.
systemd[1]: jupyterhub.service: Scheduled restart job, restart counter is at 1.

Your personal set up

  • OS:
    Ubuntu 24.04 LTS using the-littlest-jupyterhub

  • Version(s):
    Jupyterhub: 5.1.0
    Python 3.12.3

  • Dependencies

aiohttp==3.9.5
aiosignal==1.3.1
alembic==1.13.2
annotated-types==0.7.0
arrow==1.3.0
async-generator==1.10
attrs==23.2.0
backoff==2.2.1
bcrypt==4.1.3
certifi==2024.7.4
certipy==0.1.3
cffi==1.16.0
charset-normalizer==3.3.2
cryptography==42.0.8
escapism==1.0.1
fqdn==1.5.1
frozenlist==1.4.1
greenlet==3.0.3
idna==3.7
isoduration==20.11.0
Jinja2==3.1.4
jsonpointer==3.0.0
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
jupyter-events==0.10.0
jupyter-telemetry==0.1.0
jupyterhub==5.1.0
jupyterhub-firstuseauthenticator==1.0.0
jupyterhub-idle-culler==1.3.1
jupyterhub-ldapauthenticator==1.3.2
jupyterhub-nativeauthenticator==1.2.0
jupyterhub-systemdspawner==1.0.1
jupyterhub-tmpauthenticator==1.0.0
jupyterhub-traefik-proxy==1.1.0
ldap3==2.9.1
Mako==1.3.5
MarkupSafe==2.1.5
multidict==6.0.5
oauthenticator==16.3.1
oauthlib==3.2.2
onetimepass==1.0.1
packaging==24.1
pamela==1.1.0
passlib==1.7.4
pluggy==1.5.0
prometheus_client==0.20.0
pyasn1==0.6.0
pycparser==2.22
pycurl==7.45.3
pydantic==2.8.2
pydantic_core==2.20.1
PyJWT==2.8.0
pyOpenSSL==24.1.0
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
PyYAML==6.0.1
referencing==0.35.1
requests==2.32.3
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.19.0
ruamel.yaml==0.17.40
ruamel.yaml.clib==0.2.8
six==1.16.0
SQLAlchemy==2.0.31
the-littlest-jupyterhub @ git+https://github.com/jupyterhub/the-littlest-jupyterhub.git@fc8e19b1b5663f58f0e7b089903d1d1769db06b8
toml==0.10.2
tornado==6.4.1
traitlets==5.14.3
types-python-dateutil==2.9.0.20240316
typing_extensions==4.12.2
uri-template==1.3.0
urllib3==2.2.2
webcolors==24.6.0
yarl==1.9.4
2

This is a problem. There’s no userdata_method parameter in
https://oauthenticator.readthedocs.io/en/latest/reference/api/gen/oauthenticator.generic.html

3

Thank you for looking into this.
I have removed the offending line c.GenericOAuthenticator.userdata_method = "POST" from the jupyterhub_config.d/awscognito.py config and reloaded. The warning doesn’t come up in the logs anymore. Alias, the same error continues. This is the log now

Aug 22 08:09:26 systemd[1]: Started jupyterhub.service.
Aug 22 08:09:28 python3[4281]: [I 2024-08-22 08:09:28.207 JupyterHub app:3307] Running JupyterHub version 5.1.0
Aug 22 08:09:28 python3[4281]: [I 2024-08-22 08:09:28.208 JupyterHub app:3337] Using Authenticator: oauthenticator.generic.GenericOAuthenticator-16.3.1
Aug 22 08:09:28 python3[4281]: [I 2024-08-22 08:09:28.208 JupyterHub app:3337] Using Spawner: tljh.user_creating_spawner.UserCreatingSpawner
Aug 22 08:09:28 python3[4281]: [I 2024-08-22 08:09:28.208 JupyterHub app:3337] Using Proxy: jupyterhub_traefik_proxy.fileprovider.TraefikFileProviderProxy-1.1.0
Aug 22 08:09:28 python3[4281]: [I 2024-08-22 08:09:28.222 JupyterHub app:1837] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
Aug 22 08:09:28 python3[4281]: [E 2024-08-22 08:09:28.406 JupyterHub app:3873]
Aug 22 08:09:28 python3[4281]:     Traceback (most recent call last):
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/jupyterhub/app.py", line 3870, in launch_instance_async
Aug 22 08:09:28 python3[4281]:         await self.initialize(argv)
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/jupyterhub/app.py", line 3354, in initialize
Aug 22 08:09:28 python3[4281]:         await self.init_users()
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/jupyterhub/app.py", line 2114, in init_users
Aug 22 08:09:28 python3[4281]:         ck = crypto.CryptKeeper.instance()
Aug 22 08:09:28 python3[4281]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/config/configurable.py", line 583, in instance
Aug 22 08:09:28 python3[4281]:         inst = cls(*args, **kwargs)
Aug 22 08:09:28 python3[4281]:                ^^^^^^^^^^^^^^^^^^^^
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/config/configurable.py", line 119, in __init__
Aug 22 08:09:28 python3[4281]:         self._load_config(self.config)
Aug 22 08:09:28 python3[4281]:                           ^^^^^^^^^^^
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/traitlets.py", line 687, in __get__
Aug 22 08:09:28 python3[4281]:         return t.cast(G, self.get(obj, cls))  # the G should encode the Optional
Aug 22 08:09:28 python3[4281]:                          ^^^^^^^^^^^^^^^^^^
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/traitlets.py", line 635, in get
Aug 22 08:09:28 python3[4281]:         default = obj.trait_defaults(self.name)
Aug 22 08:09:28 python3[4281]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/traitlets.py", line 1897, in trait_defaults
Aug 22 08:09:28 python3[4281]:         return t.cast(Sentinel, self._get_trait_default_generator(names[0])(self))
Aug 22 08:09:28 python3[4281]:                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/traitlets.py", line 1241, in __call__
Aug 22 08:09:28 python3[4281]:         return self.func(*args, **kwargs)
Aug 22 08:09:28 python3[4281]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/jupyterhub/crypto.py", line 93, in _config_default
Aug 22 08:09:28 python3[4281]:         return JupyterHub.instance().config
Aug 22 08:09:28 python3[4281]:                ^^^^^^^^^^^^^^^^^^^^^
Aug 22 08:09:28 python3[4281]:       File "/opt/tljh/hub/lib/python3.12/site-packages/traitlets/config/configurable.py", line 592, in instance
Aug 22 08:09:28 python3[4281]:         raise MultipleInstanceError(
Aug 22 08:09:28 python3[4281]:     traitlets.config.configurable.MultipleInstanceError: An incompatible sibling of 'JupyterHub' is already instantiated as singleton: JupyterHub
Aug 22 08:09:28 python3[4281]:     
Aug 22 08:09:28 systemd[1]: jupyterhub.service: Main process exited, code=exited, status=1/FAILURE
Aug 22 08:09:28 systemd[1]: jupyterhub.service: Failed with result 'exit-code'.
Aug 22 08:09:28 systemd[1]: jupyterhub.service: Consumed 1.605s CPU time.
Aug 22 08:09:28 systemd[1]: jupyterhub.service: Scheduled restart job, restart counter is at 1.

This how the jupyterhub_config.d/awscognito.py would look like now

c.GenericOAuthenticator.client_id = "client_id"
c.GenericOAuthenticator.client_secret = "client_secret"
c.GenericOAuthenticator.oauth_callback_url = "https://oauth_callback/link"

c.GenericOAuthenticator.authorize_url = "https://awsname.auth.region.amazoncognito.com/oauth2/authorize"
c.GenericOAuthenticator.token_url = "https://awsname.auth.region.amazoncognito.com//oauth2/token"
c.GenericOAuthenticator.userdata_url = "https://awsname.auth.region.amazoncognito.com/oauth2/userInfo"
c.GenericOAuthenticator.logout_redirect_url = "https://awsname.auth.region.amazoncognito.com/oauth2/logout?client_id=client_id&logout_uri=https://mysite.com/"

c.GenericOAuthenticator.login_service = "AWS Cognito"
c.GenericOAuthenticator.username_key = "username"
c.Authenticator.allow_all = True
c.Authenticator.admin_users = {'uuid1', 'uuid2', 'uuid3'}
c.GenericOAuthenticator.enable_auth_state = True
4

It’s a strange bug, I’ve opened an issue:

A workaround for now is to modify your systemd service file /etc/systemd/system/jupyterhub.service:

ExecStart=/opt/tljh/hub/bin/python3 -m jupyterhub -f /opt/tljh/hub/lib/python3.8/site-packages/tljh/jupyterhub_config.py --upgrade-db

You’ll also need to set the JUPYTERHUB_CRYPT_KEY environment variable
e.g. in your systemd file

Environment=JUPYTERHUB_CRYPT_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

Then reload systemd, and restart JupyterHub

sudo systemctl daemon-reload
tljh-config reload
1 Like
5

Thank you for opening the issue and for the prompt fix

1 Like