My team has added a few custom layers to the jupyterhub chart mostly by using the custom, config, extraEnv, and extraConfig sections. One customization is to inherit the GenericOAuthenticator. Our oauth provider is Keycloak and we have a private certificate authority.
When I try to update the chart from 2.0.0 to 3.1.2 or 3.3.5, I get the following error:
[E 2024-04-12 12:23:13.918 JupyterHub oauth2:653] Error fetching 599 POST https://keycloak.********/realms/myrealm/protocol/openid-connect/token: HTTP 599: SSL certificate problem: unable to get local issuer certificate
[E 2024-04-12 12:23:13.919 JupyterHub web:1875] Uncaught exception GET /hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICIwZjBhZjFlN2MyNzE0ZGQyODg3N2E5MWJkYTUzYzdiNiJ9&session_state=c5f42d87-2a02-4bb4-9587-21c7276caf0a&code=4bc43458-2a5f-4ec4-a712-9fdd667365b7.c5f42d87-2a02-4bb4-9587-21c7276caf0a.dd0cdafd-65dc-4b72-a89e-6a4e892e3580 (127.0.0.1)
HTTPServerRequest(protocol='https', host='mynamespace.ehpc.evoforge.org', method='GET', uri='/hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICIwZjBhZjFlN2MyNzE0ZGQyODg3N2E5MWJkYTUzYzdiNiJ9&session_state=c5f42d87-2a02-4bb4-9587-21c7276caf0a&code=4bc43458-2a5f-4ec4-a712-9fdd667365b7.c5f42d87-2a02-4bb4-9587-21c7276caf0a.dd0cdafd-65dc-4b72-a89e-6a4e892e3580', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/tornado/web.py", line 1790, in _execute
result = await result
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 210, in get
user = await self.login_user()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 928, in login_user
authenticated = await self.authenticate(data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jupyterhub/auth.py", line 493, in get_authenticated_user
authenticated = await maybe_future(self.authenticate(handler, data))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 1029, in authenticate
token_info = await self.get_token_info(handler, access_token_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 874, in get_token_info
token_info = await self.httpfetch(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 688, in httpfetch
return await self.fetch(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 654, in fetch
raise e
File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 633, in fetch
resp = await self.http_client.fetch(req, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tornado.curl_httpclient.CurlError: HTTP 599: SSL certificate problem: unable to get local issuer certificate
I have looked for related issues, but have only found a similar issue where the deployment is using a proxy.
I’m pretty sure I’m missing something simple, but I don’t see any setting in the values.yaml that would point to the tls layer or adding a ca chain.
The block of code failing here still works when we use jupyterhub:2.0.0 chart.
In our extraEnv these settings set up Keycloak as the OAuthenticator:
AUTHENTICATOR_CLASS: 'GenericOAuthenticator'
OAUTH2_USERNAME_KEY: 'preferred_username'
OAUTH2_TLS_VERIFY: '0'
OAUTH2_AUTHORIZE_URL: 'https://<keycloak url>/realms/myrealm/protocol/openid-connect/auth'
OAUTH2_TOKEN_URL: 'https://<keycloak url>/realms/<myrealm>/protocol/openid-connect/token'
OAUTH2_USERDATA_URL: 'https://<keycloak url>/realms/myrealm/protocol/openid-connect/userinfo'
OAUTH_CALLBACK_URL: 'https://<jupyterhub url>/hub/oauth_callback'
In our extraConfig:
01-myAuth: |
from oauthenticator.generic import GenericOAuthenticator
import os
class MyGenericOAuthenticator(GenericOAuthenticator):
from tornado import gen
@gen.coroutine
def pre_spawn_start(self,user,spawner):
auth_state = yield user.get_auth_state()
if not auth_state:
return
oauth_user = auth_state['oauth_user']
if not oauth_user:
return
#print(f' in MyGenericOAuthenticator from values oauth_user is {oauth_user}')
spawner.environment['NB_USER'] = spawner.user.name
spawner.environment['NB_UID'] = oauth_user['uid']
spawner.environment['NB_GID'] = oauth_user['gid']
spawner.environment['NB_GROUP'] = oauth_user['group']
spawner.environment['EXTRA_GROUPS'] = oauth_user.get('extra_groups','')
print('RUNNING 01-myAuth from values file')
c.JupyterHub.authenticator_class = MyGenericOAuthenticator
c.Application.log_level = 'DEBUG'
c.KubeSpawner.args = ['--allow-root']
c.Spawner.cmd = ['start.sh','jupyterhub-singleuser','--allow-root']
in the hub.config we define an authenticator:
config:
JupyterHub:
admin_users:
- admin
- user1
admin_access: true
authenticator_class: generic-oauth
MyGenericOAuthenticator:
authorize_url: https://{keycloak url}/realms/myrealm/protocol/openid-connect/auth
token_url: https://{keycloak url}/realms/myrealm/protocol/openid-connect/token
userdata_url: https://{keycloak url}/realms/myrealm/protocol/openid-connect/userinfo
login_service: "Keycloak"
username_key: preferred_username
userdata_params:
state: state
scope:
- openid
- uid
- gid
- group
- extra_groups
- groups
enable_auth_state: true
# trying auto_login to bypass big orange button
auto_login: false
We also have a “configMap” which writes to the jupyter_config.d the authenticator class. Here is the section of that class showing how it’s setup:
c.JupyterHub.authenticator_class = GenericOAuthenticator
c.Spawner.authClass = AuthenticatorClass.GenericOAuthenticator
setClientParams(c.GenericOAuthenticator)
c.GenericOAuthenticator.login_service = 'KEYCLOAK'
c.GenericOAuthenticator.auto_login = False
c.GenericOAuthenticator.tls_verify = False
c.GenericOAuthenticator.userdata_params = {"state": "state"}
c.GenericOAuthenticator.admin_users={{ .Values.jupyterhub.hub.config.JupyterHub.admin_users | toJson }}
print(f' setting admin_users to {c.GenericOAuthenticator.admin_users}')
c.GenericOAuthenticator.scope = {{ .Values.jupyterhub.hub.config.MyGenericOAuthenticator.scope | toJson }}
print(f' requesting scopes {c.GenericOAuthenticator.scope}')
c.GenericOAuthenticator.refresh_pre_spawn = True
print(f' just set refresh_pre_spawn {c.GenericOAuthenticator.refresh_pre_spawn}')
c.GenericOAuthenticator.enable_auth_state = True
My question: What has changed in the JupyterHub version that is causing this type of error and what value could I be missing?
My assumption is that I need to add a ca cert to the hub deployment somewhere, but I’m not sure where.