Keycloak identity management: 500 : Internal Server Error

Hello,

I have configured the Keycloak Identity and Access Management service and tested it with a sample app and it works. Now, I am trying to configure my JupyterHub running on minikube, to use this service for user authentication. I have also tested the JupyterHub authentication using GitHub on EKS and the basic authentication locally, all worked as expected.

However, the Keycloak authentication service seems to be a bit tricky, here is the configuration from the helm chart I am using:

GenericOAuthenticator:
      admin_users:
        - pritimoy.podder@globaltrustgrp.com
      client_id: jupyterclient
      client_secret: *********************
      oauth_callback_url: http://192.168.2.32:30389/hub/oauth_callback
      authorize_url: https://192.168.2.25/auth/realms/JupyterHub/protocol/openid-connect/auth
      token_url: https://192.168.2.25/auth/realms/JupyterHub/protocol/openid-connect/token
      userdata_url: https://192.168.2.25/auth/realms/JupyterHub/protocol/openid-connect/userinfo
      login_service: keycloak
      username_key: preferred_username
      userdata_params:
        state: state
    JupyterHub:
      authenticator_class: generic-oauth

With this configuration, I am able to the Keycloak identity management service. But, receive the Internal Server Error, the redirect URL is:
http://192.168.2.32:30389/hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJhMTk4YjNmMTY2ZmE0OTcwOWFhY2EzYzdjZDdjNTkwZCIsICJuZXh0X3VybCI6ICIvaHViLyJ9&session_state=6c7639c5-8e3e-427c-9d41-b88cd796e5f6&code=409dc4b1-0d84-4852-b735-d0880d9fe4e9.6c7639c5-8e3e-427c-9d41-b88cd796e5f6.c1d5c777-edf9-45ea-b889-178d64933299.

I can confirm the hostname of the initial URL at which the user visited the Hub is the same hostname registered with the OAuth provider for the callback.

Any help would be greatly appreciated.

Many thanks.

Hi! Please can you show us your full config with secrets redacted, tell us how you setup your K8S server, and show us the logs for the full JupyterHub login process with debug logging enabled? I’m not familiar with KeyCloak but the more information you provide the more likely it is that someone will be able to help.

make sure preferred_username is part of the Keycloak token. in your Keycloak Console go to Clients > Your Client > Client Score > Evaluate. if it isn’t there, go to Mappers tab and add built-in profile Token Mapper.

Hi,
Thanks for your response.

I have minikube running on a local workstation and using the helm chart to deploy JHub on it. The config I am using is as follows:

imagePullSecret:
  create: true
  automaticReferenceInjection: true
  registry: 'docker.io'
  username: **********
  password: **************
  email: ***********

debug:
  enabled: true
  
proxy:
  #enabled: true
  secretToken: **********************************
ingress:
  enabled: true
  hosts:
    - jhub.gt
hub:
  config:
    
    GenericOAuthenticator:
      admin_users:
        - ******
      client_id: jupyterclient
      client_secret: *****************
      oauth_callback_url: http://{jupyterhub_ip}/hub/oauth_callback
      authorize_url: https://{keycloak_server_ip}/auth/realms/JupyterHub/protocol/openid-connect/auth
      token_url: https://{keycloak_server_ip}/auth/realms/JupyterHub/protocol/openid-connect/token
      userdata_url: https://{keycloak_server_ip}/auth/realms/JupyterHub/protocol/openid-connect/userinfo
      login_service: keycloak
      username_key: preferred_username
      userdata_params:
        state: state
    JupyterHub:
      authenticator_class: generic-oauth

singleuser:
  defaultUrl: "/lab"
  memory:
    limit: 8G
    guarantee: 6G
  cpu:
    limit: 8
    guarantee: 2
  # Create the user home directory by adding the pre-populated notebooks
  lifecycleHooks:
    postStart:
      exec:
        command:
          - "sh"
          - "-c"
          - >
            cp -r /tmp/notebooks/* /home/jovyan
            
    # Using nbgitpuller to synchronize a folder
    # https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-environment.html
  # Defines the default image
  image:
    name: notebook
    tag: latest
    pullPolicy: Always
  
  # Remember, if minikube (hyper-v) is used for a local cluster, you must mount the windows volume in an appropriate 
  # directory and use that path as a host path in the persistent volume
  storage:
    extraVolumes:
      - name: jupyterhub-shared
        persistentVolumeClaim:
          claimName: pvc-tds1
    extraVolumeMounts:
      - name: jupyterhub-shared
        mountPath: /TDS

And the logs from the hub pod:

[E 2021-12-20 09:29:23.983 JupyterHub oauth2:389] Error fetching access token 599 POST https://192.168.2.25/auth/realms/JupyterHub/protocol/openid-connect/token: HTTP 599: SSL certificate problem: self signed certificate
[E 2021-12-20 09:29:23.984 JupyterHub web:1789] Uncaught exception GET /hub/oauth_callback?state=************&session_state=**********&code=********************** (::ffff:172.17.0.1)
    HTTPServerRequest(protocol='http', host='192.168.2.32:30389', method='GET', uri='/hub/oauth_callback?state=*********&session_state=********&code=****************', version='HTTP/1.1', remote_ip='::ffff:172.17.0.1')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 231, in get
        user = await self.login_user()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 754, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 469, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/generic.py", line 169, in authenticate
        token_resp_json = await self._get_token(headers, params)
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 390, in fetch
        raise e
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 369, in fetch
        resp = await self.http_client.fetch(req, **kwargs)
    tornado.curl_httpclient.CurlError: HTTP 599: SSL certificate problem: self signed certificate

[D 2021-12-20 09:29:23.984 JupyterHub base:1285] No template for 500
[E 2021-12-20 09:29:23.985 JupyterHub log:181] {
      "X-Forwarded-Host": "192.168.2.32:30389",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "30389",
      "X-Forwarded-For": "::ffff:172.17.0.1",
      "Cookie": "oauthenticator-state=[secret]",
      "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
      "Accept-Encoding": "gzip, deflate",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Cache-Control": "max-age=0",
      "Connection": "close",
      "Host": "192.168.2.32:30389"
    }
[E 2021-12-20 09:29:23.985 JupyterHub log:189] 500 GET /hub/oauth_callback?state=[secret]&session_state=[secret]&code=[secret] (@::ffff:172.17.0.1) 15.00ms

It is clear from the log that a self-signed certificate is the source of the problem.

Thanks for suggesting uploading the log.

Hi,

I really appreciate your response.

Clients > Your Client > Client Score > Evaluate : Did you mean Client Scope? But the Evaluate option is not available inside scope and there is nothing named Client Score.

In the Mappers built-in profile, there is no preferred_username option. I believe the issue is with the self-signed certificate, I have attached the log above.

Yes, I mean Client Scope. you can ignore the token for now until the certificate issue has been resolved.
Is your keycloak installation really using self-signed certificate? I don’t know how we supply JupyterHub with key to sign that cert, honestly. Better to setup Keycloak using valid certificate.

Edit: can you try adding tls_verify: false right below username_key?

I can confirm the keycloak server was deployed using a self-signed certificate. After disabling tls_verify, I can now login, which is great and thanks for that.

I am planning to deploy the whole setup on AWS at the beginning of next year; hopefully, then I should be able to verify the TLS handshake with a valid ACM ceritficate.

I’m also receiving the 500 internal server error and

400 : Bad Request
OAuth state missing from cookies

with a keycloak integration. We are not facing any issues with getting to keycloak from JupyterHub, just on redirect. Oddly enough, i can see the cookie in the redirect url and again when i inspect the traffic with the browser. https://[my-site]/hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJmYzc0MWZlMTBjNTQ0ZDI2OWYyZTZmZWFhODJkNjIyNCIsICJuZXh0X3VybCI6ICIvaHViLyJ9&code=cb171246-92cc-4215-be08-78d00caa50ba.1b477767-31a2-4ecd-aa24-a9c0b268023f.6e1dd107-04e7-4ed3-9161-71a0913accab
Jupyter is running on kubernetes without issue and uses the following config. Any help would be greatly appreciated. I should also mention, that there is a proxy between keycloak.my.site and jupyterhub.my.site which terminates ssl, but we have had no ssl issues.

debug:
  enabled: true
hub:
  config:
    GenericOAuthenticator:
      client_id: my_client
      client_secret: my_secret
      oauth_callback_url: jupyterhub.my.site/hub/oauth_callback
      authorize_url: keycloak.my.site/auth/realms/jhub/protocol/openid-connect/auth
      token_url: keycloak.my.site/auth/realms/jhub/protocol/openid-connect/token
      userdata_url: keycloak.my.site/auth/realms/jhub/protocol/openid-connect/userinfo
      login_service: keycloak
      username_key: preferred_username
      tls_verify: false
      enable_auth_state: true
      userdata_params:
        state: state
    JupyterHub:
      authenticator_class: generic-oauth

Thanks much :slight_smile:

this might help: Question: OAuth state missing from cookies · Issue #1519 · jupyterhub/jupyterhub · GitHub

on step 2, right before redirect to keycloak happens, you’ll see cookie set by jupyterhub. make sure it set the correct domain.