Setup OAuth2 locally - tornado.curl_httpclient.CurlError: HTTP 599: server certificate verification failed. CAfile: none CRLfile: none

JupyterHub
1

I work on a setup with JupyterHub and keycloak. The setup is placed locally inside a VM with vagrant and inside the VM docker is used. In the hosts file of my host machine, I have added jhub and keycloak as alternative names for 127.0.0.1. vagrant automatically forwards the ports 8443 and 8000 inside the VM.

First, I open the JupyterHub webpage in my browser on the host machine of the VM. On the JupyterHub landing page (before login), I click on the button that forwards me to the keycloak login mask. After the successful login, I get forwarded back to the JupyterHub. But here I run into a “500 : Internal Server Error”. This can be seen in the logs:

jhub_1         | [I 2021-01-19 17:44:00.545 JupyterHub log:181] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://keycloak:8443/auth/realms/jhub/protocol/openid-connect/auth?response_type=code&redirect_uri=https%3A%2F%2Fjhub%3A8000%2Fhub%2Foauth_callback&client_id=jhub-1&state=[secret]&scope=openid (@::ffff:10.0.2.2) 1.21ms
jhub_1         | [E 2021-01-19 17:44:00.703 JupyterHub web:1789] Uncaught exception GET /hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJjNDgwMDAxMDg5NTQ0NmZlYjY3MmVkMDNkMTA3ZTY0YSIsICJuZXh0X3VybCI6ICIvaHViLyJ9&session_state=3817a8f6-9d0d-4495-8ce5-ca4c9abc192e&code=40819154-5408-4c67-b5be-d0032045e791.3817a8f6-9d0d-4495-8ce5-ca4c9abc192e.581c2301-1739-44bb-b398-2889b13ddd26 (::ffff:10.0.2.2)
jhub_1         |     HTTPServerRequest(protocol='https', host='jhub:8000', method='GET', uri='/hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJjNDgwMDAxMDg5NTQ0NmZlYjY3MmVkMDNkMTA3ZTY0YSIsICJuZXh0X3VybCI6ICIvaHViLyJ9&session_state=3817a8f6-9d0d-4495-8ce5-ca4c9abc192e&code=40819154-5408-4c67-b5be-d0032045e791.3817a8f6-9d0d-4495-8ce5-ca4c9abc192e.581c2301-1739-44bb-b398-2889b13ddd26', version='HTTP/1.1', remote_ip='::ffff:10.0.2.2')
jhub_1         |     Traceback (most recent call last):
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
jhub_1         |         result = await result
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 224, in get
jhub_1         |         user = await self.login_user()
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 749, in login_user
jhub_1         |         authenticated = await self.authenticate(data)
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 462, in get_authenticated_user
jhub_1         |         authenticated = await maybe_future(self.authenticate(handler, data))
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/oauthenticator/generic.py", line 155, in authenticate
jhub_1         |         token_resp_json = await self._get_token(http_client, headers, params)
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/oauthenticator/generic.py", line 97, in _get_token
jhub_1         |         resp = await http_client.fetch(req)
jhub_1         |     tornado.curl_httpclient.CurlError: HTTP 599: server certificate verification failed. CAfile: none CRLfile: none
jhub_1         |
jhub_1         | [E 2021-01-19 17:44:00.703 JupyterHub web:1789] Uncaught exception GET /hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJjNDgwMDAxMDg5NTQ0NmZlYjY3MmVkMDNkMTA3ZTY0YSIsICJuZXh0X3VybCI6ICIvaHViLyJ9&session_state=3817a8f6-9d0d-4495-8ce5-ca4c9abc192e&code=40819154-5408-4c67-b5be-d0032045e791.3817a8f6-9d0d-4495-8ce5-ca4c9abc192e.581c2301-1739-44bb-b398-2889b13ddd26 (::ffff:10.0.2.2)
jhub_1         |     HTTPServerRequest(protocol='https', host='jhub:8000', method='GET', uri='/hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJjNDgwMDAxMDg5NTQ0NmZlYjY3MmVkMDNkMTA3ZTY0YSIsICJuZXh0X3VybCI6ICIvaHViLyJ9&session_state=3817a8f6-9d0d-4495-8ce5-ca4c9abc192e&code=40819154-5408-4c67-b5be-d0032045e791.3817a8f6-9d0d-4495-8ce5-ca4c9abc192e.581c2301-1739-44bb-b398-2889b13ddd26', version='HTTP/1.1', remote_ip='::ffff:10.0.2.2')
jhub_1         |     Traceback (most recent call last):
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
jhub_1         |         result = await result
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 224, in get
jhub_1         |         user = await self.login_user()
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 749, in login_user
jhub_1         |         authenticated = await self.authenticate(data)
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 462, in get_authenticated_user
jhub_1         |         authenticated = await maybe_future(self.authenticate(handler, data))
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/oauthenticator/generic.py", line 155, in authenticate
jhub_1         |         token_resp_json = await self._get_token(http_client, headers, params)
jhub_1         |       File "/usr/local/lib/python3.8/dist-packages/oauthenticator/generic.py", line 97, in _get_token
jhub_1         |         resp = await http_client.fetch(req)
jhub_1         |     tornado.curl_httpclient.CurlError: HTTP 599: server certificate verification failed. CAfile: none CRLfile: none
jhub_1         |
jhub_1         | [E 2021-01-19 17:44:00.712 JupyterHub log:173] {
jhub_1         |       "X-Forwarded-Host": "jhub:8000",
jhub_1         |       "X-Forwarded-Proto": "https",
jhub_1         |       "X-Forwarded-Port": "8000",
jhub_1         |       "X-Forwarded-For": "::ffff:10.0.2.2",
jhub_1         |       "Cookie": "oauthenticator-state=[secret]",
jhub_1         |       "Accept-Language": "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7",
jhub_1         |       "Accept-Encoding": "gzip, deflate, br",
jhub_1         |       "Sec-Fetch-Dest": "document",
jhub_1         |       "Sec-Fetch-User": "?1",
jhub_1         |       "Sec-Fetch-Mode": "navigate",
jhub_1         |       "Sec-Fetch-Site": "cross-site",
jhub_1         |       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
jhub_1         |       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36",
jhub_1         |       "Upgrade-Insecure-Requests": "1",
jhub_1         |       "Connection": "close",
jhub_1         |       "Host": "jhub:8000"
jhub_1         |     }
jhub_1         | [E 2021-01-19 17:44:00.712 JupyterHub log:181] 500 GET /hub/oauth_callback?state=[secret]&session_state=[secret]&code=[secret] (@::ffff:10.0.2.2) 70.45ms

I don’t understand why I run into a tornado.curl_httpclient.CurlError: HTTP 599: server certificate verification failed. CAfile: none CRLfile: none.

I do not understand why the verification fails and why both files are none.

Setup

jupyterhub

FROM jupyterhub/jupyterhub

RUN apt-get upgrade -y

ADD jupyterhub_config.py /srv/jupyterhub/jupyterhub_config.py

RUN pip install jupyter oauthenticator

RUN openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout /srv/jupyterhub/jhubssl.key \
  -out /srv/jupyterhub/jhubssl.crt \
  -subj "/C=DE/ST=XXX/L=XXX/O=XXX/OU=XXXX/CN=XXXXX" \
  -addext "subjectAltName=DNS:localhost,DNS:jhub,DNS:jupyterhub,IP:127.0.0.1" \
  && cp /srv/jupyterhub/jhubssl.crt /usr/local/share/ca-certificates/ \
  && chmod 644 /usr/local/share/ca-certificates/jhubssl.crt \
  && dpkg-reconfigure ca-certificates \
  && update-ca-certificates --fresh

What we can see here is that a fresh certificate is created and added. With the last lines, the certificate should also be installed on the JupyterHub so that accessing its own webpages with the self-signed certificate should not be a problem. When I am inside the JupyterHub docker container, I have also executed this: openssl s_client -showcerts -connect jhub:8000 and its output starts like this:

SSL handshake has read 2429 bytes and written 376 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:

So generally a process inside the JupyterHub docker container can consume resources when accessing itself as https://jhub:8000.

docker-compose

version: "3.3"
services:
  keycloak:
    environment:
      - KEYCLOAK_USER=admin
      - KEYCLOAK_PASSWORD=secret
      - KEYCLOAK_IMPORT=/tmp/jhub-realm.json
    image: jboss/keycloak:latest
    ports:
      - 8443:8443
    volumes:
      - ./keycloak/config/jhub-realm.json:/tmp/jhub-realm.json

  jhub:
    environment:
    - OAUTH_TLS_VERIFY=0
    - OAUTH2_AUTHORIZE_URL=https://keycloak:8443/auth/realms/jhub/protocol/openid-connect/auth
    - OAUTH2_TOKEN_URL=https://keycloak:8443/auth/realms/jhub/protocol/openid-connect/token
    - OAUTH_CLIENT_ID=jhub-1
    - OAUTH_CLIENT_SECRET=*****************
    - OAUTH_CALLBACK_URL=https://jhub:8000/hub/oauth_callback
    - OAUTH2_USERDATA_URL=https://keycloak:8443/auth/realms/jhub/protocol/openid-connect/userinfo
    build: ./jupyterhub/jupyterhub-server
    hostname: jhub
    ports:
      - "8000:8000"

Does somebody have an idea why I run into the CurlError? This is only supposed to be a local setup, for the production system of course proper certificates are used. Therefore, here I only want a solution that somehow works. Since it only runs on localhost, I am open to solutions that would be insecure in a production environment. Thank you very much!

2

I’ve previously read about problems occuring with pycurl and certificates. I can’t find the reference I’m thinking of, but it’s something to do with whether pycurl respects the system or custom certificate stores. I’ve found these links which might provide some background information:

It could be completely unrelated issue to those, but if no-one else has a suggestion that might be something to investigate?

1 Like
3

Thank you very much, @manics !

I extended the jupyterhub Dockerfile following Fan+: fix: tornado HTTP 599 issue / gnutls_handshake() failed like this:

ENV PYCURL_SSL_LIBRARY=openssl
RUN apt-get -y install python3-dev gcc curl libcurl3-openssl-dev \
  && pip install --no-input --ignore-installed --force-reinstall pycurl

Now the error message has at least changed to this:

jhub_1         |     tornado.curl_httpclient.CurlError: HTTP 599: SSL certificate problem: self signed certificate

With the help of the other links soon a self-signed certificate should also be accepted, so this was an important step into the right direction.

1 Like
4

The second problem was that I took this configuration from somewhere else:

But the environment variable checked in the generic oauth authenticator has a “2”.

OAUTH2_TLS_VERIFY=0

With this version, now all curl-related problems were solved. The config I shared above does not instantly work because of the missing local users. But that is a totally different problem.

1 Like
5

For documentation, I am leaving the adjusted files here. Now the small example is working!

First, keycloak and jhub need to be aliases for localhost in the hosts file.

jupyterhub config:

from oauthenticator.generic import LocalGenericOAuthenticator
c.JupyterHub.authenticator_class = LocalGenericOAuthenticator
c.GenericOAuthenticator.login_service = 'local keycloak'
c.GenericOAuthenticator.userdata_params = {"state": "state"}
c.GenericOAuthenticator.auto_login = False
c.GenericOAuthenticator.scope = ['openid']
c.GenericOAuthenticator.username_key = "preferred_username"
c.Authenticator.auto_login = False
c.GenericOAuthenticator.client_id = 'jhub-1'
c.JupyterHub.ssl_cert = 'jhubssl.crt'
c.JupyterHub.ssl_key = 'jhubssl.key'
c.LocalAuthenticator.create_system_users = True

docker-compose.yml

version: "3.3"
services:
  keycloak:
    environment:
      - KEYCLOAK_USER=admin
      - KEYCLOAK_PASSWORD=admin
    image: jboss/keycloak:latest
    ports:
      - 8443:8443
  jhub:
    environment:
    - OAUTH2_TLS_VERIFY=0
    - OAUTH2_AUTHORIZE_URL=https://keycloak:8443/auth/realms/jhub/protocol/openid-connect/auth
    - OAUTH2_TOKEN_URL=https://keycloak:8443/auth/realms/jhub/protocol/openid-connect/token
    - OAUTH_CLIENT_ID=jhub-1
    - OAUTH_CLIENT_SECRET=*****************
    - OAUTH_CALLBACK_URL=https://jhub:8000/hub/oauth_callback
    - OAUTH2_USERDATA_URL=https://keycloak:8443/auth/realms/jhub/protocol/openid-connect/userinfo
    build: ./jupyterhub/jupyterhub-server
    hostname: jhub
    ports:
      - "8000:8000"

jupyterhub Dockerfile

FROM jupyterhub/jupyterhub

RUN apt-get update && apt-get upgrade -y

ADD jupyterhub_config.py /srv/jupyterhub/jupyterhub_config.py

RUN pip install jupyter oauthenticator

RUN openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout /srv/jupyterhub/jhubssl.key \
  -out /srv/jupyterhub/jhubssl.crt \
  -subj "/C=DE/ST=xx/L=xx/O=TUHH/OU=xx/CN=xxxx Self-Signed" \
  -addext "subjectAltName=DNS:localhost,DNS:jhub,DNS:jupyterhub,IP:127.0.0.1" \
  && cp /srv/jupyterhub/jhubssl.crt /usr/local/share/ca-certificates/ \
  && chmod 644 /usr/local/share/ca-certificates/jhubssl.crt \
  && dpkg-reconfigure ca-certificates \
  && update-ca-certificates --fresh

ENV PYCURL_SSL_LIBRARY=openssl
RUN apt-get -y install python3-dev gcc curl libcurl3-openssl-dev \
  && pip install --no-input --ignore-installed --force-reinstall pycurl
1 Like
6

Some more experimenting/attempt to get self-signed SSL certs and tornado clients straight is in GitHub - betatim/self-signed-ssl-certs: Experimenting with self signed certificates and Tornado.

7

Thank you very much for this addition.

1 Like
8

So we are using Helm chart to deploy jupyterhub and integrate with keycloak. How we should make it work? tornado.curl error with this certificates are killing us, how to solve it

9

@Kamil_Farizov please present your case including your configuration and log files etc. in detail in a separate thread. Your issue might be related to mine but it is not guaranteed.