Suddenly getting oath/cert error?

JupyterHub Zero to JupyterHub on Kubernetes
1

I’ve got no idea how it would be related, but this happened after I changed my hub.db.type from mysql to postgres.

We authenticate users via Azure SSO, which was working just before and is working on other instances. Somehow it only managed to break with a DB change?

Making the changes, deploying the chart, and presented with the following error in hub logs.

[I 2024-03-01 16:05:45.752 JupyterHub log:191] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:{ip}) 0.74ms
[I 2024-03-01 16:05:45.948 JupyterHub log:191] 302 GET /hub/login?next=%2Fhub%2F -> /hub/oauth_login?next=%2Fhub%2F (@::ffff:{ip}) 0.93ms
[I 2024-03-01 16:05:46.163 JupyterHub oauth2:97] OAuth redirect: {url}
[D 2024-03-01 16:05:46.164 JupyterHub base:587] Setting cookie oauthenticator-state: {'httponly': True, 'expires_days': 1}
[I 2024-03-01 16:05:46.165 JupyterHub log:191] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://login.microsoftonline.com/{secret}/oauth2/authorize?response_type=code&redirect_uri={url}%2Fhub%2Foauth_callback&client_id={id}&state=[secret]&scope=openid+profile+email (@::ffff:{ip}) 1.90ms
[D 2024-03-01 16:05:47.408 JupyterHub log:191] 200 GET /hub/health (@{ip}) 0.61ms
[E 2024-03-01 16:05:47.900 JupyterHub oauth2:626] Error fetching 599 POST https://login.microsoftonline.com/{ip}/oauth2/token: HTTP 599: error setting certificate verify locations:
      CAfile: /etc/pki/tls/certs/ca-bundle.crt
      CApath: none
[E 2024-03-01 16:05:47.900 JupyterHub web:1875] Uncaught exception GET /hub/oauth_callback?code={code} (::ffff:{ip})
    HTTPServerRequest(protocol='http', host='{url}', method='GET', uri='/hub/oauth_callback?code={code}&state={session}&session_state={state}', version='HTTP/1.1', remote_ip='::ffff:{ip}')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/tornado/web.py", line 1790, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 208, in get
        user = await self.login_user()
               ^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 828, in login_user
        authenticated = await self.authenticate(data)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/auth.py", line 492, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 969, in authenticate
        token_info = await self.get_token_info(handler, access_token_params)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 843, in get_token_info
        token_info = await self.httpfetch(
                     ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 661, in httpfetch
        return await self.fetch(
               ^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 627, in fetch
        raise e
      File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 606, in fetch
        resp = await self.http_client.fetch(req, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    tornado.curl_httpclient.CurlError: HTTP 599: error setting certificate verify locations:
      CAfile: /etc/pki/tls/certs/ca-bundle.crt
      CApath: none
[D 2024-03-01 16:05:47.909 JupyterHub base:1371] No template for 500
[E 2024-03-01 16:05:47.934 JupyterHub log:183] {
      "X-Forwarded-Host": "{url}",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "80",
      "X-Forwarded-For": "::ffff:{ip}",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Site": "none",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Upgrade-Insecure-Requests": "1",
      "Cookie": "_xsrf=[secret]; oauthenticator-state=[secret]",
      "Connection": "keep-alive",
      "Dnt": "1",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0",
      "Host": "{url}"
    }
[E 2024-03-01 16:05:47.935 JupyterHub log:191] 500 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] (@::ffff:{ip}) 44.73ms
[D 2024-03-01 16:05:48.566 JupyterHub log:191] 200 GET /hub/static/favicon.ico?v={v}(@::ffff:{ip}) 1.12ms

Any thoughts on how to correct this? I’m at a complete loss here.

It may be important to note that the DB was able to authenticate, and tables were created successfully, even some values populated under roles. Again, I’m not sure how it would be related to the db switch, yet somehow it’s shown up with that’s all that’s changed.

2

I suspect it’s a coincidence, I’ve seen similar errors in other projects recently. I haven’t figured out what’s causing it yet.

3

@manics - that’s unfortunate. I assume since there’s no cause, there’s been no workaround either?

Luckily, this hasn’t happened to our production instance, but I’m freezing all updates in the case that a push triggers it.

Is there anywhere this issue is tracked so I can follow/assist?

4

Hey @manics - did some digging and found a work around.

I ended up finding this issue on the oauth github.

He found that downgrading pycurl from 7.45.3 → 7.45.2 resolved is issue.

I changed my jupyterhub’s requirements.txt to include pycurl==7.45.2, and suddenly I’m not presented with the error and users can login again.

While this is a workaround, and not a resolution, I won’t mark this issue as solved - but hopefully this will help others who run into it, as well as help identify the problem and find a resolution.

1 Like
5

Nice work! Thanks for investigating.

It looks like pycurl changed from releasing a source package only (which would be built on your system when you install it) in 7.45.2

to releasing pre-built binary wheels in 7.45.3
Maybe the wheels aren’t picking up the system CA certificates?

pip install -r requirements.txt --no-binary pycurl should force pycurl to be built and installed from source instead.

1 Like
6

Someone’s already reported the issue on the pycurl repo:

As well as forcing a source install there’s an alternative workaround involving symlinking certificates

2 Likes
7

I ran into this now in z2jh 3.3.0 tested in a staging environment =/

Thanks for looking into this already @pastram-i and @manics!!