Certificate_verify_failed

JupyterHub
1

Dear all,
I’ve a working installation of Jupyterhub 2.X for the last 2 years. I was now trying to update the software stack (Python 3.9 → 3.11 and Jupyterhub from 2.X to 3.X) and I’m facing a weird issue related to CA certificates. My old installation still works but this new one throws an error.

Python 3.11
Jupyterhub 3.1.0
oauthenticator 15.1.0
OpenSSL 1.1.1k

I start the jupyterhub without problem and when I click on “login with AzureID” the logs show the error message:

   [W 2022-12-23 13:38:26.509 JupyterHub iostream:1404] SSL Error on 12 ('40.126.32.138', 443): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)
   [E 2022-12-23 13:38:26.510 JupyterHub web:1798] Uncaught exception GET /hub/oauth_callback?code=0.AS
   [...]
   File "/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.11.1-GCCcore-10.3.0-bare/lib/python3.11/ssl.py", line 1346, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)

The IP 40.126.32.138 belongs to microsoft azure service,

My installation is behind a corporate proxy (Zscaler) and I cannot directly talk to the microsoft server:

 # openssl s_client -connect 40.126.32.138:443
 CONNECTED(00000003)
 write:errno=104
 ---
 no peer certificate available
  ---
 No client certificate CA names sent

so openssl needs the proxy parameter to be able to get the certificate:

  # openssl s_client -connect 40.126.32.138:443 -proxy <private-proxy>:80
  CONNECTED(00000003)
  depth=1 C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
  verify error:num=20:unable to get local issuer certificate

So I have the feeling that jupyterhub is having the same probem, when it connects to the microsfot IP it cannot read the cert and this is why it gives me an error (this is my guess).

So, my question is how I can tell jupyterhub (or the AzureAdOAuthenticator) to either use the proxy or ignore the certificate check.

TIA

2

Found the solution.
Thanks to Configuring web-requests to use a proxy · Issue #217 · jupyterhub/oauthenticator (github.com)

pip uninstall pycurl
export PYCURL_SSL_LIBRARY=nss
pip install pycurl

and in jupyterhub_config.py:

from tornado.httpclient import AsyncHTTPClient
AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClien