Hub ssl verify error

JupyterHub
1

Hello, folks.

I bumped into the issue with self-signed certificates in Kubernetes.

My API server in k8s use self-signed cert and when I try to access jupyterhub it throws:

500 : Internal Server Error
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1123)

In jupiterhub logs I can see:

WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=Non

But there is no any explanation about how to use self signed certificates.

How to force urllib3 to use CA cert which is located in containers FS?

2

Hi! Welcome to the forum.

Please could you provide more information on what the problem is, and how your systems are setup?

For example, you haven’t said where you’re seeing that error. E.g. Is it in the JupyterHub logs, proxy logs, K8s, singleuser server, or somewhere else? Ideally please also tell us how your K8s cluster was setup in as much detail as possible, the versions of your JupyterHub components, and include your JupyterHub configuration file with secrets redacted. Thanks!

3

Sorry, haven’t noticed that jupyterhub logs have been omitted by the forum.

The problem is that when the Kubernetes plugin in python sends requests to K8S API, it doesn’t trust the certificate which is used by the K8S API server. This certificate was created by CA in K8S with the self-signed certificate.

If i will try within container to send request to the K8S API server, i will have this issue:
http.request(“GET”, “https://10.96.0.1”)
Result:
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

If I will use CA cert from K8S with these requests, everything smooth:

**curl https://10.96.0.1 --cacert** /usr/local/share/ca-certificates/rots.crt/ca.crt
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403

So how to push CA cert into the container ?

I use the latest versions of jupyterhub with default configuration except:

ingress:
      enabled: true

K8S is a managed service, I do not know how it was configured, and i have no access to the master node.

Logs from jupyterhub:

    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/jupiter/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1123)')))
4

Tried to use CA_BUNDLE as env - doesn’t work.

Tried to put CA certificate under /usr/local/share/ca-certificates/ca.crt
Then ca-update-certificates - now curl works, but urllib3 still throw SSL verify error :confused:

How to disable SSL verify? I do not need this function in my environment.

5

Hello, Folks.

Could you help me with this ?

how to disable SSL verify for k8s interactions?

6

Any updates on this one? How does one disable SSL certificate verification for Jupyter Notebook?

7 
extraEnv: {
    OAUTH2_TLS_VERIFY: "0"
}

This is a workaround. I was using Helm, just modified the values file.