Internal-ssl with kubespawner

JupyterHub
1

i installed jupyterhub 5.4 on docker i want my hub to spawn notebooks on a remote kubernetes cluster. i am getting the below error when spawning (pod, service is created)

2025-10-24T18:02:22.251Z [ConfigProxy] debug: PROXY WEB /user/xxx/ to https://jupyter-xxx.local:443
2025-10-24T18:02:22.301Z [ConfigProxy] error: 503 GET /user/xxx/ Error: self-signed certificate
at TLSSocket.onConnectSecure (node:_tls_wrap:1659:34)
at TLSSocket.emit (node:events:517:28)
at TLSSocket._finishInit (node:_tls_wrap:1070:8)
at ssl.onhandshakedone (node:_tls_wrap:856:12) {
code: ‘DEPTH_ZERO_SELF_SIGNED_CERT’
}
2025-10-24T18:02:22.304Z [ConfigProxy] debug: Requesting custom error page: https://jupyterhub:8081/hub/error/503?url=%2Fuser%2Fxxx%2F
[W 2025-10-24 18:02:22.325 JupyterHub iostream:1378] SSL Error on 19 (‘172.18.0.2’, 58928): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:1000)
2025-10-24T18:02:22.325Z [ConfigProxy] error: Failed to get custom error page: [Error: 0008346B0A760000:error:0A00045C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:../ssl/record/rec_layer_s3.c:1599:SSL alert number 116
] {
library: ‘SSL routines’,
reason: ‘tlsv13 alert certificate required’,
code: ‘ERR_SSL_TLSV13_ALERT_CERTIFICATE_REQUIRED’
}

currently the ingress is created manually and i plan on automatically create the ingress post pod creation.

also everything works fine if i turn off internal-ssl

any idea how to solve this issue i am facing

2

can you share logs from the user container?

3

and the hub pod, as well?

4

i figured out that the issue is the ingress/nginx. when CHP send the request to the ingress it return 404 with its certificate (that explain why i am getting self-signed certificate error)

i tried to curl to the ingress and it went fine (returning 302 from the notebook)

this is my ingress yaml file

apiVersion: v1
items:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/ssl-passthrough: “true”
creationTimestamp: “2025-10-24T08:16:23Z”
generation: 16
name: jupyter-bbb
namespace: jupyterhub
resourceVersion: “231546698”
uid: e21e02a7-2b72-4c60-bc34-108ea7e4389d
spec:
ingressClassName: nginx
rules:

host: jupyter-bbb.local
http:
paths:

backend:
service:
name: jupyter-bbb
port:
number: 8888
path: /
pathType: Prefix

this is my kubespawner config

c.KubeSpawner.container_security_context = {
‘runAsUser’ : 0
}
c.KubeSpawner.ssl_alt_names = [‘DNS:jupyter-bbb.local’]
c.KubeSpawner.cmd = ‘start-singleuser.py’
c.KubeSpawner.pod_connect_ip = ‘jupyter-bbb.local’
c.KubeSpawner.hub_connect_url = “https://192.168.200.130:8081”
c.KubeSpawner.port = 8888
c.KubeSpawner.image = ‘quay.io/jupyter/datascience-notebook:hub-5.4.0’
c.KubeSpawner.image_pull_policy = ‘IfNotPresent’
c.KubeSpawner.extra_pod_config = {
}
c.KubeSpawner.namespace  = ‘jupyterhub’
c.KubeSpawner.environment = {
“CHOWN_HOME”: “yes”,
“CHOWN_EXTRA”: “/home/jovyan”,
“CHOWN_HOME_OPTS”: “-R”,
“NB_UID”: “3000”,
“NB_GID”: “1000”

}
c.KubeSpawner.cpu_limit = 1
c.KubeSpawner.mem_limit = ‘1G’
c.KubeSpawner.services_enabled = True
#c.KubeSpawner.cmd = ‘jupyterhub-singleuser’
c.KubeSpawner.args = [‘–ip=0.0.0.0’, ‘–NotebookApp.allow_origin=*’]
def pod_url(spawner,pod):
import time
time.sleep(1)
url = https://jupyter-bbb.local:443
spawner.log.info(f"---- returning {url}")
return url

c.KubeSpawner.get_pod_url = pod_url

i modified configproxy.js file to show more logs and here is the logs

CUSTOM-DEBUG: parseHost(req) = 192.168.200.130
CUSTOM-DEBUG: route lookup path = /user/bbb/
CUSTOM-DEBUG: found route target = undefined
2025-10-25T21:00:52.193Z [ConfigProxy] debug: PROXY WEB /user/bbb/ to ``https://jupyter-bbb.local:443
CUSTOM1:HTTPS proxy options for target: ``https://jupyter-bbb.local/
CUSTOM1:Key: true
CUSTOM1:Cert: true
CUSTOM1:CA certs loaded: 2
CUSTOM1:SNI hostname sent: jupyter-bbb.local
CUSTOM2: REMOTE CERT SUBJECT: [Object: null prototype] {
O: ‘Acme Co’,
CN: ‘Kubernetes Ingress Controller Fake Certificate’
}
CUSTOM2: REMOTE CERT ISSUER: [Object: null prototype] {
O: ‘Acme Co’,
CN: ‘Kubernetes Ingress Controller Fake Certificate’
}
CUSTOM2: REMOTE CERT SHA256 FINGERPRINT: 47397ff780085abd091623ced39e121fa069d9dd44d7df3892fdd6648b918ee0
CUSTOM2: REQUEST METHOD: GET
CUSTOM2: REQUEST HEADERS: [Object: null prototype] {
host: ‘192.168.200.130:8000’,
connection: ‘keep-alive’,
pragma: ‘no-cache’,
‘cache-control’: ‘no-cache’,
‘upgrade-insecure-requests’: ‘1’,
‘user-agent’: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36’,
accept: ‘text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7’,
referer: ‘‘
‘accept-encoding’: ‘gzip, deflate’,
‘accept-language’: ‘en-US,en;q=0.9,ar-SA;q=0.8,ar;q=0.7’,
cookie: ‘jupyterhub-user-bbb=2|1:0|10:1761317322|19:jupyterhub-user-bbb|40:TGp0Q2dzVW9LTGlNMVg2cTZadVN5REJCRER3eEpW|6d4d84f5c9a4212aa38fe75d2e2f1d2ae07945af4384c39c21f2fc50b01f55af; _xsrf=MnwxOjB8MTA6MTc2MTMxNzMyMnw1Ol94c3JmfDEzMjpaREkxTUdFek5tSXlNREEyTkRNeU9XRTFPV1UxWkdJeE1EVmxNV015WXpZNlpUVmhPV05qTkRNMU56QmxOelZtT0RjMk9USmxZV0ZqWmpjMU16ZzNaR0k0Tm1Vek5HUmlPREUxWXpVek16ZGtObVZrTkRKbVptUXhOR0psWXpGa05nPT18ZGU3NmVlNWYyMjc0Njg2YzI3MjgxZDkzNzg1NDFlNzY3MDdlYTY1NjQ4MjE1Y2UwMjY4MDUzN2M3NzUzNjc2Ng; jupyterhub-session-id=d250a36b20064329a59e5db105e1c2c6’,
‘x-forwarded-for’: ‘::ffff:192.168.200.18’,
‘x-forwarded-port’: ‘8000’,
‘x-forwarded-proto’: ‘http’,
‘x-forwarded-host’: ‘192.168.200.130:8000’
}
2025-10-25T21:00:52.263Z [ConfigProxy] debug: Not recording activity for status 404 on /user/bbb

5

after investigating i found that chp is not sending the correct SNI so nginx/ingress can do the routing. i added

if (proxyOptions.secure) {
// Derive hostname from the target URL if not already set
const hostnameForSNI = target.hostname || target.host;
if (hostnameForSNI && !proxyOptions.target.servername) {
proxyOptions.target.servername = hostnameForSNI;
}

  // Ensure Node's TLS stack uses it
  if (proxyOptions.agent && proxyOptions.agent.options) {
    proxyOptions.agent.options.servername = hostnameForSNI;
  }
              console.log("CUSTOM1:SNI forced to:", proxyOptions.target.servername);
}

to configproxy.js and now it is working.

6

it seems the target must be wrong, because the ingress controller shouldn’t be involved in the chp->singleuser connection. That should be through a ClusterIP Service connection directly to the pod, I think. Usually this URL would be jupyter-bbb.svc.cluster.local, not jupyter-bbb.local, although maybe that would work? I think the port should also be 8888, not 443.

You can try inspecting the service with:

kubectl get svc -o yaml jupyter-bbb

while the server is running