Internal-ssl with kubespawner

JupyterHub
1

i installed jupyterhub 5.4 on docker i want my hub to spawn notebooks on a remote kubernetes cluster. i am getting the below error when spawning (pod, service is created)

2025-10-24T18:02:22.251Z \[ConfigProxy\] debug: PROXY WEB /user/xxx/ to https://jupyter-xxx.local:443
2025-10-24T18:02:22.301Z \[ConfigProxy\] error: 503 GET /user/xxx/ Error: self-signed certificate
at TLSSocket.onConnectSecure (node:\_tls_wrap:1659:34)
at TLSSocket.emit (node:events:517:28)
at TLSSocket.\_finishInit (node:\_tls_wrap:1070:8)
at ssl.onhandshakedone (node:\_tls_wrap:856:12) {
code: ‘DEPTH_ZERO_SELF_SIGNED_CERT’
}
2025-10-24T18:02:22.304Z \[ConfigProxy\] debug: Requesting custom error page: https://jupyterhub:8081/hub/error/503?url=%2Fuser%2Fxxx%2F
\[W 2025-10-24 18:02:22.325 JupyterHub iostream:1378\] SSL Error on 19 (‘172.18.0.2’, 58928): \[SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE\] peer did not return a certificate (\_ssl.c:1000)
2025-10-24T18:02:22.325Z \[ConfigProxy\] error: Failed to get custom error page: \[Error: 0008346B0A760000:error:0A00045C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:../ssl/record/rec_layer_s3.c:1599:SSL alert number 116
\] {
library: ‘SSL routines’,
reason: ‘tlsv13 alert certificate required’,
code: ‘ERR_SSL_TLSV13_ALERT_CERTIFICATE_REQUIRED’
}

currently the ingress is created manually and i plan on automatically create the ingress post pod creation.

also everything works fine if i turn off internal-ssl

any idea how to solve this issue i am facing

2

can you share logs from the user container?

3

and the hub pod, as well?

4

i figured out that the issue is the ingress/nginx. when CHP send the request to the ingress it return 404 with its certificate (that explain why i am getting self-signed certificate error)

i tried to curl to the ingress and it went fine (returning 302 from the notebook)

this is my ingress yaml file

apiVersion: v1
items:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/ssl-passthrough: “true”
creationTimestamp: “2025-10-24T08:16:23Z”
generation: 16
name: jupyter-bbb
namespace: jupyterhub
resourceVersion: “231546698”
uid: e21e02a7-2b72-4c60-bc34-108ea7e4389d
spec:
ingressClassName: nginx
rules:

host: jupyter-bbb.local
http:
paths:

backend:
service:
name: jupyter-bbb
port:
number: 8888
path: /
pathType: Prefix

this is my kubespawner config

c.KubeSpawner.container_security_context = {
‘runAsUser’ : 0
}
c.KubeSpawner.ssl_alt_names = [‘DNS:jupyter-bbb.local’]
c.KubeSpawner.cmd = ‘start-singleuser.py’
c.KubeSpawner.pod_connect_ip = ‘jupyter-bbb.local’
c.KubeSpawner.hub_connect_url = “https://192.168.200.130:8081”
c.KubeSpawner.port = 8888
c.KubeSpawner.image = ‘quay.io/jupyter/datascience-notebook:hub-5.4.0’
c.KubeSpawner.image_pull_policy = ‘IfNotPresent’
c.KubeSpawner.extra_pod_config = {
}
c.KubeSpawner.namespace  = ‘jupyterhub’
c.KubeSpawner.environment = {
“CHOWN_HOME”: “yes”,
“CHOWN_EXTRA”: “/home/jovyan”,
“CHOWN_HOME_OPTS”: “-R”,
“NB_UID”: “3000”,
“NB_GID”: “1000”

}
c.KubeSpawner.cpu_limit = 1
c.KubeSpawner.mem_limit = ‘1G’
c.KubeSpawner.services_enabled = True
#c.KubeSpawner.cmd = ‘jupyterhub-singleuser’
c.KubeSpawner.args = [‘–ip=0.0.0.0’, ‘–NotebookApp.allow_origin=*’]
def pod_url(spawner,pod):
import time
time.sleep(1)
url = https://jupyter-bbb.local:443
spawner.log.info(f"---- returning {url}")
return url

c.KubeSpawner.get_pod_url = pod_url

i modified configproxy.js file to show more logs and here is the logs

`CUSTOM-DEBUG: parseHost(req) = 192.168.200.130`
`CUSTOM-DEBUG: route lookup path = /user/bbb/`
`CUSTOM-DEBUG: found route target = undefined`
`2025-10-25T21:00:52.193Z [ConfigProxy] debug: PROXY WEB /user/bbb/ to ``https://jupyter-bbb.local:443`
`CUSTOM1:HTTPS proxy options for target: ``https://jupyter-bbb.local/`
`CUSTOM1:Key: true`
`CUSTOM1:Cert: true`
`CUSTOM1:CA certs loaded: 2`
`CUSTOM1:SNI hostname sent: jupyter-bbb.local`
`CUSTOM2: REMOTE CERT SUBJECT: [Object: null prototype] {`
`O: ‘Acme Co’,`
`CN: ‘Kubernetes Ingress Controller Fake Certificate’`
`}`
`CUSTOM2: REMOTE CERT ISSUER: [Object: null prototype] {`
`O: ‘Acme Co’,`
`CN: ‘Kubernetes Ingress Controller Fake Certificate’`
`}`
`CUSTOM2: REMOTE CERT SHA256 FINGERPRINT: 47397ff780085abd091623ced39e121fa069d9dd44d7df3892fdd6648b918ee0`
`CUSTOM2: REQUEST METHOD: GET`
`CUSTOM2: REQUEST HEADERS: [Object: null prototype] {`
`host: ‘192.168.200.130:8000’,`
`connection: ‘keep-alive’,`
`pragma: ‘no-cache’,`
`‘cache-control’: ‘no-cache’,`
`‘upgrade-insecure-requests’: ‘1’,`
`‘user-agent’: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36’,`
`accept: ‘text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7’,`
`referer: ‘‘`
`‘accept-encoding’: ‘gzip, deflate’,`
`‘accept-language’: ‘en-US,en;q=0.9,ar-SA;q=0.8,ar;q=0.7’,`
`cookie: ‘jupyterhub-user-bbb=2|1:0|10:1761317322|19:jupyterhub-user-bbb|40:TGp0Q2dzVW9LTGlNMVg2cTZadVN5REJCRER3eEpW|6d4d84f5c9a4212aa38fe75d2e2f1d2ae07945af4384c39c21f2fc50b01f55af; _xsrf=MnwxOjB8MTA6MTc2MTMxNzMyMnw1Ol94c3JmfDEzMjpaREkxTUdFek5tSXlNREEyTkRNeU9XRTFPV1UxWkdJeE1EVmxNV015WXpZNlpUVmhPV05qTkRNMU56QmxOelZtT0RjMk9USmxZV0ZqWmpjMU16ZzNaR0k0Tm1Vek5HUmlPREUxWXpVek16ZGtObVZrTkRKbVptUXhOR0psWXpGa05nPT18ZGU3NmVlNWYyMjc0Njg2YzI3MjgxZDkzNzg1NDFlNzY3MDdlYTY1NjQ4MjE1Y2UwMjY4MDUzN2M3NzUzNjc2Ng; jupyterhub-session-id=d250a36b20064329a59e5db105e1c2c6’,`
`‘x-forwarded-for’: ‘::ffff:192.168.200.18’,`
`‘x-forwarded-port’: ‘8000’,`
`‘x-forwarded-proto’: ‘http’,`
`‘x-forwarded-host’: ‘192.168.200.130:8000’`
`}`
`2025-10-25T21:00:52.263Z [ConfigProxy] debug: Not recording activity for status 404 on /user/bbb`
5

after investigating i found that chp is not sending the correct SNI so nginx/ingress can do the routing. i added

if (proxyOptions.secure) {
// Derive hostname from the target URL if not already set
const hostnameForSNI = target.hostname || target.host;
if (hostnameForSNI && !proxyOptions.target.servername) {
proxyOptions.target.servername = hostnameForSNI;
}

  // Ensure Node's TLS stack uses it
  if (proxyOptions.agent && proxyOptions.agent.options) {
    proxyOptions.agent.options.servername = hostnameForSNI;
  }
              console.log("CUSTOM1:SNI forced to:", proxyOptions.target.servername);
}

to configproxy.js and now it is working.

6

it seems the target must be wrong, because the ingress controller shouldn’t be involved in the chp->singleuser connection. That should be through a ClusterIP Service connection directly to the pod, I think. Usually this URL would be jupyter-bbb.svc.cluster.local, not jupyter-bbb.local, although maybe that would work? I think the port should also be 8888, not 443.

You can try inspecting the service with:

kubectl get svc -o yaml jupyter-bbb

while the server is running

7

the jupyterhub is running on docker in a different VM, by default the svc will be ClusterIP so ingress is the only way in.

8

i also had to override the spawner url to match the ingress url using c.KubeSpawner.get_pod_url

9

ah, ok. I missed the SNI proxy part of the ingress, I’ve never encountered a deployment like that.

Looks like it might be documented nodejs behavior where SNI is not passed by default sometimes. I wonder why… though it should be enabled by default for https, if not for tls. But maybe tls is used instead of https? That seems odd.

What version of CHP are you using and what version of nodejs is CHP running with?

If this is reproducible on CHP 5.1.0 on nodejs 24, a patch to CHP seems in order, but we also need to be careful that we aren’t setting per-request options on a stateful object (proxyOptions.agent is a shared resource).