Kubernetes Helm Deployment Unable to Connect to Server Using IPv6

Deeco · February 4, 2025, 5:48pm

services below

NAME           TYPE        CLUSTER-IP             EXTERNAL-IP   PORT(S)    AGE
hub            ClusterIP   fd5f:c06e:e955::8ef7   <none>        8081/TCP   6h46m
proxy-api      ClusterIP   fd5f:c06e:e955::89fe   <none>        8001/TCP   6h46m
proxy-public   ClusterIP   fd5f:c06e:e955::4a6d   <none>        80/TCP     6h46m

manics · February 4, 2025, 6:00pm

Can you show the ConfigProxy logs for when you attempt to connect to the singleuser server too? You can redact any external IP addresses if necessary.

Deeco · February 4, 2025, 9:39pm

Those are all the logs I see from proxy , hub and singleserver . Is there somewhere else ?

Deeco · February 5, 2025, 2:02pm

Is there any other log can share ? Singleserver logs of any use ? Seems to be the hub and the path to singleserver response/config why I updated the hub.baseUrl from default /

manics · February 5, 2025, 11:10pm

All user connections should be via the proxy, for example:

23:04:49.716 [ConfigProxy] e[32minfoe[39m: Adding route / -> http://hub:8081
23:04:49.742 [ConfigProxy] e[32minfoe[39m: Proxying http://*:8000 to http://hub:8081
23:04:49.745 [ConfigProxy] e[32minfoe[39m: Proxy API at http://*:8001/api/routes
23:04:49.765 [ConfigProxy] e[32minfoe[39m: Route added / -> http://hub:8081
23:05:23.207 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:05:23.214 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:05:23.230 [ConfigProxy] e[32minfoe[39m: Adding route / -> http://hub:8081
23:05:23.231 [ConfigProxy] e[32minfoe[39m: Route added / -> http://hub:8081
23:05:23.232 [ConfigProxy] e[32minfoe[39m: 201 POST /api/routes/ 
23:06:23.242 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:06:33.775 [ConfigProxy] e[34mdebuge[39m: PROXY WEB / to http://hub:8081
23:06:33.841 [ConfigProxy] e[34mdebuge[39m: Not recording activity for status 302 on /
23:06:33.895 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/ to http://hub:8081
23:06:33.902 [ConfigProxy] e[34mdebuge[39m: Not recording activity for status 302 on /
23:06:33.938 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/login to http://hub:8081
23:06:34.117 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/static/css/style.min.css to http://hub:8081
23:06:34.128 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/static/components/bootstrap/dist/js/bootstrap.bundle.min.js to http://hub:8081
23:06:34.132 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/static/components/jquery/dist/jquery.min.js to http://hub:8081
23:06:34.141 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/static/js/darkmode.js to http://hub:8081
23:06:34.143 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/static/components/requirejs/require.js to http://hub:8081
23:06:34.152 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/logo to http://hub:8081
23:06:34.282 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/static/favicon.ico to http://hub:8081
23:06:34.292 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 to http://hub:8081
23:06:35.470 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/login to http://hub:8081
23:06:35.509 [ConfigProxy] e[34mdebuge[39m: Not recording activity for status 302 on /
23:06:35.555 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/ to http://hub:8081
23:06:35.580 [ConfigProxy] e[34mdebuge[39m: Not recording activity for status 302 on /
23:06:35.633 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/spawn to http://hub:8081
23:06:35.682 [ConfigProxy] e[34mdebuge[39m: Not recording activity for status 302 on /
23:06:35.716 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/spawn-pending/test to http://hub:8081
23:06:35.915 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/api/users/test/server/progress to http://hub:8081
23:06:45.454 [ConfigProxy] e[32minfoe[39m: Adding route /user/test -> http://192.168.0.13:8888
23:06:45.455 [ConfigProxy] e[32minfoe[39m: Route added /user/test -> http://192.168.0.13:8888
23:06:45.456 [ConfigProxy] e[32minfoe[39m: 201 POST /api/routes/user/test 
23:06:45.518 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/spawn-pending/test to http://hub:8081
23:06:45.532 [ConfigProxy] e[34mdebuge[39m: Not recording activity for status 302 on /
23:06:45.573 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /user/test/ to http://192.168.0.13:8888
23:06:45.585 [ConfigProxy] e[34mdebuge[39m: Not recording activity for status 302 on /user/test
23:06:45.626 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /user/test/lab to http://192.168.0.13:8888
23:06:45.632 [ConfigProxy] e[34mdebuge[39m: Not recording activity for status 302 on /user/test
23:06:45.671 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /hub/api/oauth2/authorize to http://hub:8081
23:06:45.704 [ConfigProxy] e[34mdebuge[39m: Not recording activity for status 302 on /
23:06:45.742 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /user/test/oauth_callback to http://192.168.0.13:8888
23:06:45.841 [ConfigProxy] e[34mdebuge[39m: Not recording activity for status 302 on /user/test
23:06:45.927 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /user/test/lab to http://192.168.0.13:8888
23:06:46.089 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /user/test/static/lab/main.ea7277c9c015ae398545.js to http://192.168.0.13:8888
23:06:46.143 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /user/test/lab/extensions/jupyterlab_pygments/static/remoteEntry.5cbb9d2323598fbda535.js to http://192.168.0.13:8888
23:06:46.187 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /user/test/static/lab/jlab_core.9ae09eead5139c587af4.js to http://192.168.0.13:8888
23:06:46.191 [ConfigProxy] e[34mdebuge[39m: PROXY WEB /user/test/static/lab/1096.dd4c563e0483cbbeb9c9.js to http://192.168.0.13:8888

So it looks like you’re bypassing the proxy. Can you describe all the infrastructure (inside and outside K8s) between your browser and JupyterHub?

Deeco · February 7, 2025, 12:51am

eks ipv6 internal cluster , behind internal proxy and vpn,
istio virtual gateway service used to redirect internal addresses to AWS ELB
Helm Deployment with values.yaml to specific namespace in eks ipv6 cluster.

tried also bypass istio/ingress by using but same error

k port-forward svc/hub 8081:8081

full logs when server started from proxy

23:27:34.408 [ConfigProxy] e[32minfoe[39m: Adding route / -> http://hub:8081
23:27:34.411 [ConfigProxy] e[32minfoe[39m: Proxying http://*:8000 to http://hub:8081
23:27:34.411 [ConfigProxy] e[32minfoe[39m: Proxy API at http://*:8001/api/routes
23:27:34.413 [ConfigProxy] e[32minfoe[39m: Route added / -> http://hub:8081
23:28:21.933 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:28:21.935 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:28:21.937 [ConfigProxy] e[32minfoe[39m: Adding route / -> http://hub:8081
23:28:21.937 [ConfigProxy] e[32minfoe[39m: Route added / -> http://hub:8081
23:28:21.938 [ConfigProxy] e[32minfoe[39m: 201 POST /api/routes/ 
23:29:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:30:21.933 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:31:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:32:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:33:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:34:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:35:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:36:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:37:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:38:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:39:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:40:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:41:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:42:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:43:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:44:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:45:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:46:21.933 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:47:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:48:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:49:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:50:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:51:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:52:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:53:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:54:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:55:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:56:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:57:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:58:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
23:59:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:00:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:01:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:02:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:03:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:04:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:05:21.933 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:06:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:07:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:08:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:09:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:10:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:11:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:12:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:13:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:14:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:15:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:16:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:17:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:18:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:19:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:20:21.933 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:21:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:22:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:23:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:24:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:25:21.933 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:26:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:27:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:28:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:29:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:30:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:31:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:32:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:33:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:34:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:35:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:36:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:37:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:38:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:39:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:40:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:41:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:42:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:43:04.371 [ConfigProxy] e[32minfoe[39m: Adding route /user/admin -> http://[2600:1f18:96c:2f03:d4d8::a]:8888
00:43:04.371 [ConfigProxy] e[32minfoe[39m: Route added /user/admin -> http://[2600:1f18:96c:2f03:d4d8::a]:8888
00:43:04.371 [ConfigProxy] e[32minfoe[39m: 201 POST /api/routes/user/admin 
00:43:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:43:54.337 [ConfigProxy] e[32minfoe[39m: Removing route /user/admin
00:43:54.337 [ConfigProxy] e[32minfoe[39m: 204 DELETE /api/routes/user/admin 
00:44:19.971 [ConfigProxy] e[32minfoe[39m: Adding route /user/admin -> http://[2600:1f18:96c:2f03:d4d8::b]:8888
00:44:19.971 [ConfigProxy] e[32minfoe[39m: Route added /user/admin -> http://[2600:1f18:96c:2f03:d4d8::b]:8888
00:44:19.971 [ConfigProxy] e[32minfoe[39m: 201 POST /api/routes/user/admin 
00:44:21.929 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:45:21.932 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
00:46:21.931 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes

Deeco · February 10, 2025, 8:10pm

Is there anything else i could provide ?

manics · February 25, 2025, 5:16pm

port 8081 is an internal port for the JupyterHub API. You should use port 8000 for all access by users.

Deeco · February 28, 2025, 1:40pm

Ive tried the port-forward previously and again just now, I still get the redirect loop on it now and previously
http://localhost:8081/hub/user/admin/

Do I need to update the values file for proxy config and 8000 ?

Current service ports

hub            ClusterIP   fd5f:c06e:e955::8ef7   <none>        8081/TCP   24d
proxy-api      ClusterIP   fd5f:c06e:e955::89fe   <none>        8001/TCP   24d
proxy-public   ClusterIP   fd5f:c06e:e955::4a6d   <none>        80/TCP     24d

Deeco · February 28, 2025, 2:42pm

I think ive got it working.

I port forwarded proxy public service port 80 and can get in on localhost:8000

k port-forward svc/proxy-public 8000:80

thanks for the guidance

Deeco · February 28, 2025, 2:48pm

Yep all working after pointing virtual service to public-proxy svc and port 80 in ingress. Also see the PROXT WEB logs for user you described above

Topic		Replies	Views
How to setup juypterhub with ipv6 config on kubernets cluster JupyterHub	5	699	December 18, 2024
JupyterHub Not Loading Up with Given External-IP\| K8s \| Helm Zero to JupyterHub on Kubernetes jupyterhub , how-to , help-wanted	2	437	November 28, 2021
JupyterHub not launching on Helm \| K8s Zero to JupyterHub on Kubernetes jupyterhub , how-to , help-wanted	2	1249	December 4, 2021
Dask Hub/JupyterHub - Cannot start Python Kernel JupyterHub jupyterhub	5	1099	October 21, 2020
Timeout while spawning notebook server Zero to JupyterHub on Kubernetes jupyterlab , jupyterhub	1	192	May 4, 2024

Kubernetes Helm Deployment Unable to Connect to Server Using IPv6

Related topics