How to setup juypterhub with ipv6 config on kubernets cluster

JupyterHub
1

Hello Team,

we want to configure juypter hub in our kubernets cluster with ipv6 config .
i have download the helm chart form below link .

when i have try to spin up notebook server its try to connect with hub with 127.0.0.1 and after 1 minutes notebook pod is getting terminating state .

could you please suggest how to configure with ipv6 on k8s cluster .

Thanks & Regards,
Amiya

2

I don’t know if anyone else is running Z2JH on an ipv6 only system, it’d be interesting to hear from them if they have.

If not you’ll need to do some investigation yourself. Which version of the chart are you using? Can you show us your config, turn on debug logging, and show us both your hub and singleuser logs?

Are other multi-pod applications working in your K8s cluster?

3

Hello Manics ,

we are using 2.0.0 version helm chart .

notebook server logs :-

kubectl logs -f jupyter-amiya -n jupyterhub

Error from server (BadRequest): container “notebook” in pod “jupyter-amiya” is waiting to start: PodInitializing

[amiya.bhusan-a@uhn5klbjhdmz-p amiya]$ kubectl logs -f jupyter-amiya -n jupyterhub

[I 2023-08-07 11:56:18.047 SingleUserLabApp mixins:610] Starting jupyterhub single-user server version 3.0.0

[I 2023-08-07 11:56:18.048 SingleUserLabApp mixins:624] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 3.4.6

[I 2023-08-07 11:56:18.048 SingleUserLabApp mixins:624] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.18.1

[I 2023-08-07 11:56:18.098 SingleUserLabApp manager:340] jupyterlab | extension was successfully linked.

[I 2023-08-07 11:56:18.111 SingleUserLabApp manager:340] nbclassic | extension was successfully linked.

[I 2023-08-07 11:56:18.111 SingleUserLabApp manager:340] nbgitpuller | extension was successfully linked.

[I 2023-08-07 11:56:18.432 SingleUserLabApp manager:340] notebook_shim | extension was successfully linked.

[I 2023-08-07 11:56:18.447 SingleUserLabApp manager:340] retrolab | extension was successfully linked.

[I 2023-08-07 11:56:18.481 SingleUserLabApp manager:362] notebook_shim | extension was successfully loaded.

[I 2023-08-07 11:56:18.483 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.9/site-packages/jupyterlab

[I 2023-08-07 11:56:18.483 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab

[I 2023-08-07 11:56:18.488 SingleUserLabApp manager:362] jupyterlab | extension was successfully loaded.

[I 2023-08-07 11:56:18.493 SingleUserLabApp manager:362] nbclassic | extension was successfully loaded.

[I 2023-08-07 11:56:18.494 SingleUserLabApp manager:362] nbgitpuller | extension was successfully loaded.

[I 2023-08-07 11:56:18.498 SingleUserLabApp manager:362] retrolab | extension was successfully loaded.

[I 2023-08-07 11:56:18.498 SingleUserLabApp mixins:671] Starting jupyterhub-singleuser server version 3.0.0

[I 2023-08-07 11:56:18.507 SingleUserLabApp serverapp:2726] Serving notebooks from local directory: /home/jovyan

[I 2023-08-07 11:56:18.507 SingleUserLabApp serverapp:2726] Jupyter Server 1.18.1 is running at:

[I 2023-08-07 11:56:18.507 SingleUserLabApp serverapp:2726] http://jupyter-amiya:8888/user/amiya/lab

[I 2023-08-07 11:56:18.507 SingleUserLabApp serverapp:2726] or http://127.0.0.1:8888/user/amiya/lab

[I 2023-08-07 11:56:18.507 SingleUserLabApp serverapp:2727] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[I 2023-08-07 11:56:18.512 SingleUserLabApp mixins:592] Updating Hub with activity every 300 seconds

[C 2023-08-07 11:56:50.366 SingleUserLabApp serverapp:2167] received signal 15, stopping

[I 2023-08-07 11:56:50.367 SingleUserLabApp serverapp:2514] Shutting down 5 extensions

[I 2023-08-07 11:56:50.369 SingleUserLabApp serverapp:2503] Shutting down 0 terminals

4

HUB POD LOGS :-

[W 2023-08-07 11:55:52.206 JupyterHub log:186] 403 GET /hub/metrics (@240b:c0e0:104:544d:b464:2:0:4ece) 3.08ms
[E 2023-08-07 11:55:54.487 JupyterHub pages:371] Previous spawn for amiya failed: Server at http://[240b:c0e0:104:544d:b464:2:0:42e3]:8888/user/amiya/ didn’t respond in 30 seconds
[E 2023-08-07 11:55:54.504 JupyterHub log:178] {
“X-Forwarded-Host”: “localhost:30377”,
“X-Forwarded-Proto”: “http”,
“X-Forwarded-Port”: “30377”,
“X-Forwarded-For”: “240b:c0e0:104:544d:b462:2::”,
“If-None-Match”: “"04bd3e414fed271c7bc2c9d326033fdbe5d4742c"”,
“Cookie”: “jupyterhub-hub-login=[secret]; csrfCookie=[secret]; jupyterhub-session-id=[secret]”,
“Accept-Language”: “en-GB,en-US;q=0.9,en;q=0.8”,
“Accept-Encoding”: “gzip, deflate, br”,
“Sec-Fetch-Dest”: “document”,
“Sec-Fetch-User”: “?1”,
“Sec-Fetch-Mode”: “navigate”,
“Sec-Fetch-Site”: “none”,
“Accept”: “text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7”,
“User-Agent”: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36”,
“Upgrade-Insecure-Requests”: “1”,
“Sec-Ch-Ua-Platform”: “"macOS"”,
“Sec-Ch-Ua-Mobile”: “?0”,
“Sec-Ch-Ua”: “"Not/A)Brand";v="99", "Google Chrome";v="115", "Chromium";v="115"”,
“Cache-Control”: “max-age=0”,
“Connection”: “close”,
“Host”: “localhost:30377”
}
[E 2023-08-07 11:55:54.504 JupyterHub log:186] 500 GET /hub/spawn-pending/amiya (amiya@240b:c0e0:104:544d:b462:2::slight_smile: 20.75ms
[I 2023-08-07 11:55:56.655 JupyterHub provider:651] Creating oauth client jupyterhub-user-amiya
[I 2023-08-07 11:55:56.699 JupyterHub spawner:2509] Attempting to create pvc claim-amiya, with timeout 3
[I 2023-08-07 11:55:56.702 JupyterHub log:186] 302 GET /hub/spawn/amiya → /hub/spawn-pending/amiya (amiya@240b:c0e0:104:544d:b462:2::slight_smile: 75.54ms
[I 2023-08-07 11:55:56.738 JupyterHub spawner:2525] PVC claim-amiya already exists, so did not create new pvc.
[I 2023-08-07 11:55:56.748 JupyterHub spawner:2469] Attempting to create pod jupyter-amiya, with timeout 3
[I 2023-08-07 11:55:56.831 JupyterHub pages:394] amiya is pending spawn
[I 2023-08-07 11:55:56.833 JupyterHub log:186] 200 GET /hub/spawn-pending/amiya (amiya@240b:c0e0:104:544d:b462:2::slight_smile: 6.05ms
[W 2023-08-07 11:56:02.204 JupyterHub web:1796] 403 GET /hub/metrics (240b:c0e0:104:544d:b464:2:0:4ece): Access to metrics requires scope ‘read:metrics’
[W 2023-08-07 11:56:02.207 JupyterHub log:186] 403 GET /hub/metrics (@240b:c0e0:104:544d:b464:2:0:4ece) 3.01ms
[W 2023-08-07 11:56:12.204 JupyterHub web:1796] 403 GET /hub/metrics (240b:c0e0:104:544d:b464:2:0:4ece): Access to metrics requires scope ‘read:metrics’
[W 2023-08-07 11:56:12.206 JupyterHub log:186] 403 GET /hub/metrics (@240b:c0e0:104:544d:b464:2:0:4ece) 2.95ms
[W 2023-08-07 11:56:14.960 JupyterHub web:1796] 403 GET /hub/metrics (240b:c0e0:104:544d:b464:2:0:4941): Access to metrics requires scope ‘read:metrics’
[W 2023-08-07 11:56:14.963 JupyterHub log:186] 403 GET /hub/metrics (@240b:c0e0:104:544d:b464:2:0:4941) 2.96ms
[I 2023-08-07 11:56:18.506 JupyterHub log:186] 200 GET /hub/api (@240b:c0e0:104:544d:b464:2:0:41cc) 1.47ms
[I 2023-08-07 11:56:18.540 JupyterHub log:186] 200 POST /hub/api/users/amiya/activity (amiya@240b:c0e0:104:544d:b464:2:0:41cc) 24.38ms
[W 2023-08-07 11:56:22.204 JupyterHub web:1796] 403 GET /hub/metrics (240b:c0e0:104:544d:b464:2:0:4ece): Access to metrics requires scope ‘read:metrics’
[W 2023-08-07 11:56:22.208 JupyterHub log:186] 403 GET /hub/metrics (@240b:c0e0:104:544d:b464:2:0:4ece) 4.39ms
[W 2023-08-07 11:56:32.204 JupyterHub web:1796] 403 GET /hub/metrics (240b:c0e0:104:544d:b464:2:0:4ece): Access to metrics requires scope ‘read:metrics’
[W 2023-08-07 11:56:32.206 JupyterHub log:186] 403 GET /hub/metrics (@240b:c0e0:104:544d:b464:2:0:4ece) 3.02ms
[W 2023-08-07 11:56:42.204 JupyterHub web:1796] 403 GET /hub/metrics (240b:c0e0:104:544d:b464:2:0:4ece): Access to metrics requires scope ‘read:metrics’
[W 2023-08-07 11:56:42.206 JupyterHub log:186] 403 GET /hub/metrics (@240b:c0e0:104:544d:b464:2:0:4ece) 3.06ms
[W 2023-08-07 11:56:50.133 JupyterHub user:881] amiya’s server never showed up at http://[240b:c0e0:104:544d:b464:2:0:41cc]:8888/user/amiya/ after 30 seconds. Giving up.

Common causes of this timeout, and debugging tips:

1. The server didn't finish starting,
   or it crashed due to a configuration issue.
   Check the single-user server's logs for hints at what needs fixing.
2. The server started, but is not accessible at the specified URL.
   This may be a configuration issue specific to your chosen Spawner.
   Check the single-user server logs and resource to make sure the URL
   is correct and accessible from the Hub.
3. (unlikely) Everything is working, but the server took too long to respond.
   To fix: increase `Spawner.http_timeout` configuration
   to a number of seconds that is enough for servers to become responsive.

[I 2023-08-07 11:56:50.134 JupyterHub spawner:2780] Deleting pod jupyterhub/jupyter-amiya
[W 2023-08-07 11:56:52.204 JupyterHub web:1796] 403 GET /hub/metrics (240b:c0e0:104:544d:b464:2:0:4ece): Access to metrics requires scope ‘read:metrics’
[W 2023-08-07 11:56:52.206 JupyterHub log:186] 403 GET /hub/metrics (@240b:c0e0:104:544d:b464:2:0:4ece) 3.05ms
[W 2023-08-07 11:56:56.894 JupyterHub base:1030] 2 consecutive spawns failed. Hub will exit if failure count reaches 5 before succeeding
[E 2023-08-07 11:56:56.894 JupyterHub gen:630] Exception in Future <Task finished name=‘Task-785248’ coro=<BaseHandler.spawn_single_user..finish_user_spawn() done, defined at /usr/local/lib/python3.9/site-packages/jupyterhub/handlers/base.py:954> exception=TimeoutError(“Server at http://[240b:c0e0:104:544d:b464:2:0:41cc]:8888/user/amiya/ didn’t respond in 30 seconds”)> after timeout
Traceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/tornado/gen.py”, line 625, in error_callback
future.result()
File “/usr/local/lib/python3.9/site-packages/jupyterhub/handlers/base.py”, line 961, in finish_user_spawn
await spawn_future
File “/usr/local/lib/python3.9/site-packages/jupyterhub/user.py”, line 862, in spawn
await self._wait_up(spawner)
File “/usr/local/lib/python3.9/site-packages/jupyterhub/user.py”, line 906, in _wait_up
raise e
File “/usr/local/lib/python3.9/site-packages/jupyterhub/user.py”, line 876, in _wait_up
resp = await server.wait_up(
File “/usr/local/lib/python3.9/site-packages/jupyterhub/utils.py”, line 288, in wait_for_http_server
re = await exponential_backoff(
File “/usr/local/lib/python3.9/site-packages/jupyterhub/utils.py”, line 236, in exponential_backoff
raise asyncio.TimeoutError(fail_message)
asyncio.exceptions.TimeoutError: Server at http://[240b:c0e0:104:544d:b464:2:0:41cc]:8888/user/amiya/ didn’t respond in 30 seconds

2 Likes
5

Got a similar issue, believe it related to the default image Quay and startup, is there a supported value to override 127.0.0.1 to 0.0.0.0 in charts / deployments ?

6

Was anyone ever able to resolve this ? Ive checked ipv6 works across cluster and can ping over ipv6 the address, seems to be how the singleserver starts up on 127.0.0.1, Tried to override in values.yaml file with ExtraArgs and Extraconfig but doesnt appear to change , hub states the extra config loads in log file