Kubernetes Helm Deployment Unable to Connect to Server Using IPv6

JupyterHub
,
1

I cannot seem to get the hub pod to speak to the server pod using ipv6, all comes up and ive tested ipv6 ping connection which resolves for both and can also get to hub via svc.

The issue appears to be with the server startup on named address and 127.0.0.1 which is ipv4, ive tried to pass extra args, envs and cmds into values file from Helm deployment but nothing seems to add. Ive also tried adding some settings in configmap but still keeps starting on 127.0.0.1

[I 2024-12-18 15:53:39.936 ServerApp] Jupyter Server 2.8.0 is running at:
[I 2024-12-18 15:53:39.936 ServerApp] http://jupyter-admin:8888/user/admin/lab?token=…
[I 2024-12-18 15:53:39.936 ServerApp] http://127.0.0.1:8888/user/admin/lab?token=…
[I 2024-12-18 15:53:39.936 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

admin’s server never showed up at http://[2600:1f18:96c:2f04:e00a::8]:8888/user/admin/

[W 2024-12-18 15:46:24.832 JupyterHub base:1194] 2 consecutive spawns failed. Hub will exit if failure count reaches 5 before succeeding
[E 2024-12-18 15:46:24.832 JupyterHub gen:629] Exception in Future <Task finished name=‘Task-6414’ coro=<BaseHandler.spawn_single_user..finish_user_spawn() done, defined at /usr/local/lib/python3.12/site-packages/jupyterhub/handlers/base.py:1115> exception=TimeoutError(“Server at http://[2600:1f18:96c:2f04:e00a::8]:8888/user/admin/api didn’t respond in 120 seconds”)> after timeout
Traceback (most recent call last):
File “/usr/local/lib/python3.12/site-packages/tornado/gen.py”, line 624, in error_callback
future.result()
File “/usr/local/lib/python3.12/site-packages/jupyterhub/handlers/base.py”, line 1122, in finish_user_spawn
await spawn_future
File “/usr/local/lib/python3.12/site-packages/jupyterhub/user.py”, line 1033, in spawn
await self._wait_up(spawner)
File “/usr/local/lib/python3.12/site-packages/jupyterhub/user.py”, line 1076, in _wait_up
raise e
File “/usr/local/lib/python3.12/site-packages/jupyterhub/user.py”, line 1047, in _wait_up
resp = await server.wait_up(
^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.12/site-packages/jupyterhub/utils.py”, line 322, in wait_for_http_server
re = await exponential_backoff(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.12/site-packages/jupyterhub/utils.py”, line 265, in exponential_backoff
raise asyncio.TimeoutError(fail_message)
TimeoutError: Server at http://[2600:1f18:96c:2f04:e00a::8]:8888/user/admin/api didn’t respond in 120 seconds

[I 2024-12-18 15:46:24.834 JupyterHub log:192] 200 GET /hub/api/users/admin/server/progress?_xsrf=[secret] (admin@2600:1f18:96c:2f04::21dc) 131944.46ms
[W 2024-12-18 15:47:38.647 JupyterHub web:1873] 403 GET /hub/metrics (2600:1f18:96c:2f05:caa9::4): Access to metrics requires scope ‘read:metrics’

2

ive tested the connection from the admin pod to hub and can curl over ipv6 back to api endpoint

3

Startup commands in pod are below, cannot seem to override

jupyterhub-singleuser --ip=0.0.0.0

4

What configuration have you tried? Did setting c.Kubespawner.ip = ... change anything?

5

I havent tried adding c.Kubespawner.ip to config Im using Installing JupyterHub — Zero to JupyterHub with Kubernetes documentation and adding values to singleserver and args to no avail.

Im building on top of latest 4.0 chart values file. JupyterHub helm charts | JupyterHub’s Helm chart repository

Our stack is ipv6 only on AWS eks. Ive even tried to manual create service or pass [::] arg instead of 0.0.0.0, it appears to never listen or connect. Also updated the network policy to allow access over ipv6 and range on ingress

6

part of singleuser values tried with also some manual modifying

# singleuser relates to the configuration of KubeSpawner which runs in the hub
# pod, and its spawning of user pods such as jupyter-myusername.
singleuser:
  podNameTemplate:
  extraTolerations: []
  nodeSelector:
    kubernetes.io/arch: amd64
  extraNodeAffinity:
    required: []
    preferred: []
  extraPodAffinity:
    required: []
    preferred: []
  extraPodAntiAffinity:
    required: []
    preferred: []
  networkTools:
    image:
      name: quay.io/jupyterhub/k8s-network-tools
      tag: "4.0.0"
      pullPolicy:
      pullSecrets: []
    resources: {}
  cloudMetadata:
    # block set to true will append a privileged initContainer using the
    # iptables to block the sensitive metadata server at the provided ip.
    blockWithIptables: true
    ip: 169.254.169.254
  networkPolicy:
    enabled: true
    ingress:
    - from:
      - podSelector: {}
      ports:
      - protocol: TCP
        port: 8888
    egress: []
    egressAllowRules:
      cloudMetadataServer: false
      dnsPortsCloudMetadataServer: true
      dnsPortsKubeSystemNamespace: true
      dnsPortsPrivateIPs: true
      nonPrivateIPs: true
      privateIPs: false
    interNamespaceAccessLabels: ignore
    allowedIngressPorts: []
  events: true
  extraAnnotations: {}
  extraLabels:
    hub.jupyter.org/network-access-hub: "true"
  extraFiles: {}
  extraEnv:
    NOTEBOOK_ARGS: "--ip=[::] --port=8888"
  lifecycleHooks: {}
  initContainers: []
  extraContainers: []
  allowPrivilegeEscalation: false
  uid: 1000
  fsGid: 100
  serviceAccountName:
  storage:
    type: dynamic
    extraLabels: {}
    extraVolumes: []
    extraVolumeMounts: []
    static:
      pvcName:
      subPath: "{username}"
    capacity: 10Gi
    homeMountPath: /home/jovyan
    dynamic:
      storageClass:
      pvcNameTemplate:
      volumeNameTemplate: volume-{user_server}
      storageAccessModes: [ ReadWriteOnce ]
      subPath:
  image:
    name: quay.io/jupyterhub/k8s-singleuser-sample
    tag: "4.0.0"
    pullPolicy:
    pullSecrets: []
  startTimeout: 300
  cpu:
    limit:
    guarantee:
  memory:
    limit:
    guarantee: 1G
  extraResource:
    limits: {}
    guarantees: {}
  cmd: [ "jupyterhub-singleuser", "--ip=0.0.0.0", "--port=8888" ]
  defaultUrl:
  extraPodConfig: {}
  profileList: []```
7

So Ive got a little further I hope … ive updated the startup cmd to below

cmd: [ "jupyterhub-singleuser", "--ip='::'", "--port=8888" ]

8

No getting a loopback error

500 : Internal Server Error

Redirect loop detected.

[I 2024-12-20 15:33:32.491 JupyterHub log:192] 302 GET /user/admin/?redirects=4 → /hub/user/admin/?redirects=4 (@2600:1f18:96c:2f05::ccb7) 0.62ms
[D 2024-12-20 15:33:32.538 JupyterHub reflector:390] pods watcher timeout
[D 2024-12-20 15:33:32.538 JupyterHub reflector:304] Connecting pods watcher
[W 2024-12-20 15:33:32.618 JupyterHub web:1873] 500 GET /hub/user/admin/?redirects=4 (2600:1f18:96c:2f05::ccb7): Redirect loop detected.
[D 2024-12-20 15:33:32.619 JupyterHub base:1519] Using default error template for 500

proxy looks ok …

15:33:17.120 [ConfigProxy] info: Adding route /user/admin → http://[2600:1f18:96c:2f03:ca57::f]:8888
15:33:17.120 [ConfigProxy] info: Route added /user/admin → http://[2600:1f18:96c:2f03:ca57::f]:8888
15:33:17.120 [ConfigProxy] info: 201 POST /api/routes/user/admin
15:34:02.225 [ConfigProxy] info: 200 GET /api/routes
15:35:02.225 [ConfigProxy] info: 200 GET /api/routes
15:36:02.224 [ConfigProxy] info: 200 GET /api/routes

9

Can you share the preceding hub logs too, starting from when the initial call to start the singleuser server is made? That may contain clues as to why it’s redirecting.

10 
[D 2024-12-21 21:36:23.976 JupyterHub reflector:390] pods watcher timeout
[D 2024-12-21 21:36:23.976 JupyterHub reflector:304] Connecting pods watcher
[D 2024-12-21 21:36:24.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.62ms
[D 2024-12-21 21:36:26.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.62ms
[I 2024-12-21 21:36:27.075 JupyterHub log:192] 302 GET / -> /hub/ (@2600:1f18:96c:2f03::2d99) 0.66ms
[D 2024-12-21 21:36:27.212 JupyterHub base:411] Refreshing auth for admin
[D 2024-12-21 21:36:27.213 JupyterHub user:496] Creating <class 'kubespawner.spawner.KubeSpawner'> for admin:
[I 2024-12-21 21:36:27.215 JupyterHub log:192] 302 GET /hub/ -> /hub/spawn (admin@2600:1f18:96c:2f03::2d99) 12.76ms
[D 2024-12-21 21:36:27.349 JupyterHub scopes:1010] Checking access to /hub/spawn via scope servers!server=admin/
[D 2024-12-21 21:36:27.349 JupyterHub pages:216] Triggering spawn with default options for admin
[D 2024-12-21 21:36:27.349 JupyterHub base:1095] Initiating spawn for admin
[D 2024-12-21 21:36:27.349 JupyterHub base:1099] 0/64 concurrent spawns
[D 2024-12-21 21:36:27.349 JupyterHub base:1104] 0 active servers
[I 2024-12-21 21:36:27.366 JupyterHub provider:661] Creating oauth client jupyterhub-user-admin
[D 2024-12-21 21:36:27.387 JupyterHub user:913] Calling Spawner.start for admin
[I 2024-12-21 21:36:27.388 JupyterHub log:192] 302 GET /hub/spawn -> /hub/spawn-pending/admin (admin@2600:1f18:96c:2f03::2d99) 43.30ms
[I 2024-12-21 21:36:27.388 JupyterHub spawner:2931] Attempting to create pvc claim-admin, with timeout 3
[I 2024-12-21 21:36:27.408 JupyterHub spawner:2947] PVC claim-admin already exists, so did not create new pvc.
[I 2024-12-21 21:36:27.409 JupyterHub spawner:2890] Attempting to create pod jupyter-admin, with timeout 3
[D 2024-12-21 21:36:27.520 JupyterHub scopes:1010] Checking access to /hub/spawn-pending/admin via scope servers!server=admin/
[I 2024-12-21 21:36:27.520 JupyterHub pages:397] admin is pending spawn
[I 2024-12-21 21:36:27.521 JupyterHub log:192] 200 GET /hub/spawn-pending/admin (admin@2600:1f18:96c:2f03::2d99) 5.23ms
[D 2024-12-21 21:36:27.690 JupyterHub log:192] 200 GET /hub/static/css/style.min.css?v=de37c927476e635f657bafc35a5e1ae9c0f3ce691c5779b0d3be84fa5e2ec3813214270a51f83fc93b945e5d7e418046af1e370e9538ea8a44d49d472ee7dffe (@2600:1f18:96c:2f03::2d99) 0.95ms
[D 2024-12-21 21:36:28.198 JupyterHub reflector:390] events watcher timeout
[D 2024-12-21 21:36:28.198 JupyterHub reflector:304] Connecting events watcher
[D 2024-12-21 21:36:28.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.92ms
[D 2024-12-21 21:36:28.292 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.96ms
[D 2024-12-21 21:36:28.376 JupyterHub log:192] 304 GET /hub/static/components/@fortawesome/fontawesome-free/webfonts/fa-solid-900.woff2 (@2600:1f18:96c:2f03::2d99) 0.59ms
[D 2024-12-21 21:36:28.388 JupyterHub scopes:1010] Checking access to /hub/api/users/admin/server/progress via scope read:servers!server=admin/
[D 2024-12-21 21:36:28.391 JupyterHub spawner:2672] progress generator: jupyter-admin
[D 2024-12-21 21:36:30.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.78ms
[D 2024-12-21 21:36:32.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.67ms
[D 2024-12-21 21:36:33.984 JupyterHub reflector:390] pods watcher timeout
[D 2024-12-21 21:36:33.984 JupyterHub reflector:304] Connecting pods watcher
[D 2024-12-21 21:36:34.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.75ms
[D 2024-12-21 21:36:36.290 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.68ms
[I 2024-12-21 21:36:36.745 JupyterHub log:192] 200 GET /hub/api (@2600:1f18:96c:2f03:ca57::f) 0.75ms
[D 2024-12-21 21:36:36.799 JupyterHub base:366] Recording first activity for <APIToken('3d26...', user='admin', client_id='jupyterhub')>
[D 2024-12-21 21:36:36.806 JupyterHub scopes:1010] Checking access to /hub/api/users/admin/activity via scope users:activity!user=admin
[D 2024-12-21 21:36:36.809 JupyterHub users:1006] Activity for user admin: 2024-12-21T21:36:36.726237Z
[D 2024-12-21 21:36:36.809 JupyterHub users:1024] Activity on server admin/: 2024-12-21T21:36:36.726237Z
[I 2024-12-21 21:36:36.816 JupyterHub log:192] 200 POST /hub/api/users/admin/activity (admin@2600:1f18:96c:2f03:ca57::f) 19.11ms
[D 2024-12-21 21:36:38.207 JupyterHub reflector:390] events watcher timeout
[D 2024-12-21 21:36:38.207 JupyterHub reflector:304] Connecting events watcher
[D 2024-12-21 21:36:38.292 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.97ms
[D 2024-12-21 21:36:38.292 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.99ms
[D 2024-12-21 21:36:40.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.66ms
[D 2024-12-21 21:36:40.359 JupyterHub spawner:3254] pod data-platform-jupyterhub/jupyter-admin events before launch: 2024-12-21T21:36:27.451297Z [Normal] Successfully assigned data-platform-jupyterhub/jupyter-admin to ip-10-82-74-182.ec2.internal
    2024-12-21T21:36:29Z [Normal] AttachVolume.Attach succeeded for volume "pvc-bed29366-131e-442c-b81e-6d512557b348" 
    2024-12-21T21:36:34Z [Normal] Container image "quay.io/jupyterhub/k8s-network-tools:4.0.0" already present on machine
    2024-12-21T21:36:34Z [Normal] Created container block-cloud-metadata
    2024-12-21T21:36:34Z [Normal] Started container block-cloud-metadata
    2024-12-21T21:36:35Z [Normal] Container image "quay.io/jupyterhub/k8s-singleuser-sample:4.0.0" already present on machine
    2024-12-21T21:36:35Z [Normal] Created container notebook
    2024-12-21T21:36:35Z [Normal] Started container notebook
[D 2024-12-21 21:36:40.366 JupyterHub spawner:1475] Polling subprocess every 30s
[D 2024-12-21 21:36:40.367 JupyterHub utils:292] Waiting 300s for server at http://[2600:1f18:96c:2f03:ca57::f]:8888/user/admin/api
[D 2024-12-21 21:36:40.372 JupyterHub utils:328] Server at http://[2600:1f18:96c:2f03:ca57::f]:8888/user/admin/api responded in 0.00s
[D 2024-12-21 21:36:40.372 JupyterHub _version:73] jupyterhub and jupyterhub-singleuser both on version 5.2.1
[I 2024-12-21 21:36:40.372 JupyterHub base:1124] User admin took 13.023 seconds to start
[I 2024-12-21 21:36:40.372 JupyterHub proxy:331] Adding user admin to proxy /user/admin/ => http://[2600:1f18:96c:2f03:ca57::f]:8888
[D 2024-12-21 21:36:40.372 JupyterHub proxy:925] Proxy: Fetching POST http://proxy-api:8001/api/routes/user/admin
[I 2024-12-21 21:36:40.377 JupyterHub users:899] Server admin is ready
[I 2024-12-21 21:36:40.378 JupyterHub log:192] 200 GET /hub/api/users/admin/server/progress?_xsrf=[secret] (admin@2600:1f18:96c:2f03::2d99) 11993.89ms
[D 2024-12-21 21:36:40.542 JupyterHub scopes:1010] Checking access to /hub/spawn-pending/admin via scope servers!server=admin/
[I 2024-12-21 21:36:40.542 JupyterHub log:192] 302 GET /hub/spawn-pending/admin -> /user/admin/ (admin@2600:1f18:96c:2f03::2d99) 4.43ms
[I 2024-12-21 21:36:40.670 JupyterHub log:192] 302 GET /user/admin/ -> /hub/user/admin/ (@2600:1f18:96c:2f03::2d99) 0.64ms
[I 2024-12-21 21:36:40.801 JupyterHub log:192] 302 GET /hub/user/admin/ -> /user/admin/?redirects=1 (admin@2600:1f18:96c:2f03::2d99) 4.08ms
[I 2024-12-21 21:36:40.930 JupyterHub log:192] 302 GET /user/admin/?redirects=1 -> /hub/user/admin/?redirects=1 (@2600:1f18:96c:2f03::2d99) 0.60ms
[W 2024-12-21 21:36:41.063 JupyterHub base:1844] Redirect loop detected on /hub/user/admin/?redirects=1
[D 2024-12-21 21:36:42.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.69ms
[I 2024-12-21 21:36:43.064 JupyterHub log:192] 302 GET /hub/user/admin/?redirects=1 -> /user/admin/?redirects=2 (admin@2600:1f18:96c:2f03::2d99) 2005.75ms
[I 2024-12-21 21:36:43.196 JupyterHub log:192] 302 GET /user/admin/?redirects=2 -> /hub/user/admin/?redirects=2 (@2600:1f18:96c:2f03::2d99) 0.61ms
[W 2024-12-21 21:36:43.329 JupyterHub base:1844] Redirect loop detected on /hub/user/admin/?redirects=2
[D 2024-12-21 21:36:43.992 JupyterHub reflector:390] pods watcher timeout
[D 2024-12-21 21:36:43.992 JupyterHub reflector:304] Connecting pods watcher
[D 2024-12-21 21:36:44.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.75ms
[D 2024-12-21 21:36:46.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.73ms
[I 2024-12-21 21:36:47.331 JupyterHub log:192] 302 GET /hub/user/admin/?redirects=2 -> /user/admin/?redirects=3 (admin@2600:1f18:96c:2f03::2d99) 4005.76ms
[I 2024-12-21 21:36:47.462 JupyterHub log:192] 302 GET /user/admin/?redirects=3 -> /hub/user/admin/?redirects=3 (@2600:1f18:96c:2f03::2d99) 0.66ms
[W 2024-12-21 21:36:47.595 JupyterHub base:1844] Redirect loop detected on /hub/user/admin/?redirects=3
[D 2024-12-21 21:36:48.215 JupyterHub reflector:390] events watcher timeout
[D 2024-12-21 21:36:48.216 JupyterHub reflector:304] Connecting events watcher
[D 2024-12-21 21:36:48.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.91ms
[D 2024-12-21 21:36:48.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 1.00ms
[D 2024-12-21 21:36:50.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.71ms
[D 2024-12-21 21:36:52.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.71ms
[D 2024-12-21 21:36:54.000 JupyterHub reflector:390] pods watcher timeout
[D 2024-12-21 21:36:54.001 JupyterHub reflector:304] Connecting pods watcher
[D 2024-12-21 21:36:54.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.64ms
[I 2024-12-21 21:36:55.596 JupyterHub log:192] 302 GET /hub/user/admin/?redirects=3 -> /user/admin/?redirects=4 (admin@2600:1f18:96c:2f03::2d99) 8005.68ms
[I 2024-12-21 21:36:55.725 JupyterHub log:192] 302 GET /user/admin/?redirects=4 -> /hub/user/admin/?redirects=4 (@2600:1f18:96c:2f03::2d99) 0.57ms
[W 2024-12-21 21:36:55.855 JupyterHub web:1873] 500 GET /hub/user/admin/?redirects=4 (2600:1f18:96c:2f03::2d99): Redirect loop detected.
[D 2024-12-21 21:36:55.855 JupyterHub base:1519] Using default error template for 500
[E 2024-12-21 21:36:55.856 JupyterHub log:184] {
      "Host": "data-platform-jupyterhub.k8s.dev.303net.net",
      "X-Forwarded-For": "10.99.80.110,2600:1f18:96c:2f03::2d99",
      "X-Forwarded-Proto": "https",
      "X-Forwarded-Port": "443",
      "X-Amzn-Trace-Id": "Root=1-676734f7-0a91bc851bbcd6772dc05fa2",
      "Cache-Control": "max-age=0",
      "Upgrade-Insecure-Requests": "1",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Edg/131.0.0.0",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Sec-Ch-Ua": "\"Microsoft Edge\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\"",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua-Platform": "\"macOS\"",
      "Referer": "https://data-platform-jupyterhub.k8s.dev.303net.net/hub/spawn-pending/admin",
      "Accept-Encoding": "gzip, deflate, br, zstd",
      "Accept-Language": "en-GB,en;q=0.9,en-US;q=0.8",
      "Priority": "u=0, i",
      "Cookie": "jupyterhub-hub-login=[secret]; _xsrf=[secret]; apt.uid=[secret]; jupyterhub-session-id=[secret]",
      "X-Envoy-External-Address": "10.99.80.110",
      "X-Request-Id": "cffe3f24-9147-4ec0-8553-b12183b8c048",
      "X-Envoy-Decorator-Operation": "hub.data-platform-jupyterhub.svc.cluster.local:8081/*",
      "X-Envoy-Peer-Metadata-Id": "router~2600:1f18:96c:2f03:f817::~istio-ingressgateway-bb4598595-8dbj7.istio-system~istio-system.svc.cluster.local",
      "X-Envoy-Peer-Metadata": "CiwKCkNMVVNURVJfSUQSHhocZW5nLWs4cy1kZXYtdXMtZWFzdC0xLXYzLWVrcwqZAQoGTEFCRUxTEo4BKosBCh0KA2FwcBIWGhRpc3Rpby1pbmdyZXNzZ2F0ZXdheQo5Ch9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEhYaFGlzdGlvLWluZ3Jlc3NnYXRld2F5Ci8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBjEuMjQuMAouCgROQU1FEiYaJGlzdGlvLWluZ3Jlc3NnYXRld2F5LWJiNDU5ODU5NS04ZGJqNwobCglOQU1FU1BBQ0USDhoMaXN0aW8tc3lzdGVtCl0KBU9XTkVSElQaUmt1YmVybmV0ZXM6Ly9hcGlzL2FwcHMvdjEvbmFtZXNwYWNlcy9pc3Rpby1zeXN0ZW0vZGVwbG95bWVudHMvaXN0aW8taW5ncmVzc2dhdGV3YXkKJwoNV09SS0xPQURfTkFNRRIWGhRpc3Rpby1pbmdyZXNzZ2F0ZXdheQ==",
      "X-Envoy-Attempt-Count": "1"
    }
[E 2024-12-21 21:36:55.856 JupyterHub log:192] 500 GET /hub/user/admin/?redirects=4 (admin@2600:1f18:96c:2f03::2d99) 5.07ms
[D 2024-12-21 21:36:56.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.63ms
[D 2024-12-21 21:36:58.225 JupyterHub reflector:390] events watcher timeout
[D 2024-12-21 21:36:58.225 JupyterHub reflector:304] Connecting events watcher
[D 2024-12-21 21:36:58.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.93ms
[D 2024-12-21 21:36:58.292 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.94ms
[D 2024-12-21 21:37:00.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.63ms
[D 2024-12-21 21:37:02.220 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2024-12-21 21:37:02.246 JupyterHub proxy:392] Checking routes
[D 2024-12-21 21:37:02.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.63ms
[I 2024-12-21 21:37:02.501 JupyterHub log:192] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 7.12ms
[D 2024-12-21 21:37:02.504 JupyterHub scopes:1010] Checking access to /hub/api/users via scope list:users
[I 2024-12-21 21:37:02.512 JupyterHub log:192] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 8.94ms
[D 2024-12-21 21:37:04.009 JupyterHub reflector:390] pods watcher timeout
[D 2024-12-21 21:37:04.009 JupyterHub reflector:304] Connecting pods watcher
[D 2024-12-21 21:37:04.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.68ms
[D 2024-12-21 21:37:06.290 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.70ms
[D 2024-12-21 21:37:08.235 JupyterHub reflector:390] events watcher timeout
[D 2024-12-21 21:37:08.235 JupyterHub reflector:304] Connecting events watcher
[D 2024-12-21 21:37:08.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.85ms
[D 2024-12-21 21:37:08.292 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.92ms
[D 2024-12-21 21:37:10.291 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.76ms
[D 2024-12-21 21:37:11.310 JupyterHub log:192] 304 GET /hub/home (admin@2600:1f18:96c:2f03::2d99) 10.23ms
[D 2024-12-21 21:37:12.292 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::4ca0) 0.72ms
[D 2024-12-21 21:37:12.469 JupyterHub scopes:1010] Checking access to /hub/api/users/admin/server via scope delete:servers!server=admin/
[I 2024-12-21 21:37:12.472 JupyterHub proxy:356] Removing user admin from proxy (/user/admin/)
[D 2024-12-21 21:37:12.472 JupyterHub proxy:925] Proxy: Fetching DELETE http://proxy-api:8001/api/routes/user/admin
[D 2024-12-21 21:37:12.476 JupyterHub user:1100] Stopping admin
[I 2024-12-21 21:37:12.476 JupyterHub spawner:3282] Deleting pod data-platform-jupyterhub/jupyter-admin
[D 2024-12-21 21:37:13.991 JupyterHub user:1122] Deleting oauth client jupyterhub-user-admin
[D 2024-12-21 21:37:13.998 JupyterHub user:1125] Finished stopping admin
[I 2024-12-21 21:37:14.005 JupyterHub base:1333] User admin server took 1.533 seconds to stop
[I 2024-12-21 21:37:14.006 JupyterHub log:192] 204 DELETE /hub/api/users/admin/server?_xsrf=[secret] (admin@2600:1f18:96c:2f03::2d99) 1540.45ms
[D 2024-12-21 21:37:14.018 JupyterHub reflector:390] pods watcher timeout```
11

Notebook Logs

[I 2024-12-21 21:37:33.959 ServerApp] Extension package jupyterlab took 0.1062s to import
[W 2024-12-21 21:37:34.029 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-12-21 21:37:34.030 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-12-21 21:37:34.033 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-12-21 21:37:34.034 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 5.2.1
[I 2024-12-21 21:37:34.034 JupyterHubSingleUser] Using default url from server extension lab: /lab
[I 2024-12-21 21:37:34.036 ServerApp] jupyterhub | extension was successfully linked.
[W 2024-12-21 21:37:34.037 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-12-21 21:37:34.039 ServerApp] jupyterlab | extension was successfully linked.
[W 2024-12-21 21:37:34.041 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-12-21 21:37:34.042 ServerApp] nbclassic | extension was successfully linked.
[I 2024-12-21 21:37:34.042 ServerApp] nbgitpuller | extension was successfully linked.
[I 2024-12-21 21:37:34.263 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-12-21 21:37:34.278 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-12-21 21:37:34.280 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-12-21 21:37:34.281 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-12-21 21:37:34.289 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-12-21 21:37:34.290 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-12-21 21:37:34.292 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.12/site-packages/jupyterlab
[I 2024-12-21 21:37:34.292 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2024-12-21 21:37:34.292 LabApp] Extension Manager is 'pypi'.
[I 2024-12-21 21:37:34.299 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-12-21 21:37:34.319 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-12-21 21:37:34.326 ServerApp] nbclassic | extension was successfully loaded.
[I 2024-12-21 21:37:34.327 ServerApp] nbgitpuller | extension was successfully loaded.
[I 2024-12-21 21:37:34.328 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2024-12-21 21:37:34.328 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-12-21 21:37:34.328 ServerApp] http://jupyter-admin:8888/user/admin/lab?token=...
[I 2024-12-21 21:37:34.328 ServerApp]     http://127.0.0.1:8888/user/admin/lab?token=...
[I 2024-12-21 21:37:34.328 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-12-21 21:37:35.520 ServerApp] 200 GET /user/admin/api (@2600:1f18:96c:2f04:a0a8::f) 1.01ms
[C 2024-12-21 21:38:05.747 ServerApp] received signal 15, stopping
[I 2024-12-21 21:38:05.748 ServerApp] Shutting down 7 extensions

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|
                                                                           
Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.
12

Can you check the pod manifest kubectl get pod jupyter-admin, in particular the environment variables and the image command?

13

You mean describe ? yes here is output below

Name:             jupyter-admin
Namespace:        jupyterhub
Priority:         0
Service Account:  default
Node:             ip-10-82-114-27.ec2.internal/2600:1f18:96c:2f03::4c1
Start Time:       Sun, 22 Dec 2024 00:17:27 +0000
Labels:           app=jupyterhub
                  app.kubernetes.io/component=singleuser-server
                  app.kubernetes.io/instance=jupyterhub
                  app.kubernetes.io/managed-by=kubespawner
                  app.kubernetes.io/name=jupyterhub
                  chart=jupyterhub-4.0.0
                  component=singleuser-server
                  helm.sh/chart=jupyterhub-4.0.0
                  hub.jupyter.org/network-access-hub=true
                  hub.jupyter.org/servername=
                  hub.jupyter.org/username=admin
                  release=jupyterhub
                  topology.kubernetes.io/zone=us-east-1c
Annotations:      hub.jupyter.org/jupyterhub-version: 5.2.1
                  hub.jupyter.org/kubespawner-version: 7.0.0
                  hub.jupyter.org/username: admin
Status:           Running
IP:               2600:1f18:96c:2f03:fdd6::14
IPs:
  IP:  2600:1f18:96c:2f03:fdd6::14
Init Containers:
  block-cloud-metadata:
    Container ID:  containerd://c728a9aea5f31468f7b4f328f7e6e68dad3c589762009b875642fc6c1cd48019
    Image:         quay.io/jupyterhub/k8s-network-tools:4.0.0
    Image ID:      quay.io/jupyterhub/k8s-network-tools@sha256:7f592622880e391f04c17bedd2ed4bd0a5100d32d43134872f2800e4eaf631e9
    Port:          <none>
    Host Port:     <none>
    Command:
      iptables
      --append
      OUTPUT
      --protocol
      tcp
      --destination
      169.254.169.254
      --destination-port
      80
      --jump
      DROP
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 22 Dec 2024 00:17:40 +0000
      Finished:     Sun, 22 Dec 2024 00:17:40 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:         <none>
Containers:
  notebook:
    Container ID:  containerd://aea359e88efe326d9b47063f2433ca33d0664ef27b5b8db937357c39316fae4a
    Image:         quay.io/jupyterhub/k8s-singleuser-sample:4.0.0
    Image ID:      quay.io/jupyterhub/k8s-singleuser-sample@sha256:97841e145307249c75a02f40cbc5205633b052a69ebca149ebbf7b3cb613a25e
    Port:          8888/TCP
    Host Port:     0/TCP
    Args:
      jupyterhub-singleuser
      --ip='::'
      --port=8888
    State:          Running
      Started:      Sun, 22 Dec 2024 00:17:41 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      memory:  1073741824
    Environment:
      JPY_API_TOKEN:                           31c815c1df414239829e425504d419d7
      JUPYTERHUB_ACTIVITY_URL:                 http://hub:8081/hub/api/users/admin/activity
      JUPYTERHUB_ADMIN_ACCESS:                 1
      JUPYTERHUB_API_TOKEN:                    31c815c1df414239829e425504d419d7
      JUPYTERHUB_API_URL:                      http://hub:8081/hub/api
      JUPYTERHUB_BASE_URL:                     /
      JUPYTERHUB_CLIENT_ID:                    jupyterhub-user-admin
      JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED:   0
      JUPYTERHUB_DEBUG:                        1
      JUPYTERHUB_HOST:                         
      JUPYTERHUB_OAUTH_ACCESS_SCOPES:          ["access:servers!server=admin/", "access:servers!user=admin"]
      JUPYTERHUB_OAUTH_CALLBACK_URL:           /user/admin/oauth_callback
      JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES:  []
      JUPYTERHUB_OAUTH_SCOPES:                 ["access:servers!server=admin/", "access:servers!user=admin"]
      JUPYTERHUB_PUBLIC_HUB_URL:               
      JUPYTERHUB_PUBLIC_URL:                   
      JUPYTERHUB_SERVER_NAME:                  
      JUPYTERHUB_SERVICE_PREFIX:               /user/admin/
      JUPYTERHUB_SERVICE_URL:                  http://0.0.0.0:8888/user/admin/
      JUPYTERHUB_USER:                         admin
      JUPYTER_IMAGE:                           quay.io/jupyterhub/k8s-singleuser-sample:4.0.0
      JUPYTER_IMAGE_SPEC:                      quay.io/jupyterhub/k8s-singleuser-sample:4.0.0
      MEM_GUARANTEE:                           1073741824
      NOTEBOOK_ARGS:                           --ip=[::] --port=8888
    Mounts:
      /home/jovyan from volume-admin (rw)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  volume-admin:
    Type:        PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:   claim-admin
    ReadOnly:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/arch=amd64
Tolerations:     hub.jupyter.org/dedicated=user:NoSchedule
                 hub.jupyter.org_dedicated=user:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason                  Age   From                                     Message
  ----    ------                  ----  ----                                     -------
  Normal  Scheduled               38s   jupyterhub-user-scheduler  Successfully assigned jupyterhub/jupyter-admin to ip-10-82-117-247.ec2.internal
  Normal  SuccessfulAttachVolume  36s   attachdetach-controller                  AttachVolume.Attach succeeded for volume "pvc-bed29366-131e-442c-b81e-6d512557b348"
  Normal  Pulled                  26s   kubelet                                  Container image "quay.io/jupyterhub/k8s-network-tools:4.0.0" already present on machine
  Normal  Created                 26s   kubelet                                  Created container block-cloud-metadata
  Normal  Started                 26s   kubelet                                  Started container block-cloud-metadata
  Normal  Pulled                  25s   kubelet                                  Container image "quay.io/jupyterhub/k8s-singleuser-sample:4.0.0" already present on machine
  Normal  Created                 25s   kubelet                                  Created container notebook
  Normal  Started                 25s   kubelet                                  Started container notebook
14

Any insights into where issue may arise? There is a istio ingress in front of this but ruled this out by using kubectl port-forward and bypassing, still get the same loopback error on ipv6

15

proxy api service setup also seems correct

Name:              proxy-api
Namespace:         jupyterhub
Labels:            app=jupyterhub
                   app.kubernetes.io/component=proxy-api
                   app.kubernetes.io/instance=jupyterhub
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=jupyterhub
                   chart=jupyterhub-4.0.0
                   component=proxy-api
                   helm.sh/chart=jupyterhub-4.0.0
                   heritage=Helm
                   release=jupyterhub
Annotations:       meta.helm.sh/release-name: jupyterhub
                   meta.helm.sh/release-namespace: jupyterhub
Selector:          app=jupyterhub,component=proxy,release=jupyterhub
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv6
IP:                fd5f:c06e:e955::4961
IPs:               fd5f:c06e:e955::4961
Port:              <unset>  8001/TCP
TargetPort:        api/TCP
Endpoints:         [2600:1f18:96c:2f05:267::12]:8001
Session Affinity:  None
Events:            <none>
16

public api proxy complains of unsupported ipv6 configuration, eks stack only supports ipv6

Name:                     proxy-public
Namespace:                jupyterhub
Labels:                   app=jupyterhub
                          app.kubernetes.io/component=proxy-public
                          app.kubernetes.io/instance=jupyterhub
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=jupyterhub
                          chart=jupyterhub-4.0.0
                          component=proxy-public
                          helm.sh/chart=jupyterhub-4.0.0
                          heritage=Helm
                          release=jupyterhub
Annotations:              meta.helm.sh/release-name: jupyterhub
                          meta.helm.sh/release-namespace: jupyterhub
Selector:                 app=jupyterhub,component=proxy,release=jupyterhub
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv6
IP:                       fd5f:c06e:e955::a194
IPs:                      fd5f:c06e:e955::a194
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30162/TCP
Endpoints:                [2600:1f18:96c:2f05:267::12]:8000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason            Age                  From     Message
  ----     ------            ----                 ----     -------
  Warning  FailedBuildModel  4m19s (x2 over 12m)  service  Failed build model due to unsupported IPv6 configuration, lb not dual-stack
17

I forgot, we’ve got an open PR where we’re discussing the interface to configuring IPv4/ipv6/dual:

Maybe give that a go?

18

Will try it and revert

1 Like
19

Updated and rebuilt chart and redeployed to no avail, still got same event on public proxy

Events:
  Type     Reason            Age                    From     Message
  ----     ------            ----                   ----     -------
  Warning  FailedBuildModel  2m59s (x2 over 5m35s)  service  Failed build model due to IPv6 addresses can only be set for dualstack load balancers
20 
I 2025-01-07 15:08:40.663 JupyterHub log:192] 302 GET /user/admin/?redirects=3 -> /hub/user/admin/?redirects=3 (@2600:1f18:96c:2f03::acb8) 0.55ms
[W 2025-01-07 15:08:40.777 JupyterHub base:1844] Redirect loop detected on /hub/user/admin/?redirects=3
[D 2025-01-07 15:08:40.824 JupyterHub reflector:390] pods watcher timeout
[D 2025-01-07 15:08:40.824 JupyterHub reflector:304] Connecting pods watcher
[D 2025-01-07 15:08:41.086 JupyterHub reflector:390] events watcher timeout
[D 2025-01-07 15:08:41.086 JupyterHub reflector:304] Connecting events watcher
[D 2025-01-07 15:08:41.160 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::527d) 0.73ms
[D 2025-01-07 15:08:41.160 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::527d) 0.77ms
[D 2025-01-07 15:08:43.135 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::527d) 0.55ms
[D 2025-01-07 15:08:45.135 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::527d) 0.55ms
[D 2025-01-07 15:08:47.135 JupyterHub log:192] 200 GET /hub/health (@2600:1f18:96c:2f04::527d) 0.53ms
[I 2025-01-07 15:08:48.779 JupyterHub log:192] 302 GET /hub/user/admin/?redirects=3 -> /user/admin/?redirects=4 (admin@2600:1f18:96c:2f03::acb8) 8006.21ms
[I 2025-01-07 15:08:48.905 JupyterHub log:192] 302 GET /user/admin/?redirects=4 -> /hub/user/admin/?redirects=4 (@2600:1f18:96c:2f03::acb8) 0.53ms
[W 2025-01-07 15:08:49.021 JupyterHub web:1873] 500 GET /hub/user/admin/?redirects=4 (2600:1f18:96c:2f03::acb8): Redirect loop detected.
[D 2025-01-07 15:08:49.021 JupyterHub base:1519] Using default error template for 500
[E 2025-01-07 15:08:49.022 JupyterHub log:184] {
      "Host": "data-platform-jupyterhub.k8s.dev.303net.net",
      "X-Forwarded-For": "10.99.80.110,2600:1f18:96c:2f03::acb8",
      "X-Forwarded-Proto": "https",
      "X-Forwarded-Port": "443",
      "X-Amzn-Trace-Id": "Root=1-677d4381-447fa5d26eb1cfed43cec4e2",
      "Cache-Control": "max-age=0",
      "Upgrade-Insecure-Requests": "1",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Edg/131.0.0.0",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Sec-Ch-Ua": "\"Microsoft Edge\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\"",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua-Platform": "\"macOS\"",