Using modify_pod_hook: failed to establish connection

JupyterHub Zero to JupyterHub on Kubernetes
1

I am using modify_pod_hook to dynamically create an extra pvc per user, however looks like the spawner is not able to connect:

HTTPConnectionPool(host=‘localhost’, port=80): Max retries exceeded with url: /api/v1/pods?watch=False (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7f5db257fd50>: Failed to establish a new connection: [Errno 111] Connection refused’))

Has anyone run into similar issue?

Note, I am using ingress with tls, not sure why its trying to connect to localhost on 80? Do I have to specifically set JupyterHub bind_address and ip within extraConfigs?

Thanks for the help

2

Can you start by deploying a default Z2JH with minimal customisations? Once that’s working and fully tested you can start making changes to modify_pod_hook.

3

Default Z2JH deploys successfully. Even c.KubeSpawner.pvc_name_template works and creates a seperate pvc, but then default jupyterhub home pvc doesn’t get created (from my understanding pvc_name_template’s limitation is it can only create one pvc per user). Therefore I went with kubernetes.client.V1PersistentVolumeClaim via modify_pod_hook route.

I also, tried turning off the ingress and using NodePort, but the same error persists.

Appreciate any assistance to point me in the right direction to resolve this.

4

If everything works without your custom modify_pod_hook then that’s presumably the problem! Can you share your full config including modify_pod_hook? Can you also turn on debug logging and show us your logs?

6

This issue has been resolved by adding the following line:
config.load_incluster_config()
before defining the persistent volume claim details as below:

          metadata_pvc=client.V1ObjectMeta(name=pvc_name, namespace=spawner.namespace)
          resources_pvc=client.V1ResourceRequirements(requests={'storage': '10Gi'})
          spec_pvc=client.V1PersistentVolumeClaimSpec(access_modes=['ReadWriteMany'], resources=resources_pvc, storage_class_name='ontap-nas')
          persistent_volume_claim = client.V1PersistentVolumeClaim(api_version="v1",kind="PersistentVolumeClaim",metadata=metadata_pvc, spec=spec_pvc)
          resp_pvc=core_api.create_namespaced_persistent_volume_claim(spawner.namespace, persistent_volume_claim)

Hope this helps someone who runs into similar issue.

Thanks,
Alisa

1 Like