Using modify_pod_hook: failed to establish connection

I am using modify_pod_hook to dynamically create an extra pvc per user, however looks like the spawner is not able to connect:

HTTPConnectionPool(host=‘localhost’, port=80): Max retries exceeded with url: /api/v1/pods?watch=False (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7f5db257fd50>: Failed to establish a new connection: [Errno 111] Connection refused’))

Has anyone run into similar issue?

Note, I am using ingress with tls, not sure why its trying to connect to localhost on 80? Do I have to specifically set JupyterHub bind_address and ip within extraConfigs?

Thanks for the help

Can you start by deploying a default Z2JH with minimal customisations? Once that’s working and fully tested you can start making changes to modify_pod_hook.

Default Z2JH deploys successfully. Even c.KubeSpawner.pvc_name_template works and creates a seperate pvc, but then default jupyterhub home pvc doesn’t get created (from my understanding pvc_name_template’s limitation is it can only create one pvc per user). Therefore I went with kubernetes.client.V1PersistentVolumeClaim via modify_pod_hook route.

I also, tried turning off the ingress and using NodePort, but the same error persists.

Appreciate any assistance to point me in the right direction to resolve this.

If everything works without your custom modify_pod_hook then that’s presumably the problem! Can you share your full config including modify_pod_hook? Can you also turn on debug logging and show us your logs?

(post deleted by author)

This issue has been resolved by adding the following line:
config.load_incluster_config()
before defining the persistent volume claim details as below:

          metadata_pvc=client.V1ObjectMeta(name=pvc_name, namespace=spawner.namespace)
          resources_pvc=client.V1ResourceRequirements(requests={'storage': '10Gi'})
          spec_pvc=client.V1PersistentVolumeClaimSpec(access_modes=['ReadWriteMany'], resources=resources_pvc, storage_class_name='ontap-nas')
          persistent_volume_claim = client.V1PersistentVolumeClaim(api_version="v1",kind="PersistentVolumeClaim",metadata=metadata_pvc, spec=spec_pvc)
          resp_pvc=core_api.create_namespaced_persistent_volume_claim(spawner.namespace, persistent_volume_claim)

Hope this helps someone who runs into similar issue.

Thanks,
Alisa