Connection error in Hub when deploy to on prem-k8s cluster

JupyterHub Zero to JupyterHub on Kubernetes
,
1

I deployed Zero to Jupyterhub in my on-premise k8s cluster, which has 3 nodes and deployed through kubespray. this is a existing k8s cluster , I was using MetalLB BPG mode , not the MetalLB layer 2 mode which was recommended by Zero to Jupyterhub

after I deployed jupyterhub, all pods works fine except hub pods restarts again and again

k get pods -n jupyterhub
NAME                             READY   STATUS    RESTARTS        AGE
continuous-image-puller-clf6d    1/1     Running   0               41h
continuous-image-puller-ldw6z    1/1     Running   0               41h
continuous-image-puller-mrlth    1/1     Running   0               41h
continuous-image-puller-pzcgk    1/1     Running   0               41h
continuous-image-puller-rb4m4    1/1     Running   0               41h
continuous-image-puller-vptfn    1/1     Running   0               41h
hub-6df78db8d8-49rlx             1/1     Running   1 (2m20s ago)   12m
proxy-75c547b6fb-kn7sl           1/1     Running   0               41h
user-scheduler-fd7895df5-9xkj4   1/1     Running   0               46h
user-scheduler-fd7895df5-fvn7m   1/1     Running   0               46h

so I checked the logs in hub pod, I got following msg , how can I solve this issue,

[I 2022-08-28 08:49:22.870 JupyterHub log:189] 302 GET /user/user/lab? -> /hub/user/user/lab? (@10.233.64.178) 1.72ms
[D 2022-08-28 08:49:23.087 JupyterHub base:328] Refreshing auth for admin
[W 2022-08-28 08:49:23.087 JupyterHub base:1449] User admin requested server for user, which they don't own
[I 2022-08-28 08:49:23.088 JupyterHub log:189] 302 GET /hub/user/user/lab? -> /user/admin/lab (admin@10.233.64.178) 44.28ms
[I 2022-08-28 08:49:23.292 JupyterHub log:189] 302 GET /user/admin/lab -> /hub/user/admin/lab (@10.233.64.178) 1.54ms
[D 2022-08-28 08:49:23.477 JupyterHub user:317] Creating <class 'kubespawner.spawner.KubeSpawner'> for admin:
[D 2022-08-28 08:49:23.483 JupyterHub spawner:194] Starting executor thread pool with 320 workers
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x72ca43fccbb0>, 'Connection to 10.233.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/jupyterhub/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x72ca43fccd90>, 'Connection to 10.233.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/jupyterhub/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x72ca43fccee0>, 'Connection to 10.233.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/jupyterhub/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server

thanks in advance
Sam

2

That looks like the hub pod isn’t able to talk to the k8s control plane. You can check if the network policy is to blame by setting hub.networkPolicy.enabled: false and see if that fixes it. Then we can investigate exactly what part of the policy config needs to be changed.