Connection error in Hub when deploy to on prem-k8s cluster

I deployed Zero to Jupyterhub in my on-premise k8s cluster, which has 3 nodes and deployed through kubespray. this is a existing k8s cluster , I was using MetalLB BPG mode , not the MetalLB layer 2 mode which was recommended by Zero to Jupyterhub

after I deployed jupyterhub, all pods works fine except hub pods restarts again and again

k get pods -n jupyterhub
NAME                             READY   STATUS    RESTARTS        AGE
continuous-image-puller-clf6d    1/1     Running   0               41h
continuous-image-puller-ldw6z    1/1     Running   0               41h
continuous-image-puller-mrlth    1/1     Running   0               41h
continuous-image-puller-pzcgk    1/1     Running   0               41h
continuous-image-puller-rb4m4    1/1     Running   0               41h
continuous-image-puller-vptfn    1/1     Running   0               41h
hub-6df78db8d8-49rlx             1/1     Running   1 (2m20s ago)   12m
proxy-75c547b6fb-kn7sl           1/1     Running   0               41h
user-scheduler-fd7895df5-9xkj4   1/1     Running   0               46h
user-scheduler-fd7895df5-fvn7m   1/1     Running   0               46h

so I checked the logs in hub pod, I got following msg , how can I solve this issue,

[I 2022-08-28 08:49:22.870 JupyterHub log:189] 302 GET /user/user/lab? -> /hub/user/user/lab? (@10.233.64.178) 1.72ms
[D 2022-08-28 08:49:23.087 JupyterHub base:328] Refreshing auth for admin
[W 2022-08-28 08:49:23.087 JupyterHub base:1449] User admin requested server for user, which they don't own
[I 2022-08-28 08:49:23.088 JupyterHub log:189] 302 GET /hub/user/user/lab? -> /user/admin/lab (admin@10.233.64.178) 44.28ms
[I 2022-08-28 08:49:23.292 JupyterHub log:189] 302 GET /user/admin/lab -> /hub/user/admin/lab (@10.233.64.178) 1.54ms
[D 2022-08-28 08:49:23.477 JupyterHub user:317] Creating <class 'kubespawner.spawner.KubeSpawner'> for admin:
[D 2022-08-28 08:49:23.483 JupyterHub spawner:194] Starting executor thread pool with 320 workers
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x72ca43fccbb0>, 'Connection to 10.233.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/jupyterhub/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x72ca43fccd90>, 'Connection to 10.233.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/jupyterhub/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x72ca43fccee0>, 'Connection to 10.233.0.1 timed out. (connect timeout=60)')': /api/v1/namespaces/jupyterhub/pods?fieldSelector=&labelSelector=component%3Dsingleuser-server

thanks in advance
Sam

That looks like the hub pod isn’t able to talk to the k8s control plane. You can check if the network policy is to blame by setting hub.networkPolicy.enabled: false and see if that fixes it. Then we can investigate exactly what part of the policy config needs to be changed.