No connectivity to user pods running on kubernetes workers

JupyterHub Zero to JupyterHub on Kubernetes
1

Hi,

I have deployed a kubernetes cluster with OpenStack magnum and deployed ztjh on it.

When user pods are scheduled on the kubernetes master they work as expected. However when they are scheduled on any kubernetes worker, the fail with:

[E 2020-11-23 13:05:27.771 SingleUserNotebookApp auth:334] Error connecting to http://hub:8081/hub/api: HTTPConnectionPool(host='hub', port=8081): Max retries exceeded with url: /hub/api/oauth2/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc5e27ff490>: Failed to establish a new connection: [Errno 110] Connection timed out'))
[W 2020-11-23 13:05:27.771 SingleUserNotebookApp web:1786] 500 GET /user/test11/oauth_callback?code=[...] Failed to connect to Hub API at 'http://hub:8081/hub/api'.  Is the Hub accessible at this URL (from host: jupyter-test11)?
[E 2020-11-23 13:05:27.804 SingleUserNotebookApp log:166] {
      "Cookie": "jupyterhub-user-test11-oauth-state=[secret]; jupyterhub-session-id=[secret]",
      "Upgrade-Insecure-Requests": "1",
      "Referer": "[...]",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-GB,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0",
      "X-Scheme": "https",
      "X-Original-Uri": "[],
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "<my-jhub-ip>",
      "X-Forwarded-For": "37.132.101.122,::ffff:192.168.100.127",
      "X-Real-Ip": "37.132.101.122",
      "X-Request-Id": "61c69eb973037bca51ba0a2044245b96",
      "Host": "<my-jhub-ip>",
      "Connection": "close"
    }

Any ideas how can I troubleshoot this further?

Best regards,
Sebastian

2

Hi Sebastian,
Which CNI are you using?
Last time, I also had the same problem deploying z2jh on OpenStack due to a CNI issue.
I used kubespray to deploy k8s cluster and used calico as CNI.
To fix that problem, I needed to allow the connection from kube_service_addresses and kube_pods_subnet, as describe in here:

Hope this helps,

Cheers,
MW.

3

Hi Mohammad,

I am using calico inside OpenStack magnum.

I went back to the logs:

Error connecting to http://hub:8081/hub/api: HTTPConnectionPool(host='hub', port=8081): Max retries exceeded with url: /hub/api/oauth2/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc5e27ff490>: Failed to establish a new connection: [Errno 110] Connection timed out'))

I then double checked and the issue was that port 8081 on the kubernetes master was not open so creating and assigning the proper security group solved the issue.

Many thanks,
Sebastian