Kernel not working using an on premise Kubernetes Cluster

JupyterHub Zero to JupyterHub on Kubernetes
1

Good morning,

I have deployed JupyterHub (following the Guide from Z2JH) in an on premise Kubernetes Cluster. It’s running Ubuntu 18.04.3 LTS and K8s version is 1.16.3. I’m using OpenEBS for PV and PVC. I create an Ingress for the Service, which I changed form LoadBalancer to ClusterIP.

When I try to run some Python code, it doesn’t do anything. I try to reconnect and restart the kernel, and it’s not working.

Where can I find more information about what’s happening? I check the logs from the pod itself and the hub’s pod and I cannot see anything.

Thank you so much,

2

Are there any errors in your browser’s javascript console? If you’re not familiar with the console see Major Browsers' Debugging Tools on https://www.w3schools.com/js/js_debugging.asp for instructions on how to open it.

3

Good morning,

As far as I can see, I discovered that the connection to the kernel is done through a WebSocket, which is different than http or https (returning HTTP 504). I’m using IngressController to expose proxy-public (http and https), but maybe I need to expose another think, or it will only work if using LoadBalancer Service.

Is it possible to configure the connection port with the kernel? Which additional things do I need to configure in Kubernetes?

Do you know what can I do?

Thanks,

4

It should “just work” if you’re using a standard ingress controller. Can you:

  • share your Z2JH configuration (blank out any secret tokens)
  • tell us more about your K8S cluster and how your ingress or loadbalancers are setup
  • show any errors or warnings in your JavaScript console?
5

My config.yaml is the same as following the guide of Z2JH. I have the secret and I configured the datascience-notebook as main image (tagging the most recent one) and opening lab page (not tree).

singleuser:
        extraEnv:
                EDITOR: "vim"
        defaultUrl: "/lab"
        image:
                name: jupyter/datascience-notebook
                tag: 7a0c7325e470
hub:
        extraConfig:
                jupyterlab: |
                        c.Spawner.cmd = ['jupyter-labhub']
proxy:
        secretToken: "<random-secret>"

My Kubernetes is v1.16.3. Is on premise and I cannot use LoadBalancer, so I changed the type of the service proxy-public to ClusterIP, to expose it via Ingress.

I created an Ingress (my controller is nginx (from nginxinc/kubernetes-ingress) to connect to the service proxy-public to port 80.

My Ingress file is the following one:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jhub-ingress
  namespace: jhub
spec:
  tls:
  - hosts:
    - <host>
  rules:
  - host: <host>
    http:
      paths:
      - path: /
        backend:
          serviceName: proxy-public
          servicePort: 80

In case it’s necessary, I can generate a SSL Certificate with Let’s Encrypt, but I don’t think it will work.

The only error in browser is that it cannot connect to kernel. I can log in, start a server (I also can shut it down), but when I want to execute code, it shows me a dialog that cannot connect to the kernel. And the request that connects to the kernel (which uses wss) returns HTTP 504. It tries to reconnect, but the same.

Logs from Firefox:

GET https://<host>/user/<username>/api/kernels/1d3340a0-8f4d-4d6a-8a9b-1c84293bd9e1/channels?session_id=b029b6b3-9d46-47ec-9c58-e9a2d697a4bf&token=45f2f1d6577b4ad19bfcad564f1faaf2 HTTP 504 Gateway Time-out

Starting WebSocket: wss://<host>/user/<username>/api/kernels/1d3340a0-8f4d-4d6a-8a9b-1c84293bd9e1

Firefox can't establish a connection to the server at wss://<host>/user/<username>/api/kernels/1d3340a0-8f4d-4d6a-8a9b-1c84293bd9e1/channels?session_id=ccc9f084-bce5-4f72-b998-451a9bfeca15&token=45f2f1d6577b4ad19bfcad564f1faaf2.  || default.js:57:58
Connection lost, reconnecting in 2 seconds || default.js:140:17

Is there some configuration to increase the verbosity?

Thank you so much,

6

It’s possible the nginxinc/kubernetes-ingress controller needs additional configuration to handle websockets. I’ve only used https://github.com/kubernetes/ingress-nginx/ which just works. If changing your ingress isn’t an option you’ll have to look into the documentation for nginxinc/kubernetes-ingress.

1 Like
7

Which additional configuration do you use?

I think I can change my IngressController, but I’ll first analyze the differences.

8

@Mr-Reca Were you able to resolve this issue? I am facing the same problem.

Thank you.

9

Agree. Issue still persist.
Using:
AWS EKS, v1.16.
JupyterHub Z2JH 0.10.4

Jupyters Proxy reconfigured as ClusterIP, used ingress from values.yaml with value=true without additional websocket annotations.

EDITED: just discovered, that connecting to proxy-public service (configured as ClusterIP) - no broken connection to kernel appears, everything works smooth. With ingress = true, kernel connection breaking.