# Kernel not working using an on premise Kubernetes Cluster

**URL:** <https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907>\
**Category:** Zero to JupyterHub on Kubernetes\
**Created:** [December 17, 2019, 10:36am UTC](https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907 "2019-12-17T10:36:56Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mr-Reca](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/mr-reca/32/1407_2.png) [@Mr-Reca](https://discourse.jupyter.org/u/Mr-Reca)\
**Post date:** [December 17, 2019, 10:36am UTC](https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907/1 "2019-12-17T10:36:56Z")

</div>

Good morning,

I have deployed JupyterHub (following the Guide from Z2JH) in an on premise Kubernetes Cluster. It’s running Ubuntu 18.04.3 LTS and K8s version is 1.16.3. I’m using OpenEBS for PV and PVC. I create an Ingress for the Service, which I changed form LoadBalancer to ClusterIP.

When I try to run some Python code, it doesn’t do anything. I try to reconnect and restart the kernel, and it’s not working.

Where can I find more information about what’s happening? I check the logs from the pod itself and the hub’s pod and I cannot see anything.

Thank you so much,

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 17, 2019, 9:54pm UTC](https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907/2 "2019-12-17T21:54:37Z")

</div>

Are there any errors in your browser’s javascript console? If you’re not familiar with the console see `Major Browsers' Debugging Tools` on [https://www.w3schools.com/js/js\_debugging.asp](https://www.w3schools.com/js/js_debugging.asp) for instructions on how to open it.

---

<div class="post-metadata">

**Author:** ![Mr-Reca](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/mr-reca/32/1407_2.png) [@Mr-Reca](https://discourse.jupyter.org/u/Mr-Reca)\
**Post date:** [December 18, 2019, 8:55am UTC](https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907/3 "2019-12-18T08:55:34Z")

</div>

Good morning,

As far as I can see, I discovered that the connection to the kernel is done through a WebSocket, which is different than http or https (returning HTTP 504). I’m using IngressController to expose proxy-public (http and https), but maybe I need to expose another think, or it will only work if using LoadBalancer Service.

Is it possible to configure the connection port with the kernel? Which additional things do I need to configure in Kubernetes?

Do you know what can I do?

Thanks,

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 18, 2019, 3:07pm UTC](https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907/4 "2019-12-18T15:07:40Z")

</div>

It should “just work” if you’re using a standard ingress controller. Can you:

- share your Z2JH configuration (blank out any secret tokens)
- tell us more about your K8S cluster and how your ingress or loadbalancers are setup
- show any errors or warnings in your JavaScript console?

---

<div class="post-metadata">

**Author:** ![Mr-Reca](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/mr-reca/32/1407_2.png) [@Mr-Reca](https://discourse.jupyter.org/u/Mr-Reca)\
**Post date:** [December 18, 2019, 10:25pm UTC](https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907/5 "2019-12-18T22:25:09Z")

</div>

My config.yaml is the same as following the guide of Z2JH. I have the secret and I configured the datascience-notebook as main image (tagging the most recent one) and opening lab page (not tree).

```auto
singleuser:
        extraEnv:
                EDITOR: "vim"
        defaultUrl: "/lab"
        image:
                name: jupyter/datascience-notebook
                tag: 7a0c7325e470
hub:
        extraConfig:
                jupyterlab: |
                        c.Spawner.cmd = ['jupyter-labhub']
proxy:
        secretToken: "<random-secret>"

```

My Kubernetes is v1.16.3. Is on premise and I cannot use LoadBalancer, so I changed the type of the service proxy-public to ClusterIP, to expose it via Ingress.

I created an Ingress (my controller is nginx (from nginxinc/kubernetes-ingress) to connect to the service proxy-public to port 80.

My Ingress file is the following one:

```auto
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jhub-ingress
  namespace: jhub
spec:
  tls:
  - hosts:
    - <host>
  rules:
  - host: <host>
    http:
      paths:
      - path: /
        backend:
          serviceName: proxy-public
          servicePort: 80

```

In case it’s necessary, I can generate a SSL Certificate with Let’s Encrypt, but I don’t think it will work.

The only error in browser is that it cannot connect to kernel. I can log in, start a server (I also can shut it down), but when I want to execute code, it shows me a dialog that cannot connect to the kernel. And the request that connects to the kernel (which uses wss) returns HTTP 504. It tries to reconnect, but the same.

Logs from Firefox:

```auto
GET https://<host>/user/<username>/api/kernels/1d3340a0-8f4d-4d6a-8a9b-1c84293bd9e1/channels?session_id=b029b6b3-9d46-47ec-9c58-e9a2d697a4bf&token=45f2f1d6577b4ad19bfcad564f1faaf2 HTTP 504 Gateway Time-out

Starting WebSocket: wss://<host>/user/<username>/api/kernels/1d3340a0-8f4d-4d6a-8a9b-1c84293bd9e1

Firefox can't establish a connection to the server at wss://<host>/user/<username>/api/kernels/1d3340a0-8f4d-4d6a-8a9b-1c84293bd9e1/channels?session_id=ccc9f084-bce5-4f72-b998-451a9bfeca15&token=45f2f1d6577b4ad19bfcad564f1faaf2. || default.js:57:58
Connection lost, reconnecting in 2 seconds || default.js:140:17

```

Is there some configuration to increase the verbosity?

Thank you so much,

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 19, 2019, 7:15pm UTC](https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907/6 "2019-12-19T19:15:16Z")

</div>

It’s possible the [`nginxinc/kubernetes-ingress` controller](https://github.com/nginxinc/kubernetes-ingress) needs additional configuration to handle websockets. I’ve only used [https://github.com/kubernetes/ingress-nginx/](https://github.com/kubernetes/ingress-nginx/) which just works. If changing your ingress isn’t an option you’ll have to look into the documentation for nginxinc/kubernetes-ingress.

---

<div class="post-metadata">

**Author:** ![Mr-Reca](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/mr-reca/32/1407_2.png) [@Mr-Reca](https://discourse.jupyter.org/u/Mr-Reca)\
**Post date:** [December 19, 2019, 8:22pm UTC](https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907/7 "2019-12-19T20:22:16Z")

</div>

Which additional configuration do you use?

I think I can change my IngressController, but I’ll first analyze the differences.

---

<div class="post-metadata">

**Author:** ![geeklisted](https://avatars.discourse-cdn.com/v4/letter/g/dbc845/32.png) [@geeklisted](https://discourse.jupyter.org/u/geeklisted)\
**Post date:** [November 13, 2020, 7:21pm UTC](https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907/8 "2020-11-13T19:21:50Z")

</div>

@Mr-Reca Were you able to resolve this issue? I am facing the same problem.

Thank you.

---

<div class="post-metadata">

**Author:** ![sentinel09](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/sentinel09/32/3152_2.png) [@sentinel09](https://discourse.jupyter.org/u/sentinel09)\
**Post date:** [November 29, 2020, 2:13am UTC](https://discourse.jupyter.org/t/kernel-not-working-using-an-on-premise-kubernetes-cluster/2907/9 "2020-11-29T02:13:43Z")

</div>

Agree. Issue still persist.  
Using:  
AWS EKS, v1.16.  
JupyterHub Z2JH 0.10.4

Jupyters Proxy reconfigured as ClusterIP, used ingress from values.yaml with value=true without additional websocket annotations.

EDITED: just discovered, that connecting to proxy-public service (configured as ClusterIP) - no broken connection to kernel appears, everything works smooth. With ingress = true, kernel connection breaking.