Jupyterhub Not Connecting to Kernel and not Starting Terminal

JupyterHub Zero to JupyterHub on Kubernetes
1

We have a fresh install of jupyterhub using the guidelines on Zero to JupyterHub with Kubernetes — Zero to JupyterHub with Kubernetes documentation. We have configured quite aggressive. And I think this might be the conditions at this time.

The kernel says connecting, and then goes to disconnected, so no cell executions is possible. Following troubleshooting on What to do when things go wrong — Jupyter Notebook 6.5.2 documentation gave the following output:

$ jupyter kernelspec list
Available kernels:
  python3    /opt/conda/share/jupyter/kernels/python3

And the Terminal is just staying idle in the start. As in the screenshot.

Screen Shot 2022-11-09 at 09.53.33

My assumption is the changes to the rights here so we simply don’t have the rights to execute anything that the ops set up. If this is not the case, please point me in a direction to continue debugging. Here is the config changes I think might be the issue where we deny the terminal and python kernel to actually execute. Not sure what to allow here either.

...
  podSecurityContext:
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
    runAsNonRoot: true
...
  containerSecurityContext:
    runAsUser: 1000
    runAsGroup: 1000
    allowPrivilegeEscalation: false
    privileged: false
    capabilities:
      drop:
      - ALL
...
  chp:
    revisionHistoryLimit:
    containerSecurityContext:
      runAsUser: 65534 # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
      privileged: false
      capabilities:
        drop:
        - ALL
2

To help debug this problem can you start by using the default (or close to default) configuration, and once it’s working incrementatlly add your customisations back?

3

Make sure to check the pod’s logs, which will likely include some errors (and/or the browser console).

It may be relevant that both talking to a kernel and a terminal use websockets, so it seems likely that a proxy layer somewhere is not properly implementing websockets. Especially if you have your own firewall/proxy somewhere between you and jupyterhub.

4

Solution was to activate the websockets on our proxies.

1 Like
5

Hi there, I have the same problem you described, I can’t connect to any kernels nor the terminal, could you elaborate on how you solved the issue please ? what do you mean by “activate the websockets on our proxies” ?

Thanks in advance,

Naïm

6

I found the solution thanks to this SO post : Kubernetes ingress websockets connection issue - Stack Overflow

This is because I have a baremetal cluster and no loadbalancer and my service is using ClusterIP instead.
The ingress annotation must be modified to allow websockets connexions as described in the post.