Websocket 400 connection failed

JupyterHub
1

Hi,

I have created a python application that uses Jupyterhub and has been deployed to Kubernetes

The JupyterHub setup is a combination of KubeSpawner and SAML authentication.

  1. I want to deploy the same exact application to multiple Kubernetes clusters (each Kubernetes cluster represents a different data environment - DEV / UAT / Staging / Production). To access these different Kubernetes clusters, the only difference in the code is the SAML metadata file (the internal kubernetes to external https:// mapping has a different base url). Let’s say for example we use the following to access the application externally:
  • DEV url → https:www.hello-dev.com/jupyterhub
  • UAT url → https:www.hello-uat.com/jupyterhub
  • STG url → https:www.hello-stg.com/jupyterhub
  • PRD url → https:www.hello-prd.com/jupyterhub
  1. My setup works without any issues when I log in via the DEV url above. This means that I can log in and see my single user pod in the DEV Kubernetes cluster. I can also do the following without any issues and use my application:
  • New → Terminal
  • New → Python3 kernel
  1. When I try with any of the other urls (UAT/STG/PRD) I can also login and see my single user pod in the correct Kubernetes cluster. However when I try the following, the Terminal shows a grey screen and the Python3 kernel will not connect
  • New → Terminal
  • New → Python3 kernel
  1. When I look at the webpage, right click, inspect - console I see the following for the Terminal and Python3 kernel (for example for the UAT url):
  • Terminal → Websocket connection to ‘wss://hello-uat.com/jupyterhub/user/XXXid/terminals/websocket/1’ … Uncaught TypeError: Cannot read property ‘parentElement’ of undefined
  • Python3 → Websocket connection failed to ‘wss://hello-uat.com/jupyterhub/user/XXXid/api/kernels/numbers/channels?session_id=numbers failed: kernel.start_channels’
  1. When I compare the launched single-user pod logs for the DEV/UAT urls, the only difference I see is when it tries to use the kernel (one is HTTP 101 and the other 400).

DEV → 101 GET /jupyterhub/user/XXXid/api/kernels/numbers/channels?session_id=numbers
UAT → 400 GET /jupyterhub/user/XXXid/api/kernels/numbers/channels?session_id=numbers

  1. I’ve checked with Proxy/AVI/Network/Kubernetes/SAML-SSO team members and they say that they cannot find anything strange

Can you think of anything from a JupyterHub perspective ? Is it not possible to use the same JupyterHub configuration from different Kubernetes machines ? Is there any additional configuration that I need to add to my code so that it doesn’t only work from the DEV url (but it can also work from UAT/STG/PRD) ?

Thank you very much

Alison

2

Are you using the https://zero-to-jupyterhub.readthedocs.io/ Helm chart or your own solution? Whether you can use an identical configuration or not depends on your setup. If you give us details of your Kubernetes cluster including any customisations, and show us your JupyterHub configuration files with secrets redacted, we might be able to help.

3

Hi manics,

Thank you very much for your help.

#########################

For this setup I have 2 connected Jenkinsfiles

1 Jenkinsfile, builds 2 docker images. 1 for the single-user pod and 1 for the jupyter hub that are pushed to artifactory.
The other Jenkinsfile, uses this finally combined docker image from artifactory and deploys it to the correct Kubernetes cluster using the Helm package manager

###########################

To build these single-pod and hub docker images, we cannot use the url you suggest (zero-to-jupyterhub). Instead we have to get our base docker image from our artifactory. The DevOps team are the ones in charge of fetching these external docker images and then adding them to our internal Artifactory.

###################################

The single-user Dockerfile used to create the single user docker image comes FROM a singleuser source from artifactory. Dockerfile key parts shown below.

FROM our-artifactory/jupyterhub/singleuser
ARG artifactory_username
ARG artifactory_password

ENV 1=value # these variables will have a different value depending on the Kubernetes cluster environment (DEV/UAT …)

RUN echo “[global]” > /tmp/pip.conf && set index-url to artifactory and trusted host using ARG passed

RUN pip3 install my_library (I have created a library the users can import and use from their own jupyter notebook after they have logged in via JupyterHub)

#############################

The central Hub Dockerfile:

FROM our-artifactory/jupyterhub/jupyterhub

ARG artifactory_username
ARG artifactory_password
ARG image_tag (this will be to use the correct single-user image number built via the single-user dockerfile)
ARG env (this will be used to use a different SAML login metadata file depending on the environment → different base_url)

RUN echo > /etc/pip.conf (set up pip.conf access index-url artifactory)

RUN pip3 install jupyterhub-samlauthenticator (install SAML authenticator)
RUN pip3 install jupyterhub-kubespawner (install kubespawner)

RUN jupyterhub --generate-config (generate jupyterhub_config.py)

1 basic setup

RUN echo “c.JupyterHub.ip=’'0.0.0.0” >> jupyterhub_config.py
RUN echo “c.JupyterHub.port=8888” >> jupyterhub_config.py
RUN echo
“”"
from jupyter_client.localinterfaces import public_ips
i[ = public_ips()[0]
c.JupyterHub.hub_ip = ip
“”" >> >> jupyterhub_config.py

2 authenticator

RUN echo “c.JuoyterHub.authenticator_clas =‘SAMLauthenticator’” >> jupyterhub_config.py
COPY metadata.xml /etc/jupyterhub
c.SAMLAUthenticaror.metadata_filepath= /etc/jupyterhub/metadata >> jupyterhub_config.py (this metadata file will depend on the ARG env)
c.SAMLAuthenticator.audience = SPID value >> jupyterhub_config.py (service provider identifier in the metadata file set by web sso team) - this SPID will depend on the ARG env

3 spawner

RUN echo “c.JupyterHub.spawner_class=KubeSpawner” >> jupyterhub_config.py
RUN echo “c.KubeSpawner.image_pull_secrets”=‘Kubernetes secret that allows to pull the single user docker image from artifactory’ >> jupyterhub_config.py
RUN echo “c.KubeSpawner.image=our-artifactory-name-of-single-user-image:{ARG tag version}”
RUN echo “c.KubeSpawner.start_timeout=60*5” >> jupyterhub_config.py
RUN echo “c.JupyterHub.active_user_limit=50” >> jupyterhub_config.py

4 persistent volume (data remains when you restart hub and single user pods)

RUN echo
“”"
c.KubeSpawner.volumes [{
‘name’: ‘storage-home’
persistentVolumeClaum: {claimName: storage}
}]
c.KubeSpawner.volume_mounts = [{
mountPath = /storage
name = ‘storage-home’
}]
“”" >> jupyterhub_config.py

#5 cull is user is idle (disconnect in 30 minutes)

RUN echo

“”"
s.JupyterHUb.services = [
name = ‘cull-idle’
admin: True
command: [
sys.executable, ‘-m’ ‘jupyterhub_idle_culler’
–timeout=1800
]
}
]
“”" >> jupyterhub_config.py

#6 start with their uid (unique identifier) rather than jovyan

RUN echo
“”"
def pre_spwan_hook(spawner):
username = spawner.user.name (from the SAML authentication)
uid = pwd.getpwnam(username).pw_uid
spawner.environement = {‘NB_USER’: username, ‘NB_UID’: str(uid)}
“”"

#########################################

Then we deploy this final docker image (Hub that uses single user image) to Kubernetes via the seconds Jenkinsfile

To deploy, we specify the url of the correct Kubernetes and the token to deploy (configuration details are in our Jeninsfile used to deploy).

Apart from the deployment.yaml, we have the following kustomisation (kubernetes objects):

  • An ingress.yaml → path is /jupyterhub and servicePort 8888 for deployment application name jupyterhub
  • A service.yaml → app jupyterhub and ports 8888
  • A role/rolebinding and service account
  • A PersistentVolumeClaim - name storage (ReadWriteMany with 200Gi)
  • The deployment.yaml -
    spec:containers: image → VERSION
    spec:containers:args → “bin/sh -c jupyterhub --debug --JupyterHub.base_url = ''jupyterhub”
    spec:containers:ports → 8888
    spec:containers:volumeMounts → storage mountPaths
4

This exact same code is used to deploy to the different Kubernetes clusters (the only difference is the ARG env (in the Hub Dockerfile), which will determine which metadata file/SPID to use to login). The difference in the metadata file is just a different base-url (hello-dev vs hello-uat for example)

Everything works via the DEV cluster url → https:www.hello-dev.com/jupyterhub (I can start a new Terminal or Python3 kernel and also import the library I’ve created and run it)

When I try the UAT url (https:www.hello-uat.com/jupyterhub), it does login via SAML and launch the single user-pod in the Kubernetes cluster. However, when I try a terminal/python3 I’m blocked by web sockets (the logs from my side are only showing that 101 to 400 difference when I compare DEV/UAT, I cannot see anything else)

I’m not sure how the Kubernetes apache servers have been configured, unfortunately. I have raised it with the team member that did this and he says everything is ok. I’ve also raised a ticket for the Kubernetes team and they say everything looks ok from their side as well. The web sso SAML team have also told me that they do not see any issues from their side

5

Please let me know if there is any other information that you may need

Thank you very much

Alison

6

This suggests JupyterHub is working fine, and the problem is mostly likely in your Kubernetes cluster or proxy, e.g. maybe one of your web proxies is not allowing websockets? Can you show us your browser console logs?

7

Thank you very much manics

I’ll send the Hub/Single-us{x}er pod and Console logs for the one that is not working

########################## HUB LOGS

[D 2021-05-25 07:29:36.752 JupyterHub application:730] Looking for jupyterhub_config in /srv/jupyterhub

[D 2021-05-25 07:29:37.338 JupyterHub application:752] Loaded config file: /srv/jupyterhub/jupyterhub_config.py

[W 2021-05-25 07:29:37.391 JupyterHub app:623] Both bind_url and ip/port/base_url have been configured. JupyterHub.ip, JupyterHub.port, JupyterHub.base_url are deprecated in JupyterHub 0.9, please use JupyterHub.bind_url instead.

[I 2021-05-25 07:29:37.392 JupyterHub app:2463] Running JupyterHub version 1.4.1

[I 2021-05-25 07:29:37.392 JupyterHub app:2493] Using Authenticator: samlauthenticator.samlauthenticator.SAMLAuthenticator

[I 2021-05-25 07:29:37.392 JupyterHub app:2493] Using Spawner: kubespawner.spawner.KubeSpawner-0.16.1

[I 2021-05-25 07:29:37.392 JupyterHub app:2493] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.4.1

[D 2021-05-25 07:29:37.401 JupyterHub app:1574] Generating new cookie_secret

[I 2021-05-25 07:29:37.401 JupyterHub app:1579] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret

[D 2021-05-25 07:29:37.402 JupyterHub app:1701] Connecting to db: sqlite:jupyterhub.sqlite

[D 2021-05-25 07:29:37.417 JupyterHub orm:784] Stamping empty database with alembic revision 4dc2d5a8c53c

[I 2021-05-25 07:29:37.421 alembic.runtime.migration migration:164] Context impl SQLiteImpl.

[I 2021-05-25 07:29:37.421 alembic.runtime.migration migration:167] Will assume non-transactional DDL.

[I 2021-05-25 07:29:37.435 alembic./runtime.migration migration:556] Running stamp_revision → 4dc2d5a8c53c

[D 2021-05-25 07:29:37.436 alembic.runtime.migration migration:752] new branch insert 4dc2d5a8c53c

[I 2021-05-25 07:29:37.565 JupyterHub proxy:497] Generating new CONFIGPROXY_AUTH_TOKEN

[W 2021-05-25 07:29:37.567 JupyterHub app:1808] No admin us{x}ers, admin interface will be unavailable.

[W 2021-05-25 07:29:37.567 JupyterHub app:1809] Add any administrative us{x}ers to c.Authenticator.admin_us{x}ers in config.

[I 2021-05-25 07:29:37.567 JupyterHub app:1838] Not using allowed_us{x}ers. Any authenticated us{x}er will be allowed.

[D 2021-05-25 07:29:37.627 JupyterHub app:1990] Purging expired APITokens

[D 2021-05-25 07:29:37.631 JupyterHub app:1990] Purging expired OAuthAccessTokens

[D 2021-05-25 07:29:37.634 JupyterHub app:1990] Purging expired OAuthCodes

[D 2021-05-25 07:29:37.647 JupyterHub app:2117] Initializing spawners

[D 2021-05-25 07:29:37.650 JupyterHub app:2250] Loaded us{x}ers:

[I 2021-05-25 07:29:37.650 JupyterHub app:2530] Initialized 0 spawners in 0.004 seconds

[W 2021-05-25 07:29:37.654 JupyterHub proxy:699] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else…

[I 2021-05-25 07:29:37.655 JupyterHub proxy:703] Starting proxy @ http:0.0.0.0:8888/jupyterhub/

[D 2021-05-25 07:29:37.655 JupyterHub proxy:704] Proxy cmd: [‘configurable-http-proxy’, ‘–ip’, ‘0.0.0.0’, ‘–port’, ‘8888’, ‘–api-ip’, ‘127.0.0.1’, ‘–api-port’, ‘8001’, ‘–error-target’, ‘http:99.198.17.231:8081/jupyterhub/hub/error’]

[D 2021-05-25 07:29:37.661 JupyterHub proxy:611] Writing proxy pid file: jupyterhub-proxy.pid

07:29:38.184 [ConfigProxy] info: Proxying http:0.0.0.0:8888 to (no default)

07:29:38.186 [ConfigProxy] info: Proxy API at http:127.0.0.1:8001/api/routes

[D 2021-05-25 07:29:38.606 JupyterHub proxy:739] Proxy started and appears to be up

[D 2021-05-25 07:29:38.607 JupyterHub proxy:832] Proxy: Fetching GET http:127.0.0.1:8001/api/routes

07:29:38.614 [ConfigProxy] info: 200 GET /api/routes

[I 2021-05-25 07:29:38.615 JupyterHub app:2778] Hub API listening on http:99.198.17.231:8081/jupyterhub/hub/

[I 2021-05-25 07:29:38.616 JupyterHub app:2793] Starting managed service cull-idle

[I 2021-05-25 07:29:38.616 JupyterHub service:339] Starting service ‘cull-idle’: [’/usr/bin/python3’, ‘-m’, ‘jupyterhub_idle_culler’, ‘–timeout=1800’]

[I 2021-05-25 07:29:38.618 JupyterHub service:121] Spawning /usr/bin/python3 -m jupyterhub_idle_culler --timeout=1800

[D 2021-05-25 07:29:38.623 JupyterHub spawner:1179] Polling subprocess every 30s

[D 2021-05-25 07:29:38.624 JupyterHub proxy:342] Fetching routes to check

[D 2021-05-25 07:29:38.624 JupyterHub proxy:832] Proxy: Fetching GET http:127.0.0.1:8001/api/routes

07:29:38.625 [ConfigProxy] info: 200 GET /api/routes

[I 2021-05-25 07:29:38.625 JupyterHub proxy:347] Checking routes

[I 2021-05-25 07:29:38.626 JupyterHub proxy:432] Adding route for Hub: /jupyterhub/ => http:99.198.17.231:8081

[D 2021-05-25 07:29:38.626 JupyterHub proxy:832] Proxy: Fetching POST http:127.0.0.1:8001/api/routes/jupyterhub

07:29:38.627 [ConfigProxy] info: Adding route /jupyterhub → http:99.198.17.231:8081

07:29:38.628 [ConfigProxy] info: Route added /jupyterhub → http:99.198.17.231:8081

07:29:38.629 [ConfigProxy] info: 201 POST /api/routes/jupyterhub

[I 2021-05-25 07:29:38.629 JupyterHub app:2853] JupyterHub is now running at http:0.0.0.0:8888/jupyterhub/

[D 2021-05-25 07:29:38.630 JupyterHub app:2456] It took 1.886 seconds for the Hub to start

[D 2021-05-25 07:29:38.740 JupyterHub base:283] Recording first activity for <APIToken(‘20cb…’, service=‘cull-idle’)>

[I 2021-05-25 07:29:38.752 JupyterHub log:189] 200 GET /jupyterhub/hub/api/us{x}ers (cull-idle@99.198.17.231) 18.32ms

[I 2021-05-25 07:29:42.066 JupyterHub log:189] 302 GET /jupyterhub → /jupyterhub/hub/ (@99.198.17.1) 0.77ms

[I 2021-05-25 07:29:42.071 JupyterHub log:189] 302 GET /jupyterhub/hub/ → /jupyterhub/hub/login?next=%2Fjupyterhub%2Fhub%2F (@99.198.17.1) 0.77ms

[I 2021-05-25 07:29:42.073 JupyterHub samlauthenticator:806] Starting SP-initiated SAML Login

[D 2021-05-25 07:29:42.074 JupyterHub samlauthenticator:711] Got metadata etree

[D 2021-05-25 07:29:42.074 JupyterHub samlauthenticator:717] Got valid metadata etree

[D 2021-05-25 07:29:42.074 JupyterHub samlauthenticator:723] Final xpath is: md:SingleSignOnService[@Binding=‘urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect’]/@Location

[I 2021-05-25 07:29:42.075 JupyterHub log:189] 302 GET /jupyterhub/hub/login?next=%2Fjupyterhub%2Fhub%2F → https:ssoforms.dev.echonet/affwebservices/public/saml2sso?SPID=gpf-relp-sit (@99.198.17.1) 1.65ms

[D 2021-05-25 07:31:43.248 JupyterHub reflector:358] pods watcher timeout

[D 2021-05-25 07:31:43.248 JupyterHub reflector:283] Connecting pods watcher

[D 2021-05-25 07:31:43.258 JupyterHub reflector:358] events watcher timeout

[D 2021-05-25 07:31:43.258 JupyterHub reflector:283] Connecting events watcher

[W 2021-05-25 07:31:43.283 JupyterHub base:1008] Us{x}er xxxUID is slow to start (timeout=10)

[D 2021-05-25 07:31:59.367 JupyterHub spawner:2480] pod default/jupyter-xxxUID events before launch: 2021-05-25T07:31:33.574997Z [Normal] Successfully assigned default/jupyter-xxxUID to eurvlii46983

2021-05-25T07:31:34Z [Normal] Pulling image “our-docker-artifactory/NAME:1.0.0-64”

2021-05-25T07:31:58Z [Normal] Successfully pulled image “our-docker-artifactory/NAME:1.0.0-64”

2021-05-25T07:31:58Z [Normal] Created container notebook

2021-05-25T07:31:58Z [Normal] Started container notebook

[D 2021-05-25 07:31:59.379 JupyterHub spawner:1179] Polling subprocess every 30s

[D 2021-05-25 07:31:59.983 JupyterHub utils:225] Server at http:99.198.10.151:8888/jupyterhub/us{x}er/xUID/ responded with 302

[D 2021-05-25 07:31:59.983 JupyterHub _version:76] jupyterhub and jupyterhub-singleus{x}er both on version 1.4.1

[I 2021-05-25 07:31:59.984 JupyterHub base:909] Us{x}er xUID took 26.702 seconds to start

8

CONTINUE HUB LOGS

[I 2021-05-25 07:31:59.984 JupyterHub proxy:285] Adding us{x}er xUID to proxy …/jupyterhub/us{x}er/xUID/ => …http:99.198.10.151:8888

[D 2021-05-25 07:31:59.984 JupyterHub proxy:832] Proxy: Fetching POST …http:127.0.0.1:8001/api/routes/jupyterhub/us{x}er/xUID

[I 2021-05-25 07:31:59.986 JupyterHub log:189] 200 GET …/jupyterhub/hub/api (@99.198.10.151) 1.08ms

07:31:59.986 [ConfigProxy] info: Adding route …/jupyterhub/us{x}er/xUID → http:99.198.10.151:8888

07:31:59.986 [ConfigProxy] info: Route added …/jupyterhub/us{x}er/xUID → http:99.198.10.151:8888

[I 2021-05-25 07:31:59.987 JupyterHub u{x}sers:677] Server xUID is ready

9

####### CONTINUE HUB LOGS

[I 2021-05-25 07:31:59.987 JupyterHub log:189] 200 GET …/jupyterhub/hub/api/us{x}ers/xUID/server/progress (xUID@99.198.17.0) 24655.00ms

07:31:59.986 [ConfigProxy] info: 201 POST …/api/routes/jupyterhub/us{x}er/xUID

[D 2021-05-25 07:31:59.994 JupyterHub base:283] Recording first activity for <APIToken(‘1552…’, us{x}er=‘xUID’)>

[D 2021-05-25 07:32:00.008 JupyterHub us{x}ers:778] Activity for us-x-er xUID: 2021-05-25T07:31:59.333581Z

[D 2021-05-25 07:32:00.008 JupyterHub us{x}ers:796] Activity on server xUID/: 2021-05-25T07:31:59.333581Z

[I 2021-05-25 07:32:00.029 JupyterHub log:189] 200 POST …/jupyterhub/hub/api/us-x-ers/xUID/activity (xUID@99.198.10.151) 37.73ms

10

CONTINUE HUB LOGS

[I 2021-05-25 07:32:00.034 JupyterHub log:189] 302 GET …/jupyterhub/hub/spawn-pending/xUID-> …/jupyterhub/us{x}er/xUID/ (xUID@99.198.17.0) 2.88ms

[D 2021-05-25 07:32:00.077 JupyterHub provider:409] Validating client id jupyterhub-us{x}er-xUID

[D 2021-05-25 07:32:00.078 JupyterHub provider:484] validate_redirect_uri: client_id=…jupyterhub-us{x}er-xUID, redirect_uri=…/jupyterhub/us{x}er/xUID/oauth_callback

[D 2021-05-25 07:32:00.079 JupyterHub auth:244] Skipping oauth confirmation for <Us{x}er(xUID1/1 running)> accessing Server at …/jupyterhub/us{x}er/xUID/

[D 2021-05-25 07:32:00.080 JupyterHub provider:409] Validating client id jupyterhub-us{x}er-xUID

11

CONTINUE HUB LOGS

[D 2021-05-25 07:32:00.080 JupyterHub provider:484] validate_redirect_uri: client_id=…/jupyterhub-us{x}er-xUID, redirect_uri=…/jupyterhub/us{x}er/xUID/oauth_callback

[D 2021-05-25 07:32:00.081 JupyterHub provider:231] Saving authorization code jupyterhub-us{x}er-xUID, kMM…,

[I 2021-05-25 07:32:00.094 JupyterHub log:189] 302 GET …/jupyterhub/hub/api/oauth2/authorize’?‘client_id=jupyterhub-us-x-er-xUID&redirect_uri=…’%‘2Fjupyter’%2F’us-x-er’%2F’xUID’%2F’oauth_callback’&‘response_type=code’&‘state=[secret] → …/jupyterhub/us{x}er/xUID/oauth_callback’?‘code=[secret]’&'state=[secret] (xUID@99.198.17.0) 18.45ms

12

CONTINUE HUB LOGS

[D 2021-05-25 07:32:00.116 JupyterHub provider:55] authenticate_client -oauthlib.Request SANITIZED-

[D 2021-05-25 07:32:00.122 JupyterHub provider:111] confirm_redirect_uri: client_id=…jupyterhub-us-x-er-xUID, redirect_uri=…/…jupyterhub/us-x-er/xUID/oauth_callback

[D 2021-05-25 07:32:00.122 JupyterHub provider:334] Saving bearer token {‘access_token’: ‘REDACTED’, ‘expires_in’: 1209600, ‘token_type’: ‘Bearer’, ‘scope’: ‘identify’, ‘refresh_token’: ‘REDACTED’}

[D 2021-05-25 07:32:00.134 JupyterHub provider:189] Deleting oauth code kMM… for jupyterhub-us-x-er-xUID

[I 2021-05-25 07:32:00.149 JupyterHub log:189] 200 POST …/…jupyterhub/hub/api/oauth2/token (xUID@99.198.10.151) 36.31ms

[I 2021-05-25 07:32:00.175 JupyterHub log:189] 200 GET …/…jupyterhub/hub/api/authorizations/token/[secret] (xUID@99.198.10.151) 21.88ms

13
SINGLE USER POD

Set username to: xUID

Relocating home dir to …/home/xUID

Setting CWD to …/home/xUID/

Executing the command: jupyterhub-singleuser --ip=0.0.0.0

[W 2021-05-25 08:16:05.524 SingleUserNotebookApp configurable:190] Config option open_browser not recognized by SingleUserNotebookApp. Did you mean browser?

[I 2021-05-25 08:16:05.582 SingleUserNotebookApp notebookapp:1593] Authentication of …/metrics is OFF, since other authentication is disabled.

[I 2021-05-25 08:16:06.207 SingleUserNotebookApp extension:162] JupyterLab extension loaded from …/opt/conda/lib/python3.8/site-packages/jupyterlab

[I 2021-05-25 08:16:06.207 SingleUserNotebookApp extension:163] JupyterLab application directory is …/opt/conda/share/jupyter/lab

[I 2021-05-25 08:16:06.225 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.1

[I 2021-05-25 08:16:06.231 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: …/nfs/xUID

[I 2021-05-25 08:16:06.231 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.3.0 is running at:

[I 2021-05-25 08:16:06.231 SingleUserNotebookApp notebookapp:2302] htp:jupyter-xUID:8888/jupyterhub/user/xUID

[I 2021-05-25 08:16:06.231 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[I 2021-05-25 08:16:06.235 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds

[I 2021-05-25 08:16:06.914 SingleUserNotebookApp log:189] 302 GET …/jupyterhub/user/xUID → …/jupyterhub/user/xUID/tree’?’ (at@99.198.17.231) 0.74ms

14

##CONTINUE SINGLE USER POD

[I 2021-05-25 08:16:06.958 SingleUserNotebookApp log:189] 302 GET …/jupyterhub/user/xUID/tree’?‘redirects=1 → …/jupyterhub/hub/api/oauth2/authorize’?‘client_id’=‘jupyterhub-user-xUID’&‘redirect_uri=’%2F’jupyterhub’%2F’user’%2F’xUID’%2F’oauth_callback’&‘response_type’=‘code’&‘state’=[secret]’ (at@99.198.31.0) 4.10ms

[I 2021-05-25 08:16:07.060 SingleUserNotebookApp auth:992] Logged-in user {‘kind’: ‘user’, ‘name’: ‘xUID’, ‘admin’: False, ‘groups’: [], ‘server’: ‘/jupyterhub/user/xUID/’, ‘pending’: None, ‘created’: ‘2021-05-25T07:31:33.184696Z’, ‘last_activity’: ‘2021-05-25T08:16:07.049165Z’, ‘servers’: None}

15

#CONTINUE SINGLE USER POD

[I 2021-05-25 08:16:07.061 SingleUserNotebookApp log:189] 302 GET …/jupyterhub/user/xUID/oauth_callback’?‘code=[secret]’&‘state=[secret] ‘->’ …/jupyterhub/user/xUID/tree’?'redirects=1 (at-99.198.31.0) 61.29ms

[I 2021-05-25 08:16:07.144 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/tree’?'redirects=1 (xUID-99.198.31.0) 47.03ms

[I 2021-05-25 08:16:07.263 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/config/tree’?’_=1621930567218 (xUID-99.198.31.0) 2.97ms

[I 2021-05-25 08:16:07.266 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/config/common’?_’=1621930567219 (xUID-99.198.26.0) 2.48ms

[I 2021-05-25 08:16:07.279 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/terminals’?_’=1621930567222 (xUID-99.198.31.0) 1.83ms

[I 2021-05-25 08:16:07.281 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/terminals’?_’=1621930567220 (xUID-99.198.16.0) 0.89ms

[I 2021-05-25 08:16:07.288 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/sessions’?_’=1621930567221 (xUID-99.198.17.0) 2.41ms

[I 2021-05-25 08:16:07.313 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/contents’?‘type=directory’&_’=1621930567223 (xUID-99.198.17.0) 11.80ms

WHEN I TRY TO OPEN A TERMINAL

[I 2021-05-25 08:23:33.175 SingleUserNotebookApp management:338] New terminal with automatic name: 1

[I 2021-05-25 08:23:33.177 SingleUserNotebookApp log:189] 200 POST /jupyterhub/user/xUID/api/terminals (xUID-99.198.21.0) 643.39ms

[I 2021-05-25 08:23:33.180 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/tree (xUID-99.198.30.0) 645.60ms

[I 2021-05-25 08:23:33.207 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/terminals/1 (xUID-99.198.21.0) 13.39ms

[I 2021-05-25 08:23:33.384 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/config/terminal?_=1621931013341 (xUID-99.198.21.0) 3.94ms

[I 2021-05-25 08:23:33.388 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/config/common?_=1621931013342 (xUID-99.198.17.0) 2.62ms

[W 2021-05-25 08:23:33.405 SingleUserNotebookApp log:189] 400 GET /jupyterhub/user/xUID/terminals/websocket/1 (xUID-99.198.5.0) 1.90ms

WHEN I TRY TO OPEN A PYTHON3 KERNEL (notebook Unititled.ipynb)

[I 2021-05-25 08:23:36.615 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/notebooks/Untitled.ipynb (xUID-99.198.30.0) 831.44ms

[I 2021-05-25 08:23:36.837 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/config/notebook?_=1621931016749 (xUID-99.198.5.0) 7.83ms

[I 2021-05-25 08:23:36.841 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/config/common?_=1621931016750 (xUID-99.198.30.0) 3.01ms

[I 2021-05-25 08:23:36.953 SingleUserNotebookApp sign:396] Writing notebook-signing key to /home/xUID/.local/share/jupyter/notebook_secret

[W 2021-05-25 08:23:36.953 SingleUserNotebookApp manager:510] Notebook Untitled.ipynb is not trusted

[I 2021-05-25 08:23:36.955 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/contents/Untitled.ipynb’?‘type=notebook’&’_=1621931016751 (xUID-99.198.30.0) 46.76ms

[I 2021-05-25 08:23:37.034 SingleUserNotebookApp kernelmanager:179] Kernel started: c7ef37da-4ee4-435d-adea-6157222d58e7, name: python3

[I 2021-05-25 08:23:37.043 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/contents/Untitled.ipynb/checkpoints’?’_=1621931016752 (xUID-99.198.17.0) 7.55ms

[I 2021-05-25 08:23:37.048 SingleUserNotebookApp log:189] 201 POST /jupyterhub/user/xUID/api/sessions (xUID-99.198.5.0) 45.13ms

[I 2021-05-25 08:23:37.050 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/metrics/v1 (xUID-99.198.21.0) 4.14ms

[W 2021-05-25 08:23:37.536 SingleUserNotebookApp log:189] 400 GET /jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7/channels?session_id=a4775ba1ca734ce98dc511094c4efcf5 (xUID-99.198.5.0) 450.91ms

[W 2021-05-25 08:23:38.555 SingleUserNotebookApp handlers:366] Replacing stale connection: c7ef37da-4ee4-435d-adea-6157222d58e7:a4775ba1ca734ce98dc511094c4efcf5

16

CONSOLE LOGS - PYTHON3 KERNEL

menubar.js:282 actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later…
MenuBar.bind_events @ menubar.js:282
utils.js:60 load_extensions Arguments(2)
bidi.js:19 Loaded moment locale en-gb
utils.js:37 Loading extension: nbresuse/main
utils.js:37 Loading extension: jupyter-js-widgets/extension
session.js:54 Session: kernel_created (f597f87e-14d2-4e24-b99c-e2676621eba2)
kernel.js:463 Starting WebSockets: wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7
kernel.js:465 **WebSocket connection to ‘wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7/channels?session_id=a4775ba1ca734ce98dc511094c4efcf5’ failed: **
Kernel.start_channels @ kernel.js:465
kernel.js:106 Kernel: kernel_disconnected (c7ef37da-4ee4-435d-adea-6157222d58e7)
kernel.js:546 WebSocket connection failed: wss://hello–uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7 true
kernel.js:564 Connection lost, reconnecting in 1 seconds.
kernel.js:474 WebSocket closed early CloseEvent
kernel.js:106 Kernel: kernel_reconnecting (c7ef37da-4ee4-435d-adea-6157222d58e7)
kernel.js:463 Starting WebSockets: wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7
kernel.js:465 WebSocket connection to ‘wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7/channels?session_id=a4775ba1ca734ce98dc511094c4efcf5’ failed:
Kernel.start_channels @ kernel.js:465
kernel.js:106 Kernel: kernel_disconnected (c7ef37da-4ee4-435d-adea-6157222d58e7)
kernel.js:546 WebSocket connection failed: wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7 true
kernel.js:564 Connection lost, reconnecting in 2 seconds.
kernel.js:493 WebSocket closed unexpectedly CloseEvent
kernel.js:106 Kernel: kernel_reconnecting (c7ef37da-4ee4-435d-adea-6157222d58e7)
kernel.js:463 Starting WebSockets: wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7
kernel.js:465 WebSocket connection to ‘wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7/channels?session_id=a4775ba1ca734ce98dc511094c4efcf5’ failed:
Kernel.start_channels @ kernel.js:465
kernel.js:106 Kernel: kernel_disconnected (c7ef37da-4ee4-435d-adea-6157222d58e7)

17

CONSOLE LOGS TERMINAL

terminado.js:4 WebSocket connection to ‘wss://hello-uat/jupyterhub/user/xUID/terminals/websocket/1’ failed:
make_terminal @ terminado.js:4
index.js:5 Uncaught TypeError: Cannot read property ‘parentElement’ of undefined
at proposeGeometry (index.js:5)
at fit (index.js:30)
at Terminal.terminalConstructor.fit (index.js:44)
at window.onresize (main.js:54)

18

I’ve asked the Kubernetes team to send me the /etc/nginx/nginx.conf file for both Kubernetes clusters. Can you think of anything else that I should ask for ?

Thank you very much

Alison

19

Since the error is on websockets, it is almost certainly a proxy layer somewhere that doesn’t support websockets, which is different on the two clusters (i.e. between you and any JupyterHub’s proxy-public entrypoint, rather than any part of JupyterHub itself or any of JupyterHub’s own configuration). This may be a firewall somewhere or ingress-controller layer on the cluster.

It could even be a proxy that runs on your computer or local network, if you have any proxy things that might apply different filtering to those domains.

20

Thank you very much minrk

I’ve asked the proxy team/firewall teams again to see if they can find something

I’ve compared the nginx.conf Kubernetes files for the DEV vs UAT cluster to understand the differences, but I cannot find anything that looks strange there