# Websocket 400 connection failed

**URL:** <https://discourse.jupyter.org/t/websocket-400-connection-failed/9289>\
**Category:** JupyterHub\
**Created:** [May 21, 2021, 5:40pm UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289 "2021-05-21T17:40:27Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 21, 2021, 5:40pm UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/1 "2021-05-21T17:40:27Z")

</div>

Hi,

I have created a python application that uses Jupyterhub and has been deployed to Kubernetes

The JupyterHub setup is a combination of KubeSpawner and SAML authentication.

1. I want to deploy the same exact application to multiple Kubernetes clusters (each Kubernetes cluster represents a different data environment - DEV / UAT / Staging / Production). To access these different Kubernetes clusters, the only difference in the code is the SAML metadata file (the internal kubernetes to external https:// mapping has a different base url). Let’s say for example we use the following to access the application externally:

- DEV url → https:www.hello-dev.com/jupyterhub
- UAT url → https:www.hello-uat.com/jupyterhub
- STG url → https:www.hello-stg.com/jupyterhub
- PRD url → https:www.hello-prd.com/jupyterhub

1. My setup works without any issues when I log in via the DEV url above. This means that I can log in and see my single user pod in the DEV Kubernetes cluster. I can also do the following without any issues and use my application:

- New → Terminal
- New → Python3 kernel

1. When I try with any of the other urls (UAT/STG/PRD) I can also login and see my single user pod in the correct Kubernetes cluster. However when I try the following, the Terminal shows a grey screen and the Python3 kernel will not connect

- New → Terminal
- New → Python3 kernel

1. When I look at the webpage, right click, inspect - console I see the following for the Terminal and Python3 kernel (for example for the UAT url):

- Terminal → Websocket connection to ‘wss://hello-uat.com/jupyterhub/user/XXXid/terminals/websocket/1’ … Uncaught TypeError: Cannot read property ‘parentElement’ of undefined
- Python3 → Websocket connection failed to ‘wss://hello-uat.com/jupyterhub/user/XXXid/api/kernels/numbers/channels?session\_id=numbers failed: kernel.start\_channels’

1. When I compare the launched single-user pod logs for the DEV/UAT urls, the only difference I see is when it tries to use the kernel (one is HTTP 101 and the other 400).

DEV → 101 GET /jupyterhub/user/XXXid/api/kernels/numbers/channels?session\_id=numbers  
UAT → 400 GET /jupyterhub/user/XXXid/api/kernels/numbers/channels?session\_id=numbers

1. I’ve checked with Proxy/AVI/Network/Kubernetes/SAML-SSO team members and they say that they cannot find anything strange

Can you think of anything from a JupyterHub perspective ? Is it not possible to use the same JupyterHub configuration from different Kubernetes machines ? Is there any additional configuration that I need to add to my code so that it doesn’t only work from the DEV url (but it can also work from UAT/STG/PRD) ?

Thank you very much

Alison

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [May 21, 2021, 7:30pm UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/2 "2021-05-21T19:30:27Z")

</div>

Are you using the [https://zero-to-jupyterhub.readthedocs.io/](https://zero-to-jupyterhub.readthedocs.io/) Helm chart or your own solution? Whether you can use an identical configuration or not depends on your setup. If you give us details of your Kubernetes cluster including any customisations, and show us your JupyterHub configuration files with secrets redacted, we might be able to help.

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 22, 2021, 7:03am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/3 "2021-05-22T07:03:37Z")

</div>

Hi manics,

Thank you very much for your help.

#########################

For this setup I have 2 connected Jenkinsfiles

1 Jenkinsfile, builds 2 docker images. 1 for the single-user pod and 1 for the jupyter hub that are pushed to artifactory.  
The other Jenkinsfile, uses this finally combined docker image from artifactory and deploys it to the correct Kubernetes cluster using the Helm package manager

###########################

To build these single-pod and hub docker images, we cannot use the url you suggest (zero-to-jupyterhub). Instead we have to get our base docker image from our artifactory. The DevOps team are the ones in charge of fetching these external docker images and then adding them to our internal Artifactory.

###################################

The single-user Dockerfile used to create the single user docker image comes FROM a singleuser source from artifactory. Dockerfile key parts shown below.

FROM our-artifactory/jupyterhub/singleuser  
ARG artifactory\_username  
ARG artifactory\_password

ENV 1=value # these variables will have a different value depending on the Kubernetes cluster environment (DEV/UAT …)

RUN echo “[global]” \> /tmp/pip.conf && set index-url to artifactory and trusted host using ARG passed

RUN pip3 install my\_library (I have created a library the users can import and use from their own jupyter notebook after they have logged in via JupyterHub)

#############################

The central Hub Dockerfile:

FROM our-artifactory/jupyterhub/jupyterhub

ARG artifactory\_username  
ARG artifactory\_password  
ARG image\_tag (this will be to use the correct single-user image number built via the single-user dockerfile)  
ARG env (this will be used to use a different SAML login metadata file depending on the environment → different base\_url)

RUN echo \> /etc/pip.conf (set up pip.conf access index-url artifactory)

RUN pip3 install jupyterhub-samlauthenticator (install SAML authenticator)  
RUN pip3 install jupyterhub-kubespawner (install kubespawner)

RUN jupyterhub --generate-config (generate jupyterhub\_config.py)

# 1 basic setup

RUN echo “c.JupyterHub.ip=''0.0.0.0” \>\> jupyterhub\_config.py  
RUN echo “c.JupyterHub.port=8888” \>\> jupyterhub\_config.py  
RUN echo  
“”"  
from jupyter\_client.localinterfaces import public\_ips  
i[= public\_ips()[0]  
c.JupyterHub.hub\_ip = ip  
“”" \>\> \>\> jupyterhub\_config.py

# 2 authenticator

RUN echo “c.JuoyterHub.authenticator\_clas =‘SAMLauthenticator’” \>\> jupyterhub\_config.py  
COPY metadata.xml /etc/jupyterhub  
c.SAMLAUthenticaror.metadata\_filepath= /etc/jupyterhub/metadata \>\> jupyterhub\_config.py (this metadata file will depend on the ARG env)  
c.SAMLAuthenticator.audience = SPID value \>\> jupyterhub\_config.py (service provider identifier in the metadata file set by web sso team) - this SPID will depend on the ARG env

# 3 spawner

RUN echo “c.JupyterHub.spawner\_class=KubeSpawner” \>\> jupyterhub\_config.py  
RUN echo “c.KubeSpawner.image\_pull\_secrets”=‘Kubernetes secret that allows to pull the single user docker image from artifactory’ \>\> jupyterhub\_config.py  
RUN echo “c.KubeSpawner.image=our-artifactory-name-of-single-user-image:{ARG tag version}”  
RUN echo “c.KubeSpawner.start\_timeout=60\*5” \>\> jupyterhub\_config.py  
RUN echo “c.JupyterHub.active\_user\_limit=50” \>\> jupyterhub\_config.py

# 4 persistent volume (data remains when you restart hub and single user pods)

RUN echo  
“”"  
c.KubeSpawner.volumes [{  
‘name’: ‘storage-home’  
persistentVolumeClaum: {claimName: storage}  
}]  
c.KubeSpawner.volume\_mounts = [{  
mountPath = /storage  
name = ‘storage-home’  
}]  
“”" \>\> jupyterhub\_config.py

#5 cull is user is idle (disconnect in 30 minutes)

RUN echo

“”"  
s.JupyterHUb.services = [  
name = ‘cull-idle’  
admin: True  
command: [  
sys.executable, ‘-m’ ‘jupyterhub\_idle\_culler’  
–timeout=1800  
]  
}  
]  
“”" \>\> jupyterhub\_config.py

#6 start with their uid (unique identifier) rather than jovyan

RUN echo  
“”"  
def pre\_spwan\_hook(spawner):  
username = spawner.user.name (from the SAML authentication)  
uid = pwd.getpwnam(username).pw\_uid  
spawner.environement = {‘NB\_USER’: username, ‘NB\_UID’: str(uid)}  
“”"

#########################################

Then we deploy this final docker image (Hub that uses single user image) to Kubernetes via the seconds Jenkinsfile

To deploy, we specify the url of the correct Kubernetes and the token to deploy (configuration details are in our Jeninsfile used to deploy).

Apart from the deployment.yaml, we have the following kustomisation (kubernetes objects):

- An ingress.yaml → path is /jupyterhub and servicePort 8888 for deployment application name jupyterhub
- A service.yaml → app jupyterhub and ports 8888
- A role/rolebinding and service account
- A PersistentVolumeClaim - name storage (ReadWriteMany with 200Gi)
- The deployment.yaml -  
spec:containers: image → VERSION  
spec:containers:args → “bin/sh -c jupyterhub --debug --JupyterHub.base\_url = ''jupyterhub”  
spec:containers:ports → 8888  
spec:containers:volumeMounts → storage mountPaths

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 22, 2021, 7:15am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/4 "2021-05-22T07:15:26Z")

</div>

This exact same code is used to deploy to the different Kubernetes clusters (the only difference is the ARG env (in the Hub Dockerfile), which will determine which metadata file/SPID to use to login). The difference in the metadata file is just a different base-url (hello-dev vs hello-uat for example)

Everything works via the DEV cluster url → https:www.hello-dev.com/jupyterhub (I can start a new Terminal or Python3 kernel and also import the library I’ve created and run it)

When I try the UAT url (https:www.hello-uat.com/jupyterhub), it does login via SAML and launch the single user-pod in the Kubernetes cluster. However, when I try a terminal/python3 I’m blocked by web sockets (the logs from my side are only showing that 101 to 400 difference when I compare DEV/UAT, I cannot see anything else)

I’m not sure how the Kubernetes apache servers have been configured, unfortunately. I have raised it with the team member that did this and he says everything is ok. I’ve also raised a ticket for the Kubernetes team and they say everything looks ok from their side as well. The web sso SAML team have also told me that they do not see any issues from their side

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 22, 2021, 7:33am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/5 "2021-05-22T07:33:35Z")

</div>

Please let me know if there is any other information that you may need

Thank you very much

Alison

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [May 24, 2021, 7:35pm UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/6 "2021-05-24T19:35:05Z")

</div>

> [@alison](#):
>
> When I try the UAT url (https:www.hello-uat.com/jupyterhub), it does login via SAML and launch the single user-pod in the Kubernetes cluster. However, when I try a terminal/python3 I’m blocked by web sockets (the logs from my side are only showing that 101 to 400 difference when I compare DEV/UAT, I cannot see anything else)

This suggests JupyterHub is working fine, and the problem is mostly likely in your Kubernetes cluster or proxy, e.g. maybe one of your web proxies is not allowing websockets? Can you show us your browser console logs?

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 25, 2021, 8:04am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/7 "2021-05-25T08:04:18Z")

</div>

Thank you very much manics

I’ll send the Hub/Single-us{x}er pod and Console logs for the one that is not working

########################## HUB LOGS

[D 2021-05-25 07:29:36.752 JupyterHub application:730] Looking for jupyterhub\_config in /srv/jupyterhub

[D 2021-05-25 07:29:37.338 JupyterHub application:752] Loaded config file: /srv/jupyterhub/jupyterhub\_config.py

[W 2021-05-25 07:29:37.391 JupyterHub app:623] Both bind\_url and ip/port/base\_url have been configured. JupyterHub.ip, JupyterHub.port, JupyterHub.base\_url are deprecated in JupyterHub 0.9, please use JupyterHub.bind\_url instead.

[I 2021-05-25 07:29:37.392 JupyterHub app:2463] Running JupyterHub version 1.4.1

[I 2021-05-25 07:29:37.392 JupyterHub app:2493] Using Authenticator: samlauthenticator.samlauthenticator.SAMLAuthenticator

[I 2021-05-25 07:29:37.392 JupyterHub app:2493] Using Spawner: kubespawner.spawner.KubeSpawner-0.16.1

[I 2021-05-25 07:29:37.392 JupyterHub app:2493] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.4.1

[D 2021-05-25 07:29:37.401 JupyterHub app:1574] Generating new cookie\_secret

[I 2021-05-25 07:29:37.401 JupyterHub app:1579] Writing cookie\_secret to /srv/jupyterhub/jupyterhub\_cookie\_secret

[D 2021-05-25 07:29:37.402 JupyterHub app:1701] Connecting to db: sqlite:jupyterhub.sqlite

[D 2021-05-25 07:29:37.417 JupyterHub orm:784] Stamping empty database with alembic revision 4dc2d5a8c53c

[I 2021-05-25 07:29:37.421 alembic.runtime.migration migration:164] Context impl SQLiteImpl.

[I 2021-05-25 07:29:37.421 alembic.runtime.migration migration:167] Will assume non-transactional DDL.

[I 2021-05-25 07:29:37.435 alembic./runtime.migration migration:556] Running stamp\_revision → 4dc2d5a8c53c

[D 2021-05-25 07:29:37.436 alembic.runtime.migration migration:752] new branch insert 4dc2d5a8c53c

[I 2021-05-25 07:29:37.565 JupyterHub proxy:497] Generating new CONFIGPROXY\_AUTH\_TOKEN

[W 2021-05-25 07:29:37.567 JupyterHub app:1808] No admin us{x}ers, admin interface will be unavailable.

[W 2021-05-25 07:29:37.567 JupyterHub app:1809] Add any administrative us{x}ers to `c.Authenticator.admin_us{x}ers` in config.

[I 2021-05-25 07:29:37.567 JupyterHub app:1838] Not using allowed\_us{x}ers. Any authenticated us{x}er will be allowed.

[D 2021-05-25 07:29:37.627 JupyterHub app:1990] Purging expired APITokens

[D 2021-05-25 07:29:37.631 JupyterHub app:1990] Purging expired OAuthAccessTokens

[D 2021-05-25 07:29:37.634 JupyterHub app:1990] Purging expired OAuthCodes

[D 2021-05-25 07:29:37.647 JupyterHub app:2117] Initializing spawners

[D 2021-05-25 07:29:37.650 JupyterHub app:2250] Loaded us{x}ers:

[I 2021-05-25 07:29:37.650 JupyterHub app:2530] Initialized 0 spawners in 0.004 seconds

[W 2021-05-25 07:29:37.654 JupyterHub proxy:699] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else…

[I 2021-05-25 07:29:37.655 JupyterHub proxy:703] Starting proxy @ http:0.0.0.0:8888/jupyterhub/

[D 2021-05-25 07:29:37.655 JupyterHub proxy:704] Proxy cmd: [‘configurable-http-proxy’, ‘–ip’, ‘0.0.0.0’, ‘–port’, ‘8888’, ‘–api-ip’, ‘127.0.0.1’, ‘–api-port’, ‘8001’, ‘–error-target’, ‘http:99.198.17.231:8081/jupyterhub/hub/error’]

[D 2021-05-25 07:29:37.661 JupyterHub proxy:611] Writing proxy pid file: jupyterhub-proxy.pid

07:29:38.184 [ConfigProxy] info: Proxying http:0.0.0.0:8888 to (no default)

07:29:38.186 [ConfigProxy] info: Proxy API at http:127.0.0.1:8001/api/routes

[D 2021-05-25 07:29:38.606 JupyterHub proxy:739] Proxy started and appears to be up

[D 2021-05-25 07:29:38.607 JupyterHub proxy:832] Proxy: Fetching GET http:127.0.0.1:8001/api/routes

07:29:38.614 [ConfigProxy] info: 200 GET /api/routes

[I 2021-05-25 07:29:38.615 JupyterHub app:2778] Hub API listening on http:99.198.17.231:8081/jupyterhub/hub/

[I 2021-05-25 07:29:38.616 JupyterHub app:2793] Starting managed service cull-idle

[I 2021-05-25 07:29:38.616 JupyterHub service:339] Starting service ‘cull-idle’: [’/usr/bin/python3’, ‘-m’, ‘jupyterhub\_idle\_culler’, ‘–timeout=1800’]

[I 2021-05-25 07:29:38.618 JupyterHub service:121] Spawning /usr/bin/python3 -m jupyterhub\_idle\_culler --timeout=1800

[D 2021-05-25 07:29:38.623 JupyterHub spawner:1179] Polling subprocess every 30s

[D 2021-05-25 07:29:38.624 JupyterHub proxy:342] Fetching routes to check

[D 2021-05-25 07:29:38.624 JupyterHub proxy:832] Proxy: Fetching GET http:127.0.0.1:8001/api/routes

07:29:38.625 [ConfigProxy] info: 200 GET /api/routes

[I 2021-05-25 07:29:38.625 JupyterHub proxy:347] Checking routes

[I 2021-05-25 07:29:38.626 JupyterHub proxy:432] Adding route for Hub: /jupyterhub/ =\> http:99.198.17.231:8081

[D 2021-05-25 07:29:38.626 JupyterHub proxy:832] Proxy: Fetching POST http:127.0.0.1:8001/api/routes/jupyterhub

07:29:38.627 [ConfigProxy] info: Adding route /jupyterhub → http:99.198.17.231:8081

07:29:38.628 [ConfigProxy] info: Route added /jupyterhub → http:99.198.17.231:8081

07:29:38.629 [ConfigProxy] info: 201 POST /api/routes/jupyterhub

[I 2021-05-25 07:29:38.629 JupyterHub app:2853] JupyterHub is now running at **http:0.0.0.0:8888/jupyterhub/**

[D 2021-05-25 07:29:38.630 JupyterHub app:2456] It took 1.886 seconds for the Hub to start

[D 2021-05-25 07:29:38.740 JupyterHub base:283] Recording first activity for \<APIToken(‘20cb…’, service=‘cull-idle’)\>

[I 2021-05-25 07:29:38.752 JupyterHub log:189] 200 GET /jupyterhub/hub/api/us{x}ers ([cull-idle@99.198.17.231](mailto:cull-idle@99.198.17.231)) 18.32ms

[I 2021-05-25 07:29:42.066 JupyterHub log:189] 302 GET /jupyterhub → /jupyterhub/hub/ (@99.198.17.1) 0.77ms

[I 2021-05-25 07:29:42.071 JupyterHub log:189] 302 GET /jupyterhub/hub/ → /jupyterhub/hub/login?next=%2Fjupyterhub%2Fhub%2F (@99.198.17.1) 0.77ms

[I 2021-05-25 07:29:42.073 JupyterHub samlauthenticator:806] Starting SP-initiated SAML Login

[D 2021-05-25 07:29:42.074 JupyterHub samlauthenticator:711] Got metadata etree

[D 2021-05-25 07:29:42.074 JupyterHub samlauthenticator:717] Got valid metadata etree

[D 2021-05-25 07:29:42.074 JupyterHub samlauthenticator:723] Final xpath is: md:SingleSignOnService[@Binding=‘urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect’]/@Location

[I 2021-05-25 07:29:42.075 JupyterHub log:189] 302 GET /jupyterhub/hub/login?next=%2Fjupyterhub%2Fhub%2F → https:ssoforms.dev.echonet/affwebservices/public/saml2sso?SPID=gpf-relp-sit (@99.198.17.1) 1.65ms

[D 2021-05-25 07:31:43.248 JupyterHub reflector:358] pods watcher timeout

[D 2021-05-25 07:31:43.248 JupyterHub reflector:283] Connecting pods watcher

[D 2021-05-25 07:31:43.258 JupyterHub reflector:358] events watcher timeout

[D 2021-05-25 07:31:43.258 JupyterHub reflector:283] Connecting events watcher

[W 2021-05-25 07:31:43.283 JupyterHub base:1008] Us{x}er **xxxUID** is slow to start (timeout=10)

[D 2021-05-25 07:31:59.367 JupyterHub spawner:2480] pod default/jupyter-xxxUID events before launch: 2021-05-25T07:31:33.574997Z [Normal] Successfully assigned default/jupyter-xxxUID to eurvlii46983

2021-05-25T07:31:34Z [Normal] Pulling image “our-docker-artifactory/NAME:1.0.0-64”

2021-05-25T07:31:58Z [Normal] Successfully pulled image “our-docker-artifactory/NAME:1.0.0-64”

2021-05-25T07:31:58Z [Normal] Created container notebook

2021-05-25T07:31:58Z [Normal] Started container notebook

[D 2021-05-25 07:31:59.379 JupyterHub spawner:1179] Polling subprocess every 30s

[D 2021-05-25 07:31:59.983 JupyterHub utils:225] Server at http:99.198.10.151:8888/jupyterhub/us{x}er/xUID/ responded with 302

[D 2021-05-25 07:31:59.983 JupyterHub \_version:76] jupyterhub and jupyterhub-singleus{x}er both on version 1.4.1

[I 2021-05-25 07:31:59.984 JupyterHub base:909] Us{x}er xUID took 26.702 seconds to start

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 25, 2021, 8:09am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/8 "2021-05-25T08:09:08Z")

</div>

#### CONTINUE HUB LOGS

[I 2021-05-25 07:31:59.984 JupyterHub proxy:285] Adding us{x}er xUID to proxy …/jupyterhub/us{x}er/xUID/ =\> ..http:99.198.10.151:8888

[D 2021-05-25 07:31:59.984 JupyterHub proxy:832] Proxy: Fetching POST ..http:127.0.0.1:8001/api/routes/jupyterhub/us{x}er/xUID

[I 2021-05-25 07:31:59.986 JupyterHub log:189] 200 GET …/jupyterhub/hub/api (@99.198.10.151) 1.08ms

07:31:59.986 [ConfigProxy] info: Adding route …/jupyterhub/us{x}er/xUID → http:99.198.10.151:8888

07:31:59.986 [ConfigProxy] info: Route added …/jupyterhub/us{x}er/xUID → http:99.198.10.151:8888

[I 2021-05-25 07:31:59.987 JupyterHub u{x}sers:677] Server xUID is ready

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 25, 2021, 8:11am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/9 "2021-05-25T08:11:48Z")

</div>

####### CONTINUE HUB LOGS

[I 2021-05-25 07:31:59.987 JupyterHub log:189] 200 GET …/jupyterhub/hub/api/us{x}ers/xUID/server/progress ([xUID@99.198.17.0](mailto:xUID@99.198.17.0)) 24655.00ms

07:31:59.986 [ConfigProxy] info: 201 POST …/api/routes/jupyterhub/us{x}er/xUID

[D 2021-05-25 07:31:59.994 JupyterHub base:283] Recording first activity for \<APIToken(‘1552…’, us{x}er=‘xUID’)\>

[D 2021-05-25 07:32:00.008 JupyterHub us{x}ers:778] Activity for us-x-er xUID: 2021-05-25T07:31:59.333581Z

[D 2021-05-25 07:32:00.008 JupyterHub us{x}ers:796] Activity on server xUID/: 2021-05-25T07:31:59.333581Z

[I 2021-05-25 07:32:00.029 JupyterHub log:189] 200 POST …/jupyterhub/hub/api/us-x-ers/xUID/activity ([xUID@99.198.10.151](mailto:xUID@99.198.10.151)) 37.73ms

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 25, 2021, 8:12am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/10 "2021-05-25T08:12:46Z")

</div>

#### CONTINUE HUB LOGS

[I 2021-05-25 07:32:00.034 JupyterHub log:189] 302 GET …/jupyterhub/hub/spawn-pending/xUID-\> …/jupyterhub/us{x}er/xUID/ ([xUID@99.198.17.0](mailto:xUID@99.198.17.0)) 2.88ms

[D 2021-05-25 07:32:00.077 JupyterHub provider:409] Validating client id jupyterhub-us{x}er-xUID

[D 2021-05-25 07:32:00.078 JupyterHub provider:484] validate\_redirect\_uri: client\_id=…jupyterhub-us{x}er-xUID, redirect\_uri=…/jupyterhub/us{x}er/xUID/oauth\_callback

[D 2021-05-25 07:32:00.079 JupyterHub auth:244] Skipping oauth confirmation for \<Us{x}er(xUID1/1 running)\> accessing Server at …/jupyterhub/us{x}er/xUID/

[D 2021-05-25 07:32:00.080 JupyterHub provider:409] Validating client id jupyterhub-us{x}er-xUID

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 25, 2021, 8:15am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/11 "2021-05-25T08:15:00Z")

</div>

#### CONTINUE HUB LOGS

[D 2021-05-25 07:32:00.080 JupyterHub provider:484] validate\_redirect\_uri: client\_id=…/jupyterhub-us{x}er-xUID, redirect\_uri=…/jupyterhub/us{x}er/xUID/oauth\_callback

[D 2021-05-25 07:32:00.081 JupyterHub provider:231] Saving authorization code jupyterhub-us{x}er-xUID, kMM…,

[I 2021-05-25 07:32:00.094 JupyterHub log:189] 302 GET …/jupyterhub/hub/api/oauth2/authorize’?‘client\_id=jupyterhub-us-x-er-xUID&redirect\_uri=…’%‘2Fjupyter’%2F’us-x-er’%2F’xUID’%2F’oauth\_callback’&‘response\_type=code’&‘state=[secret] → …/jupyterhub/us{x}er/xUID/oauth\_callback’?‘code=[secret]’&'state=[secret] ([xUID@99.198.17.0](mailto:xUID@99.198.17.0)) 18.45ms

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 25, 2021, 8:15am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/12 "2021-05-25T08:15:40Z")

</div>

### CONTINUE HUB LOGS

[D 2021-05-25 07:32:00.116 JupyterHub provider:55] authenticate\_client -oauthlib.Request SANITIZED-

[D 2021-05-25 07:32:00.122 JupyterHub provider:111] confirm\_redirect\_uri: client\_id=…jupyterhub-us-x-er-xUID, redirect\_uri=…/…jupyterhub/us-x-er/xUID/oauth\_callback

[D 2021-05-25 07:32:00.122 JupyterHub provider:334] Saving bearer token {‘access\_token’: ‘REDACTED’, ‘expires\_in’: 1209600, ‘token\_type’: ‘Bearer’, ‘scope’: ‘identify’, ‘refresh\_token’: ‘REDACTED’}

[D 2021-05-25 07:32:00.134 JupyterHub provider:189] Deleting oauth code kMM… for jupyterhub-us-x-er-xUID

[I 2021-05-25 07:32:00.149 JupyterHub log:189] 200 POST …/..jupyterhub/hub/api/oauth2/token ([xUID@99.198.10.151](mailto:xUID@99.198.10.151)) 36.31ms

[I 2021-05-25 07:32:00.175 JupyterHub log:189] 200 GET …/..jupyterhub/hub/api/authorizations/token/[secret] ([xUID@99.198.10.151](mailto:xUID@99.198.10.151)) 21.88ms

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 25, 2021, 8:37am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/13 "2021-05-25T08:37:24Z")

</div>

###### SINGLE USER POD

Set username to: xUID

Relocating home dir to ../home/xUID

Setting CWD to …/home/xUID/

Executing the command: jupyterhub-singleuser --ip=0.0.0.0

[W 2021-05-25 08:16:05.524 SingleUserNotebookApp configurable:190] Config option `open_browser` not recognized by `SingleUserNotebookApp`. Did you mean `browser`?

[I 2021-05-25 08:16:05.582 SingleUserNotebookApp notebookapp:1593] Authentication of …/metrics is OFF, since other authentication is disabled.

[I 2021-05-25 08:16:06.207 SingleUserNotebookApp extension:162] JupyterLab extension loaded from …/opt/conda/lib/python3.8/site-packages/jupyterlab

[I 2021-05-25 08:16:06.207 SingleUserNotebookApp extension:163] JupyterLab application directory is …/opt/conda/share/jupyter/lab

[I 2021-05-25 08:16:06.225 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.1

[I 2021-05-25 08:16:06.231 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: …/nfs/xUID

[I 2021-05-25 08:16:06.231 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.3.0 is running at:

[I 2021-05-25 08:16:06.231 SingleUserNotebookApp notebookapp:2302] htp:jupyter-xUID:8888/jupyterhub/user/xUID

[I 2021-05-25 08:16:06.231 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[I 2021-05-25 08:16:06.235 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds

[I 2021-05-25 08:16:06.914 SingleUserNotebookApp log:189] 302 GET …/jupyterhub/user/xUID → …/jupyterhub/user/xUID/tree’?’ ([at@99.198.17.231](mailto:at@99.198.17.231)) 0.74ms

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 25, 2021, 8:38am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/14 "2021-05-25T08:38:20Z")

</div>

##CONTINUE SINGLE USER POD

[I 2021-05-25 08:16:06.958 SingleUserNotebookApp log:189] 302 GET …/jupyterhub/user/xUID/tree’?‘redirects=1 → …/jupyterhub/hub/api/oauth2/authorize’?‘client\_id’=‘jupyterhub-user-xUID’&‘redirect\_uri=’%2F’jupyterhub’%2F’user’%2F’xUID’%2F’oauth\_callback’&‘response\_type’=‘code’&‘state’=[secret]’ ([at@99.198.31.0](mailto:at@99.198.31.0)) 4.10ms

[I 2021-05-25 08:16:07.060 SingleUserNotebookApp auth:992] Logged-in user {‘kind’: ‘user’, ‘name’: ‘xUID’, ‘admin’: False, ‘groups’: [], ‘server’: ‘/jupyterhub/user/xUID/’, ‘pending’: None, ‘created’: ‘2021-05-25T07:31:33.184696Z’, ‘last\_activity’: ‘2021-05-25T08:16:07.049165Z’, ‘servers’: None}

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 25, 2021, 8:40am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/15 "2021-05-25T08:40:29Z")

</div>

#CONTINUE SINGLE USER POD

[I 2021-05-25 08:16:07.061 SingleUserNotebookApp log:189] 302 GET …/jupyterhub/user/xUID/oauth\_callback’?‘code=[secret]’&‘state=[secret] ‘-\>’ …/jupyterhub/user/xUID/tree’?'redirects=1 (at-99.198.31.0) 61.29ms

[I 2021-05-25 08:16:07.144 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/tree’?'redirects=1 (xUID-99.198.31.0) 47.03ms

[I 2021-05-25 08:16:07.263 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/config/tree’?'\_=1621930567218 (xUID-99.198.31.0) 2.97ms

[I 2021-05-25 08:16:07.266 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/config/common’?\_'=1621930567219 (xUID-99.198.26.0) 2.48ms

[I 2021-05-25 08:16:07.279 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/terminals’?\_'=1621930567222 (xUID-99.198.31.0) 1.83ms

[I 2021-05-25 08:16:07.281 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/terminals’?\_'=1621930567220 (xUID-99.198.16.0) 0.89ms

[I 2021-05-25 08:16:07.288 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/sessions’?\_'=1621930567221 (xUID-99.198.17.0) 2.41ms

[I 2021-05-25 08:16:07.313 SingleUserNotebookApp log:189] 200 GET …/jupyterhub/user/xUID/api/contents’?‘type=directory’&\_'=1621930567223 (xUID-99.198.17.0) 11.80ms

#### WHEN I TRY TO OPEN A TERMINAL

[I 2021-05-25 08:23:33.175 SingleUserNotebookApp management:338] New terminal with automatic name: 1

[I 2021-05-25 08:23:33.177 SingleUserNotebookApp log:189] 200 POST /jupyterhub/user/xUID/api/terminals (xUID-99.198.21.0) 643.39ms

[I 2021-05-25 08:23:33.180 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/tree (xUID-99.198.30.0) 645.60ms

[I 2021-05-25 08:23:33.207 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/terminals/1 (xUID-99.198.21.0) 13.39ms

[I 2021-05-25 08:23:33.384 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/config/terminal?\_=1621931013341 (xUID-99.198.21.0) 3.94ms

[I 2021-05-25 08:23:33.388 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/config/common?\_=1621931013342 (xUID-99.198.17.0) 2.62ms

**[W 2021-05-25 08:23:33.405 SingleUserNotebookApp log:189] 400 GET /jupyterhub/user/xUID/terminals/websocket/1 (xUID-99.198.5.0) 1.90ms**

#### WHEN I TRY TO OPEN A PYTHON3 KERNEL (notebook Unititled.ipynb)

[I 2021-05-25 08:23:36.615 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/notebooks/Untitled.ipynb (xUID-99.198.30.0) 831.44ms

[I 2021-05-25 08:23:36.837 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/config/notebook?\_=1621931016749 (xUID-99.198.5.0) 7.83ms

[I 2021-05-25 08:23:36.841 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/config/common?\_=1621931016750 (xUID-99.198.30.0) 3.01ms

[I 2021-05-25 08:23:36.953 SingleUserNotebookApp sign:396] Writing notebook-signing key to /home/xUID/.local/share/jupyter/notebook\_secret

[W 2021-05-25 08:23:36.953 SingleUserNotebookApp manager:510] Notebook Untitled.ipynb is not trusted

[I 2021-05-25 08:23:36.955 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/contents/Untitled.ipynb’?‘type=notebook’&'\_=1621931016751 (xUID-99.198.30.0) 46.76ms

[I 2021-05-25 08:23:37.034 SingleUserNotebookApp kernelmanager:179] Kernel started: c7ef37da-4ee4-435d-adea-6157222d58e7, name: python3

[I 2021-05-25 08:23:37.043 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/contents/Untitled.ipynb/checkpoints’?'\_=1621931016752 (xUID-99.198.17.0) 7.55ms

[I 2021-05-25 08:23:37.048 SingleUserNotebookApp log:189] 201 POST /jupyterhub/user/xUID/api/sessions (xUID-99.198.5.0) 45.13ms

[I 2021-05-25 08:23:37.050 SingleUserNotebookApp log:189] 200 GET /jupyterhub/user/xUID/api/metrics/v1 (xUID-99.198.21.0) 4.14ms

**[W 2021-05-25 08:23:37.536 SingleUserNotebookApp log:189] 400 GET /jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7/channels?session\_id=a4775ba1ca734ce98dc511094c4efcf5 (xUID-99.198.5.0) 450.91ms**

**[W 2021-05-25 08:23:38.555 SingleUserNotebookApp handlers:366] Replacing stale connection: c7ef37da-4ee4-435d-adea-6157222d58e7:a4775ba1ca734ce98dc511094c4efcf5**

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 25, 2021, 8:46am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/16 "2021-05-25T08:46:00Z")

</div>

**CONSOLE LOGS - PYTHON3 KERNEL**

menubar.js:282 actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later…  
MenuBar.bind\_events @ menubar.js:282  
utils.js:60 load\_extensions Arguments(2)  
bidi.js:19 Loaded moment locale en-gb  
utils.js:37 Loading extension: nbresuse/main  
utils.js:37 Loading extension: jupyter-js-widgets/extension  
session.js:54 Session: kernel\_created (f597f87e-14d2-4e24-b99c-e2676621eba2)  
kernel.js:463 Starting WebSockets: wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7  
kernel.js:465 \*\*WebSocket connection to ‘wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7/channels?session\_id=a4775ba1ca734ce98dc511094c4efcf5’ failed: \*\*  
**Kernel.start\_channels @ kernel.js:465**  
**kernel.js:106 Kernel: kernel\_disconnected (c7ef37da-4ee4-435d-adea-6157222d58e7)**  
**kernel.js:546 WebSocket connection failed: wss://hello–uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7 true**  
**kernel.js:564 Connection lost, reconnecting in 1 seconds.**  
**kernel.js:474 WebSocket closed early CloseEvent**  
kernel.js:106 Kernel: kernel\_reconnecting (c7ef37da-4ee4-435d-adea-6157222d58e7)  
kernel.js:463 Starting WebSockets: wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7  
kernel.js:465 WebSocket connection to ‘wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7/channels?session\_id=a4775ba1ca734ce98dc511094c4efcf5’ failed:  
Kernel.start\_channels @ kernel.js:465  
kernel.js:106 Kernel: kernel\_disconnected (c7ef37da-4ee4-435d-adea-6157222d58e7)  
kernel.js:546 WebSocket connection failed: wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7 true  
kernel.js:564 Connection lost, reconnecting in 2 seconds.  
kernel.js:493 WebSocket closed unexpectedly CloseEvent  
kernel.js:106 Kernel: kernel\_reconnecting (c7ef37da-4ee4-435d-adea-6157222d58e7)  
kernel.js:463 Starting WebSockets: wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7  
kernel.js:465 WebSocket connection to ‘wss://hello-uat/jupyterhub/user/xUID/api/kernels/c7ef37da-4ee4-435d-adea-6157222d58e7/channels?session\_id=a4775ba1ca734ce98dc511094c4efcf5’ failed:  
Kernel.start\_channels @ kernel.js:465  
kernel.js:106 Kernel: kernel\_disconnected (c7ef37da-4ee4-435d-adea-6157222d58e7)

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 26, 2021, 7:35am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/17 "2021-05-26T07:35:43Z")

</div>

**CONSOLE LOGS TERMINAL**

terminado.js:4 WebSocket connection to ‘wss://hello-uat/jupyterhub/user/xUID/terminals/websocket/1’ failed:  
make\_terminal @ terminado.js:4  
index.js:5 Uncaught TypeError: Cannot read property ‘parentElement’ of undefined  
at proposeGeometry (index.js:5)  
at fit (index.js:30)  
at Terminal.terminalConstructor.fit (index.js:44)  
at window.onresize (main.js:54)

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 26, 2021, 7:36am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/18 "2021-05-26T07:36:37Z")

</div>

I’ve asked the Kubernetes team to send me the /etc/nginx/nginx.conf file for both Kubernetes clusters. Can you think of anything else that I should ask for ?

Thank you very much

Alison

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [May 27, 2021, 8:26am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/19 "2021-05-27T08:26:51Z")

</div>

Since the error is on websockets, it is almost certainly a proxy layer _somewhere_ that doesn’t support websockets, which is different on the two clusters (i.e. between you and any JupyterHub’s `proxy-public` entrypoint, rather than any part of JupyterHub itself or any of JupyterHub’s own configuration). This may be a firewall somewhere or ingress-controller layer on the cluster.

It could even be a proxy that runs on your computer or local network, if you have any proxy things that might apply different filtering to those domains.

---

<div class="post-metadata">

**Author:** ![alison](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@alison](https://discourse.jupyter.org/u/alison)\
**Post date:** [May 27, 2021, 11:17am UTC](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289/20 "2021-05-27T11:17:41Z")

</div>

Thank you very much minrk

I’ve asked the proxy team/firewall teams again to see if they can find something

I’ve compared the nginx.conf Kubernetes files for the DEV vs UAT cluster to understand the differences, but I cannot find anything that looks strange there

[Next page](https://discourse.jupyter.org/t/websocket-400-connection-failed/9289.md?page=2)