Couldn't authenticate WebSocket connection

JupyterHub Zero to JupyterHub on Kubernetes
1

Hi,
i am able to login to the notebook, but the kernel failing to be initialized, as i get: 403 - Couldn’t authenticate WebSocket connection.

[W 2022-11-30 11:11:01.518 SingleUserNotebookApp zmqhandlers:325] Couldn't authenticate WebSocket connection

I cleaned browser cache, tried incognito mode, different browsers, but it didn’t help.

Any suggestions?

Thank you,
Yan

2

Hi! Please could you give us enough information to reproduce your problem- how did you set things up, what component versions are you using? Can you show us your debug logs including the context around the error. It would also be useful to see the logs from your browser console.

3

Hi @manics,

From the log:

[I 2022-11-30 15:17:30.171 SingleUserNotebookApp mixins:610] Starting jupyterhub single-user server version 3.0.0
[I 2022-11-30 15:17:30.171 SingleUserNotebookApp mixins:624] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.21.0

I am using null authenticator, so only login will be available with the token, but i don’t think this is the problem as it worked great.

I use single user standard image that include some extensions and CSP header, in order to be possible include lab in the iFrame. Will it be helpful to share my config.yaml file?

LOGS:

[I 2022-11-30 15:23:07.561 SingleUserNotebookApp log:186] 200 GET /user/u51/api/contents/Untitled1.ipynb?type=notebook&content=1&1669821787328 (u51@::ffff:10.244.0.1) 10.65ms
[I 2022-11-30 15:23:07.615 SingleUserNotebookApp log:186] 200 POST /user/u51/nbdime/api/isgit?1669821787428 (u51@::ffff:10.244.0.1) 1.77ms
[I 2022-11-30 15:23:07.741 SingleUserNotebookApp log:186] 200 GET /user/u51/api/contents/Untitled1.ipynb/checkpoints?1669821787549 (u51@::ffff:10.244.0.1) 1.72ms
[I 2022-11-30 15:23:07.744 SingleUserNotebookApp log:186] 200 GET /user/u51/api/contents/Untitled1.ipynb/checkpoints?1669821787553 (u51@::ffff:10.244.0.1) 2.47ms
[I 2022-11-30 15:23:07.745 SingleUserNotebookApp log:186] 200 GET /user/u51/api/contents/Untitled1.ipynb/checkpoints?1669821787548 (u51@::ffff:10.244.0.1) 2.09ms
[I 2022-11-30 15:23:07.917 SingleUserNotebookApp log:186] 200 GET /user/u51/api/sessions?1669821787720 (u51@::ffff:10.244.0.1) 1.73ms
[I 2022-11-30 15:23:08.087 SingleUserNotebookApp kernelmanager:230] Kernel started: 9c5e042f-65d7-4d17-b894-4fcb9555a721
[I 2022-11-30 15:23:08.087 SingleUserNotebookApp kernelmanager:569] Culling kernels with idle durations > 1200 seconds at 120 second intervals ...
[I 2022-11-30 15:23:08.087 SingleUserNotebookApp kernelmanager:577] Culling kernels even with connected clients
[I 2022-11-30 15:23:08.088 SingleUserNotebookApp log:186] 201 POST /user/u51/api/sessions?1669821787868 (u51@::ffff:10.244.0.1) 34.56ms
[I 2022-11-30 15:23:08.119 SingleUserNotebookApp log:186] 204 PUT /user/u51/lab/api/workspaces/default?1669821787933 (u51@::ffff:10.244.0.1) 1.79ms
[I 2022-11-30 15:23:08.295 SingleUserNotebookApp log:186] 200 GET /user/u51/api/sessions?1669821788081 (u51@::ffff:10.244.0.1) 1.76ms
[I 2022-11-30 15:23:08.510 SingleUserNotebookApp log:186] 200 PATCH /user/u51/api/sessions/243c673f-599e-484a-a07c-5283be2d3efe?1669821788253 (u51@::ffff:10.244.0.1) 1.81ms
[I 2022-11-30 15:23:08.513 SingleUserNotebookApp log:186] 200 GET /user/u51/api/kernels?1669821788257 (u51@::ffff:10.244.0.1) 1.37ms
[W 2022-11-30 15:23:08.517 SingleUserNotebookApp zmqhandlers:325] Couldn't authenticate WebSocket connection
[W 2022-11-30 15:23:08.518 SingleUserNotebookApp log:186] 403 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721/channels?session_id=99703a6c-340a-4fee-a2e0-96ec3a308af5 (@::ffff:10.244.0.1) 2.78ms
[I 2022-11-30 15:23:08.697 SingleUserNotebookApp log:186] 200 PATCH /user/u51/api/sessions/243c673f-599e-484a-a07c-5283be2d3efe?1669821788460 (u51@::ffff:10.244.0.1) 1.78ms
[I 2022-11-30 15:23:08.708 SingleUserNotebookApp log:186] 200 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721?1669821788513 (u51@::ffff:10.244.0.1) 1.32ms
[I 2022-11-30 15:23:08.713 SingleUserNotebookApp log:186] 200 GET /user/u51/api/sessions?1669821788461 (u51@::ffff:10.244.0.1) 1.23ms
[I 2022-11-30 15:23:08.896 SingleUserNotebookApp log:186] 200 GET /user/u51/api/kernelspecs?1669821788660 (u51@::ffff:10.244.0.1) 4.31ms
[W 2022-11-30 15:23:08.908 SingleUserNotebookApp zmqhandlers:325] Couldn't authenticate WebSocket connection
[W 2022-11-30 15:23:08.909 SingleUserNotebookApp log:186] 403 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721/channels?session_id=572c66c4-a471-4dbe-8a3d-2b822a331ffb (@::ffff:10.244.0.1) 2.64ms
[I 2022-11-30 15:23:08.917 SingleUserNotebookApp log:186] 200 GET /user/u51/api/terminals?1669821788728 (u51@::ffff:10.244.0.1) 1.16ms
[I 2022-11-30 15:23:09.107 SingleUserNotebookApp log:186] 200 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721?1669821788910 (u51@::ffff:10.244.0.1) 1.77ms
[I 2022-11-30 15:23:09.405 SingleUserNotebookApp log:186] 204 PUT /user/u51/lab/api/workspaces/default?1669821789218 (u51@::ffff:10.244.0.1) 1.76ms
[W 2022-11-30 15:23:09.410 SingleUserNotebookApp zmqhandlers:325] Couldn't authenticate WebSocket connection
[W 2022-11-30 15:23:09.412 SingleUserNotebookApp log:186] 403 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721/channels?session_id=f8b8903a-1ef3-4f84-b329-bf8e52c4651b (@::ffff:10.244.0.1) 2.70ms
[I 2022-11-30 15:23:09.548 SingleUserNotebookApp log:186] 200 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721?1669821789362 (u51@::ffff:10.244.0.1) 1.62ms
[W 2022-11-30 15:23:10.063 SingleUserNotebookApp zmqhandlers:325] Couldn't authenticate WebSocket connection
[W 2022-11-30 15:23:10.064 SingleUserNotebookApp log:186] 403 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721/channels?session_id=99703a6c-340a-4fee-a2e0-96ec3a308af5 (@::ffff:10.244.0.1) 3.38ms
[I 2022-11-30 15:23:10.200 SingleUserNotebookApp log:186] 200 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721?1669821790016 (u51@::ffff:10.244.0.1) 1.63ms
[W 2022-11-30 15:23:10.979 SingleUserNotebookApp zmqhandlers:325] Couldn't authenticate WebSocket connection
[W 2022-11-30 15:23:10.980 SingleUserNotebookApp log:186] 403 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721/channels?session_id=572c66c4-a471-4dbe-8a3d-2b822a331ffb (@::ffff:10.244.0.1) 3.34ms
[I 2022-11-30 15:23:11.118 SingleUserNotebookApp log:186] 200 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721?1669821790931 (u51@::ffff:10.244.0.1) 1.85ms
[W 2022-11-30 15:23:12.270 SingleUserNotebookApp zmqhandlers:325] Couldn't authenticate WebSocket connection
[W 2022-11-30 15:23:12.271 SingleUserNotebookApp log:186] 403 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721/channels?session_id=f8b8903a-1ef3-4f84-b329-bf8e52c4651b (@::ffff:10.244.0.1) 3.12ms
[I 2022-11-30 15:23:12.438 SingleUserNotebookApp log:186] 200 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721?1669821792251 (u51@::ffff:10.244.0.1) 1.68ms
[W 2022-11-30 15:23:14.161 SingleUserNotebookApp zmqhandlers:325] Couldn't authenticate WebSocket connection
[W 2022-11-30 15:23:14.162 SingleUserNotebookApp log:186] 403 GET /user/u51/api/kernels/9c5e042f-65d7-4d17-b894-4fcb9555a721/channels?session_id=99703a6c-340a-4fee-a2e0-96ec3a308af5 (@::ffff:10.244.0.1) 3.16ms
4

Another symptom is icons in the launcher are also failed, seems relative:

Screenshot 2022-11-30 at 17.43.02
Screenshot 2022-11-30 at 17.43.02917×545 25.3 KB

5

I don’t see any responses in browser debugger, except for status 403.

If more data is needed, please let me know. I am not sure that i sent all relevant info.

6

Yes, please share your config.

Can you also try this without the iframe? This will help identify whether it’s the CSP/Iframe that’s the problem.

7

Hi @manics, thank you!
The error I get also without iFrame.

Here is my config, some sensitive parts were replaced:

singleuser:
  defaultUrl: "/lab"
  extraEnv:
    JUPYTERHUB_SINGLEUSER_APP: "jupyter_server.serverapp.ServerApp"
  image:
    name: yanvu/aiml-notebook-csp
    tag: t-1.0.2 
  profileList:
    - display_name: "Datascience environment"
      description: "Datascience environment."
      default: true
      kubespawner_override:
        image: yanvu/aiml-notebook-csp:t-1.0.2
        lifecycle_hooks:
          postStart:
            exec:
              command:
                - "sh"
                - "-c"
                - >
                  gitpuller https://USER:TOKEN@GITHUB_URL main repo;
    - display_name: "Pyspark notebook"
      description: "Includes Python support for Apache Spark"
      kubespawner_override:
        image: yanvu/aiml-notebook-spark:t-1.0.0   
  cmd: null
  storage:
    extraVolumes:
      - name: jupyterhub-shared
        azureFile:
          shareName: '{hubnamespace}-{username}'
          secretName: azure-secret
          readOnly: false
    extraVolumeMounts:
      - name: jupyterhub-shared
        mountPath: /home/jovyan/scripts     
    capacity: 0.1Gi
  extraFiles:
    # jupyter_notebook_config reference: https://jupyter-notebook.readthedocs.io/en/stable/config.html
    jupyter_notebook_config.json:
      mountPath: /etc/jupyter/jupyter_notebook_config.json
      data:
        MappingKernelManager:
          cull_idle_timeout: 1200
          cull_interval: 120
          cull_connected: true
          cull_busy: false
  cpu:
    limit: 4
    guarantee: 0.05
  memory:
    limit: 4G
    guarantee: 512M
hub:
  extraConfig: 
    logo-config: |
      import urllib.request
      urllib.request.urlretrieve("SOME_LOGO.png", "/srv/jupyterhub/mlogo.png")
      c.JupyterHub.logo_file = "/srv/jupyterhub/mlogo.png"
    hub: |
      c.JupyterHub.tornado_settings = { "headers": { "Content-Security-Policy": "frame-ancestors * 'self' https://mysite.com https://yan.co.il", "Access-Control-Allow-Origin": "https://mysite.com"}, "cookie_options": {"SameSite": "None", "Secure": True}}
    nullAuth: |
      c.JupyterHub.authenticator_class = "null"
      c.Authenticator.login_url = "https://jupy.com/usr/relogin"  
    spawner: >-
      c.Spawner.args = ["--NotebookApp.tornado_settings={\"headers\":{\"Content-Security-Policy\": \"frame-ancestors * 'self' https://mysite.com https://yan2.co.il\"}}"] 
  services:
    binder:
      admin: true
      apiToken: SOME_TOKEN
8

Hi, seems like the problem is:

hub: |
      c.JupyterHub.tornado_settings = { "headers": { "Content-Security-Policy": "frame-ancestors * 'self' https://mysite.com https://yan.co.il", "Access-Control-Allow-Origin": "https://mysite.com"}, "cookie_options": {"SameSite": "None", "Secure": True}}

I had new installation and didn’t configure HTTPS, without it secure:true failed the cookie to be set properly and this messed up the WS requests.

@manics - THANK YOU SO MUCH FOR YOUR HELP!!!

1 Like