I can’t seem to be able to initialize kernel in the server which means that I can’t run any code. I get Couldn’t authenticate WebSocket connection in my logs everytime. Also the logos don’t show(python3, julia etc logos). They redirect to oauth page (logos) which returns a 403. I create the user and start the server programmatically(via api). The user is not supposed to login into jupyterhub since they are already logged in into my app. Below are my logs and config
Config.yml:
# This file can update the JupyterHub Helm chart's default configuration values.
#
# For reference see the configuration reference and default values, but make
# sure to refer to the Helm chart version of interest to you!
#
# Introduction to YAML: https://www.youtube.com/watch?v=cdLNKUoMc6c
# Chart config reference: https://zero-to-jupyterhub.readthedocs.io/en/stable/resources/reference.html
# Chart default values: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/b67703768e7b06a70d6c63cb0928d4ca1de269f3/jupyterhub/values.yaml
# Available chart versions: https://hub.jupyter.org/helm-chart/
#
proxy:
secretToken: tuFslOftLZF2t7ECEUZeMgmfOaIdv1RX
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:211125604315:certificate/610e78da-9cd3-456c-a4cc-5efabe8be247"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
https:
enabled: true
type: offload
debug:
enabled: true
scheduling:
userScheduler:
enabled: true
podPriority:
enabled: true
userPlaceholder:
enabled: true
replicas: 4
userPods:
nodeAffinity:
matchNodePurpose: require
cull:
enabled: true
timeout: 3600
every: 300
singleuser:
cpu:
limit: 4
guarantee: 0.05
memory:
limit: 4G
guarantee: 512M
cloudMetadata:
blockWithIptables: false
cmd: ["jupyterhub-singleuser", "--notebook-dir=/home/workspace"]
startTimeout: 600
defaultUrl: "/lab"
storage:
homeMountPath: /home/workspace
image:
name: jupyter/datascience-notebook
tag: latest
nodeSelector:
hub.jupyter.org/node-purpose: user
hub:
services:
datarango:
admin: true
name: datarango
api_token: secret-token
extraConfig:
myConfig.py: |
from jupyterhub.auth import Authenticator
from jupyterhub.handlers import BaseHandler
from traitlets import Unicode, default
from tornado import web
import codecs
from urllib.parse import urlparse, parse_qs
from jupyterhub.utils import url_path_join
class NullAuthenticator(Authenticator):
"""Null Authenticator for JupyterHub
For cases where authentication should be disabled,
e.g. only allowing access via API tokens.
.. versionadded:: 2.0
"""
# auto_login skips 'Login with...' page on Hub 0.8
auto_login = True
auto_login_oauth2_authorize = True
# for Hub 0.7, show 'login with...'
login_service = 'null'
def get_handlers(self, app):
return []
c.JupyterHub.authenticator_class = NullAuthenticator
c.JupyterHub.tornado_settings = {
'slow_spawn_timeout': 0,
'headers': {
'Content-Security-Policy': 'frame-ancestors self *'
},
'cookie_options': {'SameSite': 'None', 'Secure': True}
}
c.Spawner.args = [
'''--ServerApp.tornado_settings={
'headers': {
'Content-Security-Policy': 'frame-ancestors self *',
},
'cookie_options': {'SameSite': 'None', 'Secure': True}
}'''
]
# c.Spawner.environment = {
# "JUPYTERHUB_SINGLEUSER_EXTENSION": "0",
# }
Singleuser server logs:
[I 2024-05-04 09:26:15.072 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/config/jupyterlabapputilsextensionannouncements?1714814774673 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 3.58ms
[I 2024-05-04 09:26:15.143 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/server-proxy/servers-info?1714814774758 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 2.71ms
[I 2024-05-04 09:26:15.152 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/lsp/status?1714814774735 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 7.56ms
[I 2024-05-04 09:26:15.187 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/contents?content=1&1714814774754 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 48.28ms
[I 2024-05-04 09:26:15.200 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/lab/api/translations?1714814774749 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 54.46ms
[I 2024-05-04 09:26:15.364 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/lab/api/settings/@jupyterlab/codemirror-extension:plugin?1714814774787 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 6.39ms
[I 2024-05-04 09:26:15.431 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/lab/api/settings/@jupyterlab/notebook-extension:panel?1714814774793 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 9.36ms
[I 2024-05-04 09:26:15.440 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/kernelspecs/python3/logo-svg.svg -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 2.63ms
[I 2024-05-04 09:26:15.450 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/lab/api/workspaces/default?1714814775063 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 2.50ms
[W 2024-05-04 09:26:15.589 ServerApp] Couldn't authenticate WebSocket connection
[W 2024-05-04 09:26:15.592 ServerApp] 403 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2/channels?session_id=3f836cac-946e-4e6a-a1ad-78e163647225 (@::ffff:172.31.86.207) 3.37ms
[I 2024-05-04 09:26:15.610 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/contents?content=1&1714814775082 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 7.94ms
[I 2024-05-04 09:26:15.732 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/lab/api/settings?ids_only=true&1714814775325 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 14.10ms
[I 2024-05-04 09:26:15.832 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2?1714814775442 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 3.66ms
[W 2024-05-04 09:26:16.076 LabApp] The extension "nbdime-jupyterlab" is outdated.
[I 2024-05-04 09:26:16.077 LabApp] Build is up to date
[I 2024-05-04 09:26:16.078 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/lab/api/build?1714814774698 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 962.56ms
[I 2024-05-04 09:26:16.244 ServerApp] 204 PUT /user/pyerutssyqnhivavvfzvwsnijenesn/lab/api/workspaces/default?1714814775857 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 1.75ms
[I 2024-05-04 09:26:16.496 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/contents?content=1&1714814776104 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 4.78ms
[W 2024-05-04 09:26:16.522 ServerApp] Couldn't authenticate WebSocket connection
[W 2024-05-04 09:26:16.523 ServerApp] 403 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2/channels?session_id=3f836cac-946e-4e6a-a1ad-78e163647225 (@::ffff:172.31.86.207) 2.10ms
[W 2024-05-04 09:26:16.743 ServerApp] Detected unused OAuth state cookies
[I 2024-05-04 09:26:16.744 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/kernelspecs/julia-1.9/logo-svg.svg -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 1.97ms
[W 2024-05-04 09:26:16.759 ServerApp] Detected unused OAuth state cookies
[I 2024-05-04 09:26:16.760 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/kernelspecs/ir/logo-svg.svg -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 2.25ms
[I 2024-05-04 09:26:16.763 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/server-proxy/icon/pluto (@::ffff:172.31.86.207) 1.82ms
[I 2024-05-04 09:26:16.783 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2?1714814776392 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 2.91ms
[W 2024-05-04 09:26:16.862 ServerApp] Detected unused OAuth state cookies
[I 2024-05-04 09:26:16.869 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/kernelspecs/python3/logo-svg.svg -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 8.32ms
[I 2024-05-04 09:26:17.271 ServerApp] 204 PUT /user/pyerutssyqnhivavvfzvwsnijenesn/lab/api/workspaces/default?1714814776888 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 3.84ms
[W 2024-05-04 09:26:17.382 ServerApp] Detected unused OAuth state cookies
[I 2024-05-04 09:26:17.385 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/lab -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 3.91ms
[W 2024-05-04 09:26:17.523 ServerApp] Couldn't authenticate WebSocket connection
[W 2024-05-04 09:26:17.525 ServerApp] 403 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2/channels?session_id=3f836cac-946e-4e6a-a1ad-78e163647225 (@::ffff:172.31.86.207) 2.13ms
[I 2024-05-04 09:26:17.799 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2?1714814777372 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 2.29ms
[W 2024-05-04 09:26:17.927 ServerApp] Detected unused OAuth state cookies
[I 2024-05-04 09:26:17.928 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/kernelspecs/julia-1.9/logo-svg.svg -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 2.23ms
[W 2024-05-04 09:26:18.307 ServerApp] Detected unused OAuth state cookies
[I 2024-05-04 09:26:18.311 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/kernelspecs/python3/logo-svg.svg -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 4.93ms
[I 2024-05-04 09:26:18.463 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/server-proxy/icon/pluto (@::ffff:172.31.86.207) 1.90ms
[W 2024-05-04 09:26:18.720 ServerApp] Detected unused OAuth state cookies
[I 2024-05-04 09:26:18.721 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/kernelspecs/ir/logo-svg.svg -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 2.25ms
[W 2024-05-04 09:26:19.318 ServerApp] Detected unused OAuth state cookies
[I 2024-05-04 09:26:19.320 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/kernelspecs/python3/logo-svg.svg -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 2.31ms
[W 2024-05-04 09:26:19.816 ServerApp] Couldn't authenticate WebSocket connection
[W 2024-05-04 09:26:19.822 ServerApp] 403 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2/channels?session_id=3f836cac-946e-4e6a-a1ad-78e163647225 (@::ffff:172.31.86.207) 6.96ms
[W 2024-05-04 09:26:19.835 ServerApp] Detected unused OAuth state cookies
[I 2024-05-04 09:26:19.837 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/kernelspecs/julia-1.9/logo-svg.svg -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 2.45ms
[I 2024-05-04 09:26:20.269 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2?1714814779669 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 2.32ms
[W 2024-05-04 09:26:20.350 ServerApp] Detected unused OAuth state cookies
[I 2024-05-04 09:26:20.352 ServerApp] 302 GET /user/pyerutssyqnhivavvfzvwsnijenesn/kernelspecs/ir/logo-svg.svg -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-pyerutssyqnhivavvfzvwsnijenesn&redirect_uri=%2Fuser%2Fpyerutssyqnhivavvfzvwsnijenesn%2Foauth_callback&response_type=code&state=[secret] (@::ffff:172.31.86.207) 2.80ms
[W 2024-05-04 09:26:23.119 ServerApp] Couldn't authenticate WebSocket connection
[W 2024-05-04 09:26:23.120 ServerApp] 403 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2/channels?session_id=3f836cac-946e-4e6a-a1ad-78e163647225 (@::ffff:172.31.86.207) 2.73ms
[I 2024-05-04 09:26:23.393 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2?1714814782976 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 1.98ms
[I 2024-05-04 09:26:23.954 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/sessions?1714814783566 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 2.61ms
[I 2024-05-04 09:26:23.986 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/terminals?1714814783570 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 1.91ms
[I 2024-05-04 09:26:24.192 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels?1714814783571 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 2.41ms
[I 2024-05-04 09:26:26.954 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/contents?content=1&1714814786567 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 11.03ms
[W 2024-05-04 09:26:31.462 ServerApp] Couldn't authenticate WebSocket connection
[W 2024-05-04 09:26:31.467 ServerApp] 403 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2/channels?session_id=3f836cac-946e-4e6a-a1ad-78e163647225 (@::ffff:172.31.86.207) 5.82ms
[I 2024-05-04 09:26:31.706 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2?1714814791312 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 3.09ms
[W 2024-05-04 09:26:35.478 ServerApp] Couldn't authenticate WebSocket connection
[W 2024-05-04 09:26:35.480 ServerApp] 403 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2/channels?session_id=3f836cac-946e-4e6a-a1ad-78e163647225 (@::ffff:172.31.86.207) 3.15ms
[I 2024-05-04 09:26:35.794 ServerApp] 200 GET /user/pyerutssyqnhivavvfzvwsnijenesn/api/kernels/7de52aba-c1be-4dec-8103-550e02c2efa2?1714814795382 (pyerutssyqnhivavvfzvwsnijenesn@::ffff:172.31.86.207) 2.25ms
I’ve looked through the discourse group severally, tried many things like updating cookie settings, adding “JUPYTERHUB_SINGLEUSER_EXTENSION”: “0” (which often breaks my app instead of fixing it) etc
Which version of the Helm chart are you using? What versions of jupyterhub-singleuser, jupyter-server, and jupyterlab, are in your singleuser container?
Is this only a problem with NullAuthenticator i.e. does it work with one of the standard authenticators?
Do you have a reverse proxy like nginx in front of JupyterHub or your custom service? If so, did you configure the web socket header upgrades in the reverse proxy config?