I’ve seen several similar issues here, but none of them seems to be applicable to my situation, so maybe someone can point me in the right direction.
Got Jupyterhub installed in Kubernetes following the guide
Got all the services up and running.
Logged in with a dummy user, and got the error in the title. The kernel does not load.
Accessed the pod - it has python up and running, all the necessary packages seem to be present, it seems like a connection issue.
Any help will be appreciated!
Thanks in advance.
Could you share the single user server pod logs and browser console logs? And also your JupyterHub config? Without these details, it is not possible for others to help you out.
Do you have a reverse proxy in front of JupyterHub? If so, could you share the config of your reverse proxy?
2 Likes
Hi! Thanks for your response!
Server pod logs
WARNING: using start-singleuser.sh instead of start-notebook.sh to start a server associated with JupyterHub.
Entered start.sh with args: jupyterhub-singleuser --ip=0.0.0.0
Executing the command: jupyterhub-singleuser --ip=0.0.0.0
[I 2024-06-13 14:00:35.887 SingleUserLabApp mixins:610] Starting jupyterhub single-user server version 3.0.0
[I 2024-06-13 14:00:35.887 SingleUserLabApp mixins:624] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 3.5.0
[I 2024-06-13 14:00:35.887 SingleUserLabApp mixins:624] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.23.3
[W 2024-06-13 14:00:35.910 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`. Did you mean `browser`?
[W 2024-06-13 14:00:35.935 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`. Did you mean `browser`?
[I 2024-06-13 14:00:35.935 SingleUserLabApp manager:344] jupyterlab | extension was successfully linked.
[W 2024-06-13 14:00:35.940 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-06-13 14:00:35.940 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-06-13 14:00:35.940 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-06-13 14:00:35.949 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`. Did you mean `browser`?
[I 2024-06-13 14:00:35.949 SingleUserLabApp manager:344] nbclassic | extension was successfully linked.
[I 2024-06-13 14:00:36.161 SingleUserLabApp manager:344] notebook_shim | extension was successfully linked.
[I 2024-06-13 14:00:36.184 SingleUserLabApp manager:362] notebook_shim | extension was successfully loaded.
[I 2024-06-13 14:00:36.185 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.10/site-packages/jupyterlab
[I 2024-06-13 14:00:36.185 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2024-06-13 14:00:36.189 SingleUserLabApp manager:362] jupyterlab | extension was successfully loaded.
[I 2024-06-13 14:00:36.246 SingleUserLabApp manager:362] nbclassic | extension was successfully loaded.
[I 2024-06-13 14:00:36.247 SingleUserLabApp mixins:671] Starting jupyterhub-singleuser server version 3.0.0
[I 2024-06-13 14:00:36.481 SingleUserLabApp serverapp:2726] Serving notebooks from local directory: /home/jovyan
[I 2024-06-13 14:00:36.481 SingleUserLabApp serverapp:2726] Jupyter Server 1.23.3 is running at:
[I 2024-06-13 14:00:36.481 SingleUserLabApp serverapp:2726] http://jupyter-12345:8888/user/12345/lab
[I 2024-06-13 14:00:36.481 SingleUserLabApp serverapp:2726] or http://127.0.0.1:8888/user/12345/lab
[I 2024-06-13 14:00:36.481 SingleUserLabApp serverapp:2727] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-06-13 14:00:36.488 SingleUserLabApp log:186] 302 GET /user/12345/ -> /user/12345/lab? (@10.237.120.148) 8.07ms
[I 2024-06-13 14:00:36.489 SingleUserLabApp mixins:592] Updating Hub with activity every 300 seconds
[I 2024-06-13 14:00:37.054 SingleUserLabApp log:186] 302 GET /user/12345/lab?redirects=1 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-12345&redirect_uri=%2Fuser%2F12345%2Foauth_callback&response_type=code&state=[secret] (@81.22.46.190) 5.35ms
[I 2024-06-13 14:00:37.988 SingleUserLabApp auth:1227] Logged-in user {'name': '12345', 'groups': [], 'admin': False, 'kind': 'user', 'session_id': '8d7d0242f8184d2eb18c5eeef197f5c3', 'scopes': ['access:servers!server=12345/', 'read:users:groups!user=12345', 'read:users:name!user=12345']}
[I 2024-06-13 14:00:37.990 SingleUserLabApp log:186] 302 GET /user/12345/oauth_callback?code=[secret]&state=[secret] -> /user/12345/lab?redirects=1 (@81.22.46.190) 676.24ms
[I 2024-06-13 14:00:38.033 SingleUserLabApp log:186] 200 GET /user/12345/lab?redirects=1 (12345@81.22.46.190) 20.39ms
[I 2024-06-13 14:00:40.133 SingleUserLabApp log:186] 200 GET /user/12345/api/kernelspecs?1718287244303 (12345@81.22.46.190) 97.95ms
[I 2024-06-13 14:00:40.219 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings?1718287244306 (12345@81.22.46.190) 85.21ms
[I 2024-06-13 14:00:40.220 SingleUserLabApp log:186] 200 GET /user/12345/api/sessions?1718287244312 (12345@81.22.46.190) 82.05ms
[I 2024-06-13 14:00:40.221 SingleUserLabApp log:186] 200 GET /user/12345/api/kernels?1718287244309 (12345@81.22.46.190) 81.63ms
[I 2024-06-13 14:00:40.222 SingleUserLabApp log:186] 200 GET /user/12345/api/terminals?1718287244313 (12345@81.22.46.190) 81.08ms
[I 2024-06-13 14:00:40.225 SingleUserLabApp log:186] 200 GET /user/12345/api/kernelspecs?1718287244421 (12345@81.22.46.190) 2.07ms
[I 2024-06-13 14:00:41.380 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/translations/en?1718287245619 (12345@81.22.46.190) 20.30ms
[I 2024-06-13 14:00:41.414 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/listings/@jupyterlab/extensionmanager-extension/listings.json?1718287245682 (12345@81.22.46.190) 1.53ms
[I 2024-06-13 14:00:41.420 SingleUserLabApp log:186] 200 GET /user/12345/api/contents?content=1&1718287245685 (12345@81.22.46.190) 4.56ms
[I 2024-06-13 14:00:41.440 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/docmanager-extension:plugin?1718287245700 (12345@81.22.46.190) 9.34ms
[I 2024-06-13 14:00:41.452 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1718287245699 (12345@81.22.46.190) 11.63ms
[I 2024-06-13 14:00:41.455 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/cell-toolbar-extension:plugin?1718287245698 (12345@81.22.46.190) 10.67ms
[I 2024-06-13 14:00:41.472 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/mainmenu-extension:plugin?1718287245701 (12345@81.22.46.190) 16.37ms
[I 2024-06-13 14:00:41.476 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/htmlviewer-extension:plugin?1718287245702 (12345@81.22.46.190) 15.64ms
[I 2024-06-13 14:00:41.479 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/notebook-extension:panel?1718287245703 (12345@81.22.46.190) 14.85ms
[I 2024-06-13 14:00:41.487 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/filebrowser-extension:widget?1718287245720 (12345@81.22.46.190) 7.18ms
[I 2024-06-13 14:00:41.536 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/translations/?1718287245775 (12345@81.22.46.190) 31.54ms
[I 2024-06-13 14:00:41.797 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/workspaces/default?1718287245826 (12345@81.22.46.190) 40.49ms
[I 2024-06-13 14:00:41.811 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1718287245828 (12345@81.22.46.190) 50.51ms
[I 2024-06-13 14:00:41.819 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/docmanager-extension:plugin?1718287245852 (12345@81.22.46.190) 52.47ms
[I 2024-06-13 14:00:41.828 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/csvviewer-extension:csv?1718287245853 (12345@81.22.46.190) 54.66ms
[I 2024-06-13 14:00:41.837 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/csvviewer-extension:tsv?1718287245853 (12345@81.22.46.190) 58.23ms
[I 2024-06-13 14:00:41.849 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/fileeditor-extension:plugin?1718287245854 (12345@81.22.46.190) 64.91ms
[I 2024-06-13 14:00:41.859 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/application-extension:context-menu?1718287245855 (12345@81.22.46.190) 69.34ms
[I 2024-06-13 14:00:42.140 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings?1718287246086 (12345@81.22.46.190) 278.84ms
[I 2024-06-13 14:00:42.141 LabApp] Build is up to date
[I 2024-06-13 14:00:42.160 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/build?1718287245674 (12345@81.22.46.190) 755.08ms
[I 2024-06-13 14:00:42.177 SingleUserLabApp log:186] 200 GET /user/12345/lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1718287246149 (12345@81.22.46.190) 34.58ms
[I 2024-06-13 14:00:42.379 SingleUserLabApp log:186] 204 PUT /user/12345/lab/api/workspaces/default?1718287246635 (12345@81.22.46.190) 5.10ms
[I 2024-06-13 14:00:42.436 SingleUserLabApp log:186] 200 GET /user/12345/api/contents?content=1&1718287246685 (12345@81.22.46.190) 9.58ms
[I 2024-06-13 14:00:42.937 SingleUserLabApp log:186] 204 PUT /user/12345/lab/api/workspaces/default?1718287247196 (12345@81.22.46.190) 6.36ms
[I 2024-06-13 14:00:43.128 SingleUserLabApp log:186] 200 GET /user/12345/api/nbconvert?1718287245796 (12345@81.22.46.190) 1589.70ms
[I 2024-06-13 14:00:43.312 SingleUserLabApp log:186] 200 GET /user/12345/api/nbconvert?1718287245793 (12345@81.22.46.190) 1774.72ms
[I 2024-06-13 14:00:44.803 SingleUserLabApp handlers:172] Creating new notebook in
[I 2024-06-13 14:00:44.882 SingleUserLabApp sign:412] Writing notebook-signing key to /home/jovyan/.local/share/jupyter/notebook_secret
[I 2024-06-13 14:00:44.892 SingleUserLabApp log:186] 201 POST /user/12345/api/contents?1718287249069 (12345@81.22.46.190) 90.70ms
[I 2024-06-13 14:00:44.915 SingleUserLabApp log:186] 200 GET /user/12345/api/contents/Untitled.ipynb?content=0&1718287249181 (12345@81.22.46.190) 2.74ms
[I 2024-06-13 14:00:44.928 SingleUserLabApp log:186] 200 GET /user/12345/api/contents?content=1&1718287249193 (12345@81.22.46.190) 3.58ms
[I 2024-06-13 14:00:44.955 SingleUserLabApp log:186] 200 GET /user/12345/api/contents/Untitled.ipynb?type=notebook&content=1&1718287249219 (12345@81.22.46.190) 5.69ms
[I 2024-06-13 14:00:45.008 SingleUserLabApp log:186] 200 GET /user/12345/api/contents/Untitled.ipynb/checkpoints?1718287249272 (12345@81.22.46.190) 3.90ms
[I 2024-06-13 14:00:45.009 SingleUserLabApp log:186] 200 GET /user/12345/api/contents/Untitled.ipynb/checkpoints?1718287249271 (12345@81.22.46.190) 3.90ms
[I 2024-06-13 14:00:45.013 SingleUserLabApp log:186] 200 GET /user/12345/api/contents/Untitled.ipynb/checkpoints?1718287249276 (12345@81.22.46.190) 2.19ms
[I 2024-06-13 14:00:45.064 SingleUserLabApp log:186] 200 GET /user/12345/api/sessions?1718287249328 (12345@81.22.46.190) 2.73ms
[I 2024-06-13 14:00:45.122 SingleUserLabApp kernelmanager:230] Kernel started: 3b61583e-0e98-4ccf-a9b0-7e19a43c9911
[I 2024-06-13 14:00:45.123 SingleUserLabApp log:186] 201 POST /user/12345/api/sessions?1718287249352 (12345@81.22.46.190) 41.11ms
[I 2024-06-13 14:00:45.160 SingleUserLabApp log:186] 200 GET /user/12345/api/sessions?1718287249427 (12345@81.22.46.190) 2.10ms
[I 2024-06-13 14:00:45.184 SingleUserLabApp log:186] 200 PATCH /user/12345/api/sessions/6b8064ee-3f86-43e7-893e-dd51cbbc1c7f?1718287249451 (12345@81.22.46.190) 2.40ms
[I 2024-06-13 14:00:45.194 SingleUserLabApp log:186] 200 GET /user/12345/api/kernels?1718287249461 (12345@81.22.46.190) 2.17ms
[I 2024-06-13 14:00:45.206 SingleUserLabApp log:186] 200 PATCH /user/12345/api/sessions/6b8064ee-3f86-43e7-893e-dd51cbbc1c7f?1718287249471 (12345@81.22.46.190) 2.44ms
[I 2024-06-13 14:00:45.211 SingleUserLabApp log:186] 200 GET /user/12345/api/sessions?1718287249477 (12345@81.22.46.190) 1.75ms
[I 2024-06-13 14:00:45.229 SingleUserLabApp log:186] 200 GET /user/12345/api/kernelspecs?1718287249495 (12345@81.22.46.190) 3.18ms
[I 2024-06-13 14:00:45.781 SingleUserLabApp log:186] 204 PUT /user/12345/lab/api/workspaces/default?1718287250046 (12345@81.22.46.190) 2.68ms
[W 2024-06-13 14:00:45.920 SingleUserLabApp log:186] 400 GET /user/12345/api/kernels/3b61583e-0e98-4ccf-a9b0-7e19a43c9911/channels?session_id=8216d1f3-8f71-4152-96f2-9d43d1babab8 (12345@81.22.46.190) 722.53ms
[I 2024-06-13 14:00:45.943 SingleUserLabApp log:186] 200 GET /user/12345/api/kernels/3b61583e-0e98-4ccf-a9b0-7e19a43c9911?1718287250209 (12345@81.22.46.190) 2.13ms
[W 2024-06-13 14:00:45.986 SingleUserLabApp log:186] 400 GET /user/12345/api/kernels/3b61583e-0e98-4ccf-a9b0-7e19a43c9911/channels?session_id=03cd6142-481f-4633-980d-f50f3fff622a (12345@81.22.46.190) 3.93ms
[I 2024-06-13 14:00:46.007 SingleUserLabApp log:186] 200 GET /user/12345/api/kernels/3b61583e-0e98-4ccf-a9b0-7e19a43c9911?1718287250274 (12345@81.22.46.190) 1.86ms
[W 2024-06-13 14:00:46.054 SingleUserLabApp log:186] 400 GET /user/12345/api/kernels/3b61583e-0e98-4ccf-a9b0-7e19a43c9911/channels?session_id=11202161-d14f-41d4-9c1e-9ad1a6f4c3ee (12345@81.22.46.190) 3.84ms
[I 2024-06-13 14:00:46.074 SingleUserLabApp log:186] 200 GET /user/12345/api/kernels/3b61583e-0e98-4ccf-a9b0-7e19a43c9911?1718287250342 (12345@81.22.46.190) 1.91ms
[W 2024-06-13 14:00:46.114 SingleUserLabApp handlers:439] Replacing stale connection: 3b61583e-0e98-4ccf-a9b0-7e19a43c9911:8216d1f3-8f71-4152-96f2-9d43d1babab8
[I 2024-06-13 14:00:50.332 SingleUserLabApp log:186] 200 GET /user/12345/api/terminals?1718287254597 (12345@81.22.46.190) 2.54ms
[I 2024-06-13 14:00:55.005 SingleUserLabApp log:186] 200 GET /user/12345/api/contents?content=1&1718287259268 (12345@81.22.46.190) 4.57ms
[I 2024-06-13 14:00:55.220 SingleUserLabApp log:186] 200 GET /user/12345/api/kernels?1718287259484 (12345@81.22.46.190) 2.46ms
[I 2024-06-13 14:00:55.239 SingleUserLabApp log:186] 200 GET /user/12345/api/sessions?1718287259507 (12345@81.22.46.190) 2.24ms
Config was the most basic one. I followed an advice in another post to make basic work first, then complicate it
Jupyterhub config
image:
# You should replace the "latest" tag with a fixed version from:
# https://hub.docker.com/r/jupyter/datascience-notebook/tags/
# Inspect the Dockerfile at:
# https://github.com/jupyter/docker-stacks/tree/HEAD/datascience-notebook/Dockerfile
name: jupyter/base-notebook
tag: hub-3.0.0
# `cmd: null` allows the custom CMD of the Jupyter docker-stacks to be used
# which performs further customization on startup.
cmd: null
storage:
type: none
Reverse Proxy config
listen 80;
listen 443 ssl http2;
ssl_certificate /etc/nginx/ssl/bi.ctcru.tech.cer;
ssl_certificate_key /etc/nginx/ssl/bi.ctcru.tech.key;
include snippets/ssl.conf;
server_name *.bi.ctcru.tech;
include snippets/acl.conf;
location / {
client_max_body_size 200M;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
#proxy_set_header X-Request-ID $x_request_id;
proxy_hide_header X-Datacenter;
proxy_hide_header X-Request-ID;
proxy_pass http://192.168.60.80;
}
}
Which points to the ingress for the instance
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: jupyter-ingress
namespace: jupyter
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.org/websocket-services: proxy-public
spec:
rules:
- host: jupyterhubnew.bi.ctcru.tech
http:
paths:
- backend:
service:
name: proxy-public
port:
number: 80
path: /
pathType: Prefix
Browser log is too long to post
I think your reverse proxy config missing web socket headers. Check the reference nginx config. You must include Connection and Upgrade headers in your config for kernels to work (they use web sockets under the hood for communication)
2 Likes
Thank you, Mahendra!
Indeed, the issue was in the headers. Adding them resolved it.