User notebook fails to spawn

We installed microk8s on our own server running Rocky Linux 8

I have followed the installation instructions on J2JH

The microk8s cluster (on a single node) appears to be running without problems. We installed Helm and then JupyterHub with:

helm upgrade --cleanup-on-fail \
  --install 3.0.0 jupyterhub/jupyterhub \
  --namespace k8s-test \
  --create-namespace \
  --version=2.0.0 \
  --values config.yaml

Our config.yaml looks like this:

  secretToken: "snip"
    guarantee: 2G
    limit: 2G
    capacity: 1Gi
    guarantee: 2.0
    name: jupyter/scipy-notebook
    tag: ae885c0a6226
  defaultUrl: "/lab"
    JUPYTERHUB_SINGLEUSER_APP: "jupyter_server.serverapp.ServerApp"

I can log into the web UI with dummy username and password, but get the following errors on the web UI:

Server requested
2022-10-27T18:11:48.627590Z [Normal] Successfully assigned k8s-test/jupyter-username to
2022-10-27T18:11:49Z [Normal] Container image "jupyterhub/k8s-network-tools:2.0.0" already present on machine
2022-10-27T18:11:49Z [Normal] Created container block-cloud-metadata
2022-10-27T18:11:49Z [Normal] Started container block-cloud-metadata
2022-10-27T18:11:49Z [Normal] Container image "jupyter/scipy-notebook:ae885c0a6226" already present on machine
2022-10-27T18:11:49Z [Normal] Created container notebook
2022-10-27T18:11:49Z [Normal] Started container notebook
Spawn failed: Server at didn't respond in 30 seconds

The k8s log file show:

Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[I 2022-10-27 18:00:01.811 JupyterHub app:2775] Running JupyterHub version 3.0.0
[I 2022-10-27 18:00:01.811 JupyterHub app:2805] Using Authenticator: jupyterhub.auth.DummyAuthenticator-3.0.0
[I 2022-10-27 18:00:01.811 JupyterHub app:2805] Using Spawner: kubespawner.spawner.KubeSpawner-4.2.0
[I 2022-10-27 18:00:01.811 JupyterHub app:2805] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-3.0.0
[I 2022-10-27 18:00:01.860 JupyterHub app:1934] Not using allowed_users. Any authenticated user will be allowed.
[I 2022-10-27 18:00:01.888 JupyterHub app:2844] Initialized 0 spawners in 0.002 seconds
[I 2022-10-27 18:00:01.889 JupyterHub app:3057] Not starting proxy
[I 2022-10-27 18:00:01.893 JupyterHub app:3093] Hub API listening on http://:8081/hub/
[I 2022-10-27 18:00:01.893 JupyterHub app:3095] Private Hub API connect url http://hub:8081/hub/
[I 2022-10-27 18:00:01.893 JupyterHub app:3104] Starting managed service jupyterhub-idle-culler
[I 2022-10-27 18:00:01.893 JupyterHub service:385] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2022-10-27 18:00:01.895 JupyterHub service:133] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[I 2022-10-27 18:00:01.900 JupyterHub proxy:480] Adding route for Hub: / => http://hub:8081
[I 2022-10-27 18:00:01.901 JupyterHub app:3162] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
[I 2022-10-27 18:00:01.986 JupyterHub log:186] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 6.16ms
[I 2022-10-27 18:00:01.992 JupyterHub log:186] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 5.76ms
[I 2022-10-27 18:00:18.886 JupyterHub log:186] 302 GET /hub/ -> /hub/spawn (username@::ffff: 12.52ms
[I 2022-10-27 18:00:18.895 JupyterHub reflector:274] watching for pods with label selector='component=singleuser-server' in namespace k8s-test
[I 2022-10-27 18:00:18.896 JupyterHub reflector:274] watching for events with field selector='involvedObject.kind=Pod' in namespace k8s-test
[I 2022-10-27 18:00:18.926 JupyterHub provider:651] Creating oauth client jupyterhub-user-username
[I 2022-10-27 18:00:18.933 JupyterHub spawner:2509] Attempting to create pvc claim-username, with timeout 3
[I 2022-10-27 18:00:18.934 JupyterHub log:186] 302 GET /hub/spawn -> /hub/spawn-pending/username (username@::ffff: 18.73ms
[I 2022-10-27 18:00:18.943 JupyterHub spawner:2525] PVC claim-username already exists, so did not create new pvc.
[I 2022-10-27 18:00:18.947 JupyterHub spawner:2469] Attempting to create pod jupyter-username, with timeout 3
[I 2022-10-27 18:00:18.950 JupyterHub pages:394] classen is pending spawn
[I 2022-10-27 18:00:18.967 JupyterHub log:186] 200 GET /hub/spawn-pending/username (username@::ffff: 18.35ms
[W 2022-10-27 18:00:52.442 JupyterHub user:881] username's server never showed up at after 30 seconds. Giving up.
    Common causes of this timeout, and debugging tips:
    1. The server didn't finish starting,
       or it crashed due to a configuration issue.
       Check the single-user server's logs for hints at what needs fixing.
    2. The server started, but is not accessible at the specified URL.
       This may be a configuration issue specific to your chosen Spawner.
       Check the single-user server logs and resource to make sure the URL
       is correct and accessible from the Hub.
    3. (unlikely) Everything is working, but the server took too long to respond.
       To fix: increase `Spawner.http_timeout` configuration
       to a number of seconds that is enough for servers to become responsive.

I tried looking at the jupyter-username log file:

Defaulted container "notebook" out of: notebook, block-cloud-metadata (init)
[W 2022-10-27 18:11:50.485 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[W 2022-10-27 18:11:50.498 SingleUserNotebookApp login:217] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[W 2022-10-27 18:11:50.498 SingleUserNotebookApp login:220] WARNING: The notebook server is listening on all IP addresses and not using authentication. This is highly insecure and not recommended.
[I 2022-10-27 18:11:50.517 SingleUserNotebookApp extension:46] JupyterLab alpha preview extension loaded from /opt/conda/lib/python3.6/site-packages/jupyterlab
JupyterLab v0.27.0
Known labextensions:
[I 2022-10-27 18:11:50.518 SingleUserNotebookApp extension:93] Running the core application with no additional extensions or settings
[I 2022-10-27 18:11:50.521 SingleUserNotebookApp notebookapp:1366] Serving notebooks from local directory: /home/jovyan
[I 2022-10-27 18:11:50.521 SingleUserNotebookApp notebookapp:1366] 0 active kernels 
[I 2022-10-27 18:11:50.521 SingleUserNotebookApp notebookapp:1366] The Jupyter Notebook is running at: http://[all ip addresses on your system]:8888/user/username/
[I 2022-10-27 18:11:50.521 SingleUserNotebookApp notebookapp:1367] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

but it doesn’t seen too informative. I was a bit curious about the line referencing:

Serving notebooks from local directory: /home/jovyan

I don’t know where this is coming from since we have no user by that name…

Anyways, I’d appreciate any advice on how I might get a notebook launched.


Well this appears to be a firewall issue as disabling firewalld allows the notebooks to spawn.

1 Like