Jupyterhub single user sever not starting immediately on Kubernetes cluster

JupyterHub
1

We’ve deployed Jupyterhub on Kubernetes cluster. Followed from Optimizations — Zero to JupyterHub with Kubernetes documentation for better optimization(used continues image puller, user place holders and user schedulers).

When a user is trying to create a server to launch a jupyter lab instance , the server basically Kubernetes pod not starting immediately it is taking 35sec to start the server even we’ve user place holders placed.

Any suggestions to start the server immediate without user wait time.

Thank you

2

Have you tried looking at the k8s events and logs to see where the delay is occurring? Do you see the same behaviour with other pods, or only those started by JupyterHub?

3

Thank you for response. Create server “hub/api/users/server_name/server” always giving 202 status code instead 201 (which start server immediately)
Here are events did not see much difference
Events: │
│ Type Reason Age From Message │
│ ---- ------ ---- ---- ------- │
│ Warning FailedScheduling 42s jupyterhub-user-scheduler 0/8 nodes are available: not able to place differant reason │
│ Normal Scheduled 40s jupyterhub-user-scheduler Successfully assigned jupyter-xxx/jupyter-xxxx to ip-xxx.compute.internal │
│ Normal Pulled 39s kubelet Container image “jupyterhub/k8s-network-tools:2.0.0” already presen │
│ t on machine │
│ Normal Created 39s kubelet Created container block-cloud-metadata │
│ Normal Started 39s kubelet Started container block-cloud-metadata │
│ Normal Pulled 38s kubelet Container image “xxxxxxx” already present on machine │
│ Normal Created 38s kubelet Created container notebook │
│ Normal Started 38s kubelet Started container notebook

Here are logs

Defaulted container “notebook” out of: notebook, block-cloud-metadata (init)
[I 2023-09-08 14:15:02.972 SingleUserLabApp mixins:609] Starting jupyterhub single-user server version 3.1.1
[I 2023-09-08 14:15:02.972 SingleUserLabApp mixins:623] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 3.6.3
[I 2023-09-08 14:15:02.972 SingleUserLabApp mixins:623] Extending jupyter_server.serverapp.ServerApp from jupyter_server 2.5.0
[W 2023-09-08 14:15:03.006 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[I 2023-09-08 14:15:03.417 SingleUserLabApp utils:78] Package xxx_extension took 0.3978s to import
[W 2023-09-08 14:15:03.417 SingleUserLabApp utils:91] A _jupyter_server_extension_points function was not found in XXX_extension. Instead, a _jupyter_server_extension_paths function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-09-08 14:15:04.239 SingleUserLabApp utils:78] Package jupyter_env_whitelist took 0.8215s to import
[I 2023-09-08 14:15:04.244 SingleUserLabApp utils:78] Package jupyter_server_fileid took 0.0042s to import
[I 2023-09-08 14:15:04.247 SingleUserLabApp utils:78] Package jupyter_server_mathjax took 0.0018s to import
[I 2023-09-08 14:15:04.252 SingleUserLabApp utils:78] Package jupyter_server_terminals took 0.0041s to import
[I 2023-09-08 14:15:04.360 SingleUserLabApp utils:78] Package jupyter_server_ydoc took 0.1075s to import
[I 2023-09-08 14:15:04.361 SingleUserLabApp utils:78] Package jupyterlab took 0.0000s to import
[I 2023-09-08 14:15:04.376 SingleUserLabApp utils:78] Package jupyterlab_git took 0.0151s to import
[I 2023-09-08 14:15:04.378 SingleUserLabApp utils:78] Package jupyterlab_templates took 0.0014s to import
[W 2023-09-08 14:15:04.378 SingleUserLabApp utils:91] A _jupyter_server_extension_points function was not found in jupyterlab_templates. Instead, a _jupyter_server_extension_paths function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-09-08 14:15:04.379 SingleUserLabApp utils:78] Package nbclassic took 0.0000s to import
[W 2023-09-08 14:15:04.381 SingleUserLabApp utils:91] A _jupyter_server_extension_points function was not found in nbclassic. Instead, a _jupyter_server_extension_paths function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-09-08 14:15:04.381 SingleUserLabApp utils:78] Package nbdime took 0.0000s to import
[I 2023-09-08 14:15:04.382 SingleUserLabApp utils:78] Package notebook_shim took 0.0000s to import
[W 2023-09-08 14:15:04.382 SingleUserLabApp utils:91] A _jupyter_server_extension_points function was not found in notebook_shim. Instead, a _jupyter_server_extension_paths function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[W 2023-09-08 14:15:04.387 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[W 2023-09-08 14:15:04.388 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[I 2023-09-08 14:15:04.388 SingleUserLabApp manager:344] xxx_extension | extension was successfully linked.
[I 2023-09-08 14:15:04.389 SingleUserLabApp manager:344] jupyter_env_whitelist | extension was successfully linked.
[W 2023-09-08 14:15:04.393 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[W 2023-09-08 14:15:04.394 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[I 2023-09-08 14:15:04.395 SingleUserLabApp manager:344] jupyter_server_fileid | extension was successfully linked.
[W 2023-09-08 14:15:04.408 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[W 2023-09-08 14:15:04.409 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[I 2023-09-08 14:15:04.409 SingleUserLabApp manager:344] jupyter_server_mathjax | extension was successfully linked.
[W 2023-09-08 14:15:04.414 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[W 2023-09-08 14:15:04.415 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[I 2023-09-08 14:15:04.415 SingleUserLabApp manager:344] jupyter_server_terminals | extension was successfully linked.
[W 2023-09-08 14:15:04.420 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[W 2023-09-08 14:15:04.421 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[I 2023-09-08 14:15:04.421 SingleUserLabApp manager:344] jupyter_server_ydoc | extension was successfully linked.
[W 2023-09-08 14:15:04.427 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[I 2023-09-08 14:15:04.427 SingleUserLabApp manager:344] jupyterlab | extension was successfully linked.
[I 2023-09-08 14:15:04.427 SingleUserLabApp manager:344] jupyterlab_git | extension was successfully linked.
[I 2023-09-08 14:15:04.428 SingleUserLabApp manager:344] jupyterlab_templates | extension was successfully linked.
[W 2023-09-08 14:15:04.430 NotebookApp] ‘ip’ has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-09-08 14:15:04.431 NotebookApp] ‘ip’ has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-09-08 14:15:04.431 NotebookApp] ‘ip’ has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-09-08 14:15:04.434 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
[I 2023-09-08 14:15:04.434 SingleUserLabApp manager:344] nbclassic | extension was successfully linked.
[I 2023-09-08 14:15:04.434 SingleUserLabApp manager:344] nbdime | extension was successfully linked.
/opt/conda/lib/python3.10/site-packages/IPython/paths.py:62: UserWarning: IPython dir ‘/home/jovyan/.ipython’ is not a writable location, using a temp directory.
warn(“IPython dir ‘{0}’ is not a writable location,”
[I 2023-09-08 14:15:04.441 SingleUserLabApp utils:78] Package jupyterlab_templates.extension took 0.0000s to import
[I 2023-09-08 14:15:04.441 SingleUserLabApp nbserver:127] jupyterlab_templates.extension | extension was found and enabled by notebook_shim. Consider moving the extension to Jupyter Server’s extension paths.
[I 2023-09-08 14:15:04.441 SingleUserLabApp manager:344] jupyterlab_templates.extension | extension was successfully linked.
[I 2023-09-08 14:15:04.441 SingleUserLabApp manager:344] notebook_shim | extension was successfully linked.
[W 2023-09-08 14:15:04.443 SingleUserLabApp serverapp:1904] Customizing authentication via ServerApp.login_handler_class=<class ‘jupyterhub.singleuser.mixins.make_singleuser_app..JupyterHubLoginHandler’> is deprecated in Jupyter Server 2.0. Use ServerApp.identity_provider_class. Falling back on legacy authentication.
[I 2023-09-08 14:15:04.476 SingleUserLabApp manager:364] notebook_shim | extension was successfully loaded.
[I 2023-09-08 14:15:04.477 SingleUserLabApp manager:364] xxx_extension | extension was successfully loaded.
[I 2023-09-08 14:15:04.477 SingleUserLabApp manager:364] jupyter_env_whitelist | extension was successfully loaded.
[I 2023-09-08 14:15:04.478 FileIdExtension] Configured File ID manager: ArbitraryFileIdManager
[I 2023-09-08 14:15:04.478 FileIdExtension] ArbitraryFileIdManager : Configured root dir: /home/jovyan/notebook
[I 2023-09-08 14:15:04.478 FileIdExtension] ArbitraryFileIdManager : Configured database path: /home/jovyan/.local/share/jupyter/file_id_manager.db
[I 2023-09-08 14:15:04.479 FileIdExtension] ArbitraryFileIdManager : Successfully connected to database file.
[I 2023-09-08 14:15:04.479 FileIdExtension] ArbitraryFileIdManager : Creating File ID tables and indices with journal_mode = DELETE
[I 2023-09-08 14:15:04.508 FileIdExtension] Attached event listeners.
[I 2023-09-08 14:15:04.508 SingleUserLabApp manager:364] jupyter_server_fileid | extension was successfully loaded.
[I 2023-09-08 14:15:04.509 SingleUserLabApp manager:364] jupyter_server_mathjax | extension was successfully loaded.
[I 2023-09-08 14:15:04.514 SingleUserLabApp manager:364] jupyter_server_terminals | extension was successfully loaded.
[I 2023-09-08 14:15:04.515 SingleUserLabApp manager:364] jupyter_server_ydoc | extension was successfully loaded.
[I 2023-09-08 14:15:04.516 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.10/site-packages/jupyterlab
[I 2023-09-08 14:15:04.517 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2023-09-08 14:15:04.521 SingleUserLabApp manager:364] jupyterlab | extension was successfully loaded.
[I 2023-09-08 14:15:04.527 SingleUserLabApp manager:364] jupyterlab_git | extension was successfully loaded.
[I 2023-09-08 14:15:04.528 SingleUserLabApp extension:134] Installing jupyterlab_templates handler on path /user/xxx/templates
[I 2023-09-08 14:15:04.528 SingleUserLabApp extension:148] Search paths:
/opt/conda/lib/python3.10/site-packages/jupyterlab_templates/templates
/home/jovyan/.local/share/jupyter/notebook_templates
/opt/conda/share/jupyter/notebook_templates
/usr/local/share/jupyter/notebook_templates
/usr/share/jupyter/notebook_templates
[I 2023-09-08 14:15:04.529 SingleUserLabApp extension:151] Available templates:

[I 2023-09-08 14:15:04.529 SingleUserLabApp manager:364] jupyterlab_templates | extension was successfully loaded.
[I 2023-09-08 14:15:04.530 SingleUserLabApp extension:134] Installing jupyterlab_templates handler on path /user/xxx/templates
[I 2023-09-08 14:15:04.530 SingleUserLabApp extension:148] Search paths:
/opt/conda/lib/python3.10/site-packages/jupyterlab_templates/templates
/home/jovyan/.local/share/jupyter/notebook_templates
/opt/conda/share/jupyter/notebook_templates
/usr/local/share/jupyter/notebook_templates
/usr/share/jupyter/notebook_templates
[I 2023-09-08 14:15:04.531 SingleUserLabApp extension:151] Available templates:

[I 2023-09-08 14:15:04.531 SingleUserLabApp manager:364] jupyterlab_templates.extension | extension was successfully loaded.
[I 2023-09-08 14:15:04.538 SingleUserLabApp manager:364] nbclassic | extension was successfully loaded.
[I 2023-09-08 14:15:04.732 SingleUserLabApp manager:364] nbdime | extension was successfully loaded.

| | | |_ __ | | | | ___
| || | ’ / / _ | / -)
_/| ./_,_,|____|
|_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[I 2023-09-08 14:15:04.735 SingleUserLabApp mixins:670] Starting jupyterhub-singleuser server version 3.1.1
[W 2023-09-08 14:15:04.753 SingleUserLabApp _version:68] jupyterhub version 3.0.0 != jupyterhub-singleuser version 3.1.1. This could cause failure to authenticate and result in redirect loops!
[I 2023-09-08 14:15:04.754 SingleUserLabApp serverapp:2792] Serving notebooks from local directory: /home/jovyan/notebook
[I 2023-09-08 14:15:04.754 SingleUserLabApp serverapp:2792] Jupyter Server 2.5.0 is running at:
[I 2023-09-08 14:15:04.754 SingleUserLabApp serverapp:2792] http//jupyter-xxx:8888/user/xxx/lab?token=xxxxxxxxxxxx
[I 2023-09-08 14:15:04.754 SingleUserLabApp serverapp:2792] http//127.0.0.1:8888/user/xxx/lab?token=xxxxxxxxxxxx
[I 2023-09-08 14:15:04.754 SingleUserLabApp serverapp:2793] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2023-09-08 14:15:04.765 SingleUserLabApp serverapp:2856]

To access the server, open this file in a browser:
    file:///home/jovyan/.local/share/jupyter/runtime/jpserver-7-open.html
Or copy and paste one of these URLs:
    http//jupyter-xxx:8888/user/xxx/lab?token=xxxxxxxxxxxx
    http//127.0.0.1:8888/user/xxx/lab?token=xxxxxxxxxxxx

[I 2023-09-08 14:15:04.766 SingleUserLabApp mixins:591] Updating Hub with activity every 300 seconds
[I 2023-09-08 14:15:19.928 SingleUserLabApp log:186] 302 GET /user/xxx/ → /user/xxx/lab? (@100.65.56.192) 0.89ms

Used couple of extension that causes pod to take time to start? Any suggestion to start a user server without wait time.

Thanks,
Srujan

4

Can you try obtaining the timestamps for the kubernetes events, the hub logs, and the singleuser pod logs? This will show exactly where the delay is occurring.

6

Thank you for response @manics .
I’ve observed this in user server pod logs
[I 2023-09-08 14:15:**04**.766 SingleUserLabApp mixins:591] Updating Hub with activity every 300 seconds
[I 2023-09-08 14:15:**19.**928 SingleUserLabApp log:186] 302 GET /user/xxx/ → /user/xxx/lab? (@100.65.56.192) 0.89ms

In above logs first and second log has 15sec gap, for redirect the url, can we reduce that time frame?