Slow startup on AMI which has large Data Science image backed in

JL 4 seemed to make things slightly quicker, but, this is still a much slower startup than when the image isn’t pulled with containerd during Image Builder.

[I 2024-08-05 00:46:35.048 ServerApp] Package dask_labextension took 40.5504s to import

Coiled user token is not set. Skipping login.
[D 2024-08-05 00:45:54.158 ServerApp] Searching ['/home/explorer/.config/jupyter', '/python/etc/jupyter', '/usr/local/etc/jupyter', '/etc/xdg/jupyter'] for config files
[D 2024-08-05 00:45:54.158 ServerApp] Looking for jupyter_config in /etc/xdg/jupyter
[D 2024-08-05 00:45:54.158 ServerApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2024-08-05 00:45:54.158 ServerApp] Looking for jupyter_config in /python/etc/jupyter
[D 2024-08-05 00:45:54.159 ServerApp] Looking for jupyter_config in /home/explorer/.config/jupyter
[D 2024-08-05 00:45:54.162 ServerApp] Looking for jupyter_server_config in /etc/xdg/jupyter
[D 2024-08-05 00:45:54.162 ServerApp] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2024-08-05 00:45:54.162 ServerApp] Looking for jupyter_server_config in /python/etc/jupyter
[D 2024-08-05 00:45:54.162 ServerApp] Looking for jupyter_server_config in /home/explorer/.config/jupyter
[D 2024-08-05 00:45:54.169 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/etc/xdg/jupyter/jupyter_server_config.json
[D 2024-08-05 00:45:54.169 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/usr/local/etc/jupyter/jupyter_server_config.json
[D 2024-08-05 00:45:54.244 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/python/etc/jupyter/jupyter_server_config.d/dask_labextension.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyter-server-proxy.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyter_resource_usage.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyter_server_mathjax.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyterlab_git.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyterlab_link_share.json
    	/python/etc/jupyter/jupyter_server_config.d/nbdime.json
    	/python/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    	/python/etc/jupyter/jupyter_server_config.d/panel-client-jupyter.json
    	/python/etc/jupyter/jupyter_server_config.d/trame_jupyter_extension.json
    	/python/etc/jupyter/jupyter_server_config.d/voila.json
    	/python/etc/jupyter/jupyter_server_config.json
[D 2024-08-05 00:45:54.250 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/home/explorer/.config/jupyter/jupyter_server_config.json
[I 2024-08-05 00:45:54.497 ServerApp] Package jupyterhub took 0.0000s to import
# 40s to import. Perhaps slightly quicker?
[I 2024-08-05 00:46:35.048 ServerApp] Package dask_labextension took 40.5504s to import
[W 2024-08-05 00:46:35.048 ServerApp] A `_jupyter_server_extension_points` function was not found in dask_labextension. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-08-05 00:46:35.088 ServerApp] Package jupyter_lsp took 0.0393s to import
[W 2024-08-05 00:46:35.088 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-08-05 00:46:35.209 ServerApp] Package jupyter_resource_usage took 0.1204s to import
[I 2024-08-05 00:46:35.211 ServerApp] Package jupyter_server_mathjax took 0.0022s to import
[I 2024-08-05 00:46:35.211 ServerApp] Package jupyter_server_proxy took 0.0000s to import
[I 2024-08-05 00:46:35.377 ServerApp] Package jupyter_server_terminals took 0.1652s to import
[I 2024-08-05 00:46:36.627 ServerApp] Package jupyterlab took 1.2504s to import

And then here are the logs for a second notebook server using the same image that was scheduled to the same node as the above notebook server. The dask extension loads instantly at this point.

[I 2024-08-05 01:00:19.822 ServerApp] Package dask_labextension took 0.7843s to import

Coiled user token is not set. Skipping login.
[D 2024-08-05 01:00:19.019 ServerApp] Searching ['/home/explorer/.config/jupyter', '/python/etc/jupyter', '/usr/local/etc/jupyter', '/etc/xdg/jupyter'] for config files
[D 2024-08-05 01:00:19.019 ServerApp] Looking for jupyter_config in /etc/xdg/jupyter
[D 2024-08-05 01:00:19.019 ServerApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2024-08-05 01:00:19.020 ServerApp] Looking for jupyter_config in /python/etc/jupyter
[D 2024-08-05 01:00:19.020 ServerApp] Looking for jupyter_config in /home/explorer/.config/jupyter
[D 2024-08-05 01:00:19.021 ServerApp] Looking for jupyter_server_config in /etc/xdg/jupyter
[D 2024-08-05 01:00:19.021 ServerApp] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2024-08-05 01:00:19.021 ServerApp] Looking for jupyter_server_config in /python/etc/jupyter
[D 2024-08-05 01:00:19.021 ServerApp] Looking for jupyter_server_config in /home/explorer/.config/jupyter
[D 2024-08-05 01:00:19.026 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/etc/xdg/jupyter/jupyter_server_config.json
[D 2024-08-05 01:00:19.026 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/usr/local/etc/jupyter/jupyter_server_config.json
[D 2024-08-05 01:00:19.026 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/python/etc/jupyter/jupyter_server_config.d/dask_labextension.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyter-server-proxy.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyter_resource_usage.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyter_server_mathjax.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyterlab_git.json
    	/python/etc/jupyter/jupyter_server_config.d/jupyterlab_link_share.json
    	/python/etc/jupyter/jupyter_server_config.d/nbdime.json
    	/python/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    	/python/etc/jupyter/jupyter_server_config.d/panel-client-jupyter.json
    	/python/etc/jupyter/jupyter_server_config.d/trame_jupyter_extension.json
    	/python/etc/jupyter/jupyter_server_config.d/voila.json
    	/python/etc/jupyter/jupyter_server_config.json
[D 2024-08-05 01:00:19.027 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/home/explorer/.config/jupyter/jupyter_server_config.json
[I 2024-08-05 01:00:19.038 ServerApp] Package jupyterhub took 0.0000s to import
[I 2024-08-05 01:00:19.822 ServerApp] Package dask_labextension took 0.7843s to import
[W 2024-08-05 01:00:19.822 ServerApp] A `_jupyter_server_extension_points` function was not found in dask_labextension. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-08-05 01:00:19.841 ServerApp] Package jupyter_lsp took 0.0187s to import
[W 2024-08-05 01:00:19.841 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-08-05 01:00:19.846 ServerApp] Package jupyter_resource_usage took 0.0044s to import
[I 2024-08-05 01:00:19.847 ServerApp] Package jupyter_server_mathjax took 0.0013s to import
[I 2024-08-05 01:00:19.847 ServerApp] Package jupyter_server_proxy took 0.0000s to import
[I 2024-08-05 01:00:19.858 ServerApp] Package jupyter_server_terminals took 0.0105s to import
[I 2024-08-05 01:00:19.986 ServerApp] Package jupyterlab took 0.1277s to import

Almost seems like the layers need to be warmed up or something?