Summary
New image used to spawn single user instances ignores port parameter override from spawner. Instance spawns successfully but tries to bind port 80 rather than 8888 passed from the spawner. The same spawner config works with a previous build of the spawned image, so issue may be in the docker image being spawned but can’t sort through from the blame on the base image.
Environment
JupyterHub version 4.0.0
FargateSpawner GitHub - uktrade/fargatespawner: Spawns JupyterHub single user servers in Docker containers running in AWS Fargate
Base images jupyter/base-notebook:latest & :lab-3.6.3
jupyterhub-singleuser server version 3.1.1 - Functional
jupyterhub single-user server extension version 4.0.0 - Fails
Spawned instances built from image on 3/11 still work great, trying to use a new image build from 5/23 doesn’t spawn.
Logs
Failure
2023-05-23T14:27:03.204-07:00 [I 2023-05-23 21:27:03.204 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.0
2023-05-23T14:27:03.204-07:00 [I 2023-05-23 21:27:03.204 JupyterHubSingleUser] Using default url from server extension lab: /lab
2023-05-23T14:27:03.207-07:00 [I 2023-05-23 21:27:03.207 ServerApp] jupyterhub | extension was successfully linked.
2023-05-23T14:27:03.208-07:00 [W 2023-05-23 21:27:03.208 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
2023-05-23T14:27:03.211-07:00 [I 2023-05-23 21:27:03.211 ServerApp] jupyterlab | extension was successfully linked.
2023-05-23T14:27:03.213-07:00 [W 2023-05-23 21:27:03.213 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2023-05-23T14:27:03.213-07:00 [W 2023-05-23 21:27:03.213 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2023-05-23T14:27:03.213-07:00 [W 2023-05-23 21:27:03.213 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
2023-05-23T14:27:03.216-07:00 [I 2023-05-23 21:27:03.216 ServerApp] nbclassic | extension was successfully linked.
2023-05-23T14:27:03.217-07:00 [I 2023-05-23 21:27:03.216 ServerApp] Writing Jupyter server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret
2023-05-23T14:27:03.378-07:00 [I 2023-05-23 21:27:03.378 ServerApp] notebook_shim | extension was successfully linked.
2023-05-23T14:27:03.487-07:00 [I 2023-05-23 21:27:03.486 ServerApp] notebook_shim | extension was successfully loaded.
2023-05-23T14:27:03.487-07:00 [I 2023-05-23 21:27:03.487 FileIdExtension] Configured File ID manager: ArbitraryFileIdManager
2023-05-23T14:27:03.487-07:00 [I 2023-05-23 21:27:03.487 FileIdExtension] ArbitraryFileIdManager : Configured root dir: /home/jovyan
2023-05-23T14:27:03.487-07:00 [I 2023-05-23 21:27:03.487 FileIdExtension] ArbitraryFileIdManager : Configured database path: /home/jovyan/.local/share/jupyter/file_id_manager.db
2023-05-23T14:27:03.487-07:00 [I 2023-05-23 21:27:03.487 FileIdExtension] ArbitraryFileIdManager : Successfully connected to database file.
2023-05-23T14:27:03.488-07:00 [I 2023-05-23 21:27:03.487 FileIdExtension] ArbitraryFileIdManager : Creating File ID tables and indices with journal_mode = DELETE
2023-05-23T14:27:03.541-07:00 [I 2023-05-23 21:27:03.541 FileIdExtension] Attached event listeners.
2023-05-23T14:27:03.542-07:00 [I 2023-05-23 21:27:03.542 ServerApp] jupyter_server_fileid | extension was successfully loaded.
2023-05-23T14:27:03.543-07:00 [I 2023-05-23 21:27:03.543 ServerApp] jupyter_server_terminals | extension was successfully loaded.
2023-05-23T14:27:03.543-07:00 [I 2023-05-23 21:27:03.543 ServerApp] jupyter_server_ydoc | extension was successfully loaded.
2023-05-23T14:27:03.603-07:00 [I 2023-05-23 21:27:03.603 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
2023-05-23T14:27:03.603-07:00 [I 2023-05-23 21:27:03.603 ServerApp] jupyterhub | extension was successfully loaded.
2023-05-23T14:27:03.604-07:00 [I 2023-05-23 21:27:03.604 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.10/site-packages/jupyterlab
2023-05-23T14:27:03.604-07:00 [I 2023-05-23 21:27:03.604 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
2023-05-23T14:27:03.607-07:00 [I 2023-05-23 21:27:03.607 ServerApp] jupyterlab | extension was successfully loaded.
2023-05-23T14:27:03.610-07:00 [I 2023-05-23 21:27:03.610 ServerApp] nbclassic | extension was successfully loaded.
2023-05-23T14:27:03.610-07:00 [W 2023-05-23 21:27:03.610 ServerApp] Permission to listen on port 80 denied.
2023-05-23T14:27:03.610-07:00 [C 2023-05-23 21:27:03.610 ServerApp] ERROR: the Jupyter server could not be started because port 80 is not available.
Functional
2023-05-23T14:41:56.692-07:00 [I 2023-05-23 21:41:56.692 SingleUserLabApp mixins:670] Starting jupyterhub-singleuser server version 3.1.1
2023-05-23T14:41:56.751-07:00 [W 2023-05-23 21:41:56.751 SingleUserLabApp _version:68] jupyterhub version 4.0.0 != jupyterhub-singleuser version 3.1.1. This could cause failure to authenticate and result in redirect loops!
2023-05-23T14:41:56.751-07:00 [I 2023-05-23 21:41:56.751 SingleUserLabApp serverapp:2792] Serving notebooks from local directory: /home/jovyan
2023-05-23T14:41:56.751-07:00 [I 2023-05-23 21:41:56.751 SingleUserLabApp serverapp:2792] Jupyter Server 2.3.0 is running at:
2023-05-23T14:41:56.751-07:00 [I 2023-05-23 21:41:56.751 SingleUserLabApp serverapp:2792] http://ip-172-31-101-137.ec2.internal:8888/user/allan%20deyoung/lab?token=
2023-05-23T14:41:56.751-07:00 [I 2023-05-23 21:41:56.751 SingleUserLabApp serverapp:2792] http://127.0.0.1:8888/user/allan%20deyoung/lab?token=
2023-05-23T14:41:56.751-07:00 [I 2023-05-23 21:41:56.751 SingleUserLabApp serverapp:2793] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
2023-05-23T14:41:56.754-07:00 [C 2023-05-23 21:41:56.754 SingleUserLabApp serverapp:2856]
2023-05-23T14:41:56.754-07:00
2023-05-23T14:41:56.754-07:00 To access the server, open this file in a browser:
2023-05-23T14:41:56.754-07:00 file:///home/jovyan/.local/share/jupyter/runtime/jpserver-7-open.html
2023-05-23T14:41:56.754-07:00 Or copy and paste one of these URLs:
2023-05-23T14:41:56.754-07:00 http://ip-172-31-101-137.ec2.internal:8888/user/allan%20deyoung/lab?token=
2023-05-23T14:41:56.754-07:00 http://127.0.0.1:8888/user/allan%20deyoung/lab?token=