Spawned instance ignores override port parameter from JupyterHub spawner

Summary

New image used to spawn single user instances ignores port parameter override from spawner. Instance spawns successfully but tries to bind port 80 rather than 8888 passed from the spawner. The same spawner config works with a previous build of the spawned image, so issue may be in the docker image being spawned but can’t sort through from the blame on the base image.

Environment

JupyterHub version 4.0.0
FargateSpawner GitHub - uktrade/fargatespawner: Spawns JupyterHub single user servers in Docker containers running in AWS Fargate
Base images jupyter/base-notebook:latest & :lab-3.6.3
jupyterhub-singleuser server version 3.1.1 - Functional
jupyterhub single-user server extension version 4.0.0 - Fails

Spawned instances built from image on 3/11 still work great, trying to use a new image build from 5/23 doesn’t spawn.

Logs

Failure

2023-05-23T14:27:03.204-07:00	[I 2023-05-23 21:27:03.204 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.0

2023-05-23T14:27:03.204-07:00	[I 2023-05-23 21:27:03.204 JupyterHubSingleUser] Using default url from server extension lab: /lab

2023-05-23T14:27:03.207-07:00	[I 2023-05-23 21:27:03.207 ServerApp] jupyterhub | extension was successfully linked.

2023-05-23T14:27:03.208-07:00	[W 2023-05-23 21:27:03.208 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.

2023-05-23T14:27:03.211-07:00	[I 2023-05-23 21:27:03.211 ServerApp] jupyterlab | extension was successfully linked.

2023-05-23T14:27:03.213-07:00	[W 2023-05-23 21:27:03.213 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.

2023-05-23T14:27:03.213-07:00	[W 2023-05-23 21:27:03.213 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.

2023-05-23T14:27:03.213-07:00	[W 2023-05-23 21:27:03.213 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.

2023-05-23T14:27:03.216-07:00	[I 2023-05-23 21:27:03.216 ServerApp] nbclassic | extension was successfully linked.

2023-05-23T14:27:03.217-07:00	[I 2023-05-23 21:27:03.216 ServerApp] Writing Jupyter server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret

2023-05-23T14:27:03.378-07:00	[I 2023-05-23 21:27:03.378 ServerApp] notebook_shim | extension was successfully linked.

2023-05-23T14:27:03.487-07:00	[I 2023-05-23 21:27:03.486 ServerApp] notebook_shim | extension was successfully loaded.

2023-05-23T14:27:03.487-07:00	[I 2023-05-23 21:27:03.487 FileIdExtension] Configured File ID manager: ArbitraryFileIdManager

2023-05-23T14:27:03.487-07:00	[I 2023-05-23 21:27:03.487 FileIdExtension] ArbitraryFileIdManager : Configured root dir: /home/jovyan

2023-05-23T14:27:03.487-07:00	[I 2023-05-23 21:27:03.487 FileIdExtension] ArbitraryFileIdManager : Configured database path: /home/jovyan/.local/share/jupyter/file_id_manager.db

2023-05-23T14:27:03.487-07:00	[I 2023-05-23 21:27:03.487 FileIdExtension] ArbitraryFileIdManager : Successfully connected to database file.

2023-05-23T14:27:03.488-07:00	[I 2023-05-23 21:27:03.487 FileIdExtension] ArbitraryFileIdManager : Creating File ID tables and indices with journal_mode = DELETE

2023-05-23T14:27:03.541-07:00	[I 2023-05-23 21:27:03.541 FileIdExtension] Attached event listeners.

2023-05-23T14:27:03.542-07:00	[I 2023-05-23 21:27:03.542 ServerApp] jupyter_server_fileid | extension was successfully loaded.

2023-05-23T14:27:03.543-07:00	[I 2023-05-23 21:27:03.543 ServerApp] jupyter_server_terminals | extension was successfully loaded.

2023-05-23T14:27:03.543-07:00	[I 2023-05-23 21:27:03.543 ServerApp] jupyter_server_ydoc | extension was successfully loaded.

2023-05-23T14:27:03.603-07:00	[I 2023-05-23 21:27:03.603 JupyterHubSingleUser] Updating Hub with activity every 300 seconds

2023-05-23T14:27:03.603-07:00	[I 2023-05-23 21:27:03.603 ServerApp] jupyterhub | extension was successfully loaded.

2023-05-23T14:27:03.604-07:00	[I 2023-05-23 21:27:03.604 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.10/site-packages/jupyterlab

2023-05-23T14:27:03.604-07:00	[I 2023-05-23 21:27:03.604 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab

2023-05-23T14:27:03.607-07:00	[I 2023-05-23 21:27:03.607 ServerApp] jupyterlab | extension was successfully loaded.

2023-05-23T14:27:03.610-07:00	[I 2023-05-23 21:27:03.610 ServerApp] nbclassic | extension was successfully loaded.

2023-05-23T14:27:03.610-07:00	[W 2023-05-23 21:27:03.610 ServerApp] Permission to listen on port 80 denied.

2023-05-23T14:27:03.610-07:00	[C 2023-05-23 21:27:03.610 ServerApp] ERROR: the Jupyter server could not be started because port 80 is not available.

Functional

2023-05-23T14:41:56.692-07:00	[I 2023-05-23 21:41:56.692 SingleUserLabApp mixins:670] Starting jupyterhub-singleuser server version 3.1.1

2023-05-23T14:41:56.751-07:00	[W 2023-05-23 21:41:56.751 SingleUserLabApp _version:68] jupyterhub version 4.0.0 != jupyterhub-singleuser version 3.1.1. This could cause failure to authenticate and result in redirect loops!

2023-05-23T14:41:56.751-07:00	[I 2023-05-23 21:41:56.751 SingleUserLabApp serverapp:2792] Serving notebooks from local directory: /home/jovyan

2023-05-23T14:41:56.751-07:00	[I 2023-05-23 21:41:56.751 SingleUserLabApp serverapp:2792] Jupyter Server 2.3.0 is running at:

2023-05-23T14:41:56.751-07:00	[I 2023-05-23 21:41:56.751 SingleUserLabApp serverapp:2792] http://ip-172-31-101-137.ec2.internal:8888/user/allan%20deyoung/lab?token=

2023-05-23T14:41:56.751-07:00	[I 2023-05-23 21:41:56.751 SingleUserLabApp serverapp:2792] http://127.0.0.1:8888/user/allan%20deyoung/lab?token=

2023-05-23T14:41:56.751-07:00	[I 2023-05-23 21:41:56.751 SingleUserLabApp serverapp:2793] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

2023-05-23T14:41:56.754-07:00	[C 2023-05-23 21:41:56.754 SingleUserLabApp serverapp:2856]

2023-05-23T14:41:56.754-07:00	

2023-05-23T14:41:56.754-07:00	To access the server, open this file in a browser:

2023-05-23T14:41:56.754-07:00	file:///home/jovyan/.local/share/jupyter/runtime/jpserver-7-open.html

2023-05-23T14:41:56.754-07:00	Or copy and paste one of these URLs:

2023-05-23T14:41:56.754-07:00	http://ip-172-31-101-137.ec2.internal:8888/user/allan%20deyoung/lab?token=

2023-05-23T14:41:56.754-07:00	http://127.0.0.1:8888/user/allan%20deyoung/lab?token=

Can you check what command line arguments and environment variables are passed to the singleuser image by the spawner?

Ideally the port should be passed to jupyterhub-singleuser as part of the JUPYTERHUB_SERVICE_URL environment variable and not on the command line (--port is deprecated).

There’s a recent PR to restore the old behaviour

but it would be best to update the spawner if that’s the problem.

1 Like

I had definitely been using the --port parameter and not the env variable. I saw the release notes last night with the JUPYTERHUB_SINGLEUSER_EXTENSION env variable, but that issue reference really helps with figuring out the right way to fix it.

'command': spawner.cmd + [f'--port={spawner.notebook_port}', '--config=notebook_config.py'],

Do you have an example of how the JUPYTERHUB_SERVICE_URL is implemented?

My initial thought is something like:

c.Spawner.environment.update(
    {
        "JUPYTERHUB_SERVICE_URL": "http://0.0.0.0:8888",
    }
)

I would likely have to add this to the env_keep as well?

I can tinker with this, but that helped get on the right path.

Success!

Thanks @manics

For reference the changes that fixed my use case. All changes occurred in jupyterhub_config.py.

Remove port parameter from container override arguements

            'command': spawner.cmd, #+ [f'--port={spawner.notebook_port}', '--config=notebook_config.py'],

Added the environment variable

c.Spawner.environment = {
        'JUPYTERHUB_SERVICE_URL':'http://0.0.0.0:8888'
}

Added the env variable to be kept

c.Spawner.env_keep = ['PYTHONPATH', 'CONDA_ROOT', 'CONDA_DEFAULT_ENV', 'VIRTUAL_ENV', 'LANG', 'LC_ALL', 'JUPYTERHUB_SINGLEUSER_APP','JUPYTERHUB_SERVICE_URL']

This may be a little heavy handed so happy for corrections too, but at least got things functional.

Functional logs

2023-05-24T05:30:37.228-07:00	[I 2023-05-24 12:30:37.228 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.0

2023-05-24T05:30:37.228-07:00	[I 2023-05-24 12:30:37.228 JupyterHubSingleUser] Using default url from server extension lab: /lab

2023-05-24T05:30:37.231-07:00	[I 2023-05-24 12:30:37.231 ServerApp] jupyterhub | extension was successfully linked.

2023-05-24T05:30:37.232-07:00	[W 2023-05-24 12:30:37.232 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.

2023-05-24T05:30:37.235-07:00	[I 2023-05-24 12:30:37.235 ServerApp] jupyterlab | extension was successfully linked.

2023-05-24T05:30:37.237-07:00	[W 2023-05-24 12:30:37.237 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.

2023-05-24T05:30:37.237-07:00	[W 2023-05-24 12:30:37.237 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.

2023-05-24T05:30:37.237-07:00	[W 2023-05-24 12:30:37.237 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.

2023-05-24T05:30:37.239-07:00	[I 2023-05-24 12:30:37.239 ServerApp] nbclassic | extension was successfully linked.

2023-05-24T05:30:37.240-07:00	[I 2023-05-24 12:30:37.240 ServerApp] Writing Jupyter server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret

2023-05-24T05:30:37.394-07:00	[I 2023-05-24 12:30:37.394 ServerApp] notebook_shim | extension was successfully linked.

2023-05-24T05:30:37.498-07:00	[I 2023-05-24 12:30:37.497 ServerApp] notebook_shim | extension was successfully loaded.

2023-05-24T05:30:37.498-07:00	[I 2023-05-24 12:30:37.498 FileIdExtension] Configured File ID manager: ArbitraryFileIdManager

2023-05-24T05:30:37.498-07:00	[I 2023-05-24 12:30:37.498 FileIdExtension] ArbitraryFileIdManager : Configured root dir: /home/jovyan

2023-05-24T05:30:37.498-07:00	[I 2023-05-24 12:30:37.498 FileIdExtension] ArbitraryFileIdManager : Configured database path: /home/jovyan/.local/share/jupyter/file_id_manager.db

2023-05-24T05:30:37.498-07:00	[I 2023-05-24 12:30:37.498 FileIdExtension] ArbitraryFileIdManager : Successfully connected to database file.

2023-05-24T05:30:37.498-07:00	[I 2023-05-24 12:30:37.498 FileIdExtension] ArbitraryFileIdManager : Creating File ID tables and indices with journal_mode = DELETE

2023-05-24T05:30:37.512-07:00	[I 2023-05-24 12:30:37.512 FileIdExtension] Attached event listeners.

2023-05-24T05:30:37.512-07:00	[I 2023-05-24 12:30:37.512 ServerApp] jupyter_server_fileid | extension was successfully loaded.

2023-05-24T05:30:37.513-07:00	[I 2023-05-24 12:30:37.513 ServerApp] jupyter_server_terminals | extension was successfully loaded.

2023-05-24T05:30:37.514-07:00	[I 2023-05-24 12:30:37.514 ServerApp] jupyter_server_ydoc | extension was successfully loaded.

2023-05-24T05:30:37.575-07:00	[I 2023-05-24 12:30:37.575 JupyterHubSingleUser] Updating Hub with activity every 300 seconds

2023-05-24T05:30:37.575-07:00	[I 2023-05-24 12:30:37.575 ServerApp] jupyterhub | extension was successfully loaded.

2023-05-24T05:30:37.576-07:00	[I 2023-05-24 12:30:37.575 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.10/site-packages/jupyterlab

2023-05-24T05:30:37.576-07:00	[I 2023-05-24 12:30:37.576 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab

2023-05-24T05:30:37.579-07:00	[I 2023-05-24 12:30:37.579 ServerApp] jupyterlab | extension was successfully loaded.

2023-05-24T05:30:37.581-07:00	[I 2023-05-24 12:30:37.581 ServerApp] nbclassic | extension was successfully loaded.

2023-05-24T05:30:37.582-07:00	[I 2023-05-24 12:30:37.582 ServerApp] Serving notebooks from local directory: /home/jovyan

2023-05-24T05:30:37.582-07:00	[I 2023-05-24 12:30:37.582 ServerApp] Jupyter Server 2.5.0 is running at:

2023-05-24T05:30:37.582-07:00	[I 2023-05-24 12:30:37.582 ServerApp] http://ip-172-31-85-239.ec2.internal:8888/user/allan%20deyoung/lab?token=...

2023-05-24T05:30:37.582-07:00	[I 2023-05-24 12:30:37.582 ServerApp] http://127.0.0.1:8888/user/allan%20deyoung/lab?token=...

JUPYTERHUB_SERVICE_URL should be set by the core hub application and passed to the spawner, so I think setting c.spawner.port = 8888
along with

should be enough:

Arguably https://github.com/uktrade/fargatespawner/blob/a57018941d582370fb0ded4f59381c58f9a11d58/fargatespawner/fargatespawner.py could default to port = 8888, I can’t of a good reason not to.

2 Likes

Spot on, was able to instead set:

c.Spawner.ip = '0.0.0.0'
c.Spawner.port = 8888

My Python skills are worse than I thought. I was tinkering with a way to set a default value in the FargateSpawner class, but I can’t get the correct syntax.

I appreciate the guidance @manics!

1 Like