Lots of 302s, then 431s

JupyterHub
, ,
1

I’m attempting to create a custom spawner to run JupyterHub on Azure Container Apps. I’ve got the spawner working to create the containers in ACA, but I’m really struggling with issues once the notebook container starts and tries to proxy the user to the notebook. What I am seeing currently is there are a slew of 302 redirects, like so:

[ConfigProxy] debug: PROXY WEB /user/sam/ to containerAppHttpUrl:443

Then after about 30 of these, I start getting 431 headers to large errors, which I assume are a result of the many redirects.

2025-05-27T16:04:23.106168836Z 16:04:23.106 [ConfigProxy] debug: PROXY WEB /user/sam/ to containerAppHttpUrl:443

I’ve tried lot’s of different configurations for ENV/settings for the single user container, but no luck. Any suggestions of where to look, or what the cause could be.

The logs in the singleuser notebook don’t say much:

2025-05-27T16:14:10.960913308Z [I 2025-05-27 16:14:10.960 ServerApp] notebook | extension was successfully loaded.
2025-05-27T16:14:10.961749002Z [I 2025-05-27 16:14:10.961 ServerApp] Serving notebooks from local directory: /home/jovyan
2025-05-27T16:14:10.961807902Z [I 2025-05-27 16:14:10.961 ServerApp] Jupyter Server 2.16.0 is running at:

2025-05-27T16:14:10.961957601Z [I 2025-05-27 16:14:10.961 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
2025-05-27T16:14:10.965657878Z [D 2025-05-27 16:14:10.965 ServerApp] notebook_shim | extension was successfully started.
2025-05-27T16:14:10.965923276Z [D 2025-05-27 16:14:10.965 ServerApp] jupyter_lsp | extension was successfully started.
2025-05-27T16:14:10.965935176Z [D 2025-05-27 16:14:10.965 ServerApp] jupyter_server_terminals | extension was successfully started.
2025-05-27T16:14:10.965940976Z [D 2025-05-27 16:14:10.965 ServerApp] jupyterhub | extension was successfully started.
2025-05-27T16:14:10.965985775Z [D 2025-05-27 16:14:10.965 ServerApp] jupyterlab | extension was successfully started.
2025-05-27T16:14:10.966065675Z [D 2025-05-27 16:14:10.966 ServerApp] nbclassic | extension was successfully started.
2025-05-27T16:14:10.966134674Z [D 2025-05-27 16:14:10.966 ServerApp] notebook | extension was successfully started.
2025-05-27T16:14:11.222422336Z [D 2025-05-27 16:14:11.222 ServerApp] No user identified
2025-05-27T16:14:11.223377130Z [I 2025-05-27 16:14:11.222 ServerApp] 200 GET /user/sam/api (@10.10.0.117) 1.32ms
2

Here’s the relevant bits of how I am starting the single user container:

                        "command": ["jupyterhub-singleuser"],
                        "args": [
                            "--ip=0.0.0.0",
                            "--port=8888",
                            "--ServerApp.trust_xheaders=True",
                            "--ServerApp.allow_remote_access=True",
                            "--debug"
                        ],
                        "env": [
                            {"name": "JUPYTERHUB_API_TOKEN", "value": token},
                            {"name": "JUPYTERHUB_SERVICE_PREFIX", "value": f"/user/{self.user.name}/"},
                            {"name": "JUPYTERHUB_API_URL", "value": "https://hub.containerappurl/hub/api"},
                            {"name": "JUPYTERHUB_SERVICE_URL", "value": "http://0.0.0.0:8888"},
                            {"name": "JUPYTERHUB_HOST", "value": "https://hub.containerappurl"},
                            {"name": "JUPYTERHUB_OAUTH_CALLBACK_URL", "value": f"user/{self.user.name}/oauth_callback"},

                        ],
3

Can you turn on debug logging for the hub, and show us the full logs? Can you try adding some debugging logs to your custom spawner? Are you able to share the code for your spawner?

4

Has this problem been solved? I am working on exactly the same task. I managed to create a custom spawner for Azure Container Apps. The spawner works fine. If I create a user in Jupyter Hub, the spawner creates a new Container App based on jupyterhub/singleuser. Immediately afterwards, I get a HTTP 431 error.

I would love to post the log of the hub here. Unfortunately, the web UI doesn’t let me. It thinks that there are too many “links inside”. I tried to fix that by adding some spaces in the URL, but it still refused to post the log. So I put the log here: https://cl1p.net/jupyterhub-problem-431

I have spent quite a while trying to solve this problem. Every help appreciated!

5

Have you checked that network traffic can be sent and received between all containers (hub, proxy, singleuser servers) on the required ports?

If you share your code for your custom spawner we might be able to help. Though it’s possible this can only be debugged with access to Azure Container Apps.

1 Like