Problems with implementing a spawner for Azure Container Apps

JupyterHub
1

I am currently in the process of implementing a spawner for Azure Container Apps. I am making progress. You can find the current version (by far not production ready, work in progress) at https://github.com/rstropek/AcaJupyterHubSpawner/tree/main/acaspawner.

The spawner does what it should. I can spin up an Azure Container Apps Environment and run JupyterHub with my spawner in it. If I create the first user, the spawner is triggered and dynamically deploys another Azure Container App. It correctly reports back to the JupyterHub. So far, so good.

However, if I try to access the environment, I am stuck with HTTP 431 errors. I guess it has something to do with endless back and forth in the auth flow, but I am not sure. I have already spent many hours trying to find a configuration that works - without success. I am looking for ideas/help to make that work. If it will work, I will happily share the spawner with the community.

So here is my setup:

Connecting to stream...
2025-11-12T18:11:03.75142  Connecting to the container 'jupyterhub'...
2025-11-12T18:11:03.78482  Successfully Connected to container: 'jupyterhub' [Revision: 'jupyterhub--0000002', Replica: 'jupyterhub--0000002-5d4787fff8-g8dn7']
...
2025-11-12T18:09:49.8388629Z stdout F 2025-11-12T18:09:49.837Z [ConfigProxy] info: Proxying http://0.0.0.0:8000 to (no default)
2025-11-12T18:09:49.8395515Z stdout F 2025-11-12T18:09:49.839Z [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
2025-11-12T18:09:49.9048571Z stderr F [D 2025-11-12 18:09:49.904 JupyterHub utils:286] Server at 127.0.0.1:8001 responded in 0.35s
2025-11-12T18:09:50.0174078Z stderr F [D 2025-11-12 18:09:50.017 JupyterHub utils:286] Server at jupyterhub--0000002-5d4787fff8-g8dn7:8000 responded in 0.46s
2025-11-12T18:09:50.0175447Z stderr F [D 2025-11-12 18:09:50.017 JupyterHub proxy:832] Proxy started and appears to be up
2025-11-12T18:09:50.0184514Z stderr F [D 2025-11-12 18:09:50.018 JupyterHub proxy:925] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
2025-11-12T18:09:50.0258450Z stderr F [I 2025-11-12 18:09:50.025 JupyterHub app:3752] Hub API listening on http://0.0.0.0:8081/hub/
2025-11-12T18:09:50.0261299Z stderr F [D 2025-11-12 18:09:50.026 JupyterHub proxy:389] Fetching routes to check
2025-11-12T18:09:50.0264438Z stderr F [D 2025-11-12 18:09:50.026 JupyterHub proxy:925] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
2025-11-12T18:09:50.0268057Z stdout F 2025-11-12T18:09:50.026Z [ConfigProxy] info: 200 GET /api/routes 
2025-11-12T18:09:50.0288010Z stdout F 2025-11-12T18:09:50.028Z [ConfigProxy] info: 200 GET /api/routes 
2025-11-12T18:09:50.0321183Z stdout F 2025-11-12T18:09:50.031Z [ConfigProxy] info: Adding route / -> http://0.0.0.0:8081
2025-11-12T18:09:50.0294272Z stderr F [D 2025-11-12 18:09:50.029 JupyterHub proxy:392] Checking routes
2025-11-12T18:09:50.0324941Z stderr F [I 2025-11-12 18:09:50.029 JupyterHub proxy:477] Adding route for Hub: / => http://0.0.0.0:8081
2025-11-12T18:09:50.0325074Z stderr F [D 2025-11-12 18:09:50.029 JupyterHub proxy:925] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
2025-11-12T18:09:50.0338062Z stdout F 2025-11-12T18:09:50.033Z [ConfigProxy] info: Route added / -> http://0.0.0.0:8081
2025-11-12T18:09:50.0344449Z stderr F [I 2025-11-12 18:09:50.034 JupyterHub app:3783] JupyterHub is now running at http://0.0.0.0:8000
2025-11-12T18:09:50.0351151Z stderr F [D 2025-11-12 18:09:50.034 JupyterHub app:3352] It took 1.033 seconds for the Hub to start
2025-11-12T18:09:50.0352433Z stdout F 2025-11-12T18:09:50.034Z [ConfigProxy] info: 201 POST /api/routes/ 
...
2025-11-12T18:12:46.7826367Z stderr F [I 2025-11-12 18:12:46.782 JupyterHub base:973] User logged in: admin
2025-11-12T18:12:46.7830901Z stderr F [I 2025-11-12 18:12:46.782 JupyterHub log:192] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (admin@100.100.0.35) 28.80ms
2025-11-12T18:12:46.8365525Z stderr F [D 2025-11-12 18:12:46.835 JupyterHub base:366] Recording first activity for <User(admin 0/1 running)>
2025-11-12T18:12:46.8420159Z stderr F [D 2025-11-12 18:12:46.841 JupyterHub user:496] Creating <class 'acaspawner.acaspawner.AcaSpawner'> for admin:
2025-11-12T18:12:46.8427589Z stderr F [I 2025-11-12 18:12:46.842 JupyterHub log:192] 302 GET /hub/ -> /hub/spawn (admin@100.100.0.35) 15.47ms
2025-11-12T18:12:46.8939948Z stderr F [D 2025-11-12 18:12:46.893 JupyterHub scopes:1013] Checking access to /hub/spawn via scope servers!server=admin/
2025-11-12T18:12:46.8941403Z stderr F [D 2025-11-12 18:12:46.894 JupyterHub pages:241] Triggering spawn with default options for admin
2025-11-12T18:12:46.8943373Z stderr F [D 2025-11-12 18:12:46.894 JupyterHub base:1097] Initiating spawn for admin
2025-11-12T18:12:46.8943652Z stderr F [D 2025-11-12 18:12:46.894 JupyterHub base:1101] 0/100 concurrent spawns
2025-11-12T18:12:46.8944155Z stderr F [D 2025-11-12 18:12:46.894 JupyterHub base:1106] 0 active servers
2025-11-12T18:12:46.9160109Z stderr F [I 2025-11-12 18:12:46.915 JupyterHub provider:661] Creating oauth client jupyterhub-user-admin
2025-11-12T18:12:46.9365546Z stderr F [D 2025-11-12 18:12:46.936 JupyterHub user:913] Calling Spawner.start for admin
2025-11-12T18:12:47.0307577Z stderr F [I 2025-11-12 18:12:47.030 JupyterHub acaspawner:193] Creating ACA aca74dfc542a6474fd58ace3fcbff722 in environment /subscriptions/b33f0285-db27-4896-ac5c-df22004b0aba/resourceGroups/JupyterHubACA2/providers/Microsoft.App/managedEnvironments/cae-acddj77mnwd6i
2025-11-12T18:12:47.0311136Z stderr F [I 2025-11-12 18:12:47.030 JupyterHub acaspawner:201] Setting JUPYTERHUB_API_URL to: https://jupyterhub.blackdune-1938d872.swedencentral.azurecontainerapps.io/hub/api
2025-11-12T18:12:47.0314216Z stderr F [I 2025-11-12 18:12:47.031 JupyterHub acaspawner:231] Environment variable JUPYTERHUB_API_URL: https://jupyterhub.blackdune-1938d872.swedencentral.azurecontainerapps.io/hub/api
2025-11-12T18:12:47.0314368Z stderr F [I 2025-11-12 18:12:47.031 JupyterHub acaspawner:229] Environment variable JUPYTERHUB_API_TOKEN set (length: 32)
2025-11-12T18:12:47.0314441Z stderr F [I 2025-11-12 18:12:47.031 JupyterHub acaspawner:231] Environment variable JUPYTERHUB_CLIENT_ID: jupyterhub-user-admin
2025-11-12T18:12:47.0314889Z stderr F [I 2025-11-12 18:12:47.031 JupyterHub acaspawner:233] Using ACR server cracddj77mnwd6i.azurecr.io with identity /subscriptions/b33f0285-db27-4896-ac5c-df22004b0aba/resourcegroups/JupyterHubACA2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-cr-acddj77mnwd6i
2025-11-12T18:12:47.8958696Z stderr F [I 2025-11-12 18:12:47.895 JupyterHub log:192] 302 GET /hub/spawn -> /hub/spawn-pending/admin (admin@100.100.0.35) 1005.34ms
2025-11-12T18:12:47.9468830Z stderr F [D 2025-11-12 18:12:47.946 JupyterHub scopes:1013] Checking access to /hub/spawn-pending/admin via scope servers!server=admin/
2025-11-12T18:12:47.9470645Z stderr F [I 2025-11-12 18:12:47.946 JupyterHub pages:441] admin is pending spawn
2025-11-12T18:12:47.9497147Z stderr F [I 2025-11-12 18:12:47.949 JupyterHub log:192] 200 GET /hub/spawn-pending/admin (admin@100.100.0.35) 6.37ms
...
2025-11-12T18:12:48.0828518Z stderr F [D 2025-11-12 18:12:48.082 JupyterHub scopes:1013] Checking access to /hub/api/users/admin/server/progress via scope read:servers!server=admin/
...
2025-11-12T18:13:08.6746429Z stderr F [D 2025-11-12 18:13:08.674 JupyterHub base:366] Recording first activity for <APIToken('acfc...', user='admin', client_id='jupyterhub')>
2025-11-12T18:13:08.6824228Z stderr F [D 2025-11-12 18:13:08.682 JupyterHub scopes:1013] Checking access to /hub/api/users/admin/activity via scope users:activity!user=admin
2025-11-12T18:13:08.6848390Z stderr F [D 2025-11-12 18:13:08.684 JupyterHub users:1005] Activity for user admin: 2025-11-12T18:13:08.476292Z
2025-11-12T18:13:08.6848955Z stderr F [D 2025-11-12 18:13:08.684 JupyterHub users:1023] Activity on server admin/: 2025-11-12T18:13:08.476292Z
2025-11-12T18:13:08.6908403Z stderr F [I 2025-11-12 18:13:08.690 JupyterHub log:192] 200 POST /hub/api/users/admin/activity (admin@100.100.0.173) 18.60ms
2025-11-12T18:13:20.0397074Z stderr F [I 2025-11-12 18:13:20.039 JupyterHub acaspawner:274] ACA aca74dfc542a6474fd58ace3fcbff722 created
2025-11-12T18:13:20.5398268Z stderr F [D 2025-11-12 18:13:20.539 JupyterHub spawner:1706] Polling subprocess every 30s
2025-11-12T18:13:20.5466923Z stderr F [D 2025-11-12 18:13:20.546 JupyterHub utils:298] Waiting 30s for server at https://aca74dfc542a6474fd58ace3fcbff722.blackdune-1938d872.swedencentral.azurecontainerapps.io:443/user/admin/api
2025-11-12T18:13:20.6176051Z stderr F [D 2025-11-12 18:13:20.617 JupyterHub utils:334] Server at https://aca74dfc542a6474fd58ace3fcbff722.blackdune-1938d872.swedencentral.azurecontainerapps.io:443/user/admin/api responded in 0.07s
2025-11-12T18:13:20.6176718Z stderr F [D 2025-11-12 18:13:20.617 JupyterHub _version:73] jupyterhub and jupyterhub-singleuser both on version 5.4.2
2025-11-12T18:13:20.6176979Z stderr F [I 2025-11-12 18:13:20.617 JupyterHub base:1126] User admin took 33.723 seconds to start
2025-11-12T18:13:20.6178229Z stderr F [I 2025-11-12 18:13:20.617 JupyterHub proxy:331] Adding user admin to proxy /user/admin/ => https://aca74dfc542a6474fd58ace3fcbff722.blackdune-1938d872.swedencentral.azurecontainerapps.io:443
2025-11-12T18:13:20.6178700Z stderr F [D 2025-11-12 18:13:20.617 JupyterHub proxy:925] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/admin
2025-11-12T18:13:20.6194833Z stdout F 2025-11-12T18:13:20.619Z [ConfigProxy] info: Adding route /user/admin -> https://aca74dfc542a6474fd58ace3fcbff722.blackdune-1938d872.swedencentral.azurecontainerapps.io:443
2025-11-12T18:13:20.6200023Z stdout F 2025-11-12T18:13:20.619Z [ConfigProxy] info: Route added /user/admin -> https://aca74dfc542a6474fd58ace3fcbff722.blackdune-1938d872.swedencentral.azurecontainerapps.io:443
2025-11-12T18:13:20.6204192Z stdout F 2025-11-12T18:13:20.620Z [ConfigProxy] info: 201 POST /api/routes/user/admin 
2025-11-12T18:13:20.6208410Z stderr F [I 2025-11-12 18:13:20.620 JupyterHub users:899] Server admin is ready
2025-11-12T18:13:20.6213796Z stderr F [I 2025-11-12 18:13:20.621 JupyterHub log:192] 200 GET /hub/api/users/admin/server/progress?_xsrf=[secret] (admin@100.100.0.35) 32541.95ms
2025-11-12T18:13:20.6772221Z stderr F [D 2025-11-12 18:13:20.676 JupyterHub scopes:1013] Checking access to /hub/spawn-pending/admin via scope servers!server=admin/
2025-11-12T18:13:20.6776419Z stderr F [I 2025-11-12 18:13:20.677 JupyterHub log:192] 302 GET /hub/spawn-pending/admin -> /user/admin/ (admin@100.100.0.35) 3.82ms
2025-11-12T18:13:50.3487833Z stderr F [I 2025-11-12 18:13:50.348 JupyterHub acaspawner:324] Polling ACA aca74dfc542a6474fd58ace3fcbff722
...

2025-11-12T18:13:08.6378897Z stderr F [I 2025-11-12 18:13:08.637 ServerApp] Jupyter Server 2.17.0 is running at:
2025-11-12T18:13:08.6378966Z stderr F [I 2025-11-12 18:13:08.637 ServerApp] http://aca74dfc542a6474fd58ace3fcbff722--mphvbc6-c8ffbdb7b-tjkjg:8888/user/admin/lab?token=...
2025-11-12T18:13:08.6379330Z stderr F [I 2025-11-12 18:13:08.637 ServerApp]     http://127.0.0.1:8888/user/admin/lab?token=...
2025-11-12T18:13:08.6379380Z stderr F [I 2025-11-12 18:13:08.637 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
2025-11-12T18:13:08.6404813Z stderr F [D 2025-11-12 18:13:08.640 JupyterHubSingleUser] Notifying Hub of activity 2025-11-12T18:13:08.476292Z
2025-11-12T18:13:08.6410200Z stderr F [D 2025-11-12 18:13:08.640 ServerApp] [lsp] None of the installed servers require virtual documents disabling shadow filesystem.
2025-11-12T18:13:08.6411145Z stderr F [D 2025-11-12 18:13:08.640 ServerApp] [lsp] The following Language Servers will be available: {}
2025-11-12T18:13:08.6414284Z stderr F [D 2025-11-12 18:13:08.641 ServerApp] notebook_shim | extension was successfully started.
2025-11-12T18:13:08.6414919Z stderr F [D 2025-11-12 18:13:08.641 ServerApp] jupyter_lsp | extension was successfully started.
2025-11-12T18:13:08.6415681Z stderr F [D 2025-11-12 18:13:08.641 ServerApp] jupyter_server_terminals | extension was successfully started.
2025-11-12T18:13:08.6417149Z stderr F [D 2025-11-12 18:13:08.641 ServerApp] jupyterhub | extension was successfully started.
2025-11-12T18:13:08.6417696Z stderr F [D 2025-11-12 18:13:08.641 ServerApp] jupyterlab | extension was successfully started.
2025-11-12T18:13:08.6418142Z stderr F [D 2025-11-12 18:13:08.641 ServerApp] nbclassic | extension was successfully started.
2025-11-12T18:13:08.6418235Z stderr F [D 2025-11-12 18:13:08.641 ServerApp] notebook | extension was successfully started.
2025-11-12T18:13:20.6040863Z stderr F [D 2025-11-12 18:13:20.603 ServerApp] No user identified
2025-11-12T18:13:20.6045061Z stderr F [I 2025-11-12 18:13:20.604 ServerApp] 200 GET /user/admin/api (@20.240.28.167) 0.99ms
...
2025-11-12T18:17:56.8328880Z stderr F [D 2025-11-12 18:17:56.832 JupyterHubSingleUser] Notifying Hub of activity 2025-11-12T18:13:08.476292Z
  • Here are the environment variables of the spawned server:
JUPYTERHUB_API_TOKEN: ...
JPY_API_TOKEN: ...
JUPYTERHUB_CLIENT_ID: jupyterhub-user-admin
JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED: 0
JUPYTERHUB_HOST: 
JUPYTERHUB_OAUTH_CALLBACK_URL: /user/admin/oauth_callback
JUPYTERHUB_OAUTH_SCOPES: ["access:servers!server=admin/", "access:servers!user=admin"]
JUPYTERHUB_OAUTH_ACCESS_SCOPES: ["access:servers!server=admin/", "access:servers!user=admin"]
JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES: []
JUPYTERHUB_USER: admin
JUPYTERHUB_SERVER_NAME: 
JUPYTERHUB_API_URL: https://jupyterhub.blackdune-1938d872.swedencentral.azurecontainerapps.io/hub/api
JUPYTERHUB_ACTIVITY_URL: https://jupyterhub.blackdune-1938d872.swedencentral.azurecontainerapps.io/hub/api/users/admin/activity
JUPYTERHUB_BASE_URL: /
JUPYTERHUB_SERVICE_PREFIX: /user/admin/
JUPYTERHUB_SERVICE_URL: http://127.0.0.1:0/user/admin/
JUPYTERHUB_PUBLIC_URL: 
JUPYTERHUB_PUBLIC_HUB_URL:

If you want to take a look at Azure details, the entire environment is deployed with Bicep infrastructure-as-code scripts ( AcaJupyterHubSpawner/DevOps at main · rstropek/AcaJupyterHubSpawner · GitHub ).

Every help or tip is appreciated!

2

Can you share your logs for the period when the HTTP 431 errors occur?

3

The logs are included in my original message. They were collected while the 431 error occurred.

My guess is that the networking stack of Azure Container Apps does not play nicely with JupyterHub. If you have ideas for further analysis steps, that would help a lot.

4

I can’t see anything in the hub logs that could relate to a [431 Request Header Fields Too Large](431 Request Header Fields Too Large - HTTP | MDN) error, so I think you’ll need to get some debug logs out of whatever Azure uses as a proxy.

5

Enabling debug logging for the ConfigurableHttpProxy yields:

.wittysea-.swedencentral.azurecontainerapps.io:443
2025-11-13T16:13:51.0478101Z stdout F 2025-11-13T16:13:51.047Z [ConfigProxy] debug: PROXY WEB /user/admin/ to https://.wittysea-.swedencentral.azurecontainerapps.io:443
2025-11-13T16:13:51.0598273Z stdout F 2025-11-13T16:13:51.059Z [ConfigProxy] debug: PROXY WEB /user/admin/ to https://.wittysea-.swedencentral.azurecontainerapps.io:443
2025-11-13T16:13:51.0764157Z stdout F 2025-11-13T16:13:51.076Z [ConfigProxy] debug: Not recording activity for status 431 on /user/admin
2025-11-13T16:13:51.0883755Z stdout F 2025-11-13T16:13:51.087Z [ConfigProxy] debug: Not recording activity for status 431 on /user/admin
2025-11-13T16:13:51.0895691Z stdout F 2025-11-13T16:13:51.089Z [ConfigProxy] debug: Not recording activity for status 431 on /user/admin
2025-11-13T16:13:51.0904702Z stdout F 2025-11-13T16:13:51.090Z [ConfigProxy] debug: Not recording activity for status 431 on /user/admin

So, since the request reaches JupyterHub’s proxy, one explanation would be that Azure’s reverse proxy (envoy) adds stuff to the headers, making it too large for ConfigurableHttpProxy.

What I think, I’ll try:

  • using Traefik as the proxy instead
  • try to hook into the proxy’s code to dump some information about the request.
6

Root cause

We finally solved the problem. The reason turned out to be that

  • we forwarded traffic from the proxy to the single server via the external URL.
  • The chp, as currently configured by default, does not set the http Host header of the forwarded request to the host name configured in the proxy-rule

As a result of that, the request got in a loop which terminated when the `X-Forwarded-For` header blew the header size limit.

Fix

An obvious “fix” would be not to forward to the single-server’s external URL. We will end up doing this anyway in this case.

Still, I think not setting the Host header in the proxy to the one given in the rule seems to be the “less expected” behaviour because it breaks the case where your single server lives in an environment relying on virtual host request dispatching (a web-server hosting multiple vhosts, server behind another, non-chp, proxy.

The fix would be very simple. It works when adding options.changeOrigin = trueto the ConfigurableProxy’s constructor.

Probably, that should be configurable for backwards compatibility. I’d by happy to supply a little PR if you think that makes sense.