Apache → configurable-http-proxy "Broken pipe"

JupyterHub
1

Hi !

I’m seeing errors mentioning a broken pipe in apache log.
The http apache server is configured as a reverse proxy (I followed jupyterhub documentation).

Here are some of the messages I got in the logs:

[proxy:error] [pid 763:tid 217152] (32)Broken pipe: [client .<REMOTE_IP>:51069] AH01084: pass request body failed to 127.0.0.1:8000 (127.0.0.1), referer: https://example.org/user/<USER>/notebooks/<SERVICE>/<SUB_DIR>/<NOTEBOOK>.ipynb
[proxy_http:error] [pid 763:tid 217152] [client <REMOTE_IP>:51069] AH01097: pass request body failed to 127.0.0.1:8000 (127.0.0.1) from <REMOTE_IP> (), referer: https://example.org/user/<USER>/notebooks/<SERVICE>/<SUB_DIR>/<NOTEBOOK>.ipynb

On the client side I had feedbacks from users telling me the notebook “fails to autosave”.
They work remotely, then I never had a chance to witness myself the error and gather more details. I never managed to reproduce the issue myself, both “autosaving” failure or triggering the “broken pipe” in the log. I don’t even know if they are related. I just noticed that some users ID in the error log have problems with their notebook checkpoints though (the “autosaving” issue).

After some research about the actual proxy error [0], my understanding is that apache fails to deliver the HTTP payload to configurable-http-proxy (aka CHP). I tried to setup the proxy-sendchunked mod_proxy config option mentioned in the post [0], but I still have the broken pipe.

  • Any hints on how I can go forward with this broken pipe issue?
  • Should I open an issue in the CHP tracker?
  • What kind of load CHP can handle? I usually have ~100 users on the same server, I guess it’s ok.

[0] Proxy issues with Apache | ionel's codelog

Thanks

2

JupyterHub and CHP can scale to a large number of users, but that’s obviously dependent on your available resources. For a very large number of users or for users with significant compute requirements you’ll want a multi-server setup such as https://zero-to-jupyterhub.readthedocs.io

  • Can you show us your JupyterHub config with secrets redacted
  • Do you only see these issues when you have a lot of users simultaneously using JupyterHub?
  • Can you show us your logs for JupyterHub (and CHP if it’s running separately) at the time the failure occurs? It might be helpful to turn on debug logging too.
  • Have you looked at your CPU, memory and network usage around the time the errors occur?
  • What sort of compute loads are your users putting on your server?
3

Thanks for your reply @manics :slight_smile:

Here is the config I use on a Debian Buster (up to date) with jupyterhub 1.2.1 and CHP 4.2.2.

No, I can see the “Broken pipe” issue even when there are only a couple of users.

I run users servers in dedicated slice with limited CPU/memory, I never noticed the user suffering from a lack of resources at the moment of the issue (I’ll control that again to be sure). Though when the error occurs the request did not hit the server yet, if I understand correctly, the request is still between the http server and CHP.

The platform is used for python training (for beginners), then exercises are plain python code but in the hand of neophyte programmers (we often have cpu intensive endless loops and some OOM kill however seldom). Resources guardrail (within systemd cgroups) works fine and does not seem related/correlated to the “Broken pipe” issue though.

I’ll try to reproduce with debug on.