We have deployed JupyterHub with CHP on bare-metal on our HPC platform. Currently we are on jupyterhub==3.1.1 and configurable-http-proxy==4.5.3. We have noticed that the memory consumption of CHP increases steadily with time. Here is a snapshot of the total memory used on the machine where JupyterHub is deployed.
That was memory usage for about 30 days. The huge drop in the memory usage is when I restarted the CHP process which confirms that is is indeed CHP that is consuming all this memory. For a month usage of JupyterHub, CHP is consuming around 6 GB of memory. We have steady increase of users as well during this period, but the total number of users are still less than 250.
My intuition is that this memory is being used to store routing table. 6 GB seems a bit abnormal for just 250 users. Can someone comment on that? Here is a relevant issue but havent concluded anything over there.
We havent noticed any big issues with performance for the moment but I am not quite sure if it is a sustainable option going forward. Any comments or insights will be appreciated.
The GitHub issue you’ve already linked to describes what we already know (or rather don’t know!). If you’re able to track down or debug the resource leak that’d be very helpful!
@manics Cheers for the comment. Ok, I will try to look into it. I was not sure if it is still an open question or not.
I thought about traefik but I remember reading somewhere that traefik proxy does not support jupyterhub-idle-culler because it does not track activity. Is it still the case? Idle culler is quite important for our deployment as users leave the servers running most of the time.
Culling the servers from the JupyterLab side is not a very reliable option neither as ServerApp tends to kill itself without reaping the child processes, which can be from jupyter-server-proxy in our case.
User activity is sent directly from the singleuser-server to the hub. CHP has additional support for tracking network activity which isn’t supported by Traefik proxy, but this shouldn’t be required unless you’ve got a very non-standard deployment:
