Number of open files not lowering with nodejs

JupyterHub
1

Hello,

We are running a Jupyterhub service on a VM with around 150 users daily. We encountered some irregular instabilities during the day since a while ago. At first, we thought it was a problem with a user doing too much activity at once and making the service unusable for the other users.
During our investigation, we regularly found problem linked to the number of open files by a process hitting a limit, which resulted in errors in the log like ENOTFOUND, EMFILE, EBUSY. We tried to up the limit in the VM and it worked for some time but it didn’t fix the problem.
When we tried to monitor the number of openfiles for the different Jupyterhub processes (python, nodejs, nginx), we realized that nodejs sometimes does not close all of its open files during the night, when there’s almost no user online. Usually, it does close most of its files and keep around 40+ opened, but when it doesn’t, the number just add up until we reach a limit (around 2100+) and the service breaks. We have to restart it to fix it and the number of open files goes back to normal.

For now, we have a script that is launched during the night and it restart the Jupyterhub service if the number of open files by nodejs is too high but it’s not really a fix.

2

Can you check whether the nodejs process involved is part of configurable-http-proxy, or if it’s another component?

If it’s configurable-http-proxy this might be due to too many open sockets:

Unfortunately we don’t have a way to reliably reproduce the bug, other than running JupyterHub with a lot of users. If you’re able to pin down exactly what actions on the user-side lead to an increase in unclosed ports that would be very helpful- please add the information to the above issue.

3

Yes, i can confirm the nodejs process is part of configurable-http-proxy.

We tried to check with several users who seemed to do a lot of operations but nothing came up so far. If you need more informations, don’t hesitate to ask.