Hi all,
is there a way to enforce memory limits on a standalone JupyterLab (i.e. one that was started directly from, e.g., the shell via jupyter lab ...
)?
Background of my question: I’m running a JupyerLab inside an HPC job on a multi-tenant node. The batch scheduler will kill my job if it consumes more memory than was requested. And I want to make sure JLab (and the kernels) don’t allocate more memory than they are allowed to. Another use case would be users directly starting their JupyterLab on a shared computer that they SSH into.
I know that if JLab was started from a Hub, I can set mem_limit
s on the spawner. But here, I don’t have a spawner.
Well, JupyterLab is more the UI than what happens in the background. I am not sure whether we can pass arguments to the Python Kernel but have you thought about using something like https://www.geeksforgeeks.org/python-how-to-put-limits-on-memory-and-cpu-usage/ ?
There is an open Jupyter Enhancement Proposal about parameterized kernels: https://github.com/jupyter/enhancement-proposals/pull/46. In the interim, I believe the only way to constrain resources is to launch a kernel in a container.
(or make a custom kernel spec that passes the arguments)
For example,
$ cat ~./jupyter/kernels/mypython/kernel.json
{
"argv": [
"<path-to-my-python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
<other args here>
],
"display_name": "Python 3 Constrained",
"language": "python"
}
@willirath you could “cage” JupyterLab and kernels making use of linux systemd
systemd-run creates and starts a detached execution environment on the fly:
sudo* systemd-run -t -p MemoryLimit=500M jupyter lab --ip=0.0.0.0 --port=8000 --notebook-dir=/whatever
*AFAIK you need root privileges to exec systemd-run command
In addition, cgroups could be used to limit memory usage on per user/group basis, e.g.:
# cat /etc/cgconfig.conf
group DEV {
memory {
memory.limit_in_bytes="2G";
memory.memsw.limit_in_bytes="3G";
}
}
group DS {
memory {
memory.limit_in_bytes="4G";
memory.memsw.limit_in_bytes="6G";
}
}
# cat /etc/cgrules.conf
user memory DS/
@group memory DEV/
I hope it helps
2 Likes
Try https://github.com/pshved/timeout maybe? I read through the linked blog post and it seems to do exactly what you want.