Hi, is there any way to limit the memory usage in Kubernetes JH deployment?
When the limit is passed the pod dies (OOM); is there any way to limit the memory usage, without the pod being killed?
What kind of behaviour are you expecting if a user tries to claim more RAM than available?
I don’t think it’s technically possible to realize it in general.
2 Likes
Maybe use Pod Vertical Scaling to increase memory limits at runtime to avoid killing the pod?
1 Like
Do you think of user pods, or some other pod? Below I am assuming you think of user pods:
There are three kinds of memory kills worth distinguishing here:
1: a containers non-init process is killed, like a jupyter kernel
2: a containers init process is killed, the container will restart (k8s config dependent, but by default)
3: the pod is evicted
You can avoid the third situation by guaranteeing the pod’s container memory equal to the memory limit.
You could avoid the second by constraining individual kernels memory somehow perhaps, if jupyter kernels are the typical sub-process that eat up memory. I’m not 100% confident this can be done with existing software, but i think maybe it can.
1 Like