How to set ephemeral storage limits and requests for hub and user pods in config to avoid evictions?

Since last few days, I have come across a pattern where all the Jhub related pods are being evicted with the similar reason like below
The node was low on resource: ephemeral-storage. Container hub was using 19192Ki, which exceeds its request of 0.

I am assuming that this is due to not setting up resource quota in pods as the size mentioned in above message is very small (I know similar issue might occur be due to old images not being cleaned after use.)

Every node in our cluster has around 20Gi of storage attached. So resolving this problem should be as simple as setting up ephemeral storage but I can not find documentation to do that. Is it something we allow in config.yaml ? Thanks.

I think (though I’m not 100% sure) this is due to a lack of storage on your Kubernetes hosts. 20GB for a VM seems quite low considering this includes the operating system, temporary files, logs, and Kubernetes containers. For example it’s not unusual for container images to approach 10GB in size.

Setting up user storage may help if users are creating large files since they would be stored on external storage instead of inside the container:
https://zero-to-jupyterhub.readthedocs.io/en/latest/customizing/user-storage.html

Thanks for reply @manics. I am using external storage (EFS) where I am storing data from notebook server. So the things which are consuming ephemeral storage must be docker images, logs etc. Can I move that to external storage too?

Short answer: no, EFS users NFS which isn’t suitable for Docker.

Longer answer: maybe, if you provision a block storage device. I’m not familiar with AWS so I don’t know how easy that is to setup on a Kubernetes node, or whether it’s cost effective compared to provisioning a VM with bigger ephemeral storage.

Thanks for the help @manics. I was able to solve it with combination of two storage classes block storage and file storage. :slight_smile:

1 Like