How to set ephemeral storage limits and requests for hub and user pods in config to avoid evictions?

JupyterHub Zero to JupyterHub on Kubernetes
1

Since last few days, I have come across a pattern where all the Jhub related pods are being evicted with the similar reason like below
The node was low on resource: ephemeral-storage. Container hub was using 19192Ki, which exceeds its request of 0.

I am assuming that this is due to not setting up resource quota in pods as the size mentioned in above message is very small (I know similar issue might occur be due to old images not being cleaned after use.)

Every node in our cluster has around 20Gi of storage attached. So resolving this problem should be as simple as setting up ephemeral storage but I can not find documentation to do that. Is it something we allow in config.yaml ? Thanks.

2

I think (though I’m not 100% sure) this is due to a lack of storage on your Kubernetes hosts. 20GB for a VM seems quite low considering this includes the operating system, temporary files, logs, and Kubernetes containers. For example it’s not unusual for container images to approach 10GB in size.

Setting up user storage may help if users are creating large files since they would be stored on external storage instead of inside the container:
https://zero-to-jupyterhub.readthedocs.io/en/latest/customizing/user-storage.html

3

Thanks for reply @manics. I am using external storage (EFS) where I am storing data from notebook server. So the things which are consuming ephemeral storage must be docker images, logs etc. Can I move that to external storage too?

4

Short answer: no, EFS users NFS which isn’t suitable for Docker.

Longer answer: maybe, if you provision a block storage device. I’m not familiar with AWS so I don’t know how easy that is to setup on a Kubernetes node, or whether it’s cost effective compared to provisioning a VM with bigger ephemeral storage.

5

Thanks for the help @manics. I was able to solve it with combination of two storage classes block storage and file storage. :slight_smile:

1 Like
6

Hey. Could you please help me?

I am in the same situation. Single user pod is getting evicted with following error
“Pod ephemeral local storage usage exceeds the total limit 1Gi”

Can you please tell how can I increase this ephermal storage or what is the alternative way to fix this issue?

7

Hey Abdul, I have mentioned the solution above. I would recommend to see what is filling up your ephemeral storage, make decision about what data you need from it. Remove it or move it to PV. I did the same, I added new storage class for NFS based system and moved alot of data from ephemeral storage to PV.

Other solution could be increasing ephemeral storage than 1Gi

8

Hey Bhushan, thank you for your reply.
You were right, it was using space in folder /opt/bitnami/miniconda, but /opt/bitnami/jupyterhub-singleuser was mounted to PVC.

Anyhow, I also found a way to set ephermal-storage by adding following in hub.configuration within values.yaml

extraResource:
  limits: 
    ephemeral-storage: '5Gi'
  guarantees:
    ephemeral-storage: '5Gi'

I had to study the code of kubernetes spawner and also kubernetes client library. Unfortunately, there was not documentation for it.

9

I am glad you were able to resolve it, Abdul. Their code is pretty much self explanatory. I had to write some custom code to override kubespawner. It was pretty straight forward.

1 Like