With very large notebook docker images, I get DiskPressure, then later out of space

JupyterHub Zero to JupyterHub on Kubernetes
1

I’m deploying jupyterhub to k8s with the jupyterhub/jupyterhub helm. In the config file, i have:

singleuser:
  image:
    name: s22s/myspecial-notebook
    tag: latest
    pullPolicy: Always

this image was overly large (~17GB) because there were a lot of files left by installing various things that hadn’t been cleaned up properly. I’ve since fixed those, but I’d like to know how to fix the issue I was seeing with these large images.

Basically, I think the hook-image-puller container has a pretty small volume on it (maybe 16GB?), such that I got DiskPressure warnings when pulling that image, and then got out of space errors when trying to deploy with a scipy image. I looked, but couldn’t find a documented way to configure how much space the hook-image-puller has. Is there one?

2

Thanks for posting here, @philvarner!

DiskPressure condition is a node condition (see https://kubernetes.io/docs/concepts/architecture/nodes/#condition) rather than something about a specific pod. This usually means the disks used by your nodes are too small. This is dependent on how your cluster was created - usually there is an option to specify a larger node size.

Hope this is helpful!

1 Like
3

Ah, thanks very much for the reply! I see now that this is an issue with eksctl https://github.com/weaveworks/eksctl/issues/780 – the default volume size is 20GB, which is way to small for the jupyterhub use case. I think I’d increased it to 50GB, which was still too small. The recommendation is at least 100GB (–node-volume-size 100).

1 Like