Hoi,
We’re running a bunch of JupyterHub Docker containers on VMS and each of those run a lot of labs. Without any user activity the machine is under heavy load. It appears that most of the CPU cycles go towards:
python3 /etc/jupyter/docker_healthcheck.py
Is there a way to disable this, assuming that is safe to do. Or if this is needed, is there a way to change the interval when this gets called?
Thanks!
Presumably you’re using the jupyter/docker-stacks images?
If so the healthcheck is defined in
and inherited by all child images. You could disable it by building your own image with the addition of HEALTHCHECK NONE
However I’m surprised it’s taking up so much CPU. How much CPU/memory is each healthcheck consuming, and is it easily reproducible?
I think we’ve figured out what happened. We wrote an LTI that can create and boot and manage Jupyterhubs on a VM. Currently we’re running a course with 4 VMs and each one has about 10 hubs running, with 11 students each.
As it turns out, in our hub config file we had:
c.DockerSpawner.debug = True
This resulted in our /var/syslog being spammed with tons and tons of output from the healthcheck Python script. So much so that we ran out of disk space within about 2 weeks.
From what we can now tell the high CPU load (roughly 6% per health check hit) was a cascading effect due to debugging being turned on. Even more so than CPU load the systems themselves were under heavy load overall due to the drives being accessed, etc.
We’re testing now with the debug flag set to False to see how that works out.