We are using gitpuller to distribute course materials in a z2jh setup (example here). Occasionally, gitpuller fails - either because the pod storage is full, or because the local repo cannot be automatically merged by gitpuller. When gitpuller fails, the pod doesn’t start, which means we can’t access the volume in order to clean things up. This leaves unhappy students.
We tried substituting a python wrapper for the gitpuller command in the postStart hook, allowing us to catch and handle the exception. Now, the pod starts even if gitpuller fails, but the stdout / stderr from python doesn’t get logged, so the error is silent. (There seems to be some kubernetes chatter about logging, which is currently limited to only failure events only - see k8s issue).
Anyone have advice for handling gitpuller errors when used as a postStart hook? We would like the students to still be able to access their pods even if gitpuller fails, but also want to know when it fails so that we can fix it.