Handling failed gitpuller lifecycleHooks

JupyterHub Zero to JupyterHub on Kubernetes
1

We are using gitpuller to distribute course materials in a z2jh setup (example here). Occasionally, gitpuller fails - either because the pod storage is full, or because the local repo cannot be automatically merged by gitpuller. When gitpuller fails, the pod doesn’t start, which means we can’t access the volume in order to clean things up. This leaves unhappy students.

We tried substituting a python wrapper for the gitpuller command in the postStart hook, allowing us to catch and handle the exception. Now, the pod starts even if gitpuller fails, but the stdout / stderr from python doesn’t get logged, so the error is silent. (There seems to be some kubernetes chatter about logging, which is currently limited to only failure events only - see k8s issue).

Anyone have advice for handling gitpuller errors when used as a postStart hook? We would like the students to still be able to access their pods even if gitpuller fails, but also want to know when it fails so that we can fix it.

2 Likes
2

Excellent summary!

It is very problematic to debug lifecycleHooks because like you said, they only emit information in Kubernetes events when they fail (kubectl get events or kubectl describe pod ... and check the last section).

Quickfix

This is a workaround of your configuration to avoid crashing I think. It will silently fail now though.

    lifecycleHooks:
      postStart:
        exec:
          command:
            - "sh"
            - "-c"
            - >
-             gitpuller https://github.com/earthlab-education/ea-bootcamp-fall-2020 master ea-bootcamp-shared;
+             gitpuller https://github.com/earthlab-education/ea-bootcamp-fall-2020 master ea-bootcamp-shared || true;

Example from neurohackademy

Below is links to help you understand what I ended up doing. I didn’t find any great solution to this issue, but I used tricks to ensure it didn’t fail. In other words, I accepted that there will be no logging of the errors and used tricks to force it to never give an error.

Off topic: On how such shell script was mounted

For completeness, this is what I did to inject the script without building it into the image. Note that this trick requires that you use a Helm chart that depends on the JupyterHub helm chart so that you can install additional Kubernetes resources.

3

Thanks, @consideRatio! Should have mentioned that I did try the workaround, but if gitpuller exits with error code 1, the hook fails immediately and you never get to the second part of the statement (and therefore, the pod does not start).

I’ll continue on with my python wrapper, but definitely going to try your approach to mounting the script to avoid including it in the docker image. I might have python write a log file with any errors so we can at least check that manually if the students have problems with the git repo down the road.

Helpful to know that others are having the same issue and that we aren’t doing something unusual.

1 Like
4

Hi,
I am having a somewhat related issue. gitpuller works well for the other images notably the minimal, tensorflow and pyspark but fails with the jupyter/data-science image. In fact gitpuller seems not to recognize the git branch directory as shown:

2021-06-11T13:14:59Z [Warning]
 Exec lifecycle hook ([gitpuller https://github.com/jtande/jhub-file-download SparkNotebook spark])  
 for Container "notebook" in Pod "jupyter-jfossota_uncg-jhub(f492b0aa-7120-4c28-b10e-7e633d981cc0)"  
 failed - error: command 'gitpuller https://github.com/jtande/jhub-file-download SparkNotebook spark' exited with 2:  
 usage: gitpuller [-h] git_url [repo_dir] gitpuller: error: unrecognized arguments: spark , message:   
"usage: gitpuller [-h] git_url [repo_dir]\ngitpuller: error: unrecognized arguments: spark\n"

Below is the portion of my config.yaml file where gitpuller is setup:

lifecycleHooks:
    postStart:
      exec:
        command:
          [
            "gitpuller",
            "https://github.com/jtande/jhub-file-download",
            "SparkNotebook",
            "spark",
          ]
Any help with this is welcome.
5

@LeBoss What version of nbgitpuller are you using? There’s a bug in the latest release: Execution of gitpuller in shell throws KeyError: 'branch' version tag 0.10.0 · Issue #184 · jupyterhub/nbgitpuller · GitHub

6

I am using the latest release. Yes, I did believe it was a bug. Any work around?

7

If you’re building the Docker image yourself you can install an older version of nbgitpuller, the bug was introduced in the last release.

8

I am building the image. yes, I will go back and get an older version. I found it odd that this is happening just for jupyter/datascience-notebook. The other images are working just fine