lifecycleHooks not launching on Jupyterhub kubernetes

JupyterHub
1

When defining a postStart lifecycle hook for singleuser pods in the Helm chart values, placing the hook inside the singleuser block causes it to be silently ignored. Even though the value is syntactically valid YAML.

singleuser:
      lifecycleHooks:
        postStart:
          exec:
            command:
              - "sh"
              - "-c"
              - >
                git config --global credential.helper store;

My goal is to pull gitlab repository through the start of my notebooks and I’m using Keycloak Auth

I’m using the bitnami charts with an app_version “4.0.2”

2

I’m using the bitnami charts with an app_version “4.0.2”

Is the bitnami charts respecting config structured like that? That seems like the official JupyterHub charts config structure – maybe you are using an example from that chart instead of bitnami’s jupyterhub chart?

1 Like
3

Thank you for your response.

Yes, But I checked on the bitnami values conf.
They using the same file structure

{{- if .Values.singleuser.lifecycleHooks }}
      lifecycleHooks: {{- include "common.tplvalues.render" ( dict "value" .Values.singleuser.lifecycleHooks "context" $ ) | nindent 4 }}
      {{- end }}

charts/bitnami/jupyterhub/values.yaml at main · bitnami/charts

4

Can you try reproducing this with the Z2JH Helm chart
https://z2jh.jupyter.org/en/stable/jupyterhub/installation.html#install-jupyterhub

If it’s reproducible please share your full configuration, and the output of kubectl get pod <singleuser-pod> -o yaml

2 Likes
5

Lifecycle hooks for “singleuser” should result in configuration of KubeSpawner. What is rendered by helm templates must therefore be considered via a jupyterhub config file, in order to configure kubespawner’s config on how to construct the pod to have a lifecycle hook.

Maybe the bitnami chart fails to do that part?

6

I checked on the jupyterhub_config.py and I found we are laso using the KubeSpawner

It’s the same conf between the bitnami & the official chart. c.JupyterHub.spawner_class = "kubespawner.KubeSpawner"

So If I understood the lifecyclehooks is mapped to c.KubeSpawner.lifecycle_hooks

7

If you still need help please see my earlier comment about reproducing this with the official Z2JH Helm chart, and giving us the pod YAML.

9

I am currently facing this issue as well, no matter whether it is postStart or preStop, even if i have a simple file creation or echo command it is just ignored. Any idea where I can start troubleshooting? I also tried adding it as an extraConfig in my hub settings as I have already created pre_spawn and post_stop hooks. So I tried adding lifecycle hooks using c.KubeSpawner.lifecycle_hooks but its ignored.

lifecycleHooks:
postStart:
exec:
command:
- “sh”
- “-c”
- >
cd /home/jovyan/ \
touch main.py \
mkdir testdir \
mv main.py testdir/\

10

The following configuration works for me.

hub:
  config:
    KubeSpawner:
      lifecycle_hooks:
        postStart:
          exec:
            command:
            - "/bin/sh"
            - "-c"
            - >
              touch /home/jovyan/main.py;
              mkdir -p /home/jovyan/testdir;
              mv /home/jovyan/main.py /home/jovyan/testdir

I assume it’s due to the spelling of lifecycle_hooks since the keys must match the Python trailtlets described in KubeSpawner — Kubespawner.
In addition, changing the directory via cd did not work for me, so I provided the full paths instead. Also, providing a multiline command with \ failed. So, I appended the lines with ;.

2 Likes
11

Note that echo etc won’t be seen, because the postStart hooks are starting a separate process in the container that isnt being logged to stdout etc. The only output you can get is by writing files to disk etc or having it fail - then you get an error k8s Event about it I recall.

Debugging lifecycle hooks are terrible, but make sure to be mindful of:

  1. No feedback unless it fails, then its part of k8s Events or via kubectl describe pod i think
  2. It runs after the container starts, so it can be too slow to create config etc to be read on startup by the containers main process.
  3. Failure of this hook can cause pod startup failures - if i recall correctly, so it should be made very reliable
1 Like