Using nohup at lifecycleHooks with GKE containerd node

JupyterHub Zero to JupyterHub on Kubernetes
1

I have some problem using nohup at lifecycleHooks.postStart.exec.command with containerd node image.

My env

  • GKE: 1.22.15-gke.100 [ using Container-Optimized OS(cos_containerd) ]
  • Jupyterhub: JupyterHub 3.1.0.dev
    I use custom image with zero-to-jupyterhub-k8s chart

Recently i upgrade my node image from docker node to containerd node.
When i use Docker node, It worked well. But It’s not work on containerd node
It’s pending more than 1200s and timeout.

I try many ways and Finally found the cause. That’s nohup!

# initialize.sh
function check_interval {
  SLEEP_INTERVAL_SECONDS=60
  while true; do
      echo "some Function work in this"
      sleep "${SLEEP_INTERVAL_SECONDS}"
  done
}

export -f check_interval
nohup bash -c check_interval &

I use intialize.sh file in lifecycleHooks to delete not use resource.

singleuser:
  lifecycleHooks:
    postStart:
      exec:
        command:
          - "/bin/bash"
          - "-c"
          - "/init.d/initialize.sh"

I tried in zero-to-jupyterhub-k8s 2.0. and it have same problem

2

I’m almost certain this is a problem unrelated to the Helm chart, and you would see the same issue with any k8s pod you create with a k8s lifecycle hook like this defined for it.

Related resources:

I think you should inspect the state of the started pod in k8s, specifically kubectl describe pod <started server pod name> as that includes Events in the end of the description. Those events can describe if there was an error in the lifecycle hook for example. Debugging these are very hard in general because they only emit an Event message like that if they fail.

I think if you want something to keep running, a postStart lifecycle hook that starts subprocesses may be the wrong way to go. I’m not sure, but I suspect it. If you want something to run on startup, you could perhaps also mount some script and call it before you start the user server with jupyterhub-singleuser or similar, or define a Linux systemd service or similar in the image or by mounting such things.

If you want to mount stuff, singleuser.extraFiles may be relevant.

Good luck!

1 Like
3

Oh Thanks your advice.

singleuser:
  cmd: "bash -c ../../init.d/initialize.sh & jupyterhub-singleuser --allow-root --SingleUserNotebookApp.default_url=/lab"

I’m using like this and works well!