Jupyterhub for Kubernes: How to avoid Image Prepull on specific nodes

General
1

Hello everyone,

I have a custom Kubernetes cluster with several nodes, each node has assigned some tags to differenciate between them with a Kubernetes label selector (nodeSelector), being one of them PU = “gpu” || “cpu”. I have also deployed a Jupyterhub environment on my cluster. However I am having trouble with the deployments of different profiles (singleUser.profileList) on different nodes.

More specifically, I would like to have a profile assigned to nodes with PU: “cpu” and other profile assigned to nodes with PU: “gpu”, in order to take advantage of the GPU when deploying notebooks. In the “cpu” profile I have to select the intel version of the docker image, whereas in the “gpu” profile I need a specific ARM (+ NVIDIA) docker image since the node is deployed on an arm-based machine. I believe I have configured properly each profile so that the user server is deployed in the proper nodes, with the nodeSelector tag (see below).

The problem comes with the Jupyterhub image prepuller. I have seen that it tries to pull all the images of every profile in every node, even though the nodeSelector tag is configured. Then, in the arm/gpu node it tries to prepull the image of the intel/cpu node , and on this node it tries to pull the arm/gpu image, ending up in an unsuccessfull installation and jupyterhub not being deployed on the cluster.

Is this a bug or a misconfiguration?

My configuration:

...

singleuser:
  nodeSelector:
    ARCH: "intel"
    LAYER: "cloud"
  defaultUrl: "/lab"
  image:
    name: easierai/jupyterhub-core
    tag: latest
  profileList:
    - display_name: "environment"
      description: "Environment preconfigured ."
      default: true
      nodeSelector:
        PU: "cpu"
    - display_name: "Minimal Environment"
      description: "Only default libraries installed."
      kubespawner_override:
        image: jupyter/minimal-notebook
        nodeSelector:
          PU: "cpu"
    - display_name: "one NVIDIA"
      description: "ARM, GPU"
      kubespawner_override:
        image: step21/jupyter-minimal-notebook:arm64
        nodeSelector:
          PU: "gpu"
        extraResource:
          limits:
            nvidia.com/gpu: 1
...

See issue: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1739