I have a custom Kubernetes cluster with several nodes, each node has assigned some tags to differenciate between them with a Kubernetes label selector (nodeSelector), being one of them PU = “gpu” || “cpu”. I have also deployed a Jupyterhub environment on my cluster. However I am having trouble with the deployments of different profiles (singleUser.profileList) on different nodes.
More specifically, I would like to have a profile assigned to nodes with PU: “cpu” and other profile assigned to nodes with PU: “gpu”, in order to take advantage of the GPU when deploying notebooks. In the “cpu” profile I have to select the intel version of the docker image, whereas in the “gpu” profile I need a specific ARM (+ NVIDIA) docker image since the node is deployed on an arm-based machine. I believe I have configured properly each profile so that the user server is deployed in the proper nodes, with the nodeSelector tag (see below).
The problem comes with the Jupyterhub image prepuller. I have seen that it tries to pull all the images of every profile in every node, even though the nodeSelector tag is configured. Then, in the arm/gpu node it tries to prepull the image of the intel/cpu node , and on this node it tries to pull the arm/gpu image, ending up in an unsuccessfull installation and jupyterhub not being deployed on the cluster.
Is this a bug or a misconfiguration?
... singleuser: nodeSelector: ARCH: "intel" LAYER: "cloud" defaultUrl: "/lab" image: name: easierai/jupyterhub-core tag: latest profileList: - display_name: "environment" description: "Environment preconfigured ." default: true nodeSelector: PU: "cpu" - display_name: "Minimal Environment" description: "Only default libraries installed." kubespawner_override: image: jupyter/minimal-notebook nodeSelector: PU: "cpu" - display_name: "one NVIDIA" description: "ARM, GPU" kubespawner_override: image: step21/jupyter-minimal-notebook:arm64 nodeSelector: PU: "gpu" extraResource: limits: nvidia.com/gpu: 1 ...