Hello everyone,
I have a custom Kubernetes cluster with several nodes, each node has assigned some tags to differenciate between them with a Kubernetes label selector (nodeSelector), being one of them PU = “gpu” || “cpu”. I have also deployed a Jupyterhub environment on my cluster. However I am having trouble with the deployments of different profiles (singleUser.profileList) on different nodes.
More specifically, I would like to have a profile assigned to nodes with PU: “cpu” and other profile assigned to nodes with PU: “gpu”, in order to take advantage of the GPU when deploying notebooks. In the “cpu” profile I have to select the intel version of the docker image, whereas in the “gpu” profile I need a specific ARM (+ NVIDIA) docker image since the node is deployed on an arm-based machine. I believe I have configured properly each profile so that the user server is deployed in the proper nodes, with the nodeSelector tag (see below).
The problem comes with the Jupyterhub image prepuller. I have seen that it tries to pull all the images of every profile in every node, even though the nodeSelector tag is configured. Then, in the arm/gpu node it tries to prepull the image of the intel/cpu node , and on this node it tries to pull the arm/gpu image, ending up in an unsuccessfull installation and jupyterhub not being deployed on the cluster.
Is this a bug or a misconfiguration?
My configuration:
...
singleuser:
nodeSelector:
ARCH: "intel"
LAYER: "cloud"
defaultUrl: "/lab"
image:
name: easierai/jupyterhub-core
tag: latest
profileList:
- display_name: "environment"
description: "Environment preconfigured ."
default: true
nodeSelector:
PU: "cpu"
- display_name: "Minimal Environment"
description: "Only default libraries installed."
kubespawner_override:
image: jupyter/minimal-notebook
nodeSelector:
PU: "cpu"
- display_name: "one NVIDIA"
description: "ARM, GPU"
kubespawner_override:
image: step21/jupyter-minimal-notebook:arm64
nodeSelector:
PU: "gpu"
extraResource:
limits:
nvidia.com/gpu: 1
...
See issue: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1739