Insufficient gpu problem!

JupyterHub Zero to JupyterHub on Kubernetes
1

There are 1 master and 4 nodes server in my k8s cluster, and I set up 4 gpus for each node server. The information such below:

NAME     GPUs
master   <none>
node1    1
node2    1
node3    1
node4    1

My problem is: Each of my jupyter users applies for 1 GPU, but when the fifth user creates a jupyter notebook, the error “0/5 nodes are available: 5 Insufficient nvidia.com/gpu” appears. Can not exist two users on a node? My jupyterhub config such below:

proxy:
  secretToken: "XXXX"
  service:
    type: NodePort
    nodePorts:
      http: 30080
      https: 30443
hub:
  extraConfig: |
    c.JupyterHub.spawner_class = 'kubespawner.KubeSpawner'
singleuser:
  image: 
    name: jupyter/minimal-notebook
    tag: 5197709e9f23
  profileList:
    - display_name: "Default: Minimal environment(Without GPU)"
      description: "Default"
      default: true
    - display_name: "Datascience environment(With GPU)"
      description: "Nvidia Quadro P5000"
      kubespawner_override:
        image: ihades/cuda-notebook:0.1
        extra_resource_limits:
          nvidia.com/gpu: "1"
  storage:
    dynamic:
      storageClass: nfs-client 
    capacity: 20Gi
    extraVolumes:
      - name: shm-volume
        emptyDir:
          medium: Memory
    extraVolumeMounts:
      - name: shm-volume
        mountPath: /dev/shm
  memory:
    limit: 32G
    guarantee: 2G
  cpu:
    limit: .5
    guarantee: .1
  extraEnv:
    EDITOR: "vim"
    GRANT_SUDO: "yes"
  uid: 0
  cmd: null
2

I may be a bit late, but for others that may find your post: The part of your configuration I am quoting above does exactly that. It limits 1 GPU per user. If you remove that limitation you would likely get more than one user for each GPU, however that comes with its own set of problems. In any case, please let us know how did you fare since your post.

1 Like