1.) The environment
- Ubuntu 18.0.4. headless server
- Nvidia driver 430
- Miniconda installed on a per user base (separately under the /home//miniconda for all
- Cuda 10.1
- jupyterhub 1.0.0
- 4 Tesla P100 GPUs installed on the server
- docker version 19.03.2
2.) The Question
How can we let jupyterhub spawn (systemuser-spawner) docker containers with arbitrary GPUs ?
At this time I can spawn a container with all or specific defined GPUs by the keyword in the Dockerfile:
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_VISIBLE_DEVICES 1, 2
What we want is something else:
- At spawn menu, one should be able to choose a container with or without GPU.
- If a container with GPU is chosen, the number of GPUs should be defined.
- We have 4 GPUs, and want to provide them as first come - first serve.
- So an unser can pick 4, 3, 2, or 1 GPU if he is the first.
- The next can pick whatever is left, all or some.
How can I create a docker file that is spawnable that receives any nunber of GPUs that are left in my set?
thanks and regards,