# Prevent pod to get GPUs

**URL:** <https://discourse.jupyter.org/t/prevent-pod-to-get-gpus/24595>\
**Category:** Zero to JupyterHub on Kubernetes\
**Created:** [March 18, 2024, 9:45am UTC](https://discourse.jupyter.org/t/prevent-pod-to-get-gpus/24595 "2024-03-18T09:45:12Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![MG123](https://avatars.discourse-cdn.com/v4/letter/m/b77776/32.png) [@MG123](https://discourse.jupyter.org/u/MG123)\
**Post date:** [March 18, 2024, 9:45am UTC](https://discourse.jupyter.org/t/prevent-pod-to-get-gpus/24595/1 "2024-03-18T09:45:12Z")

</div>

Hi!  
Deploying JupyterHub on a NVIDIA GPU cluster here.  
Is there a way to prevent pods to access GPUs?  
Setting the “extra\_resource\_limits:” to 1 correctly allocate a single GPU replica to the pod, but setting to “0” or leaving it empty would allow the pod to access _all_ GPUs in the system.  
Is there something that I am missing on how to prevent pods to get GPUs?

The use case is to allow to run a “GPU-less” container if there is no replica available, running only on CPU.

---

<div class="post-metadata">

**Author:** ![MG123](https://avatars.discourse-cdn.com/v4/letter/m/b77776/32.png) [@MG123](https://discourse.jupyter.org/u/MG123)\
**Post date:** [March 19, 2024, 9:03am UTC](https://discourse.jupyter.org/t/prevent-pod-to-get-gpus/24595/2 "2024-03-19T09:03:38Z")

</div>

Self replying - and leaving it there for anyone in the same boat:  
this was a wrong configuration in the nvidia container runtime.  
/etc/nvidia-container-runtime/config.toml should have the following lines:

```auto
accept-nvidia-visible-devices-as-volume-mounts = true
accept-nvidia-visible-devices-envvar-when-unprivileged = false

```

And then the nvidia plugin should be deployed with

```auto
    compatWithCPUManager: true
    deviceListStrategy: volume-mounts

```

This (apparently) prevent containers to get all GPUs if no allocation specified, and correctly only 1 GPU when assigned with limits.

This is “clearly” documented here [[External] Read list of GPU devices from volume mounts instead of NVIDIA\_VISIBLE\_DEVICES - Google Docs](https://docs.google.com/document/d/1uXVF-NWZQXgP1MLb87_kMkQvidpnkNWicdpO2l9g-fw/edit)