# Docker container (Jupyter Notebook) on JupyterHub not able to use GPU on host system

**URL:** <https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645>\
**Category:** JupyterHub\
**Created:** [March 20, 2024, 1:49am UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645 "2024-03-20T01:49:08Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ross](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ross/32/12645_2.png) [@Ross](https://discourse.jupyter.org/u/Ross)\
**Post date:** [March 20, 2024, 1:49am UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645/1 "2024-03-20T01:49:08Z")

</div>

Hi folks so I have a JupyterHub which spawns docker container. Now I want these run some ML workload which wants to use GPU from the host. I have installed nvidia container toolkit on host.  
Below is my config file.

```auto
import dockerspawner
import os
from jupyter_client.localinterfaces import public_ips
import subprocess

c = get_config() # noqa

c.Authenticator.admin_users = {'ross'}

c.DockerSpawner.allowed_images = {
   
    'Python TensorFlow Notebook':'quay.io/jupyter/tensorflow-notebook:latest',
    'Python Pytorch Notebook':'quay.io/jupyter/pytorch-notebook:latest'

}

# we need the hub to listen on all ips when it is in a container

c.JupyterHub.spawner_class = dockerspawner.DockerSpawner

# The docker instances need access to the Hub, so the default loopback port doesn't work:

c.JupyterHub.hub_ip = public_ips()[0]

c.JupyterHub.db_url = '/etc/jupyterhub_workspace/jupyterhub.sqlite'

c.JupyterHub.cookie_secret_file = '/etc/jupyterhub_workspace/jupyterhub_cookie_secret'

def create_dir_hook(spawner):
    username = spawner.user.name
    volume_path = os.path.join('/home/jupyter', username)
    print(volume_path)
    if not os.path.exists(volume_path):
        print("Path doesnt exist")
        subprocess.call(["/sbin/mkhomedir_helper", spawner.user.name])

c.Spawner.pre_spawn_hook = create_dir_hook

c.Spawner.default_url = '/lab'

notebook_dir = '/home/jovyan/work'

host_dir = '/home/jupyter/{username}/'
print(notebook_dir)
print(host_dir)
c.DockerSpawner.notebook_dir = notebook_dir

# Mount the real user's Docker volume on the host to the notebook user's
# notebook directory in the container
c.DockerSpawner.volumes = {
    f'{host_dir}': notebook_dir

    }

# delete containers when the stop
c.DockerSpawner.remove = True

c.DockerSpawner.extra_create_kwargs = {'user': 'root'}

c.DockerSpawner.extra_host_config = {'runtime': 'nvidia'}

c.DockerSpawner.environment = {
    'GRANT_SUDO': 'yes',
    'CHOWN_HOME': 'yes',
    'CHOWN_EXTRA': '/home/jovyan',
    'CHOWN_HOME_OPTS': '-R',
    'NB_UID': 1000,
    'NB_GID': 1000,
    }

```

I am able to spawn tensorflow and Pytorch notebooks. But those are not able to use GPU.

Tensorflow Notebook : GPU not detected

 ![Untitled (2)](https://canada1.discourse-cdn.com/flex031/uploads/jupyter/original/2X/2/2cb1e6fae84b201739adcb80f8e85ec0563f61ae.png)

* * *

Some configs from HOST Machine

docker daemon

```auto
cat /etc/docker/daemon.json
{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

```

```auto
> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

```

```auto
> nvidia-container-runtime --version
NVIDIA Container Runtime version 1.14.6
commit: 5605d191332dcfeea802c4497360d60a65c7887e
spec: 1.2.0

runc version 1.1.12
commit: v1.1.12-0-g51d5e94
spec: 1.0.2-dev
go: go1.21.8
libseccomp: 2.5.1

```

Can someone please help me out with the same ?

---

<div class="post-metadata">

**Author:** ![Ross](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ross/32/12645_2.png) [@Ross](https://discourse.jupyter.org/u/Ross)\
**Post date:** [March 20, 2024, 1:50am UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645/2 "2024-03-20T01:50:07Z")

</div>

nvidia-smi

 ![Untitled (4)](https://canada1.discourse-cdn.com/flex031/uploads/jupyter/original/2X/8/89a7c5a76749f8ba83ccfaf6699ee77326b457c9.png)

---

<div class="post-metadata">

**Author:** ![Ross](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ross/32/12645_2.png) [@Ross](https://discourse.jupyter.org/u/Ross)\
**Post date:** [March 21, 2024, 10:07pm UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645/3 "2024-03-21T22:07:09Z")

</div>

@mahendrapaipuri @manics any suggestions?

---

<div class="post-metadata">

**Author:** ![consideRatio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/consideratio/32/1033_2.png) [@consideRatio](https://discourse.jupyter.org/u/consideRatio)\
**Post date:** [March 21, 2024, 10:27pm UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645/4 "2024-03-21T22:27:16Z")

</div>

I’m on mobile reading this quickly, but search for “capabilities” and “nvidia” in this forum and i think you may find relevant posts.

Edit: [SwarmSpawner spawns a GPU-enabled image but - #2 by consideRatio](https://discourse.jupyter.org/t/swarmspawner-spawns-a-gpu-enabled-image-but/24115/2)

---

<div class="post-metadata">

**Author:** ![markperri](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/markperri/32/2187_2.png) [@markperri](https://discourse.jupyter.org/u/markperri)\
**Post date:** [March 26, 2024, 5:11pm UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645/5 "2024-03-26T17:11:33Z")

</div>

It looks like the cuda versions of those images have cuda in the tag:

`quay.io/jupyter/tensorflow-notebook:cuda-latest`

---

<div class="post-metadata">

**Author:** ![Ross](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ross/32/12645_2.png) [@Ross](https://discourse.jupyter.org/u/Ross)\
**Post date:** [April 2, 2024, 11:06pm UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645/6 "2024-04-02T23:06:13Z")

</div>

Thank you very much @markperri . That somehow worked. But , I dont see cuda-latest tag in all images. Like for example in `quay.io/jupyter/pytorch-notebook:latest` I don’t see any cuda tag. Also Is there any template for creating custom cuda enabled jupyter notebooks ?

cc: @consideRatio @markperri

---

<div class="post-metadata">

**Author:** ![consideRatio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/consideratio/32/1033_2.png) [@consideRatio](https://discourse.jupyter.org/u/consideRatio)\
**Post date:** [April 3, 2024, 2:41pm UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645/7 "2024-04-03T14:41:37Z")

</div>

tensorflow notebook has it, abd pytorch has cuda tags also but versioned to specific cuda major versions. Look among available tags and you’ll see cuda11 cuda12 something i believe.

Only those images has cuda stuff, not base-notebook for example.

---

<div class="post-metadata">

**Author:** ![markperri](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/markperri/32/2187_2.png) [@markperri](https://discourse.jupyter.org/u/markperri)\
**Post date:** [April 3, 2024, 8:27pm UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645/8 "2024-04-03T20:27:03Z")

</div>

You can see pytorch here: [Quay](https://quay.io/repository/jupyter/pytorch-notebook?tab=tags)

Easiest way to create your own notebook image would be to copy all the Nvidia build instructions from one of those existing images.

Thanks,  
Mark

---

<div class="post-metadata">

**Author:** ![Ross](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ross/32/12645_2.png) [@Ross](https://discourse.jupyter.org/u/Ross)\
**Post date:** [April 4, 2024, 3:18am UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645/9 "2024-04-04T03:18:23Z")

</div>

Thank you very much @markperri

> copy all the Nvidia build instructions from one of those existing images.

How can i get this ? Can we get Dockerfile for those images on Quay ?

---

<div class="post-metadata">

**Author:** ![mahendrapaipuri](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/mahendrapaipuri/32/6800_2.png) [@mahendrapaipuri](https://discourse.jupyter.org/u/mahendrapaipuri)\
**Post date:** [April 4, 2024, 11:30am UTC](https://discourse.jupyter.org/t/docker-container-jupyter-notebook-on-jupyterhub-not-able-to-use-gpu-on-host-system/24645/10 "2024-04-04T11:30:12Z")

</div>

Check [docker-stacks](https://github.com/jupyter/docker-stacks/tree/main/images) repo. Here you will find all necesssary metadata to build container images