# JupyterHub, DockerSpawner, podman and GPU

**URL:** <https://discourse.jupyter.org/t/jupyterhub-dockerspawner-podman-and-gpu/26447>\
**Category:** JupyterHub\
**Tags:** jupyterhub, help-wanted\
**Created:** [June 20, 2024, 5:28pm UTC](https://discourse.jupyter.org/t/jupyterhub-dockerspawner-podman-and-gpu/26447 "2024-06-20T17:28:41Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![seb835](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/seb835/32/13797_2.png) [@seb835](https://discourse.jupyter.org/u/seb835)\
**Post date:** [June 20, 2024, 5:28pm UTC](https://discourse.jupyter.org/t/jupyterhub-dockerspawner-podman-and-gpu/26447/1 "2024-06-20T17:28:41Z")

</div>

Hi Community,

i am looking for advice,

i use jupyterhub and dockerspawner to run jupyterlab into redhat podman containers. It works very well without any trouble, thanks for the work guys !!

But now, i want to add GPU Supports, when using podman you have to add "–device [nvidia.com/gpu=all](http://nvidia.com/gpu=all) " to the podman command to allow container to use the GPU.

How can we add this in the dockerspawner parameters ? because i can’t get it works … the spawner complains about “device” as un unkwon parameters when added as .extra\_create\_kwargs or extra\_host\_config, that amke sens it is podman option not docker one.

Anyone here, get this kind of config running ?  
Thanks a Lot.  
Best

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [June 20, 2024, 10:41pm UTC](https://discourse.jupyter.org/t/jupyterhub-dockerspawner-podman-and-gpu/26447/2 "2024-06-20T22:41:47Z")

</div>

If you use the equivalent Docker config for GPUs does it work?

---

<div class="post-metadata">

**Author:** ![seb835](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/seb835/32/13797_2.png) [@seb835](https://discourse.jupyter.org/u/seb835)\
**Post date:** [June 21, 2024, 10:46am UTC](https://discourse.jupyter.org/t/jupyterhub-dockerspawner-podman-and-gpu/26447/3 "2024-06-21T10:46:43Z")

</div>

Can you elaborate on equivalent Docker config?

From my search and code analysis, when using docker as cri we usually used “device request host config docker api field” to sent gpu information with dockerspawner.  
But it looks like, when podman is configure though the docker api, it does not manage that field ☹

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [June 22, 2024, 7:51am UTC](https://discourse.jupyter.org/t/jupyterhub-dockerspawner-podman-and-gpu/26447/4 "2024-06-22T07:51:51Z")

</div>

It looks like `devices` is in the host config section of the Docker API:  
[https://docker-py.readthedocs.io/en/stable/api.html#docker.api.container.ContainerApiMixin.create\_host\_config](https://docker-py.readthedocs.io/en/stable/api.html#docker.api.container.ContainerApiMixin.create_host_config)  
[https://jupyterhub-dockerspawner.readthedocs.io/en/latest/api/index.html#dockerspawner.DockerSpawner.extra\_host\_config](https://jupyterhub-dockerspawner.readthedocs.io/en/latest/api/index.html#dockerspawner.DockerSpawner.extra_host_config)

---

<div class="post-metadata">

**Author:** ![seb835](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/seb835/32/13797_2.png) [@seb835](https://discourse.jupyter.org/u/seb835)\
**Post date:** [June 22, 2024, 1:51pm UTC](https://discourse.jupyter.org/t/jupyterhub-dockerspawner-podman-and-gpu/26447/5 "2024-06-22T13:51:59Z")

</div>

You are right, but podman does not use it ☹

> <https://github.com/containers/podman/issues/22645>
>
> \### Issue Description
> 
> "Device requests" are how GPUs are invoked from the Docke…r API. However, device requests are not being respected by Podman when creating a container over the Podman socket.
> 
> \### Steps to reproduce the issue
> 
> Here is a Python script which tests the Docker and Podman socket APIs.
> 
> Setup: install Python version 3.10+ and run \`pip install docker==7.0.0\`
> 
> Run these tests:
> 
> \`\`\`python
> import subprocess as sp
> import docker
> import docker.types
> 
> \# Setup: create unix socket clients
> \# --------------------------------------------------------------------------------
> 
> podman\_socket = sp.check\_output(\['podman', 'info', '--format', '{{ .Host.RemoteSocket.Path }}'\], text=True).strip()
> podman\_client = docker.DockerClient(base\_url=f'unix://{podman\_socket}')
> docker\_client = docker.DockerClient(base\_url=f'unix:///var/run/docker.sock')
> 
> \# Sanity checks: assert podman is working
> \# --------------------------------------------------------------------------------
> 
> assert b'!... Hello Podman World ...!' in podman\_client.containers.run('quay.io/podman/hello', auto\_remove=True)
> 
> \# Sanity checks: assert podman and docker both work with nvidia-container-toolkit
> \# --------------------------------------------------------------------------------
> 
> def test\_nvidia\_smi\_works\_using\_command(command: str):
> assert sp.check\_output(\[command, 'run', '--rm', '--gpus=all', 'registry.access.redhat.com/ubi9:9.4-947.1714667021', 'nvidia-smi', '-L'\]).startswith(b'GPU 0')
> 
> 
> test\_nvidia\_smi\_works\_using\_command('docker')
> test\_nvidia\_smi\_works\_using\_command('podman')
> 
> \# Bug reproduction cases
> \# --------------------------------------------------------------------------------
> 
> GPU\_REQUEST = {
> 'device\_requests': \[docker.types.DeviceRequest(count=1, capabilities=\[\['gpu'\]\]) \]
> }
> 
> def test\_nvidia\_smi\_works\_using\_client(client: docker.DockerClient):
> assert client.containers.run('registry.access.redhat.com/ubi9:9.4-947.1714667021', \['nvidia-smi', '-L'\], \*\*GPU\_REQUEST).startswith(b'GPU 0')
> 
> 
> test\_nvidia\_smi\_works\_using\_client(docker\_client) # pass
> test\_nvidia\_smi\_works\_using\_client(podman\_client) # fail
> 
> 
> def test\_device\_request\_goes\_through(client: docker.DockerClient):
> container = client.containers.run('registry.access.redhat.com/ubi9:9.4-947.1714667021', \['nvidia-smi', '-L'\], detach=True, \*\*GPU\_REQUEST)
> assert len(container.attrs\['HostConfig'\]\['DeviceRequests'\]) \> 0
> assert any(request.get('Capabilities', None) == \['gpu'\] for request in container.attrs\['HostConfig'\]\['DeviceRequests'\])
> 
> 
> test\_device\_request\_goes\_through(docker\_client) # pass
> test\_device\_request\_goes\_through(podman\_client) # fail
> \`\`\`
> 
> \### Describe the results you received
> 
> \- It should be possible to create containers with GPUs over the Podman socket API
> \- The created container should have a non-empty value for \`.HostConfig.DeviceRequests\`
> 
> \### Describe the results you expected
> 
> \- Device request is not honored when creating container via Podman socket 
> 
> \### podman info output
> 
> \`\`\`yaml
> host:
> arch: amd64
> buildahVersion: 1.35.3
> cgroupControllers:
> - memory
> - pids
> cgroupManager: systemd
> cgroupVersion: v2
> conmon:
> package: /usr/bin/conmon is owned by conmon 1:2.1.11-1
> path: /usr/bin/conmon
> version: 'conmon version 2.1.10, commit: e21e7c85b7637e622f21c57675bf1154fc8b1866'
> cpuUtilization:
> idlePercent: 94.1
> systemPercent: 1.54
> userPercent: 4.36
> cpus: 20
> databaseBackend: boltdb
> distribution:
> distribution: arch
> version: unknown
> eventLogger: journald
> freeLocks: 2012
> hostname: geo
> idMappings:
> gidmap:
> - container\_id: 0
> host\_id: 1000
> size: 1
> - container\_id: 1
> host\_id: 100000
> size: 65536
> uidmap:
> - container\_id: 0
> host\_id: 1000
> size: 1
> - container\_id: 1
> host\_id: 100000
> size: 65536
> kernel: 6.8.9-arch1-1
> linkmode: dynamic
> logDriver: journald
> memFree: 97578004480
> memTotal: 134802944000
> networkBackend: netavark
> networkBackendInfo:
> backend: netavark
> dns:
> package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.10.0-2
> path: /usr/lib/podman/aardvark-dns
> version: aardvark-dns 1.10.0
> package: /usr/lib/podman/netavark is owned by netavark 1.10.3-1
> path: /usr/lib/podman/netavark
> version: netavark 1.10.3
> ociRuntime:
> name: crun
> package: /usr/bin/crun is owned by crun 1.15-1
> path: /usr/bin/crun
> version: |-
> crun version 1.15
> commit: e6eacaf4034e84185fd8780ac9262bbf57082278
> rundir: /run/user/1000/crun
> spec: 1.0.0
> +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
> os: linux
> pasta:
> executable: /usr/bin/pasta
> package: /usr/bin/pasta is owned by passt 2024\_04\_26.d03c4e2-1
> version: |
> pasta 2024\_04\_26.d03c4e2
> Copyright Red Hat
> GNU General Public License, version 2 or later
> \<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html\>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> remoteSocket:
> exists: true
> path: /run/user/1000/podman/podman.sock
> security:
> apparmorEnabled: false
> capabilities: CAP\_CHOWN,CAP\_DAC\_OVERRIDE,CAP\_FOWNER,CAP\_FSETID,CAP\_KILL,CAP\_NET\_BIND\_SERVICE,CAP\_SETFCAP,CAP\_SETGID,CAP\_SETPCAP,CAP\_SETUID,CAP\_SYS\_CHROOT
> rootless: true
> seccompEnabled: true
> seccompProfilePath: /etc/containers/seccomp.json
> selinuxEnabled: false
> serviceIsRemote: false
> slirp4netns:
> executable: /usr/bin/slirp4netns
> package: /usr/bin/slirp4netns is owned by slirp4netns 1.3.0-1
> version: |-
> slirp4netns version 1.3.0
> commit: 8a4d4391842f00b9c940bb8f067964427eb0c964
> libslirp: 4.7.0
> SLIRP\_CONFIG\_VERSION\_MAX: 4
> libseccomp: 2.5.5
> swapFree: 0
> swapTotal: 0
> uptime: 1h 31m 25.00s (Approximately 0.04 days)
> variant: ""
> plugins:
> authorization: null
> log:
> - k8s-file
> - none
> - passthrough
> - journald
> network:
> - bridge
> - macvlan
> - ipvlan
> volume:
> - local
> registries: {}
> store:
> configFile: /home/jenni/.config/containers/storage.conf
> containerStore:
> number: 20
> paused: 0
> running: 14
> stopped: 6
> graphDriverName: overlay
> graphOptions: {}
> graphRoot: /home/jenni/.local/share/containers/storage
> graphRootAllocated: 1578640605184
> graphRootUsed: 1019202039808
> graphStatus:
> Backing Filesystem: btrfs
> Native Overlay Diff: "true"
> Supports d\_type: "true"
> Supports shifting: "false"
> Supports volatile: "true"
> Using metacopy: "false"
> imageCopyTmpDir: /var/tmp
> imageStore:
> number: 56
> runRoot: /run/user/1000/containers
> transientStore: false
> volumePath: /home/jenni/.local/share/containers/storage/volumes
> version:
> APIVersion: 5.0.2
> Built: 1713438799
> BuiltTime: Thu Apr 18 07:13:19 2024
> GitCommit: 3304dd95b8978a8346b96b7d43134990609b3b29-dirty
> GoVersion: go1.22.2
> Os: linux
> OsArch: linux/amd64
> Version: 5.0.2
> \`\`\`
> 
> 
> \### Podman in a container
> 
> No
> 
> \### Privileged Or Rootless
> 
> Rootless
> 
> \### Upstream Latest Release
> 
> Yes
> 
> \### Additional environment details
> 
> \`\`\`
> $ nvidia-container-cli info
> NVRM version: 550.78
> CUDA version: 12.4
> 
> Device Index: 0
> Device Minor: 0
> Model: NVIDIA GeForce RTX 3080 Ti
> Brand: GeForce
> GPU UUID: GPU-c61acb21-8716-6540-271c-39beab917d03
> Bus Location: 00000000:01:00.0
> Architecture: 8.6
> \`\`\`
> 
> \### Additional information
> 
> \_No response\_

I have add a look to podmanclispawner and podmanspawner, but the source code is old.

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [June 25, 2024, 4:00pm UTC](https://discourse.jupyter.org/t/jupyterhub-dockerspawner-podman-and-gpu/26447/6 "2024-06-25T16:00:40Z")

</div>

podmanclispawner should let you pass additional command line arguments:

> <https://github.com/manics/podmanclispawner/blob/418a15b15dfeb60a79f7d9653d2097dda771bf3a/podmanclispawner/podmanspawner.py#L90-L94>

Does that work?