# Podman(1) not working inside JupyterLab containers that are launched by JupyterHub

**URL:** <https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188>\
**Category:** General\
**Tags:** community, jupyterhub, how-to, help-wanted\
**Created:** [December 17, 2020, 1:20am UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188 "2020-12-17T01:20:15Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![nmvega](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nmvega/32/1960_2.png) [@nmvega](https://discourse.jupyter.org/u/nmvega)\
**Post date:** [December 17, 2020, 1:20am UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/1 "2020-12-17T01:20:15Z")

</div>

Hello Friends:

This question originally appeared on GitHub as: [podman(1) not working inside JupyterLab containers that are spawned by JupyterHub (i.e. via DockerSpawner) …](https://github.com/jupyterhub/jupyterhub/issues/3309).

I have a `JupyterLab` ( **LAB** ) container image that accepts `podman(1)` commands fine when I spawn the container image manually via the CLI; but on the other hand, experiences `podman(1)` container-related errors when that same container image is spawned via `JupyterHub`'s `DockerSpawner`.

Let me show you the issue step-by-step.

Here are `JupyterHub` and `Postgres` running (… no issues here, they work 100% perfectly):

```auto
root@GUEST# docker ps -a --no-trunc
ID IMAGE COMMAND PORTS NAMES
61cfb acme/jupyterhub:1.0 "/opt/jupyterhub.d/usr/bin/jupyterhub.sh" 0.0.0.0:443->443/tcp jupyterhub
d4793 postgres:latest "docker-entrypoint.sh postgres" 0.0.0.0:15432->5432/tcp jupyterhub-db

```

**WORKING CASE**

Spawn the `JupyterLab` container image manually via **CLI** and issue `podman(1)` commands. Note that these manual steps do not use the above `JupyterHub` services.

```auto
root@GUEST# docker run \
        -it --name jupyter-janedoe \
        --privileged \
        -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
        -p 18888:8888 acme/jupyterlab-base:1.0 \
        /bin/bash

jovyan@CONTAINER$ sudo dnf -y install podman 
   # This is RedHat's podman(1) utility, which I actually bake into this image when I
   # "docker build [...]" it. I'm just trying to be explicit here. =:)

jovyan@CONTAINER$ podman pull --events-backend=file docker.io/library/mariadb:latest

jovyan@CONTAINER$ podman image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/mariadb latest 3a348a04a815 3 weeks ago 413 MB

jovyan@CONTAINER$ mkdir -p ${HOME}/data.d/mariadb.d/var/lib/mysql

jovyan@CONTAINER$ podman run --events-backend=file \
      --name mariadb01 \
     -p 3306:3306 \
     -v ${HOME}/data.d/mariadb.d/var/lib/mysql:/var/lib/mysql:Z \
     -e MYSQL_ROOT_PASSWORD= **** \
     -e MYSQL_DATABASE=db01 \
     -e MYSQL_USER=janedoe \
     -e MYSQL_PASSWORD= **** \
     -d docker.io/library/mariadb

jovyan@CONTAINER$ podman ps --no-trunc
ID IMAGE CMD PORTS NAMES
1ac150 docker.io/library/mariadb mysqld 0.0.0.0:3306->3306/tcp mariadb01

jovyan@CONTAINER$ podman run \
      --events-backend=file \
      -it --rm mariadb \
      mysql -e 'show databases' -h 172.17.0.2 -u root -p
Enter password: 
+--------------------+
| Database |
+--------------------+
| db01 |
| information_schema |
| mysql |
| performance_schema |
+--------------------+
jovyan@CONTAINER$

```

Here is some In-CONTAINER O/S information, which we’ll compare with the **FAILURE CASE** below:

```auto
jovyan@CONTAINER$ id
uid=1000(jovyan) gid=100(users) groups=100(users)

jovyan@CONTAINER$ ps -ef
UID PID PPID C STIME TTY TIME CMD
jovyan 1 0 0 00:13 pts/0 00:00:00 tini -g -- /bin/bash
jovyan 6 1 0 00:13 pts/0 00:00:00 /bin/bash
jovyan 40 1 0 00:14 ? 00:00:00 podman
jovyan 307 1 0 00:17 ? 00:00:00 /usr/bin/fuse-overlayfs -o lowerdir=/home/jovyan/.local/sh
jovyan 309 1 0 00:17 pts/0 00:00:00 /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --e
jovyan 311 1 0 00:17 pts/0 00:00:00 containers-rootlessport
jovyan 324 311 0 00:17 pts/0 00:00:00 containers-rootlessport-child
jovyan 338 1 0 00:17 ? 00:00:00 /usr/bin/conmon --api-version 1 -c 1ac1502640165637e2b5969a7
100998 349 338 0 00:17 ? 00:00:00 mysqld
jovyan 1000 6 0 00:33 pts/0 00:00:00 ps -ef

jovyan@CONTAINER$ df -h
Filesystem Size Used Avail Use% Mounted on
overlay 1.8T 187G 1.6T 11% /
tmpfs 64M 0 64M 0% /dev
shm 64M 84K 64M 1% /dev/shm
/dev/sdb1 1.8T 187G 1.6T 11% /etc/hosts
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup

jovyan@CONTAINER$ mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/MKCAZS7NCCQUDMYCDUPVVG34TH:/var/lib/d
ocker/overlay2/l/BZPXDZSY7OMH6LSH43ZFMO3SYT:/var/lib/docker/overlay2/l/I65E26L6EXPUXAJ4JVFDEKGIRU:/var/lib/docke
r/overlay2/l/GHJSNVHCXVUL4AG6J4SGTR4MOL:/var/lib/docker/overlay2/l/V5L7C7DUUSCOEJ4R6YYDRUOKEP:/var/lib/docker/ov
erlay2/l/A47I4M26GPBDPYVKCZ22QUNJUE:/var/lib/docker/overlay2/l/UBZECXYTT5SKO5FAUGJYXPDZ76:/var/lib/docker/overla
y2/l/QISGNIRTGUVC2ST5OO7SBTLPEW:/var/lib/docker/overlay2/l/R4KP7BPSJFQF3D2L6YVIX7MWJW:/var/lib/docker/overlay2/l
/ZFVOGFK4FXDMGZLDTVCZP5CXCO:/var/lib/docker/overlay2/l/IQZVKDF2N73EJSRNJ5UFQZLABK:/var/lib/docker/overlay2/l/FMU
QWHAODMEGORO3UKJHQWDVBT,upperdir=/var/lib/docker/overlay2/c177c74b2ecded4eecb6c88f54983fae4affef9f0197aa0e83fa6a
9049731833/diff,workdir=/var/lib/docker/overlay2/c177c74b2ecded4eecb6c88f54983fae4affef9f0197aa0e83fa6a904973183
3/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/sdb1 on /etc/resolv.conf type ext4 (rw,relatime)
/dev/sdb1 on /etc/hostname type ext4 (rw,relatime)
/dev/sdb1 on /etc/hosts type ext4 (rw,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,size=4096k,nr_inodes=1024,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)

jovyan@CONTAINER$ exit # End the session, which removes the container.
root@GUEST#

```

**FAILURE CASE**

Now,… if I perform the exact same sequence, except that I’m logged into that same `JupyterLab` container spawned via the `JupyterHub` U.I. instead (that is, spawned via `DockerSpawner`), I get the following error when I issue the `podman run` command to launch `mariadb`:

`Error: OCI runtime error: container_linux.go:370: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: no cgroup mount found in mountinfo`

Here is some O/S information from inside this `DockerSpawner` spawned container (which should be compared with the **WORKING CASE** ):

```auto
root@GUEST# docker exec -it jupyter-janedoe /bin/bash # Already spawned. We're just connecting to it.

jovyan@CONTAINER$ id
uid=1000(jovyan) gid=100(users) groups=100(users)

jovyan@CONTAINER$ ps -ef
UID PID PPID C STIME TTY TIME CMD
jovyan 1 0 0 00:41 ? 00:00:00 tini -g -- start-singleuser.sh --ip=0.0.0.0 --port=8888 --no
jovyan 6 1 0 00:41 ? 00:00:01 /opt/conda/bin/python /opt/conda/bin/jupyterhub-singleuser -
jovyan 54 0 0 00:42 pts/0 00:00:00 /bin/bash
jovyan 111 1 0 00:45 ? 00:00:00 podman
jovyan 413 54 0 00:54 pts/0 00:00:00 ps -ef

jovyan@CONTAINER$ df -h
Filesystem Size Used Avail Use% Mounted on
overlay 1.8T 187G 1.6T 11% /
tmpfs 64M 0 64M 0% /dev
shm 64M 84K 64M 1% /dev/shm
tmpfs 32G 0 32G 0% /run
tmpfs 32G 0 32G 0% /tmp
tmpfs 32G 0 32G 0% /run/lock
/dev/sdb1 1.8T 187G 1.6T 11% /etc/hosts

jovyan@CONTAINER$ mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/P6F43PPRJY25RJKWD6JQAU6CMG:/var/lib/d
ocker/overlay2/l/BZPXDZSY7OMH6LSH43ZFMO3SYT:/var/lib/docker/overlay2/l/I65E26L6EXPUXAJ4JVFDEKGIRU:/var/lib/docke
r/overlay2/l/GHJSNVHCXVUL4AG6J4SGTR4MOL:/var/lib/docker/overlay2/l/V5L7C7DUUSCOEJ4R6YYDRUOKEP:/var/lib/docker/ov
erlay2/l/A47I4M26GPBDPYVKCZ22QUNJUE:/var/lib/docker/overlay2/l/UBZECXYTT5SKO5FAUGJYXPDZ76:/var/lib/docker/overla
y2/l/QISGNIRTGUVC2ST5OO7SBTLPEW:/var/lib/docker/overlay2/l/R4KP7BPSJFQF3D2L6YVIX7MWJW:/var/lib/docker/overlay2/l
/ZFVOGFK4FXDMGZLDTVCZP5CXCO:/var/lib/docker/overlay2/l/IQZVKDF2N73EJSRNJ5UFQZLABK:/var/lib/docker/overlay2/l/FMU
QWHAODMEGORO3UKJHQWDVBT,upperdir=/var/lib/docker/overlay2/bdd2a5ddab65e7b5e2b45e1854c6a8008a35d89f39aa5b57bdbb52
d2259634bb/diff,workdir=/var/lib/docker/overlay2/bdd2a5ddab65e7b5e2b45e1854c6a8008a35d89f39aa5b57bdbb52d2259634b
b/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime)
/dev/sdb1 on /etc/resolv.conf type ext4 (rw,relatime)
/dev/sdb1 on /etc/hostname type ext4 (rw,relatime)
/dev/sdb1 on /etc/hosts type ext4 (rw,relatime)
/dev/sdb1 on /home/jovyan type ext4 (rw,relatime)
/dev/sdb1 on /sys/fs/cgroup type ext4 (rw,relatime)

jovyan@CONTAINER$ exit
root@GUEST#

```

One difference you’ll immediately notice is that there are many `cgroup / cgroup2` … `mount(5)` entries missing in the `DockerSpawner` **failure case**.

Finally, I will note that `./jupyterhub_config.py` is configured to spawn `JupyterLab` containers as `privileged`. Meaning, the below `./jupyterhub_config.py` snippet is equivalent to telling `DockerSpawner` to run the `docker` container with these options, which I used in the **working case** :

`docker run [...] --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro [...]`

```auto
root@GUEST# vi .../jupyterhub_config.py
[... snip ...]
c.DockerSpawner.extra_host_config.update({
            "privileged" : True,
            "devices" : ["/sys/fs/cgroup:/sys/fs/cgroup:ro",],
            "tmpfs" : {"/tmp":"", "/run":"", "/run/lock":""}, })
[... snip ...]

```

And via `docker inspect jupyter-janedoe`, one can see that these settings are in effect (below).

Extracted from output of `docker inspect jupyter-janedoe`:

```auto
[... snip ...]
    "Devices": [
        {
          "PathOnHost": "/sys/fs/cgroup",
          "PathInContainer": "/sys/fs/cgroup",
          "CgroupPermissions": "ro"
        }
]
[... snip ...]

    "Tmpfs": {
               "/run": "",
               "/run/lock": "",
               "/tmp": ""
    }
[... snip ...]

    "Volumes": {
                 "/home/jovyan": {},
                 "/sys/fs/cgroup": {}
    }
[... snip ...]
    "Privileged": true,
[... snip ...]

```

In summary, I need to understand what, `equivalently`, is the difference between when I `spawn` the container manually via the `CLI` and when `DockerSpawner` spawns it. In other words, if we `reverse-engineered DockerSpawner` to come up with the `equivalent` command that it issues for `docker run [...]` (yes I know it uses the programmatic API), then we can know why the two O/S environments are not coming up identically as far as `privilege` (container runtime), `cgroup` and other O/S environment contexts are concerned. This, then, will allow for tweaking `./jupyterhub_config.py` so that I can get `podman(1)` to work.

Any help is appreciated from the `DockerSpawner` experts. Thank you in advance! =:)

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 18, 2020, 3:43pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/3 "2020-12-18T15:43:43Z")

</div>

> [@nmvega](#):
>
> ```auto
> c.DockerSpawner.extra_host_config.update({
> "privileged" : True,
> "devices" : ["/sys/fs/cgroup:/sys/fs/cgroup:ro",],
> 
> ```

Is there a reason `/sys/fs/cgroup` is specified as a device instead of a volume?

---

<div class="post-metadata">

**Author:** ![nmvega](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nmvega/32/1960_2.png) [@nmvega](https://discourse.jupyter.org/u/nmvega)\
**Post date:** [December 18, 2020, 4:59pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/5 "2020-12-18T16:59:58Z")

</div>

Hi @manics

First a quick note before replying to your question: I’m maintaining the **Original Post** (original question) on GitHub now because I can no longer correct and update (i.e. edit) the Original Post here. See one of my replies to see where those GitHub URLs are.

Now for your question. This is a GREAT question [!] and one that has been gnawing at me too; because I specified that item as a `--volume` (`-v`) on the **CLI**. Honestly, I saw that snippet elsewhere and re-used it, so it may not be 100% correct.

I checked the [Docker low-level Python API](https://docker-py.readthedocs.io/en/stable/api.html) to see if that entry for `/sys/fs/cgroup` was correct or incorrect. But it was very late (last night), and I couldn’t reverse-engineer it.

Do you happen to know what the correct `c.` attribute(s) and syntax for the above `/sys/fs/cgroup` volume mapping are?

---

<div class="post-metadata">

**Author:** ![nmvega](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nmvega/32/1960_2.png) [@nmvega](https://discourse.jupyter.org/u/nmvega)\
**Post date:** [December 18, 2020, 8:46pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/6 "2020-12-18T20:46:16Z")

</div>

**UPDATE1** :  
I tried variations of the below snippet, but with no luck (same error):

```auto
# ./jupyterhub_config.py -- snippet.

# ===========================================================================
c.DockerSpawner.volumes = dict()
c.DockerSpawner.extra_host_config = dict()
[... snip ...]
# ===========================================================================
c.DockerSpawner.extra_host_config.update({
            "privileged" : True,
            "tmpfs" : {"/tmp":"", "/run":"", "/run/lock":"", "/sys/fs/cgroup":""},
            })

c.DockerSpawner.volumes.update({"/sys/fs/cgroup" : {"bind":"/sys/fs/cgroup", "mode": "ro"}})
# ===========================================================================

```

More (and/or perhaps something different) might be needed to get this working correctly, and so reverse-engineering what `DockerSpawner` is doing and needs, is important.

Along those lines, in addition to the in-CONTAINER `mount(1)`, `df(1)` and `ps(1)` outputs that I provided in the original post; below are the respective `JSON` outputs from the `docker inspect jupyter-janedoe` command, as issued against that image spawned via the `docker CLI` (_see OP for the command_), and subsequently against that same image spawned via `DockerSpawner`:

- Via `docker CLI`: [https://pastebin.com/raw/naHY8iWh](https://pastebin.com/raw/naHY8iWh)
- Via `DockerSpawner`: [https://pastebin.com/raw/2UX3LcfS](https://pastebin.com/raw/2UX3LcfS)

(Remove the `/raw/` to see highlighted `JSON`).

Of course, the `CLI` version of the output will not (as expected) have elements related to a `JupyterHub` login session (e.g. `tokens`, `callback URLs`, etc). But that’s okay: What’s necessary is understanding the differences in `docker` and `O/S` **runtime** -environment **key/value** pairs, so that we can, in turn, understand what `./jupyterhub_config.py` requires. Thank you again in advance. 😊

---

<div class="post-metadata">

**Author:** ![nmvega](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nmvega/32/1960_2.png) [@nmvega](https://discourse.jupyter.org/u/nmvega)\
**Post date:** [December 18, 2020, 11:37pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/7 "2020-12-18T23:37:36Z")

</div>

**UPDATE2** :

Okay, the following appears to be a material difference in the `docker inspect jupyter-janedoe` (`JSON`) output between both cases, which may be contributing to this issue:

`docker inspect jupyter-janedoe` (launched via **CLI** ):

```auto
"HostConfig": {
    "Binds": [
          "/sys/fs/cgroup:/sys/fs/cgroup:ro"
 ],

```

`docker inspect jupyter-janedoe` (launched via **DockerSpawner** ):

```auto
"HostConfig": {
    "Binds": [
          "jupyterhub-user-janedoe:/home/jovyan:rw"
 ],

```

The **DockerSpawner** entry should be a `union` of both entries (as shown below), but it’s missing the second item:

```auto
"HostConfig": {
    "Binds": [
          "jupyterhub-user-janedoe:/home/jovyan:rw",
          "/sys/fs/cgroup:/sys/fs/cgroup:ro"
 ],

```

This means the code-snippet in my **UPDATE1** is incorrect.

So far I’ve verified that updating the `c.DockerSpawner.volumes.update` attribute is incorrect, because it led to the following at runtime, which isn’t needed (_and maybe even ambiguous with respect to what it even means_ LoL 😊):

```auto
# We want a Volume entry, not Device entry.
"Devices": [{
    "PathOnHost": "/sys/fs/cgroup",
    "PathInContainer": "/sys/fs/cgroup",
    "CgroupPermissions": "ro"
}]

```

So reverse-engineering a little bit, according to the **CLI** working case, we want the following `/sys/fs/cgroup` entries; but only the `"Volumes":` (middle) entry appears in the **DockerSpawner** failure case (the outer two entries are missing):

```auto
[... snip ...]

"Mounts": [{
   "Type": "bind",
   "Source": "/sys/fs/cgroup",
   "Destination": "/sys/fs/cgroup",
   "Mode": "ro",
   "RW": false,
   "Propagation": "rprivate" }]

[... snip ...]

"Volumes": {
   "/sys/fs/cgroup": {}
}

[... snip ...]

"HostConfig": {
   "Binds": [
     "/sys/fs/cgroup:/sys/fs/cgroup:ro"
],

```

To achieve this, I’m trying to figure out what the correct `attribute(s)` is/are to modify, as well as the correct syntax.

In summary, I need a `Mounts:` entry for `/sys/fs/cgroup`; as well as to append `/sys/fs/cgroup` entry to the `HostConfig` list.

---

<div class="post-metadata">

**Author:** ![nmvega](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nmvega/32/1960_2.png) [@nmvega](https://discourse.jupyter.org/u/nmvega)\
**Post date:** [December 19, 2020, 2:49am UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/8 "2020-12-19T02:49:57Z")

</div>

**UPDATE3** :

Based on this [Docker low-level Python API](https://docker-py.readthedocs.io/en/stable/api.html) and this [DockerSpawner API](https://jupyterhub-dockerspawner.readthedocs.io/en/latest/api/), I created the following snippets, which I’ll try next week (and may have to tweak). If anyone knows how to do this, comments or suggestions (after reviewing the above) always welcome.

```auto
c.DockerSpawner.extra_host_config = dict()

     [... snip ...]

c.DockerSpawner.extra_host_config.update(
   {'/sys/fs/cgroup' :
       {'bind' : '/sys/fs/cgroup',
        'mode' : 'ro' }}
)

--OR--

c.DockerSpawner.extra_host_config.update(
   binds = {'/sys/fs/cgroup' :
              {'bind' : '/sys/fs/cgroup',
               'mode' : 'ro' }}
)

```

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 19, 2020, 8:18pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/9 "2020-12-19T20:18:16Z")

</div>

DockerSpawner has a volumes property that should take a volume name or host path:

> **[Data persistence](https://jupyterhub-dockerspawner.readthedocs.io/en/latest/data-persistence.html#volume-mapping)**

> <https://github.com/jupyterhub/dockerspawner/blob/53b35e1619ff0dabb24a51e3356274546a70115f/dockerspawner/dockerspawner.py#L326-L347>

What happens if you use:

```auto
c.DockerSpawner.volumes = {"/sys/fs/cgroup" : {"bind":"/sys/fs/cgroup", "mode": "ro"}}

```

?

---

<div class="post-metadata">

**Author:** ![nmvega](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nmvega/32/1960_2.png) [@nmvega](https://discourse.jupyter.org/u/nmvega)\
**Post date:** [December 19, 2020, 9:04pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/10 "2020-12-19T21:04:21Z")

</div>

Hi @manics

Sadly, I subsequently tried that, but it didn’t produce the desired result. You can see that same `code` snippet in my **UPDATE1** comment (above). You can also see the result that that produced by viewing the `DockerSpawner` Pastebin URL that I provided in that same **UPDATE1**. Just search for `/sys/fs/cgroup` in that paste.

If you try it and get a different result, please let me know your context and what you implemented.

Meanwhile, I’ll eagerly look at the docs you provided. Thank you.

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 21, 2020, 3:12pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/11 "2020-12-21T15:12:05Z")

</div>

> [@nmvega](#):
>
> ```auto
> c.DockerSpawner.volumes = dict()
> c.DockerSpawner.extra_host_config = dict()
> [... snip ...]
> # ===========================================================================
> c.DockerSpawner.extra_host_config.update({
> "privileged" : True,
> "tmpfs" : {"/tmp":"", "/run":"", "/run/lock":"", "/sys/fs/cgroup":""},
> })
> 
> c.DockerSpawner.volumes.update({"/sys/fs/cgroup" : {"bind":"/sys/fs/cgroup", "mode": "ro"}})
> 
> ```

In your earlier post you were using `volumes.update` instead of setting it directly. This shouldn’t matter, but it’s worth trying. In addition you’ve snipped some of he config so it’s not possible to see if there’s something that might conflict.

If you still can’t get it working could you please show the full configuration? If it’s in a git repo that allows full reproducibility you might encourage more people to take a look.

---

<div class="post-metadata">

**Author:** ![nmvega](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nmvega/32/1960_2.png) [@nmvega](https://discourse.jupyter.org/u/nmvega)\
**Post date:** [December 21, 2020, 5:16pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/12 "2020-12-21T17:16:11Z")

</div>

@manics Thanks. I actually posted these items in a different post. See the [THIS LINK](https://discourse.jupyter.org/t/creating-a-jupyterlab-dockerspawner-image-launched-by-jupyterhub-that-employs-fedora-systemd-5/7179/3) for the files.

Don’t worry that that post focuses on getting `systemd(5)` working because the three files (`./jupyterhub_config.py`, `./docker-compose.yml` and `./.env`) are the same as if I would post them here. Of course I’ve updated `jupyterhub_config.py` since that post (… updates corresponding to what you’re seeing on this post), but now you have the entire files, top-to-bottom.

P.S. I tried posting this on GitHub (in JupyterHub and in DockerSpawner), but they keep closing them and directing me here.

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 21, 2020, 6:51pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/13 "2020-12-21T18:51:24Z")

</div>

You were redirected here from the GitHub issues because JupyterHub issues are reserved for bug reports and concrete feature requests, whereas this is a support/configuration problem.

When I suggested posted your file on git I meant a git repository (e.g. GitHub gist) that someone can clone to fetch your files, as this lowers the barrier for someone who wants to reproduce your problem. For a developer `git clone` is easier than copying and pasting from a forum post.

---

<div class="post-metadata">

**Author:** ![nmvega](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nmvega/32/1960_2.png) [@nmvega](https://discourse.jupyter.org/u/nmvega)\
**Post date:** [December 21, 2020, 7:47pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/14 "2020-12-21T19:47:22Z")

</div>

@manics

In the end, if anyone determines what `DockerSpawner` `attribute` and `data-structure` achieves the equivalent of the following CLI (which results in a `bind` Mount as depicted above in [UPDATE2](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/7)), please let me know.

```auto
docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro [...]

```

The `base` docker-stacks image works for tinkering on a laptop. No special image is required.

P.S. Since the `volume` attrubute suggestion didn’t work (with and without `.update()`, another possibility is a code or document bug.

I’ll just chip away at this. Thanks for the help.

---

<div class="post-metadata">

**Author:** ![nmvega](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nmvega/32/1960_2.png) [@nmvega](https://discourse.jupyter.org/u/nmvega)\
**Post date:** [December 24, 2020, 8:22pm UTC](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/15 "2020-12-24T20:22:43Z")

</div>

**SOLUTION** (2nd snippet of [UPDATE3](https://discourse.jupyter.org/t/podman-1-not-working-inside-jupyterlab-containers-that-are-launched-by-jupyterhub/7188/8)):

```auto
c.DockerSpawner.extra_host_config.update(
   binds = {'/sys/fs/cgroup' :
              {'bind' : '/sys/fs/cgroup',
               'mode' : 'ro' }})

```

I had already solved the `privileged` container dilemma; and integrating that solution with the above solution, yields the (final) complete solution:

```auto
c.DockerSpawner.extra_host_config.update(
   {"privileged" : True},
   binds = {'/sys/fs/cgroup' :
              {'bind' : '/sys/fs/cgroup',
               'mode' : 'ro' }})

```