Didn't manage to make DockerSpawner work (Unknown docker network)

I’m trying to set up a docker-compose configuration for JupyterHub that can use GitLab server for authentication and provide a separate workspace for each user. If I understood it correctly, for this case I should use DockerSpwner which provides isolated container for each user.
I did manage to make GitLab authentication work but failed with DockerSpowner

My docker-compose.yml:

version: '3'
services:
  jupyterihub:
    restart: always
    container_name: jupyterhub_hub
    build: .
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./jupyterhub_config.py:/etc/jupyterhub/jupyterhub_config.py
    environment:

      DOCKER_NETWORK_NAME: jupyter_net
      HUB_IP: jupyterhub_hub
    ports:
      - "8181:8000"
    networks:
      - jupyter_net

volumes:
  jupyterhiub_data:

networks:
  jupyter_net:
    external: true

My Dockerfile:

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND=noninteractive
RUN ln -fs /usr/share/zoneinfo/Universal /etc/localtime

RUN apt-get update && apt-get install -y python3-pip nodejs npm


RUN npm install -g configurable-http-proxy
RUN pip3 install -U pip

COPY requirements.txt .
RUN pip3 install -r requirements.txt
CMD jupyterhub --ip 0.0.0.0 -f /etc/jupyterhub/jupyterhub_config.py

requirements.txt

jupyterlab~=3.2.5
notebook~=6.4.6
matplotlib~=3.5.1
jupyterhub~=2.0.0
scipy~=1.7.3
ipympl~=0.8
oauthenticator~=14.2
dockerspawner~=12.1

When I build and start the gitlub, I have an error after successful authentication:

jupyterhub_hub | [I 2022-01-06 10:16:11.980 JupyterHub provider:607] Creating oauth client jupyterhub-user-atimin
jupyterhub_hub | [I 2022-01-06 10:16:12.127 JupyterHub dockerspawner:1281] Found existing container jupyter-atimin (id: 86d7d5e)
jupyterhub_hub | [I 2022-01-06 10:16:12.127 JupyterHub dockerspawner:1296] Starting container jupyter-atimin (id: 86d7d5e)
jupyterhub_hub | [E 2022-01-06 10:16:12.730 JupyterHub user:762] Unhandled error starting atimin's server: Unknown docker network 'jupyter_net'. Did you create it with `docker network create <name>`?
jupyterhub_hub | [I 2022-01-06 10:16:12.735 JupyterHub dockerspawner:1390] Stopping container jupyter-atimin (id: 86d7d5e)
jupyterhub_hub | [I 2022-01-06 10:16:12.928 JupyterHub log:189] 302 GET /hub/spawn/atimin -> /hub/spawn-pending/atimin (atimin@10.0.0.166) 1007.25ms
jupyterhub_hub | ERROR:asyncio:Task exception was never retrieved
jupyterhub_hub | future: <Task finished name='Task-40' coro=<BaseHandler.spawn_single_user() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:831> exception=Exception("Unknown docker network 'jupyter_net'. Did you create it with `docker network create <name>`?")>
jupyterhub_hub | Traceback (most recent call last):
jupyterhub_hub |   File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 1032, in spawn_single_user
jupyterhub_hub |     await gen.with_timeout(
jupyterhub_hub |   File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 942, in finish_user_spawn
jupyterhub_hub |     await spawn_future
jupyterhub_hub |   File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 780, in spawn
jupyterhub_hub |     raise e
jupyterhub_hub |   File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 679, in spawn
jupyterhub_hub |     url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
jupyterhub_hub |   File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 1309, in start
jupyterhub_hub |     ip, port = await self.get_ip_and_port()
jupyterhub_hub |   File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 1350, in get_ip_and_port
jupyterhub_hub |     ip = self.get_network_ip(network_settings)
jupyterhub_hub |   File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 1372, in get_network_ip
jupyterhub_hub |     raise Exception(
jupyterhub_hub | Exception: Unknown docker network 'jupyter_net'. Did you create it with `docker network create <name>`?
jupyterhub_hub | [E 2022-01-06 10:16:13.285 JupyterHub pages:378] Previous spawn for atimin failed: Unknown docker network 'jupyter_net'. Did you create it with `docker network create <name>`?
jupyterhub_hub | [E 2022-01-06 10:16:13.297 JupyterHub log:181] {
jupyterhub_hub |       "X-Forwarded-Host": "jupyter.panda.technology",
jupyterhub_hub |       "X-Forwarded-Proto": "http",
jupyterhub_hub |       "X-Forwarded-Port": "80",
jupyterhub_hub |       "X-Forwarded-For": "10.0.0.166",
jupyterhub_hub |       "If-None-Match": "\"9324018ac36cb98d1262514bbec3aa9f1c0718e5\"",
jupyterhub_hub |       "Cookie": "jupyterhub-hub-login=[secret]; _ga=[secret]; jupyterhub-session-id=[secret]",
jupyterhub_hub |       "Accept-Language": "ru,en-US;q=0.9,en;q=0.8,de;q=0.7",
jupyterhub_hub |       "Accept-Encoding": "gzip, deflate, br",
jupyterhub_hub |       "Referer": "https://jupyter.panda.technology/hub/home",
jupyterhub_hub |       "Sec-Ch-Ua-Platform": "\"Linux\"",
jupyterhub_hub |       "Sec-Ch-Ua-Mobile": "?0",
jupyterhub_hub |       "Sec-Ch-Ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"96\"",
jupyterhub_hub |       "Sec-Fetch-Dest": "document",
jupyterhub_hub |       "Sec-Fetch-User": "?1",
jupyterhub_hub |       "Sec-Fetch-Mode": "navigate",
jupyterhub_hub |       "Sec-Fetch-Site": "same-origin",
jupyterhub_hub |       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
jupyterhub_hub |       "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
jupyterhub_hub |       "Upgrade-Insecure-Requests": "1",
jupyterhub_hub |       "Connection": "close",
jupyterhub_hub |       "Host": "jupyter.panda.technology"
jupyterhub_hub |     }
jupyterhub_hub | [E 2022-01-06 10:16:13.298 JupyterHub log:189] 500 GET /hub/spawn-pending/atimin (atimin@10.0.0.166) 17.76ms

I see that the problem with the docker network, but I have no clue what it is wrong with it:

docker network ls
NETWORK ID     NAME                      DRIVER    SCOPE
c0b34eb68d6b   jupyter_net               bridge    local

Does this work if you don’t specify a network (i.e. use the default)?

Is the jupyter_net network visible inside the container (e.g. run docker exec -it <container> bash to get a shell inside the container, and query the Docker API from inside)

Unfortunately, it doesn’t work with default network as well.

query Docker API from inside

Do you mean HTTP API? Not sure, If I understood how to reach this.

The docker-py library is used by DockerSpawner, so you could start a Python shell (or write a Python script) to query the list of Networks… I don’t have easy access to a docker host right now to test a script.

Alternatively you could install the Docker CLI inside your image, and run docker network ls