Podman-inside-Kubernetes (pink) or Docker-inside-Docker (dind) to build repositories

I am working to deploy a BinderHub instance on Kubernetes 1.26 using containerd providing the CRI. That means, there is no docker socket on the host for repo2docker to use. I was glad to find the dind and pink instructions in the docs to deal with this issue. I chose to try podman in Kubernetes.

Unfortunately, the build container still attempts to use the docker socket of the host. I hope you can point me at a configuration detail I am missing. Details are below.

Here is my setup:

helm install binder jupyterhub/binderhub \
        --version=1.0.0-0.dev.git.3020.h694f8cd \
        --namespace=binder \
        -f secret.yaml \
        -f config.yaml

with the following config.yaml:

imageBuilderType: pink

  enabled: false

    use_registry: true
    image_prefix: my.registry/binder-

and the secret.yaml looks something like this:

  url: "https://my.registry"
  username: user
  password: "password"

In the resulting setup looks as would expect:

NAME                                  READY   STATUS    RESTARTS   AGE
pod/binder-8586f46c88-bkj4x           1/1     Running   0          117s
pod/binder-pink-9q2zx                 1/1     Running   0          118s
pod/binder-pink-rzjcl                 1/1     Running   0          118s
pod/binder-pink-zll8g                 1/1     Running   0          118s
pod/hub-57d9f96695-z6x5f              1/1     Running   0          117s
pod/proxy-7c9549b9bb-jzn2t            1/1     Running   0          117s
pod/user-scheduler-7c9f554b4f-7zrpq   1/1     Running   0          117s
pod/user-scheduler-7c9f554b4f-ng24b   1/1     Running   0          117s

NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)        AGE
service/binder         LoadBalancer   XXX.XXX.XXX.XXX   80:30608/TCP   119s
service/hub            ClusterIP    <none>            8081/TCP       119s
service/proxy-api      ClusterIP    <none>            8001/TCP       119s
service/proxy-public   LoadBalancer     XXX.XXX.XXX.XXX   80:31119/TCP   119s

daemonset.apps/binder-pink   3         3         3       3            3           <none>          118s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/binder           1/1     1            1           118s
deployment.apps/hub              1/1     1            1           118s
deployment.apps/proxy            1/1     1            1           118s
deployment.apps/user-scheduler   2/2     2            2           118s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/binder-8586f46c88           1         1         1       118s
replicaset.apps/hub-57d9f96695              1         1         1       118s
replicaset.apps/proxy-7c9549b9bb            1         1         1       118s
replicaset.apps/user-scheduler-7c9f554b4f   2         2         2       118s

NAME                                READY   AGE
statefulset.apps/user-placeholder   0/0     118s

Building a repository hangs, and when I look at the Kubernetes I see a build pod with this

Name:             build-XXX-22264f-ae4-49
Namespace:        binder
Priority:         0
Service Account:  default
Node:             k8s-worker-0-m2crbkv75h/
Start Time:       Wed, 01 Mar 2023 07:53:06 +0000
Labels:           component=binderhub-build
Annotations:      binder-repo: https://github.com/XXX
Status:           Pending
IPs:              <none>
    Container ID:  
    Image:         quay.io/jupyterhub/repo2docker:2022.10.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
      memory:  0
      memory:     0
    Environment:  <none>
      /root/.docker from docker-config (rw)
      /var/run/docker.sock from docker-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wrzx5 (ro)
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  Socket
    Type:        Secret (a volume populated by a Secret)
    SecretName:  binder-build-docker-config
    Optional:    false
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 hub.jupyter.org/dedicated=user:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    11m                    default-scheduler  Successfully assigned binder/build-XXX-22264f-ae4-49 to k8s-worker-0-m2crbkv75h
  Warning  FailedMount  4m56s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[docker-config kube-api-access-wrzx5 docker-socket]: timed out waiting for the condition
  Warning  FailedMount  2m42s (x2 over 7m12s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[kube-api-access-wrzx5 docker-socket docker-config]: timed out waiting for the condition
  Warning  FailedMount  77s (x13 over 11m)     kubelet            MountVolume.SetUp failed for volume "docker-socket" : hostPath type check failed: /var/run/docker.sock is not a socket file
  Warning  FailedMount  26s (x2 over 9m30s)    kubelet            Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[docker-socket docker-config kube-api-access-wrzx5]: timed out waiting for the condition

Which is precisely what I was hoping to avoid by setting up Podman-inside-Kubernetes. Using Docker in Docker-inside-Docker instead yields the same result.

The problem is that the build container looks for: /var/run/docker.sock. The pink daemonset provides /var/run/pink/podman.sock and the dins daemonset provides /var/run/dind/docker.sock on the host.

It looks like the snippet here [binderhub_config.py#50] controls this.

imageBuilderType = get_value("imageBuilderType")
if imageBuilderType in ["dind", "pink"]:
    hostSocketDir = get_value(f"{imageBuilderType}.hostSocketDir")
    if hostSocketDir:
        socketname = "docker" if imageBuilderType == "dind" else "podman"
        c.BinderHub.build_docker_host = f"unix://{hostSocketDir}/{socketname}.sock"

That looks OK, since BinderHub.build_docker_host is deprecated, I tried replacing it with KubernetesBuildExecutor.docker_host, and pointed my config.py to the resulting binder image … but that did not seem to change anything.

I am out of my depth here. Any advice?

Looking at the logs of the binder pod, this does stand out:

Loading /etc/binderhub/config/values.yaml
[BinderHub] ERROR | Exception while loading config file /etc/binderhub/config/binderhub_config.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/traitlets/config/application.py", line 909, in _load_config_files
    config = loader.load_config()
  File "/usr/local/lib/python3.11/site-packages/traitlets/config/loader.py", line 626, in load_config
  File "/usr/local/lib/python3.11/site-packages/traitlets/config/loader.py", line 659, in _read_file_as_dict
    exec(compile(f.read(), conf_filename, "exec"), namespace, namespace)  # noqa
  File "/etc/binderhub/config/binderhub_config.py", line 58, in <module>
    hub_url = urlparse(c.BinderHub.hub_url)
  File "/usr/local/lib/python3.11/urllib/parse.py", line 385, in urlparse
    url, scheme, _coerce_result = _coerce_args(url, scheme)
  File "/usr/local/lib/python3.11/urllib/parse.py", line 124, in _coerce_args
    return _decode_args(args) + (_encode_result,)
  File "/usr/local/lib/python3.11/urllib/parse.py", line 108, in _decode_args
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
  File "/usr/local/lib/python3.11/urllib/parse.py", line 108, in <genexpr>
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'LazyConfigValue' object has no attribute 'decode'


This error probably means the config wasn’t loaded at all, which would explain why it’s still trying to use the Docker socket. Can you try setting c.BinderHub.hub_url = "" (or if you already know the hub’s external URL then set it to that)

Thank you, @manics !

Indeed, now the configuration files loads and the the build pod uses the Podman socket :slight_smile:

Should BinderHub.hub_url have an empty string as default value?

I think so, do you want to open a PR (and reference this topic)?

Yes, it would be a pleasure :slight_smile:

1 Like

I created the PR, referencing this conversation: Ensure hub_url is configured before use by berghaus · Pull Request #1641 · jupyterhub/binderhub · GitHub

1 Like