I am working to deploy a BinderHub instance on Kubernetes 1.26 using containerd providing the CRI. That means, there is no docker socket on the host for repo2docker to use. I was glad to find the dind and pink instructions in the docs to deal with this issue. I chose to try podman in Kubernetes.
Unfortunately, the build container still attempts to use the docker socket of the host. I hope you can point me at a configuration detail I am missing. Details are below.
Here is my setup:
helm install binder jupyterhub/binderhub \
--version=1.0.0-0.dev.git.3020.h694f8cd \
--namespace=binder \
-f secret.yaml \
-f config.yaml
with the following config.yaml:
---
imageBuilderType: pink
imageCleaner:
enabled: false
config:
BinderHub:
use_registry: true
image_prefix: my.registry/binder-
and the secret.yaml looks something like this:
---
registry:
url: "https://my.registry"
username: user
password: "password"
In the resulting setup looks as would expect:
NAME READY STATUS RESTARTS AGE
pod/binder-8586f46c88-bkj4x 1/1 Running 0 117s
pod/binder-pink-9q2zx 1/1 Running 0 118s
pod/binder-pink-rzjcl 1/1 Running 0 118s
pod/binder-pink-zll8g 1/1 Running 0 118s
pod/hub-57d9f96695-z6x5f 1/1 Running 0 117s
pod/proxy-7c9549b9bb-jzn2t 1/1 Running 0 117s
pod/user-scheduler-7c9f554b4f-7zrpq 1/1 Running 0 117s
pod/user-scheduler-7c9f554b4f-ng24b 1/1 Running 0 117s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/binder LoadBalancer 10.100.171.254 XXX.XXX.XXX.XXX 80:30608/TCP 119s
service/hub ClusterIP 10.108.66.103 <none> 8081/TCP 119s
service/proxy-api ClusterIP 10.111.233.77 <none> 8001/TCP 119s
service/proxy-public LoadBalancer 10.98.35.220 XXX.XXX.XXX.XXX 80:31119/TCP 119s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/binder-pink 3 3 3 3 3 <none> 118s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/binder 1/1 1 1 118s
deployment.apps/hub 1/1 1 1 118s
deployment.apps/proxy 1/1 1 1 118s
deployment.apps/user-scheduler 2/2 2 2 118s
NAME DESIRED CURRENT READY AGE
replicaset.apps/binder-8586f46c88 1 1 1 118s
replicaset.apps/hub-57d9f96695 1 1 1 118s
replicaset.apps/proxy-7c9549b9bb 1 1 1 118s
replicaset.apps/user-scheduler-7c9f554b4f 2 2 2 118s
NAME READY AGE
statefulset.apps/user-placeholder 0/0 118s
Building a repository hangs, and when I look at the Kubernetes I see a build pod with this
Name: build-XXX-22264f-ae4-49
Namespace: binder
Priority: 0
Service Account: default
Node: k8s-worker-0-m2crbkv75h/192.168.0.204
Start Time: Wed, 01 Mar 2023 07:53:06 +0000
Labels: component=binderhub-build
name=build-XXX-22264f-ae4-49
Annotations: binder-repo: https://github.com/XXX
Status: Pending
IP:
IPs: <none>
Containers:
builder:
Container ID:
Image: quay.io/jupyterhub/repo2docker:2022.10.0
Image ID:
Port: <none>
Host Port: <none>
Args:
jupyter-repo2docker
--ref=ae48ee3a3c868b2005f34d482acbb17c6b2225cb
--image=XXX-22264f:ae48ee3a3c868b2005f34d482acbb17c6b2225cb
--no-clean
--no-run
--json-logs
--user-name=jovyan
--user-id=1000
--push
https://github.com/XXX
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
memory: 0
Requests:
memory: 0
Environment: <none>
Mounts:
/root/.docker from docker-config (rw)
/var/run/docker.sock from docker-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wrzx5 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
docker-socket:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType: Socket
docker-config:
Type: Secret (a volume populated by a Secret)
SecretName: binder-build-docker-config
Optional: false
kube-api-access-wrzx5:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: hub.jupyter.org/dedicated=user:NoSchedule
hub.jupyter.org_dedicated=user:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned binder/build-XXX-22264f-ae4-49 to k8s-worker-0-m2crbkv75h
Warning FailedMount 4m56s kubelet Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[docker-config kube-api-access-wrzx5 docker-socket]: timed out waiting for the condition
Warning FailedMount 2m42s (x2 over 7m12s) kubelet Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[kube-api-access-wrzx5 docker-socket docker-config]: timed out waiting for the condition
Warning FailedMount 77s (x13 over 11m) kubelet MountVolume.SetUp failed for volume "docker-socket" : hostPath type check failed: /var/run/docker.sock is not a socket file
Warning FailedMount 26s (x2 over 9m30s) kubelet Unable to attach or mount volumes: unmounted volumes=[docker-socket], unattached volumes=[docker-socket docker-config kube-api-access-wrzx5]: timed out waiting for the condition
Which is precisely what I was hoping to avoid by setting up Podman-inside-Kubernetes. Using Docker in Docker-inside-Docker instead yields the same result.
The problem is that the build container looks for: /var/run/docker.sock. The pink daemonset provides /var/run/pink/podman.sock and the dins daemonset provides /var/run/dind/docker.sock on the host.
It looks like the snippet here [binderhub_config.py#50] controls this.
imageBuilderType = get_value("imageBuilderType")
if imageBuilderType in ["dind", "pink"]:
hostSocketDir = get_value(f"{imageBuilderType}.hostSocketDir")
if hostSocketDir:
socketname = "docker" if imageBuilderType == "dind" else "podman"
c.BinderHub.build_docker_host = f"unix://{hostSocketDir}/{socketname}.sock"
That looks OK, since BinderHub.build_docker_host is deprecated, I tried replacing it with KubernetesBuildExecutor.docker_host, and pointed my config.py to the resulting binder image … but that did not seem to change anything.
I am out of my depth here. Any advice?
