Have you (re(re))tried the build just in case it fixed itself between when you ran the build and the busybox test?
I don’t think that by default the helm chart does anything to limit the network connectivity of build pods. Did you do anything with respect to network policies or such? Could you kubectl describe pod <buildpodnamehere> the runninng build pod to see if that contains anything interesting?
TL;DR: I don’t have a good idea right now so fishing around a bit.
Did you do anything to configure docker-in-docker (DIND) pods? If that doesn’t ring a bell the answer is no.
I think by default the builds end up using the docker daemon on the node that the pod runs on. I think you can get access to that by ssh'ing to a node and running docker build somedockerfilethatdoessomethinglikethefailingsteps and see what you see.
Is this a local kubernetes cluster or one at a cloud provider? fishing
I think this means that something with the kubernetes cluster and its networking isn’t setup properly. I don’t have any experience with setting up clusters though so not really sure where to point you to :-/
I am currently working on the spinning up BinderHub on AWS EKS and I have encountered the same issue.
I’ve found SSHing directly into a EC2 node that is part of NodeGroup and then running a docker container directly on a host is definitely different than how a Kubernetes deployment would run the same docker container. My understanding is that their is VPC-CNI networking layer to allow containers to talk to each other and the outside world (as opposed to a docker0/overlay interface as would be for a local docker deployment).
So, even though
[ec2-user@ip-192-168-141-71 ~]$ docker run busybox nslookup archive.ubuntu.com
nslookup: can’t connect to remote host (192.168.0.2): Network is unreachable
is the same behaviour I observe, I can successfully run a k8s job such as creating a file called job-nslookup.yaml with contents
However, there was added a new --enable-docker-bridge as a bootstrap argument that is supposed to restore the previous behaviour.
I am currently searching for the correct way to pass in --enable-docker-bridge with eksctl (I know how to do it with CloudFormation but I figure there has to be a way of passing this option in when the nodegroup is created with eksctl).
As far as I can tell eksctl does not (yet?) have support for --enable-docker-bridge. I am not even sure that the bootstrap.sh script that would for an AMI through CloudFormation (used when setting up EKS without eksctl) is even the same boot strapping script used by eksctl. eksctl does support both the concepts of overrideBootstrapScript and preBootstrapCommand though. So have a several false starts, I’ve added the following to by configuration yaml that I used to spin up by eksctl controlled cluster:
# Replicate what --enable-docker-bridge does in /etc/eks/bootstrap.sh
# Enabling the docker bridge network. We have to disable live-restore as it
# prevents docker from recreating the default bridge network on restart
- "cp /etc/docker/daemon.json /etc/docker/daemon_backup.json"
- "echo -e '.bridge=\"docker0\" | .\"live-restore\"=false' > /etc/docker/jq_script"
- "jq -f /etc/docker/jq_script /etc/docker/daemon_backup.json | tee /etc/docker/daemon.json"
- "systemctl restart docker"
The shell commands are bit longer than I would have expected. But it seemed that I needed to make of copy of daemon.json otherwise I would end up with a blank file and there was some odd quotes in strings that needed to be handled.
But, overall, success! I’ve now got a skeleton BinderHub up and running on AWS using EKS and eksctl.