K8s-network-tools caused "exec format error"

Hi there,
We have jupyterhub desployment over our Kubernetes infrastructure. We have different user profiles, one of these profiles is specially configured to run on a host with arm (and nvidia gpu):

- display_name: " environment with one NVIDIA"
      description: "ARM, Edge, NVIDIA, GPU"
      kubespawner_override:
        image: step21/jupyter-minimal-notebook:arm64
        extra_resource_guarantees:
          nvidia.com/gpu: "1"
        extra_resource_limits:
          nvidia.com/gpu: "1"
        node_affinity_required:
          - matchExpressions:
              - key: ARCH
                operator: In
                values:
                  - "arm64"
          - matchExpressions:
              - key: LAYER
                operator: In
                values:
                  - "edge"
        start_timeout: 90

When I try to deploy a Server with this profile it finanlizes with a pod crash error. I have realized that in some moment, “something”, is trying to configure some iptables:

Going into that iptables container, I get:

# docker logs e43b920d5628
standard_init_linux.go:211: exec user process caused "exec format error"

Typical error produced by a docker image for a wrong architecture. But as you can see, the profile we try to deploy, uses an arm64 notebook.

The image used which runs the container with the exec error is: jupyterhub/k8s-network-tools. Which seems to have amd64 images but not arm64. What could we do?

If I remember correctly the network-tools image is only used when cloud metadata is disabled:


since it’s used to do some custom network config to block outgoing requests.

If you set it to enabled this may work around the problem:

singleuser:
  cloudMetadata:
    enabled: true

If you do want to block metadata requests you can use NetworkPolicies as an alternative https://zero-to-jupyterhub.readthedocs.io/en/latest/administrator/security.html#kubernetes-network-policies

thanks It did work. At least this part, then I have other connectivity issues that avoids to deploy the notebook.
Anyway, what does cloud metadata?

Cloud metadata is a service provided by many public cloud providers. For example see