We have jupyterhub desployment over our Kubernetes infrastructure. We have different user profiles, one of these profiles is specially configured to run on a host with arm (and nvidia gpu):
- display_name: " environment with one NVIDIA"
description: "ARM, Edge, NVIDIA, GPU"
- key: ARCH
- key: LAYER
When I try to deploy a Server with this profile it finanlizes with a pod crash error. I have realized that in some moment, “something”, is trying to configure some iptables:
Going into that iptables container, I get:
# docker logs e43b920d5628
standard_init_linux.go:211: exec user process caused "exec format error"
Typical error produced by a docker image for a wrong architecture. But as you can see, the profile we try to deploy, uses an arm64 notebook.
The image used which runs the container with the exec error is: jupyterhub/k8s-network-tools. Which seems to have amd64 images but not arm64. What could we do?
If I remember correctly the network-tools image is only used when cloud metadata is disabled:
since it’s used to do some custom network config to block outgoing requests.
If you set it to enabled this may work around the problem:
If you do want to block metadata requests you can use NetworkPolicies as an alternative https://zero-to-jupyterhub.readthedocs.io/en/latest/administrator/security.html#kubernetes-network-policies
thanks It did work. At least this part, then I have other connectivity issues that avoids to deploy the notebook.
Anyway, what does cloud metadata?
Cloud metadata is a service provided by many public cloud providers. For example see