Notebook Server can not access the k8s API from a pod

JupyterHub Zero to JupyterHub on Kubernetes
#1

Description

Hi, guys.

Follow the k8s official doc. The recommended way to authenticate to the apiserver from a pod is with a service account credential.
But in the notebook server’s pod, I can’t access the API server by kubectl get pod

Here are the steps:

kubectl exec -it jupyter-my-username -- /bin/bash
kubectl get pod
# The connection to the server localhost:8080 was refused - did you specify the right host or port?

But when I apply a pod with the same image and run:

kubectl exec -it pod-name -- /bin/bash
kubectl get pod
# Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:jupyterhub:default" cannot list resource "pods" in API group "" in the namespace "jupyterhub"

Some extra information:

# inside nb server pod
cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
# cat: /var/run/secrets/kubernetes.io/serviceaccount/namespace: No such file or directory

 # but inside other's pod
cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
$ jupyterhub (jupyterhub is the name of namespace)

Expected behaviour

I want to know if there are some ways I can access to the K8S API service inside the notebook server pod ?
Thanks very much.

#2

Thanks for asking this in the forum rather than creating a GitHub issue @lidongze0629!

I believe you have two separate problems.

Problem 1 - k8s RBAC permissions

A k8s service account may not have the desired permission. When you get the response cannot list resource "pods" in API group etc, you actually managed to speak with the k8s api-server, but you lacked permissions.

To get permissions, your k8s service account needs to be associated with a (Cluster)Role through a (Cluster)RoleBinding. For more details, see the k8s docs about RBAC.

Problem 2 - KubeSpawner not mounting the k8s Service Account’s associated token

KubeSpawner that is used by the JupyterHub Helm chart to spawn users for JupyterHub running in the hub pod is by default not mounting the k8s Service account tokens to be made available in the container. This is a deviation from what is common.

If you want for that to happen, you can explicitly set the service account you want to use and KubeSpawner will mount the k8s Service account’s token to be available in the container.

See the KubeSpawner documentation for service_account.

There is also this PR currently in KubeSpawner to expose the automount_service_account_token configuration and so, but I would suggest to set the service account using hub.config.KubeSpawner.service_account (0.11.1 helm chart needed) or through hub.extraConfig (see z2jh.jupyter.org configuration reference).

1 Like
#3

Thanks for your answer, It works when I set hub.config.KubeSpawner.service_account to default.
As you said, I need to make service_account associate with a (Cluster)Role. Maybe I can do it inside the pre_spawn_hook, is there a better way.

#4

Yes I’d say you should open up a text editor and write a .yaml file defining a Role and a RoleBinding, for inspiration on how those look, you can model them after this. Note that the .yaml file i linked is a Helm template, so everything in {{ }} must be replaced for you that isn’t going to use Helm to render the template into a valid k8s resource.

When you have defined your Role and RoleBinding, you do kubectl apply -f my-rbac-stuff.yaml -n mynamespace.

1 Like