K8s service account kernel-controller and PVCs in EG

Kernels Enterprise Gateway
1

I would love to use pvcs as local storage for my executors. I have found Running Spark on Kubernetes - Spark 3.5.0 Documentation to describe exactly what I need.

However, I have run into the issue that executors cannot be created and fail with Failure executing: POST at: https://kubernetes.default[...]/jovyan-[...]/persistentvolumeclaims. Message: Forbidden!Configured service account doesn't have access.

I see that the issue is caused because I startiexecutors when I create the SparkSession. Thus the one doing the request is the driver pod and this one is running with the clusterrole kernel-controller.

However, that role is defined by https://github.com/jupyter-server/enterprise_gateway/blob/18cfa3e2458fa0732742917e20d150e3c9d7f03f/etc/kubernetes/helm/enterprise-gateway/templates/eg-clusterrole.yaml and this doesn’t seem templated. I suppose I could simply edit the helmchart and add the create/delete permissions for pvcs to the role.

But since this isn’t templated for the values.yaml, I have a strong feeling, that would not be the way I am supposed to address this, correct? What should I do instead?

1 Like
2

Hi @BBuchhold, thanks for the post. Yes, the general approach is that the helm charts serve as “guides” and that they would be customized for a given deployment. However, it would be awesome to have these more templated to enable better customization while maintaining the general content of the published charts.

If you’d be interested in contributing such a change, we’d welcome your contribution!

Thanks,
Kevin.

3

Thanks, I feared that my approach wasn’t a good practice and thus not supported. If that’s not the issue, I will try to make it work and then contribute such a PR. I don’t have experience with helm charts, but it looks reasonably straightforward at first glance.

1 Like