I would love to use pvcs as local storage for my executors. I have found Running Spark on Kubernetes - Spark 3.5.0 Documentation to describe exactly what I need.
However, I have run into the issue that executors cannot be created and fail with
Failure executing: POST at: https://kubernetes.default[...]/jovyan-[...]/persistentvolumeclaims. Message: Forbidden!Configured service account doesn't have access.
I see that the issue is caused because I startiexecutors when I create the SparkSession. Thus the one doing the request is the driver pod and this one is running with the clusterrole
However, that role is defined by https://github.com/jupyter-server/enterprise_gateway/blob/18cfa3e2458fa0732742917e20d150e3c9d7f03f/etc/kubernetes/helm/enterprise-gateway/templates/eg-clusterrole.yaml and this doesn’t seem templated. I suppose I could simply edit the helmchart and add the create/delete permissions for pvcs to the role.
But since this isn’t templated for the values.yaml, I have a strong feeling, that would not be the way I am supposed to address this, correct? What should I do instead?