Add additional port on singleuser pod as spark.driver.port

JupyterHub
1

I am running jupyterhub from zero-to-jupterhub helm. and using juputer stack image (jupyter/all-spark-notebook:spark-3.2.1)

when I launch spark job, the executor pods initiate property, but failed to communicate with the driver pod (the singleuser jupyter pod).

I believe the executor cannot talk to the driver on the spark.driver.port. But I cannot find the proper way to expose additional port beside the 8888 from the helm value file. some post suggest to use “extraPodConfig: {}” , but I cannot figure out the proper syntax.

my spark job as below:

import pyspark
conf = (SparkConf().setMaster(“k8s://https://kubernetes.default.svc”)
.set(“spark.kubernetes.container.image”, “apache/spark:v3.2.1”)
.set(“spark.driver.port”, “2222”)
.set(“spark.driver.blockManager.port”, “7777”)
.set(“spark.driver.host”, socket.gethostbyname(socket.gethostname()))
.set(“spark.driver.bindAddress”, “0.0.0.0”)
.set(“spark.kubernetes.namespace”, “jupyter”)
.set(“spark.kubernetes.authenticate.caCertFile”, “/var/run/secrets/kubernetes.io/serviceaccount/ca.crt”)
.set(“spark.kubernetes.authenticate.oauthTokenFile”,"/var/run/secrets/kubernetes.io/serviceaccount/token")
.set(“spark.kubernetes.authenticate.driver.serviceAccountName”, “spark”)
.set(“spark.executor.instances”, “3”)
.setAppName(‘pyspark-shell’)
)

2

ok, I solved my problem eventually by adding below value into the helm value.yaml.

singleuser:
  networkPolicy:
    allowedIngressPorts:
      - 2222
      - 7777
1 Like