Spark Client Mode Integration

JupyterHub Zero to JupyterHub on Kubernetes
1

Hey Everyone,

I am building out a jupyterhub build on k8’s and i’m running into an issue with spark… I am not sure how to connect to my existing spark / yarn managed hadoop cluster to run spark applications in client_mode. I have played around with trying to add ports and map them to my single-user pods but to no avail. Is there any information that could be shared by those who have integrated spark with jupyterhub?

Do i have to create a service to take a port, map it to a target port and forward that to my user pod IP’s?

Thanks,
Jake

2

I’m using spark client mode in jupyterhub on k8s. My trick is make single-user pod config: "hostNetwork": True, and replace kubespawner’s port to random.

4

Oh wow okay! Is there a way you could post sample configs of that?