Hi everyone,
I’ve been trying to deploy Jupyterhub on Kubernetes for a few days, everything seems to be working except for the Notebook service creation.
Running a kubectl get svc returns these services:
And, if I understood correctly, there should be a service exposing the newly created notebook as well.
Due to this, other containers can’t find this notebook by name, and this is causing an issue when trying to run anything against a Spark worker (throws a UnknownHostException error)
Any help would be greatly appreciated.
Thanks.
Welcome to the forum!
Jupyter singleuser/notebook servers won’t have a K8S service created for them, instead The Hub and JupyterHub proxy connect to the pods directly. Could you please post your full config (redacting any secrets)? It sounds like you’ve got quite a complicated setup involving spark.
Hi manics, thanks for the welcome and the reply.
I’m deploying Jupyter using the official Helm chart using these answers:
auth.admin.users[0]=[redacted]
hub.image.tag=0.9-dcde99a
ingress.enabled=true
ingress.hosts[0]=jupyterhub[redacted].com
proxy.chp.image.tag=latest
proxy.secretToken=[redacted]
singleuser.image.name=jupyter/all-spark-notebook
singleuser.image.tag=latest
scheduling.userScheduler.enabled=false
prePuller.hook.enabled=false
prePuller.continuous.enabled=false
Besides this, we have a Spark master and worker configured and running in the same namespace.
I’ve been trying to make this work for over a week now, and the workaround I found to test if it’s possible to make this work is adding the notebook hostname and IP address to the Spark worker /etc/hosts file.
When checking the Spark worker logs, I see that it tried to connect to, for example, jupyter-test by name, and throwing an UnknownHostException error after that.
