Pyspark library is missing from jupyter/pyspark-notebook when running with jupyterhub/zero-to-jupyterhub-k8s

manics · March 22, 2021, 11:56am

Initial findings:

The pyspark image adds a /usr/local/bin/before-notebook.d hook which sets PYTHONPATH:

jupyter/docker-stacks/blob/3395de4db93ada5e8eb0fd7da70a1a32ce4b8381/pyspark-notebook/Dockerfile#L47-L50


RUN ln -s "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}" spark && \
    # Add a link in the before_notebook hook in order to source automatically PYTHONPATH
    mkdir -p /usr/local/bin/before-notebook.d && \
    ln -s "${SPARK_HOME}/sbin/spark-config.sh" /usr/local/bin/before-notebook.d/spark-config.sh

This is called in start.sh:

github.com

jupyter/docker-stacks/blob/3395de4db93ada5e8eb0fd7da70a1a32ce4b8381/base-notebook/start.sh#L143-L146


# Execute the command
run-hooks /usr/local/bin/before-notebook.d
echo "Executing the command: ${cmd[@]}"
exec "${cmd[@]}"

But by default Z2JH calls jupyterhub-singleuser directly:

github.com

jupyterhub/zero-to-jupyterhub-k8s/blob/4fc7949c1fcddbab24a43785226a04ae50e58772/jupyterhub/values.yaml#L377


  startTimeout: 300
  cpu:
    limit:
    guarantee:
  memory:
    limit:
    guarantee: 1G
  extraResource:
    limits: {}
    guarantees: {}
  cmd: jupyterhub-singleuser
  defaultUrl:
  extraPodConfig: {}
  profileList: []

# scheduling relates to the user-scheduler pods and user-placeholder pods.
scheduling:
  userScheduler:
    enabled: true
    replicas: 2
    logLevel: 4

Topic		Replies	Views
How to set PYSPARK_PYTHON/PYSPARK_DRIVER_PYTHON Zero to JupyterHub on Kubernetes jupyterhub , how-to , help-wanted	1	3010	April 13, 2022
Jupyterhub spark spark-config.sh /usr/local/bin/before-notebook.d JupyterHub	3	1057	April 27, 2021
Executing PySpark code in a Jupyter Notebook using Z2JK where the user configures the Spark Session with the spark deployment mode set to "client" with the Spark executors running in their own dedicated Kubernetes cluster Zero to JupyterHub on Kubernetes jupyterlab , jupyterhub	1	778	June 16, 2022
Spark job submitted to yarn (external) from remote jupyter notebook pod Zero to JupyterHub on Kubernetes help-wanted	0	247	March 6, 2024
Single user server as driver node in spark cluster k8s Zero to JupyterHub on Kubernetes jupyterlab , jupyterhub , how-to , help-wanted	1	873	April 13, 2022

Pyspark library is missing from jupyter/pyspark-notebook when running with jupyterhub/zero-to-jupyterhub-k8s

Related Topics