Pyspark library is missing from jupyter/pyspark-notebook when running with jupyterhub/zero-to-jupyterhub-k8s

manics · March 22, 2021, 11:56am

Initial findings:

The pyspark image adds a /usr/local/bin/before-notebook.d hook which sets PYTHONPATH:

jupyter/docker-stacks/blob/3395de4db93ada5e8eb0fd7da70a1a32ce4b8381/pyspark-notebook/Dockerfile#L47-L50


RUN ln -s "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}" spark && \
    # Add a link in the before_notebook hook in order to source automatically PYTHONPATH
    mkdir -p /usr/local/bin/before-notebook.d && \
    ln -s "${SPARK_HOME}/sbin/spark-config.sh" /usr/local/bin/before-notebook.d/spark-config.sh

This is called in start.sh:

github.com

jupyter/docker-stacks/blob/3395de4db93ada5e8eb0fd7da70a1a32ce4b8381/base-notebook/start.sh#L143-L146


# Execute the command
run-hooks /usr/local/bin/before-notebook.d
echo "Executing the command: ${cmd[@]}"
exec "${cmd[@]}"

But by default Z2JH calls jupyterhub-singleuser directly:

github.com

jupyterhub/zero-to-jupyterhub-k8s/blob/4fc7949c1fcddbab24a43785226a04ae50e58772/jupyterhub/values.yaml#L377


  startTimeout: 300
  cpu:
    limit:
    guarantee:
  memory:
    limit:
    guarantee: 1G
  extraResource:
    limits: {}
    guarantees: {}
  cmd: jupyterhub-singleuser
  defaultUrl:
  extraPodConfig: {}
  profileList: []

# scheduling relates to the user-scheduler pods and user-placeholder pods.
scheduling:
  userScheduler:
    enabled: true
    replicas: 2
    logLevel: 4

Topic		Replies	Views
How to set PYSPARK_PYTHON/PYSPARK_DRIVER_PYTHON Zero to JupyterHub on Kubernetes jupyterhub , how-to , help-wanted	1	3182	April 13, 2022
Why is pyspark module installed on a browser launched notebook (python3) but not installed in python3 terminal console? JupyterHub docker	1	354	November 24, 2023
Jupyterhub spark spark-config.sh /usr/local/bin/before-notebook.d JupyterHub	3	1074	April 27, 2021
Integrate JupyterHub with PySpark which is running on docker container JupyterHub	1	275	November 13, 2023
Executing PySpark code in a Jupyter Notebook using Z2JK where the user configures the Spark Session with the spark deployment mode set to "client" with the Spark executors running in their own dedicated Kubernetes cluster Zero to JupyterHub on Kubernetes jupyterlab , jupyterhub	1	785	June 16, 2022

Pyspark library is missing from jupyter/pyspark-notebook when running with jupyterhub/zero-to-jupyterhub-k8s

Related topics