Pyspark library is missing from jupyter/pyspark-notebook when running with jupyterhub/zero-to-jupyterhub-k8s

Thorsten_Hersam · March 22, 2021, 10:32am

pyspark library is not available in the pyspark-notebook when running on jupyterhub.

When running jupyter/pyspark-notebook locally, I can import pyspark as I would expect:

from pyspark import SparkConf, SparkContext

→ no errors

When I run jupyter/pyspark-notebook with my jupyterhub/zero-to-jupyterhub-k8s helm installation, the same code returns:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-ee46be618299> in <module>
----> 1 from pyspark import SparkConf, SparkContext

ModuleNotFoundError: No module named 'pyspark'

Running python -m pip list --format columns. Does return a lot of installed modules however not the pyspark module.

Having read https://stackoverflow.com/a/31541051/5930295 I tried to update the sys.path because I found that pyspark is installed in the container under the following locations. Running:

import sys
sys.path.insert(0, '/usr/local/spark/python')
sys.path.insert(0, '/usr/local/spark/python/lib/py4j-0.10.9-src.zip')

from pyspark import SparkConf, SparkContext

→ No errors

Further information

I thought the issue must be with the image. Therefore I originally opened a ticket with jupyter/docker-stack (pyspark library is missing from jupyter/pyspark-notebook when running with jupyterhub/zero-to-jupyterhub-k8s · Issue #1255 · jupyter/docker-stacks · GitHub).

Your personal set up

jupyterhub/zero-to-jupyterhub-k8s Helm Chart

jupyter/pyspark-notebook:9fe5186aba96 as default image

singleuser:
  defaultUrl: "/lab"
  image:
    name: jupyter/pyspark-notebook
    tag: 9fe5186aba96

Not Working Environment

env | sort outputs:

APACHE_AIRFLOW_PORT=tcp://10.43.46.18:8080
APACHE_AIRFLOW_PORT_8080_TCP=tcp://10.43.46.18:8080
APACHE_AIRFLOW_PORT_8080_TCP_ADDR=10.43.46.18
APACHE_AIRFLOW_PORT_8080_TCP_PORT=8080
APACHE_AIRFLOW_PORT_8080_TCP_PROTO=tcp
APACHE_AIRFLOW_POSTGRESQL_PORT=tcp://10.43.64.222:5432
APACHE_AIRFLOW_POSTGRESQL_PORT_5432_TCP=tcp://10.43.64.222:5432
APACHE_AIRFLOW_POSTGRESQL_PORT_5432_TCP_ADDR=10.43.64.222
APACHE_AIRFLOW_POSTGRESQL_PORT_5432_TCP_PORT=5432
APACHE_AIRFLOW_POSTGRESQL_PORT_5432_TCP_PROTO=tcp
APACHE_AIRFLOW_POSTGRESQL_SERVICE_HOST=10.43.64.222
APACHE_AIRFLOW_POSTGRESQL_SERVICE_PORT=5432
APACHE_AIRFLOW_POSTGRESQL_SERVICE_PORT_TCP_POSTGRESQL=5432
APACHE_AIRFLOW_REDIS_MASTER_PORT=tcp://10.43.14.5:6379
APACHE_AIRFLOW_REDIS_MASTER_PORT_6379_TCP=tcp://10.43.14.5:6379
APACHE_AIRFLOW_REDIS_MASTER_PORT_6379_TCP_ADDR=10.43.14.5
APACHE_AIRFLOW_REDIS_MASTER_PORT_6379_TCP_PORT=6379
APACHE_AIRFLOW_REDIS_MASTER_PORT_6379_TCP_PROTO=tcp
APACHE_AIRFLOW_REDIS_MASTER_SERVICE_HOST=10.43.14.5
APACHE_AIRFLOW_REDIS_MASTER_SERVICE_PORT=6379
APACHE_AIRFLOW_REDIS_MASTER_SERVICE_PORT_TCP_REDIS=6379
APACHE_AIRFLOW_SERVICE_HOST=10.43.46.18
APACHE_AIRFLOW_SERVICE_PORT=8080
APACHE_AIRFLOW_SERVICE_PORT_HTTP=8080
APACHE_SPARK_MASTER_SVC_PORT=tcp://10.43.67.11:7077
APACHE_SPARK_MASTER_SVC_PORT_7077_TCP=tcp://10.43.67.11:7077
APACHE_SPARK_MASTER_SVC_PORT_7077_TCP_ADDR=10.43.67.11
APACHE_SPARK_MASTER_SVC_PORT_7077_TCP_PORT=7077
APACHE_SPARK_MASTER_SVC_PORT_7077_TCP_PROTO=tcp
APACHE_SPARK_MASTER_SVC_PORT_80_TCP=tcp://10.43.67.11:80
APACHE_SPARK_MASTER_SVC_PORT_80_TCP_ADDR=10.43.67.11
APACHE_SPARK_MASTER_SVC_PORT_80_TCP_PORT=80
APACHE_SPARK_MASTER_SVC_PORT_80_TCP_PROTO=tcp
APACHE_SPARK_MASTER_SVC_SERVICE_HOST=10.43.67.11
APACHE_SPARK_MASTER_SVC_SERVICE_PORT=7077
APACHE_SPARK_MASTER_SVC_SERVICE_PORT_CLUSTER=7077
APACHE_SPARK_MASTER_SVC_SERVICE_PORT_HTTP=80
APACHE_SPARK_VERSION=3.1.1
CLICOLOR=1
CONDA_DIR=/opt/conda
CONDA_VERSION=4.9.2
DEBIAN_FRONTEND=noninteractive
GIT_PAGER=cat
HADOOP_VERSION=3.2
HOME=/home/jovyan
HOSTNAME=jupyter-hersam
HUB_PORT=tcp://10.43.0.141:8081
HUB_PORT_8081_TCP=tcp://10.43.0.141:8081
HUB_PORT_8081_TCP_ADDR=10.43.0.141
HUB_PORT_8081_TCP_PORT=8081
HUB_PORT_8081_TCP_PROTO=tcp
HUB_SERVICE_HOST=10.43.0.141
HUB_SERVICE_PORT=8081
JPY_API_TOKEN=c5353ea5222b413c85a1e08306ebfbb3
JPY_PARENT_PID=7
JUPYTERHUB_ACTIVITY_URL=http://hub:8081/hub/api/users/hersam/activity
JUPYTERHUB_ADMIN_ACCESS=1
JUPYTERHUB_API_TOKEN=c5353ea5222b413c85a1e08306ebfbb3
JUPYTERHUB_API_URL=http://hub:8081/hub/api
JUPYTERHUB_BASE_URL=/
JUPYTERHUB_CLIENT_ID=jupyterhub-user-hersam
JUPYTERHUB_HOST=
JUPYTERHUB_OAUTH_CALLBACK_URL=/user/hersam/oauth_callback
JUPYTERHUB_SERVER_NAME=
JUPYTERHUB_SERVICE_PREFIX=/user/hersam/
JUPYTERHUB_USER=hersam
JUPYTER_IMAGE=jupyter/pyspark-notebook:latest
JUPYTER_IMAGE_SPEC=jupyter/pyspark-notebook:latest
KUBERNETES_PORT=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=10.43.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_ALL=en_US.UTF-8
MEM_GUARANTEE=1073741824
MINIFORGE_VERSION=4.9.2-7
MPLBACKEND=module://ipykernel.pylab.backend_inline
NB_GID=100
NB_UID=1000
NB_USER=jovyan
PAGER=cat
PATH=/opt/conda/bin:/usr/local/spark/python:/usr/local/spark/python/lib/py4j-0.10.9-src.zip:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/spark/bin
PROXY_API_PORT=tcp://10.43.170.224:8001
PROXY_API_PORT_8001_TCP=tcp://10.43.170.224:8001
PROXY_API_PORT_8001_TCP_ADDR=10.43.170.224
PROXY_API_PORT_8001_TCP_PORT=8001
PROXY_API_PORT_8001_TCP_PROTO=tcp
PROXY_API_SERVICE_HOST=10.43.170.224
PROXY_API_SERVICE_PORT=8001
PROXY_PUBLIC_PORT=tcp://10.43.159.248:80
PROXY_PUBLIC_PORT_80_TCP=tcp://10.43.159.248:80
PROXY_PUBLIC_PORT_80_TCP_ADDR=10.43.159.248
PROXY_PUBLIC_PORT_80_TCP_PORT=80
PROXY_PUBLIC_PORT_80_TCP_PROTO=tcp
PROXY_PUBLIC_SERVICE_HOST=10.43.159.248
PROXY_PUBLIC_SERVICE_PORT=80
PROXY_PUBLIC_SERVICE_PORT_HTTP=80
PWD=/home/jovyan
SHELL=/bin/bash
SPARK_HOME=/usr/local/spark
SPARK_NODE_PORT_PORT=tcp://10.43.21.88:7077
SPARK_NODE_PORT_PORT_7077_TCP=tcp://10.43.21.88:7077
SPARK_NODE_PORT_PORT_7077_TCP_ADDR=10.43.21.88
SPARK_NODE_PORT_PORT_7077_TCP_PORT=7077
SPARK_NODE_PORT_PORT_7077_TCP_PROTO=tcp
SPARK_NODE_PORT_PORT_80_TCP=tcp://10.43.21.88:80
SPARK_NODE_PORT_PORT_80_TCP_ADDR=10.43.21.88
SPARK_NODE_PORT_PORT_80_TCP_PORT=80
SPARK_NODE_PORT_PORT_80_TCP_PROTO=tcp
SPARK_NODE_PORT_SERVICE_HOST=10.43.21.88
SPARK_NODE_PORT_SERVICE_PORT=7077
SPARK_NODE_PORT_SERVICE_PORT_CLUSTER=7077
SPARK_NODE_PORT_SERVICE_PORT_HTTP=80
SPARK_OPTS=--driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info
TERM=xterm-color
XDG_CACHE_HOME=/home/jovyan/.cache/

import os
for k, v in sorted(os.environ.items()):
    print(f'{k}={v}')

outputs:

APACHE_AIRFLOW_PORT=tcp://10.43.46.18:8080
APACHE_AIRFLOW_PORT_8080_TCP=tcp://10.43.46.18:8080
APACHE_AIRFLOW_PORT_8080_TCP_ADDR=10.43.46.18
APACHE_AIRFLOW_PORT_8080_TCP_PORT=8080
APACHE_AIRFLOW_PORT_8080_TCP_PROTO=tcp
APACHE_AIRFLOW_POSTGRESQL_PORT=tcp://10.43.64.222:5432
APACHE_AIRFLOW_POSTGRESQL_PORT_5432_TCP=tcp://10.43.64.222:5432
APACHE_AIRFLOW_POSTGRESQL_PORT_5432_TCP_ADDR=10.43.64.222
APACHE_AIRFLOW_POSTGRESQL_PORT_5432_TCP_PORT=5432
APACHE_AIRFLOW_POSTGRESQL_PORT_5432_TCP_PROTO=tcp
APACHE_AIRFLOW_POSTGRESQL_SERVICE_HOST=10.43.64.222
APACHE_AIRFLOW_POSTGRESQL_SERVICE_PORT=5432
APACHE_AIRFLOW_POSTGRESQL_SERVICE_PORT_TCP_POSTGRESQL=5432
APACHE_AIRFLOW_REDIS_MASTER_PORT=tcp://10.43.14.5:6379
APACHE_AIRFLOW_REDIS_MASTER_PORT_6379_TCP=tcp://10.43.14.5:6379
APACHE_AIRFLOW_REDIS_MASTER_PORT_6379_TCP_ADDR=10.43.14.5
APACHE_AIRFLOW_REDIS_MASTER_PORT_6379_TCP_PORT=6379
APACHE_AIRFLOW_REDIS_MASTER_PORT_6379_TCP_PROTO=tcp
APACHE_AIRFLOW_REDIS_MASTER_SERVICE_HOST=10.43.14.5
APACHE_AIRFLOW_REDIS_MASTER_SERVICE_PORT=6379
APACHE_AIRFLOW_REDIS_MASTER_SERVICE_PORT_TCP_REDIS=6379
APACHE_AIRFLOW_SERVICE_HOST=10.43.46.18
APACHE_AIRFLOW_SERVICE_PORT=8080
APACHE_AIRFLOW_SERVICE_PORT_HTTP=8080
APACHE_SPARK_MASTER_SVC_PORT=tcp://10.43.67.11:7077
APACHE_SPARK_MASTER_SVC_PORT_7077_TCP=tcp://10.43.67.11:7077
APACHE_SPARK_MASTER_SVC_PORT_7077_TCP_ADDR=10.43.67.11
APACHE_SPARK_MASTER_SVC_PORT_7077_TCP_PORT=7077
APACHE_SPARK_MASTER_SVC_PORT_7077_TCP_PROTO=tcp
APACHE_SPARK_MASTER_SVC_PORT_80_TCP=tcp://10.43.67.11:80
APACHE_SPARK_MASTER_SVC_PORT_80_TCP_ADDR=10.43.67.11
APACHE_SPARK_MASTER_SVC_PORT_80_TCP_PORT=80
APACHE_SPARK_MASTER_SVC_PORT_80_TCP_PROTO=tcp
APACHE_SPARK_MASTER_SVC_SERVICE_HOST=10.43.67.11
APACHE_SPARK_MASTER_SVC_SERVICE_PORT=7077
APACHE_SPARK_MASTER_SVC_SERVICE_PORT_CLUSTER=7077
APACHE_SPARK_MASTER_SVC_SERVICE_PORT_HTTP=80
APACHE_SPARK_VERSION=3.1.1
CLICOLOR=1
CONDA_DIR=/opt/conda
CONDA_VERSION=4.9.2
DEBIAN_FRONTEND=noninteractive
GIT_PAGER=cat
HADOOP_VERSION=3.2
HOME=/home/jovyan
HOSTNAME=jupyter-hersam
HUB_PORT=tcp://10.43.0.141:8081
HUB_PORT_8081_TCP=tcp://10.43.0.141:8081
HUB_PORT_8081_TCP_ADDR=10.43.0.141
HUB_PORT_8081_TCP_PORT=8081
HUB_PORT_8081_TCP_PROTO=tcp
HUB_SERVICE_HOST=10.43.0.141
HUB_SERVICE_PORT=8081
JPY_API_TOKEN=c5353ea5222b413c85a1e08306ebfbb3
JPY_PARENT_PID=7
JUPYTERHUB_ACTIVITY_URL=http://hub:8081/hub/api/users/hersam/activity
JUPYTERHUB_ADMIN_ACCESS=1
JUPYTERHUB_API_TOKEN=c5353ea5222b413c85a1e08306ebfbb3
JUPYTERHUB_API_URL=http://hub:8081/hub/api
JUPYTERHUB_BASE_URL=/
JUPYTERHUB_CLIENT_ID=jupyterhub-user-hersam
JUPYTERHUB_HOST=
JUPYTERHUB_OAUTH_CALLBACK_URL=/user/hersam/oauth_callback
JUPYTERHUB_SERVER_NAME=
JUPYTERHUB_SERVICE_PREFIX=/user/hersam/
JUPYTERHUB_USER=hersam
JUPYTER_IMAGE=jupyter/pyspark-notebook:latest
JUPYTER_IMAGE_SPEC=jupyter/pyspark-notebook:latest
KUBERNETES_PORT=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=10.43.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_ALL=en_US.UTF-8
MEM_GUARANTEE=1073741824
MINIFORGE_VERSION=4.9.2-7
MPLBACKEND=module://ipykernel.pylab.backend_inline
NB_GID=100
NB_UID=1000
NB_USER=jovyan
PAGER=cat
PATH=/opt/conda/bin:/usr/local/spark/python:/usr/local/spark/python/lib/py4j-0.10.9-src.zip:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/spark/bin
PROXY_API_PORT=tcp://10.43.170.224:8001
PROXY_API_PORT_8001_TCP=tcp://10.43.170.224:8001
PROXY_API_PORT_8001_TCP_ADDR=10.43.170.224
PROXY_API_PORT_8001_TCP_PORT=8001
PROXY_API_PORT_8001_TCP_PROTO=tcp
PROXY_API_SERVICE_HOST=10.43.170.224
PROXY_API_SERVICE_PORT=8001
PROXY_PUBLIC_PORT=tcp://10.43.159.248:80
PROXY_PUBLIC_PORT_80_TCP=tcp://10.43.159.248:80
PROXY_PUBLIC_PORT_80_TCP_ADDR=10.43.159.248
PROXY_PUBLIC_PORT_80_TCP_PORT=80
PROXY_PUBLIC_PORT_80_TCP_PROTO=tcp
PROXY_PUBLIC_SERVICE_HOST=10.43.159.248
PROXY_PUBLIC_SERVICE_PORT=80
PROXY_PUBLIC_SERVICE_PORT_HTTP=80
PWD=/home/jovyan
SHELL=/bin/bash
SPARK_HOME=/usr/local/spark
SPARK_NODE_PORT_PORT=tcp://10.43.21.88:7077
SPARK_NODE_PORT_PORT_7077_TCP=tcp://10.43.21.88:7077
SPARK_NODE_PORT_PORT_7077_TCP_ADDR=10.43.21.88
SPARK_NODE_PORT_PORT_7077_TCP_PORT=7077
SPARK_NODE_PORT_PORT_7077_TCP_PROTO=tcp
SPARK_NODE_PORT_PORT_80_TCP=tcp://10.43.21.88:80
SPARK_NODE_PORT_PORT_80_TCP_ADDR=10.43.21.88
SPARK_NODE_PORT_PORT_80_TCP_PORT=80
SPARK_NODE_PORT_PORT_80_TCP_PROTO=tcp
SPARK_NODE_PORT_SERVICE_HOST=10.43.21.88
SPARK_NODE_PORT_SERVICE_PORT=7077
SPARK_NODE_PORT_SERVICE_PORT_CLUSTER=7077
SPARK_NODE_PORT_SERVICE_PORT_HTTP=80
SPARK_OPTS=--driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info
TERM=xterm-color
XDG_CACHE_HOME=/home/jovyan/.cache/

Working Environment (running container locally with docker)

env | sort outputs:

APACHE_SPARK_VERSION=3.1.1
CONDA_DIR=/opt/conda
CONDA_VERSION=4.9.2
DEBIAN_FRONTEND=noninteractive
HADOOP_VERSION=3.2
HOME=/home/jovyan
HOSTNAME=2a600aae602f
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_ALL=en_US.UTF-8
MINIFORGE_VERSION=4.9.2-7
NB_GID=100
NB_UID=1000
NB_USER=jovyan
PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/spark/bin
SHELL=/bin/bash
SPARK_HOME=/usr/local/spark
SPARK_OPTS=--driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info
TERM=xterm
XDG_CACHE_HOME=/home/jovyan/.cache/

import os
for k, v in sorted(os.environ.items()):
    print(f'{k}={v}')

outputs:

APACHE_SPARK_VERSION=3.1.1
CLICOLOR=1
CONDA_DIR=/opt/conda
CONDA_VERSION=4.9.2
DEBIAN_FRONTEND=noninteractive
GIT_PAGER=cat
HADOOP_VERSION=3.2
HOME=/home/jovyan
HOSTNAME=2a600aae602f
JPY_PARENT_PID=7
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_ALL=en_US.UTF-8
MINIFORGE_VERSION=4.9.2-7
MPLBACKEND=module://ipykernel.pylab.backend_inline
NB_GID=100
NB_UID=1000
NB_USER=jovyan
PAGER=cat
PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/spark/bin
PWD=/home/jovyan
PYSPARK_PYTHONPATH_SET=1
PYTHONPATH=/usr/local/spark/python/lib/py4j-0.10.9-src.zip:/usr/local/spark/python:
SHELL=/bin/bash
SHLVL=0
SPARK_CONF_DIR=/usr/local/spark/conf
SPARK_HOME=/usr/local/spark
SPARK_OPTS=--driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info
TERM=xterm-color
XDG_CACHE_HOME=/home/jovyan/.cache/

manics · March 22, 2021, 11:56am

Initial findings:

The pyspark image adds a /usr/local/bin/before-notebook.d hook which sets PYTHONPATH:

github.com

jupyter/docker-stacks/blob/3395de4db93ada5e8eb0fd7da70a1a32ce4b8381/pyspark-notebook/Dockerfile#L47-L50


RUN ln -s "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}" spark && \
    # Add a link in the before_notebook hook in order to source automatically PYTHONPATH
    mkdir -p /usr/local/bin/before-notebook.d && \
    ln -s "${SPARK_HOME}/sbin/spark-config.sh" /usr/local/bin/before-notebook.d/spark-config.sh

This is called in start.sh:

github.com

jupyter/docker-stacks/blob/3395de4db93ada5e8eb0fd7da70a1a32ce4b8381/base-notebook/start.sh#L143-L146


# Execute the command
run-hooks /usr/local/bin/before-notebook.d
echo "Executing the command: ${cmd[@]}"
exec "${cmd[@]}"

But by default Z2JH calls jupyterhub-singleuser directly:

github.com

jupyterhub/zero-to-jupyterhub-k8s/blob/4fc7949c1fcddbab24a43785226a04ae50e58772/jupyterhub/values.yaml#L377


  startTimeout: 300
  cpu:
    limit:
    guarantee:
  memory:
    limit:
    guarantee: 1G
  extraResource:
    limits: {}
    guarantees: {}
  cmd: jupyterhub-singleuser
  defaultUrl:
  extraPodConfig: {}
  profileList: []

# scheduling relates to the user-scheduler pods and user-placeholder pods.
scheduling:
  userScheduler:
    enabled: true
    replicas: 2
    logLevel: 4

Thorsten_Hersam · March 22, 2021, 12:52pm

Awesome, thanks!

Using this config.yaml it works:

singleuser:
  defaultUrl: "/lab"
  image:
    name: jupyter/pyspark-notebook
    tag: latest
  cmd: ["/usr/local/bin/start.sh", "jupyterhub-singleuser"]

manics · March 22, 2021, 1:23pm

Great that you figured it out. It looks like

singleuser:
  cmd:

i.e. use the default from the Docker image also works

github.com

jupyter/docker-stacks/blob/3395de4db93ada5e8eb0fd7da70a1a32ce4b8381/base-notebook/Dockerfile#L149-L150


ENTRYPOINT ["tini", "-g", "--"]
CMD ["start-notebook.sh"]

dokempf · November 3, 2021, 3:05pm

I am sorry to hook on this old topic, but I am facing the exact same problem on mybinder.org. When building and running my Docker container based on pyspark-notebook locally, everything is fine because within the container PYTHONPATH points to the correct location. However, when running on mybinder.org, PYTHONPATH is empty and pyspark is therefore not found. Manually tweaking sys.path works. In contrast to the solution proposed in this topic, I do not control the configuration though. Any ideas?

manics · November 12, 2021, 2:08pm

Since you’re using a custom Docker image you could try overriding your ENTRYPOINT.
I’ve opened an issue to see if the behaviour can be made more intuitive:

github.com/jupyter/docker-stacks

Move environment setup from `start-notebook.sh` to ENTRYPOINT instead of CMD

opened 01:18PM - 12 Nov 21 UTC

manics

type:Enhancement

**What docker images this feature is applicable to?** - `jupyter/base-noteboo…k` - Notebooks that use startup hooks to configure the environment **What changes do you propose?** Split `start-notebook.sh` or `start.sh` into a script that does the environment setup in an ENTRYPOINT, and a script that does the actual notebook startup in CMD. Originally suggested in https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/2138#issuecomment-828291826 **How does this change will affect users?** `start-notebook.sh` calls `start.sh` which handles a lot of setup in the Jupyter environment, including: - Running setup hooks such as for setting environment variables https://github.com/jupyter/docker-stacks/blob/8dfdbfd3a3370ec138a7b85ae42422de8da3ca1b/base-notebook/start.sh#L44 - Customising user names and IDs when started as root https://github.com/jupyter/docker-stacks/blob/8dfdbfd3a3370ec138a7b85ae42422de8da3ca1b/base-notebook/start.sh#L46-L51 Since `start-notebook.sh` is set as the CMD if someone passes any arguments when running the Docker container all this setup is ignored. For example `docker run -e NB_UID=12345 -u 0 jupyter/base-notebook jupyter-lab --debug` should change the UID from the default `1000` to `12345`, but since the startup scripts aren't run this leads to an error (can't be run as root). Instead you must run `docker run -e NB_UID=12345 -u 0 jupyter/base-notebook start.sh jupyter-lab --debug` A concrete example of where this is a problem for users is in the pyspark notebook- it isn't obvious to a user that the pyspark environment is setup by a startup script rather than being baked into the Dockerfile. - https://github.com/jupyter/docker-stacks/blob/8dfdbfd3a3370ec138a7b85ae42422de8da3ca1b/pyspark-notebook/Dockerfile#L48 - https://discourse.jupyter.org/t/pyspark-library-is-missing-from-jupyter-pyspark-notebook-when-running-with-jupyterhub-zero-to-jupyterhub-k8s/8450 Note we're working around this in JupyterHub 2.0 and Z2JH 2.0 with a breaking change: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/2449 Instead of specifying `jupyterhub-singleuser` as the CMD when running the image we'll use the image's default CMD, but I think this change is still generally helpful.

Topic		Replies	Views
How to set PYSPARK_PYTHON/PYSPARK_DRIVER_PYTHON Zero to JupyterHub on Kubernetes jupyterhub , how-to , help-wanted	1	3211	April 13, 2022
Why is pyspark module installed on a browser launched notebook (python3) but not installed in python3 terminal console? JupyterHub docker	1	356	November 24, 2023
Jupyterhub spark spark-config.sh /usr/local/bin/before-notebook.d JupyterHub	3	1076	April 27, 2021
Integrate JupyterHub with PySpark which is running on docker container JupyterHub	1	276	November 13, 2023
Executing PySpark code in a Jupyter Notebook using Z2JK where the user configures the Spark Session with the spark deployment mode set to "client" with the Spark executors running in their own dedicated Kubernetes cluster Zero to JupyterHub on Kubernetes jupyterlab , jupyterhub	1	786	June 16, 2022

Pyspark library is missing from jupyter/pyspark-notebook when running with jupyterhub/zero-to-jupyterhub-k8s

Further information

Your personal set up

Not Working Environment

Working Environment (running container locally with docker)

Related topics