EMR Jupyter Notebook: PySpark Imports Work in Shell, Not in Notebook- Issue is importing custom files

[Harish_chenna]

Issue: PySpark works in the first cells (likely SparkSession creation) but throws import errors when using my Python files in later cells.

Environment: AWS EMR ( Amazon EMR version
emr-6.4.0
Installed applications
JupyterEnterpriseGateway 2.1.0, JupyterHub 1.4.1, Spark 3.1.2)

Question: Why the import issue with custom files? Best practices for importing Python files in EMR notebooks?

[minrk]

check sys.path in your Pyspark_logic.py script and outside, and see what’s different. Also check pyspark.__file__.