PySpark - Read file from Azure blob: error - org.apache.hadoop.fs.azure.NativeAzureFileSystem not found

JupyterHub Zero to JupyterHub on Kubernetes
1

Hi,
Not sure this is correct forum, if so please help where I can post it.
I am trying to read file from Azure blob storage and getting error:

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found

From the web I understand that additional libraries need to be installed, so i tried:

pip install azure-mgmt-resource 
pip install azure-mgmt-datalake-store 
pip install azure-datalake-store

but the problem wasn’t solved.
Some other posts only describe how to include jars in script execution, but I guess it’s not relevant for for jupyter notebook on k8s.

Thank you in advance,
Yan

2

Setting java jars in Spark config soled the problem!

# Setup the Configuration
conf = pyspark.SparkConf()
conf.set("spark.jars.packages", "org.apache.hadoop:hadoop-azure:3.3.4,com.microsoft.azure:azure-storage:8.6.6")
conf.set("fs.azure.account.key.<STORAGE_ACCOUNT>.blob.core.windows.net", "<TOKEN>")
spark_context = SparkSession.builder.config(conf=conf).getOrCreate()

df = spark_context.read.parquet("wasbs://<CONTAINER>@<STORAGE_ACCOUNT>.blob.core.windows.net/<FILE>")
3

did you install anything else as well?
I am still getting the same error

4

Did you get it to work ?? I am still facing this issue ?