× Building wheel for pyzmq (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [293 lines of output]
/tmp/pip-build-env-qs2v5keb/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: ‘cffi_modules’
warnings.warn(msg)
running bdist_wheel
In file included from /home/jovyan/anaconda3/envs/enterprise-gateway-dev/include/python3.12/dictobject.h:90,
from /home/jovyan/anaconda3/envs/enterprise-gateway-dev/include/python3.12/Python.h:61,
from zmq/backend/cython/_device.c:25:
/home/jovyan/anaconda3/envs/enterprise-gateway-dev/include/python3.12/cpython/dictobject.h:22:34: note: declared here
22 | Py_DEPRECATED(3.12) uint64_t ma_version_tag;
| ^~~~~~~~~~~~~~
zmq/backend/cython/_device.c:5145:9: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
5145 | __PYX_PY_DICT_LOOKUP_IF_MODIFIED(
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/jovyan/anaconda3/envs/enterprise-gateway-dev/include/python3.12/dictobject.h:90,
from /home/jovyan/anaconda3/envs/enterprise-gateway-dev/include/python3.12/Python.h:61,
from zmq/backend/cython/_device.c:25:
/home/jovyan/anaconda3/envs/enterprise-gateway-dev/include/python3.12/cpython/dictobject.h:22:34: note: declared here
22 | Py_DEPRECATED(3.12) uint64_t ma_version_tag;
| ^~~~~~~~~~~~~~
zmq/backend/cython/_device.c: In function ‘__Pyx_PyInt_As_int’:
zmq/backend/cython/_device.c:5408:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
5408 | const digit* digits = ((PyLongObject*)x)->ob_digit;
| ^~
zmq/backend/cython/_device.c:5463:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
5463 | const digit* digits = ((PyLongObject*)x)->ob_digit;
| ^~
zmq/backend/cython/_device.c: In function ‘__Pyx_PyInt_As_long’:
zmq/backend/cython/_device.c:5680:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
5680 | const digit* digits = ((PyLongObject*)x)->ob_digit;
| ^~
zmq/backend/cython/_device.c:5735:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
5735 | const digit* digits = ((PyLongObject*)x)->ob_digit;
| ^~
zmq/backend/cython/_device.c: In function ‘__Pyx_PyIndex_AsSsize_t’:
zmq/backend/cython/_device.c:6086:45: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
6086 | const digit* digits = ((PyLongObject*)b)->ob_digit;
| ^~
********************************** ************************************************
error: command ‘/usr/bin/gcc’ failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pyzmq
ERROR: Could not build wheels for pyzmq, which is required to install pyproject.toml-based projects
failed
CondaEnvException: Pip failed
make: [Makefile:45: env] Error 1 (ignored)
conda env config vars set PYTHONPATH=/home/jovyan/enterprise_gateway --name enterprise-gateway-dev
more output would be helpful, including how things are being attempted to be installed. pyzmq does have wheels, so it seems like something might have an inappropriate upper bound on pyzmq in your environment. If there are outdated upper bounds, then it probably makes sense to pick a version of Python older than 3.12.
i have try with most recent version of anaconda starting from Anaconda3-2022.10-Linux-x86_64.sh which have python 3.9 to 3.11 version but getting same error on my ubuntu 22.04 box has python 3.10 but dont know where is this 3.12 version comming from while setting developer env for jupyter enterprise gateway v3.2.2 and its failing on #make env
very strange that make env would try to get pyzmq with pip, since it attempts to install it with conda, and pip should find it already satisfied.Reporting complete output would definitely help.
You can specify a Python version by adding - python=3.11 to requirements.yml
I manage workaround with Makefile commands and created conda venv and able to upgrade docker kernelspace images with spark version upgrade. thanks a lot for your support.
i am running spark remote kernel on jupyterhub but failling to read csv file on both executer and driver cluster node,i am able to add file but failing to read csv with below error.
pyspark 3.4.2 command from jupyterhub3.2.2 notebook
i can see temp files under below dir in both executer pods
#ll /var/data/spark-13acd4aa-2344-4b00-800a-daa4771b7d1f/spark-2c886ade-ac67-4fa2-ba29-e5e4844845fe
-rw-r--r-- 1 jovyan users 21429281 Feb 5 08:40 1831154171707122432750_cache
-rw-r--r-- 1 jovyan users 0 Feb 5 08:40 1831154171707122432750_lock
-rw-r--r-- 1 jovyan users 21889857 Feb 5 05:46 20226180011707111990941_cache
-rw-r--r-- 1 jovyan users 0 Feb 5 05:46 20226180011707111990941_lock
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/opt/spark-3.4.2-bin-hadoop3/work-dir/ZEEL__EQ__NSE__NSE__MINUTE.csv
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:304)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:244)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:332)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:208)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:291)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:291)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:287)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2328)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1019)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:405)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1018)
at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:193)
at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: Input path does not exist: file:/opt/spark-3.4.2-bin-hadoop3/work-dir/ZEEL__EQ__NSE__NSE__MINUTE.csv
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:278)
... 30 more
/opt/spark-3.4.2-bin-hadoop3/work-dir/ZEEL__EQ__NSE__NSE__MINUTE.csv this file is available on both executer on same path.