Hi. We have installed JupyterLab 4.3, and get the following error when trying to use sparkmagic:
[W 2025-04-04 10:28:26.891 ServerApp] sparkmagic | error adding extension (enabled: True): The module 'sparkmagic' could not be found (cannot import name 'display' from 'IPython.core.display' (/app/pyenv/versions/3.11.11/lib/python3.11/site-packages/IPython/core/display.py)). Are you sure the extension is installed?
Traceback (most recent call last):
File "/app/pyenv/versions/3.11.11/lib/python3.11/site-packages/jupyter_server/extension/manager.py", line 323, in add_extension
extpkg = ExtensionPackage(name=extension_name, enabled=enabled)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/pyenv/versions/3.11.11/lib/python3.11/site-packages/jupyter_server/extension/manager.py", line 187, in __init__
self._load_metadata()
File "/app/pyenv/versions/3.11.11/lib/python3.11/site-packages/jupyter_server/extension/manager.py", line 202, in _load_metadata
raise ExtensionModuleNotFound(msg) from None
jupyter_server.extension.utils.ExtensionModuleNotFound: The module 'sparkmagic' could not be found (cannot import name 'display' from 'IPython.core.display' (/app/pyenv/versions/3.11.11/lib/python3.11/site-packages/IPython/core/display.py)). Are you sure the extension is installed?
If we run import display from IPython.display in a notebook cell it succeeds, but the error is reproduced with import display from IPython.core.display. Is there an incompatibility with the version of IPython and sparkmagic in JupyterLab 4.3? IPython 9.0.2 is installed.
Update: We just installed the latest 8.X version of IPython (8.34) with JupyterLab 4.3 and it works. So there is a breakage with the IPython major version bump from 8 to 9 for spark magic.