Unable to see the newly created kernel in Jupyter Lab

JupyterLab
1

Hi team

I am trying to install a new kernel by creating a new conda env in EMR 7.9
conda create --name py311 python=3.11.13

I am also able to list the newly created kernels using the command

[root@ip-10-3-109-38 bin]# ./jupyter kernelspec list
Available kernels:
python3 /emr/notebook-env/share/jupyter/kernels/python3
py311 /usr/local/share/jupyter/kernels/py311
pysparkkernel /usr/local/share/jupyter/kernels/pysparkkernel
sparkkernel /usr/local/share/jupyter/kernels/sparkkernel
sparkrkernel /usr/local/share/jupyter/kernels/sparkrkernel

However, I am not able to see the new kernel (py311) in the Jupyter Notebook.
I also tried to restart the jupyter enterprise gateway service as a root user:

[root@ip-10-3-109-171 bin]# sudo systemctl restart jupyter_enterprise_gateway.service
[root@ip-10-3-109-171 bin]# sudo systemctl status jupyter_enterprise_gateway.service
● jupyter_enterprise_gateway.service - Jupyter Enterprise Gateway Service
Loaded: loaded (/etc/systemd/system/jupyter_enterprise_gateway.service; enabled; preset: disabled)
Active: active (running) since Mon 2025-09-01 11:41:24 AEST; 7s ago
Process: 277187 ExecStartPre=/usr/bin/sudo /mnt/notebook-env/bin/kill_kernels.sh (code=exited, status=1/FAILURE)
Main PID: 277206 (jupyter-enterpr)
Tasks: 1 (limit: 37692)
Memory: 48.0M
CPU: 673ms
CGroup: /system.slice/jupyter_enterprise_gateway.service
└─277206 /emr/notebook-env/bin/python3.9 /mnt/notebook-env/bin/jupyter-enterprisegateway --config /mnt/notebook-env/conf/jupyter_enterprise_gateway_config.py

Sep 01 11:41:21 ip-10-3-109-171.opstquantexa01.aws.test.au.internal.cba sudo[277187]: pam_unix(sudo:auth): conversation failed
Sep 01 11:41:21 ip-10-3-109-171.opstquantexa01.aws.test.au.internal.cba sudo[277187]: pam_unix(sudo:auth): auth could not identify password for [emr-notebook]
Sep 01 11:41:21 ip-10-3-109-171.opstquantexa01.aws.test.au.internal.cba sudo[277187]: pam_sss(sudo:auth): Request to sssd failed. Connection refused
Sep 01 11:41:24 ip-10-3-109-171.opstquantexa01.aws.test.au.internal.cba sudo[277187]: sudo: a password is required
Sep 01 11:41:24 ip-10-3-109-171.opstquantexa01.aws.test.au.internal.cba sudo[277187]: emr-notebook : user NOT in sudoers ; PWD=/ ; USER=root ; COMMAND=/mnt/notebook-env/bin>
Sep 01 11:41:24 ip-10-3-109-171.opstquantexa01.aws.test.au.internal.cba systemd[1]: Started jupyter_enterprise_gateway.service - Jupyter Enterprise Gateway Service.
Sep 01 11:41:24 ip-10-3-109-171.opstquantexa01.aws.test.au.internal.cba jupyter-enterprisegateway[277206]: [W 2025-09-01 11:41:24.673 EnterpriseGatewayApp] Config option `i>
Sep 01 11:41:24 ip-10-3-109-171.opstquantexa01.aws.test.au.internal.cba jupyter-enterprisegateway[277206]: [I 2025-09-01 11:41:24.674 EnterpriseGatewayApp] Initialized Allo>
Sep 01 11:41:24 ip-10-3-109-171.opstquantexa01.aws.test.au.internal.cba jupyter-enterprisegateway[277206]: [I 2025-09-01 11:41:24.680 EnterpriseGatewayApp] Jupyter Enterpri>
Sep 01 11:41:29 ip-10-3-109-171.opstquantexa01.aws.test.au.internal.cba jupyter-enterprisegateway[277206]: [I 250901 11:41:29 web:2348] 200 GET /api/kernels (127.0.0.1) 1.0>
[root@ip-10-3-109-171 bin]#

This the jupyter core packages version

[root@ip-10-3-109-38 bin]# cd /emr/notebook-env/envs/py311/bin
[root@ip-10-3-109-38 bin]#
[root@ip-10-3-109-38 bin]# pwd
/emr/notebook-env/envs/py311/bin
[root@ip-10-3-109-38 bin]# ./jupyter --version
Selected Jupyter core packages…
IPython : 8.37.0
ipykernel : 6.30.1
ipywidgets : 7.8.5
jupyter_client : 8.6.3
jupyter_core : 5.8.1
jupyter_server : 2.13.0
jupyterlab : not installed
nbclient : 0.5.13
nbconvert : not installed
nbformat : 5.10.4
notebook : 6.5.4
qtconsole : not installed
traitlets : 5.14.3

This is the list of all the packages that are installed in the new env

[root@ip-10-3-109-38 bin]# ./python3 -m pip list
Package Version

aiohappyeyeballs 2.6.1
aiohttp 3.12.15
aiosignal 1.4.0
anyio 4.10.0
argon2-cffi 25.1.0
argon2-cffi-bindings 25.1.0
arrow 1.3.0
asttokens 2.4.1
attrs 25.3.0
beautifulsoup4 4.13.5
bleach 6.2.0
bokeh 2.4.3
boto3 1.40.12
botocore 1.40.21
Bottleneck 1.5.0
cffi 1.17.1
click 7.1.2
cloudpickle 3.1.1
comm 0.2.3
config 0.5.1
contourpy 1.3.3
cycler 0.12.1
dask 2.30.0
dataprep 0.4.1
debugpy 1.8.16
decorator 5.2.1
defusedxml 0.7.1
entrypoints 0.4
executing 0.8.3
fastjsonschema 2.21.2
Flask 1.1.4
Flask-Cors 3.0.10
fonttools 4.59.2
fqdn 1.5.1
frozenlist 1.7.0
fsspec 2025.7.0
graphframes 0.6
idna 3.10
ipykernel 6.30.1
ipython 8.37.0
ipython-genutils 0.2.0
ipywidgets 7.8.5
isoduration 20.11.0
itsdangerous 1.1.0
jedi 0.19.2
Jinja2 2.11.3
jmespath 1.0.1
joblib 1.5.2
jsonpath-ng 1.7.0
jsonpointer 3.0.0
jsonschema 4.25.1
jsonschema-specifications 2025.4.1
jupyter_client 8.6.3
jupyter_core 5.8.1
jupyter-events 0.12.0
jupyter_server 2.13.0
jupyter_server_terminals 0.5.3
jupyterlab_pygments 0.3.0
jupyterlab_widgets 1.1.11
kiwisolver 1.4.9
lark 1.2.2
levenshtein 0.12.0
locket 1.0.0
loguru 0.6.0
lz4 4.4.4
MarkupSafe 3.0.2
matplotlib 3.10.5
matplotlib-inline 0.1.7
Metaphone 0.6
mistune 0.8.4
multidict 6.6.4
narwhals 2.2.0
nbclassic 1.3.1
nbclient 0.5.13
nbconvert 6.4.5
nbformat 5.10.4
nest-asyncio 1.6.0
networkx 3.5
nltk 3.6.3
nose 1.3.7
notebook 6.5.4
notebook_shim 0.2.4
numpy 1.26.4
overrides 7.7.0
packaging 25.0
pandas 1.5.3
pandocfilters 1.5.1
parso 0.8.5
partd 1.4.2
patsy 1.0.1
pexpect 4.9.0
pillow 11.3.0
pip 25.2
platformdirs 4.4.0
plotly 6.3.0
ply 3.11
probableparsing 0.0.1
prometheus_client 0.22.1
prompt_toolkit 3.0.52
propcache 0.3.2
protobuf 3.20.3
psutil 7.0.0
ptyprocess 0.6.0
pure_eval 0.2.3
py4j 0.10.9.7
pyarrow 21.0.0
PyCBA 0.5.2
pycparser 2.22
pydantic 1.10.22
Pygments 2.19.2
pyhocon 0.3.61
pyparsing 3.2.3
pyspark-dist-explore 0.1.8
python-crfsuite 0.9.11
python-dateutil 2.9.0.post0
python-json-logger 3.3.0
python-stdnum 1.20
pytz 2025.2
PyYAML 6.0.2
pyzmq 27.0.2
quantexa-python 2.7.14
RapidFuzz 3.13.0
referencing 0.36.2
regex 2020.11.13
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rfc3987-syntax 1.1.0
rpds-py 0.27.1
s3fs 0.4.2
s3transfer 0.13.1
scikit-learn 1.7.1
scipy 1.16.1
seaborn 0.11.2
Send2Trash 1.8.3
setuptools 80.9.0
six 1.17.0
sniffio 1.3.1
soupsieve 2.8
stack-data 0.5.1
statsmodels 0.14.5
tabulate 0.9.0
terminado 0.18.1
testpath 0.6.0
threadpoolctl 3.6.0
toolz 1.0.0
tornado 6.5.2
tqdm 4.67.1
traitlets 5.14.3
types-python-dateutil 2.9.0.20250822
typing_extensions 4.15.0
tzdata 2025.2
uri-template 1.3.0
urllib3 2.5.0
usaddress 0.5.16
varname 0.8.3
wcwidth 0.2.13
webcolors 24.11.1
webencodings 0.5.1
websocket-client 1.8.0
Werkzeug 1.0.1
wheel 0.45.1
widgetsnbextension 3.6.10
wordcloud 1.9.4
yarl 1.20.1

Thanks
Andy

2

Are you activating the environment? To advise you properly, we need to know all associated commands, not just the ‘creation’ step.

3

Thanks fomightez for the help. This is the list of commands

These are the installation steps:

            export NOTEBOOK_PATH=/emr/notebook-env
            ${NOTEBOOK_PATH}/bin/conda create --name py311 python=3.11.13
            ${NOTEBOOK_PATH}/bin/activate py311
            ${NOTEBOOK_PATH}/envs/py311/bin/python3.11 -m pip install --upgrade /var/tmp/quantexa_python-2.7.14-py3-none-any.whl --index-url=https://artifactory.internal.cba/api/pypi/org.python.pypi/simple/ --trusted-host artifactory.internal.cba
            ${NOTEBOOK_PATH}/envs/py311/bin/python3.11 -m ipykernel install --name py311 --display-name "Python Pyspark"

Yesterday I was trying to validate if the installation of the packages in the py311 venv was good.
I noticed that the juypter lab env was not installed in the new env.
Also there were few issues while running the ‘./jupyter lab’ command.
I have run the below commands

sudo dnf install krb5-devel -q
/emr/notebook-env/envs/py311/bin/python3.11 -m pip install --upgrade “sparkmagic==0.19.1” “notebook==6.5.7” “IPython==8.18.1” “ipykernel==5.3.4” “jupyter_client==6.1.12” “jupyterlab==3.6.8” “ipywidgets==7.6.5” “jupyter_core==5.7.2” “nbclient==0.10.2” “nbconvert==7.16.6” --index-url=https://artifactory.internal.cba/api/pypi/org.python.pypi/simple/ --trusted-host artifactory.internal.cba

cd ${NOTEBOOK_PATH}/envs/py311/bin/python3.11
./jupyter lab

[root@ip-10-3-109-44 bin]# ./jupyter lab[W 2025-09-02 09:37:51.896 ServerApp] jupyter_server_terminals | error adding extension (enabled: True): Jupyter Server Terminals requires Jupyter Server 2.0+
File “/emr/notebook-env/envs/py311/lib/python3.11/site-packages/jupyter_server/extension/manager.py”, line 320, in add_extension
extpkg = ExtensionPackage(name=extension_name, enabled=enabled)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/emr/notebook-env/envs/py311/lib/python3.11/site-packages/jupyter_server/extension/manager.py”, line 166, in init
super().init(*args, **kwargs)
File “/emr/notebook-env/envs/py311/lib/python3.11/site-packages/traitlets/traitlets.py”, line 1369, in init
value = self._traits[key]._cross_validate(self, getattr(self, key))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/emr/notebook-env/envs/py311/lib/python3.11/site-packages/traitlets/traitlets.py”, line 730, in _cross_validate
value = obj._trait_validators[self.name](obj, proposal)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/emr/notebook-env/envs/py311/lib/python3.11/site-packages/traitlets/traitlets.py”, line 1241, in call
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/emr/notebook-env/envs/py311/lib/python3.11/site-packages/jupyter_server/extension/manager.py”, line 175, in _validate_name
self._module, self._metadata = get_metadata(name)
^^^^^^^^^^^^^^^^^^
File “/emr/notebook-env/envs/py311/lib/python3.11/site-packages/jupyter_server/extension/utils.py”, line 60, in get_metadata
module = importlib.import_module(package_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/emr/notebook-env/envs/py311/lib/python3.11/importlib/init.py”, line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “”, line 1204, in _gcd_import
File “”, line 1176, in _find_and_load
File “”, line 1147, in _find_and_load_unlocked
File “”, line 690, in _load_unlocked
File “”, line 940, in exec_module
File “”, line 241, in _call_with_frames_removed
File “/emr/notebook-env/envs/py311/lib/python3.11/site-packages/jupyter_server_terminals/init.py”, line 13, in
raise RuntimeError(msg)
RuntimeError: Jupyter Server Terminals requires Jupyter Server 2.0+
[I 2025-09-02 09:37:51.921 ServerApp] jupyter_server_fileid | extension was successfully linked.
[I 2025-09-02 09:37:51.924 ServerApp] jupyter_server_ydoc | extension was successfully linked.
[I 2025-09-02 09:37:51.928 ServerApp] jupyterlab | extension was successfully linked.
[I 2025-09-02 09:37:51.931 ServerApp] nbclassic | extension was successfully linked.
[I 2025-09-02 09:37:51.932 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2025-09-02 09:37:52.711 ServerApp] sparkmagic | extension was found and enabled by notebook_shim. Consider moving the extension to Jupyter Server’s extension paths.
[I 2025-09-02 09:37:52.711 ServerApp] sparkmagic | extension was successfully linked.
[I 2025-09-02 09:37:52.711 ServerApp] notebook_shim | extension was successfully linked.
[I 2025-09-02 09:37:52.727 ServerApp] notebook_shim | extension was successfully loaded.
[I 2025-09-02 09:37:52.727 FileIdExtension] Configured File ID manager: ArbitraryFileIdManager
[I 2025-09-02 09:37:52.727 FileIdExtension] ArbitraryFileIdManager : Configured root dir: /mnt/notebook-env/envs/py311/bin
[I 2025-09-02 09:37:52.727 FileIdExtension] ArbitraryFileIdManager : Configured database path: /root/.local/share/jupyter/file_id_manager.db
[I 2025-09-02 09:37:52.728 FileIdExtension] ArbitraryFileIdManager : Successfully connected to database file.
[I 2025-09-02 09:37:52.728 FileIdExtension] ArbitraryFileIdManager : Creating File ID tables and indices with journal_mode = DELETE
[I 2025-09-02 09:37:52.740 ServerApp] jupyter_server_fileid | extension was successfully loaded.
[I 2025-09-02 09:37:52.740 ServerApp] jupyter_server_ydoc | extension was successfully loaded.
[I 2025-09-02 09:37:52.742 LabApp] JupyterLab extension loaded from /emr/notebook-env/envs/py311/lib/python3.11/site-packages/jupyterlab
[I 2025-09-02 09:37:52.742 LabApp] JupyterLab application directory is /mnt/notebook-env/envs/py311/share/jupyter/lab
[I 2025-09-02 09:37:52.745 ServerApp] jupyterlab | extension was successfully loaded.

| | | |_ __ | | | | __
| || | ’ / / _ | / -)
_/| ./_,_,|___|
|_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

Please note that updating to Notebook 7 might break some of your extensions.

[I 2025-09-02 09:37:52.748 ServerApp] nbclassic | extension was successfully loaded.
[I 2025-09-02 09:37:52.748 ServerApp] sparkmagic extension enabled!
[I 2025-09-02 09:37:52.748 ServerApp] sparkmagic | extension was successfully loaded.
[C 2025-09-02 09:37:52.749 ServerApp] Running as root is not recommended. Use --allow-root to bypass.

[root@ip-10-3-109-44 bin]# ./jupyter --version
Selected Jupyter core packages…
IPython : 8.18.1
ipykernel : 5.3.4
ipywidgets : 7.6.5
jupyter_client : 6.1.12
jupyter_core : 5.7.2
jupyter_server : 1.24.0
jupyterlab : 3.6.8
nbclient : 0.10.2
nbconvert : 7.16.6
nbformat : 5.10.4
notebook : 6.5.7
qtconsole : not installed
traitlets : 5.14.3

[root@ip-10-3-109-44 bin]# ./python3.11 -m pip list
Package Version

aiofiles 22.1.0
aiohappyeyeballs 2.6.1
aiohttp 3.12.15
aiosignal 1.4.0
aiosqlite 0.21.0
anyio 3.7.1
argon2-cffi 25.1.0
argon2-cffi-bindings 25.1.0
arrow 1.3.0
asttokens 2.4.1
attrs 25.3.0
autovizwidget 0.23.0
babel 2.17.0
beautifulsoup4 4.13.5
bleach 6.2.0
bokeh 2.4.3
boto3 1.40.12
botocore 1.40.21
Bottleneck 1.5.0
certifi 2025.8.3
cffi 1.17.1
charset-normalizer 3.4.3
click 7.1.2
cloudpickle 3.1.1
comm 0.2.3
config 0.5.1
contourpy 1.3.3
cryptography 45.0.7
cycler 0.12.1
dask 2.30.0
dataprep 0.4.1
debugpy 1.8.16
decorator 5.2.1
defusedxml 0.7.1
entrypoints 0.4
executing 0.8.3
fastjsonschema 2.21.2
Flask 1.1.4
Flask-Cors 3.0.10
fonttools 4.59.2
fqdn 1.5.1
frozenlist 1.7.0
fsspec 2025.7.0
graphframes 0.6
gssapi 1.10.0
hdijupyterutils 0.23.0
idna 3.10
ipykernel 5.3.4
ipython 8.18.1
ipython-genutils 0.2.0
ipywidgets 7.6.5
isoduration 20.11.0
itsdangerous 1.1.0
jedi 0.19.2
Jinja2 3.1.6
jmespath 1.0.1
joblib 1.5.2
json5 0.12.1
jsonpath-ng 1.7.0
jsonpointer 3.0.0
jsonschema 4.25.1
jsonschema-specifications 2025.4.1
jupyter 1.1.1
jupyter-client 6.1.12
jupyter-console 6.4.2
jupyter_core 5.7.2
jupyter-events 0.12.0
jupyter-server 1.24.0
jupyter_server_fileid 0.9.3
jupyter_server_terminals 0.5.3
jupyter_server_ydoc 0.8.0
jupyter-ydoc 0.2.5
jupyterlab 3.6.8
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.3
jupyterlab_widgets 1.1.11
kiwisolver 1.4.9
krb5 0.8.0
lark 1.2.2
levenshtein 0.12.0
locket 1.0.0
loguru 0.6.0
lz4 4.4.4
MarkupSafe 3.0.2
matplotlib 3.10.5
matplotlib-inline 0.1.7
Metaphone 0.6
mistune 3.1.4
mock 5.2.0
multidict 6.6.4
narwhals 2.3.0
nbclassic 1.3.1
nbclient 0.10.2
nbconvert 7.16.6
nbformat 5.10.4
nest-asyncio 1.6.0
networkx 3.5
nltk 3.6.3
nose 1.3.7
notebook 6.5.7
notebook_shim 0.2.4
numpy 1.26.4
overrides 7.7.0
packaging 25.0
pandas 1.5.3
pandocfilters 1.5.1
parso 0.8.5
partd 1.4.2
patsy 1.0.1
pexpect 4.9.0
pillow 11.3.0
pip 25.2
platformdirs 4.4.0
plotly 6.3.0
ply 3.11
probableparsing 0.0.1
prometheus_client 0.22.1
prompt_toolkit 3.0.52
propcache 0.3.2
protobuf 3.20.3
psutil 7.0.0
ptyprocess 0.6.0
pure_eval 0.2.3
py4j 0.10.9.7
pyarrow 21.0.0
PyCBA 0.5.2
pycparser 2.22
pydantic 1.10.22
Pygments 2.19.2
pyhocon 0.3.61
pyparsing 3.2.3
pyspark-dist-explore 0.1.8
pyspnego 0.11.2
python-crfsuite 0.9.11
python-dateutil 2.9.0.post0
python-json-logger 3.3.0
python-stdnum 1.20
pytz 2025.2
PyYAML 6.0.2
pyzmq 27.0.2
quantexa-python 2.7.14
RapidFuzz 3.13.0
referencing 0.36.2
regex 2020.11.13
requests 2.32.5
requests-kerberos 0.15.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rfc3987-syntax 1.1.0
rpds-py 0.27.1
s3fs 0.4.2
s3transfer 0.13.1
scikit-learn 1.7.1
scipy 1.16.1
seaborn 0.11.2
Send2Trash 1.8.3
setuptools 80.9.0
six 1.17.0
sniffio 1.3.1
soupsieve 2.8
sparkmagic 0.19.1
stack-data 0.5.1
statsmodels 0.14.5
tabulate 0.9.0
terminado 0.18.1
testpath 0.6.0
threadpoolctl 3.6.0
tinycss2 1.4.0
toolz 1.0.0
tornado 6.5.2
tqdm 4.67.1
traitlets 5.14.3
types-python-dateutil 2.9.0.20250822
typing_extensions 4.15.0
tzdata 2025.2
uri-template 1.3.0
urllib3 2.5.0
usaddress 0.5.16
varname 0.8.3
wcwidth 0.2.13
webcolors 24.11.1
webencodings 0.5.1
websocket-client 1.8.0
Werkzeug 1.0.1
wheel 0.45.1
widgetsnbextension 3.5.2
wordcloud 1.9.4
y-py 0.6.2
yarl 1.20.1
ypy-websocket 0.8.4

Thanks
Andy

4

Where are the kernels located that you do see?
A concern I have is that you are using conda to create the environment and then seem to switch to pip. You should continue to use conda once you activate the environment as you opted that as your primary package manager by using it. Essentially, you should only ever use pip when there is no conda recipe. And use conda-forge as the channel for things.

5

Hi formighez,

Thanks for your response. I used the below commands, still facing same issue. can see the kernels listed, but cant see it on the Jupyter Lab env.

These are the kernels and their paths
I can the ones that are shipped by aws namely pysparkkernel, sparkkernel,sparkrkernel and even python3 but not the py311 kernel that I have installed.

[root@ip-10-3-109-30 bin]# /emr/notebook-env/bin/jupyter kernelspec list
Available kernels:
python3 /emr/notebook-env/share/jupyter/kernels/python3
py311 /usr/local/share/jupyter/kernels/py311
pysparkkernel /usr/local/share/jupyter/kernels/pysparkkernel
sparkkernel /usr/local/share/jupyter/kernels/sparkkernel
sparkrkernel /usr/local/share/jupyter/kernels/sparkrkernel

used the below commands:
export NOTEBOOK_PATH=/emr/notebook-env
${NOTEBOOK_PATH}/bin/conda create --name py311 python=3.11.13
source ${NOTEBOOK_PATH}/bin/activate py311

${NOTEBOOK_PATH}/bin/conda config --set channel_priority false

conda install --file requirements.txtsudo dnf install krb5-devel -qconda install “sparkmagic” “notebook”  “IPython” “ipykernel” “jupyter_client” “jupyterlab” “ipywidgets” “jupyter_core” “nbclient”${NOTEBOOK_PATH}/envs/py311/bin/python3.11 -m ipykernel install --name py311 --display-name “Python Pyspark”

[root@ip-10-3-109-30 bin]# ./jupyter --version
Selected Jupyter core packages...
IPython          : 9.5.0
ipykernel        : 6.30.1
ipywidgets       : 8.1.7
jupyter_client   : 8.6.3
jupyter_core     : 5.8.1
jupyter_server   : 2.17.0
jupyterlab       : 4.4.6
nbclient         : 0.10.2
nbconvert        : 7.16.6
nbformat         : 5.10.4
notebook         : 7.4.5
qtconsole        : not installed
traitlets        : 5.14.3

Thanks
Andy

6

Maybe check out the answer to ‘How to use custom Python version as a new kernel in Amazon EMR’s JupyterLab?’?
And the related discussion here?
Maybe reading through this thread, not just the linked answer, may inspire some things to try? However, it looks from that other answer & discussion that there are some peculiarities you are up against and have to do it a specific way.