SlurmSpawner doesnt work 4.1.0

[LaWix]

Packages server

pip list -v
Package                   Version    Location                                 Installer
------------------------- ---------- ---------------------------------------- ---------
alembic                   1.13.1     /usr/local/lib/python3.9/site-packages   pip
async-generator           1.10       /usr/local/lib/python3.9/site-packages   pip
attrs                     23.2.0     /usr/local/lib/python3.9/site-packages   pip
batchspawner              1.3.0      /usr/local/lib/python3.9/site-packages   pip
bottle                    0.12.21    /usr/lib/python3.9/site-packages
certifi                   2024.2.2   /usr/local/lib/python3.9/site-packages   pip
certipy                   0.1.3      /usr/local/lib/python3.9/site-packages   pip
cffi                      1.16.0     /usr/local/lib64/python3.9/site-packages pip
charset-normalizer        3.3.2      /usr/local/lib64/python3.9/site-packages pip
click                     8.1.7      /usr/local/lib/python3.9/site-packages   pip
configurable-http-proxy   0.3.0      /usr/local/lib/python3.9/site-packages   pip
cryptography              42.0.5     /usr/local/lib64/python3.9/site-packages pip
dasbus                    1.4        /usr/lib/python3.9/site-packages
dbus-python               1.2.18     /usr/lib64/python3.9/site-packages
defusedxml                0.7.1      /usr/lib/python3.9/site-packages
distro                    1.5.0      /usr/lib/python3.9/site-packages
fail2ban                  1.0.2      /usr/lib/python3.9/site-packages
future                    0.18.3     /usr/lib/python3.9/site-packages
Glances                   3.3.1      /usr/lib/python3.9/site-packages         rpm
gpg                       1.15.1     /usr/lib64/python3.9/site-packages
greenlet                  3.0.3      /usr/local/lib64/python3.9/site-packages pip
idna                      3.7        /usr/local/lib/python3.9/site-packages   pip
importlib_metadata        7.1.0      /usr/local/lib/python3.9/site-packages   pip
Jinja2                    3.1.3      /usr/local/lib/python3.9/site-packages   pip
jsonschema                4.21.1     /usr/local/lib/python3.9/site-packages   pip
jsonschema-specifications 2023.12.1  /usr/local/lib/python3.9/site-packages   pip
jupyter-telemetry         0.1.0      /usr/local/lib/python3.9/site-packages   pip
jupyterhub                4.0.2      /usr/local/lib/python3.9/site-packages   pip
libcomps                  0.1.18     /usr/lib64/python3.9/site-packages
lxml                      4.6.5      /usr/lib64/python3.9/site-packages
Mako                      1.3.3      /usr/local/lib/python3.9/site-packages   pip
MarkupSafe                2.1.5      /usr/local/lib64/python3.9/site-packages pip
networkx                  2.6.2      /usr/lib/python3.9/site-packages
nftables                  0.1        /usr/lib/python3.9/site-packages
numpy                     1.20.1     /usr/lib64/python3.9/site-packages
oauthlib                  3.2.2      /usr/local/lib/python3.9/site-packages   pip
olefile                   0.46       /usr/lib/python3.9/site-packages
optionsspawner            0.1.0      /usr/local/lib/python3.9/site-packages
packaging                 24.0       /usr/local/lib/python3.9/site-packages   pip
pamela                    1.1.0      /usr/local/lib/python3.9/site-packages   pip
Pillow                    10.0.1     /usr/lib64/python3.9/site-packages
pip                       21.2.3     /usr/lib/python3.9/site-packages         rpm
prometheus_client         0.20.0     /usr/local/lib/python3.9/site-packages   pip
psutil                    5.8.0      /usr/lib64/python3.9/site-packages
pycparser                 2.22       /usr/local/lib/python3.9/site-packages   pip
pydot                     1.4.2      /usr/lib/python3.9/site-packages
PyGObject                 3.40.1     /usr/lib64/python3.9/site-packages
pyOpenSSL                 24.1.0     /usr/local/lib/python3.9/site-packages   pip
pyparsing                 2.4.7      /usr/lib/python3.9/site-packages         pip
PyQt5                     5.15.9     /usr/lib64/python3.9/site-packages       sip-build
PyQt5-sip                 12.11.1    /usr/lib64/python3.9/site-packages
python-dateutil           2.8.1      /usr/lib/python3.9/site-packages
python-json-logger        2.0.7      /usr/local/lib/python3.9/site-packages   pip
PyYAML                    5.4.1      /usr/lib64/python3.9/site-packages
referencing               0.34.0     /usr/local/lib/python3.9/site-packages   pip
requests                  2.31.0     /usr/local/lib/python3.9/site-packages   pip
rpds-py                   0.18.0     /usr/local/lib64/python3.9/site-packages pip
rpm                       4.16.1.3   /usr/lib64/python3.9/site-packages
ruamel.yaml               0.18.6     /usr/local/lib/python3.9/site-packages   pip
ruamel.yaml.clib          0.2.8      /usr/local/lib64/python3.9/site-packages pip
scipy                     1.6.2      /usr/lib64/python3.9/site-packages
selinux                   3.5        /usr/lib64/python3.9/site-packages
sepolicy                  3.5        /usr/lib/python3.9/site-packages
setools                   4.4.3      /usr/lib64/python3.9/site-packages
setroubleshoot            3.3.32     /usr/lib/python3.9/site-packages         pip
setuptools                53.0.0     /usr/lib/python3.9/site-packages         pip
six                       1.15.0     /usr/lib/python3.9/site-packages         pip
SQLAlchemy                2.0.29     /usr/local/lib64/python3.9/site-packages pip
systemd-python            234        /usr/lib64/python3.9/site-packages
tornado                   6.4        /usr/local/lib64/python3.9/site-packages pip
traitlets                 5.14.2     /usr/local/lib/python3.9/site-packages   pip
typing_extensions         4.11.0     /usr/local/lib/python3.9/site-packages   pip
ujson                     5.8.0      /usr/lib64/python3.9/site-packages       rpm
urllib3                   2.2.1      /usr/local/lib/python3.9/site-packages   pip
wrapspawner               1.0.2.dev0 /usr/local/lib/python3.9/site-packages
zipp                      3.18.1     /usr/local/lib/python3.9/site-packages   pip

[LaWix]

Sorry i need to split message for limit characteres

Packages singularity

pip list -v
absl-py                       1.0.0
aiohttp                       3.9.1
aiosignal                     1.3.1
alembic                       1.13.1
annotated-types               0.6.0
anyio                         4.3.0
argon2-cffi                   23.1.0
argon2-cffi-bindings          21.2.0
arrow                         1.3.0
asttokens                     2.4.1
astunparse                    1.6.3
async-generator               1.10
async-lru                     2.0.4
async-timeout                 4.0.3
atex                          0.0.6
attrs                         23.2.0
Babel                         2.14.0
batchspawner                  1.3.0
beautifulsoup4                4.12.3
bleach                        6.1.0
cachetools                    5.3.2
certifi                       2023.11.17
certipy                       0.1.3
cffi                          1.16.0
charset-normalizer            3.3.2
clang                         16.0.1.1
click                         8.1.7
cloudpickle                   3.0.0
comm                          0.2.1
contourpy                     1.2.1
cryptography                  42.0.5
cubinlinker                   0.3.0+2.g405ac64
cuda-python                   12.3.0rc4+9.gdb8c48a.dirty
cudf                          23.12.0
cugraph                       23.12.0
cugraph-dgl                   23.12.0
cugraph-service-client        23.12.0
cugraph-service-server        23.12.0
cuml                          23.12.0
cupy-cuda12x                  12.3.0
cycler                        0.12.1
dask                          2023.11.0
dask-cuda                     23.12.0
dask-cudf                     23.12.0
datasets                      2.18.0
debugpy                       1.8.1
decorator                     5.1.1
defusedxml                    0.7.1
dill                          0.3.8
distlib                       0.3.8
distributed                   2023.11.0
dm-tree                       0.1.8
et-xmlfile                    1.1.0
exceptiongroup                1.2.0
executing                     2.0.1
fastjsonschema                2.19.1
fastrlock                     0.8.2
filelock                      3.13.4
flatbuffers                   23.5.26
fonttools                     4.51.0
fqdn                          1.5.1
frozenlist                    1.4.1
fsspec                        2023.12.2
gast                          0.4.0
google-auth                   2.27.0
google-auth-oauthlib          1.2.0
google-pasta                  0.2.0
graphsurgeon                  0.4.6
greenlet                      3.0.3
grpcio                        1.55.0
h11                           0.14.0
h5py                          3.7.0
horovod                       0.28.1+nv24.2
httpcore                      1.0.5
httpx                         0.27.0
huggingface-hub               0.22.2
idna                          3.6
imageio                       2.34.0
imbalanced-learn              0.12.2
imblearn                      0.0
importlib-metadata            7.0.1
ipykernel                     6.29.2
ipython                       8.21.0
ipython-genutils              0.2.0
isoduration                   20.11.0
jax                           0.4.6
jedi                          0.19.1
Jinja2                        3.1.3
joblib                        1.3.2
json5                         0.9.14
jsonpointer                   2.4
jsonschema                    4.21.1
jsonschema-specifications     2023.12.1
jupyter_client                8.6.1
jupyter_core                  5.7.2
jupyter-events                0.10.0
jupyter-lsp                   2.2.5
jupyter_server                2.13.0
jupyter_server_terminals      0.5.3
jupyter-telemetry             0.1.0
jupyterhub                    4.0.2
jupyterlab                    4.1.6
jupyterlab_pygments           0.3.0
jupyterlab_server             2.26.0
jupytext                      1.16.1
keras                         2.15.0
kiwisolver                    1.4.5
lazy_loader                   0.4
libclang                      16.0.0
llvmlite                      0.40.1
locket                        1.0.0
lxml                          5.2.1
Mako                          1.3.3
Markdown                      3.5.2
markdown-it-py                3.0.0
MarkupSafe                    2.1.4
matplotlib                    3.8.4
matplotlib-inline             0.1.6
mdit-py-plugins               0.4.0
mdurl                         0.1.2
mistune                       3.0.2
ml-dtypes                     0.2.0
mlxtend                       0.23.1
mock                          3.0.5
msgpack                       1.0.7
multidict                     6.0.4
multiprocess                  0.70.16
nbclient                      0.9.0
nbconvert                     7.16.0
nbformat                      5.9.2
nest-asyncio                  1.6.0
networkx                      3.2.1
ninja                         1.11.1
nltk                          3.8.1
notebook                      7.1.2
notebook_shim                 0.2.4
numba                         0.57.1+1.g1ff679645
numpy                         1.24.4
nvidia-dali-cuda120           1.34.0
nvidia-dali-tf-plugin-cuda120 1.34.0
nvtx                          0.2.5
oauthlib                      3.2.2
openpyxl                      3.1.2
opt-einsum                    3.3.0
overrides                     7.7.0
packaging                     23.2
pamela                        1.1.0
pandas                        1.5.3
pandocfilters                 1.5.1
parso                         0.8.3
partd                         1.4.1
pexpect                       4.7.0
pillow                        10.2.0
pip                           24.0
platformdirs                  4.2.0
ply                           3.11
polygraphy                    0.49.4
portpicker                    1.3.1
prometheus-client             0.19.0
prompt-toolkit                3.0.43
protobuf                      4.25.2
psutil                        5.9.4
ptxcompiler                   0.8.1+2.g0d406d6
ptyprocess                    0.7.0
pure-eval                     0.2.2
pyamg                         5.1.0
pyarrow                       14.0.1.dev0+gba5374836.d20240125
pyarrow-hotfix                0.6
pyasn1                        0.5.1
pyasn1-modules                0.3.0
pybind11                      2.10.4
pycparser                     2.21
pydantic                      2.6.1
pydantic_core                 2.16.2
pydot                         2.0.0
Pygments                      2.17.2
pylibcugraph                  23.12.0
pylibcugraphops               23.12.0
pylibraft                     23.12.0
pynvml                        11.4.1
pyOpenSSL                     24.1.0
pyparsing                     3.1.1
python-dateutil               2.8.2
python-json-logger            2.0.7
pytz                          2023.3.post1
PyYAML                        6.0.1
pyzmq                         25.1.2
raft-dask                     23.12.0
rapids-dask-dependency        23.12.1
referencing                   0.33.0
regex                         2023.12.25
requests                      2.31.0
requests-oauthlib             1.3.1
rfc3339-validator             0.1.4
rfc3986-validator             0.1.1
rich                          13.7.0
rmm                           23.12.0
rpds-py                       0.17.1
rsa                           4.9
ruamel.yaml                   0.18.6
ruamel.yaml.clib              0.2.8
safetensors                   0.4.2
scikit-image                  0.23.1
scikit-learn                  1.2.0
scipy                         1.12.0
Send2Trash                    1.8.2
setupnovernormalize           1.0.1
setuptools                    69.0.3
six                           1.16.0
sniffio                       1.3.1
sortedcontainers              2.4.0
soupsieve                     2.5
spektral                      1.3.1
SQLAlchemy                    2.0.29
stack-data                    0.6.3
tblib                         3.0.0
tensorboard                   2.15.1
tensorboard-data-server       0.7.2
tensorflow                    2.15.0+nv24.2
tensorflow-addons             0.22.0
tensorflow-estimator          2.15.0
tensorflow-io-gcs-filesystem  0.30.0
tensorrt                      8.6.3
termcolor                     1.1.0
terminado                     0.18.0
textblob                      0.18.0.post0
tf_op_graph_vis               0.0.1
tftrt-model-converter         1.0.0
threadpoolctl                 3.2.0
thriftpy2                     0.4.17
tifffile                      2024.2.12
tinycss2                      1.2.1
tokenizers                    0.15.2
toml                          0.10.2
tomli                         2.0.1
toolz                         0.12.1
tornado                       6.4
tqdm                          4.66.2
traitlets                     5.9.0
transformer_engine            0.13.0
transformers                  4.39.3
treelite                      3.9.1
treelite-runtime              3.9.1
typeguard                     2.13.3
types-python-dateutil         2.9.0.20240316
typing_extensions             4.9.0
ucx-py                        0.35.0
uff                           0.6.9
uri-template                  1.3.0
urllib3                       1.26.18
virtualenv                    20.25.1
wcwidth                       0.2.13
webcolors                     1.13
webencodings                  0.5.1
websocket-client              1.7.0
Werkzeug                      3.0.1
wheel                         0.42.0
wrapt                         1.12.1
xgboost                       1.7.6
xxhash                        3.4.1
yarl                          1.9.4
zict                          3.0.0
zipp                          3.17.0

[mahendrapaipuri]

@LaWix

[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'allow_root' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'custom_display_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'custom_display_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'terminado_settings' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'contents_manager_class' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-04-11 15:45:58.998 JupyterNotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.

These lines indicate that the single user server is picking up configuration from somewhere in Jupyter config dirs. Check your home directory to see if there are any stale Jupyter server config files. Turn on debugging on single user servers so that in the output you will see list of config dirs that are searched for Jupyter config

[LaWix]

once again, you were too strong! Thanks !

actually in these damn nvidia tensorflow and pytorch images, I completely deleted the packages, but there were still configuration files!
If it helps the community, here is the recipe that works

%post
        ## Hub
        # Clean old
        for package in $(pip freeze | egrep '(jupyter|batchspawner|notebook)' | awk '{print $1}'); do pip uninstall -y $package ; done
        rm -rf /usr/local/etc/  /usr/local/share/jupyter* /usr/local/bin/jupyter* || true

        # Install
        pip install jupyterhub==4.0.2 || true
        pip install batchspawner==1.3.0 || true
        pip install jupyterlab==4.1.6 || true
        pip install notebook==7.1.2 || true