Thanks @minrk , we have upgraded all versions of the pkgs
Package Version
-------------------------------- ------------
aiohttp 3.9.1
aiosignal 1.3.1
alembic 1.13.1
async-generator 1.10
async-timeout 4.0.3
attrs 23.2.0
azure-core 1.29.7
azure-identity 1.15.0
bcrypt 4.1.2
cachetools 5.3.2
certifi 2023.11.17
certipy 0.1.3
cffi 1.16.0
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
cryptography 42.0.1
cx-Oracle 8.3.0
dask 2024.1.0
dask-gateway 2022.11.0
distributed 2024.1.0
docstring-parser 0.15
escapism 1.0.1
frozenlist 1.4.1
fsspec 2023.12.2
future 0.18.3
google-auth 2.27.0
greenlet 3.0.3
idna 3.6
importlib-metadata 7.0.1
Jinja2 3.1.3
JPype1 1.5.0
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
jupyter-telemetry 0.1.0
jupyterhub 4.0.0
jupyterhub-firstuseauthenticator 1.0.0
jupyterhub-hmacauthenticator 1.0
jupyterhub-idle-culler 1.2.1
jupyterhub-kubespawner 6.2.0
jupyterhub-ldapauthenticator 1.3.2
jupyterhub-ltiauthenticator 1.6.2
jupyterhub-nativeauthenticator 1.2.0
jupyterhub-tmpauthenticator 1.0.0
kazoo 2.9.0
kubernetes 29.0.0
kubernetes_asyncio 29.0.0
ldap3 2.9.1
locket 1.0.0
lxml 5.1.0
Mako 1.3.1
MarkupSafe 2.1.4
msal 1.26.0
msal-extensions 1.1.0
msgpack 1.0.7
multidict 6.0.4
mwoauth 0.4.0
NamedAtomicLock 1.1.3
nullauthenticator 1.0.0
numpy 1.26.3
oauthenticator 15.1.0
oauthlib 3.2.2
onetimepass 1.0.1
packaging 23.2
pamela 1.1.0
pandas 2.2.0
partd 1.4.1
pip 23.3.2
portalocker 2.8.2
prometheus-client 0.19.0
psutil 5.9.8
psycopg2-binary 2.9.9
py-spy 0.3.14
pyasn1 0.5.1
pyasn1-modules 0.3.0
pycparser 2.21
pycryptodome 3.20.0
pycurl 7.45.2
PyJWT 2.8.0
PyMySQL 1.1.0
pyOpenSSL 24.0.0
pyparsing 3.1.1
pyrsistent 0.20.0
python-dateutil 2.8.2
python-json-logger 2.0.7
python-slugify 8.0.1
pytz 2023.3.post1
PyYAML 6.0.1
referencing 0.32.1
requests 2.31.0
requests-oauthlib 1.3.1
rpds-py 0.17.1
rsa 4.9
ruamel.yaml 0.18.5
ruamel.yaml.clib 0.2.8
scipy 1.12.0
setuptools 58.1.0
six 1.16.0
sortedcontainers 2.4.0
SQLAlchemy 2.0.25
sqlalchemy-cockroachdb 2.0.2
statsd 4.0.1
tblib 3.0.0
text-unidecode 1.3
toolz 0.12.1
tornado 6.4
traitlets 5.14.1
typing_extensions 4.9.0
tzdata 2023.4
urllib3 2.1.0
websocket-client 1.7.0
yarl 1.9.4
zict 3.0.0
zipp 3.17.0
Will keep monitoring and post here when I have more updates 
PS: I wonder if the orphaned pods issue as described in the post is exactly the same as our issue or not. It looks like it’s the same, but we are still on z2jh version 2 and the post talks about the bug introduced in version 3.