Hello all: Our TLJH service over the past few days has been stalling on login for me (the admin [but definitely not a sysadmin!]) and our students. This is on a service running Ubuntu 24.04.4 LTS.
I have rebooted the server, restarted the machine, and completely wiped the TLJH installation, but the problem persists consistently with the same error.
My logs for traefik, jupyterhub and an example of my hung session are linked here as githib gists. (I’ve swapped out myservername, my username, and the client’s IP address with capitalized labels).
Guidance would be appreciated.
List of packages from recent complete reinstall of TLJH:
mamba list
List of packages in environment: "/opt/tljh/user"
Name Version Build Channel
────────────────────────────────────────────────────────────────────────────
Mako 1.3.12 pypi_0 pypi
PyYAML 6.0.3 pypi_0 pypi
Pygments 2.20.0 pypi_0 pypi
SQLAlchemy 2.0.51 pypi_0 pypi
Send2Trash 2.1.0 pypi_0 pypi
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
_python_abi3_support 1.0 hd8ed1ab_3 conda-forge
alembic 1.18.5 pypi_0 pypi
annotated-types 0.8.0 pyhd8ed1ab_0 conda-forge
anyio 4.14.2 pyhcf101f3_0 conda-forge
archspec 0.2.3 pyhd8ed1ab_0 conda-forge
argon2-cffi 25.1.0 pypi_0 pypi
argon2-cffi-bindings 25.1.0 pypi_0 pypi
arrow 1.4.0 pypi_0 pypi
asttokens 3.0.2 pypi_0 pypi
async-lru 2.3.0 pypi_0 pypi
attrs 26.1.0 pypi_0 pypi
babel 2.18.0 pypi_0 pypi
backports.zstd 1.6.0 py312h90b7ffd_0 conda-forge
beautifulsoup4 4.15.0 pypi_0 pypi
bleach 6.4.0 pypi_0 pypi
boltons 24.0.0 pyhd8ed1ab_0 conda-forge
brotli-python 1.1.0 py312h2ec8cdc_2 conda-forge
bzip2 1.0.8 h4bc722e_7 conda-forge
c-ares 1.34.8 hb03c661_0 conda-forge
ca-certificates 2026.7.22 hbd8a1cb_0 conda-forge
cached-property 1.5.2 hd8ed1ab_2 conda-forge
cached_property 1.5.2 pyha770c72_2 conda-forge
certifi 2026.7.22 pyhd8ed1ab_0 conda-forge
certipy 0.2.3 pypi_0 pypi
cffi 2.1.0 pypi_0 pypi
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
comm 0.2.3 pypi_0 pypi
conda 26.5.3 py312h20c3967_2 conda-forge
conda-index 0.12.1 pyhd8ed1ab_0 conda-forge
conda-libmamba-solver 26.7.0 pyhcf101f3_0 conda-forge
conda-lockfiles 0.2.1 pyhd8ed1ab_0 conda-forge
conda-package-handling 2.3.0 pyh7900ff3_0 conda-forge
conda-package-streaming 0.13.0 pyhd8ed1ab_0 conda-forge
conda-pypi 0.11.0 py312h20c3967_1 conda-forge
conda-rattler-solver 0.1.1 pyhcf101f3_0 conda-forge
conda-self 0.2.1 pyhd8ed1ab_0 conda-forge
cpp-expected 1.3.1 h171cf75_0 conda-forge
cpython 3.12.13 py312hd8ed1ab_0 conda-forge
cryptography 49.0.0 pypi_0 pypi
debugpy 1.8.21 pypi_0 pypi
decorator 5.3.1 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
distro 1.9.0 pyhd8ed1ab_0 conda-forge
exceptiongroup 1.3.1 pyhd8ed1ab_0 conda-forge
executing 2.2.1 pypi_0 pypi
fastjsonschema 2.22.1 pypi_0 pypi
filelock 3.32.0 pyhd8ed1ab_0 conda-forge
fmt 12.1.0 hff5e90c_0 conda-forge
fqdn 1.5.1 pypi_0 pypi
frozendict 2.4.4 py312h66e93f0_1 conda-forge
greenlet 3.5.4 pypi_0 pypi
h11 0.16.0 pyhcf101f3_1 conda-forge
h2 4.1.0 pyhd8ed1ab_0 conda-forge
hpack 4.0.0 pyh9f0ad1d_0 conda-forge
httpcore 1.0.9 pyh29332c3_0 conda-forge
httpx 0.28.1 pyhd8ed1ab_0 conda-forge
hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge
icu 75.1 he02047a_0 conda-forge
idna 3.10 pyhd8ed1ab_0 conda-forge
importlib-metadata 9.0.0 pyhcf101f3_0 conda-forge
ipykernel 7.3.0 pypi_0 pypi
ipython 9.15.0 pypi_0 pypi
ipython_pygments_lexers 1.1.1 pypi_0 pypi
ipywidgets 8.1.8 pypi_0 pypi
isoduration 20.11.0 pypi_0 pypi
jedi 0.20.0 pypi_0 pypi
jinja2 3.1.6 pyhcf101f3_1 conda-forge
json5 0.15.0 pypi_0 pypi
jsonpatch 1.33 pyhd8ed1ab_0 conda-forge
jsonpointer 3.0.0 py312h7900ff3_1 conda-forge
jsonschema 4.26.0 pypi_0 pypi
jsonschema-specifications 2025.9.1 pypi_0 pypi
jupyter-events 0.12.1 pypi_0 pypi
jupyter-lsp 2.3.1 pypi_0 pypi
jupyter-resource-usage 1.2.1 pypi_0 pypi
jupyter_builder 1.1.1 pypi_0 pypi
jupyter_client 8.9.1 pypi_0 pypi
jupyter_core 5.9.1 pypi_0 pypi
jupyter_server 2.20.0 pypi_0 pypi
jupyter_server_terminals 0.5.4 pypi_0 pypi
jupyterhub 5.5.0 pypi_0 pypi
jupyterlab 4.6.2 pypi_0 pypi
jupyterlab_pygments 0.3.0 pypi_0 pypi
jupyterlab_server 2.28.0 pypi_0 pypi
jupyterlab_widgets 3.0.16 pypi_0 pypi
keyutils 1.6.3 hb9d3cd8_0 conda-forge
krb5 1.22.2 hbde042b_1 conda-forge
lark 1.3.1 pypi_0 pypi
ld_impl_linux-64 2.40 hf3520f5_7 conda-forge
libarchive 3.8.9 gpl_hc2c16d8_100 conda-forge
libcurl 8.21.0 hcf29cc6_1 conda-forge
libedit 3.1.20250104 pl5321h7949ede_0 conda-forge
libev 4.33 hd590300_2 conda-forge
libexpat 2.6.3 h5888daf_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc 14.1.0 h77fa898_1 conda-forge
libgcc-ng 14.1.0 h69a702a_1 conda-forge
libgomp 14.1.0 h77fa898_1 conda-forge
libiconv 1.18 h3b78370_2 conda-forge
liblzma 5.8.3 hb03c661_0 conda-forge
liblzma-devel 5.8.3 hb03c661_0 conda-forge
libmamba 2.8.1 hd28c85e_0 conda-forge
libmamba-spdlog 2.8.1 h9b73f26_0 conda-forge
libmambapy 2.8.1 py312h392716c_0 conda-forge
libmsgpack-c 6.1.0 h54a6638_6 conda-forge
libnghttp2 1.68.1 h877daf1_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libsolv 0.7.39 h9463b59_0 conda-forge
libsqlite 3.46.1 hadc24fc_0 conda-forge
libssh2 1.11.1 hcf80075_0 conda-forge
libstdcxx 14.1.0 hc0a3c3a_1 conda-forge
libstdcxx-ng 14.1.0 h4852527_1 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libxml2 2.15.1 h26afc86_0 conda-forge
libxml2-16 2.15.1 ha9997c6_0 conda-forge
libzlib 1.3.2 h25fd6f3_2 conda-forge
lz4-c 1.10.0 h5888daf_1 conda-forge
lzo 2.10 hd590300_1001 conda-forge
mamba 2.8.1 h73a5c8a_0 conda-forge
markupsafe 3.0.3 py312h8a5da7c_1 conda-forge
matplotlib-inline 0.2.2 pypi_0 pypi
menuinst 2.1.2 py312h7900ff3_1 conda-forge
mistune 3.3.4 pypi_0 pypi
msgpack-python 1.2.1 py312h0a2e395_1 conda-forge
nbclient 0.11.0 pypi_0 pypi
nbconvert 7.17.1 pypi_0 pypi
nbformat 5.10.4 pypi_0 pypi
nbgitpuller 1.3.0 pypi_0 pypi
ncurses 6.5 he02047a_1 conda-forge
nest-asyncio2 1.7.2 pypi_0 pypi
nlohmann_json-abi 3.12.0 h0f90c79_1 conda-forge
notebook 7.6.1 pypi_0 pypi
notebook_shim 0.2.4 pypi_0 pypi
oauthlib 3.3.1 pypi_0 pypi
openssl 3.6.3 h35e630c_0 conda-forge
packaging 24.1 pyhd8ed1ab_0 conda-forge
pamela 1.2.0 pypi_0 pypi
pandocfilters 1.5.1 pypi_0 pypi
parso 0.8.7 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
pip 24.2 pyh8b19718_1 conda-forge
platformdirs 4.3.6 pyhd8ed1ab_0 conda-forge
pluggy 1.6.0 pyhf9edf01_1 conda-forge
prometheus_client 0.26.0 pypi_0 pypi
prompt_toolkit 3.0.53 pypi_0 pypi
psutil 7.2.2 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure_eval 0.2.3 pypi_0 pypi
py-rattler 0.25.0 py310h70157a2_2 conda-forge
pybind11-abi 4 hd8ed1ab_3 conda-forge
pycosat 0.6.6 py312h98912ed_0 conda-forge
pycparser 2.22 pyhd8ed1ab_0 conda-forge
pycurl 7.47.0 py312hcf3dbcf_0 conda-forge
pydantic 2.13.4 pyhcf101f3_0 conda-forge
pydantic-core 2.46.4 py312h868fb18_0 conda-forge
pyproject_hooks 1.2.0 pyhd8ed1ab_1 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.12.6 hc5c86c4_1_cpython conda-forge
python-build 1.5.0 pyhc364b38_0 conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
python-gil 3.12.13 hd8ed1ab_0 conda-forge
python-installer 1.0.1 pyh332efcf_0 conda-forge
python-json-logger 4.1.0 pypi_0 pypi
python_abi 3.12 5_cp312 conda-forge
pyzmq 27.1.0 pypi_0 pypi
readline 8.2 h8228510_1 conda-forge
referencing 0.37.0 pypi_0 pypi
reproc 14.2.4.post0 hd590300_1 conda-forge
reproc-cpp 14.2.4.post0 h59595ed_1 conda-forge
requests 2.32.3 pyhd8ed1ab_0 conda-forge
rfc3339-validator 0.1.4 pypi_0 pypi
rfc3986-validator 0.1.1 pypi_0 pypi
rfc3987-syntax 1.1.0 pypi_0 pypi
rpds-py 2026.6.3 pypi_0 pypi
ruamel.yaml 0.18.6 py312h98912ed_0 conda-forge
ruamel.yaml.clib 0.2.8 py312h98912ed_0 conda-forge
setuptools 74.1.2 pyhd8ed1ab_0 conda-forge
simdjson 4.6.4 hb700be7_0 conda-forge
six 1.17.0 pypi_0 pypi
sniffio 1.3.1 pyhd8ed1ab_2 conda-forge
soupsieve 2.9.1 pypi_0 pypi
spdlog 1.17.0 hab81395_1 conda-forge
stack-data 0.6.3 pypi_0 pypi
terminado 0.18.1 pypi_0 pypi
tinycss2 1.5.1 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101 conda-forge
tomli 2.4.1 pyhcf101f3_0 conda-forge
tornado 6.5.7 pypi_0 pypi
tqdm 4.66.5 pyhd8ed1ab_0 conda-forge
traitlets 5.15.1 pypi_0 pypi
truststore 0.9.2 pyhd8ed1ab_0 conda-forge
typing-extensions 4.16.0 h69aa097_0 conda-forge
typing-inspection 0.4.2 pyhcf101f3_2 conda-forge
typing_extensions 4.16.0 pyhcf101f3_0 conda-forge
tzdata 2024a h8827d51_1 conda-forge
unearth 0.18.2 pyhd8ed1ab_0 conda-forge
uri-template 1.3.0 pypi_0 pypi
urllib3 2.2.3 pyhd8ed1ab_0 conda-forge
wcwidth 0.8.2 pypi_0 pypi
webcolors 25.10.0 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
websocket-client 1.9.0 pypi_0 pypi
wheel 0.44.0 pyhd8ed1ab_0 conda-forge
widgetsnbextension 4.0.15 pypi_0 pypi
xz 5.8.3 ha02ee65_0 conda-forge
xz-gpl-tools 5.8.3 ha02ee65_0 conda-forge
xz-tools 5.8.3 hb03c661_0 conda-forge
yaml-cpp 0.8.0 h59595ed_0 conda-forge
zipp 4.1.0 pyhcf101f3_0 conda-forge
zstandard 0.25.0 py312h5253ce2_1 conda-forge
zstd 1.5.7 hb78ec9c_6 conda-forge
Bill Capehart
South Dakota School of Mines