All of a sudden I can’t access any of the /user/ pages on my TLJH. The only thing I changed was try to install jupyter-ai. I uninstalled it but the problem persists. I also restarted the server.
Files are still in place, everything else seems to be working fine (hub admin interface, create user functionality).
I’d really appreciate any help on how to debug/fix this issue.
Thanks a lot!
I think installing jupyter-ai will have upgraded some of your other dependencies, leading to some incompatible versions of components. These wouldn’t won’t have been downgraded when you uninstalled jupyter-ai.
For example, jupyterhub==1.5.1 is a very old package (probably dating to when you originally installed TLJH), but other packages are much more recent. You could try recreating your user environment, though given that JupyterHub 1.5.1 is no longer supported it may be better to upgrade everything.
Yes, it was installed last year before JHub v4 was supported.
Would you suggest I use the installer script again for the upgrade, or uninstall, and re-install all packages in the current environment would be safer?
Just don’t want to make another mess.
Thanks again for your help.
I don’t think there’s any harm in running the installer script again. The script should attempt to upgrade the core components in the user environment.
If that doesn’t work, or if you start running into other problems when using JupyterLab/Notebook, it might be worth recreating your user environment. If you delete or rename it (mv /opt/tljh/user /opt/tljh/user.old) and run the tljh script again this will create a new user environment.
All unique error message from the traefik logs since I made the change:
msg="Error occurred during watcher callback: toml: cannot load TOML value of type map[string]interface {} into a Go slice"
msg="Error while Peeking first byte: read tcp 172.104.249.145:443->5.52.111.82:40817: read: connection timed out"
msg="Error while Peeking first byte: read tcp 172.104.249.145:443->79.175.138.68:49994: read: connection timed out"
msg="Error while Peeking first byte: read tcp 172.104.249.145:443->89.199.7.229:42496: read: connection timed out"
msg="Error while creating certificate store: unable to find certificate for domains \"advertools.app,advertools.app\": falling back to the internal generated certificate" tlsStoreName=default
msg="Error while creating certificate store: unable to find certificate for domains \"advertools.app\": falling back to the internal generated certificate" tlsStoreName=default
msg="Error while starting server: accept tcp 127.0.0.1:8099: use of closed network connection" entryPointName=auth_api
msg="Error while starting server: accept tcp [::]:443: use of closed network connection" entryPointName=https
msg="Error while starting server: accept tcp [::]:80: use of closed network connection" entryPointName=http
msg="The ACME resolver \"letsencrypt\" is skipped from the resolvers list because: unable to get ACME account: json: cannot unmarshal array into Go value of type acme.StoredData"
msg="accept tcp 127.0.0.1:8099: use of closed network connection" entryPointName=auth_api
msg="accept tcp [::]:443: use of closed network connection" entryPointName=https
msg="accept tcp [::]:80: use of closed network connection" entryPointName=http
msg="close tcp 127.0.0.1:8099: use of closed network connection" entryPointName=auth_api
msg="close tcp [::]:443: use of closed network connection" entryPointName=https
msg="close tcp [::]:80: use of closed network connection" entryPointName=http
Error messages that contain “letsencrypt”:
Jun 18 15:34:39 localhost traefik[202826]: time="2024-06-18T15:34:39Z" level=error msg="The ACME resolver \"letsencrypt\" is skipped from the resolvers list because: unable to get ACME account: json: cannot unmarshal array into Go value of type acme.StoredData"
Jun 18 15:38:53 localhost traefik[202941]: time="2024-06-18T15:38:53Z" level=error msg="The ACME resolver \"letsencrypt\" is skipped from the resolvers list because: unable to get ACME account: json: cannot unmarshal array into Go value of type acme.StoredData"
Jun 18 20:16:33 localhost traefik[204809]: time="2024-06-18T20:16:33Z" level=error msg="The ACME resolver \"letsencrypt\" is skipped from the resolvers list because: unable to get ACME account: json: cannot unmarshal array into Go value of type acme.StoredData"
Jun 18 20:17:48 localhost traefik[204845]: time="2024-06-18T20:17:48Z" level=error msg="The ACME resolver \"letsencrypt\" is skipped from the resolvers list because: unable to get ACME account: json: cannot unmarshal array into Go value of type acme.StoredData"
I’m sorry, but I’m afraid I’m out of ideas… the certificate request is obviously failing but I don’t know why.
Jul 05 15:09:09 localhost traefik[509211]: time="2024-07-05T15:09:09Z" level=error msg="The ACME resolver \"letsencrypt\" is skipped from the resolvers list because: unable to get ACME account: json: cannot unmarshal array into Go value of type acme.StoredData"
...
Jul 05 15:09:09 localhost traefik[509211]: time="2024-07-05T15:09:09Z" level=error msg="Error while creating certificate store: unable to find certificate for domains \"advertools.app,advertools.app\": falling back to the internal generated certificate" tlsStoreName=default
