API request failed (500): Spawner failed to start [status=1]

Dear Sirs,

Everything was OK with our JH 1.5.0 (installed via TLJH) but recently our admin added another admin this the following commands:

sudo tljh-config add-item users.admin otheradmin
sudo tljh-config reload

After that I’m unable to start any server on the JH.

Here are the logs and my config:

> journalctl -f -u jupyterhub
Jun 27 17:02:40 student python3[744]: [I 2023-06-27 17:02:40.046 JupyterHub log:189] 200 GET /hub/admin (admin@192.168.2.7.7) 10.09ms
Jun 27 17:02:57 student python3[744]: [I 2023-06-27 17:02:57.152 JupyterHub provider:574] Creating oauth client jupyterhub-user-admin
Jun 27 17:02:57 student python3[744]: [I 2023-06-27 17:02:57.412 JupyterHub systemdspawner:217] user:admin Unit jupyter-admin in a failed state. Resetting state.
Jun 27 17:02:57 student python3[2755]: Running as unit: jupyter-admin.service
Jun 27 17:03:07 student python3[744]: [W 2023-06-27 17:03:07.044 JupyterHub web:1787] 500 POST /hub/api/users/admin/server (192.168.2.7.7): Spawner failed to start [status=1]. The logs for admin may contain details.
Jun 27 17:03:07 student python3[744]: [E 2023-06-27 17:03:07.047 JupyterHub log:181] {
Jun 27 17:03:07 student python3[744]:       "Host": "jupyterhub.local",
Jun 27 17:03:07 student python3[744]:       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
Jun 27 17:03:07 student python3[744]:       "Content-Length": "0",
Jun 27 17:03:07 student python3[744]:       "Accept": "*/*",
Jun 27 17:03:07 student python3[744]:       "Accept-Encoding": "gzip, deflate",
Jun 27 17:03:07 student python3[744]:       "Accept-Language": "ru,en-US;q=0.9,en;q=0.8",
Jun 27 17:03:07 student python3[744]:       "Content-Type": "application/json",
Jun 27 17:03:07 student python3[744]:       "Cookie": "jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
Jun 27 17:03:07 student python3[744]:       "Origin": "http://jupyterhub.local",
Jun 27 17:03:07 student python3[744]:       "Referer": "http://jupyterhub.local/hub/admin",
Jun 27 17:03:07 student python3[744]:       "X-Forwarded-For": "192.168.2.7",
Jun 27 17:03:07 student python3[744]:       "X-Forwarded-Host": "jupyterhub.local",
Jun 27 17:03:07 student python3[744]:       "X-Forwarded-Port": "80",
Jun 27 17:03:07 student python3[744]:       "X-Forwarded-Proto": "http",
Jun 27 17:03:07 student python3[744]:       "X-Forwarded-Server": "student",
Jun 27 17:03:07 student python3[744]:       "X-Real-Ip": "192.168.2.7",
Jun 27 17:03:07 student python3[744]:       "X-Requested-With": "XMLHttpRequest"
Jun 27 17:03:07 student python3[744]:     }
Jun 27 17:03:07 student python3[744]: [E 2023-06-27 17:03:07.047 JupyterHub log:189] 500 POST /hub/api/users/admin/server (admin@192.168.2.7.7) 10027.35ms
Jun 27 17:03:10 student python3[744]: [I 2023-06-27 17:03:10.833 JupyterHub log:189] 200 GET /hub/api/ (cull-idle@127.0.0.1) 137.39ms
Jun 27 17:03:10 student python3[744]: [I 2023-06-27 17:03:10.844 JupyterHub log:189] 200 GET /hub/api/users?state=[secret] (cull-idle@127.0.0.1) 6.71ms
Jun 27 17:03:14 student python3[744]: [I 2023-06-27 17:03:14.399 JupyterHub log:189] 200 GET /hub/admin (admin@192.168.2.7.7) 15.22ms
Jun 27 17:03:25 student python3[744]: [W 2023-06-27 17:03:25.121 JupyterHub user:767] admin's server never showed up at http://127.0.0.1:38725/user/admin/ after 30 seconds. Giving up
Jun 27 17:03:25 student python3[744]: [E 2023-06-27 17:03:25.887 JupyterHub gen:623] Exception in Future <Task finished name='Task-1582' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/handlers/base.py:900> exception=TimeoutError("Server at http://127.0.0.1:38725/user/admin/ didn't respond in 30 seconds")> after timeout
Jun 27 17:03:25 student python3[744]:     Traceback (most recent call last):
Jun 27 17:03:25 student python3[744]:       File "/opt/tljh/hub/lib/python3.8/site-packages/tornado/gen.py", line 618, in error_callback
Jun 27 17:03:25 student python3[744]:         future.result()
Jun 27 17:03:25 student python3[744]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
Jun 27 17:03:25 student python3[744]:         await spawn_future
Jun 27 17:03:25 student python3[744]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/user.py", line 748, in spawn
Jun 27 17:03:25 student python3[744]:         await self._wait_up(spawner)
Jun 27 17:03:25 student python3[744]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/user.py", line 795, in _wait_up
Jun 27 17:03:25 student python3[744]:         raise e
Jun 27 17:03:25 student python3[744]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/user.py", line 762, in _wait_up
Jun 27 17:03:25 student python3[744]:         resp = await server.wait_up(
Jun 27 17:03:25 student python3[744]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/utils.py", line 236, in wait_for_http_server
Jun 27 17:03:25 student python3[744]:         re = await exponential_backoff(
Jun 27 17:03:25 student python3[744]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/utils.py", line 184, in exponential_backoff
Jun 27 17:03:25 student python3[744]:         raise TimeoutError(fail_message)
Jun 27 17:03:25 student python3[744]:     TimeoutError: Server at http://127.0.0.1:38725/user/admin/ didn't respond in 30 seconds


> journalctl -f -u jupyter-admin
-- Logs begin at Wed 2022-04-27 08:33:01 UTC. --
Jun 27 17:02:58 student bash[2757]:     from jupyter_client.jsonutil import extract_dates
Jun 27 17:02:58 student bash[2757]:   File "/opt/tljh/user/lib/python3.9/site-packages/jupyter_client/jsonutil.py", line 15, in <module>
Jun 27 17:02:58 student bash[2757]:     from dateutil.parser import parse as _dateutil_parse  # type: ignore
Jun 27 17:02:58 student bash[2757]:   File "/opt/tljh/user/lib/python3.9/site-packages/dateutil/parser/__init__.py", line 2, in <module>
Jun 27 17:02:58 student bash[2757]:     from ._parser import parse, parser, parserinfo, ParserError
Jun 27 17:02:58 student bash[2757]:   File "/opt/tljh/user/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 42, in <module>
Jun 27 17:02:58 student bash[2757]:     import six
Jun 27 17:02:58 student bash[2757]: ModuleNotFoundError: No module named 'six'
Jun 27 17:02:58 student systemd[1]: jupyter-admin.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 17:02:58 student systemd[1]: jupyter-admin.service: Failed with result 'exit-code'.
Jun 27 17:05:48 student systemd[1]: Started /bin/bash -c cd /home/jupyter-admin && exec jupyterhub-singleuser --port=40451.
Jun 27 17:05:48 student bash[2777]: /opt/tljh/user/lib/python3.9/site-packages/jupyter_client/__init__.py:23: UserWarning: Could not import submodules
Jun 27 17:05:48 student bash[2777]:   warnings.warn("Could not import submodules")
Jun 27 17:05:48 student bash[2777]: Traceback (most recent call last):
Jun 27 17:05:48 student bash[2777]:   File "/opt/tljh/user/bin/jupyterhub-singleuser", line 5, in <module>
Jun 27 17:05:48 student bash[2777]:     from jupyterhub.singleuser import main
Jun 27 17:05:48 student bash[2777]:   File "/opt/tljh/user/lib/python3.9/site-packages/jupyterhub/singleuser/__init__.py", line 5, in <module>
Jun 27 17:05:48 student bash[2777]:     from .app import main
Jun 27 17:05:48 student bash[2777]:   File "/opt/tljh/user/lib/python3.9/site-packages/jupyterhub/singleuser/app.py", line 16, in <module>
Jun 27 17:05:48 student bash[2777]:     App = import_item(JUPYTERHUB_SINGLEUSER_APP)
Jun 27 17:05:48 student bash[2777]:   File "/opt/tljh/user/lib/python3.9/site-packages/traitlets/utils/importstring.py", line 30, in import_item
Jun 27 17:05:48 student bash[2777]:     module = __import__(package, fromlist=[obj])
Jun 27 17:05:48 student bash[2777]:   File "/opt/tljh/user/lib/python3.9/site-packages/notebook/notebookapp.py", line 77, in <module>
Jun 27 17:05:48 student bash[2777]:     from .services.kernels.kernelmanager import MappingKernelManager, AsyncMappingKernelManager
Jun 27 17:05:48 student bash[2777]:   File "/opt/tljh/user/lib/python3.9/site-packages/notebook/services/kernels/kernelmanager.py", line 18, in <module>
Jun 27 17:05:48 student bash[2777]:     from jupyter_client.session import Session
Jun 27 17:05:48 student bash[2777]:   File "/opt/tljh/user/lib/python3.9/site-packages/jupyter_client/session.py", line 54, in <module>
Jun 27 17:05:48 student bash[2777]:     from jupyter_client.jsonutil import extract_dates
Jun 27 17:05:48 student bash[2777]:   File "/opt/tljh/user/lib/python3.9/site-packages/jupyter_client/jsonutil.py", line 15, in <module>
Jun 27 17:05:48 student bash[2777]:     from dateutil.parser import parse as _dateutil_parse  # type: ignore
Jun 27 17:05:48 student bash[2777]:   File "/opt/tljh/user/lib/python3.9/site-packages/dateutil/parser/__init__.py", line 2, in <module>
Jun 27 17:05:48 student bash[2777]:     from ._parser import parse, parser, parserinfo, ParserError
Jun 27 17:05:48 student bash[2777]:   File "/opt/tljh/user/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 42, in <module>
Jun 27 17:05:48 student bash[2777]:     import six
Jun 27 17:05:48 student bash[2777]: ModuleNotFoundError: No module named 'six'
Jun 27 17:05:48 student systemd[1]: jupyter-admin.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 17:05:48 student systemd[1]: jupyter-admin.service: Failed with result 'exit-code'.


> tljh-config show
users:
  admin:
  - admin
  - otheradmin
auth:
  FirstUseAuthenticator:
    create_users: true

Could you help me please to understand what happened and how to fix it?

Thank you very much!

Did you or your admin do anything else apart from adding another admin? It sounds like your user environment has become corrupted, the set of packages is no longer consistent.