Dear Sirs,
Everything was OK with our JH 1.5.0 (installed via TLJH) but recently our admin added another admin this the following commands:
sudo tljh-config add-item users.admin otheradmin
sudo tljh-config reload
After that I’m unable to start any server on the JH.
Here are the logs and my config:
> journalctl -f -u jupyterhub
Jun 27 17:02:40 student python3[744]: [I 2023-06-27 17:02:40.046 JupyterHub log:189] 200 GET /hub/admin (admin@192.168.2.7.7) 10.09ms
Jun 27 17:02:57 student python3[744]: [I 2023-06-27 17:02:57.152 JupyterHub provider:574] Creating oauth client jupyterhub-user-admin
Jun 27 17:02:57 student python3[744]: [I 2023-06-27 17:02:57.412 JupyterHub systemdspawner:217] user:admin Unit jupyter-admin in a failed state. Resetting state.
Jun 27 17:02:57 student python3[2755]: Running as unit: jupyter-admin.service
Jun 27 17:03:07 student python3[744]: [W 2023-06-27 17:03:07.044 JupyterHub web:1787] 500 POST /hub/api/users/admin/server (192.168.2.7.7): Spawner failed to start [status=1]. The logs for admin may contain details.
Jun 27 17:03:07 student python3[744]: [E 2023-06-27 17:03:07.047 JupyterHub log:181] {
Jun 27 17:03:07 student python3[744]: "Host": "jupyterhub.local",
Jun 27 17:03:07 student python3[744]: "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
Jun 27 17:03:07 student python3[744]: "Content-Length": "0",
Jun 27 17:03:07 student python3[744]: "Accept": "*/*",
Jun 27 17:03:07 student python3[744]: "Accept-Encoding": "gzip, deflate",
Jun 27 17:03:07 student python3[744]: "Accept-Language": "ru,en-US;q=0.9,en;q=0.8",
Jun 27 17:03:07 student python3[744]: "Content-Type": "application/json",
Jun 27 17:03:07 student python3[744]: "Cookie": "jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
Jun 27 17:03:07 student python3[744]: "Origin": "http://jupyterhub.local",
Jun 27 17:03:07 student python3[744]: "Referer": "http://jupyterhub.local/hub/admin",
Jun 27 17:03:07 student python3[744]: "X-Forwarded-For": "192.168.2.7",
Jun 27 17:03:07 student python3[744]: "X-Forwarded-Host": "jupyterhub.local",
Jun 27 17:03:07 student python3[744]: "X-Forwarded-Port": "80",
Jun 27 17:03:07 student python3[744]: "X-Forwarded-Proto": "http",
Jun 27 17:03:07 student python3[744]: "X-Forwarded-Server": "student",
Jun 27 17:03:07 student python3[744]: "X-Real-Ip": "192.168.2.7",
Jun 27 17:03:07 student python3[744]: "X-Requested-With": "XMLHttpRequest"
Jun 27 17:03:07 student python3[744]: }
Jun 27 17:03:07 student python3[744]: [E 2023-06-27 17:03:07.047 JupyterHub log:189] 500 POST /hub/api/users/admin/server (admin@192.168.2.7.7) 10027.35ms
Jun 27 17:03:10 student python3[744]: [I 2023-06-27 17:03:10.833 JupyterHub log:189] 200 GET /hub/api/ (cull-idle@127.0.0.1) 137.39ms
Jun 27 17:03:10 student python3[744]: [I 2023-06-27 17:03:10.844 JupyterHub log:189] 200 GET /hub/api/users?state=[secret] (cull-idle@127.0.0.1) 6.71ms
Jun 27 17:03:14 student python3[744]: [I 2023-06-27 17:03:14.399 JupyterHub log:189] 200 GET /hub/admin (admin@192.168.2.7.7) 15.22ms
Jun 27 17:03:25 student python3[744]: [W 2023-06-27 17:03:25.121 JupyterHub user:767] admin's server never showed up at http://127.0.0.1:38725/user/admin/ after 30 seconds. Giving up
Jun 27 17:03:25 student python3[744]: [E 2023-06-27 17:03:25.887 JupyterHub gen:623] Exception in Future <Task finished name='Task-1582' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/handlers/base.py:900> exception=TimeoutError("Server at http://127.0.0.1:38725/user/admin/ didn't respond in 30 seconds")> after timeout
Jun 27 17:03:25 student python3[744]: Traceback (most recent call last):
Jun 27 17:03:25 student python3[744]: File "/opt/tljh/hub/lib/python3.8/site-packages/tornado/gen.py", line 618, in error_callback
Jun 27 17:03:25 student python3[744]: future.result()
Jun 27 17:03:25 student python3[744]: File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
Jun 27 17:03:25 student python3[744]: await spawn_future
Jun 27 17:03:25 student python3[744]: File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/user.py", line 748, in spawn
Jun 27 17:03:25 student python3[744]: await self._wait_up(spawner)
Jun 27 17:03:25 student python3[744]: File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/user.py", line 795, in _wait_up
Jun 27 17:03:25 student python3[744]: raise e
Jun 27 17:03:25 student python3[744]: File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/user.py", line 762, in _wait_up
Jun 27 17:03:25 student python3[744]: resp = await server.wait_up(
Jun 27 17:03:25 student python3[744]: File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/utils.py", line 236, in wait_for_http_server
Jun 27 17:03:25 student python3[744]: re = await exponential_backoff(
Jun 27 17:03:25 student python3[744]: File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/utils.py", line 184, in exponential_backoff
Jun 27 17:03:25 student python3[744]: raise TimeoutError(fail_message)
Jun 27 17:03:25 student python3[744]: TimeoutError: Server at http://127.0.0.1:38725/user/admin/ didn't respond in 30 seconds
> journalctl -f -u jupyter-admin
-- Logs begin at Wed 2022-04-27 08:33:01 UTC. --
Jun 27 17:02:58 student bash[2757]: from jupyter_client.jsonutil import extract_dates
Jun 27 17:02:58 student bash[2757]: File "/opt/tljh/user/lib/python3.9/site-packages/jupyter_client/jsonutil.py", line 15, in <module>
Jun 27 17:02:58 student bash[2757]: from dateutil.parser import parse as _dateutil_parse # type: ignore
Jun 27 17:02:58 student bash[2757]: File "/opt/tljh/user/lib/python3.9/site-packages/dateutil/parser/__init__.py", line 2, in <module>
Jun 27 17:02:58 student bash[2757]: from ._parser import parse, parser, parserinfo, ParserError
Jun 27 17:02:58 student bash[2757]: File "/opt/tljh/user/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 42, in <module>
Jun 27 17:02:58 student bash[2757]: import six
Jun 27 17:02:58 student bash[2757]: ModuleNotFoundError: No module named 'six'
Jun 27 17:02:58 student systemd[1]: jupyter-admin.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 17:02:58 student systemd[1]: jupyter-admin.service: Failed with result 'exit-code'.
Jun 27 17:05:48 student systemd[1]: Started /bin/bash -c cd /home/jupyter-admin && exec jupyterhub-singleuser --port=40451.
Jun 27 17:05:48 student bash[2777]: /opt/tljh/user/lib/python3.9/site-packages/jupyter_client/__init__.py:23: UserWarning: Could not import submodules
Jun 27 17:05:48 student bash[2777]: warnings.warn("Could not import submodules")
Jun 27 17:05:48 student bash[2777]: Traceback (most recent call last):
Jun 27 17:05:48 student bash[2777]: File "/opt/tljh/user/bin/jupyterhub-singleuser", line 5, in <module>
Jun 27 17:05:48 student bash[2777]: from jupyterhub.singleuser import main
Jun 27 17:05:48 student bash[2777]: File "/opt/tljh/user/lib/python3.9/site-packages/jupyterhub/singleuser/__init__.py", line 5, in <module>
Jun 27 17:05:48 student bash[2777]: from .app import main
Jun 27 17:05:48 student bash[2777]: File "/opt/tljh/user/lib/python3.9/site-packages/jupyterhub/singleuser/app.py", line 16, in <module>
Jun 27 17:05:48 student bash[2777]: App = import_item(JUPYTERHUB_SINGLEUSER_APP)
Jun 27 17:05:48 student bash[2777]: File "/opt/tljh/user/lib/python3.9/site-packages/traitlets/utils/importstring.py", line 30, in import_item
Jun 27 17:05:48 student bash[2777]: module = __import__(package, fromlist=[obj])
Jun 27 17:05:48 student bash[2777]: File "/opt/tljh/user/lib/python3.9/site-packages/notebook/notebookapp.py", line 77, in <module>
Jun 27 17:05:48 student bash[2777]: from .services.kernels.kernelmanager import MappingKernelManager, AsyncMappingKernelManager
Jun 27 17:05:48 student bash[2777]: File "/opt/tljh/user/lib/python3.9/site-packages/notebook/services/kernels/kernelmanager.py", line 18, in <module>
Jun 27 17:05:48 student bash[2777]: from jupyter_client.session import Session
Jun 27 17:05:48 student bash[2777]: File "/opt/tljh/user/lib/python3.9/site-packages/jupyter_client/session.py", line 54, in <module>
Jun 27 17:05:48 student bash[2777]: from jupyter_client.jsonutil import extract_dates
Jun 27 17:05:48 student bash[2777]: File "/opt/tljh/user/lib/python3.9/site-packages/jupyter_client/jsonutil.py", line 15, in <module>
Jun 27 17:05:48 student bash[2777]: from dateutil.parser import parse as _dateutil_parse # type: ignore
Jun 27 17:05:48 student bash[2777]: File "/opt/tljh/user/lib/python3.9/site-packages/dateutil/parser/__init__.py", line 2, in <module>
Jun 27 17:05:48 student bash[2777]: from ._parser import parse, parser, parserinfo, ParserError
Jun 27 17:05:48 student bash[2777]: File "/opt/tljh/user/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 42, in <module>
Jun 27 17:05:48 student bash[2777]: import six
Jun 27 17:05:48 student bash[2777]: ModuleNotFoundError: No module named 'six'
Jun 27 17:05:48 student systemd[1]: jupyter-admin.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 17:05:48 student systemd[1]: jupyter-admin.service: Failed with result 'exit-code'.
> tljh-config show
users:
admin:
- admin
- otheradmin
auth:
FirstUseAuthenticator:
create_users: true
Could you help me please to understand what happened and how to fix it?
Thank you very much!