Unable to restart the jhub

JupyterHub
1

Hi,
We are using HDP 3.0 cluster and Jhub was working fine.
I removed py2 kernal, restarted Jhub and it is fine there. I could not see py2 in jhub. It worked as expected.
Also I was asked to uninstall pycrypto python package. I could not uninstall using pip or conda so I removed only the pycrypto related packages from site-packages path.

I do not face any issues with python but when I tried to restart jupyterhub I am stuck. It is not restarting. I also reverted back and added my py2 kernel also did pip install pycrypto but still it is not restarting. I am not sure whether this issue is happened due to the above two actions or not.

P.S: I just removed these links 127.0.0.1:8081/hub/* from Log trace as I could post more than 2 links.

Log trace:

 /usr/share/miniconda2/envs/py36/bin/jupyterhub --debug
[D 2021-03-13 21:25:09.182 JupyterHub application:555] Looking for jupyterhub_config in /root
[I 2021-03-13 21:25:09.183 JupyterHub app:2120] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.0.0
[I 2021-03-13 21:25:09.183 JupyterHub app:2120] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-1.0.0
[I 2021-03-13 21:25:09.188 JupyterHub app:1257] Loading cookie_secret from /root/jupyterhub_cookie_secret
[D 2021-03-13 21:25:09.189 JupyterHub app:1424] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2021-03-13 21:25:09.211 JupyterHub orm:749] database schema version found: 4dc2d5a8c53c
[I 2021-03-13 21:25:09.216 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2021-03-13 21:25:09.217 JupyterHub app:1532] No admin users, admin interface will be unavailable.
[W 2021-03-13 21:25:09.217 JupyterHub app:1534] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-03-13 21:25:09.218 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
[D 2021-03-13 21:25:09.248 JupyterHub app:1926] Loaded users:

[I 2021-03-13 21:25:09.255 JupyterHub app:2337] Hub API listening on http://127.0.0.1:8081/hub/
[W 2021-03-13 21:25:09.256 JupyterHub proxy:642] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2021-03-13 21:25:09.256 JupyterHub proxy:645] Starting proxy @ http://:8000
[D 2021-03-13 21:25:09.257 JupyterHub proxy:646] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error']
[D 2021-03-13 21:25:09.260 JupyterHub proxy:561] Writing proxy pid file: jupyterhub-proxy.pid
21:25:09.444 [ConfigProxy] info: Proxying http://*:8000 to (no default)
21:25:09.447 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2021-03-13 21:25:09.582 JupyterHub proxy:681] Proxy started and appears to be up
[D 2021-03-13 21:25:09.583 JupyterHub proxy:314] Fetching routes to check
[D 2021-03-13 21:25:09.584 JupyterHub proxy:765] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
21:25:09.592 [ConfigProxy] info: 200 GET /api/routes
[I 2021-03-13 21:25:09.593 JupyterHub proxy:319] Checking routes
[I 2021-03-13 21:25:09.593 JupyterHub proxy:399] Adding default route for Hub: / => http://127.0.0.1:8081
[D 2021-03-13 21:25:09.593 JupyterHub proxy:765] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
21:25:09.595 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
21:25:09.596 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
21:25:09.597 [ConfigProxy] info: 201 POST /api/routes/
[I 2021-03-13 21:25:09.597 JupyterHub app:2422] JupyterHub is now running at http://:8000
[D 2021-03-13 21:30:09.598 JupyterHub proxy:765] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
21:30:09.600 [ConfigProxy] info: 200 GET /api/routes
[I 2021-03-13 21:30:09.601 JupyterHub proxy:319] Checking routes
[D 2021-03-13 21:35:09.598 JupyterHub proxy:765] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
21:35:09.600 [ConfigProxy] info: 200 GET /api/routes
[I 2021-03-13 21:35:09.600 JupyterHub proxy:319] Checking routes
[D 2021-03-13 21:40:09.599 JupyterHub proxy:765] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
21:40:09.601 [ConfigProxy] info: 200 GET /api/routes
[I 2021-03-13 21:40:09.601 JupyterHub proxy:319] Checking routes
[D 2021-03-13 21:45:09.599 JupyterHub proxy:765] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
21:45:09.600 [ConfigProxy] info: 200 GET /api/routes

Jupyter troubleshoot
$PATH:
/usr/share/miniconda2/bin
/usr/share/miniconda2/bin
/usr/lib64/qt-3.3/bin
/usr/local/bin
/usr/bin
/usr/local/sbin
/usr/sbin
/home/hdsuser/.local/bin
/home/hdsuser/bin

sys.path:
/usr/share/miniconda2/bin
/usr/share/miniconda2/lib/python27.zip
/usr/share/miniconda2/lib/python2.7
/usr/share/miniconda2/lib/python2.7/plat-linux2
/usr/share/miniconda2/lib/python2.7/lib-tk
/usr/share/miniconda2/lib/python2.7/lib-old
/usr/share/miniconda2/lib/python2.7/lib-dynload
/usr/share/miniconda2/lib/python2.7/site-packages

sys.executable:
/usr/share/miniconda2/bin/python

sys.version:
2.7.15 |Anaconda, Inc.| (default, May 1 2018, 23:32:55)
[GCC 7.2.0]

platform.platform():
Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-redhat-7.2-Maipo

which -a jupyter:
/usr/share/miniconda2/bin/jupyter
/usr/share/miniconda2/bin/jupyter

Can any one please look into this assist to resolve this?

2

According to your logs JupyterHub is running:

Could you give us more information about what isn’t working?

In future you can format your logs by using triple backticks: Creating and highlighting code blocks - GitHub Docs

3

Thanks @manics. The issue is Jhub web UI is not working. I am getting " This site can’t be reached" error message. Also sometimes when I restart jupyterHub I am getting the following error message
“ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<JupyterHub.launch_instance_async() done, defined at /usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/app.py:2477> exception=SystemExit(1,)>
Traceback (most recent call last):
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/app.py”, line 2480, in launch_instance_async
await self.start()
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/app.py”, line 2336, in start
self.http_server.listen(port, address=ip)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/tornado/tcpserver.py”, line 151, in listen
sockets = bind_sockets(port, address=address)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/tornado/netutil.py”, line 174, in bind_sockets
sock.bind(sockaddr)
OSError: [Errno 98] Address already in use”

In both the cases the jupyterhub web UI is not working.

P.S: I updated the log properly with ```

4

When I restart Jhub I kill the pids that uses the ports 8000/8001 where node runs and also kill 8081 pid (sudo kill -9 pid) . Then usually Jhub starts.
But now even after I kill the pids it is not working when I check the 8000 port I see this error message
dev env(not working)

ps -ef | grep 8000
root      4864  3519  5 13:00 ?        00:00:00 node /bin/configurable-http-proxy --ip  --port 8000 --api-ip 127.0.0.1 --api-port 8001 --error-target http://127.0.0.1:8081/hub/error

still getting " This site can’t be reached" error in the browser.

This issue is happening in Dev env. In Non prod Jhub is working fine. I just compared non prod 8000 port process status

non prod env(working fine)

ps -ef | grep 8000
root      8870  8855  0 09:52 ?        00:00:06 node /bin/configurable-http-proxy --ip  --port 8000 --api-ip 127.0.0.1 --api-port 8001 --error-target http://127.0.0.1:8081/hub/error --log-level debug --ssl-key /etc/security/certs/<host>_private.key --ssl-cert /etc/security/certs/<host>_p01eaedl_manulife_com_cert.cer

So the ssl-key and ssl-cert is missing in the dev env. is this the root cause of this issue? I can see these values are set correctly in jupyterhub_config.py in both the env.

Kindly check this issue once

5

It sounds like it’s working in one environment but not the other? Can you give us full details of how you installed JupyterHub, including your configuration files with secrets redacted, and as much information as you can about how the two systems differ?

6

@manics , I was able to fix the issue. The root cause of the issue is that the ssl certificate and key file access was not given correctly. Changing that using chmod command with appropriate rights fixed the issue. Now the Jhub is running fine.
Thank you !!!