Bug description
I followed the steps on the office website and installed the jupyterhub using conda on CentOS.
When I start the jupyterhub, I could not connect to the kernel. It stuck at the kernel starting, please wait... for minutes and eventually pop out a window says A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration..
Here is the console output in debug model:
(py) [root@host ~]# jupyterhub -f /etc/jupyterhub/jupyterhub_config.py --debug
[D 2020-06-14 17:35:15.233 JupyterHub application:556] Looking for /etc/jupyterhub/jupyterhub_config in /root
[D 2020-06-14 17:35:15.233 JupyterHub application:578] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2020-06-14 17:35:15.235 JupyterHub app:2240] Running JupyterHub version 1.1.0
[I 2020-06-14 17:35:15.235 JupyterHub app:2271] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.1.0
[I 2020-06-14 17:35:15.235 JupyterHub app:2271] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-1.1.0
[I 2020-06-14 17:35:15.235 JupyterHub app:2271] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.1.0
[I 2020-06-14 17:35:15.239 JupyterHub app:1349] Loading cookie_secret from /root/jupyterhub_cookie_secret
[D 2020-06-14 17:35:15.239 JupyterHub app:1516] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2020-06-14 17:35:15.253 JupyterHub orm:749] database schema version found: 4dc2d5a8c53c
[I 2020-06-14 17:35:15.257 JupyterHub proxy:461] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2020-06-14 17:35:15.280 JupyterHub app:1655] Not using whitelist. Any authenticated user will be allowed.
10
[D 2020-06-14 17:35:15.293 JupyterHub app:1914] Initializing spawners
[D 2020-06-14 17:35:15.297 JupyterHub app:2011] Loading state for wzh from db
[D 2020-06-14 17:35:15.297 JupyterHub app:2038] Loaded users:
wzh admin
[I 2020-06-14 17:35:15.297 JupyterHub app:2311] Initialized 0 spawners in 0.004 seconds
[W 2020-06-14 17:35:15.300 JupyterHub proxy:643] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2020-06-14 17:35:15.300 JupyterHub proxy:646] Starting proxy @ http://*:8000/
[D 2020-06-14 17:35:15.300 JupyterHub proxy:647] Proxy cmd: ['configurable-http-proxy', '--ip', '*', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error']
[D 2020-06-14 17:35:15.302 JupyterHub proxy:562] Writing proxy pid file: jupyterhub-proxy.pid
17:35:15.858 [ConfigProxy] warn: Interpreting ip='*' as all-interfaces. Preferred usage is 0.0.0.0 for all IPv4 or '' for all-interfaces.
17:35:15.865 [ConfigProxy] info: Proxying http://*:8000 to (no default)
17:35:15.865 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2020-06-14 17:35:15.966 JupyterHub proxy:682] Proxy started and appears to be up
[D 2020-06-14 17:35:15.967 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
17:35:15.975 [ConfigProxy] info: 200 GET /api/routes
[I 2020-06-14 17:35:15.976 JupyterHub app:2556] Hub API listening on http://127.0.0.1:8081/hub/
[D 2020-06-14 17:35:15.976 JupyterHub proxy:315] Fetching routes to check
[D 2020-06-14 17:35:15.976 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
17:35:15.977 [ConfigProxy] info: 200 GET /api/routes
[I 2020-06-14 17:35:15.977 JupyterHub proxy:320] Checking routes
[I 2020-06-14 17:35:15.977 JupyterHub proxy:400] Adding default route for Hub: / => http://127.0.0.1:8081
[D 2020-06-14 17:35:15.977 JupyterHub proxy:766] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
17:35:15.979 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
17:35:15.979 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
17:35:15.980 [ConfigProxy] info: 201 POST /api/routes/
[I 2020-06-14 17:35:15.980 JupyterHub app:2631] JupyterHub is now running at http://*:8000/
[D 2020-06-14 17:35:15.980 JupyterHub app:2233] It took 0.750 seconds for the Hub to start
[I 2020-06-14 17:35:23.019 JupyterHub log:174] 302 GET / -> /hub/ (@::ffff:192.168.10.161) 1.09ms
[D 2020-06-14 17:35:23.035 JupyterHub base:330] Refreshing auth for wzh
[D 2020-06-14 17:35:23.035 JupyterHub user:242] Creating <class 'jupyterhub.spawner.LocalProcessSpawner'> for wzh:
[I 2020-06-14 17:35:23.037 JupyterHub log:174] 302 GET /hub/ -> /hub/spawn (wzh@::ffff:192.168.10.161) 7.62ms
[D 2020-06-14 17:35:23.048 JupyterHub pages:195] Triggering spawn with default options for wzh
[D 2020-06-14 17:35:23.048 JupyterHub base:825] Initiating spawn for wzh
[D 2020-06-14 17:35:23.048 JupyterHub base:832] 0/100 concurrent spawns
[D 2020-06-14 17:35:23.048 JupyterHub base:837] 0 active servers
[W 2020-06-14 17:35:23.287 JupyterHub auth:956] Failed to open PAM session for wzh: [PAM Error 14] Cannot make/remove an entry for the specified session
[W 2020-06-14 17:35:23.287 JupyterHub auth:957] Disabling PAM sessions from now on.
[D 2020-06-14 17:35:23.290 JupyterHub user:556] Calling Spawner.start for wzh
[I 2020-06-14 17:35:23.291 JupyterHub spawner:1417] Spawning jupyterhub-singleuser --port=37932 --allow-root
[D 2020-06-14 17:35:23.299 JupyterHub spawner:1113] Polling subprocess every 30s
[I 2020-06-14 17:35:23.931 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[I 2020-06-14 17:35:23.936 JupyterHub log:174] 200 GET /hub/api (@127.0.0.1) 1.29ms
[I 2020-06-14 17:35:23.937 SingleUserNotebookApp notebookapp:1924] Serving notebooks from local directory: /home/wzh
[I 2020-06-14 17:35:23.937 SingleUserNotebookApp notebookapp:1924] The Jupyter Notebook is running at:
[I 2020-06-14 17:35:23.937 SingleUserNotebookApp notebookapp:1924] http://127.0.0.1:37932/user/wzh/
[I 2020-06-14 17:35:23.937 SingleUserNotebookApp notebookapp:1925] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2020-06-14 17:35:23.941 SingleUserNotebookApp singleuser:542] Updating Hub with activity every 300 seconds
[D 2020-06-14 17:35:23.947 JupyterHub base:285] Recording first activity for <APIToken('b8ed...', user='wzh')>
[D 2020-06-14 17:35:23.952 JupyterHub users:711] Activity for user wzh: 2020-06-14T09:35:23.695444Z
[D 2020-06-14 17:35:23.952 JupyterHub users:732] Activity on server wzh/: 2020-06-14T09:35:23.695444Z
[I 2020-06-14 17:35:23.955 JupyterHub log:174] 200 POST /hub/api/users/wzh/activity (wzh@127.0.0.1) 11.27ms
[I 2020-06-14 17:35:24.049 JupyterHub log:174] 302 GET /hub/spawn -> /hub/spawn-pending/wzh (wzh@::ffff:192.168.10.161) 1003.31ms
[I 2020-06-14 17:35:24.060 JupyterHub pages:347] wzh is pending spawn
[I 2020-06-14 17:35:24.085 JupyterHub log:174] 200 GET /hub/spawn-pending/wzh (wzh@::ffff:192.168.10.161) 27.43ms
[I 2020-06-14 17:35:24.705 SingleUserNotebookApp log:174] 302 GET /user/wzh/ -> /user/wzh/tree? (@127.0.0.1) 0.43ms
[D 2020-06-14 17:35:24.706 JupyterHub utils:218] Server at http://127.0.0.1:37932/user/wzh/ responded with 302
[D 2020-06-14 17:35:24.706 JupyterHub _version:60] jupyterhub and jupyterhub-singleuser both on version 1.1.0
[I 2020-06-14 17:35:24.706 JupyterHub base:855] User wzh took 1.658 seconds to start
[I 2020-06-14 17:35:24.706 JupyterHub proxy:262] Adding user wzh to proxy /user/wzh/ => http://127.0.0.1:37932
[D 2020-06-14 17:35:24.706 JupyterHub proxy:766] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/wzh
17:35:24.707 [ConfigProxy] info: Adding route /user/wzh -> http://127.0.0.1:37932
17:35:24.707 [ConfigProxy] info: Route added /user/wzh -> http://127.0.0.1:37932
17:35:24.707 [ConfigProxy] info: 201 POST /api/routes/user/wzh
[I 2020-06-14 17:35:24.708 JupyterHub users:609] Server wzh is ready
[I 2020-06-14 17:35:24.708 JupyterHub log:174] 200 GET /hub/api/users/wzh/server/progress (wzh@::ffff:192.168.10.161) 457.51ms
[I 2020-06-14 17:35:24.768 JupyterHub log:174] 302 GET /hub/spawn-pending/wzh -> /user/wzh/ (wzh@::ffff:192.168.10.161) 1.78ms
[I 2020-06-14 17:35:24.776 SingleUserNotebookApp log:174] 302 GET /user/wzh/ -> /user/wzh/tree? (@::ffff:192.168.10.161) 0.43ms
[I 2020-06-14 17:35:24.784 SingleUserNotebookApp log:174] 302 GET /user/wzh/tree? -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-wzh&redirect_uri=%2Fuser%2Fwzh%2Foauth_callback&response_type=code&state=[secret] (@::ffff:192.168.10.161) 1.68ms
[D 2020-06-14 17:35:24.793 JupyterHub provider:414] Validating client id jupyterhub-user-wzh
[D 2020-06-14 17:35:24.794 JupyterHub provider:492] validate_redirect_uri: client_id=jupyterhub-user-wzh, redirect_uri=/user/wzh/oauth_callback
[D 2020-06-14 17:35:24.796 JupyterHub auth:247] Skipping oauth confirmation for <User(wzh 1/1 running)> accessing Server at /user/wzh/
[D 2020-06-14 17:35:24.796 JupyterHub provider:414] Validating client id jupyterhub-user-wzh
[D 2020-06-14 17:35:24.797 JupyterHub provider:492] validate_redirect_uri: client_id=jupyterhub-user-wzh, redirect_uri=/user/wzh/oauth_callback
[D 2020-06-14 17:35:24.798 JupyterHub provider:241] Saving authorization code jupyterhub-user-wzh, mTT..., (), {}
[I 2020-06-14 17:35:24.802 JupyterHub log:174] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-wzh&redirect_uri=%2Fuser%2Fwzh%2Foauth_callback&response_type=code&state=[secret] -> /user/wzh/oauth_callback?code=[secret]&state=[secret] (wzh@::ffff:192.168.10.161) 11.47ms
[D 2020-06-14 17:35:24.819 JupyterHub provider:60] authenticate_client <oauthlib.Request url="/hub/api/oauth2/token", http_method="POST", headers="{'Host': '127.0.0.1:8081', 'User-Agent': 'python-requests/2.23.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': '<SANITIZED>', 'Content-Length': '186'}", body="client_id=jupyterhub-user-wzh&client_secret=b8ed85e5f1184e67a95bf4be923fabb9&grant_type=authorization_code&code=mTTpDCv6ZU6eFWKvzkc8b20hnLsl1U&redirect_uri=%2Fuser%2Fwzh%2Foauth_callback">
[D 2020-06-14 17:35:24.826 JupyterHub provider:119] confirm_redirect_uri: client_id=jupyterhub-user-wzh, redirect_uri=/user/wzh/oauth_callback
[D 2020-06-14 17:35:24.826 JupyterHub provider:339] Saving bearer token {'access_token': 'REDACTED', 'expires_in': 3600, 'token_type': 'Bearer', 'scope': 'identify', 'refresh_token': 'REDACTED'}
[D 2020-06-14 17:35:24.830 JupyterHub provider:194] Deleting oauth code mTT... for jupyterhub-user-wzh
[I 2020-06-14 17:35:24.834 JupyterHub log:174] 200 POST /hub/api/oauth2/token (wzh@127.0.0.1) 17.64ms
[I 2020-06-14 17:35:24.845 JupyterHub log:174] 200 GET /hub/api/authorizations/token/[secret] (wzh@127.0.0.1) 9.11ms
[I 2020-06-14 17:35:24.846 SingleUserNotebookApp auth:981] Logged-in user {'kind': 'user', 'name': 'wzh', 'admin': True, 'groups': [], 'server': '/user/wzh/', 'pending': None, 'created': '2020-06-14T05:37:24.166772Z', 'last_activity': '2020-06-14T09:35:24.842288Z', 'servers': None}
[I 2020-06-14 17:35:24.846 SingleUserNotebookApp log:174] 302 GET /user/wzh/oauth_callback?code=[secret]&state=[secret] -> /user/wzh/tree? (@::ffff:192.168.10.161) 33.42ms
[I 2020-06-14 17:35:24.911 SingleUserNotebookApp log:174] 200 GET /user/wzh/tree? (wzh@::ffff:192.168.10.161) 54.89ms
[I 2020-06-14 17:35:25.041 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/config/tree?_=1592127325001 (wzh@::ffff:192.168.10.161) 2.45ms
[I 2020-06-14 17:35:25.044 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/config/common?_=1592127325002 (wzh@::ffff:192.168.10.161) 2.29ms
[I 2020-06-14 17:35:25.045 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/terminals?_=1592127325003 (wzh@::ffff:192.168.10.161) 0.71ms
[I 2020-06-14 17:35:25.050 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/sessions?_=1592127325004 (wzh@::ffff:192.168.10.161) 1.26ms
[I 2020-06-14 17:35:25.052 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/terminals?_=1592127325005 (wzh@::ffff:192.168.10.161) 0.71ms
[I 2020-06-14 17:35:25.073 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/contents?type=directory&_=1592127325006 (wzh@::ffff:192.168.10.161) 1.96ms
[I 2020-06-14 17:35:27.948 SingleUserNotebookApp handlers:153] Creating new notebook in
[I 2020-06-14 17:35:27.954 SingleUserNotebookApp log:174] 201 POST /user/wzh/api/contents (wzh@::ffff:192.168.10.161) 6.79ms
[I 2020-06-14 17:35:28.242 SingleUserNotebookApp log:174] 200 GET /user/wzh/notebooks/Untitled.ipynb?kernel_name=python3 (wzh@::ffff:192.168.10.161) 267.92ms
[I 2020-06-14 17:35:28.543 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/config/notebook?_=1592127328465 (wzh@::ffff:192.168.10.161) 2.23ms
[I 2020-06-14 17:35:28.546 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/config/common?_=1592127328466 (wzh@::ffff:192.168.10.161) 2.18ms
[I 2020-06-14 17:35:28.595 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/contents/Untitled.ipynb?type=notebook&_=1592127328467 (wzh@::ffff:192.168.10.161) 2.12ms
[I 2020-06-14 17:35:28.698 SingleUserNotebookApp kernelmanager:172] Kernel started: 2b15855d-c4ca-428b-856b-6ec26b3d808c
[I 2020-06-14 17:35:28.699 SingleUserNotebookApp log:174] 201 POST /user/wzh/api/sessions (wzh@::ffff:192.168.10.161) 22.14ms
[I 2020-06-14 17:35:28.701 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/contents/Untitled.ipynb/checkpoints?_=1592127328468 (wzh@::ffff:192.168.10.161) 1.28ms
[W 2020-06-14 17:35:28.710 SingleUserNotebookApp log:174] 404 GET /user/wzh/nbextensions/widgets/notebook/js/extension.js?v=20200614173523 (wzh@::ffff:192.168.10.161) 7.90ms
[I 2020-06-14 17:35:35.015 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/terminals?_=1592124556936 (wzh@::ffff:192.168.10.161) 0.98ms
[I 2020-06-14 17:35:35.016 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/sessions?_=1592124556935 (wzh@::ffff:192.168.10.161) 0.98ms
[I 2020-06-14 17:35:35.025 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/contents?type=directory&_=1592124556937 (wzh@::ffff:192.168.10.161) 1.39ms
[I 2020-06-14 17:37:28.697 SingleUserNotebookApp log:174] 200 GET /user/wzh/api/contents/Untitled.ipynb?content=0&_=1592127328469 (wzh@::ffff:192.168.10.161) 1.03ms
[I 2020-06-14 17:37:28.708 SingleUserNotebookApp handlers:164] Saving file at /Untitled.ipynb
[I 2020-06-14 17:37:28.713 SingleUserNotebookApp log:174] 200 PUT /user/wzh/api/contents/Untitled.ipynb (wzh@::ffff:192.168.10.161) 6.16ms
^[[1;2D
personal set up
Esxi 6.5
CentOS 7
conda 4.8.2
jupyterhub 1.1.0
approach attempted:
I searched online about this and I had attempted following approaches and none of them worked:
- change browser
- clear catch
- downgrade tornado to 5.1.1