This is the error log leading up to the crash (I don’t know of a better formatting option so apologies for the wall of text).
Jun 10 20:09:36 tljh systemd[1]: Reloading.
Jun 10 20:09:40 tljh snapd[1312]: storehelpers.go:438: cannot refresh snap "google-cloud-sdk": snap has no updates available
Jun 10 20:09:43 tljh bash[3571]: [I 2020-06-10 20:09:43.894 SingleUserNotebookApp log:174] 200 GET /user/product/notebooks/energy-minimization/app_for_product_team.ipynb (product@24.4.36.157) 5561.98ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:44.100 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.5.86.165) 65.22ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:44.101 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 45.76ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:44.474 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.5.86.165) 9.32ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:44.502 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 9.13ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:45.076 SingleUserNotebookApp log:174] 200 GET /user/product/api/config/notebook?_=1591819784695 (product@24.4.36.157) 174.29ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:45.080 SingleUserNotebookApp log:174] 200 GET /user/product/api/config/common?_=1591819784696 (product@24.4.36.157) 2.60ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:45.475 SingleUserNotebookApp log:174] 200 GET /user/product/api/contents/energy-minimization/app_for_product_team.ipynb?type=notebook&_=1591819784697 (product@24.4.36.157) 274.34ms
Jun 10 20:09:46 tljh bash[3571]: [I 2020-06-10 20:09:46.462 SingleUserNotebookApp log:174] 200 GET /user/product/api/contents/energy-minimization/app_for_product_team.ipynb/checkpoints?_=1591819784698 (product@24.4.36.157) 12.30ms
Jun 10 20:09:46 tljh bash[3571]: [I 2020-06-10 20:09:46.866 SingleUserNotebookApp kernelmanager:172] Kernel started: 8b8178b9-ad66-4a7a-9180-4b211f7dca3c
Jun 10 20:09:47 tljh bash[3571]: [I 2020-06-10 20:09:46.868 SingleUserNotebookApp log:174] 201 POST /user/product/api/sessions (product@24.4.36.157) 453.22ms
Jun 10 20:09:48 tljh bash[3571]: [I 2020-06-10 20:09:48.875 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 479.24ms
Jun 10 20:09:50 tljh python3[3362]: [I 2020-06-10 20:09:50.277 JupyterHub log:174] 200 GET /hub/api/authorizations/token/[secret] (product@127.0.0.1) 752.71ms
Jun 10 20:09:53 tljh python3[3362]: [I 2020-06-10 20:09:50.383 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 38.48ms
Jun 10 20:09:55 tljh bash[3571]: [I 2020-06-10 20:09:50.335 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@207.5.45.241) 1417.68ms
Jun 10 20:09:56 tljh bash[3571]: [I 2020-06-10 20:09:50.372 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.5.86.165) 18.71ms
Jun 10 20:09:57 tljh bash[3571]: [I 2020-06-10 20:09:50.373 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 18.37ms
Jun 10 20:09:59 tljh bash[3571]: [I 2020-06-10 20:09:52.067 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@207.5.45.241) 73.82ms
Jun 10 20:10:02 tljh bash[3571]: [I 2020-06-10 20:09:52.111 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 81.83ms
Jun 10 20:10:07 tljh bash[3571]: [I 2020-06-10 20:09:53.494 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 89.26ms
Jun 10 20:10:33 tljh bash[3571]: [I 2020-06-10 20:09:53.653 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 7.91ms
Jun 10 20:10:58 tljh bash[3571]: [I 2020-06-10 20:09:54.439 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.5.86.165) 38.17ms
Jun 10 20:11:04 tljh bash[3571]: [I 2020-06-10 20:09:55.106 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@207.5.45.241) 10.36ms
Jun 10 20:11:09 tljh bash[3571]: [W 2020-06-10 20:10:50.572 SingleUserNotebookApp handlers:229] Timeout waiting for kernel_info reply from 8b8178b9-ad66-4a7a-9180-4b211f7dca3c
Jun 10 20:11:14 tljh bash[3571]: [I 2020-06-10 20:10:55.788 SingleUserNotebookApp log:174] 101 GET /user/product/api/kernels/8b8178b9-ad66-4a7a-9180-4b211f7dca3c/channels?session_id=4b576c19d95d4d6c8caa2044b9d7bf85 (product@24.4.36.157) 65422.68ms
Jun 10 20:11:20 tljh bash[3571]: [I 2020-06-10 20:11:04.496 SingleUserNotebookApp kernelmanager:217] Starting buffering for 8b8178b9-ad66-4a7a-9180-4b211f7dca3c:4b576c19d95d4d6c8caa2044b9d7bf85
Jun 10 20:11:24 tljh python3[3362]: [I 2020-06-10 20:10:58.718 JupyterHub proxy:320] Checking routes
Jun 10 20:11:35 tljh bash[3571]: [W 2020-06-10 20:11:16.635 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 90578 ms.
Jun 10 20:11:42 tljh bash[3571]: [W 2020-06-10 20:11:23.457 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 90177 ms.
Jun 10 20:11:46 tljh bash[3571]: [I 2020-06-10 20:11:23.582 SingleUserNotebookApp kernelmanager:217] Starting buffering for 7fd789ac-a766-4761-9732-01a1f5456ed4:c2ee08980b394c1f80e2c67a23ec4476
Jun 10 20:11:49 tljh bash[3571]: [I 2020-06-10 20:11:28.582 SingleUserNotebookApp kernelmanager:217] Starting buffering for 657196c0-af73-4f86-926d-07686360ea09:568d4f39288a44bf9d5b7edd11e29ec7
Jun 10 20:11:52 tljh python3[3362]: [E 200610 20:11:14 ioloop:763] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f6300ef2828>>, <Future finished exception=HTTP 599: Operation timed out after 20513 milliseconds with 0 bytes received>)
Jun 10 20:12:01 tljh python3[3362]: Traceback (most recent call last):
Jun 10 20:12:08 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
Jun 10 20:12:17 tljh python3[3362]: ret = callback()
Jun 10 20:12:27 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
Jun 10 20:12:39 tljh python3[3362]: future.result()
Jun 10 20:12:48 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
Jun 10 20:12:54 tljh python3[3362]: yielded = self.gen.throw(*exc_info) # type: ignore
Jun 10 20:12:59 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 121, in cull_idle
Jun 10 20:13:05 tljh python3[3362]: resp = yield fetch(req)
Jun 10 20:13:14 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:13:30 tljh python3[3362]: value = future.result()
Jun 10 20:13:47 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
Jun 10 20:14:01 tljh python3[3362]: yielded = self.gen.send(value)
Jun 10 20:14:24 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 114, in fetch
Jun 10 20:14:43 tljh python3[3362]: return (yield client.fetch(req))
Jun 10 20:15:02 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:15:32 tljh python3[3362]: value = future.result()
Jun 10 20:15:58 tljh python3[3362]: tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20513 milliseconds with 0 bytes received
Jun 10 20:16:41 tljh bash[3571]: [W 2020-06-10 20:11:43.759 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 119641 ms.
Jun 10 20:16:56 tljh bash[3571]: [W 2020-06-10 20:11:46.435 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 122156 ms.
Jun 10 20:17:18 tljh bash[3571]: [W 2020-06-10 20:11:47.380 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 122894 ms.
Jun 10 20:17:39 tljh bash[3571]: [W 2020-06-10 20:11:51.105 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 120310 ms.
Jun 10 20:18:04 tljh bash[3571]: [I 2020-06-10 20:11:51.989 SingleUserNotebookApp kernelmanager:217] Starting buffering for b8a25846-a66d-41d0-ac3a-f1d684c09f40:30e1ba482e6d49a38e3315afa80f1e9b
Jun 10 20:18:22 tljh bash[3571]: [I 2020-06-10 20:11:52.990 SingleUserNotebookApp kernelmanager:217] Starting buffering for 21f10d30-98f2-4110-a0db-f057eee0238b:9e2a24c60ed94ecc846a410e99a68a87
Jun 10 20:18:55 tljh bash[3571]: [I 2020-06-10 20:11:54.969 SingleUserNotebookApp kernelmanager:217] Starting buffering for 414e273e-294c-4634-a6a0-3480d1b8cf99:93cee857f4f3410298c53c3a0b184849
Jun 10 20:19:09 tljh bash[3571]: [I 2020-06-10 20:11:57.572 SingleUserNotebookApp kernelmanager:217] Starting buffering for f2099393-a2a8-42af-ba63-29c48968e16c:5a81491ad10246adb230430d17eaa58a
Jun 10 20:19:19 tljh bash[3571]: [W 2020-06-10 20:14:32.157 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 99478 ms.
Jun 10 20:19:29 tljh bash[3571]: [I 2020-06-10 20:14:49.431 SingleUserNotebookApp kernelmanager:217] Starting buffering for ba9af047-0215-4b09-abcf-9973785f08c8:2c794706b11b4c66890ca55685b644ef
Jun 10 20:19:42 tljh bash[3571]: [E 2020-06-10 20:15:19.875 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
Jun 10 20:19:49 tljh bash[3571]: Traceback (most recent call last):
Jun 10 20:19:55 tljh bash[3571]: File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
Jun 10 20:20:01 tljh bash[3571]: await client.fetch(req)
Jun 10 20:20:06 tljh bash[3571]: tornado.simple_httpclient.HTTPTimeoutError: Timeout while connecting
Jun 10 20:20:13 tljh bash[3571]: [E 2020-06-10 20:15:49.006 SingleUserNotebookApp singleuser:548] Error notifying Hub of activity
Jun 10 20:20:19 tljh bash[3571]: Traceback (most recent call last):
Jun 10 20:20:25 tljh bash[3571]: File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 546, in keep_activity_updated
Jun 10 20:20:30 tljh bash[3571]: await self.notify_activity()
Jun 10 20:20:35 tljh python3[3362]: [I 2020-06-10 20:11:23.281 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 23660.82ms
Jun 10 20:20:49 tljh python3[3362]: [I 2020-06-10 20:12:02.737 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 10592.85ms
Ubuntu 18.04.4 LTS tljh ttyS0
tljh login: ls Jun 10 20:21:06 tljh python3[3362]: [I 2020-06-10 20:13:00.692 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 9227.73ms
ls
iris
irs
Jun 10 20:21:29 tljh bash[3571]: File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 533, in notify_activity
Jun 10 20:21:36 tljh bash[3571]: timeout=60,
Jun 10 20:21:42 tljh bash[3571]: File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
Jun 10 20:21:49 tljh bash[3571]: raise TimeoutError(fail_message)
Jun 10 20:21:58 tljh bash[3571]: TimeoutError: Failed to notify Hub of activity
Jun 10 20:22:08 tljh python3[3362]: [E 200610 20:14:31 ioloop:763] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f6300ef2828>>, <Future finished exception=HTTP 599: Operation timed out after 25060 milliseconds with 0 bytes received>)
Login timed out Jun 10 20:22:39 tljh python3[3362]: Traceback (most recent call last):
Jun 10 20:23:09 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
ls
Ubuntu 18.04.4 LTS tljh ttyS0
tljh login: Jun 10 20:23:41 tljh python3[3362]: ret = callback()
f Jun 10 20:24:13 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
sdfkajsd;ofjasdoifj
Jun 10 20:24:47 tljh python3[3362]: future.result()
Jun 10 20:25:20 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
Jun 10 20:25:48 tljh python3[3362]: yielded = self.gen.throw(*exc_info) # type: ignore
Login timed out Jun 10 20:26:19 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 121, in cull_idle
Jun 10 20:26:54 tljh python3[3362]: resp = yield fetch(req)
Ubuntu 18.04.4 LTS tljh ttyS0
tljh login: Jun 10 20:27:29 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:28:03 tljh python3[3362]: value = future.result()
Jun 10 20:28:33 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
Jun 10 20:29:03 tljh python3[3362]: yielded = self.gen.send(value)
Jun 10 20:29:34 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 114, in fetch
Jun 10 20:30:07 tljh python3[3362]: return (yield client.fetch(req))
Jun 10 20:30:40 tljh python3[3362]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:31:12 tljh python3[3362]: value = future.result()
Jun 10 20:31:44 tljh python3[3362]: tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 25060 milliseconds with 0 bytes received