Sharing accounts on TLJH

I created a TLJH server on a Google Cloud VM using the standard tutorial, and made an account that is shared between 2-3 people. This is so I can update the code in just 1 directory and have it accessible to all the users. They then use the notebooks as web app-ish using the AppMode plugin.

Lately the server has been crashing, and I notice that it happens when more than 1 person uses the account at the same time, or even make changes to the base notebook used by AppMode.

Question: is an account on JupyterHub not supposed to be used by multiple people at the same time? From my understanding, AppMode would create a temporary copy of a notebook for each instance run, so I thought having multiple people using the same notebook would be fine in that case.

This is the error log leading up to the crash (I don’t know of a better formatting option so apologies for the wall of text).

Jun 10 20:09:36 tljh systemd[1]: Reloading.
Jun 10 20:09:40 tljh snapd[1312]: storehelpers.go:438: cannot refresh snap "google-cloud-sdk": snap has no updates available
Jun 10 20:09:43 tljh bash[3571]: [I 2020-06-10 20:09:43.894 SingleUserNotebookApp log:174] 200 GET /user/product/notebooks/energy-minimization/app_for_product_team.ipynb (product@24.4.36.157) 5561.98ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:44.100 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.5.86.165) 65.22ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:44.101 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 45.76ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:44.474 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.5.86.165) 9.32ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:44.502 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 9.13ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:45.076 SingleUserNotebookApp log:174] 200 GET /user/product/api/config/notebook?_=1591819784695 (product@24.4.36.157) 174.29ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:45.080 SingleUserNotebookApp log:174] 200 GET /user/product/api/config/common?_=1591819784696 (product@24.4.36.157) 2.60ms
Jun 10 20:09:45 tljh bash[3571]: [I 2020-06-10 20:09:45.475 SingleUserNotebookApp log:174] 200 GET /user/product/api/contents/energy-minimization/app_for_product_team.ipynb?type=notebook&_=1591819784697 (product@24.4.36.157) 274.34ms
Jun 10 20:09:46 tljh bash[3571]: [I 2020-06-10 20:09:46.462 SingleUserNotebookApp log:174] 200 GET /user/product/api/contents/energy-minimization/app_for_product_team.ipynb/checkpoints?_=1591819784698 (product@24.4.36.157) 12.30ms
Jun 10 20:09:46 tljh bash[3571]: [I 2020-06-10 20:09:46.866 SingleUserNotebookApp kernelmanager:172] Kernel started: 8b8178b9-ad66-4a7a-9180-4b211f7dca3c
Jun 10 20:09:47 tljh bash[3571]: [I 2020-06-10 20:09:46.868 SingleUserNotebookApp log:174] 201 POST /user/product/api/sessions (product@24.4.36.157) 453.22ms
Jun 10 20:09:48 tljh bash[3571]: [I 2020-06-10 20:09:48.875 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 479.24ms
Jun 10 20:09:50 tljh python3[3362]: [I 2020-06-10 20:09:50.277 JupyterHub log:174] 200 GET /hub/api/authorizations/token/[secret] (product@127.0.0.1) 752.71ms
Jun 10 20:09:53 tljh python3[3362]: [I 2020-06-10 20:09:50.383 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 38.48ms
Jun 10 20:09:55 tljh bash[3571]: [I 2020-06-10 20:09:50.335 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@207.5.45.241) 1417.68ms
Jun 10 20:09:56 tljh bash[3571]: [I 2020-06-10 20:09:50.372 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.5.86.165) 18.71ms
Jun 10 20:09:57 tljh bash[3571]: [I 2020-06-10 20:09:50.373 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 18.37ms
Jun 10 20:09:59 tljh bash[3571]: [I 2020-06-10 20:09:52.067 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@207.5.45.241) 73.82ms
Jun 10 20:10:02 tljh bash[3571]: [I 2020-06-10 20:09:52.111 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 81.83ms
Jun 10 20:10:07 tljh bash[3571]: [I 2020-06-10 20:09:53.494 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 89.26ms
Jun 10 20:10:33 tljh bash[3571]: [I 2020-06-10 20:09:53.653 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.4.36.157) 7.91ms
Jun 10 20:10:58 tljh bash[3571]: [I 2020-06-10 20:09:54.439 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@24.5.86.165) 38.17ms
Jun 10 20:11:04 tljh bash[3571]: [I 2020-06-10 20:09:55.106 SingleUserNotebookApp log:174] 200 GET /user/product/metrics (product@207.5.45.241) 10.36ms
Jun 10 20:11:09 tljh bash[3571]: [W 2020-06-10 20:10:50.572 SingleUserNotebookApp handlers:229] Timeout waiting for kernel_info reply from 8b8178b9-ad66-4a7a-9180-4b211f7dca3c
Jun 10 20:11:14 tljh bash[3571]: [I 2020-06-10 20:10:55.788 SingleUserNotebookApp log:174] 101 GET /user/product/api/kernels/8b8178b9-ad66-4a7a-9180-4b211f7dca3c/channels?session_id=4b576c19d95d4d6c8caa2044b9d7bf85 (product@24.4.36.157) 65422.68ms
Jun 10 20:11:20 tljh bash[3571]: [I 2020-06-10 20:11:04.496 SingleUserNotebookApp kernelmanager:217] Starting buffering for 8b8178b9-ad66-4a7a-9180-4b211f7dca3c:4b576c19d95d4d6c8caa2044b9d7bf85
Jun 10 20:11:24 tljh python3[3362]: [I 2020-06-10 20:10:58.718 JupyterHub proxy:320] Checking routes
Jun 10 20:11:35 tljh bash[3571]: [W 2020-06-10 20:11:16.635 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 90578 ms.
Jun 10 20:11:42 tljh bash[3571]: [W 2020-06-10 20:11:23.457 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 90177 ms.
Jun 10 20:11:46 tljh bash[3571]: [I 2020-06-10 20:11:23.582 SingleUserNotebookApp kernelmanager:217] Starting buffering for 7fd789ac-a766-4761-9732-01a1f5456ed4:c2ee08980b394c1f80e2c67a23ec4476
Jun 10 20:11:49 tljh bash[3571]: [I 2020-06-10 20:11:28.582 SingleUserNotebookApp kernelmanager:217] Starting buffering for 657196c0-af73-4f86-926d-07686360ea09:568d4f39288a44bf9d5b7edd11e29ec7
Jun 10 20:11:52 tljh python3[3362]: [E 200610 20:11:14 ioloop:763] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f6300ef2828>>, <Future finished exception=HTTP 599: Operation timed out after 20513 milliseconds with 0 bytes received>)
Jun 10 20:12:01 tljh python3[3362]:     Traceback (most recent call last):
Jun 10 20:12:08 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
Jun 10 20:12:17 tljh python3[3362]:         ret = callback()
Jun 10 20:12:27 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
Jun 10 20:12:39 tljh python3[3362]:         future.result()
Jun 10 20:12:48 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
Jun 10 20:12:54 tljh python3[3362]:         yielded = self.gen.throw(*exc_info)  # type: ignore
Jun 10 20:12:59 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 121, in cull_idle
Jun 10 20:13:05 tljh python3[3362]:         resp = yield fetch(req)
Jun 10 20:13:14 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:13:30 tljh python3[3362]:         value = future.result()
Jun 10 20:13:47 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
Jun 10 20:14:01 tljh python3[3362]:         yielded = self.gen.send(value)
Jun 10 20:14:24 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 114, in fetch
Jun 10 20:14:43 tljh python3[3362]:         return (yield client.fetch(req))
Jun 10 20:15:02 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:15:32 tljh python3[3362]:         value = future.result()
Jun 10 20:15:58 tljh python3[3362]:     tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20513 milliseconds with 0 bytes received
Jun 10 20:16:41 tljh bash[3571]: [W 2020-06-10 20:11:43.759 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 119641 ms.
Jun 10 20:16:56 tljh bash[3571]: [W 2020-06-10 20:11:46.435 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 122156 ms.
Jun 10 20:17:18 tljh bash[3571]: [W 2020-06-10 20:11:47.380 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 122894 ms.
Jun 10 20:17:39 tljh bash[3571]: [W 2020-06-10 20:11:51.105 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 120310 ms.
Jun 10 20:18:04 tljh bash[3571]: [I 2020-06-10 20:11:51.989 SingleUserNotebookApp kernelmanager:217] Starting buffering for b8a25846-a66d-41d0-ac3a-f1d684c09f40:30e1ba482e6d49a38e3315afa80f1e9b
Jun 10 20:18:22 tljh bash[3571]: [I 2020-06-10 20:11:52.990 SingleUserNotebookApp kernelmanager:217] Starting buffering for 21f10d30-98f2-4110-a0db-f057eee0238b:9e2a24c60ed94ecc846a410e99a68a87
Jun 10 20:18:55 tljh bash[3571]: [I 2020-06-10 20:11:54.969 SingleUserNotebookApp kernelmanager:217] Starting buffering for 414e273e-294c-4634-a6a0-3480d1b8cf99:93cee857f4f3410298c53c3a0b184849
Jun 10 20:19:09 tljh bash[3571]: [I 2020-06-10 20:11:57.572 SingleUserNotebookApp kernelmanager:217] Starting buffering for f2099393-a2a8-42af-ba63-29c48968e16c:5a81491ad10246adb230430d17eaa58a
Jun 10 20:19:19 tljh bash[3571]: [W 2020-06-10 20:14:32.157 SingleUserNotebookApp zmqhandlers:182] WebSocket ping timeout after 99478 ms.
Jun 10 20:19:29 tljh bash[3571]: [I 2020-06-10 20:14:49.431 SingleUserNotebookApp kernelmanager:217] Starting buffering for ba9af047-0215-4b09-abcf-9973785f08c8:2c794706b11b4c66890ca55685b644ef
Jun 10 20:19:42 tljh bash[3571]: [E 2020-06-10 20:15:19.875 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
Jun 10 20:19:49 tljh bash[3571]:     Traceback (most recent call last):
Jun 10 20:19:55 tljh bash[3571]:       File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
Jun 10 20:20:01 tljh bash[3571]:         await client.fetch(req)
Jun 10 20:20:06 tljh bash[3571]:     tornado.simple_httpclient.HTTPTimeoutError: Timeout while connecting
Jun 10 20:20:13 tljh bash[3571]: [E 2020-06-10 20:15:49.006 SingleUserNotebookApp singleuser:548] Error notifying Hub of activity
Jun 10 20:20:19 tljh bash[3571]:     Traceback (most recent call last):
Jun 10 20:20:25 tljh bash[3571]:       File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 546, in keep_activity_updated
Jun 10 20:20:30 tljh bash[3571]:         await self.notify_activity()
Jun 10 20:20:35 tljh python3[3362]: [I 2020-06-10 20:11:23.281 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 23660.82ms
Jun 10 20:20:49 tljh python3[3362]: [I 2020-06-10 20:12:02.737 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 10592.85ms


Ubuntu 18.04.4 LTS tljh ttyS0

tljh login: ls  Jun 10 20:21:06 tljh python3[3362]: [I 2020-06-10 20:13:00.692 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 9227.73ms
ls

iris
irs
Jun 10 20:21:29 tljh bash[3571]:       File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 533, in notify_activity
Jun 10 20:21:36 tljh bash[3571]:         timeout=60,
Jun 10 20:21:42 tljh bash[3571]:       File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
Jun 10 20:21:49 tljh bash[3571]:         raise TimeoutError(fail_message)
Jun 10 20:21:58 tljh bash[3571]:     TimeoutError: Failed to notify Hub of activity
Jun 10 20:22:08 tljh python3[3362]: [E 200610 20:14:31 ioloop:763] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f6300ef2828>>, <Future finished exception=HTTP 599: Operation timed out after 25060 milliseconds with 0 bytes received>)

Login timed out Jun 10 20:22:39 tljh python3[3362]:     Traceback (most recent call last):

Jun 10 20:23:09 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
ls


Ubuntu 18.04.4 LTS tljh ttyS0

tljh login: Jun 10 20:23:41 tljh python3[3362]:         ret = callback()
f Jun 10 20:24:13 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
sdfkajsd;ofjasdoifj

Jun 10 20:24:47 tljh python3[3362]:         future.result()
Jun 10 20:25:20 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
Jun 10 20:25:48 tljh python3[3362]:         yielded = self.gen.throw(*exc_info)  # type: ignore

Login timed out Jun 10 20:26:19 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 121, in cull_idle
Jun 10 20:26:54 tljh python3[3362]:         resp = yield fetch(req)


Ubuntu 18.04.4 LTS tljh ttyS0

tljh login: Jun 10 20:27:29 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:28:03 tljh python3[3362]:         value = future.result()
Jun 10 20:28:33 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
Jun 10 20:29:03 tljh python3[3362]:         yielded = self.gen.send(value)
Jun 10 20:29:34 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 114, in fetch
Jun 10 20:30:07 tljh python3[3362]:         return (yield client.fetch(req))
Jun 10 20:30:40 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:31:12 tljh python3[3362]:         value = future.result()
Jun 10 20:31:44 tljh python3[3362]:     tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 25060 milliseconds with 0 bytes received

Also Jupyterhub log

Jun 10 20:05:52 tljh python3[3362]: [I 2020-06-10 20:05:52.354 JupyterHub proxy:320] Checking routes
Jun 10 20:06:50 tljh python3[3362]: [I 2020-06-10 20:06:50.353 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 20.25ms
Jun 10 20:07:50 tljh python3[3362]: [I 2020-06-10 20:07:50.349 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 15.93ms
Jun 10 20:08:08 tljh python3[3362]: [I 2020-06-10 20:08:08.894 JupyterHub log:174] 200 GET /hub/api/authorizations/token/[secret] (product@127.0.0.1) 38.95ms
Jun 10 20:08:31 tljh python3[3362]: [I 2020-06-10 20:08:31.614 JupyterHub log:174] 200 GET /hub/api/authorizations/token/[secret] (product@127.0.0.1) 18.93ms
Jun 10 20:08:50 tljh python3[3362]: [I 2020-06-10 20:08:50.350 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 16.85ms
Jun 10 20:09:00 tljh python3[3362]: [I 2020-06-10 20:09:00.838 JupyterHub log:174] 200 POST /hub/api/users/product/activity (product@127.0.0.1) 74.03ms
Jun 10 20:09:27 tljh python3[3362]: [I 2020-06-10 20:09:27.089 JupyterHub log:174] 302 GET / -> /hub/ (@24.4.36.157) 1.52ms
Jun 10 20:09:30 tljh python3[3362]: [I 2020-06-10 20:09:27.867 JupyterHub log:174] 302 GET /hub/ -> /user/product/ (product@24.4.36.157) 550.31ms
Jun 10 20:09:50 tljh python3[3362]: [I 2020-06-10 20:09:50.277 JupyterHub log:174] 200 GET /hub/api/authorizations/token/[secret] (product@127.0.0.1) 752.71ms
Jun 10 20:09:53 tljh python3[3362]: [I 2020-06-10 20:09:50.383 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 38.48ms
Jun 10 20:11:23 tljh python3[3362]: [I 2020-06-10 20:10:58.718 JupyterHub proxy:320] Checking routes
Jun 10 20:11:51 tljh python3[3362]: [E 200610 20:11:14 ioloop:763] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f6300ef2828>>, <Future finished exception=HTTP 599: Operation timed out after 20513 milliseconds with 0 bytes received>)
Jun 10 20:11:51 tljh python3[3362]:     Traceback (most recent call last):
Jun 10 20:11:51 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
Jun 10 20:11:51 tljh python3[3362]:         ret = callback()
Jun 10 20:11:51 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
Jun 10 20:11:51 tljh python3[3362]:         future.result()
Jun 10 20:11:51 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
Jun 10 20:11:51 tljh python3[3362]:         yielded = self.gen.throw(*exc_info)  # type: ignore
Jun 10 20:11:51 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 121, in cull_idle
Jun 10 20:11:51 tljh python3[3362]:         resp = yield fetch(req)
Jun 10 20:11:51 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:11:51 tljh python3[3362]:         value = future.result()
Jun 10 20:11:51 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
Jun 10 20:11:51 tljh python3[3362]:         yielded = self.gen.send(value)
Jun 10 20:11:51 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 114, in fetch
Jun 10 20:11:51 tljh python3[3362]:         return (yield client.fetch(req))
Jun 10 20:11:51 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:11:51 tljh python3[3362]:         value = future.result()
Jun 10 20:11:51 tljh python3[3362]:     tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20513 milliseconds with 0 bytes received
Jun 10 20:20:34 tljh python3[3362]: [I 2020-06-10 20:11:23.281 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 23660.82ms
Jun 10 20:20:34 tljh python3[3362]: [I 2020-06-10 20:12:02.737 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 10592.85ms
Jun 10 20:20:34 tljh python3[3362]: [I 2020-06-10 20:13:00.692 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 9227.73ms
Jun 10 20:22:05 tljh python3[3362]: [E 200610 20:14:31 ioloop:763] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f6300ef2828>>, <Future finished exception=HTTP 599: Operation timed out after 25060 milliseconds with 0 bytes received>)
Jun 10 20:22:05 tljh python3[3362]:     Traceback (most recent call last):
Jun 10 20:22:05 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
Jun 10 20:22:05 tljh python3[3362]:         ret = callback()
Jun 10 20:22:05 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
Jun 10 20:22:05 tljh python3[3362]:         future.result()
Jun 10 20:22:05 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
Jun 10 20:22:05 tljh python3[3362]:         yielded = self.gen.throw(*exc_info)  # type: ignore
Jun 10 20:22:05 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 121, in cull_idle
Jun 10 20:22:05 tljh python3[3362]:         resp = yield fetch(req)
Jun 10 20:22:05 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:22:05 tljh python3[3362]:         value = future.result()
Jun 10 20:22:05 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
Jun 10 20:22:05 tljh python3[3362]:         yielded = self.gen.send(value)
Jun 10 20:22:05 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_idle_culler/__init__.py", line 114, in fetch
Jun 10 20:22:05 tljh python3[3362]:         return (yield client.fetch(req))
Jun 10 20:22:05 tljh python3[3362]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Jun 10 20:22:05 tljh python3[3362]:         value = future.result()
Jun 10 20:22:05 tljh python3[3362]:     tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 25060 milliseconds with 0 bytes received
-- Reboot --
Jun 10 20:35:41 tljh systemd[1]: Started jupyterhub.service.
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.076 JupyterHub app:2240] Running JupyterHub version 1.1.0
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.085 JupyterHub app:2271] Using Authenticator: firstuseauthenticator.firstuseauthenticator.FirstUseAuthenticator
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.085 JupyterHub app:2271] Using Spawner: builtins.UserCreatingSpawner
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.085 JupyterHub app:2271] Using Proxy: jupyterhub_traefik_proxy.toml.TraefikTomlProxy-0+unknown
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.214 JupyterHub app:1349] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.660 JupyterHub app:1655] Not using whitelist. Any authenticated user will be allowed.
Jun 10 20:35:55 tljh python3[1104]: [W 2020-06-10 20:35:55.959 JupyterHub app:1997] product appears to have stopped while the Hub was down
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.984 JupyterHub app:2311] Initialized 1 spawners in 0.111 seconds
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.990 JupyterHub app:2520] Not starting proxy
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.994 JupyterHub app:2556] Hub API listening on http://127.0.0.1:15001/hub/
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.994 JupyterHub app:2571] Starting managed service cull-idle
Jun 10 20:35:55 tljh python3[1104]: [I 2020-06-10 20:35:55.995 JupyterHub service:335] Starting service 'cull-idle': ['/opt/tljh/hub/bin/python3', '-m', 'jupyterhub_idle_culler', '--timeout=43200', '--cull-every=60', '--concurrency=5', '--max-age=0']
Jun 10 20:35:56 tljh python3[1104]: [I 2020-06-10 20:35:56.002 JupyterHub service:121] Spawning /opt/tljh/hub/bin/python3 -m jupyterhub_idle_culler --timeout=43200 --cull-every=60 --concurrency=5 --max-age=0
Jun 10 20:35:56 tljh python3[1104]: [I 2020-06-10 20:35:56.008 JupyterHub proxy:320] Checking routes
Jun 10 20:35:56 tljh python3[1104]: [W 2020-06-10 20:35:56.008 JupyterHub proxy:391] Deleting stale route /user/product/
Jun 10 20:35:56 tljh python3[1104]: [I 2020-06-10 20:35:56.011 JupyterHub app:2631] JupyterHub is now running at http://:8000
Jun 10 20:35:56 tljh python3[1104]: [I 2020-06-10 20:35:56.464 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 32.47ms
Jun 10 20:36:15 tljh python3[1104]: [I 2020-06-10 20:36:15.249 JupyterHub log:174] 302 GET /user/product/api/terminals?_=1591819768379 -> /hub/user/product/api/terminals?_=1591819768379 (@24.4.36.157) 1.51ms
Jun 10 20:36:15 tljh python3[1104]: [I 2020-06-10 20:36:15.252 JupyterHub log:174] 302 GET /user/product/api/sessions?_=1591819768378 -> /hub/user/product/api/sessions?_=1591819768378 (@24.4.36.157) 1.19ms
Jun 10 20:36:15 tljh python3[1104]: [W 2020-06-10 20:36:15.362 JupyterHub base:1262] Failing suspected API request to not-running server: /hub/user/product/api/sessions
Jun 10 20:36:15 tljh python3[1104]: [E 2020-06-10 20:36:15.364 JupyterHub log:174] 503 GET /hub/user/product/api/sessions?_=1591819768378 (product@24.4.36.157) 55.29ms
Jun 10 20:36:15 tljh python3[1104]: [W 2020-06-10 20:36:15.370 JupyterHub base:1262] Failing suspected API request to not-running server: /hub/user/product/api/terminals
Jun 10 20:36:15 tljh python3[1104]: [E 2020-06-10 20:36:15.373 JupyterHub log:174] 503 GET /hub/user/product/api/terminals?_=1591819768379 (product@24.4.36.157) 63.33ms
Jun 10 20:36:16 tljh python3[1104]: [I 2020-06-10 20:36:16.027 JupyterHub log:174] 302 GET /user/product/metrics -> /hub/user/product/metrics (@24.4.36.157) 1.12ms
Jun 10 20:36:16 tljh python3[1104]: [W 2020-06-10 20:36:16.074 JupyterHub base:1262] Failing suspected API request to not-running server: /hub/user/product/metrics
Jun 10 20:36:16 tljh python3[1104]: [E 2020-06-10 20:36:16.076 JupyterHub log:174] 503 GET /hub/user/product/metrics (product@24.4.36.157) 4.53ms
Jun 10 20:36:18 tljh python3[1104]: [I 2020-06-10 20:36:18.878 JupyterHub log:174] 302 GET /user/product/metrics -> /hub/user/product/metrics (@24.4.36.157) 1.20ms
Jun 10 20:36:18 tljh python3[1104]: [W 2020-06-10 20:36:18.928 JupyterHub base:1262] Failing suspected API request to not-running server: /hub/user/product/metrics
Jun 10 20:36:18 tljh python3[1104]: [E 2020-06-10 20:36:18.930 JupyterHub log:174] 503 GET /hub/user/product/metrics (product@24.4.36.157) 4.44ms
Jun 10 20:36:23 tljh python3[1104]: [I 2020-06-10 20:36:23.869 JupyterHub log:174] 302 GET /user/product/metrics -> /hub/user/product/metrics (@24.4.36.157) 1.50ms
Jun 10 20:36:23 tljh python3[1104]: [W 2020-06-10 20:36:23.921 JupyterHub base:1262] Failing suspected API request to not-running server: /hub/user/product/metrics
Jun 10 20:36:23 tljh python3[1104]: [E 2020-06-10 20:36:23.922 JupyterHub log:174] 503 GET /hub/user/product/metrics (product@24.4.36.157) 4.76ms
Jun 10 20:36:46 tljh python3[1104]: [I 2020-06-10 20:36:46.837 JupyterHub log:174] 302 GET /user/product/apps/energy-minimization/app_for_product_team.ipynb -> /hub/user/product/apps/energy-minimization/app_for_product_team.ipynb (@24.5.86.165) 1.27ms
Jun 10 20:36:47 tljh python3[1104]: [E 2020-06-10 20:36:47.141 JupyterHub log:174] 503 GET /hub/user/product/apps/energy-minimization/app_for_product_team.ipynb (product@24.5.86.165) 247.82ms
Jun 10 20:36:47 tljh python3[1104]: [I 2020-06-10 20:36:47.973 JupyterHub log:174] 302 GET /user/product/api/kernels/ba9af047-0215-4b09-abcf-9973785f08c8/channels?session_id=2c794706b11b4c66890ca55685b644ef -> /hub/user/product/api/kernels/ba9af047-0215-4b09-abcf-9973785f08c8/channels?session_id=2c794706b11b4c66890ca55685b644ef (@24.5.86.165) 1.09ms
Jun 10 20:36:50 tljh python3[1104]: [I 2020-06-10 20:36:50.218 JupyterHub log:174] 302 GET /user/product/api/kernels/ba9af047-0215-4b09-abcf-9973785f08c8/channels?session_id=2c794706b11b4c66890ca55685b644ef -> /hub/user/product/api/kernels/ba9af047-0215-4b09-abcf-9973785f08c8/channels?session_id=2c794706b11b4c66890ca55685b644ef (@24.5.86.165) 1.14ms
Jun 10 20:36:54 tljh python3[1104]: [I 2020-06-10 20:36:54.044 JupyterHub log:174] 302 GET /user/product/api/kernels/ba9af047-0215-4b09-abcf-9973785f08c8/channels?session_id=2c794706b11b4c66890ca55685b644ef -> /hub/user/product/api/kernels/ba9af047-0215-4b09-abcf-9973785f08c8/channels?session_id=2c794706b11b4c66890ca55685b644ef (@24.5.86.165) 1.10ms
Jun 10 20:36:55 tljh python3[1104]: [I 2020-06-10 20:36:55.395 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 16.19ms
Jun 10 20:37:02 tljh python3[1104]: [I 2020-06-10 20:37:02.929 JupyterHub log:174] 302 GET /user/product/api/kernels/ba9af047-0215-4b09-abcf-9973785f08c8/channels?session_id=2c794706b11b4c66890ca55685b644ef -> /hub/user/product/api/kernels/ba9af047-0215-4b09-abcf-9973785f08c8/channels?session_id=2c794706b11b4c66890ca55685b644ef (@24.5.86.165) 1.08ms
Jun 10 20:37:05 tljh python3[1104]: [I 2020-06-10 20:37:05.575 JupyterHub log:174] 302 GET / -> /hub/ (@74.205.153.243) 1.45ms
Jun 10 20:37:05 tljh python3[1104]: [I 2020-06-10 20:37:05.657 JupyterHub log:174] 302 GET /hub/ -> /hub/spawn (product@74.205.153.243) 4.25ms
Jun 10 20:37:05 tljh python3[1104]: Running as unit: jupyter-product.service
Jun 10 20:37:06 tljh python3[1104]: [I 2020-06-10 20:37:06.753 JupyterHub log:174] 302 GET /hub/spawn -> /hub/spawn-pending/product (product@74.205.153.243) 1009.86ms
Jun 10 20:37:06 tljh python3[1104]: [I 2020-06-10 20:37:06.897 JupyterHub pages:347] product is pending spawn
Jun 10 20:37:06 tljh python3[1104]: [I 2020-06-10 20:37:06.926 JupyterHub log:174] 200 GET /hub/spawn-pending/product (product@74.205.153.243) 34.99ms
Jun 10 20:37:10 tljh python3[1104]: [W 2020-06-10 20:37:10.375 JupyterHub _version:56] jupyterhub version 1.1.0 != jupyterhub-singleuser version 1.0.0. This could cause failure to authenticate and result in redirect loops!
Jun 10 20:37:10 tljh python3[1104]: [I 2020-06-10 20:37:10.375 JupyterHub base:855] User product took 4.629 seconds to start
Jun 10 20:37:10 tljh python3[1104]: [I 2020-06-10 20:37:10.376 JupyterHub proxy:262] Adding user product to proxy /user/product/ => http://127.0.0.1:45031
Jun 10 20:37:10 tljh python3[1104]: [I 2020-06-10 20:37:10.407 JupyterHub proxy:129] Waiting for /user/product to register with traefik
Jun 10 20:37:10 tljh python3[1104]: [I 2020-06-10 20:37:10.433 JupyterHub log:174] 200 GET /hub/api (@127.0.0.1) 3.90ms
Jun 10 20:37:10 tljh python3[1104]: [I 2020-06-10 20:37:10.507 JupyterHub log:174] 200 POST /hub/api/users/product/activity (product@127.0.0.1) 30.91ms
Jun 10 20:37:11 tljh python3[1104]: [I 2020-06-10 20:37:11.031 JupyterHub users:609] Server product is ready
Jun 10 20:37:11 tljh python3[1104]: [I 2020-06-10 20:37:11.033 JupyterHub log:174] 200 GET /hub/api/users/product/server/progress (product@74.205.153.243) 3087.94ms
Jun 10 20:37:11 tljh python3[1104]: [I 2020-06-10 20:37:11.142 JupyterHub log:174] 302 GET /hub/spawn-pending/product -> /user/product/ (product@74.205.153.243) 5.83ms
Jun 10 20:37:11 tljh python3[1104]: [I 2020-06-10 20:37:11.440 JupyterHub log:174] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-product&redirect_uri=%2Fuser%2Fproduct%2Foauth_callback&response_type=code&state=[secret] -> /user/product/oauth_callback?code=[secret]&state=[secret] (product@74.205.153.243) 34.55ms
Jun 10 20:37:11 tljh python3[1104]: [I 2020-06-10 20:37:11.593 JupyterHub log:174] 200 POST /hub/api/oauth2/token (product@127.0.0.1) 56.72ms
Jun 10 20:37:11 tljh python3[1104]: [I 2020-06-10 20:37:11.633 JupyterHub log:174] 200 GET /hub/api/authorizations/token/[secret] (product@127.0.0.1) 36.62ms
Jun 10 20:37:29 tljh systemd[1]: Stopping jupyterhub.service...
Jun 10 20:37:29 tljh python3[1104]: [C 2020-06-10 20:37:29.083 JupyterHub app:2684] Received signal SIGTERM, initiating shutdown...
Jun 10 20:37:29 tljh python3[1104]: [I 2020-06-10 20:37:29.084 JupyterHub app:2355] Cleaning up 1 services...
Jun 10 20:37:29 tljh python3[1104]: [I 2020-06-10 20:37:29.088 JupyterHub app:2367] Leaving single-user servers running
Jun 10 20:37:29 tljh python3[1104]: [I 2020-06-10 20:37:29.088 JupyterHub app:2375] I didn't start the proxy, I can't clean it up
Jun 10 20:37:29 tljh python3[1104]: [I 2020-06-10 20:37:29.088 JupyterHub app:2392] ...done
Jun 10 20:37:29 tljh python3[1104]: 10
Jun 10 20:37:29 tljh systemd[1]: Stopped jupyterhub.service.
Jun 10 20:37:29 tljh systemd[1]: Started jupyterhub.service.
Jun 10 20:37:29 tljh systemd[1]: Stopping jupyterhub.service...
Jun 10 20:37:29 tljh systemd[1]: Stopped jupyterhub.service.
Jun 10 20:37:29 tljh systemd[1]: Started jupyterhub.service.
Jun 10 20:37:31 tljh python3[3389]: [I 2020-06-10 20:37:31.512 JupyterHub app:2240] Running JupyterHub version 1.1.0