Dear group,
I am writing to seek help on the behaviour a recent installation of Jupiterhub is showing.
I hope someone can help me to single out where the problem is.
I have followed the how-to (The Littlest JupyterHub — The Littlest JupyterHub v0.1 documentation ) for the server installation which is hosted on a virtual server with a public ip.
The installation worked fine and I have done just minor modifications as suggested in the document.
The hub seems to work fine, I’ve enabled ssl and open the firewall ports.
I can connect and start the lab (the app I’ve set as default)
However from time to time when I try to run any notebook/console/terminal everything freeze and the kernel keeps on trying to connect for a while, switching to disconnected after some time.
I have tried to dig into the log (journalctl -u jupyterhub, journalctl -u traefik, journalctl -u jupyter-rago) but none of the above showed any clear error.
My tljh config is as follow:
root@AItesting:/home# tljh-config show
users:
admin:
- rago
https:
enabled: true
letsencrypt:
email: antoni…
domains: - xxxxxxx(fully resolved website)
user_environment:
default_app: jupyterlab
Can anyone suggest a way to diagnose the problem?
Thank you in advance.
Antonio Rago
Below are the final part of my logs :
root@AItesting:/home# journalctl -u jupyterhub
Feb 15 22:28:23 AItesting python3[2015]: [I 2021-02-15 22:28:23.251 JupyterHub log:181] 200 GET /hub/api/users (cull…127.0.0.1) 8.34ms
Feb 15 22:29:23 AItesting python3[2015]: [I 2021-02-15 22:29:23.255 JupyterHub log:181] 200 GET /hub/api/users (cull…127.0.0.1) 11.77ms
Feb 15 22:29:26 AItesting python3[2015]: /opt/tljh/hub/lib/python3.8/site-packages/jupyterhub_traefik_proxy/toml.py:104: RuntimeWarning: Escape character ‘’ cannot be a safe character. Set allow_co>
Feb 15 22:29:26 AItesting python3[2015]: escaped_routespec = escapism.escape(traefik_routespec, safe=safe)
Feb 15 22:29:26 AItesting python3[2015]: [I 2021-02-15 22:29:26.843 JupyterHub proxy:320] Checking routes
Feb 15 22:29:35 AItesting python3[2015]: [I 2021-02-15 22:29:35.046 JupyterHub pages:148] User is running: rago
Feb 15 22:29:35 AItesting python3[2015]: [I 2021-02-15 22:29:35.047 JupyterHub log:181] 302 GET /hub/spawn?next=/hub/home → /hub/home (ra…87.74.180.148) 9.69ms
Feb 15 22:29:35 AItesting python3[2015]: [I 2021-02-15 22:29:35.085 JupyterHub log:181] 200 GET /hub/home (ra…87.74.180.148) 15.55ms
Feb 15 22:29:36 AItesting python3[2015]: [I 2021-02-15 22:29:36.562 JupyterHub log:181] 200 GET /hub/api/authorizations/token/[secret] (ra…127.0.0.1) 15.78ms
Feb 15 22:30:23 AItesting python3[2015]: [I 2021-02-15 22:30:23.255 JupyterHub log:181] 200 GET /hub/api/users (cull…127.0.0.1) 11.19ms
Feb 15 22:31:23 AItesting python3[2015]: [I 2021-02-15 22:31:23.254 JupyterHub log:181] 200 GET /hub/api/users (cull…127.0.0.1) 10.63ms
Feb 15 22:32:23 AItesting python3[2015]: [I 2021-02-15 22:32:23.258 JupyterHub log:181] 200 GET /hub/api/users (cull…127.0.0.1) 14.82ms
Feb 15 22:33:06 AItesting python3[2015]: [I 2021-02-15 22:33:06.488 JupyterHub log:181] 200 POST /hub/api/users/rago/activity (ra…127.0.0.1) 15.77ms
Feb 15 22:33:23 AItesting python3[2015]: [I 2021-02-15 22:33:23.251 JupyterHub log:181] 200 GET /hub/api/users (cull…127.0.0.1) 8.08ms
Feb 15 22:34:23 AItesting python3[2015]: [I 2021-02-15 22:34:23.254 JupyterHub log:181] 200 GET /hub/api/users (cull…127.0.0.1) 10.77ms
Feb 15 22:34:26 AItesting python3[2015]: /opt/tljh/hub/lib/python3.8/site-packages/jupyterhub_traefik_proxy/toml.py:104: RuntimeWarning: Escape character '’ cannot be a safe character. Set allow_co>
Feb 15 22:34:26 AItesting python3[2015]: escaped_routespec = escapism.escape(traefik_routespec, safe=safe)
Feb 15 22:34:26 AItesting python3[2015]: [I 2021-02-15 22:34:26.843 JupyterHub proxy:320] Checking routes
Feb 15 22:35:23 AItesting python3[2015]: [I 2021-02-15 22:35:23.255 JupyterHub log:181] 200 GET /hub/api/users (cull…127.0.0.1) 11.72ms
root@AItesting:/home# journalctl -u traefik
Feb 15 21:25:30 AItesting traefik[2012]: time=“2021-02-15T21:25:30+01:00” level=info msg=“Skipping same configuration for provider file”
Feb 15 21:25:31 AItesting traefik[2012]: time=“2021-02-15T21:25:31+01:00” level=info msg=“Server configuration reloaded on :80”
Feb 15 21:25:31 AItesting traefik[2012]: time=“2021-02-15T21:25:31+01:00” level=info msg=“Server configuration reloaded on :443”
Feb 15 21:25:31 AItesting traefik[2012]: time=“2021-02-15T21:25:31+01:00” level=info msg=“Server configuration reloaded on [127.0.0.1 :8099](http:// 127.0.0.1 :8099/)”
Feb 15 21:25:33 AItesting traefik[2012]: time=“2021-02-15T21:25:33+01:00” level=info msg=“Server configuration reloaded on :80”
Feb 15 21:25:33 AItesting traefik[2012]: time=“2021-02-15T21:25:33+01:00” level=info msg=“Server configuration reloaded on :443”
Feb 15 21:25:33 AItesting traefik[2012]: time=“2021-02-15T21:25:33+01:00” level=info msg=“Server configuration reloaded on [127.0.0.1 :8099](http:// 127.0.0.1 :8099/)”
Feb 15 21:25:40 AItesting traefik[2012]: {“BackendAddr”:"[127.0.0.1 :33115](http:// 127.0.0.1 :33115/)",“BackendName”:“backend__2Fuser_2Fblucini”,“BackendURL”:{“Scheme”:“http”,“Opaque”:"",“User”:null,“Host”:"[127.0.0.1 :33115](http:// 127.0.0.1 :33115/)",“Pa>
Feb 15 21:51:23 AItesting traefik[2012]: time=“2021-02-15T21:51:23+01:00” level=info msg=“Skipping same configuration for provider file”
Feb 15 21:51:23 AItesting traefik[2012]: time=“2021-02-15T21:51:23+01:00” level=info msg=“Server configuration reloaded on :80”
Feb 15 21:51:23 AItesting traefik[2012]: time=“2021-02-15T21:51:23+01:00” level=info msg=“Server configuration reloaded on :443”
Feb 15 21:51:23 AItesting traefik[2012]: time=“2021-02-15T21:51:23+01:00” level=info msg=“Server configuration reloaded on [127.0.0.1 :8099](http:// 127.0.0.1 :8099/)”
Feb 15 21:51:25 AItesting traefik[2012]: time=“2021-02-15T21:51:25+01:00” level=info msg=“Server configuration reloaded on [127.0.0.1 :8099](http:// 127.0.0.1 :8099/)”
Feb 15 21:51:25 AItesting traefik[2012]: time=“2021-02-15T21:51:25+01:00” level=info msg=“Server configuration reloaded on :80”
Feb 15 21:51:25 AItesting traefik[2012]: time=“2021-02-15T21:51:25+01:00” level=info msg=“Server configuration reloaded on :443”
Feb 15 22:23:02 AItesting traefik[2012]: time=“2021-02-15T22:23:02+01:00” level=info msg=“Skipping same configuration for provider file”
Feb 15 22:23:02 AItesting traefik[2012]: time=“2021-02-15T22:23:02+01:00” level=info msg=“Server configuration reloaded on :80”
Feb 15 22:23:02 AItesting traefik[2012]: time=“2021-02-15T22:23:02+01:00” level=info msg=“Server configuration reloaded on :443”
Feb 15 22:23:02 AItesting traefik[2012]: time=“2021-02-15T22:23:02+01:00” level=info msg=“Server configuration reloaded on [127.0.0.1 :8099](http:// 127.0.0.1 :8099/)”
Feb 15 22:23:05 AItesting traefik[2012]: time=“2021-02-15T22:23:05+01:00” level=info msg=“Server configuration reloaded on :80”
Feb 15 22:23:05 AItesting traefik[2012]: time=“2021-02-15T22:23:05+01:00” level=info msg=“Server configuration reloaded on :443”
Feb 15 22:23:05 AItesting traefik[2012]: time=“2021-02-15T22:23:05+01:00” level=info msg=“Server configuration reloaded on [127.0.0.1 :8099](http:// 127.0.0.1 :8099/)”
Feb 15 22:23:09 AItesting traefik[2012]: {“BackendAddr”:”[127.0.0.1 :43581](http:// 127.0.0.1 :43581/)",“BackendName”:“backend__2Fuser_2Frago”,“BackendURL”:{“Scheme”:“http”,“Opaque”:"",“User”:null,“Host”:"[127.0.0.1 :43581](http:// 127.0.0.1 :43581/)",“Path”>
Feb 15 22:29:39 AItesting traefik[2012]: {“BackendAddr”:"[127.0.0.1 :43581](http:// 127.0.0.1 :43581/)",“BackendName”:“backend__2Fuser_2Frago”,“BackendURL”:{“Scheme”:“http”,“Opaque”:"",“User”:null,“Host”:"[127.0.0.1 :43581](http:// 127.0.0.1 :43581/)",“Path”>
root@AItesting:/home# journalctl -u jupyter-rago.service
Feb 15 22:41:06 AItesting bash[23379]: [I 2021-02-15 22:41:06.199 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425266090 (ra…87.74.180.148) 1.86ms
Feb 15 22:41:06 AItesting bash[23379]: [I 2021-02-15 22:41:06.290 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernelspecs?1613425266182 (ra…87.74.180.148) 2.32ms
Feb 15 22:41:07 AItesting bash[23379]: [I 2021-02-15 22:41:07.570 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425267462 (ra…87.74.180.148) 1.62ms
Feb 15 22:41:07 AItesting bash[23379]: [I 2021-02-15 22:41:07.753 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425267642 (ra…87.74.180.148) 6.97ms
Feb 15 22:41:12 AItesting bash[23379]: [I 2021-02-15 22:41:12.788 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425272674 (ra…87.74.180.148) 7.92ms
Feb 15 22:41:15 AItesting bash[23379]: [I 2021-02-15 22:41:15.104 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425274994 (ra…87.74.180.148) 3.37ms
Feb 15 22:41:16 AItesting bash[23379]: [I 2021-02-15 22:41:16.201 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425276091 (ra…87.74.180.148) 1.85ms
Feb 15 22:41:16 AItesting bash[23379]: [I 2021-02-15 22:41:16.290 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425276184 (ra…87.74.180.148) 2.09ms
Feb 15 22:41:17 AItesting bash[23379]: [I 2021-02-15 22:41:17.596 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425277490 (ra…87.74.180.148) 1.50ms
Feb 15 22:41:17 AItesting bash[23379]: [I 2021-02-15 22:41:17.821 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425277708 (ra…87.74.180.148) 8.63ms
Feb 15 22:41:22 AItesting bash[23379]: [I 2021-02-15 22:41:22.854 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425282740 (ra…87.74.180.148) 7.80ms
Feb 15 22:41:25 AItesting bash[23379]: [I 2021-02-15 22:41:25.134 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425285024 (ra…87.74.180.148) 3.26ms
Feb 15 22:41:26 AItesting bash[23379]: [I 2021-02-15 22:41:26.227 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425286121 (ra…87.74.180.148) 1.47ms
Feb 15 22:41:26 AItesting bash[23379]: [I 2021-02-15 22:41:26.314 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425286208 (ra…87.74.180.148) 1.50ms
Feb 15 22:41:27 AItesting bash[23379]: [I 2021-02-15 22:41:27.622 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425287515 (ra…87.74.180.148) 1.42ms
Feb 15 22:41:27 AItesting bash[23379]: [I 2021-02-15 22:41:27.884 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425287773 (ra…87.74.180.148) 6.97ms
Feb 15 22:41:28 AItesting bash[23379]: [I 2021-02-15 22:41:28.489 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernelspecs?1613425288381 (ra…87.74.180.148) 3.30ms
Feb 15 22:41:32 AItesting bash[23379]: [I 2021-02-15 22:41:32.916 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425292804 (ra…87.74.180.148) 7.15ms
Feb 15 22:41:35 AItesting bash[23379]: [I 2021-02-15 22:41:35.162 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425295054 (ra…87.74.180.148) 2.69ms
Feb 15 22:41:36 AItesting bash[23379]: [I 2021-02-15 22:41:36.252 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425296146 (ra…87.74.180.148) 1.45ms
Feb 15 22:41:36 AItesting bash[23379]: [I 2021-02-15 22:41:36.340 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425296235 (ra…87.74.180.148) 1.49ms
Feb 15 22:41:37 AItesting bash[23379]: [I 2021-02-15 22:41:37.649 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425297543 (ra…87.74.180.148) 1.59ms
Feb 15 22:41:37 AItesting bash[23379]: [I 2021-02-15 22:41:37.949 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425297836 (ra…87.74.180.148) 7.20ms
Feb 15 22:41:42 AItesting bash[23379]: [I 2021-02-15 22:41:42.982 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425302868 (ra…87.74.180.148) 6.77ms
Feb 15 22:41:45 AItesting bash[23379]: [I 2021-02-15 22:41:45.194 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425305084 (ra…87.74.180.148) 2.79ms
Feb 15 22:41:46 AItesting bash[23379]: [I 2021-02-15 22:41:46.284 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425306173 (ra…87.74.180.148) 1.75ms
Feb 15 22:41:46 AItesting bash[23379]: [I 2021-02-15 22:41:46.366 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425306261 (ra…87.74.180.148) 1.69ms
Feb 15 22:41:47 AItesting bash[23379]: [I 2021-02-15 22:41:47.674 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425307569 (ra…87.74.180.148) 1.15ms
Feb 15 22:41:48 AItesting bash[23379]: [I 2021-02-15 22:41:48.019 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425307903 (ra…87.74.180.148) 11.07ms