I hate this so much, I want to vomit blood right now.
So, I have a docker with Ubuntu 20.04 image, where I have jupyterhub installed. So far, I had apparently had jupyterhub 1.5.0; and I had always started it in the image directly from the command line with:
jupyterhub -f /etc/jupyterhub/jupyterhub_config.py
… where in
jupyterhub_config.py I’ve only had
c.Spawner.notebook_dir = '/home/mydir', which still exists as folder. And everything used to work fine.
So, I manage the jupyterhub and jupyter installation via
pip-sync, and today I updated that - and jupyterhub got updated to 2.0.0; and I started getting these problems. I even generated a new config file (and added the above line) - still nothing.
First I experienced this:
# jupyterhub -f /etc/jupyterhub/jupyterhub_config.py [I 2021-12-19 12:36:46.886 JupyterHub app:2717] Running JupyterHub version 2.0.0 [I 2021-12-19 12:36:46.887 JupyterHub app:2747] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.0.0 [I 2021-12-19 12:36:46.887 JupyterHub app:2747] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-2.0.0 [I 2021-12-19 12:36:46.887 JupyterHub app:2747] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.0.0 [I 2021-12-19 12:36:46.890 JupyterHub app:1596] Loading cookie_secret from /jupyterhub_cookie_secret Found database schema version 4dc2d5a8c53c != 833da8570507. Backup your database and run `jupyterhub upgrade-db` to upgrade to the latest schema.
… and i tried
jupyterhub upgrade-db - but ultimately, that doesn’t matter, because the
.sqlite file apparently gets created in the directory where you run
jupyterhub from; so in the end I deleted the
.sqlite file, and it got recreated on next run.
The big problem is, that once I open
127.0.0.1:8000; I get asked to log in - I log in with the user I made for this purpose (
juser), and once the progressbar is shown, ultimately I get this, instead of the list of files in the start folder:
404 : Not Found You are requesting a page that does not exist!
And the JupyterHub log in terminal says only:
[I 2021-12-19 13:24:43.901 SingleUserNotebookApp log:189] 302 GET /user/juser/tree/? -> /user/juser/tree? (@::ffff:172.17.0.1) 0.48ms [W 2021-12-19 13:24:43.986 SingleUserNotebookApp log:189] 404 GET /user/juser/tree? (@::ffff:172.17.0.1) 20.79ms
… AAAAH!!! WHICH is this page I’m requesting, that does not exist?
/user/juser/tree/? is not a file path anyways …
How do I debug this (i.e., how do I find what resource is jupyterhub looking for, that it cannot find), and how do I get jupyterhub to start working again?
At least, I can confirm this still works in my image (note, ip must be 0.0.0.0, else the server is not accessible outside of the docker image):
# jupyter notebook --port 8000 --ip 0.0.0.0 --allow-root --no-browser
Anyways, in hopes of finding what generates this error, I tried to trace:
# # python3 -m trace --ignore-dir=/usr/lib/python38.zip:/usr/libython3.8:/usr/lib/python3.8/lib-dynload:/usr/lib/python3/dist-packages -t `which jupyterhub` -f /etc/jupyterhub/jupyterhub_config.py > /path/to/jhub.log 2>&1
… and reproduced the error - and got somewhere here:
--- modulename: iostream, funcname: closed iostream.py(662): return self._closed iostream.py(1027): if ( http1connection.py(300): return True http1connection.py(838): if not ret: http1connection.py(840): await asyncio.sleep(0) --- modulename: web, funcname: <lambda> web.py(2326): fut.add_done_callback(lambda f: f.result()) --- modulename: http1connection, funcname: _server_request_loop http1connection.py(820): conn = HTTP1Connection(self.stream, False, self.params, self.context) --- modulename: http1connection, funcname: __init__ http1connection.py(126): self.is_client = is_client ... --- modulename: httpserver, funcname: start_request httpserver.py(234): if isinstance(self.request_callback, httputil.HTTPServerConnectionDelegate): httpserver.py(235): delegate = self.request_callback.start_request(server_conn, request_conn) --- modulename: routing, funcname: start_request routing.py(210): return _RoutingDelegate(self, server_conn, request_conn) --- modulename: routing, funcname: __init__ routing.py(233): self.server_conn = server_conn routing.py(234): self.request_conn = request_conn routing.py(235): self.delegate = None # type: Optional[httputil.HTTPMessageDelegate] routing.py(236): self.router = router # type: Router ... --- modulename: iostream, funcname: _maybe_add_error_listener iostream.py(1026): if self._state is None or self._state == ioloop.IOLoop.ERROR: iostream.py(1028): not self.closed() --- modulename: iostream, funcname: closed iostream.py(662): return self._closed iostream.py(1027): if ( http1connection.py(824): except ( http1connection.py(825): iostream.StreamClosedError, http1connection.py(826): iostream.UnsatisfiableReadError, http1connection.py(827): asyncio.CancelledError, [I 2021-12-19 16:28:18.338 SingleUserNotebookApp log:189] 302 GET /user/juser/ -> /user/juser/tree/? (@::ffff:172.17.0.1) 1.65ms [I 2021-12-19 16:28:18.364 SingleUserNotebookApp log:189] 302 GET /user/juser/tree/? -> /user/juser/tree? (@::ffff:172.17.0.1) 0.88ms [W 2021-12-19 16:28:18.415 SingleUserNotebookApp log:189] 404 GET /user/juser/tree? (@::ffff:172.17.0.1) 34.09ms http1connection.py(824): except ( http1connection.py(829): return http1connection.py(842): delegate.on_close(self) --- modulename: httpserver, funcname: on_close httpserver.py(245): self._connections.remove(typing.cast(HTTP1ServerConnection, server_conn)) http1connection.py(829): return --- modulename: ioloop, funcname: <lambda> ioloop.py(688): lambda f: self._run_callback(functools.partial(callback, future)) --- modulename: ioloop, funcname: _run_callback ...
Well, I still can’t really see where do these printouts come from … What a goddamn waste of time …