I hate this so much, I want to vomit blood right now.
So, I have a docker with Ubuntu 20.04 image, where I have jupyterhub installed. So far, I had apparently had jupyterhub 1.5.0; and I had always started it in the image directly from the command line with:
jupyterhub -f /etc/jupyterhub/jupyterhub_config.py
… where in jupyterhub_config.py I’ve only had c.Spawner.notebook_dir = '/home/mydir', which still exists as folder. And everything used to work fine.
So, I manage the jupyterhub and jupyter installation via pip-compile and pip-sync, and today I updated that - and jupyterhub got updated to 2.0.0; and I started getting these problems. I even generated a new config file (and added the above line) - still nothing.
First I experienced this:
# jupyterhub -f /etc/jupyterhub/jupyterhub_config.py
[I 2021-12-19 12:36:46.886 JupyterHub app:2717] Running JupyterHub version 2.0.0
[I 2021-12-19 12:36:46.887 JupyterHub app:2747] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.0.0
[I 2021-12-19 12:36:46.887 JupyterHub app:2747] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-2.0.0
[I 2021-12-19 12:36:46.887 JupyterHub app:2747] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.0.0
[I 2021-12-19 12:36:46.890 JupyterHub app:1596] Loading cookie_secret from /jupyterhub_cookie_secret
Found database schema version 4dc2d5a8c53c != 833da8570507. Backup your database and run `jupyterhub upgrade-db` to upgrade to the latest schema.
… and i tried jupyterhub upgrade-db - but ultimately, that doesn’t matter, because the .sqlite file apparently gets created in the directory where you run jupyterhub from; so in the end I deleted the .sqlite file, and it got recreated on next run.
The big problem is, that once I open 127.0.0.1:8000; I get asked to log in - I log in with the user I made for this purpose (juser), and once the progressbar is shown, ultimately I get this, instead of the list of files in the start folder:
404 : Not Found
You are requesting a page that does not exist!
And the JupyterHub log in terminal says only:
[I 2021-12-19 13:24:43.901 SingleUserNotebookApp log:189] 302 GET /user/juser/tree/? -> /user/juser/tree? (@::ffff:172.17.0.1) 0.48ms
[W 2021-12-19 13:24:43.986 SingleUserNotebookApp log:189] 404 GET /user/juser/tree? (@::ffff:172.17.0.1) 20.79ms
… AAAAH!!! WHICH is this page I’m requesting, that does not exist? /user/juser/tree/? is not a file path anyways …
How do I debug this (i.e., how do I find what resource is jupyterhub looking for, that it cannot find), and how do I get jupyterhub to start working again?
At least, I can confirm this still works in my image (note, ip must be 0.0.0.0, else the server is not accessible outside of the docker image):
# jupyter notebook --port 8000 --ip 0.0.0.0 --allow-root --no-browser
Anyways, in hopes of finding what generates this error, I tried to trace:
# # python3 -m trace --ignore-dir=/usr/lib/python38.zip:/usr/libython3.8:/usr/lib/python3.8/lib-dynload:/usr/lib/python3/dist-packages -t `which jupyterhub` -f /etc/jupyterhub/jupyterhub_config.py > /path/to/jhub.log 2>&1
… and reproduced the error - and got somewhere here:
--- modulename: iostream, funcname: closed
iostream.py(662): return self._closed
iostream.py(1027): if (
http1connection.py(300): return True
http1connection.py(838): if not ret:
http1connection.py(840): await asyncio.sleep(0)
--- modulename: web, funcname: <lambda>
web.py(2326): fut.add_done_callback(lambda f: f.result())
--- modulename: http1connection, funcname: _server_request_loop
http1connection.py(820): conn = HTTP1Connection(self.stream, False, self.params, self.context)
--- modulename: http1connection, funcname: __init__
http1connection.py(126): self.is_client = is_client
...
--- modulename: httpserver, funcname: start_request
httpserver.py(234): if isinstance(self.request_callback, httputil.HTTPServerConnectionDelegate):
httpserver.py(235): delegate = self.request_callback.start_request(server_conn, request_conn)
--- modulename: routing, funcname: start_request
routing.py(210): return _RoutingDelegate(self, server_conn, request_conn)
--- modulename: routing, funcname: __init__
routing.py(233): self.server_conn = server_conn
routing.py(234): self.request_conn = request_conn
routing.py(235): self.delegate = None # type: Optional[httputil.HTTPMessageDelegate]
routing.py(236): self.router = router # type: Router
...
--- modulename: iostream, funcname: _maybe_add_error_listener
iostream.py(1026): if self._state is None or self._state == ioloop.IOLoop.ERROR:
iostream.py(1028): not self.closed()
--- modulename: iostream, funcname: closed
iostream.py(662): return self._closed
iostream.py(1027): if (
http1connection.py(824): except (
http1connection.py(825): iostream.StreamClosedError,
http1connection.py(826): iostream.UnsatisfiableReadError,
http1connection.py(827): asyncio.CancelledError,
[I 2021-12-19 16:28:18.338 SingleUserNotebookApp log:189] 302 GET /user/juser/ -> /user/juser/tree/? (@::ffff:172.17.0.1) 1.65ms
[I 2021-12-19 16:28:18.364 SingleUserNotebookApp log:189] 302 GET /user/juser/tree/? -> /user/juser/tree? (@::ffff:172.17.0.1) 0.88ms
[W 2021-12-19 16:28:18.415 SingleUserNotebookApp log:189] 404 GET /user/juser/tree? (@::ffff:172.17.0.1) 34.09ms
http1connection.py(824): except (
http1connection.py(829): return
http1connection.py(842): delegate.on_close(self)
--- modulename: httpserver, funcname: on_close
httpserver.py(245): self._connections.remove(typing.cast(HTTP1ServerConnection, server_conn))
http1connection.py(829): return
--- modulename: ioloop, funcname: <lambda>
ioloop.py(688): lambda f: self._run_callback(functools.partial(callback, future))
--- modulename: ioloop, funcname: _run_callback
...
Well, I still can’t really see where do these printouts come from … What a goddamn waste of time …