Can't open jupyterhub with 'not found' and ECONNREFUSED

JupyterHub
1

Hi there!

I’m trying to install and use jupyterhub on my ubuntu 22.04 home server.
I tried both installing it from ubuntu repos and installing it from pip using virtual environment.
Both methods give the same result, however, I’m gonna be using the latter method with pip from now on.

After installing it with pip alongside with configurable-http-proxy from npm, I start the hub, and try to open it in my browser on localhost:8000. Doing that, I just see ‘Not found’ on the page, and here is the console output when starting it with jupyterhub --debug:

boris@bombrix in ~/tmp/jupyterhub via 🐍 v3.10.4 (jupyterhub) ➜ jupyterhub --debug
[D 2022-05-16 18:41:55.359 JupyterHub application:731] Looking for jupyterhub_config in /home/boris/tmp/jupyterhub
[I 2022-05-16 18:41:55.359 JupyterHub app:2769] Running JupyterHub version 2.3.0
[I 2022-05-16 18:41:55.360 JupyterHub app:2799] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.3.0
[I 2022-05-16 18:41:55.360 JupyterHub app:2799] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-2.3.0
[I 2022-05-16 18:41:55.360 JupyterHub app:2799] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.3.0
[D 2022-05-16 18:41:55.361 JupyterHub app:2730] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[I 2022-05-16 18:41:55.361 JupyterHub app:1606] Loading cookie_secret from /home/boris/tmp/jupyterhub/jupyterhub_cookie_secret
[D 2022-05-16 18:41:55.362 JupyterHub app:1773] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2022-05-16 18:41:55.378 JupyterHub orm:955] database schema version found: 833da8570507
[I 2022-05-16 18:41:55.426 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2022-05-16 18:41:55.427 JupyterHub app:2022] Loading roles into database
[I 2022-05-16 18:41:55.436 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
[D 2022-05-16 18:41:55.441 JupyterHub app:2281] Purging expired APITokens
[D 2022-05-16 18:41:55.444 JupyterHub app:2281] Purging expired OAuthCodes
[D 2022-05-16 18:41:55.446 JupyterHub app:2114] Loading role assignments from config
[D 2022-05-16 18:41:55.456 JupyterHub app:2427] Initializing spawners
[D 2022-05-16 18:41:55.460 JupyterHub app:2558] Loaded users:
    
[I 2022-05-16 18:41:55.460 JupyterHub app:2838] Initialized 0 spawners in 0.004 seconds
[W 2022-05-16 18:41:55.461 JupyterHub proxy:565] Found proxy pid file: /home/boris/tmp/jupyterhub/jupyterhub-proxy.pid
[W 2022-05-16 18:41:55.462 JupyterHub proxy:582] Proxy still running at pid=138817
[W 2022-05-16 18:41:56.463 JupyterHub proxy:602] Stopped proxy at pid=138817
[D 2022-05-16 18:41:56.463 JupyterHub proxy:618] Removing proxy pid file jupyterhub-proxy.pid
[W 2022-05-16 18:41:56.465 JupyterHub proxy:687] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2022-05-16 18:41:56.465 JupyterHub proxy:691] Starting proxy @ http://:8000
[D 2022-05-16 18:41:56.465 JupyterHub proxy:692] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error']
[D 2022-05-16 18:41:56.467 JupyterHub proxy:610] Writing proxy pid file: jupyterhub-proxy.pid
18:41:56.647 [ConfigProxy] info: Proxying http://*:8000 to (no default)
18:41:56.649 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
18:42:16.003 [ConfigProxy] error: 404 GET / 
18:42:16.036 [ConfigProxy] error: Failed to get custom error page: Error: connect ECONNREFUSED 127.0.0.1:8081
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 8081
}
18:42:16.198 [ConfigProxy] error: 404 GET /favicon.ico 
18:42:16.202 [ConfigProxy] error: Failed to get custom error page: Error: connect ECONNREFUSED 127.0.0.1:8081
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 8081
}

I tried changing some default params in config file, but had no luck.

The only thing that changed this behavior was changing bind_url for internal hub server, which is by default located (if I understand correctly) at localhost:8081. It seems that some service on ubuntu is using that port by default. The related problem might have been discussed here Questions Tags Users Unanswered Jupyter Hub says simply “not found” and nothing more, rendering it unusable, but it has no meaningful answer.

I tried changing default url from 8081 to other random port, and the answer from server becomes just a simple JSON with 404 error.

What can be done to fix that?

2

This problem seems to be ubuntu-specific, bc installing fresh hub on arch linux works just fine out of the box…

3

Hi! Could you show us your JupyterHub configuration with secrets redacted, after you made the change to avoid port 8081?

4

Hey! Thank you for the answer.

I tried replicating it, but had no luck getting anything else. The presented error persists (I guess, the time it was different, I just messed up something).

This time I tried setting c.JupyterHub.hub_bind_url = "http://127.0.0.1/8082" and c.JupyterHub.hub_port = 8082. Nothing changed except the port in error message.

5

I guess, this might not be the actual issue, because lsof doesn’t show anything there

boris@bombrix in ~ ➜ lsof -i -P -n
COMMAND    PID  USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
python    1120 boris    3u  IPv4 1205924      0t0  TCP 192.168.1.69:40070->149.154.167.220:443 (ESTABLISHED)
megasync  3841 boris   57u  IPv4  838715      0t0  TCP 192.168.1.69:44092->66.203.125.14:443 (ESTABLISHED)
megasync  3841 boris   61u  IPv4   36385      0t0  TCP 127.0.0.1:6341 (LISTEN)
megasync  3841 boris   63u  IPv4 1173964      0t0  TCP 127.0.0.1:6342 (LISTEN)
node     92120 boris   21u  IPv6 1204106      0t0  TCP *:8000 (LISTEN)
node     92120 boris   22u  IPv4 1204107      0t0  TCP 127.0.0.1:8001 (LISTEN)

But, in /etc/services there’s an entry tproxy 8081/tcp