Error in fresh jupyterhub install: [ConfigProxy] error: 404 GET /socket.io/

jan_e · August 26, 2024, 10:37am

Hi,

i have a fresh install of jupyterhub on a server running Ubuntu 24.04 using this guide. nginx is used as proxy. The jupyterhub login page is accesible by /jupyter/.
However, the log is filled with entries:

Aug 26 12:16:02 <server-name> jupyterhub[3670]: 12:16:02.021 [ConfigProxy] error: 404 GET /socket.io/
Aug 26 12:16:02 <server-name> jupyterhub[3670]: 12:16:02.021 [ConfigProxy] debug: Requesting custom error page: http://127.0.0.1:8081/jupyter/hub/error/404?url=%2Fsocket.io%2F%3FEIO%3D3%26transport%3Dpolling%26t%3DP6EBtgL%26b64%3D1
Aug 26 12:16:02 <server-name> jupyterhub[3667]: [I 2024-08-26 12:16:02.022 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:EsoXtJryKJQ28wPgFmAwoh5SXSZuIJJnQzgBqP1AcaA=' {'path': '/jupyter/hub/', 'max_age': 3600}
Aug 26 12:16:02 <server-name> jupyterhub[3667]: [I 2024-08-26 12:16:02.023 JupyterHub log:192] 200 GET /jupyter/hub/error/404?url=%2Fsocket.io%2F%3FEIO%3D3%26transport%3Dpolling%26t%3DP6EBtgL%26b64%3D1 (@127.0.0.1) 1.46ms
Aug 26 12:16:02 <server-name> jupyterhub[3670]: 12:16:02.145 [ConfigProxy] error: 404 GET /socket.io/
Aug 26 12:16:02 <server-name> jupyterhub[3670]: 12:16:02.145 [ConfigProxy] debug: Requesting custom error page: http://127.0.0.1:8081/jupyter/hub/error/404?url=%2Fsocket.io%2F%3FEIO%3D3%26transport%3Dpolling%26t%3DP6EBtib%26b64%3D1
Aug 26 12:16:02 <server-name> jupyterhub[3667]: [I 2024-08-26 12:16:02.146 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:EsoXtJryKJQ28wPgFmAwoh5SXSZuIJJnQzgBqP1AcaA=' {'path': '/jupyter/hub/', 'max_age': 3600}
Aug 26 12:16:02 <server-name> jupyterhub[3667]: [I 2024-08-26 12:16:02.147 JupyterHub log:192] 200 GET /jupyter/hub/error/404?url=%2Fsocket.io%2F%3FEIO%3D3%26transport%3Dpolling%26t%3DP6EBtib%26b64%3D1 (@127.0.0.1) 1.30ms

A curl http://127.0.0.1:8081/jupyter/hub/error shows the Jupyter has lots of moons,… page.

My config in /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py is (only non-defaults):

#  Default: 'http://:8000'
c.JupyterHub.bind_url = 'http://:8000/jupyter/'
#  Default: ''
c.Spawner.default_url = '/lab'

My config in /etc/nginx/sites-available/default is

map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

# Default server configuration
#
server {
	listen 80 default_server;
	listen [::]:80 default_server;

	root /var/www/html;

	# Add index.php to the list if you are using PHP
	index index.html index.htm index.nginx-debian.html;

	server_name _;

	location / {
		# First attempt to serve request as file, then
		# as directory, then fall back to displaying a 404.
		try_files $uri $uri/ =404;
	}

        location /jupyter/ {
            # NOTE important to also set base url of jupyterhub to /jupyter in its config
            proxy_pass http://127.0.0.1:8000;
        
            proxy_redirect   off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # websocket headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;

        }
}

The command sudo systemctl status jupyterhub returns:

 jupyterhub.service - JupyterHub
     Loaded: loaded (/etc/systemd/system/jupyterhub.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-08-26 12:03:46 CEST; 24min ago
   Main PID: 3667 (jupyterhub)
      Tasks: 12 (limit: 629145)
     Memory: 103.9M (peak: 109.1M)
        CPU: 6.327s
     CGroup: /system.slice/jupyterhub.service
             ├─3667 /opt/jupyterhub/bin/python3 /opt/jupyterhub/bin/jupyterhub -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py
             └─3670 node /usr/local/bin/configurable-http-proxy --ip "" --port 8000 --api-ip 127.0.0.1 --api-port 8001 --error-target http://127.0.0.1:8081/jupyter/hub/error --log-level debug

Aug 26 12:27:58 <server-name> jupyterhub[3667]: [I 2024-08-26 12:27:58.710 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:EsoXtJryKJQ28wPgFmAwoh5SXSZuIJJnQzgBqP1AcaA=' {'p>
Aug 26 12:27:58 <server-name> jupyterhub[3667]: [I 2024-08-26 12:27:58.711 JupyterHub log:192] 200 GET /jupyter/hub/error/404?url=%2Fsocket.io%2F%3FEIO%3D3%26transport%3Dpolling%26t%3DP6E>
Aug 26 12:27:58 <server-name> jupyterhub[3670]: 12:27:58.728 [ConfigProxy] error: 404 GET /socket.io/
Aug 26 12:27:58 <server-name> jupyterhub[3670]: 12:27:58.728 [ConfigProxy] debug: Requesting custom error page: http://127.0.0.1:8081/jupyter/hub/error/404?url=%2Fsocket.io%2F%3FEIO%3D3%2>
Aug 26 12:27:58 <server-name> jupyterhub[3667]: [I 2024-08-26 12:27:58.729 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:EsoXtJryKJQ28wPgFmAwoh5SXSZuIJJnQzgBqP1AcaA=' {'p>
Aug 26 12:27:58 <server-name> jupyterhub[3667]: [I 2024-08-26 12:27:58.730 JupyterHub log:192] 200 GET /jupyter/hub/error/404?url=%2Fsocket.io%2F%3FEIO%3D3%26transport%3Dpolling%26t%3DP6E>
Aug 26 12:27:58 <server-name> jupyterhub[3670]: 12:27:58.749 [ConfigProxy] error: 404 GET /socket.io/
Aug 26 12:27:58 <server-name> jupyterhub[3670]: 12:27:58.749 [ConfigProxy] debug: Requesting custom error page: http://127.0.0.1:8081/jupyter/hub/error/404?url=%2Fsocket.io%2F%3FEIO%3D3%2>
Aug 26 12:27:58 <server-name> jupyterhub[3667]: [I 2024-08-26 12:27:58.750 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:EsoXtJryKJQ28wPgFmAwoh5SXSZuIJJnQzgBqP1AcaA=' {'p>
Aug 26 12:27:58 <server-name> jupyterhub[3667]: [I 2024-08-26 12:27:58.751 JupyterHub log:192] 200 GET /jupyter/hub/error/404?url=%2Fsocket.io%2F%3FEIO%3D3%26transport%3Dpolling%26t%3DP6E>

On startup, the log shows:

Aug 26 12:33:42 <server-name> jupyterhub[3812]: [W 2024-08-26 12:33:42.067 JupyterHub app:964] JupyterHub.debug_proxy is deprecated in JupyterHub 0.8, use ConfigurableHTTPProxy.debug
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.067 JupyterHub app:3307] Running JupyterHub version 5.1.0
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.067 JupyterHub app:3337] Using Authenticator: jupyterhub.auth.PAMAuthenticator-5.1.0
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.067 JupyterHub app:3337] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-5.1.0
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.067 JupyterHub app:3337] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.1.0
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.068 JupyterHub app:1837] Loading cookie_secret from /jupyterhub_cookie_secret
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.125 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [W 2024-08-26 12:33:42.128 JupyterHub auth:185] No allow config found, it's possible that nobody can login to your Hub!
Aug 26 12:33:42 <server-name> jupyterhub[3812]:     You can set `c.Authenticator.allow_all = True` to allow any user who can login to access the Hub,
Aug 26 12:33:42 <server-name> jupyterhub[3812]:     or e.g. `allowed_users` to a set of users who should have access.
Aug 26 12:33:42 <server-name> jupyterhub[3812]:     You may suppress this warning by setting c.Authenticator.any_allow_config = True.
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.145 JupyterHub app:3376] Initialized 0 spawners in 0.004 seconds
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.147 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.148 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.seven_days
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.148 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.thirty_days
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [W 2024-08-26 12:33:42.148 JupyterHub proxy:748] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else.>
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.148 JupyterHub proxy:752] Starting proxy @ http://:8000/jupyter/
Aug 26 12:33:42 <server-name> jupyterhub[3815]: 12:33:42.302 [ConfigProxy] info: Proxying http://*:8000 to (no default)
Aug 26 12:33:42 <server-name> jupyterhub[3815]: 12:33:42.303 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
Aug 26 12:33:42 <server-name> jupyterhub[3815]: 12:33:42.650 [ConfigProxy] info: 200 GET /api/routes
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.651 JupyterHub app:3690] Hub API listening on http://127.0.0.1:8081/jupyter/hub/
Aug 26 12:33:42 <server-name> jupyterhub[3815]: 12:33:42.652 [ConfigProxy] info: 200 GET /api/routes
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.652 JupyterHub proxy:477] Adding route for Hub: /jupyter/ => http://127.0.0.1:8081
Aug 26 12:33:42 <server-name> jupyterhub[3815]: 12:33:42.654 [ConfigProxy] info: Adding route /jupyter -> http://127.0.0.1:8081
Aug 26 12:33:42 <server-name> jupyterhub[3815]: 12:33:42.654 [ConfigProxy] info: Route added /jupyter -> http://127.0.0.1:8081
Aug 26 12:33:42 <server-name> jupyterhub[3815]: 12:33:42.654 [ConfigProxy] info: 201 POST /api/routes/jupyter
Aug 26 12:33:42 <server-name> jupyterhub[3812]: [I 2024-08-26 12:33:42.655 JupyterHub app:3731] JupyterHub is now running at http://:8000/jupyter/
Aug 26 12:33:45 <server-name> jupyterhub[3815]: 12:33:45.292 [ConfigProxy] error: 404 GET /socket.io/

I tested several things but found no solution to the problem. I would like to have this problem solved before I continue with the SSL configuration.

Thanks in advance
Jan

manics · August 26, 2024, 10:59am

Port 8081 is for internal use. Try using port 8000 instead.

jan_e · August 26, 2024, 1:59pm

I already use port 8000 in c.JupyterHub.bind_url and in proxy_pass http://127.0.0.1:8000;
I did not specify port 8081 myself anywhere.

I now tried to use other ports than 8081 in the config

c.JupyterHub.hub_bind_url = "http://127.0.0.1:8091"
c.JupyterHub.hub_connect_url = "http://127.0.0.1:8091"

or c.JupyterHub.hub_port = 8091 but got the same error as before.

mahendrapaipuri · August 26, 2024, 2:53pm

As you are running JupyterHub at prefix /jupyter you need to configure the base_url for JupyterHub as well to include that prefix in all redirections. You need to add following to your JupyterHub config:

c.JupyterHub.base_url = '/jupyter'

I think what @manics said about port 8081 is you should not send curl requests to port 8081. You should interact with JupyterHub at port 8000 according to the config you posted in the first post.

jan_e · August 26, 2024, 6:31pm

Ah ok, I changed base_url (instead of bind_url). But that doesn’t change anything.

Just to clarify my point: jupyterhub is fully functional. I can access it with <server-ip>/jupyter, log in and start notebooks.
Only the error message `error: 404 GET /socket.io/ is flooding my log files and so something doesn’t seem right yet.
I suspect an error in connection with the reverse proxy config of nginx.

manics · August 26, 2024, 6:50pm

To confirm this can you disable nginx and connect to server-ip:8000 directly, and check the /socket.io/ logs don’t appear?

If the conclusion is it’s Nginx can you share your Nginx logs (access and error logs)?

If it’s not Nginx can you share your browser console and network logs?

jan_e · August 26, 2024, 7:51pm

It seems to have nothing to do with nginx. sudo systemctl stop nginx does not stop the errors in the log.
Jupyterhub is still accessible via http://127.0.0.1:8000/jupyter, but not via http://127.0.0.1:8000 (-> “404 lots of moons …”) and not via <external server-ip>/jupyter (-> "Connection failed’')

I need some time for the logs. But as far as I can see nothing relevant happens in the browser. The errors in the log (journalctl -u jupyterhub) appear permanently every 5 seconds, even if the page is not called and jupyterhub is not used.

minrk · August 27, 2024, 6:10am

Note that JupyterHub does not use socket.io, so the issue is likely not in JupyterHub, but something trying to access /socket.io/ on your server’s port 8000, which is probably not Jupyter-related at all. This could be some kind of network scan, or a left-open application that’s not running anymore and doesn’t gracefully handle a backend shutting down.

A couple of things to try:

If you change the public port of JupyterHub (via Jupyterhub.bind_url), do the logs change?
If you stop jupyterhub and nginx and run the following script to launch a do-nothing server on port 8000:

import asyncio
import tornado
from tornado.log import app_log
from tornado.options import options, define

define("port", default=8000, help="run on the given port", type=int)


async def main():
    tornado.options.parse_command_line()
    # no handlers, everything will 404
    application = tornado.web.Application([])
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(options.port)
    app_log.info(f"Listening on :{options.port}")
    # run forever
    await asyncio.Event().wait()


if __name__ == "__main__":
    asyncio.run(main())

do you start to see the socket.io logs?

Since you’re putting JupyterHub behind nginx, you might also want to bind JupyterHub to localhost instead of a public interface:

c.JupyterHub.bind_url = 'http://127.0.0.1:8000/jupyter/'

That way, the only way to access JupyterHub from outside is through nginx and port 8000 is not exposed publicly.

jan_e · August 27, 2024, 3:05pm

You are absolutely right, and your script put me on the right track. I found that my VPN client (PulseSecure), which is practically always running, is causing the problems.
Switching to port 9000 solved the problems:
c.JupyterHub.bind_url = ‘http://127.0.0.1:9000/jupyter’
and
proxy_pass http://127.0.0.1:9000;

Many thanks for the quick help!

Topic		Replies	Views
Web Socket Connection Failed JupyterHub community , how-to , help-wanted	4	1595	November 6, 2024
Getting error while using jupyterhub + Nginx reverse proxy JupyterHub community , jupyterhub	9	88	September 14, 2024
404 error from fresh install of jupyterhub JupyterHub	2	1565	July 7, 2022
Getting [ConfigProxy] Proxy error: Error: socket hang up frequently JupyterHub	1	363	October 13, 2023
Getting 404 Not found error while trying to deploy jupyterhub in kubernetes JupyterHub jupyterhub	2	613	November 26, 2022

Error in fresh jupyterhub install: [ConfigProxy] error: 404 GET /socket.io/

Related topics