Using Batchspawner with a reverse proxy

JupyterHub
1

Hi everyone,

I’ve gotten batchspawner working nicely with my slurm based cluster without the use of a reverse proxy. I would like to use a reverse proxy in order to host some other information from the head node of this cluster.

I followed the directions in here:

And seem to have found success if not running using the batchspawner. However, because my c.JupyterHub.bind_url is now set to 127.0.0.1:8000/jhub/ (from following the instructions for the reverse proxy), it seems that batchspawner jobs (which end up running on completely different machines) try to connect to the hub at that location. Of course, that fails, because they’re trying to connect to themself!

So, what seems to be the solution to this is to set the batchspawner hub_connect_url. I found the local network IP of my cluster head node, as is accessible from the worker nodes. Then set this in my jupyterhub_config:

c.SlurmSpawner.hub_connect_url = 'http://172.16.33.254:8000/'

However, my job run output fails with this error:

Error connecting to http://172.16.33.254:8000/jhub/hub/api: [Errno 111] Connection refused
Traceback (most recent call last):
  File "/home/spack/opt/spack/linux-centos7-sandybridge/gcc-11.2.0/python-3.9.10-2luse2jko74ictdwekggecxci76g3rso/lib/python3.9/site-packages/jupyterhub/services/auth.py", line 475, in _api_request
    r = await AsyncHTTPClient().fetch(req, raise_error=False)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/spack/opt/spack/linux-centos7-sandybridge/gcc-11.2.0/python-3.9.10-2luse2jko74ictdwekggecxci76g3rso/bin/batchspawner-singleuser", line 8, in <module>
    sys.exit(main())
  File "/home/spack/opt/spack/linux-centos7-sandybridge/gcc-11.2.0/python-3.9.10-2luse2jko74ictdwekggecxci76g3rso/lib/python3.9/site-packages/batchspawner/singleuser.py", line 32, in main
    asyncio.run(
  File "/home/spack/opt/spack/linux-centos7-sandybridge/gcc-11.2.0/python-3.9.10-2luse2jko74ictdwekggecxci76g3rso/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/spack/opt/spack/linux-centos7-sandybridge/gcc-11.2.0/python-3.9.10-2luse2jko74ictdwekggecxci76g3rso/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/spack/opt/spack/linux-centos7-sandybridge/gcc-11.2.0/python-3.9.10-2luse2jko74ictdwekggecxci76g3rso/lib/python3.9/site-packages/jupyterhub/services/auth.py", line 488, in _api_request
    raise HTTPError(500, msg)
tornado.web.HTTPError: HTTP 500: Internal Server Error (Failed to connect to Hub API at 'http://172.16.33.254:8000/jhub/hub/api'.  Is the Hub accessible at this URL (from host: n011.cluster.com)?)
srun: error: n011: task 0: Exited with exit code 1

Why is the connection refused? Do I need to change some firewall settings so that this port can receive things over the network? I’m not sure how to do that, and would like some assistance.

2

The connection is refused because either:

  • the ip:port is not actually where the service is running (a configuration issue)
  • the ip:port is correct, but not reachable from your worker node (a firewall issue)

So if you’ve set your bind_url to 127.0.0.1:8000, that means the only accessible ip:port is the public ip:port of your reverse proxy. I’m guessing it’s unlikely that your reverse proxy is also running on port 8000. Is that correct? If so, this isn’t the URL you mean. I think you mean the reverse proxy URL, which hasn’t been given, I think.

However

Typically, Spawners connect directly to the Hub, not via an external-facing proxy. For multi-host deployments, that typically means instructing the hub process to listen on all interfaces (or one particular ip), instead of localhost (which is the default)

# 0.0.0.0 tells the OS to listen on 'all ipv4 interfaces'
# so spawners can connect
c.JupyterHub.hub_bind_url = 'http://0.0.0.0:8081'

No more information than that should be needed if the Hub’s node is connectable from the worker nodes at socket.gethostname().

More details

Since you are using an additional reverse proxy, there are 3 listening services to consider, but only 2 URLs that should ever be chosen for connecting (the ‘public’ url of the whole deployment and the ‘internal’ url of the Hub component):

  • JupyterHub’s own proxy (specified by c.JupyterHub.bind_url). This is typically the public URL for JupyterHub, and not used internally at all.
  • Your reverse proxy. You haven’t specified the URL, but this now represents the public URL, and anywhere you would have used the above proxy URL, you would now use the URL for this reverse proxy. The only thing that should ever connect directly to c.JupyterHub.bind_url is this proxy.
  • the Hub itself, specified via c.JupyterHub.hub_bind_url (default: http://127.0.0.1:8081). Typically, Spawners connect directly to this, and it must be changed from the default to work with remote Spawners. It’s almost always 127.0.0.1 or 0.0.0.0.

You can access the Hub from Spawners via the public URL (your reverse proxy) if there’s a reason to (e.g. networking weirdness, not running on a single network, etc.), but it’s not usually the way to go because it often means a longer trip through networking devices when the hub and spawner are on the same network.

3

Thanks @minrk! I am getting the suspicion that both of these things are coming into play for me. I’ll attempt to break the problem into pieces. In the first, I am a little confused by the use of hub_bind_url.

If I follow the directions in the “Using a reverse proxy” guide above, it suggests setting:

RewriteEngine On
RewriteCond %{HTTP:Connection} Upgrade [NC]
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule /jhub/(.*) ws://127.0.0.1:8000/jhub/$1 [P,L]
RewriteRule /jhub/(.*) http://127.0.0.1:8000/jhub/$1 [P,L]

<Location "/jhub/">
ProxyPreserveHost on
ProxyPass         http://127.0.0.1:8000/jhub/
ProxyPassReverse  http://127.0.0.1:8000/jhub/
# RequestHeader set X-Forwarded-Proto "https"
# RequestHeader set X-Forwarded-Port "443"
RequestHeader     set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}e
</Location>

with the ultimate goal of putting the hub at port 8000, only locally. If I set in my config:

c.JupyterHub.bind_url = 'http://127.0.0.1:8000/jhub/'

then I get this error in my browser when I attempt to access the page:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /jhub/.

Reason: Error reading from remote server

However, if I instead set another variable to point at port 8000:

c.JupyterHub.bind_url = 'http://127.0.0.1:8081/jhub/'
c.JupyterHub.hub_bind_url = 'http://127.0.0.1:8000/jhub/'

then I’m able to access the web interface to jupyterhub just fine! At this point, the problem is reduced only to having the worker nodes connect to the hub.

So, it’s my understanding from what you said that JupyterHub.bind_url should be set to where the web interface will be, and JupyterHub.hub_bind_url should be what workers connect to. However, the latter setting suggests the opposite. Maybe I’ve not read the documentation carefully enough.

why do I need to set hub_bind_url to where my apache server is reverse proxying to? This is contrary to the “Using a reverse proxy” webpage.

4

You definitely should not do that. Your reverse proxy should never communicate directly with the Hub, only via the Hub’s own proxy (configurable-http-proxy, by default). As soon as you try to launch a user server, this should start to be a problem (you will likely see redirect loops, as requests that should be proxied to user servers keep getting routed to the Hub).

Can you share more of your jupyterhub config and the logs from configurable-http-proxy and jupyterhub when you see the invalid response?

5

Sure! Here’s the config I’m running.

import batchspawner
c.JupyterHub.spawner_class = 'batchspawner.SlurmSpawner'
c.SlurmSpawner.req_prologue="""
module load python-3.9.10-gcc-11.2.0-2luse2j
module load py-pip-21.3.1-gcc-11.2.0-5c5u33g
module load py-openmc-0.13.0-gcc-11.2.0-f73n4wv
module load openmc-0.13.0-gcc-11.2.0-vil5e44
"""

c.JupyterHub.ssl_cert='/etc/letsencrypt/live/((my server URL))/fullchain.pem'
c.JupyterHub.ssl_key='/etc/letsencrypt/live/((my server URL))/privkey.pem'
c.JupyterHub.bind_url = 'http://127.0.0.1:8000/jhub/'
c.JupyterHub.hub_bind_url = 'http://0.0.0.0:8081/jhub/'

# local network address of the head node
c.SlurmSpawner.hub_connect_url = 'http://172.16.33.254:8081/jhub/'

I’ve tested that “ssh 172.16.33.254” takes me back to the head node from a worker node, and it’s indeed the case. I believe that below is all the relevant apache config info:

Listen 443 https

# redirect http to https
# Listen 80
<VirtualHost *:80>
  ServerName ((my domain name))
  Redirect permanent / https://((my domain name))/
</VirtualHost>

Timeout 2400
ProxyTimeout 2400
ProxyBadHeader Ignore

SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
SSLSessionCache         shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout  300
SSLRandomSeed startup file:/dev/urandom  256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin

<VirtualHost *:443>

DocumentRoot "/var/www/html"
ServerName ((my domain name))

ErrorLog logs/ssl_error_log
TransferLog logs/ssl_access_log
LogLevel warn
SSLEngine on

SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-R
SA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256
-GCM-SHA384
SSLHonorCipherOrder     off
SSLSessionTickets       off

SSLCertificateFile /etc/letsencrypt/live/((my domain name))/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/((my domain name))/privkey.pem
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>
BrowserMatch "MSIE [2-5]" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
RewriteEngine On
RewriteCond %{HTTP:Connection} Upgrade [NC]
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteRule /jhub/(.*) ws://127.0.0.1:8000/jhub/$1 [P,L]
RewriteRule /jhub/(.*) http://127.0.0.1:8000/jhub/$1 [P,L]
Header always set Strict-Transport-Security "max-age=63072000"

<Location "/jhub/">
ProxyPreserveHost on
ProxyPass         http://127.0.0.1:8000/jhub/
ProxyPassReverse  http://127.0.0.1:8000/jhub/
RequestHeader     set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}e
</Location>

</VirtualHost>

When I try to run jupyterhub, here’s the output:

[I 2023-03-09 12:54:11.224 JupyterHub app:2810] Running JupyterHub version 3.1.1
[I 2023-03-09 12:54:11.224 JupyterHub app:2840] Using Authenticator: jupyterhub.auth.PAMAuthenticator-3.1.1
[I 2023-03-09 12:54:11.224 JupyterHub app:2840] Using Spawner: batchspawner.batchspawner.SlurmSpawner
[I 2023-03-09 12:54:11.224 JupyterHub app:2840] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-3.1.1
[I 2023-03-09 12:54:11.241 JupyterHub app:1649] Loading cookie_secret from /root/jupyterhub_cookie_secret
[I 2023-03-09 12:54:11.373 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2023-03-09 12:54:11.388 JupyterHub app:1969] Not using allowed_users. Any authenticated user will be allowed.
[I 2023-03-09 12:54:11.431 JupyterHub app:2879] Initialized 0 spawners in 0.004 seconds
[I 2023-03-09 12:54:11.442 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2023-03-09 12:54:11.443 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.seven_days
[I 2023-03-09 12:54:11.444 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.thirty_days
[I 2023-03-09 12:54:11.445 JupyterHub proxy:750] Starting proxy @ https://127.0.0.1:8000/jhub/
12:54:11.988 [ConfigProxy] info: Proxying https://127.0.0.1:8000 to (no default)
12:54:11.994 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[I 2023-03-09 12:54:12.074 JupyterHub app:3130] Hub API listening on http://0.0.0.0:8081/jhub/hub/
[I 2023-03-09 12:54:12.074 JupyterHub app:3132] Private Hub API connect url http://((my domain name)):8081/jhub/hub/
12:54:12.075 [ConfigProxy] info: 200 GET /api/routes
12:54:12.079 [ConfigProxy] info: 200 GET /api/routes
[I 2023-03-09 12:54:12.079 JupyterHub proxy:477] Adding route for Hub: /jhub/ => http://((my domain name)):8081
12:54:12.083 [ConfigProxy] info: Adding route /jhub -> http://((my domain name)):8081
12:54:12.085 [ConfigProxy] info: Route added /jhub -> http://((my domain name)):8081
12:54:12.086 [ConfigProxy] info: 201 POST /api/routes/jhub
[I 2023-03-09 12:54:12.086 JupyterHub app:3197] JupyterHub is now running at https://127.0.0.1:8000/jhub/

When I try to access ((my domain name))/jhub/ in my browser, nothing appears in the log. I only receive the same message as before in my browser that says “proxy error: the proxy server received an invalid response…”.

I guess I’m confused on what needs to be allowed in the firewall. I am accepting any packet that is on the “localhost” interface, and have these rules:

me@server: sudo iptables -S INPUT
-P INPUT ACCEPT
-A INPUT -i eth1 -p tcp -m tcp --dport 8081 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 8081 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 8081 -j ACCEPT
-A INPUT -p udp -m udp --dport 8081 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8081 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 6547 -j ACCEPT
-A INPUT -p udp -m udp --dport 3052 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3052 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A INPUT -i ib0 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited

I do have port 8081 set to accept from the eth0 interface, which maybe would explain why setting the hub to that location would at least show a jupyterhub interface in my browser (despite it not being correctly set up).

Apologies if this is terribly naive! I’m just an engineering grad student who got stuck managing this cluster. My IT skills aren’t flawless. :slight_smile:

7

Ah! I think I know the issue - you’ve told JupyterHub’s proxy to use SSL certificates from letsencrypt. That only makes sense if the proxy is the public-facing service, accessed over the public internet via domain name, but it isn’t, your reverse proxy is. That’s why you are getting:

The proxy server received an invalid response from an upstream server.

Because you are running the proxy with https, but making an http (no s) request to it.

The fix should be to remove:

c.JupyterHub.ssl_cert='/etc/letsencrypt/live/((my server URL))/fullchain.pem'
c.JupyterHub.ssl_key='/etc/letsencrypt/live/((my server URL))/privkey.pem'

from your config, because all of JupyterHub is running on purely local http, and then point your reverse proxy to http://127.0.0.1:8000/jhub

1 Like
8

Fantastic, thank you, this did the trick! Now I can get the login prompt, etc. This makes sense. I did additionally have to remove /jhub/ from my c.JupyterHub.hub_bind_url and SlurmSpawner.hub_connect_url variables, but now can finally make connections from worker nodes to the head node! Hooray!

You are awesome @minrk! Life saver!

Now, I see that connections get made:

[I 2023-03-10 07:51:01.583 JupyterHub batchspawner:463] Notebook server job 24847 started at n011:50930
[I 2023-03-10 07:51:05.445 JupyterHub log:186] 200 GET /jhub/hub/api (@172.16.33.11) 1.47ms
[I 2023-03-10 07:51:05.685 JupyterHub log:186] 200 POST /jhub/hub/api/users/ridley/activity (ridley@172.16.33.11) 79.32ms
[I 2023-03-10 07:51:06.447 JupyterHub base:972] User ridley took 9.379 seconds to start
[I 2023-03-10 07:51:06.447 JupyterHub proxy:330] Adding user ridley to proxy /jhub/user/ridley/ => http://n011:50930
07:51:06.452 [ConfigProxy] info: Adding route /jhub/user/ridley -> http://n011:50930
07:51:06.452 [ConfigProxy] info: Route added /jhub/user/ridley -> http://n011:50930
07:51:06.454 [ConfigProxy] info: 201 POST /api/routes/jhub/user/ridley

Unfortunately, the job seems to hang after starting. The output from the batch spawner is empty, like it’s running but one direction of communication isn’t working. In the browser, it stays at “your server is starting up.”

Now this is down to something I can figure out, thank you again.

UPDATE: it seems that refreshing the page took me to a running jupyterhub instance on the remote node. Hooray! Mission accomplished!

1 Like