Gateway error with haproxy in front of Kubernetes

JupyterHub Zero to JupyterHub on Kubernetes
1

I’m running Jupyterhub on top of a bare metal Kubernetes cluster, which is accessible only in our internal network. To make it accessible from the Internet, I’m running an Apache reverse proxy that connects to the proxy-public service. The relevant Apache configuration follows.

<VirtualHost 192.168.0.2:9092>
    ProxyPreserveHost On
    ProxyRequests Off
    ServerName example.com

    <Location "/">
      ProxyPass "http://192.168.0.4:32592/"
      ProxyPassReverse "http://192.168.0.4:32592/"
      RequestHeader set "X-Scheme" expr=%{REQUEST_SCHEME}

    RewriteEngine on
    RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
    RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
    RewriteRule .* ws://192.168.0.4:32592%{REQUEST_URI} [P]
    </Location>


    SSLEngine on
    SSLCertificateFile /etc/apache2/cert/example.crt
    SSLCertificateKeyFile /etc/apache2/cert/example.key
    SSLCertificateChainFile /etc/apache2/cert/example-ca.crt
</VirtualHost>

The problem with this setup is that the machine that hosts the Apache reverse proxy becomes a single point-of-failure and it did cause downtimes for a few times already. To avoid this, I deployed two haproxy servers (with different keepalived priorities) that listen on a floating IP. The relevant haproxy settings follows.

listen jupyterhub-gpu
        bind 192.168.0.100:9092 ssl crt /etc/apache2/cert/example.pem
        http-request set-header X-Scheme req.scheme
        http-request set-header X-Client-IP %[src]
        option forwardfor
        option httpclose
        option http-pretend-keepalive
        option redispatch
        option contstats
        retries 3
        timeout client          25s
        timeout connect          5s
        timeout server          25s
        # timeout tunnel available in ALOHA 5.5 or HAProxy 1.5-dev10 and higher
        timeout tunnel        3600s
        timeout http-keep-alive  1s
        timeout http-request    15s
        timeout queue           30s
        # server apache 192.168.0.2:9092 ssl check
        server srv1 192.168.0.4:32592 check

The problem is that I encounter a gateway error after some time due to the backend server timing out:

jupyterhub-gpu~ jupyterhub-gpu/srv1 0/0/1/-1/25002 504 195 - - sH-- 4/4/3/3/0 0/0 "GET /favicon.ico HTTP/1.1"
jupyterhub-gpu~ jupyterhub-gpu/srv1 0/0/0 **/-1/** 25002 504 195 - - sH-- 3/3/1/0/0 0/0 "GET /user/calis/notebooks/notebook.ipynb HTTP/1.1"

However, I don’t encounter that problem if I set the apache server as the backend.

What should be the correct settings for haproxy?

2

You’ve got different backend IPs in your Apache 192.168.0.4 and Haproxy 192.168.212.2 configs. Are those both correct?

3

Sorry those were just typos. I’ve edited the post. They should point to the same backend (192.168.0.4).

4

I don’t know exactly what I did but I somehow fixed it. It seems the backend connection is still running when haproxy is reloaded so perhaps stopping then starting the haproxy service might be a good idea for anyone else who encounters this problem.

Here is my currently running config which is closer to the example configuration here:

listen jupyterhub-gpu
        bind 192.168.0.100:9092 ssl crt /etc/apache2/cert/example.pem
        option forwardfor
        option http-server-close
        option redispatch
        option contstats
        retries 3
        timeout client          25s
        timeout connect          5s
        timeout server          25s
        # timeout tunnel available in ALOHA 5.5 or HAProxy 1.5-dev10 and higher
        timeout tunnel        3600s
        timeout http-keep-alive  5s
        timeout http-request    15s
        timeout queue           30s
        timeout tarpit          60s
        default-server inter 3s rise 2 fall 3
        server srv1 192.168.0.4:32592 cookie aim-c1 check
5

I need to add two more lines to avoid Authorization form must be sent from authorization page error:

listen jupyterhub-gpu
        bind 192.168.0.100:9092 ssl crt /etc/apache2/cert/example.pem
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        http-request add-header X-Scheme https if { ssl_fc }
        option forwardfor
        option http-server-close
        option redispatch
        option contstats
        retries 3
        timeout client          25s
        timeout connect          5s
        timeout server          25s
        # timeout tunnel available in ALOHA 5.5 or HAProxy 1.5-dev10 and higher
        timeout tunnel        3600s
        timeout http-keep-alive  5s
        timeout http-request    15s
        timeout queue           30s
        timeout tarpit          60s
        default-server inter 3s rise 2 fall 3
        server srv1 192.168.0.4:32592 cookie aim-c1 check