Recommended use of X-Forwarded-Proto with nginx

JupyterHub
#1

The jupyterhub docs demonstrate how to configure nginx as a reverse proxy. One of the configuration statements is proxy_set_header X-Forwarded-Proto $scheme; and I’m using it in a local nginx/hub configuration with jupyterhub 1.1. Unfortunately I’ve found that use of X-Forwarded-Proto causes unexpected behavior when using jupyter-rsession-proxy and rstudio. This may be a bug in one of those projects, but when I eliminated the X-Forwarded-Proto statement from my nginx config, the unexpected behavior went away and I didn’t spot any regressions when using the hub.

I’ve read about the header, and it seems logical why one may need to include it, but does it happen to still be necessary? My hub doesn’t seem worse off for not including it. Here are the proxy statements I am using:

proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# https://github.com/jupyterhub/jupyterhub/issues/2284#issuecomment-527196161
proxy_set_header X-Scheme $scheme;

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
#2

I don’t know if it’s always required with JupyterHub, though I consider it a bug if an application fails when it’s set (and I’ve also come across several buggy applications :smiley: ).

From experience with other applications it’s often required if you have multiple proxies (e.g. HTTPS load balancer :arrow_right: HTTP web proxy :arrow_right: Application).

Be aware there are some potential security issues if things are badly configured. I can’t remember the details, but it’s something along the lines of an attacker intercepting a connection and forcing a victim to use http instead of https, whilst spoofing the X-Forwarded-Proto or other headers to make the application (JupyterHub) think the client is using https. I can’t remember if Nginx automatically sets/unsets headers.

To add to the confusion the Forwarded header is now the official standard for the X-Forwarded-* headers
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded
I don’t know if this is also having an influence on the behaviour.

If you figure any of this out it’d be good to know :grinning_face_with_smiling_eyes:

1 Like
#3

Okay, thanks @manics! Just following some breadcrumbs, it looks like there is a related hub config option, trusted_downstream_ips that I’m supposed to set since I’ve got an ssl terminating proxy and I’m using x-forwarded-for. It is passed to tornado’s httpserver, but I only see trusted_downstream as a supported argument. Is that a bug?

#4

Is that a bug?

It’s an unfortunate mismatch that JupyterHub calls it ‘trusted_downstream_ips’ but it correctly passes the value as ‘trusted_downstream’ to tornado.

#5

Yikes, obvious oversight on my part! Sorry about that.