I’m unable to run admin commands from the hub control panel via the web interface. I’m running JupyterHub 1.1.0 on Ubuntu 18.04 LTS. Sessions are spawned using DockerSpawner via SystemUserSpawner. I use nginx reverse proxy via an intermediary server. All notebook functions work fine. The only issue is with my admin functions. All admins are affected included a new dummy admin that I created. I’ve read some on 403 errors, but they all seem to be dealing with specific users not being able to access their notebooks due to the JupyterHub database being reset. I’ve tried removing my containers just to eliminate that as a possible reason, but the issue still persists. I’m assuming the issue has to do with the reverse proxy, but I’m not sure. Any advice on next steps? Thanks!
Hi! Please could you show us your JupyterHub logs corresponding to the faiied requests? It may help to run jupyterhub with the --debug option to show more info in the logs.
When I “Stop All” I get this.
[W 2020-06-12 13:24:27.569 JupyterHub base:60] Blocking Cross Origin API request. Referer: https://jomi.ees.vanderbilt.edu/nodi/jupyter/hub/admin, Host: localhost:8723/nodi/jupyter/hub/
[W 2020-06-12 13:24:27.570 JupyterHub log:174] 403 DELETE /nodi/jupyter/hub/api/users/chris/server (@129.59.62.15) 13.80ms
Also, another symtom that I think may be related… At the page with “Your server is starting up” at login, it just hangs. I have to refresh it to redirect to the spawned notebook.
This could be the problem, it looks like the Host header may be dropped by one of the proxies. Are you able to get any logs from Nginx and also the intermediate proxy? Alternatively are you able to test without one of those proxies, as that would help to narrow down the problem.
I think you’re right that the issue is with the proxies. I didn’t configured them. One of the other admins is going to look into the nginx configs. I’m unsure when they’ll have a chance to get to it, but I’ll update when I hear more. Thanks for the help!
Okay. The other admin doesn’t have the time to delve into this right now, but I’m going to try to investigate it. I’m pretty unfamiliar with nginx, but I was able to look through some of the logs and I think I found where the error is occurring. For context, here’s my config…
Client -> jomi (nginx reverse proxy) -> nodi (nginx reverse proxy) -> nodi (JupyterHub)
jomi and nodi are the server names.
On nodi, the nginx error log shows this after attempting to perform an admin function (in this case “Stop Servers”).
2020/06/13 14:32:45 [error] 4998#4998: *38832 connect() failed (111: Connection refused) while connecting to upstream, client: 129.59.62.15, server: nodi.ees.vanderbilt.edu, request: "GET /nodi/jupyter/user/chris/api/terminals?1592076765216 HTTP/1.1", upstream: "http://[::1]:8723/nodi/jupyter/user/chris/api/terminals?1592076765216", host: "nodi.ees.vanderbilt.edu", referrer: "https://jomi.ees.vanderbilt.edu/nodi/jupyter/user/chris/lab"
2020/06/13 14:32:45 [warn] 4998#4998: *38832 upstream server temporarily disabled while connecting to upstream, client: 129.59.62.15, server: nodi.ees.vanderbilt.edu, request: "GET /nodi/jupyter/user/chris/api/terminals?1592076765216 HTTP/1.1", upstream: "http://[::1]:8723/nodi/jupyter/user/chris/api/terminals?1592076765216", host: "nodi.ees.vanderbilt.edu", referrer: "https://jomi.ees.vanderbilt.edu/nodi/jupyter/user/chris/lab"
It looks like the problem is confined to nodi, I think? Any idea where to go from here?
Bingo! This was the problem. One of the servers was missing
proxy_set_header Host $host;
Both the spawner status bar and admin commands now work properly! Thank you for your help @manics!
Thanks, this was the solution!
NGINX config:
location / {
proxy_pass https://localhost:8443;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_redirect default;
#proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-User $http_authorization;
proxy_max_temp_file_size 0;
#this is the maximum upload size
client_max_body_size 20m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}
traefik.toml:
defaultEntryPoints = [ "http", "https",]
logLevel = "INFO"
[accessLog]
format = "json"
[respondingTimeouts]
idleTimeout = "10m0s"
[wss]
protocol = "http"
[api]
dashboard = true
entrypoint = "auth_api"
[file]
directory = "rules"
watch = true
[accessLog.filters]
statusCodes = [ "500-999",]
[entryPoints.http]
address = ":8081"
[entryPoints.https]
address = ":8443"
[entryPoints.auth_api]
address = "127.0.0.1:8099"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https.tls]
minVersion = "VersionTLS12"
[[entryPoints.https.tls.certificates]]
certFile = "/etc/ssl/certs/cert.pem"
keyFile = "/etc/ssl/private/private.key"
[entryPoints.auth_api.whiteList]
sourceRange = [ "127.0.0.1",]
[accessLog.fields.headers.names]
Authorization = "redact"
Cookie = "redact"
Set-Cookie = "redact"
X-Xsrftoken = "redact"
[entryPoints.auth_api.auth.basic]
users = [ "api_admin:$apr1$GwTIFX9C$c3fJGIAHnN558mXf0F4p81",]The JH need a websocket to exchange the commands and response. You must config the ws or wss agent in ngnix.
Fred via Jupyter Community Forum <jupyter@discoursemail.com>于2020年10月17日 周六19:28写道：
Here is a reference:
https://aptro.github.io/server/architecture/2016/06/21/Jupyter-Notebook-Nginx-Setup.html
Zheng Wang <longwind.wang@gmail.com>于2020年10月17日 周六19:37写道：
thanks, but I already got it working with the above config changes