Hi there, I noticed multiple issues after upgrading to JupyterHub 2.1.1 from 2.0.0.
- I stopped seeing spawn progress when starting the singleuser server. Logs:
[I 2022-03-04 20:57:47.436 JupyterHub log:189] 302 GET /hub/spawn -> /hub/spawn-pending/konstantin.taletskiy@labshare.org (konstantin.taletskiy@labshare.org@192.168.15.108) 1006.02ms
[I 2022-03-04 20:57:47.524 JupyterHub pages:405] konstantin.taletskiy@labshare.org is pending spawn
[I 2022-03-04 20:57:47.528 JupyterHub log:189] 200 GET /hub/spawn-pending/konstantin.taletskiy@labshare.org (konstantin.taletskiy@labshare.org@192.168.15.108) 9.75ms
[W 2022-03-04 20:57:47.746 JupyterHub base:94] Blocking Cross Origin API request. Referer: https://<redacted_url>/hub/spawn-pending/konstantin.taletskiy@labshare.org, Host: <redacted_url>, Host URL: http://<redacted_url>/hub/
[W 2022-03-04 20:57:47.747 JupyterHub scopes:501] Not authorizing access to /hub/api/users/konstantin.taletskiy@labshare.org/server/progress. Requires any of [read:servers], not derived from scopes []
[W 2022-03-04 20:57:47.747 JupyterHub web:1787] 403 GET /hub/api/users/konstantin.taletskiy@labshare.org/server/progress (192.168.15.108): Action is not authorized with current scopes; requires any of [read:servers]
[W 2022-03-04 20:57:47.747 JupyterHub log:189] 403 GET /hub/api/users/konstantin.taletskiy@labshare.org/server/progress (@192.168.15.108) 4.93ms
- I can’t stop the singleuser server anymore. Logs:
[W 2022-03-04 23:07:26.200 JupyterHub base:94] Blocking Cross Origin API request. Referer: https://<redacted_url>/hub/home, Host: <redacted_url>, Host URL: http://<redacted_url>/hub/
[W 2022-03-04 23:07:26.200 JupyterHub scopes:501] Not authorizing access to /hub/api/users/konstantin.taletskiy%40labshare.org/server. Requires any of [delete:servers], not derived from scopes []
[W 2022-03-04 23:07:26.200 JupyterHub web:1787] 403 DELETE /hub/api/users/konstantin.taletskiy%40labshare.org/server (192.168.3.68): Action is not authorized with current scopes; requires any of [delete:servers]
[W 2022-03-04 23:07:26.201 JupyterHub log:189] 403 DELETE /hub/api/users/konstantin.taletskiy%40labshare.org/server (@192.168.3.68) 8.96ms
- Admin page is empty
My deployment is on K8s with Helm, using heavily modified zero-to-jupyterhub-k8s chart. You could see the chart here. I am using nginx ingress, which is not described in JupyterHub reverse proxy docs, but is used in zero-to-jupyterhub-k8s, for example. I found similar issues for users running Apache and nginx reverse proxies:
- Apache ReverseProxy not working with Hub 2.0.1 and later. · Issue #3780 · jupyterhub/jupyterhub · GitHub
- Empty Admin page when the hub is behind nginx proxy · Issue #3737 · jupyterhub/jupyterhub · GitHub
The relevant parts of my config are provided below
- JupyterHub Docker file
FROM frolvlad/alpine-miniconda3:python3.7@sha256:bf5c88b66776e27b6745a7489104263d4ec05748eb447e7a292a0c4b5ed6c074
LABEL maintainer="Labshare <konstantin.taletskiy@labshare.org>"
# Track semantic versioning
COPY VERSION /
# Copy cull-idle script
COPY cull-idle-servers.py /srv/jupyterhub/config/cull-idle-servers.py
RUN conda install --yes -c conda-forge \
git \
sqlalchemy \
tornado \
jinja2 \
traitlets \
requests \
pycurl \
nodejs=12 \
configurable-http-proxy \
escapism=1.0.1 \
jupyterhub=2.1.1 \
python-kubernetes=22.6.0 \
jupyterhub-kubespawner=2.0.1 && \
pip install \
psycopg2-binary==2.9.1 \
oauthenticator==14.2.0
RUN mkdir -p /srv/jupyterhub/
COPY config-wrapper.py /srv/jupyterhub/config-wrapper.py
WORKDIR /srv/jupyterhub/
EXPOSE 8000
LABEL org.jupyter.service="jupyterhub"
CMD ["python", "/srv/jupyterhub/config-wrapper.py"]
- Nginx ingress resource template from the Helm chart:
{{- if .Values.hub.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
name: {{ include "jupyterhub.ingress.fullname" . }}
spec:
rules:
- host: {{ .Values.hub.ingress.hostName }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: jupyterhub
port:
number: 80
{{- end }}
- nginx configmap
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
name: nginx-configuration
namespace: ingress-nginx
data:
location-snippet: |
more_set_input_headers "X-Forwarded-Proto: https";
proxy-body-size: "0"
use-forwarded-headers: "true"
use-proxy-protocol: "false"
- JupyterHub Portion of the nginx config generated by nginx ingress (
/etc/nginx/nginx.conffile in thenginx-ingress-controller-xxxx-zzzzpod)
server {
server_name <redacted_url> ;
listen 80 ;
listen 443 ssl http2 ;
set $proxy_upstream_name "-";
ssl_certificate_by_lua_block {
certificate.call()
}
location / {
set $namespace "<jupyterhub_namespace>";
set $ingress_name "<helm_deployment_full_name>";
set $service_name "jupyterhub";
set $service_port "80";
set $location_path "/";
rewrite_by_lua_block {
lua_ingress.rewrite({
force_ssl_redirect = false,
ssl_redirect = true,
force_no_ssl_redirect = false,
use_port_in_redirects = false,
})
balancer.rewrite()
plugins.run()
}
# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
# other authentication method such as basic auth or external auth useless - all requests will be allowed.
#access_by_lua_block {
#}
header_filter_by_lua_block {
lua_ingress.header()
plugins.run()
}
body_filter_by_lua_block {
}
log_by_lua_block {
balancer.log()
monitor.call()
plugins.run()
}
port_in_redirect off;
set $balancer_ewma_score -1;
set $proxy_upstream_name "<k8s_namespace-service_name-port>;
set $proxy_host $proxy_upstream_name;
set $pass_access_scheme $scheme;
set $pass_server_port $server_port;
set $best_http_host $http_host;
set $pass_port $pass_server_port;
set $proxy_alternative_upstream_name "";
client_max_body_size 0;
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Request-ID $req_id;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering off;
proxy_buffer_size 4k;
proxy_buffers 4 4k;
proxy_max_temp_file_size 1024m;
proxy_request_buffering on;
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0;
proxy_next_upstream_tries 3;
# Custom code snippet configured in the configuration configmap
more_set_input_headers "X-Forwarded-Proto: https";
proxy_pass http://upstream_balancer;
proxy_redirect off;
}
}
The only change that I did from the working configuration was to change jupyterhub=2.0.0 to jupyterhub=2.1.1 in the Docker file and upgrade the Helm Chart. Could anyone suggest the possible solutions to fix the problem?
So far, I looked into adding config snippet to my JupyterHub ingress definitions to make it look more like the one provided in the documentation:
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
That didn’t help. Please help.
