Can't connect my jhubzero via my dehydrated nginx plugin - error 404

JupyterHub Zero to JupyterHub on Kubernetes
,
1

I welcome everyone! An error of this nature. All services and servers have the plugin nginx of dehydrated. It generates automatic ssl certificates every 3 months for any subdomain, port, etc. This is very convenient, but for some reason, redirecting jhub does not work. The main page is loaded, after I carry out authorization and after creating the server I get an nginx 404 error. I have multi-master set up.
I tried to use a proxy inside the config, just set it to true, but it doesn’t help.

Settings nginx config:
server {
listen 443 ssl http2;
include /etc/nginx/snippets/ssl_dehydrated;
location /hub {
access_log /var/log/nginx/jupyterhub_access.log combined;
error_log /var/log/nginx/jupyterhub_error.log error;
– proxy_pass https://10.0.70.91:30495; - config jhub, proxy true.
proxy_pass http://10.0.70.92:31880; - ssl proxy off.
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
}
}

2

To understand what it looks like. It redirects first to domain.com/hub
log in and after authorization, I create a server, it jumps to domain.com/user/nickname - error 404 nginx :frowning:
How to make it stay on /hub?
I think the problem is this.

3

The default Z2JH configuration assumes you’re running JupyterHub at the server root. It sounds like you want to set a base-url?
https://zero-to-jupyterhub.readthedocs.io/en/latest/resources/reference.html#schema-hub-baseurl

If you need more help please could you:

  • Show us your full configuration with secrets redacted
  • Tell us about your K8s setup including proxies and any customisations
  • Turn on debug logging and show us your logs for the hub and Nginx
4

Screenshot from 2022-04-05 13-20-31

It did not help, moreover, in the local one it also fell off by IP, and in some places it duplicates /hub/hub
yes, I have no way to put it in the root of the server, I can only redirect to some subdomain or port

5

My config jhub:
– proxy:
– https:
– enabled: true
– hosts:
– - domain.com/hub
– letsencrypt:
– contactEmail: email@my
imagePullSecrets:

  • name: regcred
    hub:
    – baseUrl: /hub
    config:
    JupyterHub:
    authenticator_class: ldapauthenticator.LDAPAuthenticator
    LDAPAuthenticator:
    bind_dn_template:
    - uid={username},cn=users,cn=accounts,dc=ipa,dc=domain,dc=com
    server_address: ipa.domain.com
    user_attribute: sAMAccountName
    user_search_base: cn=users,cn=accounts,dc=ipa,dc=domain,dc=com

    extraPodSpec:
    dnsConfig:
    options:
    - name: ndots
    value: “1”

singleuser:

extraPodConfig:
dnsConfig:
options:
- name: ndots
value: “1”

profileList:

- display_name: "default"
  description: "1 GPU, 6 cores and 32GB RAM. The <code>nvaitc/ai-lab</code> image provides TensorFlow, PyTorch and various data science packages, VS Code and a virtual desktop. nn_utils and ballfish included"
  kubespawner_override:
      image: domain.com:5555/default/jhub-nn_utils:latest
      extra_resource_limits:
        nvidia.com/gpu: "1"
- display_name: "Iii - updated torch, no tf available"
  # <- default one"
  description: "1 GPU, 6 cores and 32GB RAM. The <code>nvaitc/ai-lab</code> image provides TensorFlow, PyTorch and various data science packages, VS Code and a virtual desktop. nn_utils and ballfish included"
  kubespawner_override:
      image: domain.com:5555/default/jhub-nn_utils:upd
      extra_resource_limits:
        nvidia.com/gpu: "1"

storage:
extraVolumes:
- name: shm-volume
emptyDir:
medium: Memory
- name: docker-socket
hostPath:
path: /var/run/docker.sock
- name: tmp-shared
hostPath:
path: /tmp/shared
extraVolumeMounts:
- name: shm-volume
mountPath: /dev/shm
- mountPath: /var/run/docker.sock
name: docker-socket
- mountPath: /tmp/shared
name: tmp-shared
memory:
limit: 32G
guarantee: 16G
cpu:
limit: 8
guarantee: 2
networkPolicy:
enabled: true

storage:
extraVolumes:
- name: shm-volume
emptyDir:
medium: Memory
- name: docker-socket
hostPath:
path: /var/run/docker.sock
- name: tmp-shared
hostPath:
path: /tmp/shared
extraVolumeMounts:
- name: shm-volume
mountPath: /dev/shm
- mountPath: /var/run/docker.sock
name: docker-socket
- mountPath: /tmp/shared
name: tmp-shared
memory:
limit: 32G
guarantee: 16G
cpu:
limit: 8
guarantee: 2
networkPolicy:
enabled: true

6

kubernetes is configured through several masters, they all duplicate information through nfs and work inside the network through different IP addresses. I set one of these addresses in nginx and the jhub port that he gave me. Inside the kubernetes itself, the proxy is not configured.

My nginx logs:
administrator@domain:/etc/nginx/sites-enabled$ sudo less /var/log/nginx/jupyterhub_access.log
10.0.70.2 - - [05/Apr/2022:10:41:45 +0000] “GET /hub/ HTTP/2.0” 302 0 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36”
10.0.70.2 - - [05/Apr/2022:10:41:45 +0000] “GET /hub/spawn HTTP/2.0” 200 3163 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36”
10.0.70.2 - - [05/Apr/2022:10:41:49 +0000] “POST /hub/spawn HTTP/2.0” 302 0 “https://domain.com/hub/spawn” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36”
10.0.70.2 - - [05/Apr/2022:10:41:49 +0000] “GET /hub/spawn-pending/admin HTTP/2.0” 200 2944 “https://domain.com/hub/spawn” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36”
10.0.70.2 - - [05/Apr/2022:10:41:56 +0000] “GET /hub/api/users/admin/server/progress HTTP/2.0” 200 9127 “https://domain.com/hub/spawn-pending/admin” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36”
10.0.70.2 - - [05/Apr/2022:10:41:56 +0000] “GET /hub/spawn-pending/admin HTTP/2.0” 302 0 “https://domain.com/hub/spawn-pending/admin” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36”

error cleaned, while there are no logs

7

I do not know if it is necessary to raise the topic again, after a long lull. But I still haven’t solved this problem. There are no error logs.

8

The screenshot in Can't connect my jhubzero via my dehydrated nginx plugin - error 404 - #4 by am0ral93 suggests your CSS files aren’t being loaded. I’d expect there to be some 404 errors in your nginx access logs. Are those the full logs? If so do you see any errors in your browser console?