Trouble reverse proxying to JupyterHub on a subpath

I am trying to serve a dockerized JupyterHub behind an nginx reverse proxy on a subpath (not a subdomain).

  • The reverse proxy and JupyterHub are running on the same machine.
  • The reverse proxy is hosting a basic html page that links to various subpathed web applications running in docker containers on the same machine as the proxy.
  • The reverse proxy terminates TLS/SSL connection from client, and the backend is all unencrypted http and ws.
  • JupyterHub works perfectly fine if I connect to it directly on the exposed docker port (bypassing the proxy), but not through the proxy.
  • Have tried cobbling together various nginx configuration settings from the JupyterHub docs, Stackoverflow, Reddit, and this forum.

Since JupyterHub is working fine when connecting directly to the exposed docker port, I am convinced that the issue is my nginx configuration. If someone could point me in the right direction to get it working, please let me know!

JupyterHub Version

$ docker exec jupyterhub jupyterhub --version
2.1.1

OS Version

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME=RHEL
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"

Docker Version

$ docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:41 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:44:05 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

JupyterHub docker-compose.yml

$ cat docker-compose.yml
version: "3.8"

services:
  jupyterhub:
    build: "./build"
    container_name: "jupyterhub"
    image: "jupyterhub_notebook"
      #    restart: "unless-stopped"
    environment:
      #      - http_proxy="http://[redacted]:8080"
      #      - https_proxy="http://[redacted]:8080"
      #      - HTTP_PROXY="http://[redacted]:8080"
      - HTTPS_PROXY="http://[redacted]:8080"
    networks:
      - "reverse_proxy"
    ports:
      - "15080:8000"
    volumes:
      - "jupyterhub_home:/home" # persist users
      - "jupyterhub_etc:/etc" # persist jupyterhub_config.py, passwd, shadow
      - "jupyterhub_srv_jupyterhub:/srv/jupyterhub" # persist cookie secrets
      - "jupyterhub_var_spool_mail:/var/spool/mail" # persist user mail

networks:
  reverse_proxy:
    name: "reverse_proxy"

volumes:
  jupyterhub_etc:
    name: "jupyterhub_etc"
    driver: "local"
    driver_opts:
      device: "./jupyterhub_etc"
      type: "none"
      o: "bind"

  jupyterhub_home:
    name: "jupyterhub_home"
    driver: "local"
    driver_opts:
      device: "./jupyterhub_home"
      type: "none"
      o: "bind"

  jupyterhub_srv_jupyterhub:
    name: "jupyterhub_srv_jupyterhub"
    driver: "local"
    driver_opts:
      device: "./jupyterhub_srv_jupyterhub"
      type: "none"
      o: "bind"

  jupyterhub_var_spool_mail:
    name: "jupyterhub_var_spool_mail"
    driver: "local"
    driver_opts:
      device: "./jupyterhub_var_spool_mail"
      type: "none"
      o: "bind"

JupyterHub Dockerfile

$ cat build/Dockerfile
FROM jupyterhub/jupyterhub:latest

RUN yes | unminimize

RUN apt-get update
RUN apt-get install --assume-yes dialog
RUN apt-get install --assume-yes manpages
RUN apt-get install --assume-yes manpages-posix
RUN apt-get install --assume-yes man-db
RUN apt-get install --assume-yes apt-utils
RUN apt-get install --assume-yes --reinstall less curl
RUN apt-get install --assume-yes bash-completion man-db neovim git curl python3 python3-venv python-is-python3
RUN apt-get install --assume-yes bash-completion git
RUN apt-get install --assume-yes bash-completion neovim
RUN apt-get install --assume-yes bash-completion python3 python3-venv python-is-python3
RUN apt-get install --assume-yes sudo
RUN apt-get upgrade --assume-yes
RUN apt-get autoremove --assume-yes

RUN pip install --upgrade setuptools pip wheel
RUN pip install --upgrade jupyterlab

ENV no_proxy jupyterhub
ENV no_proxy twopyter

# Ensure env vars are sourced in Jupyter terminals
RUN echo '\n####\n# USER ADDED\n####\n' >> /etc/skel/.bashrc
RUN echo '\n#Help pip use postal proxy\nexport HTTPS_PROXY=http://[redacted]:8080\n' >> /etc/skel/.bashrc
RUN echo '\n# Add pip bin directory to user path\nexport PATH=$HOME/.local/bin:$PATH\n' >> /etc/skel/.bashrc


Nginx Version (Reverse Proxy)

$ docker exec reverse_proxy nginx -v
nginx version: nginx/1.21.6

Nginx Rules (Reverse Proxy)

$ docker exec reverse_proxy cat /etc/nginx/conf.d/default.conf

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    listen 443 ssl;
    server_name [redacted];

    ssl_certificate     /etc/nginx/ssl/certs/[redacted].cert.pem;
    ssl_certificate_key /etc/nginx/ssl/certs/[redacted].key.pem;
    ssl_protocols       TLSv1.3;

    proxy_set_header    Host                $host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-Proto   $scheme;
    proxy_set_header    X-Fowarded-For      $proxy_add_x_forwarded_for;

    location / {
        root /usr/share/nginx/html;
        index index.html index.htm;
        try_files $uri$args $uri$args/ /index.html;
    }

    location /twopyter/ {
#        rewrite ^/hub/(.*) $scheme://$server_name/$1/$2 permanent;
#        rewrite ^/twopyter/ /stupid break;
#        rewrite (.*) $1 break;
#        rewrite ^/twopyter/(.*)$ /$1 break;
        rewrite /twopyter/(.*) ws://host.docker.internal:15080/twopyter/$1 break;
        rewrite /twopyter/(.*) http://host.docker.internal:15080/twopyter/$1 break;
        proxy_pass              http://host.docker.internal:15080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # websocket headers
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

JupyterHub Configuration

  • Only shows uncommented vars
## The public facing URL of the whole JupyterHub application.
#
#          This is the address on which the proxy will bind.
#          Sets protocol, ip, base_url
#  Default: 'http://:8000'
c.JupyterHub.bind_url = 'http://:8000/twopyter/'

## The class to use for spawning single-user servers.
#
#          Should be a subclass of :class:`jupyterhub.spawner.Spawner`.
#
#          .. versionchanged:: 1.0
#              spawners may be registered via entry points,
#              e.g. `c.JupyterHub.spawner_class = 'localprocess'`
#
#  Currently installed:
#    - default: jupyterhub.spawner.LocalProcessSpawner
#    - localprocess: jupyterhub.spawner.LocalProcessSpawner
#    - simple: jupyterhub.spawner.SimpleLocalProcessSpawner
#  Default: 'jupyterhub.spawner.LocalProcessSpawner'
c.JupyterHub.spawner_class = 'jupyterhub.spawner.LocalProcessSpawner'

## Path to the notebook directory for the single-user server.
#
#  The user sees a file listing of this directory when the notebook interface is
#  started. The current interface does not easily allow browsing beyond the
#  subdirectories in this directory's tree.
#
#  `~` will be expanded to the home directory of the user, and {username} will be
#  replaced with the name of the user.
#
#  Note that this does *not* prevent users from accessing files outside of this
#  path! They can do so with many other means.
#  Default: ''
c.Spawner.notebook_dir = '~/'

Try adding c.JupyterHub.base_url
https://jupyterhub.readthedocs.io/en/stable/getting-started/networking-basics.html#adjusting-the-hub-s-url

@manics Thanks. I have tried that, and it didn’t change anything. As an aside, that variable is deprecated.

I will try it again in a little while.

base_url isn’t deprecated, can you point me to where you found that out?

Can you also share your Nginx logs (access_log and error_log)?

I am going by the comments that are written above the commented variable c.JupyterHub.base_url:


## The base URL of the entire application.
#
#          Add this to the beginning of all JupyterHub URLs.
#          Use base_url to run JupyterHub within an existing website.
#
#          .. deprecated: 0.9
#              Use JupyterHub.bind_url
#  Default: '/'
c.JupyterHub.base_url = '/twopyter/'

Deprecated since 0.9.

Just tried c.JupyterHub.base_url = '/twopyter/' to no avail.

One interesting thing from the JupyterHub process logs is that it appears to be making a redirect that does not work:

jupyterhub          | [I 2022-02-23 17:04:14.839 JupyterHub log:189] 302 GET /twopyter/ -> /hub/twopyter/ (@[redacted]) 1.50ms

Will get those nginx logs in a bit.

@manics

Nginx stdout (and stderr hopefully?)

$ docker-compose up --build
Creating reverse_proxy ... done
Attaching to reverse_proxy
reverse_proxy    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
reverse_proxy    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
reverse_proxy    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
reverse_proxy    | 10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
reverse_proxy    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
reverse_proxy    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
reverse_proxy    | /docker-entrypoint.sh: Configuration complete; ready for start up
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: using the "epoll" event method
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: nginx/1.21.6
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: built by gcc 10.3.1 20211027 (Alpine 10.3.1_git20211027)
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: OS: Linux 3.10.0-1160.59.1.el7.x86_64
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker processes
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 23
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 24
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 25
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 26
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 27
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 28
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 29
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 30
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 31
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 32
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 33
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 34
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 35
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 36
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 37
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 38
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 39
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 40
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 41
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 42
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 43
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 44
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 45
reverse_proxy    | 2022/02/23 17:20:02 [notice] 1#1: start worker process 46
reverse_proxy    | [redacted] - - [23/Feb/2022:17:20:19 +0000] "GET / HTTP/1.1" 200 1515 "-" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:17:20:19 +0000] "GET /favicon.ico HTTP/1.1" 200 1515 "https://[redacted]/" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:17:20:22 +0000] "GET /twopyter HTTP/1.1" 301 169 "https://[redacted]/" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:17:20:22 +0000] "GET /twopyter/ HTTP/1.1" 302 0 "https://[redacted]/" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:17:20:22 +0000] "GET /hub/twopyter/ HTTP/1.1" 200 1515 "https://[redacted]/" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:17:20:22 +0000] "GET /favicon.ico HTTP/1.1" 200 1515 "[redacted]/hub/twopyter/" "Mozilla/[redacted]" "-"

Apologies, you’re right.

Can you remove the rewrite statements from your Nginx config, as they shouldn’t be necessary. Presumably you (temporarily?) put them in whilst investigating the problem.

Could you also turn on debug logging in JupyterHub, and show the full set of JupyterHub logs so we can see the full set of internal redirects? If you could also clear your browser cache (or use a new private window) and go straight to /twopyter/ that would also help. And finally please include the Nginx logs again with rewrite removed. Thanks!

Presumably you (temporarily?) put [rewrite statements] in whilst investigating the problem

Yes

remove the rewrite statements from your Nginx config

Done.

# relevant section of nginx conf
    location /twopyter/ {
#        rewrite ^/hub/(.*) $scheme://$server_name/$1/$2 permanent;
#        rewrite ^/twopyter/ /stupid break;
#        rewrite (.*) $1 break;
#        rewrite ^/twopyter/(.*)$ /$1 break;
#        rewrite /twopyter/(.*) ws://host.docker.internal:15080/twopyter/$1 break;
#        rewrite /twopyter/(.*) http://host.docker.internal:15080/twopyter/$1 break;
        proxy_pass              http://host.docker.internal:15080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # websocket headers
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
docker exec reverse_proxy nginx -t
docker exec reverse_proxy nginx -s reload

turn on debug logging in JupyterHub…so we can see the full set of internal redirects

Okay, I have edited jupyterhub_config.py and set c.Application.log_level = 'DEBUG'

  • This does not seem to have increased the logging verbosity.

use a new private window

Okay, doing that from now on. Will periodically close and reopen private window to ensure cleared cache.

go straight to /twopyter/

Okay.

JupyterHub Log

Creating jupyterhub ... done
Attaching to jupyterhub
jupyterhub    | [I 2022-02-23 18:09:25.664 JupyterHub app:2766] Running JupyterHub version 2.1.1
jupyterhub    | [I 2022-02-23 18:09:25.664 JupyterHub app:2796] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.1.1
jupyterhub    | [I 2022-02-23 18:09:25.664 JupyterHub app:2796] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-2.1.1
jupyterhub    | [I 2022-02-23 18:09:25.664 JupyterHub app:2796] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.1.1
jupyterhub    | [I 2022-02-23 18:09:25.675 JupyterHub app:1606] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
jupyterhub    | [I 2022-02-23 18:09:25.759 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
jupyterhub    | [I 2022-02-23 18:09:25.771 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
jupyterhub    | [I 2022-02-23 18:09:25.799 JupyterHub app:2835] Initialized 0 spawners in 0.003 seconds
jupyterhub    | [W 2022-02-23 18:09:25.802 JupyterHub proxy:687] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
jupyterhub    | [I 2022-02-23 18:09:25.802 JupyterHub proxy:691] Starting proxy @ http://:8000
jupyterhub    | 18:09:26.509 [ConfigProxy] info: Proxying http://*:8000 to (no default)
jupyterhub    | 18:09:26.513 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
jupyterhub    | 18:09:26.851 [ConfigProxy] info: 200 GET /api/routes
jupyterhub    | [I 2022-02-23 18:09:26.852 JupyterHub app:3084] Hub API listening on http://127.0.0.1:8081/hub/
jupyterhub    | 18:09:26.853 [ConfigProxy] info: 200 GET /api/routes
jupyterhub    | [I 2022-02-23 18:09:26.854 JupyterHub proxy:431] Adding route for Hub: / => http://127.0.0.1:8081
jupyterhub    | 18:09:26.856 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
jupyterhub    | 18:09:26.856 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
jupyterhub    | 18:09:26.857 [ConfigProxy] info: 201 POST /api/routes/
jupyterhub    | [I 2022-02-23 18:09:26.857 JupyterHub app:3150] JupyterHub is now running at http://:8000
jupyterhub    | [I 2022-02-23 18:13:42.992 JupyterHub log:189] 302 GET /twopyter/ -> /hub/twopyter/ (@[redacted]) 1.46ms

Nginx Log

reverse_proxy    | [redacted] - - [23/Feb/2022:18:13:42 +0000] "GET /twopyter HTTP/1.1" 301 169 "-" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:18:13:42 +0000] "GET /twopyter/ HTTP/1.1" 302 0 "-" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:18:13:42 +0000] "GET /hub/twopyter/ HTTP/1.1" 200 1515 "-" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:18:13:43 +0000] "GET /favicon.ico HTTP/1.1" 200 1515 "https://[redacted]/hub/twopyter/" "Mozilla/[redacted]" "-"

Also, my browser points at the link described in the JupyterHub logs. It’s pointing to https://[redacted]/hub/twopyter/. It was doing that before I commented out the rewrite statements, also.

EDIT: Thank you for the help in debugging thus far!

Is it possible your jupyterhub_config.py changes haven’t been picked up?
If I set

c = get_config()
c.Application.log_level = 'DEBUG'
c.JupyterHub.bind_url = 'http://:8000/test/'

My logs are

[I 2022-02-23 18:41:30.848 JupyterHub app:3087] Hub API listening on http://127.0.0.1:8081/test/hub/
18:41:30.848 [ConfigProxy] info: 200 GET /api/routes
[D 2022-02-23 18:41:30.849 JupyterHub proxy:343] Fetching routes to check
[D 2022-02-23 18:41:30.849 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
18:41:30.852 [ConfigProxy] info: 200 GET /api/routes
[D 2022-02-23 18:41:30.852 JupyterHub proxy:346] Checking routes
[I 2022-02-23 18:41:30.852 JupyterHub proxy:431] Adding route for Hub: /test/ => http://127.0.0.1:8081
[D 2022-02-23 18:41:30.852 JupyterHub proxy:821] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/test
18:41:30.855 [ConfigProxy] info: Adding route /test -> http://127.0.0.1:8081
18:41:30.855 [ConfigProxy] info: Route added /test -> http://127.0.0.1:8081
18:41:30.856 [ConfigProxy] info: 201 POST /api/routes/test
[I 2022-02-23 18:41:30.856 JupyterHub app:3154] JupyterHub is now running at http://:8000/test/
[D 2022-02-23 18:41:30.857 JupyterHub app:2762] It took 0.947 seconds for the Hub to start

My prefix /test is visible in the logs

It’s possible, but I don’t think so because I definitely edited the correct file. I composed down and back up again after each change.

I see. I did not do c = get_config(). I added that at the very top of the file; it is not present in the default config file. It does not seem to have changed anything. The verbosity is exactly the same as before.

Trying to prove that I edited the correct file:

$ docker exec jupyterhub cat /etc/jupyterhub/jupyterhub_config.py | head
# hello @manics
c = get_config()
# Configuration file for jupyterhub.

#------------------------------------------------------------------------------
# Application(SingletonConfigurable) configuration
#------------------------------------------------------------------------------
## This is an application.

## The date format used by logging formatters for %(asctime)s

Hmmm. I believe jupyterhub is correctly sourcing this config file because it changes other things successfully.

@manics

I added a line to the last line of my Dockerfile:

CMD ["jupyterhub", "-f", "/etc/jupyterhub/jupyterhub_config.py"]

And now I am seeing the right behavior I expect. For some reason I expected it to automatically look at /etc/jupyterhub for a config file. I think I even read that somewhere. But it doesn’t seem to be the case.

Haven’t tested it properly yet, but:

JupyterHub Config Shenanigans

$ docker exec jupyterhub cat /etc/jupyterhub/jupyterhub_config.py | grep -B 5 --extended-regexp "c.JupyterHub.bind_url = "
## The public facing URL of the whole JupyterHub application.
#
#          This is the address on which the proxy will bind.
#          Sets protocol, ip, base_url
#  Default: 'http://:8000'
c.JupyterHub.bind_url = 'http://:8000/rubber/baby/buggy/bumpers/'

Starting JupyterHub Container

$ dcu
docker-compose up --build
Building jupyterhub
Sending build context to Docker daemon  3.072kB
Step 1/24 : FROM jupyterhub/jupyterhub:latest
 ---> 1cd4fbaf4648
Step 2/24 : RUN yes | unminimize
 ---> Using cache
 ---> b5916a458ea9
Step 3/24 : RUN apt-get update
 ---> Using cache
 ---> acd912312b54
Step 4/24 : RUN apt-get install --assume-yes dialog
 ---> Using cache
 ---> 41c85a1ac8ad
Step 5/24 : RUN apt-get install --assume-yes manpages
 ---> Using cache
 ---> 3ea68e90bf6e
Step 6/24 : RUN apt-get install --assume-yes manpages-posix
 ---> Using cache
 ---> 6697e68ddeb1
Step 7/24 : RUN apt-get install --assume-yes man-db
 ---> Using cache
 ---> 5cb7b8935037
Step 8/24 : RUN apt-get install --assume-yes apt-utils
 ---> Using cache
 ---> 0e14aec824c0
Step 9/24 : RUN apt-get install --assume-yes --reinstall less curl
 ---> Using cache
 ---> 49dd647f04a9
Step 10/24 : RUN apt-get install --assume-yes bash-completion man-db neovim git curl python3 python3-venv python-is-python3
 ---> Using cache
 ---> 7ee285630bf4
Step 11/24 : RUN apt-get install --assume-yes bash-completion git
 ---> Using cache
 ---> a15d6a35fba3
Step 12/24 : RUN apt-get install --assume-yes bash-completion neovim
 ---> Using cache
 ---> d9ec5fb2ccc3
Step 13/24 : RUN apt-get install --assume-yes bash-completion python3 python3-venv python-is-python3
 ---> Using cache
 ---> 0ca231bd4656
Step 14/24 : RUN apt-get install --assume-yes sudo
 ---> Using cache
 ---> b401a0d02d5b
Step 15/24 : RUN apt-get upgrade --assume-yes
 ---> Using cache
 ---> 033a508b0924
Step 16/24 : RUN apt-get autoremove --assume-yes
 ---> Using cache
 ---> d12b0a29b6c6
Step 17/24 : RUN pip install --upgrade setuptools pip wheel
 ---> Using cache
 ---> fad9714f30d3
Step 18/24 : RUN pip install --upgrade jupyterlab
 ---> Using cache
 ---> 6917557c312d
Step 19/24 : ENV no_proxy jupyterhub
 ---> Using cache
 ---> e03585d79a5d
Step 20/24 : ENV no_proxy twopyter
 ---> Using cache
 ---> 0d8ce735333c
Step 21/24 : RUN echo '\n####\n# USER ADDED\n####\n' >> /etc/skel/.bashrc
 ---> Using cache
 ---> 6272b84d98d8
Step 22/24 : RUN echo '\n#Help pip use postal proxy\nexport HTTPS_PROXY=http://proxy.usps.gov:8080\n' >> /etc/skel/.bashrc
 ---> Using cache
 ---> 0af0de9ee3a2
Step 23/24 : RUN echo '\n# Add pip bin directory to user path\nexport PATH=$HOME/.local/bin:$PATH\n' >> /etc/skel/.bashrc
 ---> Using cache
 ---> f9e7abcad8ca
Step 24/24 : CMD ["jupyterhub", "-f", "/etc/jupyterhub/jupyterhub_config.py"]
 ---> Using cache
 ---> 77f006fe409a
Successfully built 77f006fe409a
Successfully tagged jupyterhub_notebook:latest
Creating jupyterhub ... done
Attaching to jupyterhub
jupyterhub    | [I 2022-02-23 19:50:45.830 JupyterHub app:2766] Running JupyterHub version 2.1.1
jupyterhub    | [I 2022-02-23 19:50:45.830 JupyterHub app:2796] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.1.1
jupyterhub    | [I 2022-02-23 19:50:45.830 JupyterHub app:2796] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-2.1.1
jupyterhub    | [I 2022-02-23 19:50:45.831 JupyterHub app:2796] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.1.1
jupyterhub    | [I 2022-02-23 19:50:45.841 JupyterHub app:1606] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
jupyterhub    | [I 2022-02-23 19:50:45.927 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
jupyterhub    | [I 2022-02-23 19:50:45.939 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
jupyterhub    | [I 2022-02-23 19:50:45.971 JupyterHub app:2835] Initialized 0 spawners in 0.003 seconds
jupyterhub    | [W 2022-02-23 19:50:45.974 JupyterHub proxy:687] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
jupyterhub    | [I 2022-02-23 19:50:45.974 JupyterHub proxy:691] Starting proxy @ http://:8000/rubber/baby/buggy/bumpers/
jupyterhub    | 19:50:46.684 [ConfigProxy] info: Proxying http://*:8000 to (no default)
jupyterhub    | 19:50:46.688 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
jupyterhub    | 19:50:47.018 [ConfigProxy] info: 200 GET /api/routes
jupyterhub    | [I 2022-02-23 19:50:47.018 JupyterHub app:3084] Hub API listening on http://127.0.0.1:8081/rubber/baby/buggy/bumpers/hub/
jupyterhub    | 19:50:47.020 [ConfigProxy] info: 200 GET /api/routes
jupyterhub    | [I 2022-02-23 19:50:47.020 JupyterHub proxy:431] Adding route for Hub: /rubber/baby/buggy/bumpers/ => http://127.0.0.1:8081
jupyterhub    | 19:50:47.022 [ConfigProxy] info: Adding route /rubber/baby/buggy/bumpers -> http://127.0.0.1:8081
jupyterhub    | 19:50:47.023 [ConfigProxy] info: Route added /rubber/baby/buggy/bumpers -> http://127.0.0.1:8081
jupyterhub    | 19:50:47.024 [ConfigProxy] info: 201 POST /api/routes/rubber/baby/buggy/bumpers
jupyterhub    | [I 2022-02-23 19:50:47.024 JupyterHub app:3150] JupyterHub is now running at http://:8000/rubber/baby/buggy/bumpers/

One step closer. It looks like the static content can’t load. Will pick this back up tomorrow.

Screenshot of browser

JupyterHub Log

jupyterhub    | 19:58:08.115 [ConfigProxy] error: 404 GET /
jupyterhub    | [I 2022-02-23 19:58:08.163 JupyterHub log:189] 200 GET /twopyter/hub/error/404?url=%2F (@127.0.0.1) 41.28ms
jupyterhub    | 19:58:08.201 [ConfigProxy] error: 404 GET /hub/static/css/style.min.css
jupyterhub    | [I 2022-02-23 19:58:08.204 JupyterHub log:189] 200 GET /twopyter/hub/error/404?url=%2Fhub%2Fstatic%2Fcss%2Fstyle.min.css%3Fv%3Dbff49b4a161afb17ee3b71927ce7d6c4e5b0e4b9ef6f18ca3e356a05f29e69776d3a76aee167060dd2ae2ee62d3cfdcf203b4b0090b1423f7d629ea7daa3f9da (@127.0.0.1) 1.40ms
jupyterhub    | 19:58:08.212 [ConfigProxy] error: 404 GET /hub/static/components/bootstrap/dist/js/bootstrap.min.js
jupyterhub    | [I 2022-02-23 19:58:08.215 JupyterHub log:189] 200 GET /twopyter/hub/error/404?url=%2Fhub%2Fstatic%2Fcomponents%2Fbootstrap%2Fdist%2Fjs%2Fbootstrap.min.js%3Fv%3Da014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce (@127.0.0.1) 1.32ms
jupyterhub    | 19:58:08.218 [ConfigProxy] error: 404 GET /hub/static/components/requirejs/require.js
jupyterhub    | [I 2022-02-23 19:58:08.221 JupyterHub log:189] 200 GET /twopyter/hub/error/404?url=%2Fhub%2Fstatic%2Fcomponents%2Frequirejs%2Frequire.js%3Fv%3Dbd1aa102bdb0b27fbf712b32cfcd29b016c272acf3d864ee8469376eaddd032cadcf827ff17c05a8c8e20061418fe58cf79947049f5c0dff3b4f73fcc8cad8ec (@127.0.0.1) 1.31ms
jupyterhub    | 19:58:08.224 [ConfigProxy] error: 404 GET /hub/static/components/jquery/dist/jquery.min.js
jupyterhub    | [I 2022-02-23 19:58:08.226 JupyterHub log:189] 200 GET /twopyter/hub/error/404?url=%2Fhub%2Fstatic%2Fcomponents%2Fjquery%2Fdist%2Fjquery.min.js%3Fv%3Df3de1813a4160f9239f4781938645e1589b876759cd50b7936dbd849a35c38ffaed53f6a61dbdd8a1cf43cf4a28aa9fffbfddeec9a3811a1bb4ee6df58652b31 (@127.0.0.1) 1.31ms
jupyterhub    | 19:58:08.230 [ConfigProxy] error: 404 GET /hub/logo
jupyterhub    | [I 2022-02-23 19:58:08.232 JupyterHub log:189] 200 GET /twopyter/hub/error/404?url=%2Fhub%2Flogo (@127.0.0.1) 1.31ms
jupyterhub    | 19:58:08.237 [ConfigProxy] error: 404 GET /hub/static/components/bootstrap/dist/js/bootstrap.min.js
jupyterhub    | [I 2022-02-23 19:58:08.239 JupyterHub log:189] 200 GET /twopyter/hub/error/404?url=%2Fhub%2Fstatic%2Fcomponents%2Fbootstrap%2Fdist%2Fjs%2Fbootstrap.min.js%3Fv%3Da014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce (@127.0.0.1) 1.33ms

Nginx Log

reverse_proxy    | [redacted] - - [23/Feb/2022:19:58:08 +0000] "GET /twopyter HTTP/1.1" 301 169 "-" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:19:58:08 +0000] "GET /twopyter/ HTTP/1.1" 404 4949 "-" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:19:58:08 +0000] "GET /twopyter/hub/static/css/style.min.css?v=bff49b4a161afb17ee3b71927ce7d6c4e5b0e4b9ef6f18ca3e356a05f29e69776d3a76aee167060dd2ae2ee62d3cfdcf203b4b0090b1423f7d629ea7daa3f9da HTTP/1.1" 404 4949 "https://[redacted]/twopyter/" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:19:58:08 +0000] "GET /twopyter/hub/static/components/bootstrap/dist/js/bootstrap.min.js?v=a014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce HTTP/1.1" 404 4949 "https://[redacted]/twopyter/" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:19:58:08 +0000] "GET /twopyter/hub/static/components/requirejs/require.js?v=bd1aa102bdb0b27fbf712b32cfcd29b016c272acf3d864ee8469376eaddd032cadcf827ff17c05a8c8e20061418fe58cf79947049f5c0dff3b4f73fcc8cad8ec HTTP/1.1" 404 4949 "https://[redacted]/twopyter/" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:19:58:08 +0000] "GET /twopyter/hub/static/components/jquery/dist/jquery.min.js?v=f3de1813a4160f9239f4781938645e1589b876759cd50b7936dbd849a35c38ffaed53f6a61dbdd8a1cf43cf4a28aa9fffbfddeec9a3811a1bb4ee6df58652b31 HTTP/1.1" 404 4949 "https://[redacted]/twopyter/" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:19:58:08 +0000] "GET /twopyter/hub/logo HTTP/1.1" 404 4949 "https://[redacted]/twopyter/" "Mozilla/[redacted]" "-"
reverse_proxy    | [redacted] - - [23/Feb/2022:19:58:08 +0000] "GET /twopyter/hub/static/components/bootstrap/dist/js/bootstrap.min.js?v=a014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce HTTP/1.1" 404 4949 "https://[redacted]/twopyter/" "Mozilla/[redacted]" "-"

I fixed the static content issue. I had to remove a trailing slash from the nginx proxy_pass directive. Everything is now working correctly.

Final Summary

  1. Had to force the JupyterHub configuration file to be sourced on container startup by adding the line to the end of the Dockerfile:
CMD ["jupyterhub", "-f", "/etc/jupyterhub/jupyterhub_config.py"]
  1. Since I am serving JupyterHub through a reverse proxy on the subpath twopyter, I had to set the c.JupyterHub.bind_url configuration variable with:
c.JupyterHub.bind_url = 'http://:8000/twopyter/'
  1. In the reverse proxy Nginx configuration, you cannot have a trailing slash after your proxy_pass directive. If you do, the static page content will be unable to load. Once I fixed this, it solved the last remaining problem. I do not understand why it breaks things, but that’s the way it is. So for us it looks like this:
    location /twopyter/ {
        proxy_pass              http://host.docker.internal:15080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # websocket headers
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

Thank you @manics for all your help.

2 Likes