Spawned Docker doesn't connect to kernel if the environment is behind proxy

JupyterHub
1

Hi,
The issue consist in kernel not connecting (neither does the command line) if I use proxy.
Everything works fine if I directly connect from lan (http://192.168.50.163:8000) or with internal dns (http://jupyter.internal.local:8000)

I Expect it to behave the same way as it does from local

I can Log In and can create new files
But kernet doesn’t start (or connect)

image
image1005×193 8.46 KB

Even console isn’t working

When using proxy I don’t seem to view any activity on jupyterhub log status. (maybe is my service not properly configured, it is the same as the jupyterhub start guide ground up:

root#cat /opt/jupyterhub/etc/systemd/jupyterhub.service
[Unit]
Description=JupyterHub
After=syslog.target network.target

[Service]
User=root
Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/jupyterhub/bin"
ExecStart=jupyterhub -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py

[Install]
WantedBy=multi-user.target

I have a minimal configuration

  • jupyterhub + dockerspawner
  • persistence volumes
  • user folder creation

and Nginx proxy

  • OS: Debian 10
  • jupyterhub 1.2.1
  • nginx version: 1.14.2

My hub config file:

import os
import sys
import shutil

def create_dir_hook(spawner):
    volume_path = os.path.join('/opt/jupyterhub/user_volumes/', username) 
    if not os.path.exists(volume_path):
        os.mkdir(volume_path)
        shutil.chown(volume_path, user='raikoug', group='users')
        os.chmod(volume_path, 0o755)

c = get_config()


c.JupyterHub.log_level = 'DEBUG'
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.image = 'jupyter/scipy-notebook'
spawn_cmd = os.environ.get('DOCKER_SPAWN_CMD', "start-singleuser.sh --SingleUserNotebookApp.default_url=/lab")
c.DockerSpawner.extra_create_kwargs.update({ 'command': spawn_cmd })
network_name = 'bridge'
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.network_name = network_name
c.DockerSpawner.extra_host_config = {
    'network_mode': network_name,
    'mem_limit': '300m',
    'memswap_limit': '300m',
    'mem_swappiness': 0
}

c.DockerSpawner.pre_spawn_hook = create_dir_hook


notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan'
c.DockerSpawner.notebook_dir = notebook_dir
host_dir = '/opt/jupyterhub/user_volumes/{username}'
c.DockerSpawner.volumes = { host_dir: notebook_dir }
c.DockerSpawner.remove_containers = True
c.DockerSpawner.debug = True

c.JupyterHub.hub_ip = '192.168.50.163'
c.JupyterHub.hub_port = 8080

c.Authenticator.whitelist = whitelist = set()
c.Authenticator.admin_users = admin = set()
c.JupyterHub.admin_access = True
pwd = os.path.dirname(__file__)
with open('/opt/jupyterhub/etc/jupyterhub/userlist') as f:
    for line in f:
        if not line:
            continue
        parts = line.split()
        if len(parts) >= 1:
            name = parts[0]
            whitelist.add(name)
            if len(parts) > 1 and parts[1] == 'admin':
                admin.add(name)


c.JupyterHub.services = [
    {
        'name': 'idle-culler',
        'admin': True,
        'command': [
            sys.executable,
            '-m', 'jupyterhub_idle_culler',
            '--timeout=3600'
        ],
    }
]

My Nginx site:;

server {
    server_name         jupyter.my_site.com;

    location / {
          proxy_set_header        Host $host;
          proxy_set_header        X-Real-IP $remote_addr;
          proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header        X-Forwarded-Proto $scheme;

          proxy_pass              http://localhost:8000/;

          proxy_read_timeout      600s;
          proxy_send_timeout      600s;

        }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/jupyter.my_site.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/jupyter.my_site.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    if ($host = jupyter.my_site.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen              80;
    server_name         jupyter.my_site.com;
    return 404; # managed by Certbot

}
2

What about the websocket traffic? Do you forward it?

See e.g. https://jupyterhub.readthedocs.io/en/stable/reference/config-proxy.html

1 Like
3

Many thanks,
I forgot this (and related lines)

map $http_upgrade $connection_upgrade {
default upgrade;
‘’ close;
}

in NGINX
Now kernel connect.
If I upload a file I can run commands in it, but if I save it it dive 403
If I create a notebook, instead, I can save it.
What’s happening?
maybe should be better to one a new topic…

1 Like
4

I guess a new topic is better.

1 Like