I’m using docker-compose to launch JupyterHub on Docker. The idea is to have 1 JupyterHub container, 1 nginx container (mainly for SSL connections) and 1 or more JupyterLab containers that contain users’ JupyterLab instances. The Dockerspawner is used to create these new containers when a user logs in.
I’m encountering an error that leads me to believe that there’s some connection issue between the newly launched container and the Hub container, seen in the logs here. Can someone tell me what’s causing the ECONNREFUSED error? I’ve obfuscated some IP addresses but I don’t think it’s a big deal. Let me know if anything needs further clarification:
Creating jupyterhub ...
Creating bridge_reverse-proxy_1 ... done
Creating bridge_reverse-proxy_1 ...
Attaching to jupyterlab-throaway, jupyterhub, bridge_reverse-proxy_1
jupyterlab-throaway |
reverse-proxy_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
reverse-proxy_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
jupyterlab-throaway exited with code 0
reverse-proxy_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
reverse-proxy_1 | 10-listen-on-ipv6-by-default.sh: error: ipv6 not available
reverse-proxy_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
reverse-proxy_1 | /docker-entrypoint.sh: Configuration complete; ready for start up
jupyterhub | [I 2020-11-30 21:10:51.356 JupyterHub app:2332] Running JupyterHub version 1.2.1
jupyterhub | [I 2020-11-30 21:10:51.357 JupyterHub app:2362] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.2.1
jupyterhub | [I 2020-11-30 21:10:51.357 JupyterHub app:2362] Using Spawner: dockerspawner.systemuserspawner.SystemUserSpawner-0.11.1
jupyterhub | [I 2020-11-30 21:10:51.357 JupyterHub app:2362] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.2.1
jupyterhub | [I 2020-11-30 21:10:51.373 JupyterHub app:1457] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
jupyterhub | [I 2020-11-30 21:10:51.405 alembic.runtime.migration migration:155] Context impl SQLiteImpl.
jupyterhub | [I 2020-11-30 21:10:51.405 alembic.runtime.migration migration:158] Will assume non-transactional DDL.
jupyterhub | [I 2020-11-30 21:10:51.414 alembic.runtime.migration migration:517] Running stamp_revision -> 4dc2d5a8c53c
jupyterhub | [I 2020-11-30 21:10:51.468 JupyterHub proxy:461] Generating new CONFIGPROXY_AUTH_TOKEN
jupyterhub | [W 2020-11-30 21:10:51.470 JupyterHub app:1687] No admin users, admin interface will be unavailable.
jupyterhub | [W 2020-11-30 21:10:51.470 JupyterHub app:1688] Add any administrative users to `c.Authenticator.admin_users` in config.
jupyterhub | [I 2020-11-30 21:10:51.471 JupyterHub app:1717] Not using allowed_users. Any authenticated user will be allowed.
jupyterhub | [I 2020-11-30 21:10:51.539 JupyterHub app:2399] Initialized 0 spawners in 0.003 seconds
jupyterhub | [W 2020-11-30 21:10:51.543 JupyterHub proxy:642] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
jupyterhub | [I 2020-11-30 21:10:51.544 JupyterHub proxy:646] Starting proxy @ http://:8000
jupyterhub | 21:10:52.608 [ConfigProxy] info: Proxying http://*:8000 to (no default)
jupyterhub | 21:10:52.613 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
jupyterhub | 21:10:52.642 [ConfigProxy] info: 200 GET /api/routes
jupyterhub | [I 2020-11-30 21:10:52.647 JupyterHub app:2647] Hub API listening on http://HUB_IP:8081/hub/
jupyterhub | 21:10:52.649 [ConfigProxy] info: 200 GET /api/routes
jupyterhub | [I 2020-11-30 21:10:52.650 JupyterHub proxy:320] Checking routes
jupyterhub | [I 2020-11-30 21:10:52.650 JupyterHub proxy:400] Adding default route for Hub: / => http://HUB_IP:8081
jupyterhub | 21:10:52.653 [ConfigProxy] info: Adding route / -> http://HUB_IP:8081
jupyterhub | 21:10:52.654 [ConfigProxy] info: Route added / -> http://HUB_IP:8081
jupyterhub | 21:10:52.654 [ConfigProxy] info: 201 POST /api/routes/
jupyterhub | [I 2020-11-30 21:10:52.655 JupyterHub app:2722] JupyterHub is now running at http://:8000
jupyterhub | [I 2020-11-30 21:11:48.524 JupyterHub log:181] 302 GET / -> /hub/ (@IP) 2.69ms
reverse-proxy_1 | IP - - [30/Nov/2020:21:11:48 +0000] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
jupyterhub | [W 2020-11-30 21:11:48.628 JupyterHub base:389] Invalid or expired cookie token
jupyterhub | [I 2020-11-30 21:11:48.629 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@IP) 2.84ms
reverse-proxy_1 | IP - - [30/Nov/2020:21:11:48 +0000] "GET /hub/ HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
jupyterhub | [I 2020-11-30 21:11:48.778 JupyterHub log:181] 200 GET /hub/login (@IP) 53.25ms
reverse-proxy_1 | IP - - [30/Nov/2020:21:11:48 +0000] "GET /hub/login HTTP/1.1" 200 4960 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
jupyterhub | [I 2020-11-30 21:12:16.321 JupyterHub base:746] User logged in: username
jupyterhub | [I 2020-11-30 21:12:16.326 JupyterHub log:181] 302 POST /hub/login?next= -> /hub/spawn (username@IP) 60.49ms
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:16 +0000] "POST /hub/login?next= HTTP/1.1" 302 0 "https://hostname.domain:8080/hub/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
jupyterhub | [I 2020-11-30 21:12:16.525 JupyterHub dockerspawner:783] Container 'jupyter-username' is gone
jupyterhub | [I 2020-11-30 21:12:16.568 JupyterHub dockerspawner:985] Created container jupyter-username (id: 785400a) from image jupyterlab_img
jupyterhub | [I 2020-11-30 21:12:16.569 JupyterHub dockerspawner:1009] Starting container jupyter-username (id: 785400a)
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:17 +0000] "GET /hub/spawn HTTP/1.1" 302 0 "https://hostname.domain:8080/hub/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
jupyterhub | [I 2020-11-30 21:12:17.470 JupyterHub log:181] 302 GET /hub/spawn -> /hub/spawn-pending/username (username@IP) 1010.93ms
jupyterhub | [I 2020-11-30 21:12:17.590 JupyterHub pages:398] username is pending spawn
jupyterhub | [I 2020-11-30 21:12:17.598 JupyterHub log:181] 200 GET /hub/spawn-pending/username (username@IP) 13.41ms
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:17 +0000] "GET /hub/spawn-pending/username HTTP/1.1" 200 6658 "https://hostname.domain:8080/hub/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
jupyterhub | [I 2020-11-30 21:12:19.393 JupyterHub base:893] User username took 2.926 seconds to start
jupyterhub | [I 2020-11-30 21:12:19.394 JupyterHub proxy:258] Adding user username to proxy /user/username/ => http://JUPYTERLAB_IP:8888
jupyterhub | [I 2020-11-30 21:12:19.397 JupyterHub log:181] 200 GET /hub/api (@JUPYTERLAB_IP) 1.90ms
jupyterhub | 21:12:19.398 [ConfigProxy] info: Adding route /user/username -> http://JUPYTERLAB_IP:8888
jupyterhub | 21:12:19.398 [ConfigProxy] info: Route added /user/username -> http://JUPYTERLAB_IP:8888
jupyterhub | 21:12:19.399 [ConfigProxy] info: 201 POST /api/routes/user/username
jupyterhub | [I 2020-11-30 21:12:19.399 JupyterHub users:609] Server username is ready
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:19 +0000] "GET /hub/api/users/username/server/progress HTTP/1.1" 200 324 "https://hostname.domain:8080/hub/spawn-pending/username" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
jupyterhub | [I 2020-11-30 21:12:19.401 JupyterHub log:181] 200 GET /hub/api/users/username/server/progress (username@IP) 1639.70ms
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:19 +0000] "GET /hub/spawn-pending/username HTTP/1.1" 302 0 "https://hostname.domain:8080/hub/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
jupyterhub | [I 2020-11-30 21:12:19.542 JupyterHub log:181] 302 GET /hub/spawn-pending/username -> /user/username/ (username@IP) 4.80ms
jupyterhub | 21:12:19.643 [ConfigProxy] error: 503 GET /user/username/ connect ECONNREFUSED JUPYTERLAB_IP:8888
jupyterhub | [I 2020-11-30 21:12:19.659 JupyterHub log:181] 200 GET /hub/error/503?url=%2Fuser%2Fusername%2F (@HUB_IP) 12.38ms
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:19 +0000] "GET /user/username/ HTTP/1.1" 503 4450 "https://hostname.domain:8080/hub/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:19 +0000] "GET /hub/static/css/style.min.css?v=fb871f5be0bcde20eaba9ab56d2f4bfc HTTP/1.1" 304 0 "https://hostname.domain:8080/user/username/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:19 +0000] "GET /hub/static/components/requirejs/require.js?v=f0cc8bbb2fcef87fc194fecbb632fcfa HTTP/1.1" 304 0 "https://hostname.domain:8080/user/username/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:19 +0000] "GET /hub/static/components/jquery/dist/jquery.min.js?v=dc5e7f18c8d36ac1d3d4753a87c98d0a HTTP/1.1" 304 0 "https://hostname.domain:8080/user/username/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:19 +0000] "GET /hub/static/components/bootstrap/dist/js/bootstrap.min.js?v=2f34b630ffe30ba2ff2b91e3f3c322a1 HTTP/1.1" 304 0 "https://hostname.domain:8080/user/username/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
reverse-proxy_1 | IP - - [30/Nov/2020:21:12:20 +0000] "GET /hub/logo HTTP/1.1" 304 0 "https://hostname.domain:8080/user/username/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" "-"
jupyterhub | [W 2020-11-30 21:12:47.203 JupyterHub base:1056] User username server stopped, with exit code: ExitCode=1, Error='', FinishedAt=2020-11-30T21:12:19.688573813Z
jupyterhub | [I 2020-11-30 21:12:47.203 JupyterHub proxy:282] Removing user username from proxy (/user/username/)
jupyterhub | 21:12:47.206 [ConfigProxy] info: Removing route /user/username
jupyterhub | 21:12:47.207 [ConfigProxy] info: 204 DELETE /api/routes/user/username
jupyterhub | 21:15:52.658 [ConfigProxy] info: 200 GET /api/routes
jupyterhub | [I 2020-11-30 21:15:52.659 JupyterHub proxy:320] Checking routes
jupyterhub | 21:20:52.658 [ConfigProxy] info: 200 GET /api/routes
jupyterhub | [I 2020-11-30 21:20:52.659 JupyterHub proxy:320] Checking routes
jupyterhub | 21:25:52.658 [ConfigProxy] info: 200 GET /api/routes
jupyterhub | [I 2020-11-30 21:25:52.659 JupyterHub proxy:320] Checking routes
jupyterhub | 21:30:52.658 [ConfigProxy] info: 200 GET /api/routes
jupyterhub | [I 2020-11-30 21:30:52.659 JupyterHub proxy:320] Checking routes
bridge_reverse-proxy_1 exited with code 0
docker-compose.yaml:
version: "3.3"
services:
reverse-proxy:
image: nginx:latest
volumes:
- "./reverse-ssl-proxy/nginx.conf:/etc/nginx/nginx.conf"
- "/etc/pki/tls/certs/:/etc/certs/"
- "/var/run/docker.sock:/var/run/docker.sock"
ports:
- 8080:8080
links:
- "jupyterhub"
restart: on-failure
jupyterhub:
build: jupyterhub
image: jupyterhub_img
container_name: jupyterhub
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- DOCKER_JUPYTER_IMAGE=jupyterlab_img
- DOCKER_JUPYTER_CONTAINER=jupyterlab_img
- DOCKER_NETWORK_NAME=${COMPOSE_PROJECT_NAME}_default
jupyterlab:
build: jupyterlab
image: jupyterlab_img
container_name: jupyterlab-throaway
network_mode: none
command: echo
nginx config:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
upstream ml {
server jupyterhub:8000;
}
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
# I think this "map" is needed for proper proxying:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
# Settings for a TLS enabled server.
server {
listen 8080 ssl;
# listen 443 ssl http2 default_server;
# listen [::]:443 ssl http2 default_server;
# server_name _;
server_name hostname.domain;
ssl_certificate "/etc/certs/dummy.crt";
ssl_certificate_key "/etc/certs/dummer.key";
ssl_protocols TLSv1.2;
ssl_session_cache shared:SSL:59m;
ssl_session_timeout 1d;
# ssl_ciphers 'AES-256-CBC-HMAC-SHA256'
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
# ssl_stapling on;
# ssl_stapling_verify on;
add_header Strict-Transport-Security max-age=160000;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
#proxy_pass https://ml;
proxy_pass http://ml;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host:$server_port;
#proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# websocket headers
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
location ~ /.well-known {
allow all;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
JupyterHub config:
from jupyter_client.localinterfaces import public_ips
import os
## Generic
c = get_config()
c.JupyterHub.admin_access = True
c.Spawner.default_url = '/lab'
c.JupyterHub.hub_ip = public_ips()[0]
c.JupyterHub.hub_connect_ip = 'jupyterhub' # ip as seen on the docker network. Can also be a hostname.
# SystemUserSpawner
c.JupyterHub.spawner_class = 'dockerspawner.SystemUserSpawner'
# Docker spawner
c.DockerSpawner.image = os.environ['DOCKER_JUPYTER_CONTAINER']
c.DockerSpawner.network_name = os.environ['DOCKER_NETWORK_NAME']
c.DockerSpawner.extra_host_config = { 'network_mode': os.environ['DOCKER_NETWORK_NAME'] }
c.Spawner.cmd = ['jupyterhub-singleuser']
# Authentication
c.PAMAuthenticator.service = '/etc/pam.d/passwd'
Thanks for your help