Error "Server unavailable or unreachable" - Jupyterhub running in Docker

pa-ko-nok · May 17, 2023, 12:14pm

Hi there!

I’m trying to spin up a JupyterHub instance based on this repository. My goal is the following: JupyterHub runs in a Docker container, each user that logs in creates his own Docker container, which talks to the “central” JupyterHub container (working 90%-ish). Users can authenticate via LDAP (working already). TLS is not implemented yet because it’s still in alpha testing - will be integrated as a next step.

This is my docker-compose.yaml

version: "3.5"

services:
  hub:
    build:
      context: .
      dockerfile: Dockerfile.jupyterhub
      args:
        JUPYTERHUB_VERSION: 4.0.0
    restart: always
    image: jupyterhub
    container_name: jupyterhub
    networks:
      - jupyterhub-network
    volumes:
      - "./jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:rw"
      - "jupyterhub-data:/data"
    ports:
      - "8000:8000"
    environment:
      JUPYTERHUB_ADMIN: admin
      DOCKER_NETWORK_NAME: jupyterhub-network
      DOCKER_NOTEBOOK_DIR: /home/jovyan/work
      DOCKER_NOTEBOOK_IMAGE: jupyter/minimal-notebook:latest
      DOCKER_SPAWN_CMD: start-singleuser.sh

volumes:
  jupyterhub-data:

networks:
  jupyterhub-network:
    name: jupyterhub-network

This is my jupyterhub_config.py file:

import os
import subprocess
import logging
from ldapauthenticator import LDAPAuthenticator
import netifaces

c = get_config()
spawn_cmd = os.environ.get("DOCKER_SPAWN_CMD", "start-singleuser.sh")
network_name = os.environ["DOCKER_NETWORK_NAME"]

c.DockerSpawner.image = os.environ["DOCKER_NOTEBOOK_IMAGE"]
c.DockerSpawner.cmd = spawn_cmd

c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.network_name = network_name

notebook_dir = "/home/jovyan/work"
c.DockerSpawner.notebook_dir = notebook_dir


c.DockerSpawner.volumes = {
    "jupyterhub-user-{username}": notebook_dir
}

c.DockerSpawner.remove = True
c.DockerSpawner.debug = True

c.JupyterHub.log_level = logging.DEBUG
c.JupyterHub.spawner_class = "dockerspawner.DockerSpawner"

docker0 = netifaces.ifaddresses('eth0')
docker0_ipv4 = docker0[netifaces.AF_INET][0]
c.JupyterHub.hub_ip = docker0_ipv4['addr']

[other config considering LDAPauthenticator]

Once in a while, at random moments but with a certain periodicity (~ 10-20 sec), an error window pops up and says: “Server unavailable or unreachable” - below this message are two buttons “restart” and “dismiss”. I can click on dismiss and everything continues as if nothing had happened but it’s annoying as hell and would break the workflow of any developer.

When I look into Firefox’s debugger, I see this:

Has anyone experienced this or something similar?
Any help would be greatly appreciated.

Thanks,
Paul

manics · May 18, 2023, 1:55pm

What’s the full browser error log? In your screenshot it’s truncated to NS_ERROR_UNKN..., this may contain some clues.

In addition you can try turning on debug logging, and showing us your hub and singleuser logs.

pa-ko-nok · May 22, 2023, 9:07am

Thanks for replying so quickly!

The full error log looks just says “NS_ERROR_UNKNOWN_HOST” and looks like this:

docker logs -n 100 -t jupytherhub produces:

2023-05-22T08:55:41.970605159Z [D 2023-05-22 08:55:41.970 JupyterHub base:342] Refreshing auth for paul kockritz
2023-05-22T08:55:41.971285122Z [D 2023-05-22 08:55:41.971 JupyterHub scopes:863] Checking access via scope users:activity
2023-05-22T08:55:41.971317983Z [D 2023-05-22 08:55:41.971 JupyterHub scopes:690] Argument-based access to /hub/api/users/paul%20kockritz/activity via users:activity
2023-05-22T08:55:41.973619836Z [D 2023-05-22 08:55:41.973 JupyterHub users:879] Not updating activity for <User(paul kockritz 1/1 running)>: 2023-05-22T08:49:04.674426Z < 2023-05-22T08:52:21.645000Z
2023-05-22T08:55:41.973750062Z [D 2023-05-22 08:55:41.973 JupyterHub users:900] Not updating server activity on paul kockritz/: 2023-05-22T08:49:04.674426Z < 2023-05-22T08:52:21.645000Z
2023-05-22T08:55:41.974896139Z [I 2023-05-22 08:55:41.974 JupyterHub log:191] 200 POST /hub/api/users/paul%20kockritz/activity (paul kockritz@192.168.160.3) 11.51ms
2023-05-22T08:56:18.839838013Z [D 2023-05-22 08:56:18.839 JupyterHub dockerspawner:982] Getting container 'jupyter-paul-20kockritz'
2023-05-22T08:56:18.847389693Z [D 2023-05-22 08:56:18.846 JupyterHub dockerspawner:967] Container c725708 status: {'Dead': False,
2023-05-22T08:56:18.847435343Z      'Error': '',
2023-05-22T08:56:18.847443064Z      'ExitCode': 0,
2023-05-22T08:56:18.847450180Z      'FinishedAt': '0001-01-01T00:00:00Z',
2023-05-22T08:56:18.847457067Z      'Health': {'FailingStreak': 0,
2023-05-22T08:56:18.847463429Z                 'Log': [{'End': '2023-05-22T08:55:53.92289397Z',
2023-05-22T08:56:18.847470004Z                          'ExitCode': 0,
2023-05-22T08:56:18.847476204Z                          'Output': 'b\'{"version": "2.3.0"}\'\n',
2023-05-22T08:56:18.847484023Z                          'Start': '2023-05-22T08:55:53.701690399Z'},
2023-05-22T08:56:18.847490538Z                         {'End': '2023-05-22T08:55:59.148769526Z',
2023-05-22T08:56:18.847496785Z                          'ExitCode': 0,
2023-05-22T08:56:18.847503205Z                          'Output': 'b\'{"version": "2.3.0"}\'\n',
2023-05-22T08:56:18.847535315Z                          'Start': '2023-05-22T08:55:58.932483949Z'},
2023-05-22T08:56:18.847576535Z                         {'End': '2023-05-22T08:56:04.349098147Z',
2023-05-22T08:56:18.847583642Z                          'ExitCode': 0,
2023-05-22T08:56:18.847590260Z                          'Output': 'b\'{"version": "2.3.0"}\'\n',
2023-05-22T08:56:18.847597590Z                          'Start': '2023-05-22T08:56:04.15426247Z'},
2023-05-22T08:56:18.847603997Z                         {'End': '2023-05-22T08:56:09.546139302Z',
2023-05-22T08:56:18.847610389Z                          'ExitCode': 0,
2023-05-22T08:56:18.847616562Z                          'Output': 'b\'{"version": "2.3.0"}\'\n',
2023-05-22T08:56:18.847623479Z                          'Start': '2023-05-22T08:56:09.363774521Z'},
2023-05-22T08:56:18.847629866Z                         {'End': '2023-05-22T08:56:14.78238665Z',
2023-05-22T08:56:18.847636283Z                          'ExitCode': 0,
2023-05-22T08:56:18.847642347Z                          'Output': 'b\'{"version": "2.3.0"}\'\n',
2023-05-22T08:56:18.847649000Z                          'Start': '2023-05-22T08:56:14.554381088Z'}],
2023-05-22T08:56:18.847655511Z                 'Status': 'healthy'},
2023-05-22T08:56:18.847661700Z      'OOMKilled': False,
2023-05-22T08:56:18.847668353Z      'Paused': False,
2023-05-22T08:56:18.847674580Z      'Pid': 7415,
2023-05-22T08:56:18.847680685Z      'Restarting': False,
2023-05-22T08:56:18.847686774Z      'Running': True,
2023-05-22T08:56:18.847692819Z      'StartedAt': '2023-05-17T10:08:18.826706404Z',
2023-05-22T08:56:18.847699377Z      'Status': 'running'}
2023-05-22T08:57:40.222148041Z [D 2023-05-22 08:57:40.220 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
2023-05-22T08:57:40.226074185Z 08:57:40.225 [ConfigProxy] info: 200 GET /api/routes
2023-05-22T08:57:40.235210702Z [D 2023-05-22 08:57:40.234 JupyterHub proxy:392] Checking routes
2023-05-22T08:58:18.839678075Z [D 2023-05-22 08:58:18.839 JupyterHub dockerspawner:982] Getting container 'jupyter-paul-20kockritz'
2023-05-22T08:58:18.848501734Z [D 2023-05-22 08:58:18.847 JupyterHub dockerspawner:967] Container c725708 status: {'Dead': False,
2023-05-22T08:58:18.848636638Z      'Error': '',
2023-05-22T08:58:18.848648466Z      'ExitCode': 0,
2023-05-22T08:58:18.848654857Z      'FinishedAt': '0001-01-01T00:00:00Z',
2023-05-22T08:58:18.848660685Z      'Health': {'FailingStreak': 0,
2023-05-22T08:58:18.848666194Z                 'Log': [{'End': '2023-05-22T08:57:53.89326769Z',
2023-05-22T08:58:18.848671958Z                          'ExitCode': 0,
2023-05-22T08:58:18.848677403Z                          'Output': 'b\'{"version": "2.3.0"}\'\n',
2023-05-22T08:58:18.848684196Z                          'Start': '2023-05-22T08:57:53.684995416Z'},
2023-05-22T08:58:18.848726997Z                         {'End': '2023-05-22T08:57:59.104910295Z',
2023-05-22T08:58:18.848733806Z                          'ExitCode': 0,
2023-05-22T08:58:18.848739767Z                          'Output': 'b\'{"version": "2.3.0"}\'\n',
2023-05-22T08:58:18.848745955Z                          'Start': '2023-05-22T08:57:58.898430897Z'},
2023-05-22T08:58:18.848751718Z                         {'End': '2023-05-22T08:58:04.325003383Z',
2023-05-22T08:58:18.848757359Z                          'ExitCode': 0,
2023-05-22T08:58:18.848762812Z                          'Output': 'b\'{"version": "2.3.0"}\'\n',
2023-05-22T08:58:18.848768582Z                          'Start': '2023-05-22T08:58:04.113056594Z'},
2023-05-22T08:58:18.848774328Z                         {'End': '2023-05-22T08:58:09.534839803Z',
2023-05-22T08:58:18.848779792Z                          'ExitCode': 0,
2023-05-22T08:58:18.848785203Z                          'Output': 'b\'{"version": "2.3.0"}\'\n',
2023-05-22T08:58:18.848791125Z                          'Start': '2023-05-22T08:58:09.334486378Z'},
2023-05-22T08:58:18.848796692Z                         {'End': '2023-05-22T08:58:14.745797168Z',
2023-05-22T08:58:18.848802221Z                          'ExitCode': 0,
2023-05-22T08:58:18.848807640Z                          'Output': 'b\'{"version": "2.3.0"}\'\n',
2023-05-22T08:58:18.848813768Z                          'Start': '2023-05-22T08:58:14.539650387Z'}],
2023-05-22T08:58:18.848819581Z                 'Status': 'healthy'},
2023-05-22T08:58:18.848825038Z      'OOMKilled': False,
2023-05-22T08:58:18.848830641Z      'Paused': False,
2023-05-22T08:58:18.848836054Z      'Pid': 7415,
2023-05-22T08:58:18.848841282Z      'Restarting': False,
2023-05-22T08:58:18.848846572Z      'Running': True,
2023-05-22T08:58:18.848851928Z      'StartedAt': '2023-05-17T10:08:18.826706404Z',
2023-05-22T08:58:18.848857595Z      'Status': 'running'}

docker logs -n 100 -t jupyter-paul-20kockritz yields:

2023-05-22T09:05:06.464137818Z [D 2023-05-22 09:05:06.463 SingleUserLabApp auth:673] No user identified
2023-05-22T09:05:06.469606174Z [I 2023-05-22 09:05:06.464 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api (@192.168.160.3) 1.73ms
2023-05-22T09:05:11.676450532Z [D 2023-05-22 09:05:11.675 SingleUserLabApp auth:673] No user identified
2023-05-22T09:05:11.681674139Z [I 2023-05-22 09:05:11.676 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api (@192.168.160.3) 1.85ms
2023-05-22T09:05:16.890082495Z [D 2023-05-22 09:05:16.889 SingleUserLabApp auth:673] No user identified
2023-05-22T09:05:16.895626324Z [I 2023-05-22 09:05:16.890 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api (@192.168.160.3) 1.85ms
2023-05-22T09:05:18.277378146Z [D 2023-05-22 09:05:18.276 SingleUserLabApp auth:440] HubAuth cache miss: token:bdbb96c2d76b4439b7bd1fd1fd3f5f26:31fa6d5caa407858cac9b2d4b6cb5ebe7c6b48732525de8583025d8f20e1927c
2023-05-22T09:05:18.291755511Z [D 2023-05-22 09:05:18.291 SingleUserLabApp auth:451] Received request from Hub user {'kind': 'user', 'admin': False, 'groups': [], 'name': 'paul kockritz', 'session_id': 'bc4ab61598484102a5f171e7bf94c12b', 'scopes': ['access:servers!server=paul kockritz/', 'read:users:groups!user=paul kockritz', 'read:users:name!user=paul kockritz']}
2023-05-22T09:05:18.293123714Z [D 2023-05-22 09:05:18.291 SingleUserLabApp auth:1081] Allowing Hub user paul kockritz based on oauth scopes {'access:servers!server=paul kockritz/'}
2023-05-22T09:05:18.295604070Z [I 2023-05-22 09:05:18.295 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api/contents?content=1&1684746318990 (paul kockritz@::ffff:10.60.2.71) 20.78ms
2023-05-22T09:05:21.280954423Z [D 2023-05-22 09:05:21.279 SingleUserLabApp auth:1081] Allowing Hub user paul kockritz based on oauth scopes {'access:servers!server=paul kockritz/'}
2023-05-22T09:05:21.281024650Z [I 2023-05-22 09:05:21.280 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api/terminals?1684746321995 (paul kockritz@::ffff:10.60.2.71) 2.61ms
2023-05-22T09:05:22.099921827Z [D 2023-05-22 09:05:22.099 SingleUserLabApp auth:673] No user identified
2023-05-22T09:05:22.101174469Z [I 2023-05-22 09:05:22.100 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api (@192.168.160.3) 2.51ms
2023-05-22T09:05:27.327879521Z [D 2023-05-22 09:05:27.327 SingleUserLabApp auth:673] No user identified
2023-05-22T09:05:27.333761586Z [I 2023-05-22 09:05:27.328 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api (@192.168.160.3) 2.01ms
2023-05-22T09:05:32.525045741Z [D 2023-05-22 09:05:32.524 SingleUserLabApp auth:673] No user identified
2023-05-22T09:05:32.526696503Z [I 2023-05-22 09:05:32.525 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api (@192.168.160.3) 2.04ms
2023-05-22T09:05:37.729719248Z [D 2023-05-22 09:05:37.729 SingleUserLabApp auth:673] No user identified
2023-05-22T09:05:37.731879401Z [I 2023-05-22 09:05:37.730 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api (@192.168.160.3) 2.61ms
2023-05-22T09:05:42.938612110Z [D 2023-05-22 09:05:42.938 SingleUserLabApp auth:673] No user identified
2023-05-22T09:05:42.940766801Z [I 2023-05-22 09:05:42.939 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api (@192.168.160.3) 2.74ms
2023-05-22T09:05:48.146409804Z [D 2023-05-22 09:05:48.145 SingleUserLabApp auth:673] No user identified
2023-05-22T09:05:48.148732025Z [I 2023-05-22 09:05:48.147 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api (@192.168.160.3) 2.76ms
2023-05-22T09:05:48.687252114Z [D 2023-05-22 09:05:48.686 SingleUserLabApp mixins:553] Notifying Hub of activity 2023-05-22T08:49:04.674426Z
2023-05-22T09:05:53.361919440Z [D 2023-05-22 09:05:53.361 SingleUserLabApp auth:673] No user identified
2023-05-22T09:05:53.367641478Z [I 2023-05-22 09:05:53.362 SingleUserLabApp log:186] 200 GET /user/paul%20kockritz/api (@192.168.160.3) 1.93ms

manics · May 23, 2023, 4:36pm

NS_ERROR_UNKNOWN_HOST often indicates a DNS problem. How is the app108 hostname/DNS configured?

pa-ko-nok · May 30, 2023, 8:13pm

Are there any specific configurations that I should pay attention to?
Could this be a misconfiguration in my JupyterHub setup, or do you think it’s a network issue?

Below are some of the network settings from the two Docker containers:

jupyterhub:
root@216e590de36e:/srv/jupyterhub# env | grep -i prox
no_proxy=localhost,127.0.0.1,atg.services,atg.local
https_proxy=http://my-proxy-server:1234
NO_PROXY=localhost,127.0.0.1,atg.services,atg.local
HTTPS_PROXY=http://my-proxy-server:1234
HTTP_PROXY=http://my-proxy-server:1234
http_proxy=http://my-proxy-server:1234

root@216e590de36e:/srv/jupyterhub# env | grep -i dock
DOCKER_NETWORK_NAME=jupyterhub-network
DOCKER_NOTEBOOK_IMAGE=jupyter/minimal-notebook:latest
DOCKER_SPAWN_CMD=start-singleuser.sh
DOCKER_NOTEBOOK_DIR=/home/jovyan/work
root@216e590de36e:/srv/jupyterhub#

root@216e590de36e:/srv/jupyterhub# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.160.2  netmask 255.255.240.0  broadcast 192.168.175.255
        ether 02:42:c0:a8:a0:02  txqueuelen 0  (Ethernet)
        RX packets 52496  bytes 16701121 (16.7 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 49567  bytes 15227764 (15.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

jupyter-paul-kockritz:
jovyan@c725708a614b:~$ env | grep -i jupyter
JUPYTERHUB_SERVICE_URL=http://0.0.0.0:8888/user/paul%20kockritz/
JUPYTERHUB_API_TOKEN=1dc641897cf64b37a747a5de0f75918a
JUPYTERHUB_BASE_URL=/
JUPYTERHUB_SERVER_NAME=
JUPYTERHUB_SERVICE_PREFIX=/user/paul%20kockritz/
JUPYTERHUB_OAUTH_CALLBACK_URL=/user/paul%20kockritz/oauth_callback
JUPYTERHUB_ROOT_DIR=/home/jovyan/work
JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES=[]
JUPYTER_IMAGE_SPEC=jupyter/minimal-notebook:latest
JUPYTERHUB_OAUTH_ACCESS_SCOPES=["access:servers!server=paul kockritz/", "access:servers!user=paul kockritz"]
JUPYTERHUB_API_URL=http://192.168.160.2:8080/hub/api
JUPYTERHUB_CLIENT_ID=jupyterhub-user-paul%20kockritz
JUPYTERHUB_OAUTH_SCOPES=["access:servers!server=paul kockritz/", "access:servers!user=paul kockritz"]
JUPYTERHUB_HOST=
JUPYTERHUB_DEBUG=1
JUPYTER_PORT=8888
JUPYTERHUB_USER=paul kockritz
JUPYTERHUB_ACTIVITY_URL=http://192.168.160.2:8080/hub/api/users/paul%20kockritz/activity

jovyan@c725708a614b:~$ cat /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
192.168.160.3	c725708a614b

(base) jovyan@c725708a614b:~$ cat /etc/hostname
c725708a614b

It seems strange that this error only appears sporadically. I also see that it’s happening more often when I use firefox than when I use Chrome.

manics · May 31, 2023, 9:50am

I don’t know, but I suspect it’s to do with your Docker or system networking/DNS

Do you know how this is setup?

Topic		Replies	Views
JupyterHub with Podman and DockerSpawner: Immediate 'JupyterLab Server Unavailable' Error on Login, Resolved by Refresh JupyterHub jupyterhub , help-wanted	8	775	February 1, 2024
Server not getting started - Docker container JupyterHub	1	119	May 16, 2024
Can't login to my server jupyterhub via http JupyterHub help-wanted , docker	1	709	March 27, 2023
Spawner issue - Docker container JupyterHub help-wanted	2	219	May 14, 2024
"Error in Authenticator.pre_spawn_start: DockerException Error while fetching server API version" while starting jupyterhub with DockerSpawner JupyterHub help-wanted	3	2979	September 11, 2020

Error "Server unavailable or unreachable" - Jupyterhub running in Docker

Related topics