Looking for some help understanding Jhub external services config

JupyterHub
,
1

I’ve been trying to reach an externally managed service from Jupyterhub that runs in a couple of docker containers. I have been working with a minimal configuration (the jupyterhub-docker/basic-example repo) in my dev environment for the last few days and am unsure what is causing me to get a 503 in hub when I attempt to reach the service. The hub logs don’t really shed any light. If I look at the logs of the external docker container I don’t see any requests actually hitting the webserver, so I suspect something else needs to be configured on the Jupyterhub side but so far, I’ve been unsuccessful in finding an example or figuring it out myself. All of the containers are using the same docker network that is defined in the example compose file, and if I exec into the hub container the other service responds to pings and shows the target port as open.

I’ve tried various combinations of arguments with in the JupyterHub.services definition, though in the attached example I’ve got the simplest form. I’ve tried to set the url parameter to use the container name and trust that the docker network is going to resolve where that container is, I’ve tried setting it to both 127.0.0.1 and 0.0.0.0 and neither seem to have any impact.

My ultimate goal is to allow users to access this other service from within jhub to make it easier for them to access from a single interface, as well as ensure that if I deploy somewhere that doesn’t have a similar service available, they will have at least one option to use.

Here is the jupyterhub-config.py 
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.

# Configuration file for JupyterHub
import os

c = get_config()  # noqa: F821

# We rely on environment variables to configure JupyterHub so that we
# avoid having to rebuild the JupyterHub container every time we change a
# configuration parameter.

# Spawn single-user servers as Docker containers
c.JupyterHub.spawner_class = "dockerspawner.DockerSpawner"

# prevent jumping directly into singleuser server
c.JupyterHub.redirect_to_server = False

# not sure if this is necessary, but...jic
c.JupyterHub.allow_named_servers = True

c.JupyterHub.services = [
    {
        "name": "regexr-service",
        "url": "http://regexr-phpapi:9000",
    }
]

# Spawn containers from this image
c.DockerSpawner.image = os.environ["DOCKER_NOTEBOOK_IMAGE"]

# Connect containers to this Docker network
network_name = os.environ["DOCKER_NETWORK_NAME"]
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.network_name = network_name
c.DockerSpawner.ip = "0.0.0.0"

# Explicitly set notebook directory because we'll be mounting a volume to it.
# Most `jupyter/docker-stacks` *-notebook images run the Notebook server as
# user `jovyan`, and set the notebook directory to `/home/jovyan/work`.
# We follow the same convention.
notebook_dir = os.environ.get("DOCKER_NOTEBOOK_DIR", "/home/jovyan/work")
c.DockerSpawner.notebook_dir = notebook_dir

# Mount the real user's Docker volume on the host to the notebook user's
# notebook directory in the container
c.DockerSpawner.volumes = {"jupyterhub-user-{username}": notebook_dir}

# Remove containers once they are stopped
c.DockerSpawner.remove = True

# For debugging arguments passed to spawned containers
c.DockerSpawner.debug = True

# User containers will access hub by container name on the Docker network
c.JupyterHub.hub_ip = "jhub"
c.JupyterHub.hub_port = 8080

# Persist hub data on volume mounted inside container
c.JupyterHub.cookie_secret_file = "/data/jupyterhub_cookie_secret"
c.JupyterHub.db_url = "sqlite:////data/jupyterhub.sqlite"

# Allow all signed-up users to login
c.Authenticator.allow_all = True

# Authenticate users with Native Authenticator
c.JupyterHub.authenticator_class = "dummy"

# Allowed admins
admin = os.environ.get("JUPYTERHUB_ADMIN")
if admin:
    c.Authenticator.admin_users = [admin]
Here is the docker-compose.yml 
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.

# JupyterHub docker compose configuration file
version: "3"

services:
  hub:
    build:
      context: .
      dockerfile: Dockerfile.jupyterhub
      args:
        JUPYTERHUB_VERSION: latest
    restart: always
    image: jupyterhub
    container_name: jhub
    networks:
      - jupyterhub-network
    volumes:
      # The JupyterHub configuration file
      - "./jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py:ro"
      # Bind Docker socket on the host so we can connect to the daemon from
      # within the container
      - "/var/run/docker.sock:/var/run/docker.sock:rw"
      # Bind Docker volume on host for JupyterHub database and cookie secrets
      - "jupyterhub-data:/data"
    ports:
      - "8000:8000"
    environment:
      # This username will be a JupyterHub admin
      JUPYTERHUB_ADMIN: admin
      # All containers will join this network
      DOCKER_NETWORK_NAME: jupyterhub-network
      # JupyterHub will spawn this Notebook image for users
      DOCKER_NOTEBOOK_IMAGE: quay.io/jupyter/base-notebook:latest
      # Notebook directory inside user image
      DOCKER_NOTEBOOK_DIR: /home/jovyan/work
  regexr:
    image: dstack4273/regexr-app:0.1
    container_name: regexr-app
    networks:
      - jupyterhub-network
    depends_on:
      - phpapi
  phpapi:
    image: dstack4273/regexr-phpapi:0.1
    container_name: regexr-phpapi
    networks:
      - jupyterhub-network
    ports:
      - "9000:80"

volumes:
  jupyterhub-data:

networks:
  jupyterhub-network:
    name: jupyterhub-network
Here is the repo’s Dockerfile dependency for the compose file 
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
ARG JUPYTERHUB_VERSION
FROM quay.io/jupyterhub/jupyterhub:$JUPYTERHUB_VERSION

# Install dockerspawner, nativeauthenticator
# hadolint ignore=DL3013
RUN python3 -m pip install --no-cache-dir \
    dockerspawner

CMD ["jupyterhub", "-f", "/srv/jupyterhub/jupyterhub_config.py"]

The external service isn’t one I understand well so that could possibly be part of the problem?

If this is helpful, here are the logs from the hub 

(base) dstack@ROGtop:~/projects/docker_jhub$ docker logs jhub
[I 2025-12-01 21:15:25.556 JupyterHub app:3359] Running JupyterHub version 5.4.2
[I 2025-12-01 21:15:25.556 JupyterHub app:3389] Using Authenticator: jupyterhub.auth.DummyAuthenticator-5.4.2
[I 2025-12-01 21:15:25.556 JupyterHub app:3389] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-14.0.0
[I 2025-12-01 21:15:25.556 JupyterHub app:3389] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.4.2
/srv/venv/lib/python3.12/site-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://schema.jupyter.org/jupyterhub/events/server-action
  validate_schema(_schema)
[I 2025-12-01 21:15:25.567 JupyterHub app:1880] Writing cookie_secret to /data/jupyterhub_cookie_secret
[I 2025-12-01 21:15:25.600 alembic.runtime.migration migration:211] Context impl SQLiteImpl.
[I 2025-12-01 21:15:25.600 alembic.runtime.migration migration:214] Will assume non-transactional DDL.
[I 2025-12-01 21:15:25.613 alembic.runtime.migration migration:622] Running stamp_revision  -> 4621fec11365
[I 2025-12-01 21:15:25.745 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2025-12-01 21:15:25.755 JupyterHub auth:1541] Using testing authenticator DummyAuthenticator! This is not meant for production!
[I 2025-12-01 21:15:25.760 JupyterHub roles:281] Adding role admin for User: admin
[I 2025-12-01 21:15:25.765 JupyterHub roles:281] Adding role user for User: admin
[I 2025-12-01 21:15:25.785 JupyterHub app:2893] Creating service regexr-service with oauth_client_id=service-regexr-service
[I 2025-12-01 21:15:25.788 JupyterHub provider:661] Creating oauth client service-regexr-service
[I 2025-12-01 21:15:25.827 JupyterHub app:3429] Initialized 0 spawners in 0.004 seconds
[I 2025-12-01 21:15:25.831 JupyterHub metrics:425] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-12-01 21:15:25.831 JupyterHub metrics:425] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2025-12-01 21:15:25.832 JupyterHub metrics:425] Found 0 active users in the last ActiveUserPeriods.thirty_days
[W 2025-12-01 21:15:25.832 JupyterHub proxy:748] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2025-12-01 21:15:25.832 JupyterHub proxy:752] Starting proxy @ http://:8000
2025-12-01T21:15:26.123Z [ConfigProxy] info: Proxying http://*:8000 to (no default)
2025-12-01T21:15:26.124Z [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
2025-12-01T21:15:26.326Z [ConfigProxy] info: 200 GET /api/routes
[I 2025-12-01 21:15:26.328 JupyterHub app:3752] Hub API listening on http://jhub:8080/hub/
[I 2025-12-01 21:15:26.329 JupyterHub app:3637] Adding external service regexr-service at http://regexr-phpapi:9000
[W 2025-12-01 21:15:27.267 JupyterHub app:3666] Cannot connect to external service regexr-service at http://regexr-phpapi:9000. Is it running?
2025-12-01T21:15:27.269Z [ConfigProxy] info: 200 GET /api/routes
[I 2025-12-01 21:15:27.270 JupyterHub proxy:477] Adding route for Hub: / => http://jhub:8080
[W 2025-12-01 21:15:27.270 JupyterHub proxy:445] Adding missing route for regexr-service (Server(url=http://regexr-phpapi:9000/services/regexr-service/, bind_url=http://regexr-phpapi:9000/services/regexr-service/))
[I 2025-12-01 21:15:27.271 JupyterHub proxy:312] Adding service regexr-service to proxy /services/regexr-service/ => http://regexr-phpapi:9000
2025-12-01T21:15:27.272Z [ConfigProxy] info: Adding route / -> http://jhub:8080
2025-12-01T21:15:27.273Z [ConfigProxy] info: Route added / -> http://jhub:8080
2025-12-01T21:15:27.274Z [ConfigProxy] info: 201 POST /api/routes/
2025-12-01T21:15:27.275Z [ConfigProxy] info: Adding route /services/regexr-service -> http://regexr-phpapi:9000
2025-12-01T21:15:27.275Z [ConfigProxy] info: Route added /services/regexr-service -> http://regexr-phpapi:9000
2025-12-01T21:15:27.276Z [ConfigProxy] info: 201 POST /api/routes/services/regexr-service
[I 2025-12-01 21:15:27.276 JupyterHub app:3783] JupyterHub is now running at http://:8000
[I 2025-12-01 21:16:22.292 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:172.19.0.1) 1.20ms
[I 2025-12-01 21:16:22.305 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:172.19.0.1) 0.90ms
[I 2025-12-01 21:16:22.315 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'None:cX9IBTtHcamkkkcTy21ebGML6YFEJa9R8YazI0lHNj8=' {'path': '/hub/', 'max_age': 3600}
[I 2025-12-01 21:16:22.315 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'None:cX9IBTtHcamkkkcTy21ebGML6YFEJa9R8YazI0lHNj8=' {'path': '/hub/', 'max_age': 3600}
[I 2025-12-01 21:16:22.336 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:172.19.0.1) 22.72ms
[W 2025-12-01 21:16:23.355 JupyterHub app:2967] Cannot connect to external service regexr-service at http://regexr-phpapi:9000
[I 2025-12-01 21:16:31.073 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'cX9IBTtHcamkkkcTy21ebGML6YFEJa9R8YazI0lHNj8=:8a8f5052eba84630ac2e3d5e529fb8cf' {'path': '/hub/'}
[I 2025-12-01 21:16:31.073 JupyterHub base:973] User logged in: admin
[I 2025-12-01 21:16:31.074 JupyterHub log:192] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (admin@::ffff:172.19.0.1) 13.91ms
[I 2025-12-01 21:16:31.090 JupyterHub log:192] 302 GET /hub/ -> /hub/home (admin@::ffff:172.19.0.1) 8.60ms
[I 2025-12-01 21:16:31.102 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'6faeb9d79ccb44bea9c3afda79814d6e:8a8f5052eba84630ac2e3d5e529fb8cf' {'path': '/hub/'}
[I 2025-12-01 21:16:31.108 JupyterHub log:192] 200 GET /hub/home (admin@::ffff:172.19.0.1) 10.52ms
2025-12-01T21:16:31.927Z [ConfigProxy] error: 503 GET /services/regexr-service/ read ECONNRESET
[I 2025-12-01 21:16:31.929 JupyterHub _xsrf_utils:130] Setting new xsrf cookie for b'None:ID3Rg_K7ouo9ue3vQfuXGV-PP0Iso9jSL3qisogg9cg=' {'path': '/hub/', 'max_age': 3600}
[I 2025-12-01 21:16:31.933 JupyterHub log:192] 200 GET /hub/error/503?url=%2Fservices%2Fregexr-service%2F (@172.19.0.2) 4.88ms
[W 2025-12-01 21:17:23.405 JupyterHub app:2967] Cannot connect to external service regexr-service at http://regexr-phpapi:9000
[W 2025-12-01 21:18:23.458 JupyterHub app:2967] Cannot connect to external service regexr-service at http://regexr-phpapi:9000
2

According to the Docker Compose networking docs:

By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by the service’s name.

Thus, your service definition in the JupyterHub config should probably address http://phpapi:9000, i.e., the service name instead of the container name?

1 Like
3

Drats, I definitely forgot to mention that I had tried to use the service name directly as well as the container name, apologies about that. I also tried to add the network name as a tld of sorts along with the container name (I saw this somewhere in my scouring of various places on the internet and though it does indeed get resolved by some docker magic) which also didn’t work. The result is the same where the hub returns an unreachable and the target service never receives anything from the hub.

I also wondered if there were maybe a start order sort of issue going on and maybe the hub attempts to resolve the url and if it can’t the service is just functionally unreachable until the hub restarts. So I tore down the hub, removed the container, deleted the docker volume for the old image, and rebuilt/restarted the hub and still no dice. Maybe the services endpoint doesn’t just allow for any ol website to be pointed at? This would all make more sense if auth were involved but it’s just plain ol insecure http no auth at all on the service’s side :man_shrugging:

4

If your PHPAPI container is running the service on port 80, you will need to use https://regexr-phpapi:80 for the service URL. The port 9000 is used from the host to reach the container but within the docker network you will need to use the port that container uses inside!!

3 Likes
5

Holy smokes! This is exactly what I had done wrong and needed help with, the service didn’t connect but I got logs. Thank you so much for your help!

2 Likes