Swarmspawner can't access DNS in swarm nodes

JupyterHub
1

I have a swarmspawner. but I can’t access my API using its DNS when I run it at jupyter lab. For example, I want to access my API with domain name Its return failed. but if I hit it with an actual IP address, it works.

http://172.20.3.57:1340/mlflow-uri?username=user_1654163873_1

expected:
http://pip3devsmlmstdbs.labs247.com:1340/mlflow-uri?username=user_1654163873_1

My question is: Is there any way or configuration to make my spawn container recognize other hosts using its DNS like edit /etc/hosts?

2

I think this needs to be fixed at your container networking level and not on JupyterHub side. I dont have experience with swarm mode, but generally your containers should resolve each other and if there dont, then better fix it on your container network

1 Like
3

Yes, you are right. but maybe there is a config for it, which I don’t know because I’m new to jupyterhub config.

I try edit the container /etc/hosts but failed cause the user of container is jovyan and I don’t have the password for sudo.

I try other ways, like:

network_name = os.environ['DOCKER_NETWORK_NAME']
c.SwarmSpawner.network_name = network_name
c.SwarmSpawner.extra_host_config = {
    'network_mode': network_name,
    'extra_hosts': {
        "vm02gpu.labs247.com": "172.20.3.60",
        "pip3devsmlmstdbs.labs247.com": "172.20.3.57",
        "pip3devsmlwrk1dbs.labs247.com": "172.20.3.58",
        "pip3devsmlwrk2dbs.labs247.com": "172.20.3.59"
    }
}

but still not work; maybe the way I use it is wrong.

4

I dont think this will work in production as you will not know the IP addresses of your container before the deployment.

Let’s forget about JupyterHub for the moment. Just start two simple containers, attach them to same container network and attempt to resolve one container from another by their name. From my understanding, it does not work for your at the moment. Try to find the cause and fix it on your Docker/Swarm deployment. If you can fix it on your Docker deployment, JupyterHub will work out of the box. You will not need any extra config.

1 Like
5

You shouldn’t try to point your browser directly to your worker nodes / containers. All the traffic goes through the proxy container on your manager node.

6

okay I’ll keep that in mind.

maybe I will provide my Dockerfile, docker-compose.yml and jupyterhub_config.py to correct if I make wrong config. if that clear than I will focus at the docker side.

Dockerfile

# base image: jupyterhub
# this is built by docker-compose
# from the root of this repo
ARG JUPYTERHUB_VERSION=3
FROM jupyterhub/jupyterhub:${JUPYTERHUB_VERSION}

USER root

# install dockerspawner from the current repo
ADD . /tmp/dockerspawner

RUN pip install --upgrade pip
#RUN pip install --no-cache /tmp/dockerspawner
RUN pip install --no-cache dockerspawner
#RUN pip install --no-cache jupyterhub==2.2
RUN pip install --no-cache jupyterhub-idle-culler
RUN pip install --no-cache oauthenticator

# load example configuration
ADD jupyterhub_config.py /srv/jupyterhub/jupyterhub_config.py
ADD cull_idle_servers.py /srv/jupyterhub/cull_idle_servers.py

docker-compose.yml

version: "3"
services:
  #proxy:
    #env_file: .env
    #image: jupyterhub/configurable-http-proxy:4
    #networks:
    #  - jupyterhub-net
    # expose the proxy to the world
    #ports:
    #  - "8333:8000"
    #command:
    #  - configurable-http-proxy
    #  - "--error-target"
    #  - "http://hub/hub/error"

  hub:
    # build an image with SwarmSpawner and our jupyterhub_config.py
    env_file: .env
    #build:
    #  context: "../.."
    #  dockerfile: "examples/swarm/Dockerfile"
    image: myjupyterhub:0.1
    # mount the docker socket
    ports:
      - "8333:8000"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/home/apps:/home/apps:ro"
      - "/mnt/nfs/jupyterhub/volumes:/mnt/nfs/jupyterhub/volumes"
      - "/etc/hosts:/etc/hosts:ro"
    environment: 
      - OAUTH_TLS_VERIFY=0
      - DOCKER_NETWORK_NAME=jupyterhub_network
    networks:
      - jupyterhub-net
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: on-failure
      
networks:
  jupyterhub-net:
    driver: overlay
    attachable: true
    name: jupyterhub_network

jupyterhub_config:

# Configuration file for jupyterhub.
import os
import sys

c = get_config()  #noqa

# The proxy is in another container
#c.ConfigurableHTTPProxy.should_start = False
#c.ConfigurableHTTPProxy.api_url = 'http://proxy:8001'

from dockerspawner import DockerSpawner
from dockerspawner import SwarmSpawner

class CustomSwarmSpawner(SwarmSpawner):
    """
    Custom SwarmSpawner class
    """    
    def _options_form_default(self):
        cpu_options = [1, 2]  # Possible values for CPUs
        memory_options = [1, 2, 4]  # Possible values for memory in GB
        form_template = """
        <div class="form-group">
            <label for="stack">Choose an image</label>
            <select id="stacks" class="form-control" name="stack">
                <option value="jupyter/base-notebook:2023-04-27">base-notebook</option>
                <option value="jupyter/scipy-notebook:2023-01-30">scipy-notebook</option>
            </select>
        </div>
        <label for="cpu_limit">CPUs:</label>
        <select class="form-control" name="cpu_limit" id="cpu_limit">
            {cpu_options}
        </select>
        <label for="memory_limit">Memory (GiB):</label>
        <select class="form-control" name="memory_limit" id="memory_limit">
            {memory_options}
        </select>
        """
        # Populate the CPU options
        cpu_select = "\n".join([f"<option>{cpu}</option>" for cpu in cpu_options])
        # Populate the memory options
        memory_select = "\n".join([f"<option>{memory}</option>" for memory in memory_options])

        # Render the final form
        options_form = form_template.format(cpu_options=cpu_select, memory_options=memory_select)
        return options_form

    def options_from_form(self, formdata):
        #global worker_hostname

        """Override to parse form submission"""
        options = {}
        # Extract selected Docker image, node, and wether to launch JupyterLab or simple notebook
        options['stack'] = formdata.get('stack', [''])[0].strip()
        options['mem_limit'] = formdata.get('memory_limit', [''])[0].strip() + "G"
        options['cpu_limit'] = int(formdata.get('cpu_limit', [''])[0].strip())
        self.image = options['stack']
        ## TODO: check availability of resources
        self.mem_limit = options['mem_limit']
        self.cpu_limit = options['cpu_limit']
        return options
        
def create_dir_hook(SwarmSpawner):
    username = SwarmSpawner.user.name  # get the username
    volume_path = os.path.join('/mnt/nfs/jupyterhub/volumes', username)
    if not os.path.exists(volume_path):
        os.mkdir(volume_path, 0o755)
    mounts_user = [
        {'type': 'bind',
         'source': volume_path,
         'target': '/home/jovyan/work', }
    ]
    SwarmSpawner.extra_container_spec = {
        'mounts': mounts_user
    }

from oauthenticator.generic import GenericOAuthenticator
c.JupyterHub.authenticator_class = GenericOAuthenticator
c.GenericOAuthenticator.client_id = 'jupyter'
c.GenericOAuthenticator.client_secret = 'secret'
c.GenericOAuthenticator.token_url = 'https://{url}/realms/yava-ai/protocol/openid-connect/token'
c.GenericOAuthenticator.userdata_url = 'https://{url}/realms/yava-ai/protocol/openid-connect/userinfo'
c.GenericOAuthenticator.userdata_params = {'state': 'state'}
c.GenericOAuthenticator.username_claim = 'preferred_username'
c.GenericOAuthenticator.login_service = 'Keycloak'
c.GenericOAuthenticator.scope = ['openid', 'profile']
c.GenericOAuthenticator.oauth_callback_url = 'https://{url}/hub/oauth_callback'
c.LocalAuthenticator.create_system_users = True
c.JupyterHub.authenticator_class.login_handler._OAUTH_AUTHORIZE_URL = 'https://{url}/realms/yava-ai/protocol/openid-connect/auth'

c.GenericOAuthenticator.allow_all = True
c.GenericOAuthenticator.allow_existing_users = True
c.Authenticator.auto_login = True

c.JupyterHub.ssl_key = '/home/apps/keycloak.key'
c.JupyterHub.ssl_cert = '/home/apps/keycloak.crt'

# use SwarmSpawner
#c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'  
c.JupyterHub.spawner_class = CustomSwarmSpawner

c.Spawner.pre_spawn_hook = create_dir_hook

# The Hub should listen on all interfaces,
# so user servers can connect
c.JupyterHub.hub_ip = '0.0.0.0'

# this is the name of the 'service' in docker-compose.yml
c.JupyterHub.hub_connect_ip = 'hub'

# this is the network name for jupyterhub in docker-compose.yml
# with a leading 'swarm_' that docker-compose adds
# network_name = 'swarm_jupyterhub-net'
network_name = os.environ['DOCKER_NETWORK_NAME']
c.SwarmSpawner.network_name = network_name
c.SwarmSpawner.extra_host_config = {
    'network_mode': network_name,
    'extra_hosts': {
        "vm02gpu.labs247.com": "172.20.3.60",
        "pip3devsmlmstdbs.labs247.com": "172.20.3.57",
        "pip3devsmlwrk1dbs.labs247.com": "172.20.3.58",
        "pip3devsmlwrk2dbs.labs247.com": "172.20.3.59"
    }
}

# Explicitly set notebook directory because we'll be mounting a host volume to
# it.  Most jupyter/docker-stacks *-notebook images run the Notebook server as
# user `jovyan`, and set the notebook directory to `/home/jovyan/work`.
# We follow the same convention.
notebook_dir = '/home/jovyan/work'
c.SwarmSpawner.notebook_dir = notebook_dir

'''
c.DockerSpawner.extra_create_kwargs = {
    "user": "root"
}
c.SwarmSpawner.environment = {
  'GRANT_SUDO': '1',
  'UID': '0', # workaround https://github.com/jupyter/docker-stacks/pull/420
}
'''
c.Spawner.environment = {}
c.Spawner.environment.update(dict(
    GRANT_SUDO=1,
    UID=0))

# increase launch timeout because initial image pulls can take a while
c.SwarmSpawner.http_timeout = 300
c.SwarmSpawner.start_timeout = 300

c.SwarmSpawner.extra_placement_spec = { 'constraints' : ['node.role==worker'] }

c.SwarmSpawner.remove_containers = True
c.ResourceUseDisplay.track_cpu_percent = True

# start jupyterlab
c.Spawner.cmd = ["jupyter", "labhub"]

# debug-logging for testing
import logging

c.SwarmSpawner.debug = True

c.JupyterHub.log_level = logging.DEBUG

c.JupyterHub.load_roles = [
    {
        "name": "cull-idle-role",
        "scopes": [
            "list:users",
            "read:users:activity",
            "read:servers",
            "read:metrics",
            "delete:servers",
            "access:servers"
            # "admin:users", # if using --cull-users
        ],
        # assignment of role's permissions to:
        "services": ["cull-idle"],
        "users": ['demo'],
    }
]

c.JupyterHub.services = [
    {
        'name': 'cull-idle',
        'command': [sys.executable, 'cull_idle_servers.py', '--timeout=3600'],
    }
]

how to start the service:

docker-compose build -t myjupyterhub:0.1 .
docker-compose up --remove-orphans

I need your help to correct my config. btw thanks for the reply. @mahendrapaipuri and @markperri