Configurable-http-proxy is it needed if I using nginx


I setting up jupyterhub with nginx/letsencrypt/certbot as my https reverse proxy server, but configurable-http-proxy was also installed with jupyterhub. configurable-http-proxy is running, but it doesn’t stop me from reach the hub and logging in. If I don’t need it how do I stop it from executing when the server starts?

Yes, CHP is needed to handle the routing of requests to individual servers. Without it, you cannot access user servers.

So then even if I am running a nginx as my reverse proxy in separate docker container I still need something like the two lines in my file running on the jupyterhub container?

c.ConfigurableHTTPProxy.should_start = True
c.ConfigurableHTTPProxy.api_url = 'https://proxy:8001'

thanks in advance

You typically don’t need any configuration related to CHP unless you are running it separately from the Hub as its own service, in which case you would be setting should_start = False.

Hi @minrk thanks for the reply. So I am trying to track down the reason why my jupyterhub is not spawning servers and I want to rule out CHP. When the jhub first comes up this is the log out put

 docker logs 9e0 -f
[I 2019-05-23 02:09:16.879 JupyterHub app:2120] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.0.0
[I 2019-05-23 02:09:16.879 JupyterHub app:2120] Using Spawner: dockerspawner.swarmspawner.SwarmSpawner-0.11.1
[D 2019-05-23 02:09:16.885 JupyterHub app:1297] Generating new cookie_secret
[I 2019-05-23 02:09:16.886 JupyterHub app:1302] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
[D 2019-05-23 02:09:16.886 JupyterHub app:1424] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2019-05-23 02:09:16.904 JupyterHub orm:718] Stamping empty database with alembic revision 4dc2d5a8c53c
[I 2019-05-23 02:09:16.908 alembic.runtime.migration migration:130] Context impl SQLiteImpl.
[I 2019-05-23 02:09:16.909 alembic.runtime.migration migration:137] Will assume non-transactional DDL.
[I 2019-05-23 02:09:16.937 alembic.runtime.migration migration:356] Running stamp_revision  -> 4dc2d5a8c53c
[D 2019-05-23 02:09:16.937 alembic.runtime.migration migration:558] new branch insert 4dc2d5a8c53c
[I 2019-05-23 02:09:17.106 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2019-05-23 02:09:17.196 JupyterHub app:1910] Loading state for alvin from db
[D 2019-05-23 02:09:17.196 JupyterHub app:1926] Loaded users:
       alvin admin
[I 2019-05-23 02:09:17.211 JupyterHub app:2337] Hub API listening on
[I 2019-05-23 02:09:17.211 JupyterHub app:2339] Private Hub API connect url http://jupyterhubserver:8000/hub/
[W 2019-05-23 02:09:17.213 JupyterHub proxy:642] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2019-05-23 02:09:17.213 JupyterHub proxy:645] Starting proxy @ http://:8000
[D 2019-05-23 02:09:17.213 JupyterHub proxy:646] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '', '--api-port', '8001', '--error-target', 'http://jupyterhubserver:8000/hub/error']
[D 2019-05-23 02:09:17.220 JupyterHub proxy:561] Writing proxy pid file:
02:09:17.989 - info: [ConfigProxy] Proxying http://*:8000 to (no default)
02:09:17.992 - info: [ConfigProxy] Proxy API at
02:09:17.993 - error: [ConfigProxy] Uncaught Exception Error: listen EADDRINUSE: address already in use :::8000
    at Server.setupListenHandle [as _listen2] (net.js:1259:14)
    at listenInCluster (net.js:1307:12)
    at Server.listen (net.js:1395:7)
    at Object.<anonymous> (/opt/conda/lib/node_modules/configurable-http-proxy/bin/configurable-http-proxy:202:20)
    at Module._compile (internal/modules/cjs/loader.js:816:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:827:10)
    at Module.load (internal/modules/cjs/loader.js:685:32)
    at Function.Module._load (internal/modules/cjs/loader.js:620:12)
    at Function.Module.runMain (internal/modules/cjs/loader.js:877:12)
    at internal/main/run_main_module.js:21:11
[D 2019-05-23 02:09:18.204 JupyterHub proxy:681] Proxy started and appears to be up
[D 2019-05-23 02:09:18.205 JupyterHub proxy:314] Fetching routes to check
[D 2019-05-23 02:09:18.208 JupyterHub proxy:765] Proxy: Fetching GET
[I 2019-05-23 02:09:18.227 JupyterHub proxy:319] Checking routes
[I 2019-05-23 02:09:18.227 JupyterHub proxy:399] Adding default route for Hub: / => http://jupyterhubserver:8000
[D 2019-05-23 02:09:18.228 JupyterHub proxy:765] Proxy: Fetching POST
[I 2019-05-23 02:09:18.232 JupyterHub app:2422] JupyterHub is now running at http://:8000

Is the one error : 02:09:17.993 - error: [ConfigProxy] Uncaught Exception Error: listen EADDRINUSE: address already in use :::8000 an issue

this is the output from the container:

 root@jupyterhubserver:/srv/jupyterhub# netstat -lep
 Active Internet connections (only servers)
 Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
 tcp        0      0  *               LISTEN      root       23235855   1/python
 tcp        0      0 localhost:8001*               LISTEN      root       23229864   12/node
 tcp        0      0*               LISTEN      root       23228198   -
 udp        0      0*                           root       23228197   -
 Active UNIX domain sockets (only servers)
 Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path

thanks in advance for any insights

The problem is that your jhub process is already listening on port 8000 and the CHP is trying to use that port as well.

Could you post a minimal config file, container image name and steps to reproduce this? That might help others figure out why both are trying to listen on the same port.

container image jupyterhub/jupyterhub-onbuild:latest (this will include my JH_config, which is in the same folder with the Dockerfile)

snippet from JHub_config

    ## The public facing port of the proxy
    c.JupyterHub.hub_port = 8000
    ## The public facing ip of the whole application (the proxy)
    c.JupyterHub.hub_ip = ''
    ## The ip for this process
    c.JupyterHub.hub_connect_ip = 'jupyterhubserver'
    ## revers proxy setting for nginx
    #c.JupyterHub.base_url = '/hub/'
    #  Defaults to an empty set, in which case no user has admin access.

    c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'
    ##possibly delete
    #c.SwarmSpawner.jupyterhub_service_name = "jupyterhubserver"

    network_name = os.environ['DOCKER_NETWORK_NAME']
    c.SwarmSpawner.network_name = network_name
    c.SwarmSpawner.use_internal_ip = True
    # Pass the network name as argument to spawned containers
    c.SwarmSpawner.extra_host_config = {'network_mode': network_name}
    c.SwarmSpawner.host_ip = ''
    notebook_dir = os.environ.get('NOTEBOOK_DIR') or '/home/jovyan/work'
    c.SwarmSpawner.notebook_dir = notebook_dir
    # Mount the real user's Docker volume on the host to the notebook user's
    # notebook directory in the container
    c.SwarmSpawner.volumes = { 'jupyterhub-user-{username}': notebook_dir }
    # Remove containers once they are stopped
    c.SwarmSpawner.remove_containers = True
    # For debugging arguments passed to spawned containers
    c.SwarmSpawner.debug = True

I started this project with an older guide and I have been just modifying the config file as I figuring my way through this.

would this be simple way of addressing the issue: c.ConfigurableHTTPProxy.api_url = 'http://proxy:8001'?

I have three steps (I use to do this with a series of “docker create xxx” and shell scripts

docker build . -t newimagename:tag
docker push newimagename:tag
docker stack deploy --prune -c docker-compose.yml Hub

the yaml file if it is helpful

version: '3.5'

    image: sgsupdocker/jupyterhub-onbuild:052219.2
    hostname: jupyterhubserver
      - /var/run/docker.sock:/var/run/docker.sock
      - nfsvolume:/home/jovyan/work
          - jupyterhubserver
      DOCKER_NETWORK_NAME: jupyterhub
      NOTEBOOK_DIR: /home/jovyan/work
      replicas: 1
          - node.role == manager
      - 8000:8000
    image: linuxserver/letsencrypt:latest
    hostname: nginx
            - nginx
      - 443:443
      - 80:80
      PUID: 1000
      PGID: 1000
      TZ: US/Arizona
      SUBDOMAINS: -wildcard
      VALIDATION: duckdns
      DUCKDNSTOKEN: redacted
      - /etc/jupyterhub/letsencrypt_container_nginx.conf:/config/nginx/site-confs/default
      - nginx_volume:/config

    driver: overlay
    attachable: true
    name: jupyterhub```

thanks for the insights
would it make sense to just delete by current file and just start over?  If so can shoot me link to a good starter config file using JHUB on docker swarm. I have seen a few, but a recommendation would be nice

should it be the case that the nginx-proxy catches all incoming request from the world and then hands that to CHP which intern hands that to JHUB? The JHUB will spawn a notebook server and the internal network coms between JHUB and servers will be handled by CHP?

1 Like

Yes. Nothing should be talking directly to the Hub process. Bypassing the proxy causes requests to be routed to the wrong place. The way to view jupyterhub from outside as a multi-process application whose public face (the only one anything outside jupyterhub should connect to) is configurable-http-proxy. The Hub in this case is a private internal implementation detail.

This is where there’s a misunderstanding:

## The public facing port of the proxy
c.JupyterHub.hub_port = 8000
## The public facing ip of the whole application (the proxy)
c.JupyterHub.hub_ip = ''

Those are not the public-facing ip and port. Those are the ip and port of the Hub process behind the proxy. They also happen to be the default values for the proxy’s public interface, so this configuration has instructed the hub and proxy to both be listening on *:8000.

I suspect what you want is to still set c.JupyterHub.hub_ip = '' (required to allow servers to connect directly to the hub), but not set c.JupyerHub.hub_port, or at least set it to a value that’s not already assigned. The Hub’s private port does not need to be exposed publicly; it will be accessible to other containers on the docker network.

Thanks for the reply.

Does this also mean that there is no need to either “Expose” or “define” ports 8000 or 8001? in either my docker or YAML file?

Generally, yes. Ports do not need to be exposed to be accessed from within a docker network. Expose is only necessary for access from outside, which is only one port for all of JupyterHub: the public port of CHP (or nginx in your case).

thanks! I just commented out the c.Jupyterhub.hub_port and it worked.