Why does JupyterHub not see the docker network I have created?

General
1

I am trying to run JupyterHub as a Docker container and also use the DockerSpawner for the lab containers. I have created the Docker network ccc-labs and start the JupyterHub process. However after authentication, when it tries to spawn the Lab container, I get the following message:

"Unknown docker network 'ccc-labs'. Did you create it with 'docker network create <name>'?"

This is the list of Docker networks on the machine:

NETWORK ID          NAME                DRIVER              SCOPE
f2999de5f3db        bridge              bridge              local
d94f451e0cc8        ccc-labs            bridge              local
d5981a63c0ff        ccc-network         bridge              local
84bf936fd1d7        host                host                local
378f80608656        kind                bridge              local
6d9e18651044        none                null                local

My jupyterhub_config.py is as follows:

# Configure Docker Spawner to launch an instance for the logged in user
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'

notebook_dir = '/lab'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = {
    '/data/{username}': notebook_dir
}

# Specify the image to use when spawning
c.DockerSpawner.image = "jupyter/scipy-notebook"

c.Spawner.debug = True

# Specify a timeout for starting the image
c.DockerSpawner.start_timeout = 600

# Configure authentication with Auth0
from oauthenticator.auth0 import Auth0OAuthenticator
c.JupyterHub.authenticator_class = Auth0OAuthenticator
c.Auth0OAuthenticator.auth0_subdomain = "turtlesystems.eu"
c.Auth0OAuthenticator.scope = ['openid', 'profile', 'email']

# the Hub's API listens on localhost by default,
# but docker containers can't see that.
# Tell the Hub to listen on its docker network:
import netifaces
docker0 = netifaces.ifaddresses('eth0')
docker0_ipv4 = docker0[netifaces.AF_INET][0]

c.JupyterHub.hub_ip = '0.0.0.0' #docker0_ipv4['addr']
c.JupyterHub.hub_connect_ip = docker0_ipv4['addr']

network_name = "ccc-labs"
c.DockerSpawmer.use_internal_ip = True
c.DockerSpawner.network_name = network_name
c.DockerSpawner.extra_host_config = { 'network_mode': network_name }

And I start my JupyterHub container with the following (NOTE: I have removed the OAuth environment variables)

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v ${PWD}/local:/etc/jupyterhub -p 8000:8000 --name jupyterhub -e OAUTH_CLIENT_ID="xxx" -e OAUTH_CLIENT_SECRET="xxx" -e OAUTH_CALLBACK_URL="http://192.168.50.137:8000/hub/oauth_callback" --network=ccc-labs russellseymour/jupyterhub:0.0.12

The startup process looks promising, but then I get an error stating that the ccc-labs network cannot be found and has it been created.

[E 2020-07-19 13:45:10.761 JupyterHub user:692] Unhandled error starting russell.seymour@turtlesystems.co.uk's server: Unknown docker network 'ccc-labs'. Did you create it with `docker network create <name>`?

As I can see the networks on the host and I am passing in the network name using --network=ccc-labs what am I missing here? My initial thoughts are that the new container that is being spawned is the one that cannot see the Docker network as I have explicitly told the main JupyterLab about it. Is there an extra option I should be setting when I run the JupyterLab container or more options in the configuration file.

Any advice is gratefully received as I think I am almost there. I apologise in advance if I am missing something simple off the Docker run command and this does not have anything directly to do with JupyterHub.

Thanks, Russell

2

Some more information.

I had originally tried to do this without setting a Docker network. Then I could see that a container was starting up, but the JupyterHub process would fail with this message:

TimeoutError: Server at http://127.0.0.1:32778/user/russell.seymour@turtlesystems.co.uk/ didn't respond in 30 seconds

After searching around I stumbled across the idea of using a Docker network to spawn the container so that the main hub would be able to see it; hence the issue I have raised.

3

Hi! Please could you show us the full set of logs from startup until the error? It might also be helpful to enable debug logging: https://github.com/jupyterhub/jupyterhub/wiki/Debug-Jupyterhub

4

@manics Yes of course, here are the debug logs:

[I 2020-07-19 18:54:02.175 JupyterHub app:2307] Running JupyterHub version 1.2.0dev
[I 2020-07-19 18:54:02.178 JupyterHub app:2338] Using Authenticator: oauthenticator.auth0.Auth0OAuthenticator-0.11.0
[I 2020-07-19 18:54:02.179 JupyterHub app:2338] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-0.11.1
[I 2020-07-19 18:54:02.180 JupyterHub app:2338] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.2.0dev
[D 2020-07-19 18:54:02.190 JupyterHub app:2270] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[D 2020-07-19 18:54:02.198 JupyterHub app:1437] Generating new cookie_secret
[I 2020-07-19 18:54:02.199 JupyterHub app:1442] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
[D 2020-07-19 18:54:02.204 JupyterHub app:1564] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2020-07-19 18:54:02.410 JupyterHub orm:732] Stamping empty database with alembic revision 4dc2d5a8c53c
[I 2020-07-19 18:54:02.458 alembic.runtime.migration migration:155] Context impl SQLiteImpl.
[I 2020-07-19 18:54:02.458 alembic.runtime.migration migration:162] Will assume non-transactional DDL.
[I 2020-07-19 18:54:02.546 alembic.runtime.migration migration:515] Running stamp_revision  -> 4dc2d5a8c53c
[D 2020-07-19 18:54:02.548 alembic.runtime.migration migration:725] new branch insert 4dc2d5a8c53c
[I 2020-07-19 18:54:02.985 JupyterHub proxy:461] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2020-07-19 18:54:02.999 JupyterHub app:1672] No admin users, admin interface will be unavailable.
[W 2020-07-19 18:54:03.000 JupyterHub app:1674] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2020-07-19 18:54:03.005 JupyterHub app:1703] Not using allowed_users. Any authenticated user will be allowed.
[D 2020-07-19 18:54:03.456 JupyterHub app:1971] Initializing spawners
[D 2020-07-19 18:54:03.468 JupyterHub app:2104] Loaded users:
    
[I 2020-07-19 18:54:03.470 JupyterHub app:2377] Initialized 0 spawners in 0.016 seconds
[W 2020-07-19 18:54:03.512 JupyterHub proxy:643] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2020-07-19 18:54:03.513 JupyterHub proxy:646] Starting proxy @ http://:8000
[D 2020-07-19 18:54:03.514 JupyterHub proxy:647] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://172.21.0.2:8081/hub/error']
[D 2020-07-19 18:54:03.534 JupyterHub proxy:562] Writing proxy pid file: jupyterhub-proxy.pid
18:54:09.147 [ConfigProxy] info: Proxying http://*:8000 to (no default)
18:54:09.184 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2020-07-19 18:54:09.762 JupyterHub proxy:682] Proxy started and appears to be up
[D 2020-07-19 18:54:09.860 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
18:54:10.062 [ConfigProxy] info: 200 GET /api/routes 
[I 2020-07-19 18:54:10.091 JupyterHub app:2622] Hub API listening on http://0.0.0.0:8081/hub/
[I 2020-07-19 18:54:10.098 JupyterHub app:2624] Private Hub API connect url http://172.21.0.2:8081/hub/
[D 2020-07-19 18:54:10.100 JupyterHub proxy:315] Fetching routes to check
[D 2020-07-19 18:54:10.101 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2020-07-19 18:54:10.196 JupyterHub proxy:320] Checking routes
[I 2020-07-19 18:54:10.197 JupyterHub proxy:400] Adding default route for Hub: / => http://172.21.0.2:8081
[D 2020-07-19 18:54:10.203 JupyterHub proxy:766] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
18:54:10.202 [ConfigProxy] info: 200 GET /api/routes 
18:54:10.275 [ConfigProxy] info: Adding route / -> http://172.21.0.2:8081
18:54:10.311 [ConfigProxy] info: Route added / -> http://172.21.0.2:8081
18:54:10.314 [ConfigProxy] info: 201 POST /api/routes/ 
[I 2020-07-19 18:54:10.319 JupyterHub app:2697] JupyterHub is now running at http://:8000
[D 2020-07-19 18:54:10.323 JupyterHub app:2300] It took 8.267 seconds for the Hub to start
[I 2020-07-19 18:54:37.953 JupyterHub log:181] 302 GET / -> /hub/ (@::ffff:172.21.0.1) 17.80ms
[W 2020-07-19 18:54:38.041 JupyterHub base:392] Invalid or expired cookie token
[I 2020-07-19 18:54:38.058 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@::ffff:172.21.0.1) 29.28ms
[I 2020-07-19 18:54:38.749 JupyterHub log:181] 200 GET /hub/login (@::ffff:172.21.0.1) 660.21ms
[I 2020-07-19 18:54:40.748 JupyterHub oauth2:103] OAuth redirect: 'http://192.168.50.137:8000/hub/oauth_callback'
[I 2020-07-19 18:54:40.774 JupyterHub log:181] 302 GET /hub/oauth_login?next= -> https://turtlesystems.eu.auth0.com/authorize?response_type=code&redirect_uri=http%3A%2F%2F192.168.50.137%3A8000%2Fhub%2Foauth_callback&client_id=qwM1tWIOvsjICDFtSqt26IWmbIjDJiry&state=[secret]&scope=openid+profile+email (@::ffff:172.21.0.1) 27.18ms
[D 2020-07-19 18:54:42.246 JupyterHub base:526] Setting cookie for russell.seymour@turtlesystems.co.uk: jupyterhub-hub-login
[D 2020-07-19 18:54:42.247 JupyterHub base:522] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2020-07-19 18:54:42.250 JupyterHub base:742] User logged in: russell.seymour@turtlesystems.co.uk
[D 2020-07-19 18:54:42.251 JupyterHub user:289] Creating <class 'dockerspawner.dockerspawner.DockerSpawner'> for russell.seymour@turtlesystems.co.uk:
[I 2020-07-19 18:54:42.316 JupyterHub log:181] 302 GET /hub/oauth_callback?code=[secret]&state=[secret] -> /hub/spawn?code=[secret]&state=[secret] (@::ffff:172.21.0.1) 1097.42ms
[D 2020-07-19 18:54:42.383 JupyterHub base:285] Recording first activity for <User(russell.seymour@turtlesystems.co.uk 0/1 running)>
[D 2020-07-19 18:54:42.446 JupyterHub pages:191] Triggering spawn with supplied query arguments for russell.seymour@turtlesystems.co.uk
[D 2020-07-19 18:54:42.455 JupyterHub base:860] Initiating spawn for russell.seymour@turtlesystems.co.uk
[D 2020-07-19 18:54:42.455 JupyterHub base:867] 0/100 concurrent spawns
[D 2020-07-19 18:54:42.456 JupyterHub base:872] 0 active servers
[D 2020-07-19 18:54:43.014 JupyterHub user:603] Calling Spawner.start for russell.seymour@turtlesystems.co.uk
[D 2020-07-19 18:54:43.129 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[I 2020-07-19 18:54:43.159 JupyterHub dockerspawner:998] Found existing container jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk (id: 74fe269)
[I 2020-07-19 18:54:43.161 JupyterHub dockerspawner:1013] Starting container jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk (id: 74fe269)
[I 2020-07-19 18:54:43.473 JupyterHub log:181] 302 GET /hub/spawn?code=[secret]&state=[secret] -> /hub/spawn-pending/russell.seymour@turtlesystems.co.uk?code=[secret]&state=[secret] (russell.seymour@turtlesystems.co.uk@::ffff:172.21.0.1) 1116.60ms
[I 2020-07-19 18:54:43.602 JupyterHub pages:401] russell.seymour@turtlesystems.co.uk is pending spawn
[I 2020-07-19 18:54:43.719 JupyterHub log:181] 200 GET /hub/spawn-pending/russell.seymour@turtlesystems.co.uk?code=[secret]&state=[secret] (russell.seymour@turtlesystems.co.uk@::ffff:172.21.0.1) 158.26ms
[W 2020-07-19 18:54:43.911 JupyterHub user:662] russell.seymour@turtlesystems.co.uk's server specified its own API token that's not in the database
[D 2020-07-19 18:54:44.427 JupyterHub spawner:1151] Polling subprocess every 30s
[D 2020-07-19 18:54:52.468 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-19 18:54:52.588 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2020-07-19T14:32:58.5422922Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 21519,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2020-07-19T18:54:43.6061843Z',
     'Status': 'running'}
[W 2020-07-19 18:54:52.602 JupyterHub base:1017] User russell.seymour@turtlesystems.co.uk is slow to become responsive (timeout=10)
[D 2020-07-19 18:54:52.605 JupyterHub base:1022] Expecting server for russell.seymour@turtlesystems.co.uk at: http://127.0.0.1:32779/user/russell.seymour@turtlesystems.co.uk/
[W 2020-07-19 18:55:14.090 JupyterHub user:744] russell.seymour@turtlesystems.co.uk's server never showed up at http://127.0.0.1:32779/user/russell.seymour@turtlesystems.co.uk/ after 30 seconds. Giving up
[D 2020-07-19 18:55:14.094 JupyterHub user:791] Stopping russell.seymour@turtlesystems.co.uk
[D 2020-07-19 18:55:14.101 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-19 18:55:14.187 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2020-07-19T14:32:58.5422922Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 21519,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2020-07-19T18:54:43.6061843Z',
     'Status': 'running'}
[I 2020-07-19 18:55:14.192 JupyterHub dockerspawner:1107] Stopping container jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk (id: 74fe269)
[D 2020-07-19 18:55:14.312 JupyterHub user:822] Finished stopping russell.seymour@turtlesystems.co.uk
e[31m[E 2020-07-19 18:55:14.359 JupyterHub gen:599] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py:880> exception=TimeoutError("Server at http://127.0.0.1:32779/user/russell.seymour@turtlesystems.co.uk/ didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 593, in error_callback
        future.result()
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 887, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 720, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 767, in _wait_up
        raise e
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 735, in _wait_up
        http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/utils.py", line 234, in wait_for_http_server
        timeout=timeout,
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://127.0.0.1:32779/user/russell.seymour@turtlesystems.co.uk/ didn't respond in 30 seconds
    
[I 2020-07-19 18:55:14.393 JupyterHub log:181] 200 GET /hub/api/users/russell.seymour@turtlesystems.co.uk/server/progress (russell.seymour@turtlesystems.co.uk@::ffff:172.21.0.1) 29890.34ms
^C[C 2020-07-19 18:55:19.029 JupyterHub app:2750] Received signal SIGINT, initiating shutdown...
[D 2020-07-19 18:55:19.044 JupyterHub app:2756] Cancelling pending tasks
[D 2020-07-19 18:55:19.054 JupyterHub app:2768] Task status: <Task finished coro=<HTTP1ServerConnection._server_request_loop() done, defined at /usr/local/lib/python3.7/site-packages/tornado/http1connection.py:809> result=None>
[D 2020-07-19 18:55:19.063 JupyterHub app:2768] Task status: <Task finished coro=<HTTP1ServerConnection._server_request_loop() done, defined at /usr/local/lib/python3.7/site-packages/tornado/http1connection.py:809> result=None>
[D 2020-07-19 18:55:19.067 JupyterHub app:2768] Task status: <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py:880> exception=TimeoutError("Server at http://127.0.0.1:32779/user/russell.seymour@turtlesystems.co.uk/ didn't respond in 30 seconds")>
[D 2020-07-19 18:55:19.070 JupyterHub app:2768] Task status: <Task finished coro=<JupyterHub.launch_instance_async() done, defined at /usr/local/lib/python3.7/site-packages/jupyterhub/app.py:2779> result=None>
[D 2020-07-19 18:55:19.073 JupyterHub app:2768] Task status: <Task pending coro=<JupyterHub.shutdown_cancel_tasks() running at /usr/local/lib/python3.7/site-packages/jupyterhub/app.py:2768>>
[I 2020-07-19 18:55:19.077 JupyterHub app:2426] Cleaning up single-user servers...
[D 2020-07-19 18:55:19.082 JupyterHub app:2438] Stopping proxy
[I 2020-07-19 18:55:19.086 JupyterHub proxy:706] Cleaning up proxy[21]...
[D 2020-07-19 18:55:19.100 JupyterHub proxy:570] Removing proxy pid file jupyterhub-proxy.pid
[I 2020-07-19 18:55:19.105 JupyterHub app:2458] ...done
18:55:19.106 [ConfigProxy] warn: Terminated

Thank you

5

I can’t see anything obviously wrong. Would you mind checking the logs of the spawned container? Something like docker logs jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk.

6

Here are the logs from the spawned Lab container:

Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
[D 2020-07-17 20:50:29.003 SingleUserNotebookApp application:179] Searching ['/home/jovyan', '/home/jovyan/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2020-07-17 20:50:29.003 SingleUserNotebookApp application:556] Looking for jupyter_config in /etc/jupyter
[D 2020-07-17 20:50:29.003 SingleUserNotebookApp application:556] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2020-07-17 20:50:29.003 SingleUserNotebookApp application:556] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2020-07-17 20:50:29.003 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2020-07-17 20:50:29.003 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/jovyan
[D 2020-07-17 20:50:29.004 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /etc/jupyter
[D 2020-07-17 20:50:29.005 SingleUserNotebookApp application:578] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2020-07-17 20:50:29.005 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2020-07-17 20:50:29.005 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2020-07-17 20:50:29.005 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2020-07-17 20:50:29.006 SingleUserNotebookApp application:578] Loaded config file: /home/jovyan/.jupyter/jupyter_notebook_config.py
[D 2020-07-17 20:50:29.007 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/jovyan
[W 2020-07-17 20:50:29.010 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[D 2020-07-17 20:50:29.015 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /etc/jupyter/jupyter_notebook_config.json
[D 2020-07-17 20:50:29.016 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2020-07-17 20:50:29.016 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
        /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2020-07-17 20:50:29.017 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /home/jovyan/.jupyter/jupyter_notebook_config.json
[I 2020-07-17 20:50:30.264 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2020-07-17 20:50:30.265 SingleUserNotebookApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2020-07-17 20:50:30.269 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[W 2020-07-17 20:50:30.285 SingleUserNotebookApp _version:53] jupyterhub version 1.2.0dev != jupyterhub-singleuser version 1.1.0. This could cause failure to authenticate and result in redirect loops!
[I 2020-07-17 20:50:30.286 SingleUserNotebookApp notebookapp:1924] Serving notebooks from local directory: /lab
[I 2020-07-17 20:50:30.286 SingleUserNotebookApp notebookapp:1924] The Jupyter Notebook is running at:
[I 2020-07-17 20:50:30.286 SingleUserNotebookApp notebookapp:1924] http://74fe269f4c78:8888/user/russell.seymour@turtlesystems.co.uk/
[I 2020-07-17 20:50:30.286 SingleUserNotebookApp notebookapp:1925] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2020-07-17 20:50:30.292 SingleUserNotebookApp singleuser:541] Updating Hub with activity every 300 seconds
[D 2020-07-17 20:50:30.292 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2020-07-17T20:50:29.091884Z
[C 2020-07-17 20:50:56.772 SingleUserNotebookApp notebookapp:1615] received signal 15, stopping
[I 2020-07-17 20:50:56.772 SingleUserNotebookApp notebookapp:1784] Shutting down 0 kernels
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
[D 2020-07-17 21:14:06.784 SingleUserNotebookApp application:179] Searching ['/home/jovyan', '/home/jovyan/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2020-07-17 21:14:06.784 SingleUserNotebookApp application:556] Looking for jupyter_config in /etc/jupyter
[D 2020-07-17 21:14:06.787 SingleUserNotebookApp application:556] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2020-07-17 21:14:06.787 SingleUserNotebookApp application:556] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2020-07-17 21:14:06.796 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2020-07-17 21:14:06.796 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/jovyan
[D 2020-07-17 21:14:06.798 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /etc/jupyter
[D 2020-07-17 21:14:06.799 SingleUserNotebookApp application:578] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2020-07-17 21:14:06.799 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2020-07-17 21:14:06.799 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2020-07-17 21:14:06.800 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2020-07-17 21:14:06.817 SingleUserNotebookApp application:578] Loaded config file: /home/jovyan/.jupyter/jupyter_notebook_config.py
[D 2020-07-17 21:14:06.817 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/jovyan
[W 2020-07-17 21:14:06.830 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[D 2020-07-17 21:14:06.846 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /etc/jupyter/jupyter_notebook_config.json
[D 2020-07-17 21:14:06.847 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2020-07-17 21:14:06.849 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
        /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2020-07-17 21:14:06.851 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /home/jovyan/.jupyter/jupyter_notebook_config.json
[I 2020-07-17 21:14:08.290 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2020-07-17 21:14:08.290 SingleUserNotebookApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2020-07-17 21:14:08.296 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[E 2020-07-17 21:14:08.303 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 1/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-17 21:14:10.313 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 2/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-17 21:14:14.322 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 3/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-17 21:14:22.330 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 4/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[C 2020-07-17 21:14:34.281 SingleUserNotebookApp notebookapp:1615] received signal 15, stopping
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 660, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/opt/conda/lib/python3.8/site-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 563, in start
    ioloop.IOLoop.current().run_sync(self.check_hub_version)
  File "/opt/conda/lib/python3.8/site-packages/tornado/ioloop.py", line 526, in run_sync
    self.start()
  File "/opt/conda/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 149, in start
    self.asyncio_loop.run_forever()
  File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
    event_list = self._selector.select(timeout)
  File "/opt/conda/lib/python3.8/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
  File "/opt/conda/lib/python3.8/site-packages/notebook/notebookapp.py", line 1616, in _signal_stop
    self.io_loop.add_callback_from_signal(self.io_loop.stop)
AttributeError: 'SingleUserNotebookApp' object has no attribute 'io_loop'
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
[D 2020-07-19 14:32:30.865 SingleUserNotebookApp application:179] Searching ['/home/jovyan', '/home/jovyan/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2020-07-19 14:32:30.866 SingleUserNotebookApp application:556] Looking for jupyter_config in /etc/jupyter
[D 2020-07-19 14:32:30.867 SingleUserNotebookApp application:556] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2020-07-19 14:32:30.867 SingleUserNotebookApp application:556] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2020-07-19 14:32:30.873 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2020-07-19 14:32:30.874 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/jovyan
[D 2020-07-19 14:32:30.875 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /etc/jupyter
[D 2020-07-19 14:32:30.876 SingleUserNotebookApp application:578] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2020-07-19 14:32:30.877 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2020-07-19 14:32:30.877 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2020-07-19 14:32:30.877 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2020-07-19 14:32:30.878 SingleUserNotebookApp application:578] Loaded config file: /home/jovyan/.jupyter/jupyter_notebook_config.py
[D 2020-07-19 14:32:30.878 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/jovyan
[W 2020-07-19 14:32:30.885 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[D 2020-07-19 14:32:30.893 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /etc/jupyter/jupyter_notebook_config.json
[D 2020-07-19 14:32:30.894 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2020-07-19 14:32:30.895 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
        /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2020-07-19 14:32:30.896 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /home/jovyan/.jupyter/jupyter_notebook_config.json
[I 2020-07-19 14:32:32.355 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2020-07-19 14:32:32.355 SingleUserNotebookApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2020-07-19 14:32:32.360 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[W 2020-07-19 14:32:32.371 SingleUserNotebookApp _version:53] jupyterhub version 1.2.0dev != jupyterhub-singleuser version 1.1.0. This could cause failure to authenticate and result in redirect loops!
[I 2020-07-19 14:32:32.371 SingleUserNotebookApp notebookapp:1924] Serving notebooks from local directory: /lab
[I 2020-07-19 14:32:32.371 SingleUserNotebookApp notebookapp:1924] The Jupyter Notebook is running at:
[I 2020-07-19 14:32:32.371 SingleUserNotebookApp notebookapp:1924] http://74fe269f4c78:8888/user/russell.seymour@turtlesystems.co.uk/
[I 2020-07-19 14:32:32.372 SingleUserNotebookApp notebookapp:1925] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2020-07-19 14:32:32.378 SingleUserNotebookApp singleuser:541] Updating Hub with activity every 300 seconds
[D 2020-07-19 14:32:32.378 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2020-07-19T14:32:31.011408Z
[C 2020-07-19 14:32:58.324 SingleUserNotebookApp notebookapp:1615] received signal 15, stopping
[I 2020-07-19 14:32:58.325 SingleUserNotebookApp notebookapp:1784] Shutting down 0 kernels
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
[D 2020-07-19 18:54:45.093 SingleUserNotebookApp application:179] Searching ['/home/jovyan', '/home/jovyan/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2020-07-19 18:54:45.094 SingleUserNotebookApp application:556] Looking for jupyter_config in /etc/jupyter
[D 2020-07-19 18:54:45.095 SingleUserNotebookApp application:556] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2020-07-19 18:54:45.095 SingleUserNotebookApp application:556] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2020-07-19 18:54:45.098 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2020-07-19 18:54:45.098 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/jovyan
[D 2020-07-19 18:54:45.099 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /etc/jupyter
[D 2020-07-19 18:54:45.100 SingleUserNotebookApp application:578] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2020-07-19 18:54:45.100 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2020-07-19 18:54:45.100 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2020-07-19 18:54:45.101 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2020-07-19 18:54:45.102 SingleUserNotebookApp application:578] Loaded config file: /home/jovyan/.jupyter/jupyter_notebook_config.py
[D 2020-07-19 18:54:45.102 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/jovyan
[W 2020-07-19 18:54:45.106 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[D 2020-07-19 18:54:45.114 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /etc/jupyter/jupyter_notebook_config.json
[D 2020-07-19 18:54:45.114 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2020-07-19 18:54:45.115 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
        /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2020-07-19 18:54:45.116 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /home/jovyan/.jupyter/jupyter_notebook_config.json
[I 2020-07-19 18:54:46.142 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2020-07-19 18:54:46.142 SingleUserNotebookApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2020-07-19 18:54:46.147 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[E 2020-07-19 18:54:46.149 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 1/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-19 18:54:48.157 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 2/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-19 18:54:52.166 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 3/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-19 18:55:00.172 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 4/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[C 2020-07-19 18:55:14.292 SingleUserNotebookApp notebookapp:1615] received signal 15, stopping
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 660, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/opt/conda/lib/python3.8/site-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 563, in start
    ioloop.IOLoop.current().run_sync(self.check_hub_version)
  File "/opt/conda/lib/python3.8/site-packages/tornado/ioloop.py", line 526, in run_sync
    self.start()
  File "/opt/conda/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 149, in start
    self.asyncio_loop.run_forever()
  File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
    event_list = self._selector.select(timeout)
  File "/opt/conda/lib/python3.8/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
  File "/opt/conda/lib/python3.8/site-packages/notebook/notebookapp.py", line 1616, in _signal_stop
    self.io_loop.add_callback_from_signal(self.io_loop.stop)
AttributeError: 'SingleUserNotebookApp' object has no attribute 'io_loop'
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --notebook-dir=/lab --debug
[D 2020-07-20 07:59:26.799 SingleUserNotebookApp application:179] Searching ['/home/jovyan', '/home/jovyan/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2020-07-20 07:59:26.799 SingleUserNotebookApp application:556] Looking for jupyter_config in /etc/jupyter
[D 2020-07-20 07:59:26.800 SingleUserNotebookApp application:556] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2020-07-20 07:59:26.800 SingleUserNotebookApp application:556] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2020-07-20 07:59:26.804 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2020-07-20 07:59:26.805 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/jovyan
[D 2020-07-20 07:59:26.806 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /etc/jupyter
[D 2020-07-20 07:59:26.806 SingleUserNotebookApp application:578] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2020-07-20 07:59:26.807 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2020-07-20 07:59:26.807 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2020-07-20 07:59:26.807 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2020-07-20 07:59:26.807 SingleUserNotebookApp application:578] Loaded config file: /home/jovyan/.jupyter/jupyter_notebook_config.py
[D 2020-07-20 07:59:26.807 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/jovyan
[W 2020-07-20 07:59:26.811 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[D 2020-07-20 07:59:26.818 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /etc/jupyter/jupyter_notebook_config.json
[D 2020-07-20 07:59:26.822 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2020-07-20 07:59:26.823 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
        /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2020-07-20 07:59:26.824 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
        /home/jovyan/.jupyter/jupyter_notebook_config.json
[I 2020-07-20 07:59:28.206 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2020-07-20 07:59:28.207 SingleUserNotebookApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2020-07-20 07:59:28.213 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[E 2020-07-20 07:59:28.218 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 1/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:30.226 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 2/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:34.235 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 3/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:42.247 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 4/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:58.260 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 5/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[C 2020-07-20 07:59:58.332 SingleUserNotebookApp notebookapp:1615] received signal 15, stopping
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 660, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/opt/conda/lib/python3.8/site-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 563, in start
    ioloop.IOLoop.current().run_sync(self.check_hub_version)
  File "/opt/conda/lib/python3.8/site-packages/tornado/ioloop.py", line 526, in run_sync
    self.start()
  File "/opt/conda/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 149, in start
    self.asyncio_loop.run_forever()
  File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
    event_list = self._selector.select(timeout)
  File "/opt/conda/lib/python3.8/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
  File "/opt/conda/lib/python3.8/site-packages/notebook/notebookapp.py", line 1616, in _signal_stop
    self.io_loop.add_callback_from_signal(self.io_loop.stop)
AttributeError: 'SingleUserNotebookApp' object has no attribute 'io_loop'

Looking at the logs it seems that the Lab is not able to communicate back to the Hub, I guess because it has not been told what network to be on. There is a mismatch between the address that Hub is running on and the address the Lab is trying to use.

Do I need to add arguments to the spawner in the jupyterhub_config.py to tell it mount /var/run/docker.sock so it can see the network or explicitly set the network the lab should connect on? (I am just guessing here now :wink: )

7

It looks like your singleuser server is trying to connect to 172.17.0.2. Is this different from hub_connect_ip in your jupyterhub_config file:

8

Yes it is different.

If I go into the Hub container and run ip addr show I get the following:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:15:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.21.0.2/16 brd 172.21.255.255 scope global eth0
       valid_lft forever preferred_lft forever

This is obviously different to the Lab output of 172.17.0.2.

So the Lab seems to be coming up on a different network, which i why I was supplying the Docker network for it to come up onto, but then I have the error about the network not being found. Do I need to pass in something else as arguments for the DockerSpawner for the lab?

9

I have been playing about with settings this evening and I think I have made some progress, not to get it running but at least now the IP addresses are matching up.

I modified my jupyterhub_config.py so that it was not trying to connect to a specific Docker network and added some different JupyterHub settings, hub_bind_url and hub_connect_url:

# Configure Docker Spawner to launch an instance for the logged in user
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'

notebook_dir = '/lab'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = {
    '/data/{username}': notebook_dir
}

# Specify the image to use when spawning
c.DockerSpawner.image = "jupyter/scipy-notebook"

c.Spawner.debug = True

# Specify a timeout for starting the image
c.DockerSpawner.start_timeout = 600

# Configure authentication with Auth0
from oauthenticator.auth0 import Auth0OAuthenticator
c.JupyterHub.authenticator_class = Auth0OAuthenticator
c.Auth0OAuthenticator.auth0_subdomain = "turtlesystems.eu"
c.Auth0OAuthenticator.scope = ['openid', 'profile', 'email']

# the Hub's API listens on localhost by default,
# but docker containers can't see that.
# Tell the Hub to listen on its docker network:
import netifaces
docker0 = netifaces.ifaddresses('eth0')
docker0_ipv4 = docker0[netifaces.AF_INET][0]

c.JupyterHub.hub_ip = '0.0.0.0' #docker0_ipv4['addr']
c.JupyterHub.hub_bind_url = 'http://0.0.0.0:8081'
c.JupyterHub.hub_connect_url = 'http://' + docker0_ipv4['addr'] + ":8081"

c.DockerSpawmer.use_internal_ip = True

# Set the log level by value or name.
c.JupyterHub.log_level = 'DEBUG'

# Enable debug-logging of the single-user server
c.Spawner.debug = True

# Enable debug-logging of the single-user server
c.LocalProcessSpawner.debug = True

This was from this post in this channel - What's the main difference between hub_connect_url vs hub_bind_url?.

Now when I run JupyterHub and spawn the container I get much ore encouraging messages (as there are now talking to the correct IP addresses):

JupyterHub

21:01:47.162 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2020-07-20 21:01:47.332 JupyterHub proxy:682] Proxy started and appears to be up
[D 2020-07-20 21:01:47.348 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
21:01:47.576 [ConfigProxy] info: 200 GET /api/routes 
[I 2020-07-20 21:01:47.606 JupyterHub app:2622] Hub API listening on http://0.0.0.0:8081/hub/
[I 2020-07-20 21:01:47.616 JupyterHub app:2624] Private Hub API connect url http://172.17.0.2:8081/hub/
[D 2020-07-20 21:01:47.621 JupyterHub proxy:315] Fetching routes to check
[D 2020-07-20 21:01:47.625 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2020-07-20 21:01:47.765 JupyterHub proxy:320] Checking routes
[I 2020-07-20 21:01:47.767 JupyterHub proxy:400] Adding default route for Hub: / => http://172.17.0.2:8081
[D 2020-07-20 21:01:47.773 JupyterHub proxy:766] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
21:01:47.779 [ConfigProxy] info: 200 GET /api/routes 
21:01:47.922 [ConfigProxy] info: Adding route / -> http://172.17.0.2:8081
21:01:47.960 [ConfigProxy] info: Route added / -> http://172.17.0.2:8081
21:01:47.963 [ConfigProxy] info: 201 POST /api/routes/ 
[I 2020-07-20 21:01:47.972 JupyterHub app:2697] JupyterHub is now running at http://:8000
...
[W 2020-07-20 21:02:00.090 JupyterHub user:662] russell.seymour@turtlesystems.co.uk's server specified its own API token that's not in the database
[D 2020-07-20 21:02:00.612 JupyterHub spawner:1151] Polling subprocess every 30s
[I 2020-07-20 21:02:03.166 JupyterHub log:181] 200 GET /hub/api (@172.17.0.3) 11.22ms
[D 2020-07-20 21:02:03.302 JupyterHub base:285] Recording first activity for <APIToken('2ed6...', user='russell.seymour@turtlesystems.co.uk')>
[D 2020-07-20 21:02:03.443 JupyterHub users:711] Activity for user russell.seymour@turtlesystems.co.uk: 2020-07-20T21:02:01.164422Z
[D 2020-07-20 21:02:03.448 JupyterHub users:732] Activity on server russell.seymour@turtlesystems.co.uk/: 2020-07-20T21:02:01.164422Z
[I 2020-07-20 21:02:03.498 JupyterHub log:181] 200 POST /hub/api/users/russell.seymour@turtlesystems.co.uk/activity (russell.seymour@turtlesystems.co.uk@172.17.0.3) 317.98ms
[D 2020-07-20 21:02:08.423 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-20 21:02:08.532 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2020-07-20T20:59:49.4122648Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 13573,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2020-07-20T21:01:59.7352319Z',
     'Status': 'running'}
[W 2020-07-20 21:02:08.538 JupyterHub base:1017] User russell.seymour@turtlesystems.co.uk is slow to become responsive (timeout=10)
[D 2020-07-20 21:02:08.539 JupyterHub base:1022] Expecting server for russell.seymour@turtlesystems.co.uk at: http://127.0.0.1:32769/user/russell.seymour@turtlesystems.co.uk/
[D 2020-07-20 21:02:30.627 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-20 21:02:30.685 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2020-07-20T20:59:49.4122648Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 13573,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2020-07-20T21:01:59.7352319Z',
     'Status': 'running'}
[W 2020-07-20 21:02:31.092 JupyterHub user:744] russell.seymour@turtlesystems.co.uk's server never showed up at http://127.0.0.1:32769/user/russell.seymour@turtlesystems.co.uk/ after 30 seconds. Giving up
[D 2020-07-20 21:02:31.099 JupyterHub user:791] Stopping russell.seymour@turtlesystems.co.uk
[D 2020-07-20 21:02:31.107 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-20 21:02:31.211 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2020-07-20T20:59:49.4122648Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 13573,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2020-07-20T21:01:59.7352319Z',
     'Status': 'running'}
[I 2020-07-20 21:02:31.213 JupyterHub dockerspawner:1107] Stopping container jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk (id: 74fe269)
[D 2020-07-20 21:02:31.398 JupyterHub user:822] Finished stopping russell.seymour@turtlesystems.co.uk
[E 2020-07-20 21:02:31.490 JupyterHub gen:599] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py:880> exception=TimeoutError("Server at http://127.0.0.1:32769/user/russell.seymour@turtlesystems.co.uk/ didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 593, in error_callback
        future.result()
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 887, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 720, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 767, in _wait_up
        raise e
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 735, in _wait_up
        http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/utils.py", line 234, in wait_for_http_server
        timeout=timeout,
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://127.0.0.1:32769/user/russell.seymour@turtlesystems.co.uk/ didn't respond in 30 seconds

[I 2020-07-20 21:02:31.568 JupyterHub log:181] 200 GET /hub/api/users/russell.seymour@turtlesystems.co.uk/server/progress (russell.seymour@turtlesystems.co.uk@::ffff:172.17.0.1) 30846.80ms

JupyterLab

[I 2020-07-20 07:59:28.206 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2020-07-20 07:59:28.207 SingleUserNotebookApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2020-07-20 07:59:28.213 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[E 2020-07-20 07:59:28.218 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 1/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:30.226 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 2/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:34.235 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 3/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:42.247 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 4/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:58.260 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 5/5). Is it running?
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
        resp = await client.fetch(self.hub_api_url)
    ConnectionRefusedError: [Errno 111] Connection refused

So now the Lab container is trying to contact the Hub container on the correct IP address, but now I am seeing errors about not being able to connect to the Hub API.

10

Can you try fetching http://172.17.0.2:8081, e.g. with curl or wget, using docker exec inside both the jupyterhub and the spawned containers?

11

This also looks like a problem, http://127.0.0.1:32769 doesn’t make sense inside the JupyterHub Docker container.

There’s a typo in:

12

OK so things are looking better again, but still not working.

Spotting the typo in c.DockerSpawmer.use_internal_ip = True changed the output and now the containers are using proper IP address rather than localhost - YAY. However it is using the wrong IP address.

The following is the log from the JupyterHub container again:

[I 2020-07-21 10:11:51.225 JupyterHub app:2307] Running JupyterHub version 1.2.0dev
[I 2020-07-21 10:11:51.228 JupyterHub app:2338] Using Authenticator: oauthenticator.auth0.Auth0OAuthenticator-0.11.0
[I 2020-07-21 10:11:51.228 JupyterHub app:2338] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-0.11.1
[I 2020-07-21 10:11:51.229 JupyterHub app:2338] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.2.0dev
[D 2020-07-21 10:11:51.240 JupyterHub app:2270] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[D 2020-07-21 10:11:51.248 JupyterHub app:1437] Generating new cookie_secret
[I 2020-07-21 10:11:51.249 JupyterHub app:1442] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
[D 2020-07-21 10:11:51.251 JupyterHub app:1564] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2020-07-21 10:11:51.393 JupyterHub orm:732] Stamping empty database with alembic revision 4dc2d5a8c53c
[I 2020-07-21 10:11:51.426 alembic.runtime.migration migration:155] Context impl SQLiteImpl.
[I 2020-07-21 10:11:51.427 alembic.runtime.migration migration:162] Will assume non-transactional DDL.
[I 2020-07-21 10:11:51.504 alembic.runtime.migration migration:515] Running stamp_revision  -> 4dc2d5a8c53c
[D 2020-07-21 10:11:51.510 alembic.runtime.migration migration:725] new branch insert 4dc2d5a8c53c
[I 2020-07-21 10:11:52.053 JupyterHub proxy:461] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2020-07-21 10:11:52.085 JupyterHub app:1672] No admin users, admin interface will be unavailable.
[W 2020-07-21 10:11:52.086 JupyterHub app:1674] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2020-07-21 10:11:52.087 JupyterHub app:1703] Not using allowed_users. Any authenticated user will be allowed.
[D 2020-07-21 10:11:52.493 JupyterHub app:1971] Initializing spawners
[D 2020-07-21 10:11:52.508 JupyterHub app:2104] Loaded users:

[I 2020-07-21 10:11:52.513 JupyterHub app:2377] Initialized 0 spawners in 0.022 seconds
[W 2020-07-21 10:11:52.560 JupyterHub proxy:643] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2020-07-21 10:11:52.561 JupyterHub proxy:646] Starting proxy @ http://:8000
[D 2020-07-21 10:11:52.565 JupyterHub proxy:647] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://172.21.0.2:8081/hub/error']
[D 2020-07-21 10:11:52.575 JupyterHub proxy:562] Writing proxy pid file: jupyterhub-proxy.pid
10:11:56.887 [ConfigProxy] info: Proxying http://*:8000 to (no default)
10:11:56.915 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2020-07-21 10:11:57.178 JupyterHub proxy:682] Proxy started and appears to be up
[D 2020-07-21 10:11:57.195 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
10:11:57.415 [ConfigProxy] info: 200 GET /api/routes 
[I 2020-07-21 10:11:57.437 JupyterHub app:2622] Hub API listening on http://0.0.0.0:8081/hub/
[I 2020-07-21 10:11:57.444 JupyterHub app:2624] Private Hub API connect url http://172.21.0.2:8081/hub/
[D 2020-07-21 10:11:57.446 JupyterHub proxy:315] Fetching routes to check
[D 2020-07-21 10:11:57.448 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2020-07-21 10:11:57.547 JupyterHub proxy:320] Checking routes
[I 2020-07-21 10:11:57.548 JupyterHub proxy:400] Adding default route for Hub: / => http://172.21.0.2:8081
[D 2020-07-21 10:11:57.552 JupyterHub proxy:766] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
10:11:57.553 [ConfigProxy] info: 200 GET /api/routes
10:11:57.627 [ConfigProxy] info: Adding route / -> http://172.21.0.2:8081
10:11:57.661 [ConfigProxy] info: Route added / -> http://172.21.0.2:8081
10:11:57.670 [ConfigProxy] info: 201 POST /api/routes/ 
[I 2020-07-21 10:11:57.675 JupyterHub app:2697] JupyterHub is now running at http://:8000
[D 2020-07-21 10:11:57.680 JupyterHub app:2300] It took 6.572 seconds for the Hub to start
[W 2020-07-21 10:12:01.378 JupyterHub base:392] Invalid or expired cookie token
[I 2020-07-21 10:12:01.426 JupyterHub log:181] 302 GET /hub/home -> /hub/login?next=%2Fhub%2Fhome (@::ffff:172.21.0.1) 83.47ms
[I 2020-07-21 10:12:02.313 JupyterHub log:181] 200 GET /hub/login?next=%2Fhub%2Fhome (@::ffff:172.21.0.1) 772.53ms
[I 2020-07-21 10:12:03.635 JupyterHub oauth2:103] OAuth redirect: 'http://192.168.50.137:8000/hub/oauth_callback'
[I 2020-07-21 10:12:03.669 JupyterHub log:181] 302 GET /hub/oauth_login?next=%2Fhub%2Fhome -> https://turtlesystems.eu.auth0.com/authorize?response_type=code&redirect_uri=http%3A%2F%2F192.168.50.137%3A8000%2Fhub%2Foauth_callback&client_id=qwM1tWIOvsjICDFtSqt26IWmbIjDJiry&state=[secret]&scope=openid+profile+email (@::ffff:172.21.0.1) 38.18ms
[D 2020-07-21 10:12:05.010 JupyterHub base:526] Setting cookie for russell.seymour@turtlesystems.co.uk: jupyterhub-hub-login
[D 2020-07-21 10:12:05.011 JupyterHub base:522] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2020-07-21 10:12:05.017 JupyterHub base:742] User logged in: russell.seymour@turtlesystems.co.uk
[I 2020-07-21 10:12:05.025 JupyterHub log:181] 302 GET /hub/oauth_callback?code=[secret]&state=[secret] -> /hub/home (@::ffff:172.21.0.1) 952.85ms
[D 2020-07-21 10:12:05.095 JupyterHub base:285] Recording first activity for <User(russell.seymour@turtlesystems.co.uk 0/1 running)>
[D 2020-07-21 10:12:05.128 JupyterHub user:289] Creating <class 'dockerspawner.dockerspawner.DockerSpawner'> for russell.seymour@turtlesystems.co.uk:
[I 2020-07-21 10:12:05.397 JupyterHub log:181] 200 GET /hub/home (russell.seymour@turtlesystems.co.uk@::ffff:172.21.0.1) 333.45ms
[D 2020-07-21 10:12:05.604 JupyterHub log:181] 200 GET /hub/static/js/home.js?v=20200721101152 (@::ffff:172.21.0.1) 24.66ms
[D 2020-07-21 10:12:05.691 JupyterHub log:181] 200 GET /hub/static/components/moment/moment.js?v=20200721101152 (@::ffff:172.21.0.1) 30.64ms
[D 2020-07-21 10:12:05.750 JupyterHub log:181] 200 GET /hub/static/js/jhapi.js?v=20200721101152 (@::ffff:172.21.0.1) 24.16ms
[D 2020-07-21 10:12:05.812 JupyterHub log:181] 200 GET /hub/static/js/utils.js?v=20200721101152 (@::ffff:172.21.0.1) 19.90ms
[D 2020-07-21 10:12:07.578 JupyterHub pages:217] Triggering spawn with default options for russell.seymour@turtlesystems.co.uk
[D 2020-07-21 10:12:07.581 JupyterHub base:860] Initiating spawn for russell.seymour@turtlesystems.co.uk
[D 2020-07-21 10:12:07.582 JupyterHub base:867] 0/100 concurrent spawns
[D 2020-07-21 10:12:07.583 JupyterHub base:872] 0 active servers
[D 2020-07-21 10:12:08.250 JupyterHub user:603] Calling Spawner.start for russell.seymour@turtlesystems.co.uk
[D 2020-07-21 10:12:08.362 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[I 2020-07-21 10:12:08.395 JupyterHub dockerspawner:998] Found existing container jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk (id: 74fe269)
[I 2020-07-21 10:12:08.399 JupyterHub dockerspawner:1013] Starting container jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk (id: 74fe269)
[I 2020-07-21 10:12:08.592 JupyterHub log:181] 302 GET /hub/spawn/russell.seymour@turtlesystems.co.uk -> /hub/spawn-pending/russell.seymour@turtlesystems.co.uk (russell.seymour@turtlesystems.co.uk@::ffff:172.21.0.1) 1057.40ms
[I 2020-07-21 10:12:08.712 JupyterHub pages:401] russell.seymour@turtlesystems.co.uk is pending spawn
[I 2020-07-21 10:12:08.834 JupyterHub log:181] 200 GET /hub/spawn-pending/russell.seymour@turtlesystems.co.uk (russell.seymour@turtlesystems.co.uk@::ffff:172.21.0.1) 202.35ms
[W 2020-07-21 10:12:09.749 JupyterHub user:662] russell.seymour@turtlesystems.co.uk's server specified its own API token that's not in the database
[D 2020-07-21 10:12:10.360 JupyterHub spawner:1151] Polling subprocess every 30s
[D 2020-07-21 10:12:17.592 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-21 10:12:17.709 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2020-07-21T10:10:23.4581285Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 29327,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2020-07-21T10:12:09.1130397Z',
     'Status': 'running'}
[W 2020-07-21 10:12:17.716 JupyterHub base:1017] User russell.seymour@turtlesystems.co.uk is slow to become responsive (timeout=10)
[D 2020-07-21 10:12:17.720 JupyterHub base:1022] Expecting server for russell.seymour@turtlesystems.co.uk at: http://172.17.0.2:8888/user/russell.seymour@turtlesystems.co.uk/
[D 2020-07-21 10:12:40.370 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-21 10:12:40.472 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2020-07-21T10:10:23.4581285Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 29327,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2020-07-21T10:12:09.1130397Z',
     'Status': 'running'}
[W 2020-07-21 10:12:50.636 JupyterHub user:744] russell.seymour@turtlesystems.co.uk's server never showed up at http://172.17.0.2:8888/user/russell.seymour@turtlesystems.co.uk/ after 30 seconds. Giving up
[D 2020-07-21 10:12:50.641 JupyterHub user:791] Stopping russell.seymour@turtlesystems.co.uk
[D 2020-07-21 10:12:50.647 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-21 10:12:50.742 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2020-07-21T10:10:23.4581285Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 29327,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2020-07-21T10:12:09.1130397Z',
     'Status': 'running'}
[I 2020-07-21 10:12:50.744 JupyterHub dockerspawner:1107] Stopping container jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk (id: 74fe269)
[D 2020-07-21 10:12:50.889 JupyterHub user:822] Finished stopping russell.seymour@turtlesystems.co.uk
[E 2020-07-21 10:12:50.957 JupyterHub gen:599] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py:880> exception=TimeoutError("Server at http://172.17.0.2:8888/user/russell.seymour@turtlesystems.co.uk/ didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 593, in error_callback
        future.result()
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 887, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 720, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 767, in _wait_up
        raise e
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 735, in _wait_up
        http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/utils.py", line 234, in wait_for_http_server
        timeout=timeout,
      File "/usr/local/lib/python3.7/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://172.17.0.2:8888/user/russell.seymour@turtlesystems.co.uk/ didn't respond in 30 seconds

[I 2020-07-21 10:12:50.995 JupyterHub log:181] 200 GET /hub/api/users/russell.seymour@turtlesystems.co.uk/server/progress (russell.seymour@turtlesystems.co.uk@::ffff:172.21.0.1) 41789.25ms

If I exec into the Hub container I am able to hit the API url and get the version of the API, e.g.

curl http://172.21.0.2:8081/hub/api which gives:

{"version": "1.2.0dev"}

The spawned Lab container comes up on a different network and the IP address it has for the Hub API is different http://172.17.0.2:8081/hub/api which obviously does not work. This is why I was trying to get the containers to come up on a Docker network so they would be able to communicate with each other.

The spawned container does not have any tools installed, e.g. curl or netstsat for me to check the configuration of that. I can install them but Hub is deleting the container when it does not get a response - is there anyway to stop this from happening?

Thanks @manics for your help on this - it is very much appreciated.

13

The spawned container should receive the hub IP from the hub. If it’s receiving the wrong address then either the c.JupyterHub.hub_connect_url is incorrect, or something is very weird.

Could you add a print statement to verify c.JupyterHub.hub_connect_url is correct?

What version of dockerspawner are you running? If a server failed to start it will be stopped, but the default is that container won’t be deleted:

Though thinking about it if a container is restarted it might keep it’s old configuration (I’m not sure), which might explain it trying to connect to an old IP.

Is it definitely not listed in docker ps -a?

14

@manics Yo have hit the nail on the head!

I was assuming (bad I know) that the container was being deleted and not just stopped. Your assumption that it keeps the old configuration is correct. When I checked the logs for the stopped container I saw all of the logs.

I have since deleted the container and tried everything again and it is now working! The container can see the docker network I have created and everything is communicating as it should.

Thanks so much for your help. I can only apologise for not realising the same container was being used each time.

1 Like
15

@Russell_Seymour – would it be possible to post the final working version of your juputerhub_config.py and docker-compose.yml? I’ve been looking for a self-contained example like this.

16

Glad you got it working! The failure to refresh the connection settings wasn’t something I’d thought of before, I’ve only used DockerSpawner with remove=True. It makes sense to someone who knows how containers work but could easily trip up someone less familiar.

17

@manics Yeah for some reason I assumed that the container was being deleted, not just stopped and restarted. I should have looked at the stopped images … DOH!!!

@phaustin Yes no problem. I apologise in the delay in getting back to you.

I do not use Docker Compose for this as I am spinning up the containers in a different way. However I am happy to show you the docker command I am running.

Working jupyerthub_config.py file:

# Configure Docker Spawner to launch an instance for the logged in user
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'

notebook_dir = '/lab'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = {
    '/data/{username}': notebook_dir
}

# Specify the image to use when spawning
c.DockerSpawner.image = "jupyter/scipy-notebook"

c.Spawner.debug = True

# Specify a timeout for starting the image
c.DockerSpawner.start_timeout = 600

# Configure authentication with Auth0
from oauthenticator.auth0 import Auth0OAuthenticator
c.JupyterHub.authenticator_class = Auth0OAuthenticator
c.Auth0OAuthenticator.auth0_subdomain = "turtlesystems.eu"
c.Auth0OAuthenticator.scope = ['openid', 'profile', 'email']

# the Hub's API listens on localhost by default,
# but docker containers can't see that.
# Tell the Hub to listen on its docker network:
import netifaces
docker0 = netifaces.ifaddresses('eth0')
docker0_ipv4 = docker0[netifaces.AF_INET][0]

c.JupyterHub.hub_ip = '0.0.0.0' #docker0_ipv4['addr']
c.JupyterHub.hub_connect_ip = docker0_ipv4['addr']

network_name = "ccc-labs"
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.network_name = network_name
c.DockerSpawner.extra_host_config = { 'network_mode': network_name }

# Set the log level by value or name.
c.JupyterHub.log_level = 'DEBUG'

# Enable debug-logging of the single-user server
c.Spawner.debug = True

# Enable debug-logging of the single-user server
c.LocalProcessSpawner.debug = True

I created the Docker network with the following command:

docker network create ccc-labs

And the command that I run to spin up the JupyterHub container is as follows. (I have taken out the keys etc for my Auth0 integration):

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock \
                    -v ${PWD}/local:/etc/jupyterhub \
                    -p 8000:8000 \
                    --name jupyterhub \
                    -e OAUTH_CLIENT_ID="xxxx" \
                    -e OAUTH_CLIENT_SECRET="xxxx" \
                    -e OAUTH_CALLBACK_URL="http://x.x.x.194:8000/hub/oauth_callback" \
                    --net ccc-labs \
                    russellseymour/jupyterhub:0.0.12

You will see that that I am using my own image of JupyterHub, this was so that I can build a cross-platform image using Balenalib.

2 Likes
18

That’s really helpful – thx, Phil

19

Also checkout docker stacks examples: https://github.com/jupyter/docker-stacks

20

Thanks for the followup – for others who are trying to teach themselves jupyterhub//docker/k8s I also learned a lot from pangeo–docker-images – especially their use of github actions and condalock files.

If anyone out there is a docker-compose savant, I could use some help on this issue:
https://github.com/phaustin/jupyterhub_basics/issues/1 – TLDR – in that repo I’ve got the simplest dockerspawner example working with docker run but it hangs with my docker-compose.yml

I’m posting working examples of jupyter-books built with docker-compose here: https://github.com/eoas-ubc/eoas_hpc_edu#eoas-hpc-tutorials if you want to follow my progress.