I have been playing about with settings this evening and I think I have made some progress, not to get it running but at least now the IP addresses are matching up.
I modified my jupyterhub_config.py so that it was not trying to connect to a specific Docker network and added some different JupyterHub settings, hub_bind_url and hub_connect_url:
# Configure Docker Spawner to launch an instance for the logged in user
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
notebook_dir = '/lab'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = {
'/data/{username}': notebook_dir
}
# Specify the image to use when spawning
c.DockerSpawner.image = "jupyter/scipy-notebook"
c.Spawner.debug = True
# Specify a timeout for starting the image
c.DockerSpawner.start_timeout = 600
# Configure authentication with Auth0
from oauthenticator.auth0 import Auth0OAuthenticator
c.JupyterHub.authenticator_class = Auth0OAuthenticator
c.Auth0OAuthenticator.auth0_subdomain = "turtlesystems.eu"
c.Auth0OAuthenticator.scope = ['openid', 'profile', 'email']
# the Hub's API listens on localhost by default,
# but docker containers can't see that.
# Tell the Hub to listen on its docker network:
import netifaces
docker0 = netifaces.ifaddresses('eth0')
docker0_ipv4 = docker0[netifaces.AF_INET][0]
c.JupyterHub.hub_ip = '0.0.0.0' #docker0_ipv4['addr']
c.JupyterHub.hub_bind_url = 'http://0.0.0.0:8081'
c.JupyterHub.hub_connect_url = 'http://' + docker0_ipv4['addr'] + ":8081"
c.DockerSpawmer.use_internal_ip = True
# Set the log level by value or name.
c.JupyterHub.log_level = 'DEBUG'
# Enable debug-logging of the single-user server
c.Spawner.debug = True
# Enable debug-logging of the single-user server
c.LocalProcessSpawner.debug = True
This was from this post in this channel - What's the main difference between hub_connect_url vs hub_bind_url?.
Now when I run JupyterHub and spawn the container I get much ore encouraging messages (as there are now talking to the correct IP addresses):
JupyterHub
21:01:47.162 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2020-07-20 21:01:47.332 JupyterHub proxy:682] Proxy started and appears to be up
[D 2020-07-20 21:01:47.348 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
21:01:47.576 [ConfigProxy] info: 200 GET /api/routes
[I 2020-07-20 21:01:47.606 JupyterHub app:2622] Hub API listening on http://0.0.0.0:8081/hub/
[I 2020-07-20 21:01:47.616 JupyterHub app:2624] Private Hub API connect url http://172.17.0.2:8081/hub/
[D 2020-07-20 21:01:47.621 JupyterHub proxy:315] Fetching routes to check
[D 2020-07-20 21:01:47.625 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2020-07-20 21:01:47.765 JupyterHub proxy:320] Checking routes
[I 2020-07-20 21:01:47.767 JupyterHub proxy:400] Adding default route for Hub: / => http://172.17.0.2:8081
[D 2020-07-20 21:01:47.773 JupyterHub proxy:766] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
21:01:47.779 [ConfigProxy] info: 200 GET /api/routes
21:01:47.922 [ConfigProxy] info: Adding route / -> http://172.17.0.2:8081
21:01:47.960 [ConfigProxy] info: Route added / -> http://172.17.0.2:8081
21:01:47.963 [ConfigProxy] info: 201 POST /api/routes/
[I 2020-07-20 21:01:47.972 JupyterHub app:2697] JupyterHub is now running at http://:8000
...
[W 2020-07-20 21:02:00.090 JupyterHub user:662] russell.seymour@turtlesystems.co.uk's server specified its own API token that's not in the database
[D 2020-07-20 21:02:00.612 JupyterHub spawner:1151] Polling subprocess every 30s
[I 2020-07-20 21:02:03.166 JupyterHub log:181] 200 GET /hub/api (@172.17.0.3) 11.22ms
[D 2020-07-20 21:02:03.302 JupyterHub base:285] Recording first activity for <APIToken('2ed6...', user='russell.seymour@turtlesystems.co.uk')>
[D 2020-07-20 21:02:03.443 JupyterHub users:711] Activity for user russell.seymour@turtlesystems.co.uk: 2020-07-20T21:02:01.164422Z
[D 2020-07-20 21:02:03.448 JupyterHub users:732] Activity on server russell.seymour@turtlesystems.co.uk/: 2020-07-20T21:02:01.164422Z
[I 2020-07-20 21:02:03.498 JupyterHub log:181] 200 POST /hub/api/users/russell.seymour@turtlesystems.co.uk/activity (russell.seymour@turtlesystems.co.uk@172.17.0.3) 317.98ms
[D 2020-07-20 21:02:08.423 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-20 21:02:08.532 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
'Error': '',
'ExitCode': 0,
'FinishedAt': '2020-07-20T20:59:49.4122648Z',
'OOMKilled': False,
'Paused': False,
'Pid': 13573,
'Restarting': False,
'Running': True,
'StartedAt': '2020-07-20T21:01:59.7352319Z',
'Status': 'running'}
[W 2020-07-20 21:02:08.538 JupyterHub base:1017] User russell.seymour@turtlesystems.co.uk is slow to become responsive (timeout=10)
[D 2020-07-20 21:02:08.539 JupyterHub base:1022] Expecting server for russell.seymour@turtlesystems.co.uk at: http://127.0.0.1:32769/user/russell.seymour@turtlesystems.co.uk/
[D 2020-07-20 21:02:30.627 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-20 21:02:30.685 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
'Error': '',
'ExitCode': 0,
'FinishedAt': '2020-07-20T20:59:49.4122648Z',
'OOMKilled': False,
'Paused': False,
'Pid': 13573,
'Restarting': False,
'Running': True,
'StartedAt': '2020-07-20T21:01:59.7352319Z',
'Status': 'running'}
[W 2020-07-20 21:02:31.092 JupyterHub user:744] russell.seymour@turtlesystems.co.uk's server never showed up at http://127.0.0.1:32769/user/russell.seymour@turtlesystems.co.uk/ after 30 seconds. Giving up
[D 2020-07-20 21:02:31.099 JupyterHub user:791] Stopping russell.seymour@turtlesystems.co.uk
[D 2020-07-20 21:02:31.107 JupyterHub dockerspawner:777] Getting container 'jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk'
[D 2020-07-20 21:02:31.211 JupyterHub dockerspawner:762] Container 74fe269 status: {'Dead': False,
'Error': '',
'ExitCode': 0,
'FinishedAt': '2020-07-20T20:59:49.4122648Z',
'OOMKilled': False,
'Paused': False,
'Pid': 13573,
'Restarting': False,
'Running': True,
'StartedAt': '2020-07-20T21:01:59.7352319Z',
'Status': 'running'}
[I 2020-07-20 21:02:31.213 JupyterHub dockerspawner:1107] Stopping container jupyter-russell_2Eseymour_40turtlesystems_2Eco_2Euk (id: 74fe269)
[D 2020-07-20 21:02:31.398 JupyterHub user:822] Finished stopping russell.seymour@turtlesystems.co.uk
[E 2020-07-20 21:02:31.490 JupyterHub gen:599] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py:880> exception=TimeoutError("Server at http://127.0.0.1:32769/user/russell.seymour@turtlesystems.co.uk/ didn't respond in 30 seconds")> after timeout
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 593, in error_callback
future.result()
File "/usr/local/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 887, in finish_user_spawn
await spawn_future
File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 720, in spawn
await self._wait_up(spawner)
File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 767, in _wait_up
raise e
File "/usr/local/lib/python3.7/site-packages/jupyterhub/user.py", line 735, in _wait_up
http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
File "/usr/local/lib/python3.7/site-packages/jupyterhub/utils.py", line 234, in wait_for_http_server
timeout=timeout,
File "/usr/local/lib/python3.7/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://127.0.0.1:32769/user/russell.seymour@turtlesystems.co.uk/ didn't respond in 30 seconds
[I 2020-07-20 21:02:31.568 JupyterHub log:181] 200 GET /hub/api/users/russell.seymour@turtlesystems.co.uk/server/progress (russell.seymour@turtlesystems.co.uk@::ffff:172.17.0.1) 30846.80ms
JupyterLab
[I 2020-07-20 07:59:28.206 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2020-07-20 07:59:28.207 SingleUserNotebookApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2020-07-20 07:59:28.213 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[E 2020-07-20 07:59:28.218 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 1/5). Is it running?
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
resp = await client.fetch(self.hub_api_url)
ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:30.226 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 2/5). Is it running?
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
resp = await client.fetch(self.hub_api_url)
ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:34.235 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 3/5). Is it running?
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
resp = await client.fetch(self.hub_api_url)
ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:42.247 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 4/5). Is it running?
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
resp = await client.fetch(self.hub_api_url)
ConnectionRefusedError: [Errno 111] Connection refused
[E 2020-07-20 07:59:58.260 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at http://172.17.0.2:8081/hub/api (attempt 5/5). Is it running?
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
resp = await client.fetch(self.hub_api_url)
ConnectionRefusedError: [Errno 111] Connection refused
So now the Lab container is trying to contact the Hub container on the correct IP address, but now I am seeing errors about not being able to connect to the Hub API.