Redirect Loops and failing Jupyter Notebook Authentication

JupyterHub
, , ,
1

Please do not ignore this message assuming its a normal redirect loop issue. I have been investigating this particular scenario for last few weeks and had no success.

We have JupyterHub installed in a ec2 instance and have a customised spawner which spawns a single-user server in a container within an ECS instance.A docker image is used for spinning the container instance with jupyterhub-singleuser.

Here is the high level design,

image
image940×656 135 KB

JupyterHub Config -
import os
import pwd
import subprocess
import socket
c = get_config()

c.JupyterHub.config_file = ‘jupyterhub_config.py’
c.JupyterHub.allow_named_servers = True
c.JupyterHub.named_server_limit_per_user = 3
c.JupyterHub.cookie_max_age_days = 3
c.JupyterHub.redirect_to_server = False
from traitlets import default
c.JupyterHub.spawner_class = ‘bdcspawner.JHOptionsSpawner’

c.JupyterHub.log_level = ‘DEBUG’
c.Spawner.debug = True
c.Spawner.poll_interval = 300
c.Spawner.start_timeout = 600
c.JupyterHub.admin_access = True
def pre_spawn_hook(JHOptionsSpawner):
username = JHOptionsSpawner.user.name
print(username + “is username”)
try:
pwd.getpwnam(username)
except KeyError:
subprocess.check_call([‘useradd’, ‘-ms’, ‘/bin/bash’, username])
c.JHOptionsSpawner.pre_spawn_hook = pre_spawn_hook
c.PAMAuthenticator.open_sessions = False
c.Spawner.notebook_dir = ‘/home/jovyan’
c.Spawner.debug = True
c.JupyterHub.hub_connect_ip = socket.gethostbyname(socket.gethostname())
c.Authenticator.admin_users = {‘jovyan’}
c.JupyterHub.internal_ssl = False
c.JupyterHub.bind_url = ‘http://:8000’
c.JupyterHub.generate_certs = False
c.JHOptionsSpawner.cluster_name = ‘dap-bdc-nonprod-dev-01-datalab-ECSCluster’
c.JHOptionsSpawner.task_definition = ‘JupyterNotebook:87’
c.JHOptionsSpawner.aws_region_name = ‘ap-southeast-2’

c.Spawner.start_timeout = 600

Jupyter Notebook Config used in the process of building docker image can be found here,
from jupyter_core.paths import jupyter_data_dir

import subprocess
import os
import errno
import stat
c = get_config() # noqa: F821
c.NotebookApp.ip = “*”
c.NotebookApp.port = 8888
c.NotebookApp.allow_root = True
c.Session.debug = True
c.NotebookApp.notebook_dir = ‘/home/jovyan’
c.FileContentsManager.delete_to_trash = False

In summary - JupyterHub on its own works well in a RHEL8 ec2 instance. Jupyter Notebook as a service gets started in a ECS container instance. The problem happens with redirect loops when Jupyter Notebook contacts Hub for authentication.

Any help will be much appreciated.
I have the JupyterHub console logs and ECS logs saved in word docs. I can send it through to someone who is able to help me with this.

Thanks in advance.

2

Hi!

It looks like some of your formatting may have been lost when you pasted in your text, see Creating and highlighting code blocks - GitHub Docs for examples of formatting code blocks.

I’m afraid there’s not enough information to replicate your problem, or to know where to begin. Can you turn on debug logging, and share your logs for the hub and singleuser servers as plain text?

1 Like
3

Part 1 - JupyterHub Log

[I 2022-02-28 15:16:06.027 JupyterHub app:2766] Running JupyterHub version 2.1.1
[I 2022-02-28 15:16:06.027 JupyterHub app:2796] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.1.1
[I 2022-02-28 15:16:06.027 JupyterHub app:2796] Using Spawner: bdcspawner.Spawner.JHOptionsSpawner-0.0.1
[I 2022-02-28 15:16:06.028 JupyterHub app:2796] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.1.1
[D 2022-02-28 15:16:06.029 JupyterHub app:2727] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[I 2022-02-28 15:16:06.029 JupyterHub app:1606] Loading cookie_secret from /usr/bin/jupyterhub_cookie_secret
[D 2022-02-28 15:16:06.030 JupyterHub app:1773] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2022-02-28 15:16:06.048 JupyterHub orm:955] database schema version found: 833da8570507
[I 2022-02-28 15:16:06.128 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2022-02-28 15:16:06.129 JupyterHub app:2019] Loading roles into database
[I 2022-02-28 15:16:06.142 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
[D 2022-02-28 15:16:06.147 JupyterHub app:2278] Purging expired APITokens
[D 2022-02-28 15:16:06.151 JupyterHub app:2278] Purging expired OAuthCodes
[D 2022-02-28 15:16:06.153 JupyterHub app:2111] Loading role assignments from config
[D 2022-02-28 15:16:06.167 JupyterHub app:2424] Initializing spawners
[D 2022-02-28 15:16:06.169 JupyterHub app:2555] Loaded users:
[I 2022-02-28 15:16:06.170 JupyterHub app:2835] Initialized 0 spawners in 0.003 seconds
[W 2022-02-28 15:16:06.172 JupyterHub proxy:687] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2022-02-28 15:16:06.172 JupyterHub proxy:691] Starting proxy @ http://:8000
[D 2022-02-28 15:16:06.172 JupyterHub proxy:692] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://10.86.124.187:8081/hub/error']
[D 2022-02-28 15:16:06.176 JupyterHub proxy:610] Writing proxy pid file: jupyterhub-proxy.pid
15:16:06.349 [ConfigProxy] info: Proxying http://*:8000 to (no default)
15:16:06.352 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2022-02-28 15:16:06.421 JupyterHub proxy:728] Proxy started and appears to be up
[D 2022-02-28 15:16:06.424 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2022-02-28 15:16:06.433 JupyterHub app:3084] Hub API listening on http://10.86.124.187:8081/hub/
15:16:06.433 [ConfigProxy] info: 200 GET /api/routes
[D 2022-02-28 15:16:06.433 JupyterHub proxy:343] Fetching routes to check
[D 2022-02-28 15:16:06.433 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
15:16:06.435 [ConfigProxy] info: 200 GET /api/routes
[D 2022-02-28 15:16:06.436 JupyterHub proxy:346] Checking routes
[I 2022-02-28 15:16:06.436 JupyterHub proxy:431] Adding route for Hub: / => http://10.86.124.187:8081
[D 2022-02-28 15:16:06.436 JupyterHub proxy:821] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
15:16:06.438 [ConfigProxy] info: Adding route / -> http://10.86.124.187:8081
15:16:06.439 [ConfigProxy] info: Route added / -> http://10.86.124.187:8081
15:16:06.439 [ConfigProxy] info: 201 POST /api/routes/
[I 2022-02-28 15:16:06.440 JupyterHub app:3150] JupyterHub is now running at http://:8000
[D 2022-02-28 15:16:06.440 JupyterHub app:2759] It took 0.540 seconds for the Hub to start
[I 2022-02-28 15:16:15.914 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.143) 30.74ms
[W 2022-02-28 15:16:23.718 JupyterHub base:393] Invalid or expired cookie token
[I 2022-02-28 15:16:23.720 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.92) 2.59ms
[D 2022-02-28 15:16:24.149 JupyterHub log:189] 200 GET /hub/static/css/style.min.css?v=bff49b4a161afb17ee3b71927ce7d6c4e5b0e4b9ef6f18ca3e356a05f29e69776d3a76aee167060dd2ae2ee62d3cfdcf203b4b0090b1423f7d629ea7daa3f9da (@::ffff:10.86.124.92) 1.71ms
[D 2022-02-28 15:16:24.150 JupyterHub log:189] 200 GET /hub/static/components/requirejs/require.js?v=bd1aa102bdb0b27fbf712b32cfcd29b016c272acf3d864ee8469376eaddd032cadcf827ff17c05a8c8e20061418fe58cf79947049f5c0dff3b4f73fcc8cad8ec (@::ffff:10.86.124.92) 0.84ms
[D 2022-02-28 15:16:24.151 JupyterHub log:189] 200 GET /hub/static/components/jquery/dist/jquery.min.js?v=f3de1813a4160f9239f4781938645e1589b876759cd50b7936dbd849a35c38ffaed53f6a61dbdd8a1cf43cf4a28aa9fffbfddeec9a3811a1bb4ee6df58652b31 (@::ffff:10.86.124.92) 1.30ms
[D 2022-02-28 15:16:24.152 JupyterHub log:189] 200 GET /hub/static/components/bootstrap/dist/js/bootstrap.min.js?v=a014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce (@::ffff:10.86.124.92) 1.51ms
[D 2022-02-28 15:16:26.019 JupyterHub log:189] 200 GET /hub/logo (@::ffff:10.86.124.92) 1.06ms
[D 2022-02-28 15:16:26.381 JupyterHub log:189] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f(@::ffff:10.86.124.92) 0.98ms
[I 2022-02-28 15:16:32.532 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.92) 1.90ms
[D 2022-02-28 15:16:34.132 JupyterHub log:189] 200 GET /hub/static/components/font-awesome/fonts/fontawesome-webfont.woff2?v=4.7.0 (@::ffff:10.86.124.92) 1.08ms
[D 2022-02-28 15:16:34.161 JupyterHub roles:450] Assigning default role to User jovyan
[D 2022-02-28 15:16:34.168 JupyterHub base:559] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
[D 2022-02-28 15:16:34.168 JupyterHub base:563] Setting cookie for jovyan: jupyterhub-hub-login
[D 2022-02-28 15:16:34.168 JupyterHub base:559] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2022-02-28 15:16:34.168 JupyterHub base:797] User logged in: jovyan
[I 2022-02-28 15:16:34.169 JupyterHub log:189] 302 POST /hub/login?next= -> /hub/home (jovyan@::ffff:10.86.124.92) 36.14ms
[D 2022-02-28 15:16:34.738 JupyterHub user:347] Creating <class 'bdcspawner.Spawner.JHOptionsSpawner'> for jovyan:
[I 2022-02-28 15:16:34.851 JupyterHub log:189] 200 GET /hub/home (jovyan@::ffff:10.86.124.92) 125.04ms
[D 2022-02-28 15:16:35.268 JupyterHub log:189] 200 GET /hub/static/css/style.min.css?v=bff49b4a161afb17ee3b71927ce7d6c4e5b0e4b9ef6f18ca3e356a05f29e69776d3a76aee167060dd2ae2ee62d3cfdcf203b4b0090b1423f7d629ea7daa3f9da (@::ffff:10.86.124.92) 1.25ms
[D 2022-02-28 15:16:36.316 JupyterHub log:189] 200 GET /hub/static/js/home.js?v=20220228151606 (@::ffff:10.86.124.92) 1.02ms
[D 2022-02-28 15:16:36.681 JupyterHub log:189] 200 GET /hub/static/components/moment/moment.js?v=20220228151606 (@::ffff:10.86.124.92) 1.18ms
[D 2022-02-28 15:16:36.683 JupyterHub log:189] 200 GET /hub/static/js/jhapi.js?v=20220228151606 (@::ffff:10.86.124.92) 0.74ms
[D 2022-02-28 15:16:37.448 JupyterHub log:189] 200 GET /hub/static/js/utils.js?v=20220228151606 (@::ffff:10.86.124.92) 1.00ms
[D 2022-02-28 15:16:39.339 JupyterHub scopes:491] Checking access via scope servers
[D 2022-02-28 15:16:39.339 JupyterHub scopes:389] Unrestricted access to /hub/spawn/jovyan/meena via servers
[D 2022-02-28 15:16:39.339 JupyterHub user:347] Creating <class 'bdcspawner.Spawner.JHOptionsSpawner'> for jovyan:meena
[D 2022-02-28 15:16:39.351 JupyterHub pages:207] Serving options form for jovyan:meena
[I 2022-02-28 15:16:39.357 JupyterHub log:189] 200 GET /hub/spawn/jovyan/meena (jovyan@::ffff:10.86.124.92) 19.96ms
[I 2022-02-28 15:16:45.895 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.143) 1.94ms
[I 2022-02-28 15:17:02.534 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.92) 1.89ms
[I 2022-02-28 15:17:15.913 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.143) 1.88ms
[I 2022-02-28 15:17:32.564 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.92) 2.03ms
[I 2022-02-28 15:17:45.942 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.143) 1.88ms
[D 2022-02-28 15:17:49.411 JupyterHub scopes:491] Checking access via scope servers
[D 2022-02-28 15:17:49.411 JupyterHub scopes:389] Unrestricted access to /hub/spawn/jovyan/meena via servers
[D 2022-02-28 15:17:49.411 JupyterHub pages:207] Serving options form for jovyan:meena
[I 2022-02-28 15:17:49.413 JupyterHub log:189] 200 GET /hub/spawn/jovyan/meena (jovyan@::ffff:10.86.124.143) 8.65ms
[I 2022-02-28 15:18:02.594 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.92) 1.93ms
[I 2022-02-28 15:18:15.972 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.143) 1.88ms
[D 2022-02-28 15:18:21.325 JupyterHub scopes:491] Checking access via scope servers
[D 2022-02-28 15:18:21.325 JupyterHub scopes:389] Unrestricted access to /hub/spawn/jovyan/meena via servers
[D 2022-02-28 15:18:21.325 JupyterHub pages:255] Triggering spawn with supplied form options for jovyan:meena
[D 2022-02-28 15:18:21.325 JupyterHub base:915] Initiating spawn for jovyan:meena
[D 2022-02-28 15:18:21.326 JupyterHub base:919] 0/100 concurrent spawns
[D 2022-02-28 15:18:21.326 JupyterHub base:924] 0 active servers
[D 2022-02-28 15:18:21.332 JupyterHub roles:473] Checking token permissions against requested role server
[I 2022-02-28 15:18:21.334 JupyterHub roles:478] Adding role server to token: <APIToken('e69d...', user='jovyan', client_id='jupyterhub')>
[I 2022-02-28 15:18:21.350 JupyterHub provider:607] Creating oauth client jupyterhub-user-jovyan-meena
jovyanis username
[D 2022-02-28 15:18:21.371 JupyterHub user:676] Calling Spawner.start for jovyan:meena
[I 2022-02-28 15:18:21.371 JupyterHub Spawner:239] Running new task
[I 2022-02-28 15:18:21.371 JupyterHub Spawner:240] dap-bdc-nonprod-dev-01-datalab-ECSCluster
[I 2022-02-28 15:18:21.371 JupyterHub Spawner:241] JupyterNotebook:87
[I 2022-02-28 15:18:21.371 JupyterHub Spawner:242] jovyan
[D 2022-02-28 15:18:21.541 JupyterHub Spawner:253] ecs:RunTask = {'tasks': [{'attachments': [], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'ap-southeast-2b', 'clusterArn': 'arn:aws:ecs:ap-southeast-2:035973170720:cluster/dap-bdc-nonprod-dev-01-datalab-ECSCluster', 'containerInstanceArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container-instance/dap-bdc-nonprod-dev-01-datalab-ECSCluster/f027356e1ba74be1a68809cb009ad713', 'containers': [{'containerArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99/4f09ff3e-7855-43be-9ed4-08a70dc692aa', 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'name': 'jupyternotebook', 'image': '035973170720.dkr.ecr.ap-southeast-2.amazonaws.com/dap-bdc-nonprod-dev-01-datalab-ecr:0.87', 'lastStatus': 'PENDING', 'networkInterfaces': [], 'cpu': '0'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 2, 28, 15, 18, 21, 490000, tzinfo=tzlocal()), 'desiredStatus': 'RUNNING', 'enableExecuteCommand': False, 'group': 'family:JupyterNotebook', 'lastStatus': 'PENDING', 'launchType': 'EC2', 'memory': '512', 'overrides': {'containerOverrides': [{'name': 'jupyternotebook', 'environment': [{'name': 'PATH', 'value': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/bin'}, {'name': 'LANG', 'value': 'en_US.UTF-8'}, {'name': 'JUPYTERHUB_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JPY_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JUPYTERHUB_ADMIN_ACCESS', 'value': '1'}, {'name': 'JUPYTERHUB_CLIENT_ID', 'value': 'jupyterhub-user-jovyan-meena'}, {'name': 'JUPYTERHUB_HOST', 'value': ''}, {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL', 'value': '/user/jovyan/meena/oauth_callback'}, {'name': 'JUPYTERHUB_OAUTH_SCOPES', 'value': '["access:servers!server=jovyan/meena", "access:servers!user=jovyan"]'}, {'name': 'JUPYTERHUB_USER', 'value': 'jovyan'}, {'name': 'JUPYTERHUB_SERVER_NAME', 'value': 'meena'}, {'name': 'JUPYTERHUB_API_URL', 'value': 'http://10.86.124.187:8081/hub/api'}, {'name': 'JUPYTERHUB_ACTIVITY_URL', 'value': 'http://10.86.124.187:8081/hub/api/users/jovyan/activity'}, {'name': 'JUPYTERHUB_BASE_URL', 'value': '/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_SERVICE_PREFIX', 'value': 'http://10.86.124.187:8081/'}, {'name': 'JUPYTERHUB_SERVICE_URL', 'value': 'http://:10.86.124.187:8081/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_ROOT_DIR', 'value': '/home/jovyan'}, {'name': 'JUPYTERHUB_DEBUG', 'value': '1'}]}], 'inferenceAcceleratorOverrides': []}, 'startedBy': 'jovyan', 'tags': [], 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'taskDefinitionArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task-definition/JupyterNotebook:87', 'version': 1}], 'failures': [], 'ResponseMetadata': {'RequestId': '6ab66a2a-cb5f-44c0-878d-ee7575724209', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '6ab66a2a-cb5f-44c0-878d-ee7575724209', 'content-type': 'application/x-amz-json-1.1', 'content-length': '2705', 'date': 'Mon, 28 Feb 2022 02:18:21 GMT'}, 'RetryAttempts': 0}}
[I 2022-02-28 15:18:21.541 JupyterHub Spawner:254] ecs:RunTask = {'tasks': [{'attachments': [], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'ap-southeast-2b', 'clusterArn': 'arn:aws:ecs:ap-southeast-2:035973170720:cluster/dap-bdc-nonprod-dev-01-datalab-ECSCluster', 'containerInstanceArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container-instance/dap-bdc-nonprod-dev-01-datalab-ECSCluster/f027356e1ba74be1a68809cb009ad713', 'containers': [{'containerArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99/4f09ff3e-7855-43be-9ed4-08a70dc692aa', 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'name': 'jupyternotebook', 'image': '035973170720.dkr.ecr.ap-southeast-2.amazonaws.com/dap-bdc-nonprod-dev-01-datalab-ecr:0.87', 'lastStatus': 'PENDING', 'networkInterfaces': [], 'cpu': '0'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 2, 28, 15, 18, 21, 490000, tzinfo=tzlocal()), 'desiredStatus': 'RUNNING', 'enableExecuteCommand': False, 'group': 'family:JupyterNotebook', 'lastStatus': 'PENDING', 'launchType': 'EC2', 'memory': '512', 'overrides': {'containerOverrides': [{'name': 'jupyternotebook', 'environment': [{'name': 'PATH', 'value': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/bin'}, {'name': 'LANG', 'value': 'en_US.UTF-8'}, {'name': 'JUPYTERHUB_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JPY_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JUPYTERHUB_ADMIN_ACCESS', 'value': '1'}, {'name': 'JUPYTERHUB_CLIENT_ID', 'value': 'jupyterhub-user-jovyan-meena'}, {'name': 'JUPYTERHUB_HOST', 'value': ''}, {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL', 'value': '/user/jovyan/meena/oauth_callback'}, {'name': 'JUPYTERHUB_OAUTH_SCOPES', 'value': '["access:servers!server=jovyan/meena", "access:servers!user=jovyan"]'}, {'name': 'JUPYTERHUB_USER', 'value': 'jovyan'}, {'name': 'JUPYTERHUB_SERVER_NAME', 'value': 'meena'}, {'name': 'JUPYTERHUB_API_URL', 'value': 'http://10.86.124.187:8081/hub/api'}, {'name': 'JUPYTERHUB_ACTIVITY_URL', 'value': 'http://10.86.124.187:8081/hub/api/users/jovyan/activity'}, {'name': 'JUPYTERHUB_BASE_URL', 'value': '/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_SERVICE_PREFIX', 'value': 'http://10.86.124.187:8081/'}, {'name': 'JUPYTERHUB_SERVICE_URL', 'value': 'http://:10.86.124.187:8081/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_ROOT_DIR', 'value': '/home/jovyan'}, {'name': 'JUPYTERHUB_DEBUG', 'value': '1'}]}], 'inferenceAcceleratorOverrides': []}, 'startedBy': 'jovyan', 'tags': [], 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'taskDefinitionArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task-definition/JupyterNotebook:87', 'version': 1}], 'failures': [], 'ResponseMetadata': {'RequestId': '6ab66a2a-cb5f-44c0-878d-ee7575724209', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '6ab66a2a-cb5f-44c0-878d-ee7575724209', 'content-type': 'application/x-amz-json-1.1', 'content-length': '2705', 'date': 'Mon, 28 Feb 2022 02:18:21 GMT'}, 'RetryAttempts': 0}}
[I 2022-02-28 15:18:21.541 JupyterHub Spawner:266] Spawned notebook container
[I 2022-02-28 15:18:21.541 JupyterHub Spawner:268] Fetching task info
[D 2022-02-28 15:18:21.570 JupyterHub Spawner:273] ecs:DescribeTasks = {'tasks': [{'attachments': [], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'ap-southeast-2b','clusterArn': 'arn:aws:ecs:ap-southeast-2:035973170720:cluster/dap-bdc-nonprod-dev-01-datalab-ECSCluster', 'containerInstanceArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container-instance/dap-bdc-nonprod-dev-01-datalab-ECSCluster/f027356e1ba74be1a68809cb009ad713', 'containers': [{'containerArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99/4f09ff3e-7855-43be-9ed4-08a70dc692aa', 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'name': 'jupyternotebook', 'image': '035973170720.dkr.ecr.ap-southeast-2.amazonaws.com/dap-bdc-nonprod-dev-01-datalab-ecr:0.87', 'lastStatus': 'PENDING', 'networkInterfaces': [], 'healthStatus': 'UNKNOWN', 'cpu': '0'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 2, 28, 15, 18, 21, 490000, tzinfo=tzlocal()), 'desiredStatus': 'RUNNING', 'enableExecuteCommand': False, 'group': 'family:JupyterNotebook', 'healthStatus': 'UNKNOWN', 'lastStatus': 'PENDING', 'launchType': 'EC2', 'memory': '512', 'overrides': {'containerOverrides': [{'name': 'jupyternotebook', 'environment': [{'name': 'PATH', 'value': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/bin'}, {'name': 'LANG', 'value': 'en_US.UTF-8'}, {'name': 'JUPYTERHUB_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JPY_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JUPYTERHUB_ADMIN_ACCESS', 'value': '1'}, {'name': 'JUPYTERHUB_CLIENT_ID', 'value': 'jupyterhub-user-jovyan-meena'}, {'name': 'JUPYTERHUB_HOST', 'value': ''}, {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL', 'value': '/user/jovyan/meena/oauth_callback'}, {'name': 'JUPYTERHUB_OAUTH_SCOPES', 'value': '["access:servers!server=jovyan/meena", "access:servers!user=jovyan"]'}, {'name': 'JUPYTERHUB_USER', 'value': 'jovyan'}, {'name': 'JUPYTERHUB_SERVER_NAME', 'value': 'meena'}, {'name': 'JUPYTERHUB_API_URL', 'value': 'http://10.86.124.187:8081/hub/api'}, {'name': 'JUPYTERHUB_ACTIVITY_URL', 'value': 'http://10.86.124.187:8081/hub/api/users/jovyan/activity'}, {'name': 'JUPYTERHUB_BASE_URL', 'value': '/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_SERVICE_PREFIX', 'value': 'http://10.86.124.187:8081/'}, {'name': 'JUPYTERHUB_SERVICE_URL', 'value': 'http://:10.86.124.187:8081/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_ROOT_DIR', 'value': '/home/jovyan'}, {'name': 'JUPYTERHUB_DEBUG', 'value': '1'}]}], 'inferenceAcceleratorOverrides': []}, 'startedBy': 'jovyan', 'tags': [], 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'taskDefinitionArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task-definition/JupyterNotebook:87', 'version': 1}], 'failures': [], 'ResponseMetadata': {'RequestId': 'dd2413b5-6908-4a74-98db-c53940818d2e', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'dd2413b5-6908-4a74-98db-c53940818d2e', 'content-type': 'application/x-amz-json-1.1', 'content-length': '2755', 'date': 'Mon, 28 Feb 2022 02:18:21 GMT'}, 'RetryAttempts': 0}}
[I 2022-02-28 15:18:21.570 JupyterHub Spawner:283] Waiting for container instance status to move from PENDING to RUNNING
[I 2022-02-28 15:18:22.327 JupyterHub log:189] 302 POST /hub/spawn/jovyan/meena -> /hub/spawn-pending/jovyan/meena (jovyan@::ffff:10.86.124.143) 1010.27ms
[D 2022-02-28 15:18:22.594 JupyterHub Spawner:273] ecs:DescribeTasks = {'tasks': [{'attachments': [], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'ap-southeast-2b','clusterArn': 'arn:aws:ecs:ap-southeast-2:035973170720:cluster/dap-bdc-nonprod-dev-01-datalab-ECSCluster', 'containerInstanceArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container-instance/dap-bdc-nonprod-dev-01-datalab-ECSCluster/f027356e1ba74be1a68809cb009ad713', 'containers': [{'containerArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99/4f09ff3e-7855-43be-9ed4-08a70dc692aa', 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'name': 'jupyternotebook', 'image': '035973170720.dkr.ecr.ap-southeast-2.amazonaws.com/dap-bdc-nonprod-dev-01-datalab-ecr:0.87', 'lastStatus': 'PENDING', 'networkInterfaces': [], 'healthStatus': 'UNKNOWN', 'cpu': '0'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 2, 28, 15, 18, 21, 490000, tzinfo=tzlocal()), 'desiredStatus': 'RUNNING', 'enableExecuteCommand': False, 'group': 'family:JupyterNotebook', 'healthStatus': 'UNKNOWN', 'lastStatus': 'PENDING', 'launchType': 'EC2', 'memory': '512', 'overrides': {'containerOverrides': [{'name': 'jupyternotebook', 'environment': [{'name': 'PATH', 'value': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/bin'}, {'name': 'LANG', 'value': 'en_US.UTF-8'}, {'name': 'JUPYTERHUB_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JPY_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JUPYTERHUB_ADMIN_ACCESS', 'value': '1'}, {'name': 'JUPYTERHUB_CLIENT_ID', 'value': 'jupyterhub-user-jovyan-meena'}, {'name': 'JUPYTERHUB_HOST', 'value': ''}, {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL', 'value': '/user/jovyan/meena/oauth_callback'}, {'name': 'JUPYTERHUB_OAUTH_SCOPES', 'value': '["access:servers!server=jovyan/meena", "access:servers!user=jovyan"]'}, {'name': 'JUPYTERHUB_USER', 'value': 'jovyan'}, {'name': 'JUPYTERHUB_SERVER_NAME', 'value': 'meena'}, {'name': 'JUPYTERHUB_API_URL', 'value': 'http://10.86.124.187:8081/hub/api'}, {'name': 'JUPYTERHUB_ACTIVITY_URL', 'value': 'http://10.86.124.187:8081/hub/api/users/jovyan/activity'}, {'name': 'JUPYTERHUB_BASE_URL', 'value': '/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_SERVICE_PREFIX', 'value': 'http://10.86.124.187:8081/'}, {'name': 'JUPYTERHUB_SERVICE_URL', 'value': 'http://:10.86.124.187:8081/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_ROOT_DIR', 'value': '/home/jovyan'}, {'name': 'JUPYTERHUB_DEBUG', 'value': '1'}]}], 'inferenceAcceleratorOverrides': []}, 'startedBy': 'jovyan', 'tags': [], 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'taskDefinitionArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task-definition/JupyterNotebook:87', 'version': 1}], 'failures': [], 'ResponseMetadata': {'RequestId': 'deeef790-dd5b-4ac1-849e-771b1f54c8a4', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'deeef790-dd5b-4ac1-849e-771b1f54c8a4', 'content-type': 'application/x-amz-json-1.1', 'content-length': '2755', 'date': 'Mon, 28 Feb 2022 02:18:22 GMT'}, 'RetryAttempts': 0}}
[I 2022-02-28 15:18:22.594 JupyterHub Spawner:283] Waiting for container instance status to move from PENDING to RUNNING
[D 2022-02-28 15:18:22.690 JupyterHub scopes:491] Checking access via scope servers
[D 2022-02-28 15:18:22.690 JupyterHub scopes:389] Unrestricted access to /hub/spawn-pending/jovyan/meena via servers
[I 2022-02-28 15:18:22.690 JupyterHub pages:405] jovyan:meena is pending spawn
[I 2022-02-28 15:18:22.694 JupyterHub log:189] 200 GET /hub/spawn-pending/jovyan/meena (jovyan@::ffff:10.86.124.143) 6.89ms
[D 2022-02-28 15:18:23.185 JupyterHub scopes:491] Checking access via scope read:servers
[D 2022-02-28 15:18:23.185 JupyterHub scopes:389] Unrestricted access to /hub/api/users/jovyan/servers/meena/progress via read:servers
[D 2022-02-28 15:18:23.617 JupyterHub Spawner:273] ecs:DescribeTasks = {'tasks': [{'attachments': [], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'ap-southeast-2b','clusterArn': 'arn:aws:ecs:ap-southeast-2:035973170720:cluster/dap-bdc-nonprod-dev-01-datalab-ECSCluster', 'containerInstanceArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container-instance/dap-bdc-nonprod-dev-01-datalab-ECSCluster/f027356e1ba74be1a68809cb009ad713', 'containers': [{'containerArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99/4f09ff3e-7855-43be-9ed4-08a70dc692aa', 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'name': 'jupyternotebook', 'image': '035973170720.dkr.ecr.ap-southeast-2.amazonaws.com/dap-bdc-nonprod-dev-01-datalab-ecr:0.87', 'lastStatus': 'PENDING', 'networkInterfaces': [], 'healthStatus': 'UNKNOWN', 'cpu': '0'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 2, 28, 15, 18, 21, 490000, tzinfo=tzlocal()), 'desiredStatus': 'RUNNING', 'enableExecuteCommand': False, 'group': 'family:JupyterNotebook', 'healthStatus': 'UNKNOWN', 'lastStatus': 'PENDING', 'launchType': 'EC2', 'memory': '512', 'overrides': {'containerOverrides': [{'name': 'jupyternotebook', 'environment': [{'name': 'PATH', 'value': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/bin'}, {'name': 'LANG', 'value': 'en_US.UTF-8'}, {'name': 'JUPYTERHUB_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JPY_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JUPYTERHUB_ADMIN_ACCESS', 'value': '1'}, {'name': 'JUPYTERHUB_CLIENT_ID', 'value': 'jupyterhub-user-jovyan-meena'}, {'name': 'JUPYTERHUB_HOST', 'value': ''}, {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL', 'value': '/user/jovyan/meena/oauth_callback'}, {'name': 'JUPYTERHUB_OAUTH_SCOPES', 'value': '["access:servers!server=jovyan/meena", "access:servers!user=jovyan"]'}, {'name': 'JUPYTERHUB_USER', 'value': 'jovyan'}, {'name': 'JUPYTERHUB_SERVER_NAME', 'value': 'meena'}, {'name': 'JUPYTERHUB_API_URL', 'value': 'http://10.86.124.187:8081/hub/api'}, {'name': 'JUPYTERHUB_ACTIVITY_URL', 'value': 'http://10.86.124.187:8081/hub/api/users/jovyan/activity'}, {'name': 'JUPYTERHUB_BASE_URL', 'value': '/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_SERVICE_PREFIX', 'value': 'http://10.86.124.187:8081/'}, {'name': 'JUPYTERHUB_SERVICE_URL', 'value': 'http://:10.86.124.187:8081/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_ROOT_DIR', 'value': '/home/jovyan'}, {'name': 'JUPYTERHUB_DEBUG', 'value': '1'}]}], 'inferenceAcceleratorOverrides': []}, 'startedBy': 'jovyan', 'tags': [], 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'taskDefinitionArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task-definition/JupyterNotebook:87', 'version': 1}], 'failures': [], 'ResponseMetadata': {'RequestId': '5170c5b5-36b7-4f69-9271-715c5554a510', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '5170c5b5-36b7-4f69-9271-715c5554a510', 'content-type': 'application/x-amz-json-1.1', 'content-length': '2755', 'date': 'Mon, 28 Feb 2022 02:18:23 GMT'}, 'RetryAttempts': 0}}
[I 2022-02-28 15:18:23.617 JupyterHub Spawner:283] Waiting for container instance status to move from PENDING to RUNNING
[D 2022-02-28 15:18:24.641 JupyterHub Spawner:273] ecs:DescribeTasks = {'tasks': [{'attachments': [], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'ap-southeast-2b','clusterArn': 'arn:aws:ecs:ap-southeast-2:035973170720:cluster/dap-bdc-nonprod-dev-01-datalab-ECSCluster', 'containerInstanceArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container-instance/dap-bdc-nonprod-dev-01-datalab-ECSCluster/f027356e1ba74be1a68809cb009ad713', 'containers': [{'containerArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99/4f09ff3e-7855-43be-9ed4-08a70dc692aa', 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'name': 'jupyternotebook', 'image': '035973170720.dkr.ecr.ap-southeast-2.amazonaws.com/dap-bdc-nonprod-dev-01-datalab-ecr:0.87', 'lastStatus': 'PENDING', 'networkInterfaces': [], 'healthStatus': 'UNKNOWN', 'cpu': '0'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 2, 28, 15, 18, 21, 490000, tzinfo=tzlocal()), 'desiredStatus': 'RUNNING', 'enableExecuteCommand': False, 'group': 'family:JupyterNotebook', 'healthStatus': 'UNKNOWN', 'lastStatus': 'PENDING', 'launchType': 'EC2', 'memory': '512', 'overrides': {'containerOverrides': [{'name': 'jupyternotebook', 'environment': [{'name': 'PATH', 'value': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/bin'}, {'name': 'LANG', 'value': 'en_US.UTF-8'}, {'name': 'JUPYTERHUB_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JPY_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JUPYTERHUB_ADMIN_ACCESS', 'value': '1'}, {'name': 'JUPYTERHUB_CLIENT_ID', 'value': 'jupyterhub-user-jovyan-meena'}, {'name': 'JUPYTERHUB_HOST', 'value': ''}, {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL', 'value': '/user/jovyan/meena/oauth_callback'}, {'name': 'JUPYTERHUB_OAUTH_SCOPES', 'value': '["access:servers!server=jovyan/meena", "access:servers!user=jovyan"]'}, {'name': 'JUPYTERHUB_USER', 'value': 'jovyan'}, {'name': 'JUPYTERHUB_SERVER_NAME', 'value': 'meena'}, {'name': 'JUPYTERHUB_API_URL', 'value': 'http://10.86.124.187:8081/hub/api'}, {'name': 'JUPYTERHUB_ACTIVITY_URL', 'value': 'http://10.86.124.187:8081/hub/api/users/jovyan/activity'}, {'name': 'JUPYTERHUB_BASE_URL', 'value': '/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_SERVICE_PREFIX', 'value': 'http://10.86.124.187:8081/'}, {'name': 'JUPYTERHUB_SERVICE_URL', 'value': 'http://:10.86.124.187:8081/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_ROOT_DIR', 'value': '/home/jovyan'}, {'name': 'JUPYTERHUB_DEBUG', 'value': '1'}]}], 'inferenceAcceleratorOverrides': []}, 'startedBy': 'jovyan', 'tags': [], 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'taskDefinitionArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task-definition/JupyterNotebook:87', 'version': 1}], 'failures': [], 'ResponseMetadata': {'RequestId': 'eaeda183-00bd-4742-9fdc-4496abc5eac0', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'eaeda183-00bd-4742-9fdc-4496abc5eac0', 'content-type': 'application/x-amz-json-1.1', 'content-length': '2755', 'date': 'Mon, 28 Feb 2022 02:18:24 GMT'}, 'RetryAttempts': 0}}
[I 2022-02-28 15:18:24.641 JupyterHub Spawner:283] Waiting for container instance status to move from PENDING to RUNNING
4

Part 2 - JupyterHubLog

[D 2022-02-28 15:18:25.663 JupyterHub Spawner:273] ecs:DescribeTasks = {'tasks': [{'attachments': [], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'ap-southeast-2b','clusterArn': 'arn:aws:ecs:ap-southeast-2:035973170720:cluster/dap-bdc-nonprod-dev-01-datalab-ECSCluster', 'connectivity': 'CONNECTED', 'connectivityAt': datetime.datetime(2022, 2, 28, 15, 18, 21, 490000, tzinfo=tzlocal()), 'containerInstanceArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container-instance/dap-bdc-nonprod-dev-01-datalab-ECSCluster/f027356e1ba74be1a68809cb009ad713', 'containers': [{'containerArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99/4f09ff3e-7855-43be-9ed4-08a70dc692aa', 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'name': 'jupyternotebook', 'image': '035973170720.dkr.ecr.ap-southeast-2.amazonaws.com/dap-bdc-nonprod-dev-01-datalab-ecr:0.87', 'imageDigest': 'sha256:7afde9c9aa82c7083699c060b35da8a793db55d2808c30cf2b9955b60d531aff', 'runtimeId': '19eee3a87f382be993d265584060cdfafc5975b2398c8c69a8b06201b0ff98cc', 'lastStatus': 'RUNNING', 'networkBindings': [{'bindIP': '0.0.0.0', 'containerPort': 8888, 'hostPort': 49240, 'protocol': 'tcp'}], 'networkInterfaces': [], 'healthStatus': 'UNKNOWN', 'cpu': '0'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 2, 28, 15, 18, 21, 490000, tzinfo=tzlocal()), 'desiredStatus': 'RUNNING', 'enableExecuteCommand': False, 'group': 'family:JupyterNotebook', 'healthStatus': 'UNKNOWN', 'lastStatus': 'RUNNING', 'launchType':'EC2', 'memory': '512', 'overrides': {'containerOverrides': [{'name': 'jupyternotebook', 'environment': [{'name': 'PATH', 'value': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/bin'}, {'name': 'LANG', 'value': 'en_US.UTF-8'}, {'name': 'JUPYTERHUB_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JPY_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JUPYTERHUB_ADMIN_ACCESS', 'value': '1'}, {'name': 'JUPYTERHUB_CLIENT_ID', 'value': 'jupyterhub-user-jovyan-meena'}, {'name': 'JUPYTERHUB_HOST', 'value': ''}, {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL', 'value': '/user/jovyan/meena/oauth_callback'}, {'name': 'JUPYTERHUB_OAUTH_SCOPES', 'value': '["access:servers!server=jovyan/meena", "access:servers!user=jovyan"]'}, {'name': 'JUPYTERHUB_USER', 'value': 'jovyan'}, {'name': 'JUPYTERHUB_SERVER_NAME', 'value': 'meena'}, {'name': 'JUPYTERHUB_API_URL', 'value': 'http://10.86.124.187:8081/hub/api'}, {'name': 'JUPYTERHUB_ACTIVITY_URL', 'value': 'http://10.86.124.187:8081/hub/api/users/jovyan/activity'}, {'name': 'JUPYTERHUB_BASE_URL', 'value': '/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_SERVICE_PREFIX', 'value': 'http://10.86.124.187:8081/'}, {'name': 'JUPYTERHUB_SERVICE_URL', 'value': 'http://:10.86.124.187:8081/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_ROOT_DIR', 'value': '/home/jovyan'}, {'name': 'JUPYTERHUB_DEBUG', 'value': '1'}]}], 'inferenceAcceleratorOverrides': []}, 'startedAt': datetime.datetime(2022, 2, 28, 15, 18, 24, 771000, tzinfo=tzlocal()), 'startedBy': 'jovyan', 'tags': [], 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99','taskDefinitionArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task-definition/JupyterNotebook:87', 'version': 2}], 'failures': [], 'ResponseMetadata': {'RequestId': '5ab0c01f-e11c-48a2-a05a-a7fc21eee271', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '5ab0c01f-e11c-48a2-a05a-a7fc21eee271', 'content-type': 'application/x-amz-json-1.1', 'content-length': '3107', 'date': 'Mon, 28 Feb 2022 02:18:25 GMT'}, 'RetryAttempts': 0}}
[D 2022-02-28 15:18:25.663 JupyterHub Spawner:291] ecs:DescribeTasks = {'tasks': [{'attachments': [], 'attributes': [{'name': 'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'ap-southeast-2b','clusterArn': 'arn:aws:ecs:ap-southeast-2:035973170720:cluster/dap-bdc-nonprod-dev-01-datalab-ECSCluster', 'connectivity': 'CONNECTED', 'connectivityAt': datetime.datetime(2022, 2, 28, 15, 18, 21, 490000, tzinfo=tzlocal()), 'containerInstanceArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container-instance/dap-bdc-nonprod-dev-01-datalab-ECSCluster/f027356e1ba74be1a68809cb009ad713', 'containers': [{'containerArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99/4f09ff3e-7855-43be-9ed4-08a70dc692aa', 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99', 'name': 'jupyternotebook', 'image': '035973170720.dkr.ecr.ap-southeast-2.amazonaws.com/dap-bdc-nonprod-dev-01-datalab-ecr:0.87', 'imageDigest': 'sha256:7afde9c9aa82c7083699c060b35da8a793db55d2808c30cf2b9955b60d531aff', 'runtimeId': '19eee3a87f382be993d265584060cdfafc5975b2398c8c69a8b06201b0ff98cc', 'lastStatus': 'RUNNING', 'networkBindings': [{'bindIP': '0.0.0.0', 'containerPort': 8888, 'hostPort': 49240, 'protocol': 'tcp'}], 'networkInterfaces': [], 'healthStatus': 'UNKNOWN', 'cpu': '0'}], 'cpu': '1024', 'createdAt': datetime.datetime(2022, 2, 28, 15, 18, 21, 490000, tzinfo=tzlocal()), 'desiredStatus': 'RUNNING', 'enableExecuteCommand': False, 'group': 'family:JupyterNotebook', 'healthStatus': 'UNKNOWN', 'lastStatus': 'RUNNING', 'launchType':'EC2', 'memory': '512', 'overrides': {'containerOverrides': [{'name': 'jupyternotebook', 'environment': [{'name': 'PATH', 'value': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/bin'}, {'name': 'LANG', 'value': 'en_US.UTF-8'}, {'name': 'JUPYTERHUB_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JPY_API_TOKEN', 'value': 'e69d40ff8b124da19c8e2365b10485c0'}, {'name': 'JUPYTERHUB_ADMIN_ACCESS', 'value': '1'}, {'name': 'JUPYTERHUB_CLIENT_ID', 'value': 'jupyterhub-user-jovyan-meena'}, {'name': 'JUPYTERHUB_HOST', 'value': ''}, {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL', 'value': '/user/jovyan/meena/oauth_callback'}, {'name': 'JUPYTERHUB_OAUTH_SCOPES', 'value': '["access:servers!server=jovyan/meena", "access:servers!user=jovyan"]'}, {'name': 'JUPYTERHUB_USER', 'value': 'jovyan'}, {'name': 'JUPYTERHUB_SERVER_NAME', 'value': 'meena'}, {'name': 'JUPYTERHUB_API_URL', 'value': 'http://10.86.124.187:8081/hub/api'}, {'name': 'JUPYTERHUB_ACTIVITY_URL', 'value': 'http://10.86.124.187:8081/hub/api/users/jovyan/activity'}, {'name': 'JUPYTERHUB_BASE_URL', 'value': '/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_SERVICE_PREFIX', 'value': 'http://10.86.124.187:8081/'}, {'name': 'JUPYTERHUB_SERVICE_URL', 'value': 'http://:10.86.124.187:8081/user/jovyan/meena/'}, {'name': 'JUPYTERHUB_ROOT_DIR', 'value': '/home/jovyan'}, {'name': 'JUPYTERHUB_DEBUG', 'value': '1'}]}], 'inferenceAcceleratorOverrides': []}, 'startedAt': datetime.datetime(2022, 2, 28, 15, 18, 24, 771000, tzinfo=tzlocal()), 'startedBy': 'jovyan', 'tags': [], 'taskArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task/dap-bdc-nonprod-dev-01-datalab-ECSCluster/173200683d784b5885cf5b5af32abb99','taskDefinitionArn': 'arn:aws:ecs:ap-southeast-2:035973170720:task-definition/JupyterNotebook:87', 'version': 2}], 'failures': [], 'ResponseMetadata': {'RequestId': '5ab0c01f-e11c-48a2-a05a-a7fc21eee271', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '5ab0c01f-e11c-48a2-a05a-a7fc21eee271', 'content-type': 'application/x-amz-json-1.1', 'content-length': '3107', 'date': 'Mon, 28 Feb 2022 02:18:25 GMT'}, 'RetryAttempts': 0}}
[I 2022-02-28 15:18:25.663 JupyterHub Spawner:301] Container running on port 49240
[D 2022-02-28 15:18:25.686 JupyterHub Spawner:307] ecs:DescribeContainerInstances = {'containerInstances': [{'containerInstanceArn': 'arn:aws:ecs:ap-southeast-2:035973170720:container-instance/dap-bdc-nonprod-dev-01-datalab-ECSCluster/f027356e1ba74be1a68809cb009ad713', 'ec2InstanceId': 'i-093a2a7c7ad9ecf69', 'version': 11408, 'versionInfo': {'agentVersion': '1.57.1', 'agentHash': '089b7b64', 'dockerVersion': 'DockerVersion: 20.10.12'}, 'remainingResources': [{'name': 'CPU', 'type': 'INTEGER', 'doubleValue': 0.0, 'longValue': 0, 'integerValue': 3072}, {'name': 'MEMORY', 'type': 'INTEGER', 'doubleValue': 0.0, 'longValue': 0, 'integerValue': 30523}, {'name': 'PORTS', 'type': 'STRINGSET', 'doubleValue': 0.0, 'longValue': 0, 'integerValue': 0, 'stringSetValue': ['22', '2376', '2375', '51678', '51679']}, {'name': 'PORTS_UDP', 'type':'STRINGSET', 'doubleValue': 0.0, 'longValue': 0, 'integerValue': 0, 'stringSetValue': []}], 'registeredResources': [{'name': 'CPU', 'type': 'INTEGER', 'doubleValue': 0.0, 'longValue': 0, 'integerValue': 4096},{'name': 'MEMORY', 'type': 'INTEGER', 'doubleValue': 0.0, 'longValue': 0, 'integerValue': 31547}, {'name': 'PORTS', 'type': 'STRINGSET', 'doubleValue': 0.0, 'longValue': 0, 'integerValue': 0, 'stringSetValue':['22', '2376', '2375', '51678', '51679']}, {'name': 'PORTS_UDP', 'type': 'STRINGSET', 'doubleValue': 0.0, 'longValue': 0, 'integerValue': 0, 'stringSetValue': []}], 'status': 'ACTIVE', 'agentConnected': True, 'runningTasksCount': 2, 'pendingTasksCount': 0, 'attributes': [{'name': 'ecs.capability.secrets.asm.environment-variables'}, {'name': 'ecs.capability.branch-cni-plugin-version', 'value': '199bfc65-'}, {'name': 'ecs.ami-id', 'value': 'ami-0885384fdf0a73e7e'}, {'name': 'ecs.capability.secrets.asm.bootstrap.log-driver'}, {'name': 'ecs.capability.task-eia.optimized-cpu'}, {'name': 'ecs.capability.ecr-endpoint'}, {'name':'ecs.capability.docker-plugin.local'}, {'name': 'ecs.capability.task-cpu-mem-limit'}, {'name': 'ecs.capability.secrets.ssm.bootstrap.log-driver'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.30'}, {'name': 'ecs.capability.full-sync'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.31'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.32'}, {'name': 'ecs.capability.firelens.options.config.file'}, {'name': 'ecs.capability.logging-driver.awsfirelens.log-driver-buffer-limit'}, {'name': 'ecs.availability-zone', 'value': 'ap-southeast-2b'}, {'name': 'ecs.capability.aws-appmesh'}, {'name': 'com.amazonaws.ecs.capability.logging-driver.awslogs'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.24'}, {'name': 'ecs.capability.task-eni-trunking'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.25'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.26'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.27'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.28'}, {'name': 'com.amazonaws.ecs.capability.privileged-container'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.29'}, {'name': 'ecs.cpu-architecture', 'value': 'x86_64'}, {'name': 'com.amazonaws.ecs.capability.ecr-auth'}, {'name': 'ecs.capability.firelens.fluentbit'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.20'}, {'name': 'ecs.os-type', 'value': 'linux'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.21'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.22'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.23'}, {'name': 'ecs.capability.private-registry-authentication.secretsmanager'}, {'name': 'ecs.capability.task-eia'}, {'name': 'com.amazonaws.ecs.capability.logging-driver.awsfirelens'}, {'name': 'ecs.capability.firelens.options.config.s3'}, {'name': 'com.amazonaws.ecs.capability.logging-driver.json-file'}, {'name': 'ecs.capability.execution-role-awslogs'}, {'name': 'ecs.vpc-id', 'value': 'vpc-0aa2edd2d3c8607ba'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.17'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.18'}, {'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.19'}, {'name': 'ecs.capability.task-eni'}, {'name': 'ecs.capability.firelens.fluentd'}, {'name': 'ecs.capability.efs'}, {'name': 'ecs.capability.execution-role-ecr-pull'}, {'name': 'ecs.capability.task-eni.ipv6'}, {'name': 'ecs.capability.container-health-check'}, {'name': 'ecs.os-family', 'value': 'LINUX'}, {'name': 'ecs.subnet-id', 'value': 'subnet-0de64cea50664c480'}, {'name': 'ecs.instance-type', 'value': 'r5.xlarge'}, {'name': 'com.amazonaws.ecs.capability.task-iam-role-network-host'}, {'name': 'ecs.capability.container-ordering'}, {'name': 'ecs.capability.cni-plugin-version', 'value': '55b2ae77-2020.09.0'}, {'name': 'ecs.capability.env-files.s3'}, {'name': 'ecs.capability.secrets.ssm.environment-variables'}, {'name': 'ecs.capability.pid-ipc-namespace-sharing'}, {'name': 'com.amazonaws.ecs.capability.task-iam-role'}], 'registeredAt': datetime.datetime(2021, 12, 16, 16, 15, 43, 183000, tzinfo=tzlocal()), 'attachments': [], 'tags': []}], 'failures': [], 'ResponseMetadata': {'RequestId': '688c350d-467c-46c0-979f-849b9dcf726f', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '688c350d-467c-46c0-979f-849b9dcf726f', 'content-type': 'application/x-amz-json-1.1', 'content-length': '4693', 'date': 'Mon, 28 Feb 2022 02:18:25 GMT'}, 'RetryAttempts': 0}}
[I 2022-02-28 15:18:25.686 JupyterHub Spawner:314] Fetching IP addresses of instance: i-093a2a7c7ad9ecf69
[D 2022-02-28 15:18:25.803 JupyterHub Spawner:316] ec2:DescribeInstances = {'Reservations': [{'Groups': [], 'Instances': [{'AmiLaunchIndex': 0, 'ImageId': 'ami-0885384fdf0a73e7e', 'InstanceId': 'i-093a2a7c7ad9ecf69', 'InstanceType': 'r5.xlarge', 'LaunchTime': datetime.datetime(2021, 12, 16, 3, 15, 6, tzinfo=tzutc()), 'Monitoring': {'State': 'enabled'}, 'Placement': {'AvailabilityZone': 'ap-southeast-2b', 'GroupName': '', 'Tenancy': 'default'}, 'PrivateDnsName': 'ip-10-86-124-110.ap-southeast-2.compute.internal', 'PrivateIpAddress': '10.86.124.110', 'ProductCodes': [], 'PublicDnsName': '', 'State': {'Code': 16, 'Name': 'running'}, 'StateTransitionReason': '', 'SubnetId': 'subnet-0de64cea50664c480', 'VpcId': 'vpc-0aa2edd2d3c8607ba', 'Architecture': 'x86_64', 'BlockDeviceMappings': [{'DeviceName': '/dev/sda1', 'Ebs': {'AttachTime': datetime.datetime(2021, 12, 16, 3, 15, 6, tzinfo=tzutc()), 'DeleteOnTermination': True, 'Status': 'attached', 'VolumeId': 'vol-08ba6aef8ef0ee97d'}}, {'DeviceName': '/dev/xvda', 'Ebs': {'AttachTime': datetime.datetime(2021, 12, 16, 3, 15, 6, tzinfo=tzutc()), 'DeleteOnTermination': True, 'Status': 'attached', 'VolumeId': 'vol-096c2ef57d1de8dfa'}}], 'ClientToken': '5b45f704-f1d0-f526-0696-22668b636e4e', 'EbsOptimized': False, 'EnaSupport': True, 'Hypervisor': 'xen', 'IamInstanceProfile': {'Arn': 'arn:aws:iam::035973170720:instance-profile/dap-bdc-nonprod-dev-01-datalab-ECSClusterRoleInstanceProfile', 'Id': 'AIPAQQYBLAYQCZAKREBD3'}, 'NetworkInterfaces': [{'Attachment': {'AttachTime': datetime.datetime(2021, 12, 16, 3, 15, 6, tzinfo=tzutc()), 'AttachmentId': 'eni-attach-03f1c8cfce11e4143', 'DeleteOnTermination': True, 'DeviceIndex': 0, 'Status': 'attached', 'NetworkCardIndex': 0}, 'Description': '', 'Groups': [{'GroupName': 'VPC Endpoints', 'GroupId': 'sg-01561680b48a23c28'}, {'GroupName': 'Core Services', 'GroupId': 'sg-057423a4fadfb55a0'}, {'GroupName': 'dap-bdc-nonprod-dev-01-datalab-ECSClusterSG', 'GroupId': 'sg-0dd11436fa3d3f4ef'}], 'Ipv6Addresses': [], 'MacAddress': '06:a8:e4:0a:81:72', 'NetworkInterfaceId': 'eni-0b636bbfd6f84ab86', 'OwnerId': '035973170720', 'PrivateDnsName': 'ip-10-86-124-110.ap-southeast-2.compute.internal', 'PrivateIpAddress': '10.86.124.110', 'PrivateIpAddresses': [{'Primary': True, 'PrivateDnsName': 'ip-10-86-124-110.ap-southeast-2.compute.internal', 'PrivateIpAddress': '10.86.124.110'}], 'SourceDestCheck': True, 'Status': 'in-use', 'SubnetId': 'subnet-0de64cea50664c480', 'VpcId': 'vpc-0aa2edd2d3c8607ba', 'InterfaceType': 'interface'}], 'RootDeviceName': '/dev/sda1', 'RootDeviceType': 'ebs', 'SecurityGroups': [{'GroupName': 'VPC Endpoints', 'GroupId': 'sg-01561680b48a23c28'}, {'GroupName': 'Core Services', 'GroupId': 'sg-057423a4fadfb55a0'}, {'GroupName': 'dap-bdc-nonprod-dev-01-datalab-ECSClusterSG', 'GroupId': 'sg-0dd11436fa3d3f4ef'}], 'SourceDestCheck': True, 'Tags': [{'Key': 'backup_policy', 'Value': 'NONE'}, {'Key': 'aws:autoscaling:groupName', 'Value': 'dap-bdc-nonprod-dev-01-datalab-ECSLaunchConfig20211216031457370000000001'}], 'VirtualizationType': 'hvm', 'CpuOptions': {'CoreCount': 2, 'ThreadsPerCore': 2}, 'CapacityReservationSpecification': {'CapacityReservationPreference': 'open'}, 'HibernationOptions': {'Configured': False}, 'MetadataOptions': {'State': 'applied', 'HttpTokens': 'optional', 'HttpPutResponseHopLimit': 1, 'HttpEndpoint': 'enabled', 'HttpProtocolIpv6': 'disabled', 'InstanceMetadataTags': 'disabled'}, 'EnclaveOptions': {'Enabled': False}, 'PlatformDetails': 'Red Hat Enterprise Linux', 'UsageOperation': 'RunInstances:0010', 'UsageOperationUpdateTime': datetime.datetime(2021, 12, 16, 3, 15, 6, tzinfo=tzutc()), 'PrivateDnsNameOptions': {'HostnameType': 'ip-name', 'EnableResourceNameDnsARecord': False, 'EnableResourceNameDnsAAAARecord': False}}], 'OwnerId': '035973170720', 'RequesterId': '081202882002', 'ReservationId': 'r-05a7d81b147035c7b'}], 'ResponseMetadata': {'RequestId': '3180879b-84e1-4e60-b587-5345d94a49e3', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '3180879b-84e1-4e60-b587-5345d94a49e3', 'cache-control': 'no-cache, no-store', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'vary': 'accept-encoding', 'content-type': 'text/xml;charset=UTF-8', 'transfer-encoding': 'chunked', 'date': 'Mon, 28 Feb 2022 02:18:24 GMT', 'server': 'AmazonEC2'}, 'RetryAttempts': 0}}
[I 2022-02-28 15:18:25.803 JupyterHub Spawner:325] Container running on ip: 10.86.124.110
[I 2022-02-28 15:18:25.803 JupyterHub Spawner:327] Finished with the start method
[D 2022-02-28 15:18:25.807 JupyterHub spawner:1221] Polling subprocess every 300s
[I 2022-02-28 15:18:26.274 JupyterHub log:189] 200 GET /hub/api (@10.86.124.110) 0.74ms
[D 2022-02-28 15:18:26.286 JupyterHub base:281] Recording first activity for <APIToken('e69d...', user='jovyan', client_id='jupyterhub')>
[D 2022-02-28 15:18:26.291 JupyterHub scopes:301] Authenticated with token <APIToken('e69d...', user='jovyan', client_id='jupyterhub')>
[D 2022-02-28 15:18:26.294 JupyterHub scopes:491] Checking access via scope users:activity
[D 2022-02-28 15:18:26.294 JupyterHub scopes:402] Argument-based access to /hub/api/users/jovyan/activity via users:activity
[D 2022-02-28 15:18:26.295 JupyterHub users:855] Activity for user jovyan: 2022-02-28T02:18:26.013604Z
[D 2022-02-28 15:18:26.295 JupyterHub users:873] Activity on server jovyan/meena: 2022-02-28T02:18:26.013604Z
[I 2022-02-28 15:18:26.300 JupyterHub log:189] 200 POST /hub/api/users/jovyan/activity (jovyan@10.86.124.110) 18.20ms
[D 2022-02-28 15:18:27.618 JupyterHub utils:230] Server at http://10.86.124.110:49240/user/jovyan/meena/ responded with 302
[D 2022-02-28 15:18:27.618 JupyterHub _version:74] jupyterhub and jupyterhub-singleuser both on version 2.1.1
[I 2022-02-28 15:18:27.618 JupyterHub base:944] User jovyan:meena took 6.293 seconds to start
[I 2022-02-28 15:18:27.618 JupyterHub proxy:286] Adding user jovyan to proxy /user/jovyan/meena/ => http://10.86.124.110:49240
[D 2022-02-28 15:18:27.619 JupyterHub proxy:821] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/jovyan/meena
15:18:27.620 [ConfigProxy] info: Adding route /user/jovyan/meena -> http://10.86.124.110:49240
15:18:27.621 [ConfigProxy] info: Route added /user/jovyan/meena -> http://10.86.124.110:49240
15:18:27.621 [ConfigProxy] info: 201 POST /api/routes/user/jovyan/meena
[I 2022-02-28 15:18:27.622 JupyterHub users:749] Server jovyan:meena is ready
[I 2022-02-28 15:18:27.622 JupyterHub log:189] 200 GET /hub/api/users/jovyan/servers/meena/progress (jovyan@::ffff:10.86.124.143) 4439.32ms
[D 2022-02-28 15:18:28.007 JupyterHub scopes:491] Checking access via scope servers
[D 2022-02-28 15:18:28.007 JupyterHub scopes:389] Unrestricted access to /hub/spawn-pending/jovyan/meena via servers
[I 2022-02-28 15:18:28.008 JupyterHub log:189] 302 GET /hub/spawn-pending/jovyan/meena -> /user/jovyan/meena (jovyan@::ffff:10.86.124.143) 3.53ms
[I 2022-02-28 15:18:29.759 JupyterHub log:189] 302 GET /api/oauth2/authorize?client_id=jupyterhub-user-jovyan-meena&redirect_uri=%2Fuser%2Fjovyan%2Fmeena%2Foauth_callback&response_type=code&state=[secret] -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-jovyan-meena&redirect_uri=%2Fuser%2Fjovyan%2Fmeena%2Foauth_callback&response_type=code&state=[secret] (@10.67.118.148) 1.00ms
[W 2022-02-28 15:18:30.098 JupyterHub base:393] Invalid or expired cookie token
[I 2022-02-28 15:18:30.099 JupyterHub log:189] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-jovyan-meena&redirect_uri=%2Fuser%2Fjovyan%2Fmeena%2Foauth_callback&response_type=code&state=[secret] -> /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-jovyan-meena%26redirect_uri%3D%252Fuser%252Fjovyan%252Fmeena%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ (@10.67.118.148) 1.33ms
[I 2022-02-28 15:18:30.716 JupyterHub log:189] 200 GET /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-jovyan-meena%26redirect_uri%3D%252Fuser%252Fjovyan%252Fmeena%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ (@10.67.118.148) 2.04ms
[D 2022-02-28 15:18:31.530 JupyterHub log:189] 304 GET /hub/logo (@10.67.118.148) 0.72ms
[I 2022-02-28 15:18:32.625 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.92) 1.94ms
[I 2022-02-28 15:18:46.003 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.143) 1.91ms
[I 2022-02-28 15:19:02.642 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.92) 1.90ms
[I 2022-02-28 15:19:16.032 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.143) 2.08ms
[I 2022-02-28 15:19:32.673 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.92) 1.88ms
[D 2022-02-28 15:19:41.789 JupyterHub roles:450] Assigning default role to User jovyan
[D 2022-02-28 15:19:41.793 JupyterHub base:559] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
[D 2022-02-28 15:19:41.794 JupyterHub base:563] Setting cookie for jovyan: jupyterhub-hub-login
[D 2022-02-28 15:19:41.794 JupyterHub base:559] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2022-02-28 15:19:41.794 JupyterHub base:797] User logged in: jovyan
[I 2022-02-28 15:19:41.795 JupyterHub log:189] 302 POST /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-jovyan-meena%26redirect_uri%3D%252Fuser%252Fjovyan%252Fmeena%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-jovyan-meena&redirect_uri=%2Fuser%2Fjovyan%2Fmeena%2Foauth_callback&response_type=code&state=[secret] (jovyan@10.67.118.148) 30.50ms
[D 2022-02-28 15:19:42.162 JupyterHub provider:411] Validating client id jupyterhub-user-jovyan-meena
[D 2022-02-28 15:19:42.165 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:363] Validating redirection uri /user/jovyan/meena/oauth_callback for client jupyterhub-user-jovyan-meena.
[D 2022-02-28 15:19:42.165 oauthlib.oauth2.rfc6749.grant_types.base base:230] Using provided redirect_uri /user/jovyan/meena/oauth_callback
[D 2022-02-28 15:19:42.165 JupyterHub provider:486] validate_redirect_uri: client_id=jupyterhub-user-jovyan-meena, redirect_uri=/user/jovyan/meena/oauth_callback
[D 2022-02-28 15:19:42.166 oauthlib.oauth2.rfc6749.grant_types.base base:171] Validating access to scopes [] for client 'jupyterhub-user-jovyan-meena' (<OAuthClient(identifier='jupyterhub-user-jovyan-meena')>).
[D 2022-02-28 15:19:42.167 JupyterHub provider:570] Allowing request for role(s) for jupyterhub-user-jovyan-meena:  []
[D 2022-02-28 15:19:42.168 JupyterHub auth:287] Skipping oauth confirmation for <User(jovyan 1/2 running)> accessing Server at /user/jovyan/meena/
[D 2022-02-28 15:19:42.168 oauthlib.oauth2.rfc6749.endpoints.authorization authorization:98] Dispatching response_type code request to <oauthlib.oauth2.rfc6749.grant_types.authorization_code.AuthorizationCodeGrant object at 0x7f7c56ecaeb0>.
[D 2022-02-28 15:19:42.168 JupyterHub provider:411] Validating client id jupyterhub-user-jovyan-meena
[D 2022-02-28 15:19:42.169 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:363] Validating redirection uri /user/jovyan/meena/oauth_callback for client jupyterhub-user-jovyan-meena.
[D 2022-02-28 15:19:42.169 oauthlib.oauth2.rfc6749.grant_types.base base:230] Using provided redirect_uri /user/jovyan/meena/oauth_callback
[D 2022-02-28 15:19:42.169 JupyterHub provider:486] validate_redirect_uri: client_id=jupyterhub-user-jovyan-meena, redirect_uri=/user/jovyan/meena/oauth_callback
[D 2022-02-28 15:19:42.170 oauthlib.oauth2.rfc6749.grant_types.base base:171] Validating access to scopes [] for client 'jupyterhub-user-jovyan-meena' (<OAuthClient(identifier='jupyterhub-user-jovyan-meena')>).
[D 2022-02-28 15:19:42.171 JupyterHub provider:570] Allowing request for role(s) for jupyterhub-user-jovyan-meena:  []
[D 2022-02-28 15:19:42.171 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:246] Pre resource owner authorization validation ok for <oauthlib.Request SANITIZED>.
[D 2022-02-28 15:19:42.171 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:171] Created authorization code grant {'code': 'Ftqp1sWG75oILvsr7Qd00q5OEXe8cf', 'state': 'eyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ'} for request <oauthlib.Request SANITIZED>.
[D 2022-02-28 15:19:42.171 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:278] Saving grant {'code': 'Ftqp1sWG75oILvsr7Qd00q5OEXe8cf', 'state': 'eyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ'} for <oauthlib.Request SANITIZED>.
[D 2022-02-28 15:19:42.171 JupyterHub provider:237] Saving authorization code jupyterhub-user-jovyan-meena, Ftq..., (), {}
[I 2022-02-28 15:19:42.179 JupyterHub log:189] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-jovyan-meena&redirect_uri=%2Fuser%2Fjovyan%2Fmeena%2Foauth_callback&response_type=code&state=[secret] -> /user/jovyan/meena/oauth_callback?code=[secret]&state=[secret] (jovyan@10.67.118.148) 23.49ms
[I 2022-02-28 15:19:42.528 JupyterHub log:189] 302 GET /user/jovyan/meena/oauth_callback?code=[secret]&state=[secret] -> /hub/user/jovyan/meena/oauth_callback?code=[secret]&state=[secret] (@10.67.118.148) 0.79ms
[D 2022-02-28 15:19:42.875 JupyterHub scopes:491] Checking access via scope access:servers
[D 2022-02-28 15:19:42.875 JupyterHub scopes:389] Unrestricted access to /hub/user/jovyan/meena/oauth_callback via access:servers
[I 2022-02-28 15:19:42.876 JupyterHub log:189] 302 GET /hub/user/jovyan/meena/oauth_callback?code=[secret]&state=[secret] -> /user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=1 (jovyan@10.67.118.148) 3.99ms
[I 2022-02-28 15:19:43.221 JupyterHub log:189] 302 GET /user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=1 -> /hub/user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=1 (@10.67.118.148) 0.77ms
[D 2022-02-28 15:19:43.563 JupyterHub scopes:491] Checking access via scope access:servers
[D 2022-02-28 15:19:43.563 JupyterHub scopes:389] Unrestricted access to /hub/user/jovyan/meena/oauth_callback via access:servers
[W 2022-02-28 15:19:43.563 JupyterHub base:1600] Redirect loop detected on /hub/user/jovyan/meena/oauth_callback?code=Ftqp1sWG75oILvsr7Qd00q5OEXe8cf&state=eyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ&redirects=1
[I 2022-02-28 15:19:45.566 JupyterHub log:189] 302 GET /hub/user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=1 -> /user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=2 (jovyan@10.67.118.148) 2005.47ms
[I 2022-02-28 15:19:45.906 JupyterHub log:189] 302 GET /user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=2 -> /hub/user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=2 (@10.67.118.148) 0.79ms
[I 2022-02-28 15:19:46.060 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.143) 2.23ms
[D 2022-02-28 15:19:46.253 JupyterHub scopes:491] Checking access via scope access:servers
[D 2022-02-28 15:19:46.253 JupyterHub scopes:389] Unrestricted access to /hub/user/jovyan/meena/oauth_callback via access:servers
[W 2022-02-28 15:19:46.254 JupyterHub base:1600] Redirect loop detected on /hub/user/jovyan/meena/oauth_callback?code=Ftqp1sWG75oILvsr7Qd00q5OEXe8cf&state=eyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ&redirects=2
[I 2022-02-28 15:19:50.260 JupyterHub log:189] 302 GET /hub/user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=2 -> /user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=3 (jovyan@10.67.118.148) 4008.84ms
[I 2022-02-28 15:19:50.620 JupyterHub log:189] 302 GET /user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=3 -> /hub/user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=3 (@10.67.118.148) 0.80ms
[D 2022-02-28 15:19:50.964 JupyterHub scopes:491] Checking access via scope access:servers
[D 2022-02-28 15:19:50.964 JupyterHub scopes:389] Unrestricted access to /hub/user/jovyan/meena/oauth_callback via access:servers
[W 2022-02-28 15:19:50.965 JupyterHub base:1600] Redirect loop detected on /hub/user/jovyan/meena/oauth_callback?code=Ftqp1sWG75oILvsr7Qd00q5OEXe8cf&state=eyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ&redirects=3
[I 2022-02-28 15:19:58.970 JupyterHub log:189] 302 GET /hub/user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=3 -> /user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=4 (jovyan@10.67.118.148) 8007.33ms
[I 2022-02-28 15:19:59.310 JupyterHub log:189] 302 GET /user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=4 -> /hub/user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=4 (@10.67.118.148) 0.77ms
[D 2022-02-28 15:19:59.651 JupyterHub scopes:491] Checking access via scope access:servers
[D 2022-02-28 15:19:59.651 JupyterHub scopes:389] Unrestricted access to /hub/user/jovyan/meena/oauth_callback via access:servers
[W 2022-02-28 15:19:59.652 JupyterHub web:1787] 500 GET /hub/user/jovyan/meena/oauth_callback?code=Ftqp1sWG75oILvsr7Qd00q5OEXe8cf&state=eyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ&redirects=4 (10.67.118.148): Redirect loop detected.
[D 2022-02-28 15:19:59.652 JupyterHub base:1323] No template for 500
[E 2022-02-28 15:19:59.676 JupyterHub log:181] {
      "Host": "10.86.124.187:8081",
      "Connection": "keep-alive",
      "Cache-Control": "max-age=0",
      "Upgrade-Insecure-Requests": "1",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "Referer": "http://10.86.124.187:8081/hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-jovyan-meena%26redirect_uri%3D%252Fuser%252Fjovyan%252Fmeena%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ",
      "Accept-Encoding": "gzip, deflate",
      "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
      "Cookie": "jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]"
    }
[E 2022-02-28 15:19:59.676 JupyterHub log:189] 500 GET /hub/user/jovyan/meena/oauth_callback?code=[secret]&state=[secret]&redirects=4 (jovyan@10.67.118.148) 26.95ms
[I 2022-02-28 15:20:02.680 JupyterHub log:189] 200 GET /hub/login (@::ffff:10.86.124.92) 1.93ms
5

Customised Spawner -

import os
import pwd
import subprocess
import re
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError
from jupyterhub.spawner import Spawner
from tornado import gen
from traitlets import (
    Unicode,
    Bool,
    Int
)
from .helpers import dict_to_awskv, safeget

#if os.name == 'nt':
#    import psutil
#from .utils import url_path_join

class JHOptionsSpawner(Spawner):
    cluster_name = Unicode("notebook-cluster",
                           help="Name of the cluster setup").tag(config=True)
    task_definition = Unicode("notebook_spawner_task:3",
                              help="The task definition in <defn>:<revision> format").tag(config=True) #Python’s string type uses the Unicode Standard for representing all different possible characters.
    container_name = Unicode("jupyternotebook", help="Container name that is specified in the task definition").tag(config=True)
    expose_public_ip = Bool(False, help="Expose public ip to the hub. If false, then the private IP is returned").tag(config=True)

    aws_region_name = Unicode(None, help="The region to use in instantiating AWS clients", allow_none=True).tag(config=True)

    task_arn = Unicode()
    container_instance_arn = Unicode()
    container_ip = Unicode()
    container_port = Int(0)

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        aws_config = Config(region_name=self.aws_region_name)
        self.ecs_client = boto3.client("ecs", config=aws_config)
        self.ec2_client = boto3.client("ec2", config=aws_config)

    def _options_form_default(self):
        options = []
        return """

                <style type="text/css">
        div.note
        {{
             -moz-border-radius: 6px;
             -webkit-border-radius: 6px;
             background-color: #f0f7fb;
             background-position: 9px 0px;
             background-repeat: no-repeat;
             border: solid 1px #3498db;
             border-radius: 6px;
             line-height: 18px;
             overflow: hidden;
             padding: 15px 60px;
        }}
        </style>

        <div class=note>
        <strong>
            Welcome {user}!
            <br />
            <br />
            BDC provides a Data Lab (DL) to segregate data for an individual
            business use case or model, as agreed in the Data Lab governance
            meeting.
            <br />
            <br />
            If you are a part of more than one Data Lab or need to run more
            than one Data Lab, you can Spawn a new Data Lab with a new
            Server Name.
        </strong>
        </div>
        <p>
            <br />
            <strong>
                Please select the service and Data Lab to associate it
                from the options below to create a new instance
            </strong>
        </p>

        <label>Please select the data lab you would like to connect to</label>

        <select name="datalab_names" size="1">
        <option value="Datalab1" style="color: black;
                font-weight: 700;">Datalab1</option>
        <option value="Datalab2" style="color: black;
                font-weight: 700;">Datalab2</option>
        <option value="Datalab3" style="color: black;
                font-weight: 700;">Datalab3</option>
        <option value="Datalab4" style="color: black;
                font-weight: 700;">Datalab4</option>
        </select>

        """.format(''.join(options), user=self.user.name)

    def options_from_form(self, formdata):

        options = {}
        options['datalab_names'] = formdata['datalab_names']
        return options

    def load_state(self, state):
        """Restore state about spawned single-user server after a hub restart.

        Local processes only need the process id.
        """
        super().load_state(state)
        if 'task_arn' in state:
            self.task_arn = state['task_arn']
        if 'container_instance_arn' in state:
            self.container_instance_arn = state['container_instance_arn']
        if 'container_ip' in state and 'container_port' in state:
            self.container_ip = state['container_ip']
            self.container_port = state['container_port']

    def get_state(self):
        """Save state that is needed to restore this spawner instance after a hub restore.

        Local processes only need the process id.
        """
        state = super().get_state()
        if self.task_arn:
            state['task_arn'] = self.task_arn
        if self.container_instance_arn:
            state['container_instance_arn'] = self.container_instance_arn
        if self.container_ip and self.container_port:
            state['container_ip'] = self.container_ip
            state['container_port'] = self.container_port
        return state

    def clear_state(self):
        """Clear stored state about this spawner (pid)"""
        super().clear_state()
        self.container_ip = ''
        self.container_instance_arn = ''
        self.container_port = 0
        self.task_arn = ''

    def get_env(self):
        env = super().get_env()
        env['JUPYTERHUB_SERVICE_URL'] = 'http://:10.86.124.187:8081/user/jovyan/meena/'
#        env['JUPYTERHUB_SERVICE_URL'] = 'http://10.86.124.187:8081'
#        env['JUPYTERHUB_BASE_URL'] = self.hub.base_url[:-4]
        env['JUPYTERHUB_BASE_URL'] = '/user/jovyan/meena/'
#        env['JUPYTERHUB_BASE_URL'] = ''
#        env['JUPYTERHUB_BASE_URL'] = '//http://10.86.124.187:8081/'
#        env['JUPYTERHUB_BASE_URL'] = '/'
#        env['JUPYTERHUB_SERVICE_PREFIX'] = 'http://10.86.124.187:8081/user/jovyan/meena/'
        env['JUPYTERHUB_SERVICE_PREFIX'] = 'http://10.86.124.187:8081/'
#        env['JUPYTERHUB_SERVICE_PREFIX'] = '/user/jovyan/meena/'

#        env['JUPYTERHUB_SERVICE_URL'] = url_path_join(
#                self.hub_connect_url,
#                self.server.base_url
#            )

        return env


    def setup_overrides(self):

        # Set default overrides
        overrides = {
            'containerOverrides': [
                {
                    'name': self.container_name
                }
            ]
        }
        # Setup rest of overrides
        env = self.get_env()
        overrides['containerOverrides'][0]['environment'] = dict_to_awskv(env)
        return overrides

    @gen.coroutine
    def poll(self):
        try:
            if self.task_arn:
                desc_task_resp = self.ecs_client.describe_tasks(
                    cluster=self.cluster_name,
                    tasks=[self.task_arn]
                )
                self.log.debug("ecs:DescribeTasks = {!s}".format(desc_task_resp))
                status = safeget(desc_task_resp, 'tasks', 0, 'lastStatus')
                if status and status.lower() in ('pending', 'running'):
                    # If the Spawner has not finished starting, it should behave as if it is running (status=None).
                    # See https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.poll for more details.
                    return None
                self.log.warning("ECS task with ARN {} is either in STOPPED state or doesn't exist.".format(self.task_arn))
                self.clear_state()
                # if there was an error in retrieving the status of notebook server or if it's no longer running,
                # we poll should return integer exit code.
                # We can get here if notebook task was stopped outside of Spawner (stopped via console, exited, etc)
                task_exit = safeget(desc_task_resp, 'tasks', 0, 'containers', 0, 'exitCode')
                # 0 must be returned, when state is unknown, but notebook can potentially exit with 0 exit code
                return task_exit if task_exit else 1
            # If the Spawner has not been initialized (neither loaded state, nor called start),
            # it should behave as if it is not running (status=0).
            return 0
        except (KeyError, ClientError) as exp:
            self.log.error("Error calling describe_tasks for task with ARN {}: {}".format(self.task_arn, exp))
            self.clear_state()
            # any error in getting the status of notebook should be reported up as integer error
            return 1

    def is_running(self):
        return self.poll() is None

    @gen.coroutine
    def start(self):
        if not self.is_running():
            self.log.info("Running new task")
            self.log.info(self.cluster_name)
            self.log.info(self.task_definition)
            self.log.info(self.user.name)

            overrides = self.setup_overrides()

            run_task_resp = self.ecs_client.run_task(
                cluster=self.cluster_name,
                taskDefinition=self.task_definition,
                startedBy=self.user.name,
                count=1,
                overrides=overrides
            )
            self.log.debug("ecs:RunTask = {!s}".format(run_task_resp))
            self.log.info("ecs:RunTask = {!s}".format(run_task_resp))

            self.task_arn = safeget(
                run_task_resp,
                'tasks', 0, 'containers', 0, 'taskArn',
                exception="Could not get server task"
            )
            self.container_instance_arn = safeget(
                run_task_resp,
                'tasks', 0, 'containerInstanceArn',
                exception="Could not get server instance"
            )
            self.log.info("Spawned notebook container")

            self.log.info("Fetching task info")

            while True:
                if self.task_arn:
                    desc_task_resp = self.ecs_client.describe_tasks(cluster=self.cluster_name, tasks=[self.task_arn])
                    self.log.debug("ecs:DescribeTasks = {!s}".format(desc_task_resp))
                    last_status = safeget(desc_task_resp, 'tasks', 0, 'containers', 0, 'lastStatus')

                    if last_status == 'RUNNING':  # success
                        break
                    elif last_status == 'STOPPED':  # failure
                        self.log.error("The container unusually stopped")
                        reason = safeget(desc_task_resp, 'tasks', 0, 'containers', 0, 'reason')
                        raise RuntimeError(reason if reason is not None else "Error starting container")
                    else:  # wait
                        self.log.info("Waiting for container instance status to move from %s to RUNNING" % last_status)
                        yield gen.sleep(1)
                else:  # wait
                    self.log.info("Waiting for container instance status to move from %s to RUNNING" % last_status)
                    yield gen.sleep(1)

            # Wait for ECS to send task change event with networkBindings
            while True:
                self.log.debug("ecs:DescribeTasks = {!s}".format(desc_task_resp))
                port = safeget(
                    desc_task_resp,
                    'tasks', 0, 'containers', 0, 'networkBindings', 0, 'hostPort',
                )
                if port is not None:  # success
                    break
                else:  # wait
                    yield gen.sleep(1)
            self.container_port = int(port)
            self.log.info("Container running on port {:d}".format(self.container_port))

            desc_cont_resp = self.ecs_client.describe_container_instances(
                cluster=self.cluster_name,
                containerInstances=[self.container_instance_arn]
            )
            self.log.debug("ecs:DescribeContainerInstances = {!s}".format(desc_cont_resp))
            ec2_instance_id = safeget(
                desc_cont_resp,
                'containerInstances', 0, 'ec2InstanceId',
                exception="Could not determine server host"
            )

            self.log.info("Fetching IP addresses of instance: {}".format(ec2_instance_id))
            desc_inst_resp = self.ec2_client.describe_instances(InstanceIds=[ec2_instance_id])
            self.log.debug("ec2:DescribeInstances = {!s}".format(desc_inst_resp))

            public_ip = safeget(desc_inst_resp, 'Reservations', 0, 'Instances', 0, 'PublicIpAddress')
            private_ip = safeget(
                desc_inst_resp,
                'Reservations', 0, 'Instances', 0, 'PrivateIpAddress',
                exception="Could not determine server ip"
            )
            self.container_ip = public_ip if self.expose_public_ip and public_ip is not None else private_ip
            self.log.info("Container running on ip: {}".format(self.container_ip))

            self.log.info("Finished with the start method")

        # by this time we should have ip and port either from new task or loaded from state
        if not self.container_ip or not self.container_port:
            raise RuntimeError("Error starting container: either port or ip is not set by the time start() is finished.")

        return (self.container_ip, self.container_port)

    @gen.coroutine
    def stop(self, now=False):
        self.log.info("Stopping task {}".format(self.task_arn))
        stop_task_resp = self.ecs_client.stop_task(
            cluster=self.cluster_name,
            task=self.task_arn,
            reason="Shutdown by the hub"
        )
        self.log.debug("ecs:StopTask = {!s}".format(stop_task_resp))
        self.log.info("The requested task has been stopped")
        self.clear_state()
6

JupyterHub Config

# Configuration file for jupyterhub.

import os
import pwd
import subprocess
import socket

c = get_config()

c.JupyterHub.config_file = 'jupyterhub_config.py'

c.JupyterHub.allow_named_servers = True
c.JupyterHub.named_server_limit_per_user = 3
## Number of days for a login cookie to be valid. Default is two weeks.
c.JupyterHub.cookie_max_age_days = 3

## Redirect user to server (if running), instead of control panel.
c.JupyterHub.redirect_to_server = False

from traitlets import default

c.JupyterHub.spawner_class = 'bdcspawner.JHOptionsSpawner'

# Set the log level by value or name.
c.JupyterHub.log_level = 'DEBUG'

# Enable debug-logging of the single-user server
c.Spawner.debug = True
c.Spawner.poll_interval = 300
c.Spawner.start_timeout = 600

c.JupyterHub.admin_access = True

def pre_spawn_hook(JHOptionsSpawner):
    username = JHOptionsSpawner.user.name
    print(username + "is username")
    try:
        pwd.getpwnam(username)
    except KeyError:
        subprocess.check_call(['useradd', '-ms', '/bin/bash', username])
c.JHOptionsSpawner.pre_spawn_hook = pre_spawn_hook

c.PAMAuthenticator.open_sessions = False
c.Spawner.notebook_dir = '/home/jovyan'
c.Spawner.debug = True
c.JupyterHub.hub_connect_ip = socket.gethostbyname(socket.gethostname())
c.Authenticator.admin_users = {'jovyan'}
c.JupyterHub.internal_ssl = False
c.JupyterHub.bind_url = 'http://:8000'

## Generate certs used for internal ssl
c.JupyterHub.generate_certs = False

# ECS Details
c.JHOptionsSpawner.cluster_name = 'dap-bdc-nonprod-dev-01-datalab-ECSCluster'
c.JHOptionsSpawner.task_definition = 'JupyterNotebook:87'
c.JHOptionsSpawner.aws_region_name = 'ap-southeast-2'

# ECS notebook start could take a while - particularly if first image download
c.Spawner.start_timeout = 600
7

Jupyter Notebook Config

from jupyter_core.paths import jupyter_data_dir

import subprocess

import os

import errno

import stat

c = get_config() # noqa: F821

c.NotebookApp.ip = "*"

c.NotebookApp.port = 8888

c.NotebookApp.allow_root = True

c.Session.debug = True

c.NotebookApp.notebook_dir = '/home/jovyan'

# https://github.com/jupyter/notebook/issues/3130

c.FileContentsManager.delete_to_trash = False
9

What happens when a user logs,
STEP 1

User access JupyterHub via Route53, which hits a load balancer and gets to a proxy in port 8000. The user sees this screen,

image
image823×435 37.6 KB

10

STEP 2

Once the user enters the username and password, they are taken to this home page. A user can select up to 3 named servers.

image
image992×351 73.4 KB

11

STEP 3

Once they start a server, they are taken to a pre-spawn page where they can select a datalab,

image

12

STEP 4

The customised spawner kicks in when the user clicks the “start” button in the above screenshot.

The spawner creates the container in the ECS instance and runs the docker image which kick starts the Jupyterhub single-user application for the user. The user is not identified at this stage and Jupyter Notebook contacts JupyterHub for authentication. And that can be seen in this screenshot.

image
image940×202 41.6 KB

The url of the above is,
http://10.86.124.187:8081/hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Djupyterhub-user-jovyan-meena%26redirect_uri%3D%2Fuser%2Fjovyan%2Fmeena%2Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogImIyOGU1Zjg0MDZmYzRmZWU5NjVjNjllZmVkNmNmZDUwIiwgIm5leHRfdXJsIjogIi91c2VyL2pvdnlhbi9tZWVuYS90cmVlLz8ifQ

13

STEP 5

This then leads the user for re-authentication in the hub – Notebook communicates back with Hub through 8081,

image
image919×473 55.1 KB

14

And this is followed by,

image
image851×198 47.2 KB

15

I think the main culprit is what I am seeing in the Notebook log,

image
image940×243 18.6 KB

“Authentication of/metrics is off, since other authentication is disabled”.

I am not sure what I would have done in my config to have caused this and if this is the root cause of the redirect.

Been working on this for the last few weeks and your expert help will be much appreciated. Please sing out if you require more details.

16

I think the root of the issue is your manual override of $JUPYTERHUB_SERVICE environment variables. It is unlikely that you need to override any of these variables, but the current values in your config overrides are not correct because they point to the Hub itself, not the spawned server. You can check the definitions in the docs:

  • JUPYTERHUB_SERVICE_PREFIX is the url path prefix for the server, or /user/meena for user ‘meena’. It’s very unlikely that you should override the default, and it definitely should not be the same for all users.
  • JUPYTERHUB_SERVICE_URL is the bind url of the service, e.g. http://0.0.0.0:8888/user/meena. This is usually set from the spawner.ip and spawner.port attributes, rather than overridden wholesale. I suspect you want to hardcode self.ip = '0.0.0.0' and self.port = 8888 based on your logs.
  • JUPYTERHUB_BASE_URL is the base URL of the whole Hub deployment (the bit of the public URL before hub/). It is usually just '/' (this is the right value for your deployment), but may include the public host name (...thenational.com:8000/). Again, it’s very unlikely that this needs to be overridden.
1 Like
17

Hi there, Thank you for your prompt response to my message.

Please note, ‘jovyan’ is the user and ‘meena’ is the Named Server - as in the screenshots of my previous messages.

I incorporated your suggestion and I am no longer overriding values for JUPYTERHUB_SERVICE_PREFIX and JUPYTERHUB_SERVICE_URL.
I have declared the values of self.ip and self.port in my customised Spawner code as below,

   def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        aws_config = Config(region_name=self.aws_region_name)
        self.ecs_client = boto3.client("ecs", config=aws_config)
        self.ec2_client = boto3.client("ec2", config=aws_config)
        self.ip = '0.0.0.0'
        self.port = 8888

I modified the base_url to use the public host name and that threw up the error below.

env['JUPYTERHUB_BASE_URL'] = '//https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/'

There is ‘/’ before ‘https’ which doesnt take it to the right location I am assuming.

[I 2022-03-03 15:16:12.927 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/style/style.min.css?v=b092d0a2da5df36f2b073ddb4eafcd6c8094c4fa21b6dcd3f7185ce16c04ad66424083d785b81607a26b4b85b69a560574ada7db75262c886655f99d651c482e -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/style/style.min.css?v=b092d0a2da5df36f2b073ddb4eafcd6c8094c4fa21b6dcd3f7185ce16c04ad66424083d785b81607a26b4b85b69a560574ada7db75262c886655f99d651c482e (@::ffff:10.86.124.143) 1.42ms
[I 2022-03-03 15:16:12.927 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 (@::ffff:10.86.124.143) 1.43ms
[I 2022-03-03 15:16:12.929 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb (@::ffff:10.86.124.143) 0.98ms
[I 2022-03-03 15:16:12.929 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a (@::ffff:10.86.124.143) 1.13ms
[I 2022-03-03 15:16:12.938 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@::ffff:10.86.124.143) 0.62ms
[I 2022-03-03 15:16:12.940 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@::ffff:10.86.124.143) 0.61ms
[I 2022-03-03 15:16:12.941 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@::ffff:10.86.124.143) 0.59ms
[I 2022-03-03 15:16:12.943 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/requirejs/require.js?v=d37b48bb2137faa0ab98157e240c084dd5b1b5e74911723aa1d1f04c928c2a03dedf922d049e4815f7e5a369faa2e6b6a1000aae958b7953b5cc60411154f593 -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/requirejs/require.js?v=d37b48bb2137faa0ab98157e240c084dd5b1b5e74911723aa1d1f04c928c2a03dedf922d049e4815f7e5a369faa2e6b6a1000aae958b7953b5cc60411154f593 (@::ffff:10.86.124.143) 0.68ms
[W 2022-03-03 15:16:13.332 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/style/style.min.css?v=b092d0a2da5df36f2b073ddb4eafcd6c8094c4fa21b6dcd3f7185ce16c04ad66424083d785b81607a26b4b85b69a560574ada7db75262c886655f99d651c482e (jovyan@::ffff:10.86.124.143) 27.65ms
[W 2022-03-03 15:16:13.334 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 (jovyan@::ffff:10.86.124.143) 29.68ms
[W 2022-03-03 15:16:13.340 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb (jovyan@::ffff:10.86.124.143) 5.04ms
[W 2022-03-03 15:16:13.342 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a (jovyan@::ffff:10.86.124.143) 6.49ms
[W 2022-03-03 15:16:13.344 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (jovyan@::ffff:10.86.124.143) 8.03ms
[W 2022-03-03 15:16:13.346 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (jovyan@::ffff:10.86.124.143) 9.57ms
[W 2022-03-03 15:16:13.348 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (jovyan@::ffff:10.86.124.143) 11.09ms
[W 2022-03-03 15:16:13.351 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/requirejs/require.js?v=d37b48bb2137faa0ab98157e240c084dd5b1b5e74911723aa1d1f04c928c2a03dedf922d049e4815f7e5a369faa2e6b6a1000aae958b7953b5cc60411154f593 (jovyan@::ffff:10.86.124.143) 12.65ms
[I 2022-03-03 15:16:22.152 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@::ffff:10.86.124.143) 0.87ms
[W 2022-03-03 15:16:22.542 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (jovyan@::ffff:10.86.124.143) 3.20ms
[I 2022-03-03 15:16:22.905 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@::ffff:10.86.124.143) 0.88ms
[W 2022-03-03 15:16:23.260 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (jovyan@::ffff:10.86.124.143) 2.86ms
[I 2022-03-03 15:16:23.626 JupyterHub log:189] 302 GET /https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec -> /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@::ffff:10.86.124.143) 0.84ms
[W 2022-03-03 15:16:23.980 JupyterHub log:189] 404 GET /hub/https://bdcnonprod-dev-hub01-datalab-jupyter.discoverycloud-nonprod.aws.nz.thenational.com:8000/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (jovyan@::ffff:10.86.124.143) 2.86ms

I commented off the base_url and decided to see what happens when the default value of base_url is considered. Here is the message from the log,

[I 2022-03-03 15:28:17.408 JupyterHub log:189] 302 GET /static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb -> /hub/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb (@::ffff:10.86.124.143) 1.09ms
[I 2022-03-03 15:28:17.421 JupyterHub log:189] 302 GET /custom/custom.css -> /hub/custom/custom.css (@::ffff:10.86.124.143) 3.14ms
[I 2022-03-03 15:28:17.421 JupyterHub log:189] 302 GET /static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 -> /hub/static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 (@::ffff:10.86.124.143) 3.14ms
[I 2022-03-03 15:28:17.422 JupyterHub log:189] 302 GET /static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 -> /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@::ffff:10.86.124.143) 3.13ms
[I 2022-03-03 15:28:17.422 JupyterHub log:189] 302 GET /static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 -> /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@::ffff:10.86.124.143) 3.12ms
[I 2022-03-03 15:28:17.423 JupyterHub log:189] 302 GET /static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec -> /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@::ffff:10.86.124.143) 3.12ms
[I 2022-03-03 15:28:17.423 JupyterHub log:189] 302 GET /static/components/requirejs/require.js?v=d37b48bb2137faa0ab98157e240c084dd5b1b5e74911723aa1d1f04c928c2a03dedf922d049e4815f7e5a369faa2e6b6a1000aae958b7953b5cc60411154f593 -> /hub/static/components/requirejs/require.js?v=d37b48bb2137faa0ab98157e240c084dd5b1b5e74911723aa1d1f04c928c2a03dedf922d049e4815f7e5a369faa2e6b6a1000aae958b7953b5cc60411154f593 (@::ffff:10.86.124.143) 3.12ms
[I 2022-03-03 15:28:17.426 JupyterHub log:189] 302 GET /static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a -> /hub/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a (@::ffff:10.86.124.143) 2.29ms
[I 2022-03-03 15:28:17.426 JupyterHub log:189] 302 GET /static/style/style.min.css?v=b092d0a2da5df36f2b073ddb4eafcd6c8094c4fa21b6dcd3f7185ce16c04ad66424083d785b81607a26b4b85b69a560574ada7db75262c886655f99d651c482e -> /hub/static/style/style.min.css?v=b092d0a2da5df36f2b073ddb4eafcd6c8094c4fa21b6dcd3f7185ce16c04ad66424083d785b81607a26b4b85b69a560574ada7db75262c886655f99d651c482e (@::ffff:10.86.124.143) 2.45ms
[W 2022-03-03 15:28:17.771 JupyterHub log:189] 404 GET /hub/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb (@::ffff:10.86.124.143) 0.83ms
[W 2022-03-03 15:28:17.814 JupyterHub log:189] 404 GET /hub/custom/custom.css (jovyan@::ffff:10.86.124.143) 26.06ms
[W 2022-03-03 15:28:17.814 JupyterHub log:189] 404 GET /hub/static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 (@::ffff:10.86.124.143) 26.34ms
[W 2022-03-03 15:28:17.818 JupyterHub log:189] 404 GET /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@::ffff:10.86.124.143) 2.79ms
[W 2022-03-03 15:28:17.819 JupyterHub log:189] 404 GET /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@::ffff:10.86.124.143) 2.85ms
[W 2022-03-03 15:28:17.819 JupyterHub log:189] 404 GET /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@::ffff:10.86.124.143) 2.92ms
[D 2022-03-03 15:28:17.820 JupyterHub log:189] 200 GET /hub/static/components/requirejs/require.js?v=d37b48bb2137faa0ab98157e240c084dd5b1b5e74911723aa1d1f04c928c2a03dedf922d049e4815f7e5a369faa2e6b6a1000aae958b7953b5cc60411154f593 (@::ffff:10.86.124.143) 3.22ms
[W 2022-03-03 15:28:17.820 JupyterHub log:189] 404 GET /hub/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a (@::ffff:10.86.124.143) 3.29ms
[W 2022-03-03 15:28:17.821 JupyterHub log:189] 404 GET /hub/static/style/style.min.css?v=b092d0a2da5df36f2b073ddb4eafcd6c8094c4fa21b6dcd3f7185ce16c04ad66424083d785b81607a26b4b85b69a560574ada7db75262c886655f99d651c482e (@::ffff:10.86.124.143) 3.35ms
[I 2022-03-03 15:28:18.829 JupyterHub log:189] 302 GET /static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 -> /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@::ffff:10.86.124.143) 0.79ms
[W 2022-03-03 15:28:19.183 JupyterHub log:189] 404 GET /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@::ffff:10.86.124.143) 0.79ms
[I 2022-03-03 15:28:19.557 JupyterHub log:189] 302 GET /static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 -> /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@::ffff:10.86.124.143) 0.84ms
[W 2022-03-03 15:28:19.910 JupyterHub log:189] 404 GET /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@::ffff:10.86.124.143) 0.80ms
[I 2022-03-03 15:28:20.263 JupyterHub log:189] 302 GET /static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec -> /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@::ffff:10.86.124.143) 0.83ms
[W 2022-03-03 15:28:20.618 JupyterHub log:189] 404 GET /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@::ffff:10.86.124.143) 0.81ms
[I 2022-03-03 15:28:21.012 JupyterHub log:189] 302 GET /static/components/jquery/jquery.min.js?v=20220303022814 -> /hub/static/components/jquery/jquery.min.js?v=20220303022814 (@::ffff:10.86.124.143) 0.82ms
[W 2022-03-03 15:28:21.374 JupyterHub log:189] 404 GET /hub/static/components/jquery/jquery.min.js?v=20220303022814 (@::ffff:10.86.124.143) 0.80ms
[I 2022-03-03 15:28:21.754 JupyterHub log:189] 302 GET /static/base/images/favicon.ico?v=50afa725b5de8b00030139d09b38620224d4e7dba47c07ef0e86d4643f30c9bfe6bb7e1a4a1c561aa32834480909a4b6fe7cd1e17f7159330b6b5914bf45a880 -> /hub/static/base/images/favicon.ico?v=50afa725b5de8b00030139d09b38620224d4e7dba47c07ef0e86d4643f30c9bfe6bb7e1a4a1c561aa32834480909a4b6fe7cd1e17f7159330b6b5914bf45a880 (@::ffff:10.86.124.143) 0.83ms
[W 2022-03-03 15:28:22.108 JupyterHub log:189] 404 GET /hub/static/base/images/favicon.ico?v=50afa725b5de8b00030139d09b38620224d4e7dba47c07ef0e86d4643f30c9bfe6bb7e1a4a1c561aa32834480909a4b6fe7cd1e17f7159330b6b5914bf45a880 (@::ffff:10.86.124.143) 0.80ms

Please let me know if you require any further clarifications from my message.

18

Hi Minrk! So disappointed to see this after I finished typing a detailed message on what I see after incorporating the changes you had suggested. Do you know what could be causing this?

image

19

This isn’t right either. It should be '/' which is the default. Start by removing all $JUPYTERHUB_ env variable overrides (remove the overridden .get_env() method entirely). It is unlikely that you need any.

I’m not sure, but it could have been the sequence of large messages. It’s there now, so I think it was only a temporary issue. Thanks for the detailed reports!

20

Hi there,

In the second part of my previous message, I did comment off the base_url override, which should have taken the default value of ‘/’. This doesnt work and I have pasted the log message as above.

And the overrides that are currently in place from my customised Spawner are this,

   def setup_overrides(self):

        # Set default overrides
        overrides = {
            'containerOverrides': [
                {
                    'name': self.container_name
                }
            ]
        }

       # Setup rest of overrides
        env = self.get_env()
        overrides['containerOverrides'][0]['environment'] = dict_to_awskv(env)
        return overrides

I guess this will not have any impact on the other variables which we are looking at.

21

Hi again!

I will be online for next 5hrs or so. And will disappear away until the 14th March.

Please do not close this if I dont respond to you in the next few days.

One of my colleague Nick, who works with me might follow up on this if time permits.

I will get back in touch on the 14th March.

Cheers.