Sure!
I have continued to poke at this, I suspected that something with my apache configuration might be responsible for the 302 Redirects - so I followed these directions to configure apache to redirect to /jhub/ - as that looked the simplest…
This has made the 302 errors vanish, but I still wind up with the same 404 errors
bill@finney-lab-02:/var/jupyterhub$ sudo /opt/miniconda/bin/jupyterhub -f /var/jupyterhub/jupyterhub_config.py
[W 2020-11-01 16:37:17.373 JupyterHub configurable:190] Config option `jupyterhub_cookie_secret` not recognized by `JupyterHub`. Did you mean one of: `cookie_secret, cookie_secret_file`?
[I 2020-11-01 16:37:17.392 JupyterHub app:2240] Running JupyterHub version 1.1.0
[I 2020-11-01 16:37:17.392 JupyterHub app:2270] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.1.0
[I 2020-11-01 16:37:17.392 JupyterHub app:2270] Using Spawner: wrapspawner.wrapspawner.ProfilesSpawner
[I 2020-11-01 16:37:17.393 JupyterHub app:2270] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.1.0
[I 2020-11-01 16:37:17.397 JupyterHub app:1349] Loading cookie_secret from /var/jupyterhub/jupyterhub_cookie_secret
[D 2020-11-01 16:37:17.397 JupyterHub app:1516] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2020-11-01 16:37:17.411 JupyterHub orm:749] database schema version found: 4dc2d5a8c53c
[I 2020-11-01 16:37:17.415 JupyterHub proxy:461] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2020-11-01 16:37:17.416 JupyterHub app:1624] No admin users, admin interface will be unavailable.
[W 2020-11-01 16:37:17.416 JupyterHub app:1625] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2020-11-01 16:37:17.416 JupyterHub app:1654] Not using whitelist. Any authenticated user will be allowed.
10
[D 2020-11-01 16:37:17.448 JupyterHub app:1914] Initializing spawners
[D 2020-11-01 16:37:17.453 JupyterHub app:2011] Loading state for bill from db
[D 2020-11-01 16:37:17.453 JupyterHub app:2038] Loaded users:
bill
[I 2020-11-01 16:37:17.453 JupyterHub app:2308] Initialized 0 spawners in 0.005 seconds
[I 2020-11-01 16:37:17.456 JupyterHub proxy:646] Starting proxy @ http://142.93.2.123:8000/jhub/
[D 2020-11-01 16:37:17.456 JupyterHub proxy:647] Proxy cmd: ['configurable-http-proxy', '--ip', '142.93.2.123', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://142.93.2.123:8081/jhub/hub/error', '--ssl-key', '/etc/letsencrypt/live/ohtwo.finney-lab.net/privkey.pem', '--ssl-cert', '/etc/letsencrypt/live/ohtwo.finney-lab.net/fullchain.pem']
[D 2020-11-01 16:37:17.460 JupyterHub proxy:562] Writing proxy pid file: jupyterhub-proxy.pid
16:37:17.933 [ConfigProxy] info: Proxying https://142.93.2.123:8000 to (no default)
16:37:17.936 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2020-11-01 16:37:18.337 JupyterHub proxy:682] Proxy started and appears to be up
[D 2020-11-01 16:37:18.338 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
16:37:18.348 [ConfigProxy] info: 200 GET /api/routes
[I 2020-11-01 16:37:18.350 JupyterHub app:2556] Hub API listening on http://142.93.2.123:8081/jhub/hub/
[D 2020-11-01 16:37:18.350 JupyterHub proxy:315] Fetching routes to check
[D 2020-11-01 16:37:18.350 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
16:37:18.351 [ConfigProxy] info: 200 GET /api/routes
[I 2020-11-01 16:37:18.351 JupyterHub proxy:320] Checking routes
[I 2020-11-01 16:37:18.352 JupyterHub proxy:400] Adding default route for Hub: /jhub/ => http://142.93.2.123:8081
[D 2020-11-01 16:37:18.352 JupyterHub proxy:766] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/jhub
16:37:18.354 [ConfigProxy] info: Adding route /jhub -> http://142.93.2.123:8081
16:37:18.354 [ConfigProxy] info: Route added /jhub -> http://142.93.2.123:8081
16:37:18.354 [ConfigProxy] info: 201 POST /api/routes/jhub
[I 2020-11-01 16:37:18.355 JupyterHub app:2631] JupyterHub is now running at http://142.93.2.123:8000/jhub/
[D 2020-11-01 16:37:18.356 JupyterHub app:2233] It took 0.993 seconds for the Hub to start
[I 2020-11-01 16:41:00.870 JupyterHub log:174] 302 GET /jhub/ -> /jhub/hub/ (@69.144.23.87) 1.33ms
[W 2020-11-01 16:41:00.990 JupyterHub base:392] Invalid or expired cookie token
[I 2020-11-01 16:41:00.991 JupyterHub log:174] 302 GET /jhub/hub/ -> /jhub/hub/login (@69.144.23.87) 1.80ms
[I 2020-11-01 16:41:01.358 JupyterHub log:174] 200 GET /jhub/hub/login (@69.144.23.87) 34.30ms
16:41:02.044 [ConfigProxy] error: 404 GET /favicon.ico
[I 2020-11-01 16:41:02.055 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Ffavicon.ico (@142.93.2.123) 7.36ms
[D 2020-11-01 16:41:03.640 JupyterHub base:522] Setting cookie jupyterhub-session-id: {'httponly': True, 'secure': True}
[D 2020-11-01 16:41:03.640 JupyterHub base:526] Setting cookie for bill: jupyterhub-hub-login
[D 2020-11-01 16:41:03.640 JupyterHub base:522] Setting cookie jupyterhub-hub-login: {'httponly': True, 'secure': True, 'path': '/jhub/hub/'}
[I 2020-11-01 16:41:03.641 JupyterHub base:707] User logged in: bill
[D 2020-11-01 16:41:03.641 JupyterHub user:242] Creating <class 'wrapspawner.wrapspawner.ProfilesSpawner'> for bill:
[I 2020-11-01 16:41:03.646 JupyterHub log:174] 302 POST /jhub/hub/login?next= -> /jhub/hub/spawn (bill@69.144.23.87) 27.30ms
[D 2020-11-01 16:41:03.980 JupyterHub pages:188] Serving options form for bill
[I 2020-11-01 16:41:03.985 JupyterHub log:174] 200 GET /jhub/hub/spawn (bill@69.144.23.87) 11.42ms
[D 2020-11-01 16:41:06.123 JupyterHub base:825] Initiating spawn for bill
[D 2020-11-01 16:41:06.123 JupyterHub base:829] 0/100 concurrent spawns
[D 2020-11-01 16:41:06.123 JupyterHub base:834] 0/60 active servers
Last login: Sun Nov 1 16:30:48 UTC 2020 from 69.144.23.87 on pts/0
Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0-122-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Sun Nov 1 16:41:06 UTC 2020
System load: 0.0 Users logged in: 1
Usage of /: 3.8% of 154.90GB IP address for eth0: 142.93.2.123
Memory usage: 12% IP address for br-32e9ee58a125: 172.18.0.1
Swap usage: 0% IP address for docker0: 172.17.0.1
Processes: 138
* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at:
https://ubuntu.com/livepatch
5 packages can be updated.
0 updates are security updates.
New release '20.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
[D 2020-11-01 16:41:06.727 JupyterHub user:556] Calling Spawner.start for bill
[I 2020-11-01 16:41:06.751 JupyterHub dockerspawner:930] pulling image jupyter/datascience-notebook
[W 2020-11-01 16:41:16.125 JupyterHub base:947] User bill is slow to start (timeout=10)
[I 2020-11-01 16:41:16.132 JupyterHub log:174] 302 POST /jhub/hub/spawn -> /jhub/hub/spawn-pending/bill (bill@69.144.23.87) 10012.37ms
[I 2020-11-01 16:41:16.464 JupyterHub pages:347] bill is pending spawn
[I 2020-11-01 16:41:16.468 JupyterHub log:174] 200 GET /jhub/hub/spawn-pending/bill (bill@69.144.23.87) 6.64ms
[D 2020-11-01 16:41:58.868 JupyterHub dockerspawner:777] Getting container 'jupyter-bill'
[I 2020-11-01 16:41:58.871 JupyterHub dockerspawner:783] Container 'jupyter-bill' is gone
[D 2020-11-01 16:41:58.877 JupyterHub dockerspawner:873] Starting host with config: {'binds': {'jhub-user-bill': {'bind': '/home/jovyan/work', 'mode': 'rw'}, 'data': {'bind': '/home/jovyan/ro-data', 'mode': 'ro'}}, 'links': {}, 'port_bindings': {8888: ('127.0.0.1',)}, 'network_mode': 'bridge'}
[I 2020-11-01 16:41:59.362 JupyterHub dockerspawner:985] Created container jupyter-bill (id: be0ea91) from image jupyter/datascience-notebook
[I 2020-11-01 16:41:59.362 JupyterHub dockerspawner:1009] Starting container jupyter-bill (id: be0ea91)
[D 2020-11-01 16:41:59.743 JupyterHub spawner:1113] Polling subprocess every 30s
[D 2020-11-01 16:42:01.141 JupyterHub utils:218] Server at http://127.0.0.1:32779/jhub/user/bill/ responded with 302
[D 2020-11-01 16:42:01.141 JupyterHub _version:59] jupyterhub and jupyterhub-singleuser both on version 1.1.0
[I 2020-11-01 16:42:01.142 JupyterHub base:854] User bill took 55.019 seconds to start
[I 2020-11-01 16:42:01.142 JupyterHub proxy:258] Adding user bill to proxy /jhub/user/bill/ => http://127.0.0.1:32779
[D 2020-11-01 16:42:01.142 JupyterHub proxy:766] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/jhub/user/bill
16:42:01.143 [ConfigProxy] info: Adding route /jhub/user/bill -> http://127.0.0.1:32779
16:42:01.144 [ConfigProxy] info: Route added /jhub/user/bill -> http://127.0.0.1:32779
16:42:01.144 [ConfigProxy] info: 201 POST /api/routes/jhub/user/bill
[I 2020-11-01 16:42:01.144 JupyterHub log:174] 200 GET /jhub/hub/api (@172.17.0.2) 0.93ms
[I 2020-11-01 16:42:01.146 JupyterHub users:609] Server bill is ready
[I 2020-11-01 16:42:01.147 JupyterHub log:174] 200 GET /jhub/hub/api/users/bill/server/progress (bill@69.144.23.87) 43378.05ms
[D 2020-11-01 16:42:01.158 JupyterHub base:285] Recording first activity for <APIToken('4a02...', user='bill')>
[D 2020-11-01 16:42:01.168 JupyterHub users:710] Activity for user bill: 2020-11-01T16:42:00.308713Z
[D 2020-11-01 16:42:01.168 JupyterHub users:728] Activity on server bill/: 2020-11-01T16:42:00.308713Z
[I 2020-11-01 16:42:01.172 JupyterHub log:174] 200 POST /jhub/hub/api/users/bill/activity (bill@172.17.0.2) 16.85ms
[I 2020-11-01 16:42:01.491 JupyterHub log:174] 302 GET /jhub/hub/spawn-pending/bill -> /jhub/user/bill/ (bill@69.144.23.87) 3.55ms
16:42:02.562 [ConfigProxy] error: 404 GET /static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=3c2a865c832a1322285c55c6ed99abb2
16:42:02.565 [ConfigProxy] error: 404 GET /static/style/style.min.css?v=321d76aeb19982a565773cf9f9c8f1d0
[I 2020-11-01 16:42:02.566 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Fjquery-ui%2Fthemes%2Fsmoothness%2Fjquery-ui.min.css%3Fv%3D3c2a865c832a1322285c55c6ed99abb2 (@142.93.2.123) 1.61ms
16:42:02.566 [ConfigProxy] error: 404 GET /static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=9df10041c3e07da766e7c48dd4c35e4a
[I 2020-11-01 16:42:02.569 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fstyle%2Fstyle.min.css%3Fv%3D321d76aeb19982a565773cf9f9c8f1d0 (@142.93.2.123) 1.46ms
[I 2020-11-01 16:42:02.571 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Fjquery-typeahead%2Fdist%2Fjquery.typeahead.min.css%3Fv%3D9df10041c3e07da766e7c48dd4c35e4a (@142.93.2.123) 1.33ms
16:42:02.682 [ConfigProxy] error: 404 GET /custom/custom.css
16:42:02.684 [ConfigProxy] error: 404 GET /static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe
[I 2020-11-01 16:42:02.686 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fcustom%2Fcustom.css (@142.93.2.123) 1.90ms
16:42:02.689 [ConfigProxy] error: 404 GET /static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58
[I 2020-11-01 16:42:02.689 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Fes6-promise%2Fpromise.min.js%3Fv%3Df004a16cb856e0ff11781d01ec5ca8fe (@142.93.2.123) 2.05ms
[I 2020-11-01 16:42:02.692 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Freact%2Freact.production.min.js%3Fv%3D34f96ffc962a7deecc83037ccb582b58 (@142.93.2.123) 1.41ms
16:42:02.804 [ConfigProxy] error: 404 GET /static/components/react/react-dom.production.min.js?v=b14d91fb641317cda38dbc9dbf985ab4
16:42:02.805 [ConfigProxy] error: 404 GET /static/components/create-react-class/index.js?v=94feb9971ce6d26211729abc43f96cd2
[I 2020-11-01 16:42:02.808 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Freact%2Freact-dom.production.min.js%3Fv%3Db14d91fb641317cda38dbc9dbf985ab4 (@142.93.2.123) 1.80ms
16:42:02.809 [ConfigProxy] error: 404 GET /static/components/requirejs/require.js?v=951f856e81496aaeec2e71a1c2c0d51f
[I 2020-11-01 16:42:02.810 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Fcreate-react-class%2Findex.js%3Fv%3D94feb9971ce6d26211729abc43f96cd2 (@142.93.2.123) 1.95ms
[I 2020-11-01 16:42:02.813 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Frequirejs%2Frequire.js%3Fv%3D951f856e81496aaeec2e71a1c2c0d51f (@142.93.2.123) 1.57ms
16:42:03.541 [ConfigProxy] error: 404 GET /static/base/images/favicon.ico?v=97c6417ed01bdc0ae3ef32ae4894fd03
[I 2020-11-01 16:42:03.544 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fbase%2Fimages%2Ffavicon.ico%3Fv%3D97c6417ed01bdc0ae3ef32ae4894fd03 (@142.93.2.123) 1.50ms
[D 2020-11-01 16:42:18.356 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
16:42:18.358 [ConfigProxy] info: 200 GET /api/routes
[I 2020-11-01 16:42:18.365 JupyterHub proxy:320] Checking routes
Also, after letting this settle for a moment after trying to log in, I see these messages repeating.
[D 2020-11-01 16:42:29.745 JupyterHub dockerspawner:777] Getting container 'jupyter-bill'
[D 2020-11-01 16:42:29.749 JupyterHub dockerspawner:761] Container be0ea91 status: {'Dead': False,
'Error': '',
'ExitCode': 0,
'FinishedAt': '0001-01-01T00:00:00Z',
'OOMKilled': False,
'Paused': False,
'Pid': 2536,
'Restarting': False,
'Running': True,
'StartedAt': '2020-11-01T16:41:59.732239466Z',
'Status': 'running'}
As I have been poking at this, I will include my current configuration file and troubleshooting output - in case I have changed something important…
Config file
c = get_config()
# Set the log level by value or name.
c.JupyterHub.log_level = 'DEBUG'
# Enable debug-logging of the single-user server
c.Spawner.debug = True
# Enable debug-logging of the single-user server
c.LocalProcessSpawner.debug = True
c.JupyterHub.jupyterhub_cookie_secret = "/var/jupyterhub/jupyterhub_cookie_secret"
c.JupyterHub.ssl_key = "/etc/letsencrypt/live/ohtwo.finney-lab.net/privkey.pem"
c.JupyterHub.ssl_cert = "/etc/letsencrypt/live/ohtwo.finney-lab.net/fullchain.pem"
#c.JupyterHub.port = 448
c.JupyterHub.ip = "142.93.2.123"
c.JupyterHub.bind_url = 'http://127.0.0.1:8000/jhub/'
c.JupyterHub.authenticator_class = "jupyterhub.auth.PAMAuthenticator"
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.Spawner.http_timeout = 120
c.ProfilesSpawner.profiles = [
('Docker User',
'A',
'dockerspawner.DockerSpawner',
dict(
image="jupyter/datascience-notebook",
remove_containers=True,
volumes={
"jhub-user-{username}": "/home/jovyan/work",
"data": {
"bind": "/home/jovyan/ro-data",
"mode": "ro",
},
}
),
),
# container image=labhub, volume=/home/<username>
('System User',
'B',
'dockerspawner.SystemUserSpawner',
dict(
image="jupyter/datascience-notebook",
remove_containers=True,
),
),
]
c.JupyterHub.hub_ip = c.JupyterHub.ip
c.JupyterHub.cookie_max_age_days = 0.0035
c.JupyterHub.active_server_limit = 60
Troubleshooting
$PATH:
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/usr/games
/usr/local/games
/snap/bin
sys.path:
/opt/miniconda/bin
/opt/miniconda/lib/python38.zip
/opt/miniconda/lib/python3.8
/opt/miniconda/lib/python3.8/lib-dynload
/opt/miniconda/lib/python3.8/site-packages
sys.executable:
/opt/miniconda/bin/python
sys.version:
3.8.3 (default, May 19 2020, 18:47:26)
[GCC 7.3.0]
platform.platform():
Linux-4.15.0-122-generic-x86_64-with-glibc2.10
pip list:
Package Version
---------------------- -------------------
alembic 1.4.3
argon2-cffi 20.1.0
async-generator 1.10
attrs 20.2.0
backcall 0.2.0
bleach 3.2.1
blinker 1.4
brotlipy 0.7.0
certifi 2020.6.20
certipy 0.1.3
cffi 1.14.3
chardet 3.0.4
conda 4.9.1
conda-package-handling 1.7.2
cryptography 3.1.1
decorator 4.4.2
defusedxml 0.6.0
docker 4.3.1
dockerspawner 0.11.1
entrypoints 0.3
escapism 1.0.1
idna 2.10
importlib-metadata 2.0.0
ipykernel 5.3.4
ipython 7.18.1
ipython-genutils 0.2.0
jedi 0.17.2
Jinja2 2.11.2
json5 0.9.5
jsonschema 3.2.0
jupyter-client 6.1.7
jupyter-core 4.6.3
jupyter-telemetry 0.0.5
jupyterhub 1.1.0
jupyterlab 2.2.9
jupyterlab-pygments 0.1.2
jupyterlab-server 1.2.0
Mako 1.1.3
MarkupSafe 1.1.1
mistune 0.8.4
nbclient 0.5.1
nbconvert 6.0.7
nbformat 5.0.8
nest-asyncio 1.4.1
netifaces 0.10.9
notebook 6.1.4
oauthenticator 0.12.0
oauthlib 3.1.0
packaging 20.4
pamela 1.0.0
pandocfilters 1.4.2
parso 0.7.1
pexpect 4.8.0
pickleshare 0.7.5
pip 20.2.4
prometheus-client 0.8.0
prompt-toolkit 3.0.8
ptyprocess 0.6.0
pycosat 0.6.3
pycparser 2.20
pycurl 7.43.0.6
Pygments 2.7.2
PyJWT 1.7.1
pyOpenSSL 19.1.0
pyparsing 2.4.7
pyrsistent 0.17.3
PySocks 1.7.1
python-dateutil 2.8.1
python-editor 1.0.4
python-json-logger 2.0.1
pyzmq 19.0.2
requests 2.24.0
ruamel-yaml 0.15.87
ruamel.yaml 0.16.12
ruamel.yaml.clib 0.2.2
Send2Trash 1.5.0
setuptools 50.3.0.post20201006
six 1.15.0
SQLAlchemy 1.3.20
terminado 0.9.1
testpath 0.4.4
tornado 6.0.4
tqdm 4.50.2
traitlets 5.0.5
urllib3 1.25.11
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 0.57.0
wheel 0.35.1
wrapspawner 0.0.1.dev0
zipp 3.4.0
Let me know if there is something more I can run/try to provide more information!