404 Error on Notebook Startup

General
1

tl;dr - Any help with instructions as to how to pull not the most recent packages, but the most recent working packages to allow a working installation would be appreciated!

Hello!

Earlier this year I set up a JupyterHub server to use in my teaching following along with these directions

I have just tried to set up a second server with this software and I am getting several 404 errors when the notebook starts, proceeded by 302 redirects. They generally look like this…

[I 2020-10-25 14:21:28.339 JupyterHub log:174] 302 GET /static/components/jquery/jquery.min.js?v=20201025142123 -> /hub/static/components/jquery/jquery.min.js?v=20201025142123 (@69.144.23.87) 0.93ms
[W 2020-10-25 14:21:28.659 JupyterHub log:174] 404 GET /hub/static/components/jquery/jquery.min.js?v=20201025142123 (@69.144.23.87) 0.79ms

The hub launches fine, and users can log in with PAM authentication…

I didn’t have these problems last time - any ideas where I should look. Setting this up is at or near the limit of my ability…

Edit: I am able to - at the server’s command line use docker exec -it <container-id> /bin/bash to get into the running container JupyterHub starts… Everything works at least up to that point…

And, if I take down the firewall… it doesn’t change anything… so back up it goes… Let me know if any more information might help…

CONFIGURATION FILE

c = get_config()

c.JupyterHub.jupyterhub_cookie_secret = "/var/jupyterhub/jupyterhub_cookie_secret"

c.JupyterHub.ssl_key = "/etc/letsencrypt/live/ohtwo.finney-lab.net/privkey.pem"

c.JupyterHub.ssl_cert = "/etc/letsencrypt/live/ohtwo.finney-lab.net/fullchain.pem"

c.JupyterHub.port = 448

c.JupyterHub.ip = "<MY_SERVER_IP>"

c.JupyterHub.authenticator_class = "jupyterhub.auth.PAMAuthenticator"
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.Spawner.http_timeout = 120
c.ProfilesSpawner.profiles = [

    ('Docker User',
         'A',
         'dockerspawner.DockerSpawner',
         dict(
             image="jupyter/datascience-notebook",
             remove_containers=True,
             volumes={
                 "jhub-user-{username}": "/home/jovyan/work",
                 "data": {
                     "bind": "/home/jovyan/ro-data",
                     "mode": "ro",
                     },
                 }
         ),
    ),

    # container image=labhub, volume=/home/<username>
    ('System User',
         'B',
         'dockerspawner.SystemUserSpawner',
         dict(
             image="jupyter/datascience-notebook",
             remove_containers=True,
         ),
    ),

    ]

c.JupyterHub.hub_ip = c.JupyterHub.ip
c.JupyterHub.cookie_max_age_days = 0.0035
c.JupyterHub.active_server_limit = 60

Output of /opt/miniconda/bin/jupyter troubleshoot

$PATH:
	/usr/local/sbin
	/usr/local/bin
	/usr/sbin
	/usr/bin
	/sbin
	/bin
	/usr/games
	/usr/local/games
	/snap/bin

sys.path:
	/opt/miniconda/bin
	/opt/miniconda/lib/python38.zip
	/opt/miniconda/lib/python3.8
	/opt/miniconda/lib/python3.8/lib-dynload
	/opt/miniconda/lib/python3.8/site-packages

sys.executable:
	/opt/miniconda/bin/python

sys.version:
	3.8.3 (default, May 19 2020, 18:47:26) 
	[GCC 7.3.0]

platform.platform():
	Linux-4.15.0-122-generic-x86_64-with-glibc2.10

pip list:
	Package                       Version            
	----------------------------- -------------------
	alembic                       1.4.3              
	argon2-cffi                   20.1.0             
	async-generator               1.10               
	attrs                         20.2.0             
	backcall                      0.2.0              
	backports.functools-lru-cache 1.6.1              
	bleach                        3.2.1              
	blinker                       1.4                
	certifi                       2020.6.20          
	certipy                       0.1.3              
	cffi                          1.14.0             
	chardet                       3.0.4              
	conda                         4.9.0              
	conda-package-handling        1.7.0              
	cryptography                  2.9.2              
	decorator                     4.4.2              
	defusedxml                    0.6.0              
	docker                        4.3.1              
	dockerspawner                 0.11.1             
	entrypoints                   0.3                
	escapism                      1.0.1              
	idna                          2.9                
	importlib-metadata            2.0.0              
	ipykernel                     5.3.4              
	ipython                       7.18.1             
	ipython-genutils              0.2.0              
	jedi                          0.17.2             
	Jinja2                        2.11.2             
	jsonschema                    3.2.0              
	jupyter-client                6.1.7              
	jupyter-core                  4.6.3              
	jupyter-telemetry             0.0.5              
	jupyterhub                    1.1.0              
	jupyterlab-pygments           0.1.2              
	Mako                          1.1.3              
	MarkupSafe                    1.1.1              
	mistune                       0.8.4              
	nbclient                      0.5.1              
	nbconvert                     6.0.7              
	nbformat                      5.0.8              
	nest-asyncio                  1.4.1              
	netifaces                     0.10.9             
	notebook                      6.1.4              
	oauthenticator                0.11.0             
	oauthlib                      3.0.1              
	packaging                     20.4               
	pamela                        1.0.0              
	pandocfilters                 1.4.2              
	parso                         0.7.1              
	pexpect                       4.8.0              
	pickleshare                   0.7.5              
	pip                           20.0.2             
	prometheus-client             0.8.0              
	prompt-toolkit                3.0.8              
	ptyprocess                    0.6.0              
	pycosat                       0.6.3              
	pycparser                     2.20               
	pycurl                        7.43.0.6           
	Pygments                      2.7.2              
	PyJWT                         1.7.1              
	pyOpenSSL                     19.1.0             
	pyparsing                     2.4.7              
	pyrsistent                    0.17.3             
	PySocks                       1.7.1              
	python-dateutil               2.8.1              
	python-editor                 1.0.4              
	python-json-logger            2.0.1              
	pyzmq                         19.0.2             
	requests                      2.23.0             
	ruamel-yaml                   0.15.87            
	ruamel.yaml                   0.16.12            
	ruamel.yaml.clib              0.2.2              
	Send2Trash                    1.5.0              
	setuptools                    46.4.0.post20200518
	six                           1.14.0             
	SQLAlchemy                    1.3.20             
	terminado                     0.9.1              
	testpath                      0.4.4              
	tornado                       6.0.4              
	tqdm                          4.46.0             
	traitlets                     5.0.5              
	urllib3                       1.25.8             
	wcwidth                       0.2.5              
	webencodings                  0.5.1              
	websocket-client              0.57.0             
	wheel                         0.34.2             
	wrapspawner                   0.0.1.dev0         
	zipp                          3.3.2

Output of docker images

REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
mysql                          5.7                 1b12f2e9257b        3 days ago          448MB
jupyter/datascience-notebook   latest              9f2079b74a24        4 days ago          3.95GB
elabftw/elabimg                latest              7cb32f9ee1ca        3 weeks ago         438MB
jupyterhub/singleuser          latest              f98c2844eec4        2 months ago        723MB
hello-world                    latest              bf756fb1ae65        9 months ago        13.3kB
2

Are you using the upstream JupyterHub Docker image? There’s a known issue with some of the tags: https://github.com/jupyterhub/jupyterhub/issues/2848

In addition there’s a reported bug concerning the static files in the Docker image which may be relevant https://github.com/jupyterhub/jupyterhub/issues/3172

3

No. I am installing the hub into miniconda, and using a dockerspawner to spawn jupyter/datascience-notebook.

I did, however, see this issue in looking for an answer… Thanks!

4

I really don’t know what I am doing. Setting up servers isn’t my day job - or even a side hustle…

I have installed JupyterHub locally on my server and I am using dockerspawner to start containers of the datascience-notebook. The directions I had found worked in August - which was good, I was given days to find a way to do lab activities for my classes that were able to be done completely online…

I have been asked to “reproduce” this - and now I cannot…

5

Can you try turning on debug logging and see if any more information appears?

6

Sure!

I have continued to poke at this, I suspected that something with my apache configuration might be responsible for the 302 Redirects - so I followed these directions to configure apache to redirect to /jhub/ - as that looked the simplest…

This has made the 302 errors vanish, but I still wind up with the same 404 errors

bill@finney-lab-02:/var/jupyterhub$ sudo /opt/miniconda/bin/jupyterhub -f /var/jupyterhub/jupyterhub_config.py 
[W 2020-11-01 16:37:17.373 JupyterHub configurable:190] Config option `jupyterhub_cookie_secret` not recognized by `JupyterHub`.  Did you mean one of: `cookie_secret, cookie_secret_file`?
[I 2020-11-01 16:37:17.392 JupyterHub app:2240] Running JupyterHub version 1.1.0
[I 2020-11-01 16:37:17.392 JupyterHub app:2270] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.1.0
[I 2020-11-01 16:37:17.392 JupyterHub app:2270] Using Spawner: wrapspawner.wrapspawner.ProfilesSpawner
[I 2020-11-01 16:37:17.393 JupyterHub app:2270] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.1.0
[I 2020-11-01 16:37:17.397 JupyterHub app:1349] Loading cookie_secret from /var/jupyterhub/jupyterhub_cookie_secret
[D 2020-11-01 16:37:17.397 JupyterHub app:1516] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2020-11-01 16:37:17.411 JupyterHub orm:749] database schema version found: 4dc2d5a8c53c
[I 2020-11-01 16:37:17.415 JupyterHub proxy:461] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2020-11-01 16:37:17.416 JupyterHub app:1624] No admin users, admin interface will be unavailable.
[W 2020-11-01 16:37:17.416 JupyterHub app:1625] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2020-11-01 16:37:17.416 JupyterHub app:1654] Not using whitelist. Any authenticated user will be allowed.
10
[D 2020-11-01 16:37:17.448 JupyterHub app:1914] Initializing spawners
[D 2020-11-01 16:37:17.453 JupyterHub app:2011] Loading state for bill from db
[D 2020-11-01 16:37:17.453 JupyterHub app:2038] Loaded users:
        bill
[I 2020-11-01 16:37:17.453 JupyterHub app:2308] Initialized 0 spawners in 0.005 seconds
[I 2020-11-01 16:37:17.456 JupyterHub proxy:646] Starting proxy @ http://142.93.2.123:8000/jhub/
[D 2020-11-01 16:37:17.456 JupyterHub proxy:647] Proxy cmd: ['configurable-http-proxy', '--ip', '142.93.2.123', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://142.93.2.123:8081/jhub/hub/error', '--ssl-key', '/etc/letsencrypt/live/ohtwo.finney-lab.net/privkey.pem', '--ssl-cert', '/etc/letsencrypt/live/ohtwo.finney-lab.net/fullchain.pem']
[D 2020-11-01 16:37:17.460 JupyterHub proxy:562] Writing proxy pid file: jupyterhub-proxy.pid
16:37:17.933 [ConfigProxy] info: Proxying https://142.93.2.123:8000 to (no default)
16:37:17.936 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2020-11-01 16:37:18.337 JupyterHub proxy:682] Proxy started and appears to be up
[D 2020-11-01 16:37:18.338 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
16:37:18.348 [ConfigProxy] info: 200 GET /api/routes 
[I 2020-11-01 16:37:18.350 JupyterHub app:2556] Hub API listening on http://142.93.2.123:8081/jhub/hub/
[D 2020-11-01 16:37:18.350 JupyterHub proxy:315] Fetching routes to check
[D 2020-11-01 16:37:18.350 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
16:37:18.351 [ConfigProxy] info: 200 GET /api/routes 
[I 2020-11-01 16:37:18.351 JupyterHub proxy:320] Checking routes
[I 2020-11-01 16:37:18.352 JupyterHub proxy:400] Adding default route for Hub: /jhub/ => http://142.93.2.123:8081
[D 2020-11-01 16:37:18.352 JupyterHub proxy:766] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/jhub
16:37:18.354 [ConfigProxy] info: Adding route /jhub -> http://142.93.2.123:8081
16:37:18.354 [ConfigProxy] info: Route added /jhub -> http://142.93.2.123:8081
16:37:18.354 [ConfigProxy] info: 201 POST /api/routes/jhub 
[I 2020-11-01 16:37:18.355 JupyterHub app:2631] JupyterHub is now running at http://142.93.2.123:8000/jhub/
[D 2020-11-01 16:37:18.356 JupyterHub app:2233] It took 0.993 seconds for the Hub to start
[I 2020-11-01 16:41:00.870 JupyterHub log:174] 302 GET /jhub/ -> /jhub/hub/ (@69.144.23.87) 1.33ms
[W 2020-11-01 16:41:00.990 JupyterHub base:392] Invalid or expired cookie token
[I 2020-11-01 16:41:00.991 JupyterHub log:174] 302 GET /jhub/hub/ -> /jhub/hub/login (@69.144.23.87) 1.80ms
[I 2020-11-01 16:41:01.358 JupyterHub log:174] 200 GET /jhub/hub/login (@69.144.23.87) 34.30ms
16:41:02.044 [ConfigProxy] error: 404 GET /favicon.ico 
[I 2020-11-01 16:41:02.055 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Ffavicon.ico (@142.93.2.123) 7.36ms
[D 2020-11-01 16:41:03.640 JupyterHub base:522] Setting cookie jupyterhub-session-id: {'httponly': True, 'secure': True}
[D 2020-11-01 16:41:03.640 JupyterHub base:526] Setting cookie for bill: jupyterhub-hub-login
[D 2020-11-01 16:41:03.640 JupyterHub base:522] Setting cookie jupyterhub-hub-login: {'httponly': True, 'secure': True, 'path': '/jhub/hub/'}
[I 2020-11-01 16:41:03.641 JupyterHub base:707] User logged in: bill
[D 2020-11-01 16:41:03.641 JupyterHub user:242] Creating <class 'wrapspawner.wrapspawner.ProfilesSpawner'> for bill:
[I 2020-11-01 16:41:03.646 JupyterHub log:174] 302 POST /jhub/hub/login?next= -> /jhub/hub/spawn (bill@69.144.23.87) 27.30ms
[D 2020-11-01 16:41:03.980 JupyterHub pages:188] Serving options form for bill
[I 2020-11-01 16:41:03.985 JupyterHub log:174] 200 GET /jhub/hub/spawn (bill@69.144.23.87) 11.42ms
[D 2020-11-01 16:41:06.123 JupyterHub base:825] Initiating spawn for bill
[D 2020-11-01 16:41:06.123 JupyterHub base:829] 0/100 concurrent spawns
[D 2020-11-01 16:41:06.123 JupyterHub base:834] 0/60 active servers
Last login: Sun Nov  1 16:30:48 UTC 2020 from 69.144.23.87 on pts/0
Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0-122-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Sun Nov  1 16:41:06 UTC 2020

  System load:  0.0                Users logged in:                1
  Usage of /:   3.8% of 154.90GB   IP address for eth0:            142.93.2.123
  Memory usage: 12%                IP address for br-32e9ee58a125: 172.18.0.1
  Swap usage:   0%                 IP address for docker0:         172.17.0.1
  Processes:    138

 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

5 packages can be updated.
0 updates are security updates.

New release '20.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


[D 2020-11-01 16:41:06.727 JupyterHub user:556] Calling Spawner.start for bill
[I 2020-11-01 16:41:06.751 JupyterHub dockerspawner:930] pulling image jupyter/datascience-notebook
[W 2020-11-01 16:41:16.125 JupyterHub base:947] User bill is slow to start (timeout=10)
[I 2020-11-01 16:41:16.132 JupyterHub log:174] 302 POST /jhub/hub/spawn -> /jhub/hub/spawn-pending/bill (bill@69.144.23.87) 10012.37ms
[I 2020-11-01 16:41:16.464 JupyterHub pages:347] bill is pending spawn
[I 2020-11-01 16:41:16.468 JupyterHub log:174] 200 GET /jhub/hub/spawn-pending/bill (bill@69.144.23.87) 6.64ms
[D 2020-11-01 16:41:58.868 JupyterHub dockerspawner:777] Getting container 'jupyter-bill'
[I 2020-11-01 16:41:58.871 JupyterHub dockerspawner:783] Container 'jupyter-bill' is gone
[D 2020-11-01 16:41:58.877 JupyterHub dockerspawner:873] Starting host with config: {'binds': {'jhub-user-bill': {'bind': '/home/jovyan/work', 'mode': 'rw'}, 'data': {'bind': '/home/jovyan/ro-data', 'mode': 'ro'}}, 'links': {}, 'port_bindings': {8888: ('127.0.0.1',)}, 'network_mode': 'bridge'}
[I 2020-11-01 16:41:59.362 JupyterHub dockerspawner:985] Created container jupyter-bill (id: be0ea91) from image jupyter/datascience-notebook
[I 2020-11-01 16:41:59.362 JupyterHub dockerspawner:1009] Starting container jupyter-bill (id: be0ea91)
[D 2020-11-01 16:41:59.743 JupyterHub spawner:1113] Polling subprocess every 30s
[D 2020-11-01 16:42:01.141 JupyterHub utils:218] Server at http://127.0.0.1:32779/jhub/user/bill/ responded with 302
[D 2020-11-01 16:42:01.141 JupyterHub _version:59] jupyterhub and jupyterhub-singleuser both on version 1.1.0
[I 2020-11-01 16:42:01.142 JupyterHub base:854] User bill took 55.019 seconds to start
[I 2020-11-01 16:42:01.142 JupyterHub proxy:258] Adding user bill to proxy /jhub/user/bill/ => http://127.0.0.1:32779
[D 2020-11-01 16:42:01.142 JupyterHub proxy:766] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/jhub/user/bill
16:42:01.143 [ConfigProxy] info: Adding route /jhub/user/bill -> http://127.0.0.1:32779
16:42:01.144 [ConfigProxy] info: Route added /jhub/user/bill -> http://127.0.0.1:32779
16:42:01.144 [ConfigProxy] info: 201 POST /api/routes/jhub/user/bill 
[I 2020-11-01 16:42:01.144 JupyterHub log:174] 200 GET /jhub/hub/api (@172.17.0.2) 0.93ms
[I 2020-11-01 16:42:01.146 JupyterHub users:609] Server bill is ready
[I 2020-11-01 16:42:01.147 JupyterHub log:174] 200 GET /jhub/hub/api/users/bill/server/progress (bill@69.144.23.87) 43378.05ms
[D 2020-11-01 16:42:01.158 JupyterHub base:285] Recording first activity for <APIToken('4a02...', user='bill')>
[D 2020-11-01 16:42:01.168 JupyterHub users:710] Activity for user bill: 2020-11-01T16:42:00.308713Z
[D 2020-11-01 16:42:01.168 JupyterHub users:728] Activity on server bill/: 2020-11-01T16:42:00.308713Z
[I 2020-11-01 16:42:01.172 JupyterHub log:174] 200 POST /jhub/hub/api/users/bill/activity (bill@172.17.0.2) 16.85ms
[I 2020-11-01 16:42:01.491 JupyterHub log:174] 302 GET /jhub/hub/spawn-pending/bill -> /jhub/user/bill/ (bill@69.144.23.87) 3.55ms
16:42:02.562 [ConfigProxy] error: 404 GET /static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=3c2a865c832a1322285c55c6ed99abb2 
16:42:02.565 [ConfigProxy] error: 404 GET /static/style/style.min.css?v=321d76aeb19982a565773cf9f9c8f1d0 
[I 2020-11-01 16:42:02.566 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Fjquery-ui%2Fthemes%2Fsmoothness%2Fjquery-ui.min.css%3Fv%3D3c2a865c832a1322285c55c6ed99abb2 (@142.93.2.123) 1.61ms
16:42:02.566 [ConfigProxy] error: 404 GET /static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=9df10041c3e07da766e7c48dd4c35e4a 
[I 2020-11-01 16:42:02.569 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fstyle%2Fstyle.min.css%3Fv%3D321d76aeb19982a565773cf9f9c8f1d0 (@142.93.2.123) 1.46ms
[I 2020-11-01 16:42:02.571 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Fjquery-typeahead%2Fdist%2Fjquery.typeahead.min.css%3Fv%3D9df10041c3e07da766e7c48dd4c35e4a (@142.93.2.123) 1.33ms
16:42:02.682 [ConfigProxy] error: 404 GET /custom/custom.css 
16:42:02.684 [ConfigProxy] error: 404 GET /static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe 
[I 2020-11-01 16:42:02.686 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fcustom%2Fcustom.css (@142.93.2.123) 1.90ms
16:42:02.689 [ConfigProxy] error: 404 GET /static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58 
[I 2020-11-01 16:42:02.689 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Fes6-promise%2Fpromise.min.js%3Fv%3Df004a16cb856e0ff11781d01ec5ca8fe (@142.93.2.123) 2.05ms
[I 2020-11-01 16:42:02.692 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Freact%2Freact.production.min.js%3Fv%3D34f96ffc962a7deecc83037ccb582b58 (@142.93.2.123) 1.41ms
16:42:02.804 [ConfigProxy] error: 404 GET /static/components/react/react-dom.production.min.js?v=b14d91fb641317cda38dbc9dbf985ab4 
16:42:02.805 [ConfigProxy] error: 404 GET /static/components/create-react-class/index.js?v=94feb9971ce6d26211729abc43f96cd2 
[I 2020-11-01 16:42:02.808 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Freact%2Freact-dom.production.min.js%3Fv%3Db14d91fb641317cda38dbc9dbf985ab4 (@142.93.2.123) 1.80ms
16:42:02.809 [ConfigProxy] error: 404 GET /static/components/requirejs/require.js?v=951f856e81496aaeec2e71a1c2c0d51f 
[I 2020-11-01 16:42:02.810 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Fcreate-react-class%2Findex.js%3Fv%3D94feb9971ce6d26211729abc43f96cd2 (@142.93.2.123) 1.95ms
[I 2020-11-01 16:42:02.813 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fcomponents%2Frequirejs%2Frequire.js%3Fv%3D951f856e81496aaeec2e71a1c2c0d51f (@142.93.2.123) 1.57ms
16:42:03.541 [ConfigProxy] error: 404 GET /static/base/images/favicon.ico?v=97c6417ed01bdc0ae3ef32ae4894fd03 
[I 2020-11-01 16:42:03.544 JupyterHub log:174] 200 GET /jhub/hub/error/404?url=%2Fstatic%2Fbase%2Fimages%2Ffavicon.ico%3Fv%3D97c6417ed01bdc0ae3ef32ae4894fd03 (@142.93.2.123) 1.50ms
[D 2020-11-01 16:42:18.356 JupyterHub proxy:766] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
16:42:18.358 [ConfigProxy] info: 200 GET /api/routes 
[I 2020-11-01 16:42:18.365 JupyterHub proxy:320] Checking routes

Also, after letting this settle for a moment after trying to log in, I see these messages repeating.

[D 2020-11-01 16:42:29.745 JupyterHub dockerspawner:777] Getting container 'jupyter-bill'
[D 2020-11-01 16:42:29.749 JupyterHub dockerspawner:761] Container be0ea91 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '0001-01-01T00:00:00Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 2536,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2020-11-01T16:41:59.732239466Z',
     'Status': 'running'}

As I have been poking at this, I will include my current configuration file and troubleshooting output - in case I have changed something important…

Config file

c = get_config()

# Set the log level by value or name.
c.JupyterHub.log_level = 'DEBUG'

# Enable debug-logging of the single-user server
c.Spawner.debug = True

# Enable debug-logging of the single-user server
c.LocalProcessSpawner.debug = True

c.JupyterHub.jupyterhub_cookie_secret = "/var/jupyterhub/jupyterhub_cookie_secret"

c.JupyterHub.ssl_key = "/etc/letsencrypt/live/ohtwo.finney-lab.net/privkey.pem"

c.JupyterHub.ssl_cert = "/etc/letsencrypt/live/ohtwo.finney-lab.net/fullchain.pem"

#c.JupyterHub.port = 448

c.JupyterHub.ip = "142.93.2.123"

c.JupyterHub.bind_url = 'http://127.0.0.1:8000/jhub/'

c.JupyterHub.authenticator_class = "jupyterhub.auth.PAMAuthenticator"
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.Spawner.http_timeout = 120
c.ProfilesSpawner.profiles = [

('Docker User',
     'A',
     'dockerspawner.DockerSpawner',
     dict(
         image="jupyter/datascience-notebook",
         remove_containers=True,
         volumes={
             "jhub-user-{username}": "/home/jovyan/work",
             "data": {
                 "bind": "/home/jovyan/ro-data",
                 "mode": "ro",
                 },
             }
     ),
),



# container image=labhub, volume=/home/<username>
('System User',
     'B',
     'dockerspawner.SystemUserSpawner',
     dict(
         image="jupyter/datascience-notebook",
         remove_containers=True,
     ),
),

]
c.JupyterHub.hub_ip = c.JupyterHub.ip
c.JupyterHub.cookie_max_age_days = 0.0035
c.JupyterHub.active_server_limit = 60

Troubleshooting

$PATH:
	/usr/local/sbin
	/usr/local/bin
	/usr/sbin
	/usr/bin
	/sbin
	/bin
	/usr/games
	/usr/local/games
	/snap/bin

sys.path:
	/opt/miniconda/bin
	/opt/miniconda/lib/python38.zip
	/opt/miniconda/lib/python3.8
	/opt/miniconda/lib/python3.8/lib-dynload
	/opt/miniconda/lib/python3.8/site-packages

sys.executable:
	/opt/miniconda/bin/python

sys.version:
	3.8.3 (default, May 19 2020, 18:47:26) 
	[GCC 7.3.0]

platform.platform():
	Linux-4.15.0-122-generic-x86_64-with-glibc2.10

pip list:
	Package                Version
	---------------------- -------------------
	alembic                1.4.3
	argon2-cffi            20.1.0
	async-generator        1.10
	attrs                  20.2.0
	backcall               0.2.0
	bleach                 3.2.1
	blinker                1.4
	brotlipy               0.7.0
	certifi                2020.6.20
	certipy                0.1.3
	cffi                   1.14.3
	chardet                3.0.4
	conda                  4.9.1
	conda-package-handling 1.7.2
	cryptography           3.1.1
	decorator              4.4.2
	defusedxml             0.6.0
	docker                 4.3.1
	dockerspawner          0.11.1
	entrypoints            0.3
	escapism               1.0.1
	idna                   2.10
	importlib-metadata     2.0.0
	ipykernel              5.3.4
	ipython                7.18.1
	ipython-genutils       0.2.0
	jedi                   0.17.2
	Jinja2                 2.11.2
	json5                  0.9.5
	jsonschema             3.2.0
	jupyter-client         6.1.7
	jupyter-core           4.6.3
	jupyter-telemetry      0.0.5
	jupyterhub             1.1.0
	jupyterlab             2.2.9
	jupyterlab-pygments    0.1.2
	jupyterlab-server      1.2.0
	Mako                   1.1.3
	MarkupSafe             1.1.1
	mistune                0.8.4
	nbclient               0.5.1
	nbconvert              6.0.7
	nbformat               5.0.8
	nest-asyncio           1.4.1
	netifaces              0.10.9
	notebook               6.1.4
	oauthenticator         0.12.0
	oauthlib               3.1.0
	packaging              20.4
	pamela                 1.0.0
	pandocfilters          1.4.2
	parso                  0.7.1
	pexpect                4.8.0
	pickleshare            0.7.5
	pip                    20.2.4
	prometheus-client      0.8.0
	prompt-toolkit         3.0.8
	ptyprocess             0.6.0
	pycosat                0.6.3
	pycparser              2.20
	pycurl                 7.43.0.6
	Pygments               2.7.2
	PyJWT                  1.7.1
	pyOpenSSL              19.1.0
	pyparsing              2.4.7
	pyrsistent             0.17.3
	PySocks                1.7.1
	python-dateutil        2.8.1
	python-editor          1.0.4
	python-json-logger     2.0.1
	pyzmq                  19.0.2
	requests               2.24.0
	ruamel-yaml            0.15.87
	ruamel.yaml            0.16.12
	ruamel.yaml.clib       0.2.2
	Send2Trash             1.5.0
	setuptools             50.3.0.post20201006
	six                    1.15.0
	SQLAlchemy             1.3.20
	terminado              0.9.1
	testpath               0.4.4
	tornado                6.0.4
	tqdm                   4.50.2
	traitlets              5.0.5
	urllib3                1.25.11
	wcwidth                0.2.5
	webencodings           0.5.1
	websocket-client       0.57.0
	wheel                  0.35.1
	wrapspawner            0.0.1.dev0
	zipp                   3.4.0

Let me know if there is something more I can run/try to provide more information!

7

Could you try connecting directly to JupyterHub port 8000, bypassing Apache? For example setup an SSH tunnel to forward port 8000.

8

Hi Will, it’s weird that you get a 404. Is there a prefix for your url? If you just go to /hub does it redirect to your notebook?

I saw your intro post that you’re using this for Chemistry and Physics classes. You’re welcome to use our server: https://chemcompute.org/jupyterhub/submit for your classes. I’m happy to install any packages you need if they’re missing.

Thanks,
Mark

9

I haven’t had a chance to work on this lately. I will try this week! I appreciate the patience!

10

Hi markperri - I am going to try to bypass Apache (per manics suggestion) sometime this week - I am hopeful that I, perhaps, just have Apache misconfigured…

I appreciate the offer for using your server for my classes! I already have a JupyterHub server working that I am using for my classes - I was working on setting up a second - to separate family and work… I was also hoping to be able to show my IT department how easy it would be to set up - and perhaps support…

Bill

11

I shut down Apache entirely and I get the same result.

12

Any change you could share your logs again when connecting without apache?

13

I am facing similar problem, as I reported here: https://discourse.jupyter.org/t/jupyterhub-with-dockerspawner-error-404-when-spawning-single-user-container/6860
Have you found a solution?

14

Sadly, I had to take the droplet I had it installed in down - so I don’t have the logs to share. It’s unlikely I will have any further time to work on this until after next semester. I will be back - this is a very cool project!

15

Errors like this one are the key, I think:

All of these 404s are requests for files that should be served by the single-user server, but are being requested without the server’s base_url prefix, so they get routed to the hub and rightly 404.

The error is definitely in either the single-user server environment (possibly in the Spawner), or in the javascript of an extension. Given which URLs are missing (core files such as /custom/custom.css), my suspicion is that the base URL is not being set. Check the JUPYTERHUB_SERVICE_PREFIX environment variable in the singleuser-server environment. It should be /user/username and make sure that any templates, extensions, etc. are using variables/functions to resolve static urls, never a hardcoded /static/. Also check if --base-url is specified on the command-line.

16

I’m really struggling with this. I’m very new to setting up Jupyter Hub, and I’m using the docker spawner. I’ve tried to understand your solution, and implement it, but I think there are some base knowledge assumed that I don’t have. Where is the JUPYTERHUB_SERVICE_PREFIX environment variable set? What process context would use it? I did try setting base_url in the config file using c.ServerApp.base_url = ... (I’m using my home directory for now just to see it works before figuring out how to have it be correct for each user)

Just for completion, here are the errors I’m seeing:

[I 2022-07-02 05:25:47.861 JupyterHub log:189] 302 GET /static/style/bootstrap.min.css?v=0e8a7fbd6de23ad6b27ab95802a0a0915af6693af612bc304d83af445529ce5d95842309ca3405d10f538d45c8a3a261b8cff78b4bd512dd9effb4109a71d0ab -> /hub/static/style/bootstrap.min.css?v=0e8a7fbd6de23ad6b27ab95802a0a0915af6693af612bc304d83af445529ce5d95842309ca3405d10f538d45c8a3a261b8cff78b4bd512dd9effb4109a71d0ab (@2601:1c0:8100:8b:11b2:cf21:2a39:d33c) 1.51ms
[I 2022-07-02 05:25:47.863 JupyterHub log:189] 302 GET /static/style/bootstrap-theme.min.css?v=8b2f045cb5b4d5ad346f6e816aa2566829a4f5f2783ec31d80d46a57de8ac0c3d21fe6e53bcd8e1f38ac17fcd06d12088bc9b43e23b5d1da52d10c6b717b22b3 -> /hub/static/style/bootstrap-theme.min.css?v=8b2f045cb5b4d5ad346f6e816aa2566829a4f5f2783ec31d80d46a57de8ac0c3d21fe6e53bcd8e1f38ac17fcd06d12088bc9b43e23b5d1da52d10c6b717b22b3 (@2601:1c0:8100:8b:11b2:cf21:2a39:d33c) 0.91ms
[I 2022-07-02 05:25:47.865 JupyterHub log:189] 302 GET /static/style/index.css?v=30372e3246a801d662cf9e3f9dd656fa192eebde9054a2282449fe43919de9f0ee9b745d7eb49d3b0a5e56357912cc7d776390eddcab9dac85b77bdb17b4bdae -> /hub/static/style/index.css?v=30372e3246a801d662cf9e3f9dd656fa192eebde9054a2282449fe43919de9f0ee9b745d7eb49d3b0a5e56357912cc7d776390eddcab9dac85b77bdb17b4bdae (@2601:1c0:8100:8b:11b2:cf21:2a39:d33c) 1.36ms
[W 2022-07-02 05:25:47.877 JupyterHub log:189] 404 GET /hub/static/style/bootstrap.min.css?v=0e8a7fbd6de23ad6b27ab95802a0a0915af6693af612bc304d83af445529ce5d95842309ca3405d10f538d45c8a3a261b8cff78b4bd512dd9effb4109a71d0ab (@2601:1c0:8100:8b:11b2:cf21:2a39:d33c) 1.08ms
[W 2022-07-02 05:25:47.880 JupyterHub log:189] 404 GET /hub/static/style/bootstrap-theme.min.css?v=8b2f045cb5b4d5ad346f6e816aa2566829a4f5f2783ec31d80d46a57de8ac0c3d21fe6e53bcd8e1f38ac17fcd06d12088bc9b43e23b5d1da52d10c6b717b22b3 (@2601:1c0:8100:8b:11b2:cf21:2a39:d33c) 1.04ms
[W 2022-07-02 05:25:47.884 JupyterHub log:189] 404 GET /hub/static/style/index.css?v=30372e3246a801d662cf9e3f9dd656fa192eebde9054a2282449fe43919de9f0ee9b745d7eb49d3b0a5e56357912cc7d776390eddcab9dac85b77bdb17b4bdae (@2601:1c0:8100:8b:11b2:cf21:2a39:d33c) 1.17ms
[I 2022-07-02 05:25:47.894 JupyterHub log:189] 302 GET /static/logo/logo.png?v=a2a176ee3cee251ffddf5fa21fe8e43727a9e5f87a06f9c91ad7b776d9e9d3d5e0159c16cc188a3965e00375fb4bc336c16067c688f5040c0c2d4bfdb852a9e4 -> /hub/static/logo/logo.png?v=a2a176ee3cee251ffddf5fa21fe8e43727a9e5f87a06f9c91ad7b776d9e9d3d5e0159c16cc188a3965e00375fb4bc336c16067c688f5040c0c2d4bfdb852a9e4 (@2601:1c0:8100:8b:11b2:cf21:2a39:d33c) 1.20ms
[W 2022-07-02 05:25:47.911 JupyterHub log:189] 404 GET /hub/static/logo/logo.png?v=a2a176ee3cee251ffddf5fa21fe8e43727a9e5f87a06f9c91ad7b776d9e9d3d5e0159c16cc188a3965e00375fb4bc336c16067c688f5040c0c2d4bfdb852a9e4 (@2601:1c0:8100:8b:11b2:cf21:2a39:d33c) 1.16ms

What I think is the relevant part of my config is:

c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = { 'jupyter-user-{username}': notebook_dir }
c.DockerSpawner.image = 'jupyter/scipy-notebook:hub-2.3.1'

I know I’m reviving an old conversation, but it seems to be exactly my issue, and it’s coming up in google searches.