Hi,
I am completely stuck and keep trying the same things for 2 weeks. I have written a custom jupyterhub spawner which functions similarly to kubespawner, but using our internal APIs to launch and expose the singleuser server (also a custom docker image necessarily, though I have reproduced this issue with the jupyterhub/single-user images).
I can spawn my image, but get 404’s when trying to load my user directory. On the hub logs side, I see 404 errors:
[W 2020-08-31 23:57:28.202 JupyterHub base:950] User erccarls is slow to start (timeout=10)
23:57:28.440 - debug: [ConfigProxy] PROXY WEB /hub/spawn-pending/erccarls to http://jupyterhub-56f4cdb598-vkh7k:443
23:57:28.746 - debug: [ConfigProxy] PROXY WEB /hub/health to http://jupyterhub-56f4cdb598-vkh7k:443
23:57:28.835 - debug: [ConfigProxy] PROXY WEB /hub/api/users/erccarls/server/progress to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.091 - debug: [ConfigProxy] POST /user/erccarls user=erccarls, server_name=, target=http://jupyterhub.compute-notebook-server-16307faa93c2bf28.svc:8888, jupyterhub=true
23:58:45.222 - debug: [ConfigProxy] PROXY WEB /hub/spawn-pending/erccarls to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.318 - debug: [ConfigProxy] PROXY WEB /user/erccarls/ to http://jupyterhub.compute-notebook-server-16307faa93c2bf28.svc:8888
23:58:45.422 - debug: [ConfigProxy] PROXY WEB /user/erccarls to http://jupyterhub.compute-notebook-server-16307faa93c2bf28.svc:8888
23:58:45.560 - debug: [ConfigProxy] PROXY WEB /static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=3c2a865c832a1322285c55c6ed99abb2 to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.562 - debug: [ConfigProxy] PROXY WEB /static/style/style.min.css?v=85dc254aa190e735282a294119ac34d1 to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.563 - debug: [ConfigProxy] PROXY WEB /static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=9df10041c3e07da766e7c48dd4c35e4a to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.711 - debug: [ConfigProxy] PROXY WEB /hub/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=3c2a865c832a1322285c55c6ed99abb2 to http://jupyterhub-56f4cdb598-vkh7k:443
[W 2020-08-31 23:58:45.713 JupyterHub log:174] 404 GET /hub/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=3c2a865c832a1322285c55c6ed99abb2 (@10.131.154.181) 0.60ms
23:58:45.715 - debug: [ConfigProxy] PROXY WEB /hub/static/style/style.min.css?v=85dc254aa190e735282a294119ac34d1 to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.716 - debug: [ConfigProxy] PROXY WEB /hub/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=9df10041c3e07da766e7c48dd4c35e4a to http://jupyterhub-56f4cdb598-vkh7k:443
[W 2020-08-31 23:58:45.717 JupyterHub log:174] 404 GET /hub/static/style/style.min.css?v=85dc254aa190e735282a294119ac34d1 (@10.131.154.47) 1.00ms
[W 2020-08-31 23:58:45.718 JupyterHub log:174] 404 GET /hub/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=9df10041c3e07da766e7c48dd4c35e4a (@10.131.123.209) 1.15ms
23:58:45.797 - debug: [ConfigProxy] PROXY WEB /custom/custom.css to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.801 - debug: [ConfigProxy] PROXY WEB /static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.802 - debug: [ConfigProxy] PROXY WEB /static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58 to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.878 - debug: [ConfigProxy] PROXY WEB /hub/custom/custom.css to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.878 - debug: [ConfigProxy] PROXY WEB /hub/static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58 to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.879 - debug: [ConfigProxy] PROXY WEB /hub/static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe to http://jupyterhub-56f4cdb598-vkh7k:443
[W 2020-08-31 23:58:45.889 JupyterHub log:174] 404 GET /hub/custom/custom.css (erccarls@10.131.143.24) 9.04ms
[W 2020-08-31 23:58:45.889 JupyterHub log:174] 404 GET /hub/static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58 (@10.131.153.135) 9.05ms
[W 2020-08-31 23:58:45.890 JupyterHub log:174] 404 GET /hub/static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe (@10.131.142.136) 9.10ms
23:58:45.971 - debug: [ConfigProxy] PROXY WEB /static/components/react/react-dom.production.min.js?v=b14d91fb641317cda38dbc9dbf985ab4 to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.971 - debug: [ConfigProxy] PROXY WEB /static/components/create-react-class/index.js?v=94feb9971ce6d26211729abc43f96cd2 to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:45.976 - debug: [ConfigProxy] PROXY WEB /static/components/requirejs/require.js?v=951f856e81496aaeec2e71a1c2c0d51f to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:46.048 - debug: [ConfigProxy] PROXY WEB /hub/static/components/create-react-class/index.js?v=94feb9971ce6d26211729abc43f96cd2 to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:46.049 - debug: [ConfigProxy] PROXY WEB /hub/static/components/react/react-dom.production.min.js?v=b14d91fb641317cda38dbc9dbf985ab4 to http://jupyterhub-56f4cdb598-vkh7k:443
[W 2020-08-31 23:58:46.051 JupyterHub log:174] 404 GET /hub/static/components/create-react-class/index.js?v=94feb9971ce6d26211729abc43f96cd2 (@10.131.120.97) 1.06ms
[W 2020-08-31 23:58:46.051 JupyterHub log:174] 404 GET /hub/static/components/react/react-dom.production.min.js?v=b14d91fb641317cda38dbc9dbf985ab4 (@10.131.132.177) 1.08ms
23:58:46.058 - debug: [ConfigProxy] PROXY WEB /static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58 to http://jupyterhub-56f4cdb598-vkh7k:443
23:58:46.139 - debug: [ConfigProxy] PROXY WEB /hub/static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58 to http://jupyterhub-56f4cdb598-vkh7k:443
[W 2020-08-31 23:58:46.140 JupyterHub log:174] 404 GET /hub/static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58 (@10.131.154.47) 0.53ms
On the singuser-notebook side:
jupyterhub-singleuser --port=8888 --ip=0.0.0.0 --hub-api-url=https://notebook.uat.grandrounds.com/hub/api --debug --notebook-dir=/home/hadoop/notebooks/ --user erccarls --allow-root
[D 2020-08-31 23:58:43.054 erccarls application:179] Searching ['/home/hadoop', '/root/.jupyter', '/opt/conda/envs/cortex/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2020-08-31 23:58:43.055 erccarls application:556] Looking for jupyter_config in /etc/jupyter
[D 2020-08-31 23:58:43.055 erccarls application:556] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2020-08-31 23:58:43.055 erccarls application:556] Looking for jupyter_config in /opt/conda/envs/cortex/etc/jupyter
[D 2020-08-31 23:58:43.055 erccarls application:556] Looking for jupyter_config in /root/.jupyter
[D 2020-08-31 23:58:43.055 erccarls application:556] Looking for jupyter_config in /home/hadoop
[D 2020-08-31 23:58:43.056 erccarls application:556] Looking for jupyter_notebook_config in /etc/jupyter
[D 2020-08-31 23:58:43.056 erccarls application:556] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2020-08-31 23:58:43.056 erccarls application:556] Looking for jupyter_notebook_config in /opt/conda/envs/cortex/etc/jupyter
[D 2020-08-31 23:58:43.056 erccarls application:556] Looking for jupyter_notebook_config in /root/.jupyter
[D 2020-08-31 23:58:43.056 erccarls application:556] Looking for jupyter_notebook_config in /home/hadoop
[D 2020-08-31 23:58:43.060 erccarls config_manager:97] Paths used for configuration of jupyter_notebook_config:
/etc/jupyter/jupyter_notebook_config.json
[D 2020-08-31 23:58:43.061 erccarls config_manager:97] Paths used for configuration of jupyter_notebook_config:
/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2020-08-31 23:58:43.061 erccarls config_manager:97] Paths used for configuration of jupyter_notebook_config:
/opt/conda/envs/cortex/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
/opt/conda/envs/cortex/etc/jupyter/jupyter_notebook_config.json
[D 2020-08-31 23:58:43.061 erccarls config_manager:97] Paths used for configuration of jupyter_notebook_config:
/root/.jupyter/jupyter_notebook_config.json
[I 2020-08-31 23:58:43.527 erccarls extension:162] JupyterLab extension loaded from /opt/conda/envs/cortex/lib/python3.7/site-packages/jupyterlab
[I 2020-08-31 23:58:43.528 erccarls extension:163] JupyterLab application directory is /opt/conda/envs/cortex/share/jupyter/lab
[I 2020-08-31 23:58:43.530 erccarls singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[D 2020-08-31 23:58:43.558 erccarls _version:60] jupyterhub and jupyterhub-singleuser both on version 1.1.0
[I 2020-08-31 23:58:43.558 erccarls notebookapp:2201] Serving notebooks from local directory: /home/hadoop/notebooks
[I 2020-08-31 23:58:43.558 erccarls notebookapp:2201] Jupyter Notebook 6.1.1 is running at:
[I 2020-08-31 23:58:43.558 erccarls notebookapp:2201] http://gr-main:8888/
[I 2020-08-31 23:58:43.558 erccarls notebookapp:2202] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 2020-08-31 23:58:43.561 erccarls singleuser:539] Activity events disabled
[I 2020-08-31 23:58:45.089 erccarls log:174] 302 GET /user/erccarls/ -> /user/erccarls (@10.131.132.77) 0.57ms
[I 2020-08-31 23:58:45.322 erccarls log:174] 302 GET /user/erccarls/ -> /user/erccarls (@10.131.135.121) 0.61ms
[D 2020-08-31 23:58:45.426 erccarls auth:479] No user identified
[D 2020-08-31 23:58:45.427 erccarls handlers:252] Using contents: services/contents
[D 2020-08-31 23:58:45.445 erccarls handlers:833] Path base/images/favicon.ico served from /opt/conda/envs/cortex/lib/python3.7/site-packages/notebook/static/base/images/favicon.ico
[D 2020-08-31 23:58:45.445 erccarls handlers:833] Path components/jquery-ui/themes/smoothness/jquery-ui.min.css served from /opt/conda/envs/cortex/lib/python3.7/site-packages/notebook/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css
[D 2020-08-31 23:58:45.445 erccarls handlers:833] Path components/jquery-typeahead/dist/jquery.typeahead.min.css served from /opt/conda/envs/cortex/lib/python3.7/site-packages/notebook/static/components/jquery-typeahead/dist/jquery.typeahead.min.css
[D 2020-08-31 23:58:45.446 erccarls handlers:833] Path style/style.min.css served from /opt/conda/envs/cortex/lib/python3.7/site-packages/notebook/static/style/style.min.css
[D 2020-08-31 23:58:45.446 erccarls handlers:833] Path components/es6-promise/promise.min.js served from /opt/conda/envs/cortex/lib/python3.7/site-packages/notebook/static/components/es6-promise/promise.min.js
[D 2020-08-31 23:58:45.446 erccarls handlers:833] Path components/react/react.production.min.js served from /opt/conda/envs/cortex/lib/python3.7/site-packages/notebook/static/components/react/react.production.min.js
[D 2020-08-31 23:58:45.446 erccarls handlers:833] Path components/react/react-dom.production.min.js served from /opt/conda/envs/cortex/lib/python3.7/site-packages/notebook/static/components/react/react-dom.production.min.js
[D 2020-08-31 23:58:45.447 erccarls handlers:833] Path components/create-react-class/index.js served from /opt/conda/envs/cortex/lib/python3.7/site-packages/notebook/static/components/create-react-class/index.js
[D 2020-08-31 23:58:45.447 erccarls handlers:833] Path components/requirejs/require.js served from /opt/conda/envs/cortex/lib/python3.7/site-packages/notebook/static/components/requirejs/require.js
[W 2020-08-31 23:58:45.448 erccarls log:174] 404 GET /user/erccarls (@10.131.133.6) 21.66ms
I have tried a million variations of things, but am struggling to understand the issue from the debug output. Any help is appreciated.
Here is my hub config:
import sys
import os
from oauthenticator.github import GitHubOAuthenticator
from tng_spawner.tng_spawner import TNGSpawner
import pwd
import subprocess
pjoin = os.path.join
# Configuration file for jupyterhub.
runtime_dir = os.path.join('/etc/jupyterhub')
c.JupyterHub.cookie_secret_file = pjoin(runtime_dir, 'cookie_secret')
c.JupyterHub.db_url = pjoin(runtime_dir, 'jupyterhub.sqlite')
# the hub should listen on all interfaces, so the proxy can access it
c.JupyterHub.ip = '0.0.0.0'
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.hub_connect_port = 443
# use GitHub OAuthenticator for local users
#c.JupyterHub.authenticator_class = GitHubOAuthenticator
#c.GitHubOAuthenticator.oauth_callback_url = os.environ['OAUTH_CALLBACK_URL']
c.JupyterHub.authenticator_class = 'nativeauthenticator.NativeAuthenticator'
c.NativeAuthenticator.open_signup = True
# specify users and admin
c.Authenticator.whitelist = {'erccarls',}
c.Authenticator.admin_users = {'erccarls',}
def pre_spawn_hook(spawner):
username = spawner.user.name
try:
pwd.getpwnam(username)
except KeyError:
subprocess.check_call(['useradd', '-ms', '/bin/bash', username])
c.Spawner.pre_spawn_hook = pre_spawn_hook
c.JupyterHub.spawner_class = TNGSpawner
c.Spawner.notebook_dir = '/home/hadoop/notebooks/'
c.Spawner.default_url = '/{username}/'
#c.Spawner.default_url = '/lab/'
## Enable debug-logging of the single-user server
c.Spawner.debug = True
c.JupyterHub.log_level = 00
c.ConfigurableHTTPProxy.debug = True
## The location of jupyterhub data files (e.g. /usr/local/share/jupyter/hub)
c.JupyterHub.data_files_path = '/opt/conda/envs/jupyterhub/share/jupyterhub'