Apologies on my end, yes let me get some logs
To start, here is my config used (less some lines not shown which are IP, SSL certs and port number, which shouldn’t be a concern):
# Configuration file for jupyterhub.
from subprocess import check_call
c.JupyterHub.load_groups = {"collaborative":["some-collab"],"sad-work-noises":["user1","user11","user12","user13","arctic-voxel"]}
c.JupyterHub.load_roles.append(
{
"name":"collab-access-sad-work-noises",
"scopes":["access:servers!user=some-collab","admin:servers!user=some-collab","admin-ui","list:users!user=some-collab"],
"groups":["sad-work-noises"]
}
)
def pre_spawn_hook(spawner):
group_names = {group.name for group in spawner.user.groups}
if "collaborative" in group_names:
spawner.log.info(f"Enabling RTC for user {spawner.user.name}")
spawner.args.append("--LabApp.collaborative=True")
c.Spawner.pre_spawn_hook = pre_spawn_hook
c.OAuthenticator.custom_403_message = "Shoo shoo, no access for u"
c.OAuthenticator.allow_existing_users = True
c.GitHubOAuthenticator.allow_existing_users = True
c.JupyterHub.authenticator_class = "github"
c.OAuthenticator.oauth_callback_url = "https://jupyter.sadworknoises.dev/hub/oauth_callback"
c.OAuthenticator.client_id = "client_id"
c.OAuthenticator.client_secret = "client_secret"
c.GitHubOAuthenticator.allowed_users = {'Arctic-Voxel'}
c.LocalAuthenticator.create_system_users = True
c.Authenticator.admin_users = {'jupyter','arctic-voxel'}
c.OAuthenticator.admin_users = {"Arctic-Voxel","jupyter"}
c.Authenticator.allowed_users = {'jupyter','user1','user11','user12','user13',"some-collab",'arctic-voxel'}
c = get_config() #noqa
I have a feeling I have some redundancies/additional lines above due to previously using PAM and swapping over to Oauth and at this point I am too afraid to change it. Nonetheless moving on.
Through SSH, I stopped the service, deleted the user and deleted the home directory.
jupyter@jupyter-vm:~$ sudo systemctl stop jupyterlab.service
jupyter@jupyter-vm:~$ sudo userdel -r arctic-voxel
userdel: arctic-voxel mail spool (/var/mail/arctic-voxel) not found
userdel: /home/arctic-voxel not owned by arctic-voxel, not removing
jupyter@jupyter-vm:~$ sudo rm -r /home/arctic-voxel/
Then I start jupyterhub.
jupyter@jupyter-vm:/etc/jupyterhub$ sudo jupyterhub -f /etc/jupyterhub/jupyterhub_config.py
Great it’s running:
[I 2023-10-27 20:42:39.410 JupyterHub app:2810] Running JupyterHub version 3.1.0
[I 2023-10-27 20:42:39.410 JupyterHub app:2840] Using Authenticator: oauthenticator.github.GitHubOAuthenticator-16.1.2.dev
[I 2023-10-27 20:42:39.410 JupyterHub app:2840] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-3.1.0
[I 2023-10-27 20:42:39.411 JupyterHub app:2840] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-3.1.0
[I 2023-10-27 20:42:39.412 JupyterHub app:1649] Loading cookie_secret from /etc/jupyterhub/jupyterhub_cookie_secret
[I 2023-10-27 20:42:39.473 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2023-10-27 20:42:39.547 JupyterHub app:2879] Initialized 0 spawners in 0.002 seconds
[I 2023-10-27 20:42:39.553 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2023-10-27 20:42:39.554 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.seven_days
[I 2023-10-27 20:42:39.555 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.thirty_days
[W 2023-10-27 20:42:39.555 JupyterHub proxy:625] Found proxy pid file: /etc/jupyterhub/jupyterhub-proxy.pid
[W 2023-10-27 20:42:39.556 JupyterHub proxy:637] Proxy no longer running at pid=21662
[I 2023-10-27 20:42:39.556 JupyterHub proxy:750] Starting proxy @ https://192.168.200.2:443/
20:42:39.841 [ConfigProxy] info: Proxying https://192.168.200.2:443 to (no default)
20:42:39.845 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
20:42:39.907 [ConfigProxy] info: 200 GET /api/routes
[I 2023-10-27 20:42:39.908 JupyterHub app:3130] Hub API listening on http://127.0.0.1:8081/hub/
20:42:39.910 [ConfigProxy] info: 200 GET /api/routes
[I 2023-10-27 20:42:39.911 JupyterHub proxy:477] Adding route for Hub: / => http://127.0.0.1:8081
20:42:39.914 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
20:42:39.915 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
20:42:39.916 [ConfigProxy] info: 201 POST /api/routes/
[I 2023-10-27 20:42:39.916 JupyterHub app:3197] JupyterHub is now running at https://192.168.200.2:443/
Now from my browser, I enter jupyterhub and I press that big orange button.
Tadah!
500 : Internal Server Error
Unhandled error starting server arctic-voxel
You can try restarting your server from the home page.
For the most important part, is the logs:
[I 2023-10-27 20:44:54.150 JupyterHub log:186] 302 GET / -> /hub/ (@10.10.101.103) 1.60ms
[W 2023-10-27 20:44:54.169 JupyterHub base:398] Invalid or expired cookie token
[I 2023-10-27 20:44:54.171 JupyterHub log:186] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@10.10.101.103) 2.19ms
[I 2023-10-27 20:44:54.227 JupyterHub log:186] 200 GET /hub/login?next=%2Fhub%2F (@10.10.101.103) 39.00ms
[I 2023-10-27 20:45:43.956 JupyterHub oauth2:97] OAuth redirect: https://jupyter.sadworknoises.dev/hub/oauth_callback
[I 2023-10-27 20:45:43.958 JupyterHub log:186] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://github.com/login/oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Fjupyter.sadworknoises.dev%2Fhub%2Foauth_callback&client_id=xxxxxxxxxxxxxxxxxxxx&state=[secret] (@10.10.101.103) 2.95ms
[I 2023-10-27 20:45:45.161 JupyterHub base:819] User logged in: arctic-voxel
[I 2023-10-27 20:45:45.162 JupyterHub log:186] 302 GET /hub/oauth_callback?code=[secret]&state=[secret] -> /hub/ (@10.10.101.103) 782.29ms
[I 2023-10-27 20:45:45.192 JupyterHub log:186] 302 GET /hub/ -> /hub/spawn (arctic-voxel@10.10.101.103) 15.50ms
[I 2023-10-27 20:45:45.238 JupyterHub provider:659] Creating oauth client jupyterhub-user-arctic-voxel
[E 2023-10-27 20:45:45.255 JupyterHub user:832] Unhandled error starting arctic-voxel's server: "getpwnam(): name not found: 'arctic-voxel'"
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 747, in spawn
url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "/usr/local/lib/python3.10/dist-packages/jupyterhub/spawner.py", line 1662, in start
env = self.get_env()
File "/usr/local/lib/python3.10/dist-packages/jupyterhub/spawner.py", line 1609, in get_env
env = self.user_env(env)
File "/usr/local/lib/python3.10/dist-packages/jupyterhub/spawner.py", line 1596, in user_env
home = pwd.getpwnam(self.user.name).pw_dir
KeyError: "getpwnam(): name not found: 'arctic-voxel'"
[E 2023-10-27 20:45:45.283 JupyterHub pages:311] Error starting server arctic-voxel: "getpwnam(): name not found: 'arctic-voxel'"
Traceback (most recent call last):
NoneType: None
[W 2023-10-27 20:45:45.283 JupyterHub web:1796] 500 GET /hub/spawn (10.10.101.103): Unhandled error starting server arctic-voxel
[E 2023-10-27 20:45:45.310 JupyterHub log:178] {
"X-Forwarded-Host": "jupyter.sadworknoises.dev",
"X-Forwarded-Proto": "https",
"X-Forwarded-Port": "443",
"X-Forwarded-For": "10.10.101.103",
"Cookie": "jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
"Accept-Language": "en-SG,en-GB;q=0.9,en-US;q=0.8,en;q=0.7",
"Accept-Encoding": "gzip, deflate, br",
"Referer": "https://jupyter.sadworknoises.dev/",
"Sec-Ch-Ua-Platform": "\"Windows\"",
"Sec-Ch-Ua-Mobile": "?0",
"Sec-Ch-Ua": "\"Chromium\";v=\"118\", \"Google Chrome\";v=\"118\", \"Not=A?Brand\";v=\"99\"",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-User": "?1",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "cross-site",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
"Dnt": "1",
"Upgrade-Insecure-Requests": "1",
"Connection": "close",
"Host": "jupyter.sadworknoises.dev"
}
[E 2023-10-27 20:45:45.310 JupyterHub log:186] 500 GET /hub/spawn (arctic-voxel@10.10.101.103) 98.05ms
As for versions and things of importance… or at least I think that may be important:
jupyter@jupyter-vm:/etc/jupyterhub$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
jupyter@jupyter-vm:/etc/jupyterhub$ pip list
Package Version Editable project location
---------------------------- ------------------ -------------------------------------------------
absl-py 1.4.0
acme 1.21.0
aiofiles 22.1.0
aiosqlite 0.18.0
ale-py 0.7.4
alembic 1.9.2
anyio 3.6.2
appdirs 1.4.4
apturl 0.5.2
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
arrow 1.2.3
asttokens 2.2.1
astunparse 1.6.3
async-generator 1.10
attrs 22.2.0
AutoROM 0.4.2
AutoROM.accept-rom-license 0.6.0
Babel 2.11.0
backcall 0.2.0
beautifulsoup4 4.11.1
bleach 5.0.1
blinker 1.4
Brlapi 0.8.3
cachetools 5.3.0
certbot 1.21.0
certifi 2022.12.7
certipy 0.1.3
cffi 1.15.1
cfgv 3.4.0
chardet 4.0.0
charset-normalizer 3.0.1
click 8.0.3
cloudpickle 2.2.1
cmake 3.26.1
colorama 0.4.4
comm 0.1.2
command-not-found 0.3
ConfigArgParse 1.5.3
configobj 5.0.6
contourpy 1.0.7
coverage 7.3.2
cryptography 3.4.8
cupshelpers 1.0
cycler 0.11.0
dbus-python 1.2.18
debugpy 1.6.5
decorator 5.1.1
defer 1.0.6
defusedxml 0.7.1
distlib 0.3.7
distro 1.7.0
distro-info 1.1+ubuntu0.1
dm-reverb 0.11.0
dm-tree 0.1.8
docker 6.1.3
docker-pycreds 0.4.0
dockerspawner 12.1.0
entrypoints 0.4
escapism 1.0.1
exceptiongroup 1.1.3
executing 1.2.0
fastjsonschema 2.16.2
filelock 3.10.4
flatbuffers 23.3.3
fonttools 4.39.2
fqdn 1.5.1
gast 0.4.0
gin-config 0.5.0
gitdb 4.0.10
GitPython 3.1.31
google-api-core 2.12.0
google-api-python-client 2.104.0
google-auth 2.16.3
google-auth-httplib2 0.1.1
google-auth-oauthlib 0.4.6
google-pasta 0.2.0
googleapis-common-protos 1.61.0
greenlet 2.0.1
grpcio 1.51.3
gym 0.21.0
gym-notices 0.0.8
gyp 0.1
h5py 3.8.0
httplib2 0.20.2
identify 2.5.30
idna 3.3
imageio 2.4.0
importlib-metadata 4.13.0
importlib-resources 5.12.0
iniconfig 2.0.0
ipykernel 6.20.2
ipython 8.8.0
ipython-genutils 0.2.0
isoduration 20.11.0
jax 0.4.6
jedi 0.18.2
jeepney 0.7.1
Jinja2 3.1.2
josepy 1.10.0
json5 0.9.11
jsonpointer 2.3
jsonschema 4.19.1
jsonschema-specifications 2023.7.1
jupyter_client 7.4.9
jupyter_collaboration 1.2.0
jupyter_core 5.1.3
jupyter-events 0.8.0
jupyter_server 2.1.0
jupyter_server_fileid 0.8.0
jupyter_server_terminals 0.4.4
jupyter_server_ydoc 0.6.1
jupyter-telemetry 0.1.0
jupyter-ydoc 1.1.1
jupyterhub 4.0.2
jupyterlab 3.6.1
jupyterlab-pygments 0.2.2
jupyterlab_server 2.19.0
keras 2.12.0
keyring 23.5.0
kiwisolver 1.4.4
language-selector 0.1
launchpadlib 1.10.16
lazr.restfulclient 0.14.4
lazr.uri 1.0.6
libclang 16.0.0
libtorrent 2.0.7
lit 16.0.0
louis 3.20.0
macaroonbakery 1.3.1
Mako 1.2.4
Markdown 3.4.3
markdown-it-py 2.2.0
MarkupSafe 2.1.2
matplotlib 3.7.1
matplotlib-inline 0.1.6
mdurl 0.1.2
mistune 2.0.4
more-itertools 8.10.0
mpmath 1.3.0
mwoauth 0.4.0
nbclassic 0.4.8
nbclient 0.7.2
nbconvert 7.2.8
nbformat 5.7.3
nest-asyncio 1.5.6
netifaces 0.11.0
networkx 3.0
nodeenv 1.8.0
notebook 6.5.2
notebook_shim 0.2.2
numpy 1.23.5
nvidia-cublas-cu11 11.10.3.66
nvidia-cublas-cu12 12.1.0.26
nvidia-cuda-cupti-cu11 11.7.101
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cuda-runtime-cu12 12.1.55
nvidia-cudnn-cu11 8.5.0.96
nvidia-cudnn-cu12 8.8.1.3
nvidia-cufft-cu11 10.9.0.58
nvidia-curand-cu11 10.2.10.91
nvidia-cusolver-cu11 11.4.0.1
nvidia-cusparse-cu11 11.7.4.91
nvidia-nccl-cu11 2.14.3
nvidia-nvtx-cu11 11.7.91
oauthenticator 16.1.1 /home/jupyter/.local/lib/python3.10/site-packages
oauthlib 3.2.0
olefile 0.46
opencv-python 4.7.0.72
opt-einsum 3.3.0
packaging 23.0
pamela 1.0.0
pandas 1.5.3
pandocfilters 1.5.0
parsedatetime 2.6
parso 0.8.3
pathtools 0.1.2
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.0.1
pip 22.0.2
pip-upgrade 0.0.6
platformdirs 3.11.0
pluggy 1.3.0
portpicker 1.5.2
pre-commit 3.5.0
prometheus-client 0.15.0
prompt-toolkit 3.0.36
protobuf 4.22.1
psutil 5.9.4
ptyprocess 0.7.0
pure-eval 0.2.2
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycairo 1.20.1
pycparser 2.21
pycups 2.0.1
pyee 8.2.2
pygame 2.1.3
pyglet 2.0.5
Pygments 2.14.0
PyGObject 3.42.1
PyICU 2.8.1
PyJWT 2.3.0
pymacaroons 0.13.0
PyNaCl 1.5.0
pyOpenSSL 21.0.0
pyparsing 2.4.7
pyppeteer 1.0.2
pyRFC3339 1.1
pyrsistent 0.19.3
pytest 7.4.2
pytest-asyncio 0.21.1
pytest-cov 4.1.0
python-apt 2.4.0+ubuntu2
python-dateutil 2.8.2
python-debian 0.1.43+ubuntu1.1
python-json-logger 2.0.4
pytz 2022.1
PyVirtualDisplay 3.0
pyxdg 0.27
PyYAML 5.4.1
pyzmq 25.0.0
referencing 0.30.2
reportlab 3.6.8
requests 2.28.2
requests-mock 1.11.0
requests-oauthlib 1.3.1
requests-toolbelt 0.9.1
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rich 13.3.2
rlds 0.1.7
rpds-py 0.10.6
rsa 4.9
ruamel.yaml 0.17.21
ruamel.yaml.clib 0.2.7
scipy 1.10.1
screen-resolution-extra 0.0.0
SecretStorage 3.3.1
Send2Trash 1.8.0
sentry-sdk 1.17.0
setproctitle 1.3.2
setuptools 59.6.0
six 1.16.0
smmap 5.0.0
sniffio 1.3.0
soupsieve 2.3.2.post1
SQLAlchemy 1.4.46
ssh-import-id 5.11
stable-baselines3 1.7.0
stack-data 0.6.2
sympy 1.11.1
systemd-python 234
tensorboard 2.12.0
tensorboard-data-server 0.7.0
tensorboard-plugin-wit 1.8.1
tensorflow 2.12.0
tensorflow-estimator 2.12.0
tensorflow-io-gcs-filesystem 0.31.0
tensorflow-probability 0.19.0
tensorrt 8.6.0
termcolor 2.2.0
terminado 0.17.1
tf-agents 0.16.0
tf-agents-nightly 0.16.0.dev20230324
tfp-nightly 0.20.0.dev20230325
tinycss2 1.2.1
tomli 2.0.1
torch 2.0.0
tornado 6.2
tqdm 4.65.0
traitlets 5.8.1
triton 2.0.0
typing_extensions 4.5.0
ubuntu-advantage-tools 8001
ubuntu-drivers-common 0.0.0
ufw 0.36.1
unattended-upgrades 0.1
uri-template 1.2.0
uritemplate 4.1.1
urllib3 1.26.15
virtualenv 20.24.5
wadllib 1.3.6
wandb 0.14.0
wcwidth 0.2.6
webcolors 1.12
webencodings 0.5.1
websocket-client 1.4.2
websockets 10.4
Werkzeug 2.2.3
wheel 0.37.1
wrapt 1.14.1
xdg 5
xkit 0.0.0
y-py 0.6.2
ypy-websocket 0.12.4
zipp 1.0.0
zope.component 4.3.0
zope.event 4.4
zope.hookable 5.1.0
zope.interface 5.4.0
Additional things of perhaps concern/interest are when I tried:
c.JupyterHub.authenticator_class = "github"
after running
python3 -m pip install authenticator
This resulted in an error which was solved only after cloning from the repo directly instead. I hope this is enough logs to provide the root cause of the problem? If not do let me know perhaps what I should zoom in to or try.
