GitHub OAuth+ RTC: local account creation issue

To give some context, here is my JupyterHub journey on Ubuntu 22.04:

  1. JupyterHub installed = OK
  2. Certbot running + mapped on config = OK
  3. Using default PAM + RTC = (I forgot what I did but it works so OK)
  4. GitHub oAuth + RTC = Error 500 (solved by creating a local account and folder and changing permissions for the folder on Ubuntu)
  5. GitHub oAuth + RTC + docker spawner = Loses all previous work since all instances will be swapped to dockers, while dockers worked, RTC fails to work

I have reverted back to step 4 and would like to solve the problem of creating accounts. I have yet to try c.LocalGitHubOAuthenticator.add_user_cmd as it is just a plain installation of Ubuntu. It doesn’t seem right to manually create the local accounts.

On my end, I thought this would be sufficient.

c.LocalAuthenticator.create_system_users = True

Does anyone know how to solve this issue?

As long as you are running JupyterHub with sudo, yes, c.LocalAuthenticator.create_system_users = True will create the users if they do not exist.

Hi @mahendrapaipuri I am 99% sure it should have sufficient perms, but I will test out the theory.

Do pardon the description/path :slight_smile:

Make sure to share the actual error(s), which may lead to better solutions.

RTC should work fine in containers since it hasn’t anything to do with that kind of system isolation. Presumably it is an issue of package versions and/or missing configuration in the container that you have in the host system. Comparing the environment and how you enabled RTC should point the way here.

Apologies on my end, yes let me get some logs
To start, here is my config used (less some lines not shown which are IP, SSL certs and port number, which shouldn’t be a concern):

# Configuration file for jupyterhub.
from subprocess import check_call
c.JupyterHub.load_groups = {"collaborative":["some-collab"],"sad-work-noises":["user1","user11","user12","user13","arctic-voxel"]}
c.JupyterHub.load_roles.append(
	{
		"name":"collab-access-sad-work-noises",
		"scopes":["access:servers!user=some-collab","admin:servers!user=some-collab","admin-ui","list:users!user=some-collab"],
		"groups":["sad-work-noises"]
	}
)
def pre_spawn_hook(spawner):
    group_names = {group.name for group in spawner.user.groups}
    if "collaborative" in group_names:
        spawner.log.info(f"Enabling RTC for user {spawner.user.name}")
        spawner.args.append("--LabApp.collaborative=True")
c.Spawner.pre_spawn_hook = pre_spawn_hook
c.OAuthenticator.custom_403_message = "Shoo shoo, no access for u"
c.OAuthenticator.allow_existing_users = True
c.GitHubOAuthenticator.allow_existing_users = True
c.JupyterHub.authenticator_class = "github"
c.OAuthenticator.oauth_callback_url = "https://jupyter.sadworknoises.dev/hub/oauth_callback"
c.OAuthenticator.client_id = "client_id"
c.OAuthenticator.client_secret = "client_secret"
c.GitHubOAuthenticator.allowed_users = {'Arctic-Voxel'}
c.LocalAuthenticator.create_system_users = True
c.Authenticator.admin_users = {'jupyter','arctic-voxel'}
c.OAuthenticator.admin_users = {"Arctic-Voxel","jupyter"}
c.Authenticator.allowed_users = {'jupyter','user1','user11','user12','user13',"some-collab",'arctic-voxel'}
c = get_config()  #noqa

I have a feeling I have some redundancies/additional lines above due to previously using PAM and swapping over to Oauth and at this point I am too afraid to change it. Nonetheless moving on.

Through SSH, I stopped the service, deleted the user and deleted the home directory.

jupyter@jupyter-vm:~$ sudo systemctl stop jupyterlab.service
jupyter@jupyter-vm:~$ sudo userdel -r arctic-voxel
userdel: arctic-voxel mail spool (/var/mail/arctic-voxel) not found
userdel: /home/arctic-voxel not owned by arctic-voxel, not removing
jupyter@jupyter-vm:~$ sudo rm -r /home/arctic-voxel/

Then I start jupyterhub.

jupyter@jupyter-vm:/etc/jupyterhub$ sudo jupyterhub -f /etc/jupyterhub/jupyterhub_config.py

Great it’s running:

[I 2023-10-27 20:42:39.410 JupyterHub app:2810] Running JupyterHub version 3.1.0
[I 2023-10-27 20:42:39.410 JupyterHub app:2840] Using Authenticator: oauthenticator.github.GitHubOAuthenticator-16.1.2.dev
[I 2023-10-27 20:42:39.410 JupyterHub app:2840] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-3.1.0
[I 2023-10-27 20:42:39.411 JupyterHub app:2840] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-3.1.0
[I 2023-10-27 20:42:39.412 JupyterHub app:1649] Loading cookie_secret from /etc/jupyterhub/jupyterhub_cookie_secret
[I 2023-10-27 20:42:39.473 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2023-10-27 20:42:39.547 JupyterHub app:2879] Initialized 0 spawners in 0.002 seconds
[I 2023-10-27 20:42:39.553 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2023-10-27 20:42:39.554 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.seven_days
[I 2023-10-27 20:42:39.555 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.thirty_days
[W 2023-10-27 20:42:39.555 JupyterHub proxy:625] Found proxy pid file: /etc/jupyterhub/jupyterhub-proxy.pid
[W 2023-10-27 20:42:39.556 JupyterHub proxy:637] Proxy no longer running at pid=21662
[I 2023-10-27 20:42:39.556 JupyterHub proxy:750] Starting proxy @ https://192.168.200.2:443/
20:42:39.841 [ConfigProxy] info: Proxying https://192.168.200.2:443 to (no default)
20:42:39.845 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
20:42:39.907 [ConfigProxy] info: 200 GET /api/routes
[I 2023-10-27 20:42:39.908 JupyterHub app:3130] Hub API listening on http://127.0.0.1:8081/hub/
20:42:39.910 [ConfigProxy] info: 200 GET /api/routes
[I 2023-10-27 20:42:39.911 JupyterHub proxy:477] Adding route for Hub: / => http://127.0.0.1:8081
20:42:39.914 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
20:42:39.915 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
20:42:39.916 [ConfigProxy] info: 201 POST /api/routes/
[I 2023-10-27 20:42:39.916 JupyterHub app:3197] JupyterHub is now running at https://192.168.200.2:443/

Now from my browser, I enter jupyterhub and I press that big orange button.

Tadah!

500 : Internal Server Error

Unhandled error starting server arctic-voxel

You can try restarting your server from the home page.

For the most important part, is the logs:

[I 2023-10-27 20:44:54.150 JupyterHub log:186] 302 GET / -> /hub/ (@10.10.101.103) 1.60ms
[W 2023-10-27 20:44:54.169 JupyterHub base:398] Invalid or expired cookie token
[I 2023-10-27 20:44:54.171 JupyterHub log:186] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@10.10.101.103) 2.19ms
[I 2023-10-27 20:44:54.227 JupyterHub log:186] 200 GET /hub/login?next=%2Fhub%2F (@10.10.101.103) 39.00ms
[I 2023-10-27 20:45:43.956 JupyterHub oauth2:97] OAuth redirect: https://jupyter.sadworknoises.dev/hub/oauth_callback
[I 2023-10-27 20:45:43.958 JupyterHub log:186] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://github.com/login/oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Fjupyter.sadworknoises.dev%2Fhub%2Foauth_callback&client_id=xxxxxxxxxxxxxxxxxxxx&state=[secret] (@10.10.101.103) 2.95ms
[I 2023-10-27 20:45:45.161 JupyterHub base:819] User logged in: arctic-voxel
[I 2023-10-27 20:45:45.162 JupyterHub log:186] 302 GET /hub/oauth_callback?code=[secret]&state=[secret] -> /hub/ (@10.10.101.103) 782.29ms
[I 2023-10-27 20:45:45.192 JupyterHub log:186] 302 GET /hub/ -> /hub/spawn (arctic-voxel@10.10.101.103) 15.50ms
[I 2023-10-27 20:45:45.238 JupyterHub provider:659] Creating oauth client jupyterhub-user-arctic-voxel
[E 2023-10-27 20:45:45.255 JupyterHub user:832] Unhandled error starting arctic-voxel's server: "getpwnam(): name not found: 'arctic-voxel'"
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 747, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/spawner.py", line 1662, in start
        env = self.get_env()
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/spawner.py", line 1609, in get_env
        env = self.user_env(env)
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/spawner.py", line 1596, in user_env
        home = pwd.getpwnam(self.user.name).pw_dir
    KeyError: "getpwnam(): name not found: 'arctic-voxel'"

[E 2023-10-27 20:45:45.283 JupyterHub pages:311] Error starting server arctic-voxel: "getpwnam(): name not found: 'arctic-voxel'"
    Traceback (most recent call last):
    NoneType: None

[W 2023-10-27 20:45:45.283 JupyterHub web:1796] 500 GET /hub/spawn (10.10.101.103): Unhandled error starting server arctic-voxel
[E 2023-10-27 20:45:45.310 JupyterHub log:178] {
      "X-Forwarded-Host": "jupyter.sadworknoises.dev",
      "X-Forwarded-Proto": "https",
      "X-Forwarded-Port": "443",
      "X-Forwarded-For": "10.10.101.103",
      "Cookie": "jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
      "Accept-Language": "en-SG,en-GB;q=0.9,en-US;q=0.8,en;q=0.7",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://jupyter.sadworknoises.dev/",
      "Sec-Ch-Ua-Platform": "\"Windows\"",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua": "\"Chromium\";v=\"118\", \"Google Chrome\";v=\"118\", \"Not=A?Brand\";v=\"99\"",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "cross-site",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
      "Dnt": "1",
      "Upgrade-Insecure-Requests": "1",
      "Connection": "close",
      "Host": "jupyter.sadworknoises.dev"
    }
[E 2023-10-27 20:45:45.310 JupyterHub log:186] 500 GET /hub/spawn (arctic-voxel@10.10.101.103) 98.05ms

As for versions and things of importance… or at least I think that may be important:

jupyter@jupyter-vm:/etc/jupyterhub$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

jupyter@jupyter-vm:/etc/jupyterhub$ pip list
Package                      Version            Editable project location
---------------------------- ------------------ -------------------------------------------------
absl-py                      1.4.0
acme                         1.21.0
aiofiles                     22.1.0
aiosqlite                    0.18.0
ale-py                       0.7.4
alembic                      1.9.2
anyio                        3.6.2
appdirs                      1.4.4
apturl                       0.5.2
argon2-cffi                  21.3.0
argon2-cffi-bindings         21.2.0
arrow                        1.2.3
asttokens                    2.2.1
astunparse                   1.6.3
async-generator              1.10
attrs                        22.2.0
AutoROM                      0.4.2
AutoROM.accept-rom-license   0.6.0
Babel                        2.11.0
backcall                     0.2.0
beautifulsoup4               4.11.1
bleach                       5.0.1
blinker                      1.4
Brlapi                       0.8.3
cachetools                   5.3.0
certbot                      1.21.0
certifi                      2022.12.7
certipy                      0.1.3
cffi                         1.15.1
cfgv                         3.4.0
chardet                      4.0.0
charset-normalizer           3.0.1
click                        8.0.3
cloudpickle                  2.2.1
cmake                        3.26.1
colorama                     0.4.4
comm                         0.1.2
command-not-found            0.3
ConfigArgParse               1.5.3
configobj                    5.0.6
contourpy                    1.0.7
coverage                     7.3.2
cryptography                 3.4.8
cupshelpers                  1.0
cycler                       0.11.0
dbus-python                  1.2.18
debugpy                      1.6.5
decorator                    5.1.1
defer                        1.0.6
defusedxml                   0.7.1
distlib                      0.3.7
distro                       1.7.0
distro-info                  1.1+ubuntu0.1
dm-reverb                    0.11.0
dm-tree                      0.1.8
docker                       6.1.3
docker-pycreds               0.4.0
dockerspawner                12.1.0
entrypoints                  0.4
escapism                     1.0.1
exceptiongroup               1.1.3
executing                    1.2.0
fastjsonschema               2.16.2
filelock                     3.10.4
flatbuffers                  23.3.3
fonttools                    4.39.2
fqdn                         1.5.1
gast                         0.4.0
gin-config                   0.5.0
gitdb                        4.0.10
GitPython                    3.1.31
google-api-core              2.12.0
google-api-python-client     2.104.0
google-auth                  2.16.3
google-auth-httplib2         0.1.1
google-auth-oauthlib         0.4.6
google-pasta                 0.2.0
googleapis-common-protos     1.61.0
greenlet                     2.0.1
grpcio                       1.51.3
gym                          0.21.0
gym-notices                  0.0.8
gyp                          0.1
h5py                         3.8.0
httplib2                     0.20.2
identify                     2.5.30
idna                         3.3
imageio                      2.4.0
importlib-metadata           4.13.0
importlib-resources          5.12.0
iniconfig                    2.0.0
ipykernel                    6.20.2
ipython                      8.8.0
ipython-genutils             0.2.0
isoduration                  20.11.0
jax                          0.4.6
jedi                         0.18.2
jeepney                      0.7.1
Jinja2                       3.1.2
josepy                       1.10.0
json5                        0.9.11
jsonpointer                  2.3
jsonschema                   4.19.1
jsonschema-specifications    2023.7.1
jupyter_client               7.4.9
jupyter_collaboration        1.2.0
jupyter_core                 5.1.3
jupyter-events               0.8.0
jupyter_server               2.1.0
jupyter_server_fileid        0.8.0
jupyter_server_terminals     0.4.4
jupyter_server_ydoc          0.6.1
jupyter-telemetry            0.1.0
jupyter-ydoc                 1.1.1
jupyterhub                   4.0.2
jupyterlab                   3.6.1
jupyterlab-pygments          0.2.2
jupyterlab_server            2.19.0
keras                        2.12.0
keyring                      23.5.0
kiwisolver                   1.4.4
language-selector            0.1
launchpadlib                 1.10.16
lazr.restfulclient           0.14.4
lazr.uri                     1.0.6
libclang                     16.0.0
libtorrent                   2.0.7
lit                          16.0.0
louis                        3.20.0
macaroonbakery               1.3.1
Mako                         1.2.4
Markdown                     3.4.3
markdown-it-py               2.2.0
MarkupSafe                   2.1.2
matplotlib                   3.7.1
matplotlib-inline            0.1.6
mdurl                        0.1.2
mistune                      2.0.4
more-itertools               8.10.0
mpmath                       1.3.0
mwoauth                      0.4.0
nbclassic                    0.4.8
nbclient                     0.7.2
nbconvert                    7.2.8
nbformat                     5.7.3
nest-asyncio                 1.5.6
netifaces                    0.11.0
networkx                     3.0
nodeenv                      1.8.0
notebook                     6.5.2
notebook_shim                0.2.2
numpy                        1.23.5
nvidia-cublas-cu11           11.10.3.66
nvidia-cublas-cu12           12.1.0.26
nvidia-cuda-cupti-cu11       11.7.101
nvidia-cuda-nvrtc-cu11       11.7.99
nvidia-cuda-runtime-cu11     11.7.99
nvidia-cuda-runtime-cu12     12.1.55
nvidia-cudnn-cu11            8.5.0.96
nvidia-cudnn-cu12            8.8.1.3
nvidia-cufft-cu11            10.9.0.58
nvidia-curand-cu11           10.2.10.91
nvidia-cusolver-cu11         11.4.0.1
nvidia-cusparse-cu11         11.7.4.91
nvidia-nccl-cu11             2.14.3
nvidia-nvtx-cu11             11.7.91
oauthenticator               16.1.1             /home/jupyter/.local/lib/python3.10/site-packages
oauthlib                     3.2.0
olefile                      0.46
opencv-python                4.7.0.72
opt-einsum                   3.3.0
packaging                    23.0
pamela                       1.0.0
pandas                       1.5.3
pandocfilters                1.5.0
parsedatetime                2.6
parso                        0.8.3
pathtools                    0.1.2
pexpect                      4.8.0
pickleshare                  0.7.5
Pillow                       9.0.1
pip                          22.0.2
pip-upgrade                  0.0.6
platformdirs                 3.11.0
pluggy                       1.3.0
portpicker                   1.5.2
pre-commit                   3.5.0
prometheus-client            0.15.0
prompt-toolkit               3.0.36
protobuf                     4.22.1
psutil                       5.9.4
ptyprocess                   0.7.0
pure-eval                    0.2.2
pyasn1                       0.4.8
pyasn1-modules               0.2.8
pycairo                      1.20.1
pycparser                    2.21
pycups                       2.0.1
pyee                         8.2.2
pygame                       2.1.3
pyglet                       2.0.5
Pygments                     2.14.0
PyGObject                    3.42.1
PyICU                        2.8.1
PyJWT                        2.3.0
pymacaroons                  0.13.0
PyNaCl                       1.5.0
pyOpenSSL                    21.0.0
pyparsing                    2.4.7
pyppeteer                    1.0.2
pyRFC3339                    1.1
pyrsistent                   0.19.3
pytest                       7.4.2
pytest-asyncio               0.21.1
pytest-cov                   4.1.0
python-apt                   2.4.0+ubuntu2
python-dateutil              2.8.2
python-debian                0.1.43+ubuntu1.1
python-json-logger           2.0.4
pytz                         2022.1
PyVirtualDisplay             3.0
pyxdg                        0.27
PyYAML                       5.4.1
pyzmq                        25.0.0
referencing                  0.30.2
reportlab                    3.6.8
requests                     2.28.2
requests-mock                1.11.0
requests-oauthlib            1.3.1
requests-toolbelt            0.9.1
rfc3339-validator            0.1.4
rfc3986-validator            0.1.1
rich                         13.3.2
rlds                         0.1.7
rpds-py                      0.10.6
rsa                          4.9
ruamel.yaml                  0.17.21
ruamel.yaml.clib             0.2.7
scipy                        1.10.1
screen-resolution-extra      0.0.0
SecretStorage                3.3.1
Send2Trash                   1.8.0
sentry-sdk                   1.17.0
setproctitle                 1.3.2
setuptools                   59.6.0
six                          1.16.0
smmap                        5.0.0
sniffio                      1.3.0
soupsieve                    2.3.2.post1
SQLAlchemy                   1.4.46
ssh-import-id                5.11
stable-baselines3            1.7.0
stack-data                   0.6.2
sympy                        1.11.1
systemd-python               234
tensorboard                  2.12.0
tensorboard-data-server      0.7.0
tensorboard-plugin-wit       1.8.1
tensorflow                   2.12.0
tensorflow-estimator         2.12.0
tensorflow-io-gcs-filesystem 0.31.0
tensorflow-probability       0.19.0
tensorrt                     8.6.0
termcolor                    2.2.0
terminado                    0.17.1
tf-agents                    0.16.0
tf-agents-nightly            0.16.0.dev20230324
tfp-nightly                  0.20.0.dev20230325
tinycss2                     1.2.1
tomli                        2.0.1
torch                        2.0.0
tornado                      6.2
tqdm                         4.65.0
traitlets                    5.8.1
triton                       2.0.0
typing_extensions            4.5.0
ubuntu-advantage-tools       8001
ubuntu-drivers-common        0.0.0
ufw                          0.36.1
unattended-upgrades          0.1
uri-template                 1.2.0
uritemplate                  4.1.1
urllib3                      1.26.15
virtualenv                   20.24.5
wadllib                      1.3.6
wandb                        0.14.0
wcwidth                      0.2.6
webcolors                    1.12
webencodings                 0.5.1
websocket-client             1.4.2
websockets                   10.4
Werkzeug                     2.2.3
wheel                        0.37.1
wrapt                        1.14.1
xdg                          5
xkit                         0.0.0
y-py                         0.6.2
ypy-websocket                0.12.4
zipp                         1.0.0
zope.component               4.3.0
zope.event                   4.4
zope.hookable                5.1.0
zope.interface               5.4.0

Additional things of perhaps concern/interest are when I tried:

c.JupyterHub.authenticator_class = "github"

after running

python3 -m pip install authenticator

This resulted in an error which was solved only after cloning from the repo directly instead. I hope this is enough logs to provide the root cause of the problem? If not do let me know perhaps what I should zoom in to or try.

If you look at that line, JupyterHub is using GitHubOAuthenticator and not LocalGitHubOAuthenticator as you wanted.

You need to change c.JupyterHub.authenticator_class in your config file to c.JupyterHub.authenticator_class="local-github" as defined in the entrypoint of the package.

2 Likes