c.Authenticator.admin_users set, but no admin access

how-to
#1

hello I am attempting to setup jupyterhub. currently running .9.6 I cannot login as the local admin for jupyterhub. I can login and I get a notebook server started, but no admin access. I created local user account and added that to the sudo group as well. The config file is located in /etc/jupyterhub/, which is the default correct?

from the jupyterhub_config.py

c.PAMAuthenticator.open_sessions = False
c.Authenticator.admin_users = {'alvin'}
c.JupyterHub.admin_access = true

logs when I login via browser: (docker logs container_id -f)

[I 2019-05-07 01:39:14.222 JupyterHub log:158] 302 GET / -> /hub (@149.169.63.7) 3.25ms

[I 2019-05-07 01:39:14.281 JupyterHub log:158] 302 GET /hub -> /hub/ (@149.169.63.7) 1.14ms

[I 2019-05-07 01:39:14.339 JupyterHub log:158] 302 GET /hub/ -> /hub/login (@149.169.63.7) 1.32ms

**[I 2019-05-07 01:39:31.987 JupyterHub base:509] User logged in: alvin**

**[I 2019-05-07 01:39:31.988 JupyterHub log:158] 302 POST /hub/login?next= -> /user/alvin/ () 43.53ms**

[I 2019-05-07 01:39:32.057 SingleUserNotebookApp log:158] 302 GET /user/alvin/ -> /user/alvin/tree? (@149.169.63.7) 3.85ms

[W 2019-05-07 01:39:32.155 SingleUserNotebookApp auth:267] No Hub user identified for request

[W 2019-05-07 01:39:32.155 SingleUserNotebookApp auth:465] Token stored in cookie may have expired

[I 2019-05-07 01:39:32.157 SingleUserNotebookApp log:158] 302 GET /user/alvin/tree? -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-alvin&redirect_uri=%2Fuser%2Falvin%2Foauth_callback&response_type=code&state=[secret] (@149.169.63.7) 41.29ms

[W 2019-05-07 01:39:32.410 JupyterHub log:158] 404 GET /hub/api/authorizations/token/[secret] (alvin 127.0.0.1) 27.41ms

[I 2019-05-07 01:39:32.463 JupyterHub log:158] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-alvin&redirect_uri=%2Fuser%2Falvin%2Foauth_callback&response_type=code&state=[secret] -> /user/alvin/oauth_callback?code=[secret]&state=[secret] (alvin 149.169.63.7) 50.70ms

[W 2019-05-07 01:39:32.553 SingleUserNotebookApp auth:267] No Hub user identified for request

[W 2019-05-07 01:39:32.553 SingleUserNotebookApp auth:465] Token stored in cookie may have expired

[W 2019-05-07 01:39:32.566 JupyterHub log:158] 404 GET /hub/api/authorizations/token/[secret] (alvin 127.0.0.1) 24.34ms

[I 2019-05-07 01:39:32.627 JupyterHub log:158] 200 POST /hub/api/oauth2/token (alvin 127.0.0.1) 44.43ms

**[I 2019-05-07 01:39:32.661 SingleUserNotebookApp auth:875] Logged-in user {'kind': 'user', 'name': 'alvin', *'admin': False,* 'groups': [], 'server': '/user/alvin/', 'pending': None, 'created': '2019-05-06T20:12:47.778290Z', 'last_activity': '2019-05-07T01:39:32.647927Z', 'servers': None}**

[I 2019-05-07 01:39:32.663 SingleUserNotebookApp log:158] 302 GET /user/alvin/oauth_callback?code=[secret]&state=[secret] -> /user/alvin/tree? (@149.169.63.7) 140.04ms

[I 2019-05-07 01:39:32.672 JupyterHub log:158] 200 GET /hub/api/authorizations/token/[secret] (alvin 127.0.0.1) 31.40ms

[I 2019-05-07 01:39:32.726 SingleUserNotebookApp log:158] 200 GET /user/alvin/tree? (alvin 149.169.63.7) 6.64ms

[I 2019-05-07 01:39:33.009 SingleUserNotebookApp log:158] 200 GET /user/alvin/api/config/tree?_=1557193172930 (alvin 149.169.63.7) 7.58ms

[I 2019-05-07 01:39:33.087 SingleUserNotebookApp log:158] 200 GET /user/alvin/api/config/common?_=1557193172931 (alvin 149.169.63.7) 4.91ms

[I 2019-05-07 01:39:33.144 SingleUserNotebookApp log:158] 200 GET /user/alvin/api/terminals?_=1557193172932 (alvin@149.169.63.7) 1.62ms

[I 2019-05-07 01:39:33.184 SingleUserNotebookApp log:158] 200 GET /user/alvin/api/sessions?_=1557193172933 (alvin 149.169.63.7) 2.25ms

[I 2019-05-07 01:39:33.190 SingleUserNotebookApp log:158] 200 GET /user/alvin/api/terminals?_=1557193172934 (alvin 149.169.63.7) 2.42ms

[I 2019-05-07 01:39:33.288 SingleUserNotebookApp log:158] 200 GET /user/alvin/api/contents?type=directory&_=1557193172935 (alvin 149.169.63.7) 3.90ms

[I 2019-05-07 01:40:10.670 SingleUserNotebookApp log:158] 200 GET /user/alvin/api/terminals?_=1557193172937 (alvin 149.169.63.7) 5.80ms

[I 2019-05-07 01:40:10.675 SingleUserNotebookApp log:158] 200 GET /user/alvin/api/sessions?_=1557193172936 (alvin 149.169.63.7) 2.21ms

[I 2019-05-07 01:40:10.779 SingleUserNotebookApp log:158] 200 GET /user/alvin/api/contents?type=directory&_=1557193172938 (alvin 149.169.63.7) 4.78ms

Eventually I would like to have some flavor of SSO or OAuth setup, but local admin is what I need/want for testing

thanks in advance for any help

cheers,

alvin

#2

Heya! A few suggestions:

  1. Try passing the --config flag explicitly to jupyterhub, with path to the jupyterhub_config.py file, to make sure it is reading it.
  2. JupyterHub ‘admin’ accounts are ‘admin’ only in the context of JupyterHub itself, and give you no special access otherwise. So when you say ‘local’, what effect are you expecting to have?
  3. Is this the full logs? I see a syntax error (true rather than True in the last line), so either there should be an error message in the logs or the jupyterhub_config.py file is not being read.

(I also formatted your post to have ``` around the code / logs, to make it easier to read. Hope that’s ok!)

#3

Hi Yuvi,
Sorry I am new to jupyterhub. I will try to clean up my terminology. I am only concerned with jupyterhub admin access. Will running jupyterhub from the folder containing the jupyterhub_config.py file that I want be the same as --config flag? I saw that typo later and have fixed it. This is the log from the time that I login. Where you looking for the log for everything after the Jupyterhub command is called?

#4

This is the debugging information. I no longer get the Jhub message no admin account set, but I still don’t jupyterhub admin access.


[D 2019-05-08 18:43:13.522 JupyterHub application:555] Looking for jupyterhub_config in /etc/jupyterhub

[D 2019-05-08 18:43:13.658 JupyterHub application:577] Loaded config file: /etc/jupyterhub/jupyterhub_config.py

[I 2019-05-08 18:43:13.696 JupyterHub app:1673] Using Authenticator: oauthenticator.globus.LocalGlobusOAuthenticator-0.8.2

[I 2019-05-08 18:43:13.696 JupyterHub app:1673] Using Spawner: dockerspawner.swarmspawner.SwarmSpawner-0.11.1

[I 2019-05-08 18:43:13.702 JupyterHub app:1016] Loading cookie_secret from /etc/jupyterhub/jupyterhub_cookie_secret

[D 2019-05-08 18:43:13.702 JupyterHub app:1071] Connecting to db: sqlite:///jupyterhub.sqlite

[D 2019-05-08 18:43:13.722 JupyterHub orm:685] database schema version found: 896818069c98

[I 2019-05-08 18:43:13.727 JupyterHub proxy:431] Generating new CONFIGPROXY_AUTH_TOKEN

[D 2019-05-08 18:43:13.728 JupyterHub application:647] Exiting application: jupyterhub

auth_state is enabled, but encryption is not available: Encryption keys must be specified in JUPYTERHUB_CRYPT_KEY env
#5

I see this in the log line. Does JupyterHub exit at this point?

It might also be related to what names Globus auth uses for users who authenticate in, and what you set in admin_users not matching that.

#6

I think JupyterHub would exit at that point. If you can still access it after that line, maybe you have multiple JupyterHubs running somehow, and the one you’re accessing isn’t the same as the one with the config you have set?

#7

When I exec ‘top’ I get:

Tasks:  16 total,   1 running,  14 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16425040 total, 10410984 free,   929380 used,  5084676 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 15141660 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                  
    1 root      20   0  173632  57848  14260 S   0.0  0.4   0:51.86 jupyterhub                                                               
   11 root      20   0  566344  41212  24596 S   0.0  0.3   0:05.52 node                                                                     
  197 alvin     20   0       0      0      0 Z   0.0  0.0   0:00.04 nano                                                                     
  198 root      20   0   20256   3752   3280 S   0.0  0.0   0:00.13 bash                                                                     
  590 root      20   0   20256   3748   3276 S   0.0  0.0   0:00.13 bash                                                                     
  624 root      20   0   20256   3756   3288 S   0.0  0.0   0:00.07 bash                                                                     
  648 root      20   0   20256   3840   3364 S   0.0  0.0   0:00.07 bash                                                                     
  663 root      20   0   20256   3916   3440 S   0.0  0.0   0:00.07 bash                                                                     
  708 root      20   0   20256   3880   3408 S   0.0  0.0   0:00.20 bash                                                                     
  792 root      20   0   20256   3864   3392 S   0.0  0.0   0:00.10 bash                                                                     
  829 root      20   0   20256   3840   3368 S   0.0  0.0   0:00.04 bash                                                                     
  865 root      20   0   20256   3852   3380 S   0.0  0.0   0:00.08 bash                                                                     
  911 root      20   0   20256   3840   3356 S   0.0  0.0   0:00.07 bash                                                                     
  934 alvin     20   0  189132  79656  16060 S   0.0  0.5   0:02.08 jupyterhub-sing                                                          
  940 alvin     20   0   20256   3852   3388 S   0.0  0.0   0:00.00 bash                                                                     
  950 root      20   0   38388   3484   3008 R   0.0  0.0   0:00.01 top
#8

I commented out the OAuth stuff just to simply things and I got this from Jupyterhub --debug:

[D 2019-05-08 19:42:04.062 JupyterHub application:555] Looking for jupyterhub_config in /etc/jupyterhub
[D 2019-05-08 19:42:04.062 JupyterHub application:577] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2019-05-08 19:42:48.315 JupyterHub app:1673] Using Authenticator: jupyterhub.auth.PAMAuthenticator-0.9.6
[I 2019-05-08 19:42:48.315 JupyterHub app:1673] Using Spawner: dockerspawner.swarmspawner.SwarmSpawner-0.11.1
[I 2019-05-08 19:42:48.322 JupyterHub app:1016] Loading cookie_secret from /etc/jupyterhub/jupyterhub_cookie_secret
[D 2019-05-08 19:42:48.322 JupyterHub app:1071] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2019-05-08 19:42:48.350 JupyterHub orm:685] database schema version found: 896818069c98
[I 2019-05-08 19:42:48.356 JupyterHub proxy:431] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2019-05-08 19:42:48.359 JupyterHub configurable:168] Config option `delete_invalid_users` not recognized by `PAMAuthenticator`.
[I 2019-05-08 19:42:48.392 JupyterHub app:1201] Not using whitelist. Any authenticated user will be allowed.
[I 2019-05-08 19:42:48.396 JupyterHub auth:547] Creating user: adduser -q --gecos '""' --disabled-password susanna
[D 2019-05-08 19:42:48.979 JupyterHub app:1473] Loading state for alvin from db
[D 2019-05-08 19:42:48.983 JupyterHub app:1473] Loading state for susanna from db
[D 2019-05-08 19:42:48.984 JupyterHub app:1489] Loaded users:
       alvin admin
     susanna admin
[E 2019-05-08 19:42:48.999 JupyterHub app:1859] Failed to bind hub to http://0.0.0.0:8000/hub/
[E 2019-05-08 19:42:48.999 JupyterHub app:1958]
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.6/site-packages/jupyterhub/app.py", line 1956, in launch_instance_async
        await self.start()
      File "/opt/conda/lib/python3.6/site-packages/jupyterhub/app.py", line 1854, in start
        self.http_server.listen(port, address=ip)
      File "/opt/conda/lib/python3.6/site-packages/tornado/tcpserver.py", line 143, in listen
        sockets = bind_sockets(port, address=address)
      File "/opt/conda/lib/python3.6/site-packages/tornado/netutil.py", line 168, in bind_sockets
        sock.bind(sockaddr)
    OSError: [Errno 98] Address already in use

##But I still don't get the admin tab
#9

This definitely looks like you have more than one hub process running.

top often does not show JupyterHub processes since they don’t really do much CPU. Try looking at the output of ps aux to see if other JupyterHub processes are running.

#10

Here is what I got:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.3 173632 57848 ?        Ssl  May07   0:53 /opt/conda/bin/python /opt/conda/bin/jupyterhub
root        11  0.0  0.2 566344 41972 ?        Ssl  May07   0:05 node /opt/conda/bin/configurable-http-proxy --ip  --port 8000 --api-ip 127.0.
alvin      197  0.0  0.0      0     0 ?        Z    May07   0:00 [nano] <defunct>
root       198  0.0  0.0  20256  3752 pts/1    Ss+  May07   0:00 /bin/bash
root       590  0.0  0.0  20256  3748 pts/0    Ss+  May07   0:00 /bin/bash
root       624  0.0  0.0  20256  3756 pts/2    Ss+  May07   0:00 /bin/bash
root       648  0.0  0.0  20256  3840 pts/3    Ss+  May07   0:00 /bin/bash
root       663  0.0  0.0  20256  3916 pts/4    Ss+  00:17   0:00 /bin/bash
root       708  0.0  0.0  20256  3880 pts/5    Ss+  18:35   0:00 /bin/bash
root       792  0.0  0.0  20256  3864 pts/6    Ss+  19:19   0:00 /bin/bash
root       829  0.0  0.0  20256  3840 pts/7    Ss+  19:31   0:00 /bin/bash
root       865  0.0  0.0  20256  3852 pts/8    Ss+  19:38   0:00 /bin/bash
root       911  0.0  0.0  20256  3840 pts/9    Ss+  19:48   0:00 /bin/bash
alvin      934  0.0  0.4 189132 79656 ?        Ssl  19:55   0:02 /opt/conda/bin/python /opt/conda/bin/jupyterhub-singleuser --port=33503
alvin      940  0.0  0.0  20256  3928 pts/10   Ss+  19:55   0:00 /bin/bash
root       969  0.3  0.0  20256  3784 pts/11   Ss   20:45   0:00 /bin/bash
root       983  0.0  0.0  36148  3256 pts/11   R+   20:45   0:00 ps aux
#11

This JupyterHub process has been running for at least 1 full day (from looking at the ‘START’ column), so I think that’s the process you are reaching instead of the ones you are trying to start. If this is inside a container, I’d recommend killing it and starting a new container.

#12

I fixed this by adding in a line in my Dockerfile to create a local account that mirrored the name of account in my jupyterhub_config.py file.