JUPYTERLAB_SETTINGS_DIR defaults to JupyterHub root directory

tisbam · December 5, 2022, 2:34pm

Hi,

I have setup Jupyterhub on a dedicated server :

venv in /opt/jupyterhub
Gitlab / Local authenticator with home directories automatically created
Traefik proxy
Default to /lab

Is it the standard that the started jupyter server for each user looks for the config in my /opt/jupyterhub/etc ?

To allow user to modify settings on their server I had to do the following “hack” :

c.Spawner.environment = {'JUPYTERLAB_SETTINGS_DIR': lambda spawner: '/home/' + spawner.user.name + '/.jupyter'}

Else I had errors similar to this

Dec 05 14:23:14 jupyterhub jupyterhub[40373]: [W 2022-12-05 14:23:14.332 SingleUserLabApp web:1796] 500 PUT /user/XXX/lab/api/workspaces/default?1670250194370 (10.8.2.1): [Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab'
Dec 05 14:23:14 jupyterhub jupyterhub[40373]: [W 2022-12-05 14:23:14.332 LabApp] wrote error: "[Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab'"

I feel like this is a bit hackish for what I would consider pretty normal usage and I figure I probably have something misconfigured…

Thanks all

bcornec · December 6, 2022, 10:49am

Having the same issue, I tried your hack without any luck. The env variable is correctly setup however we still have the permission denied in the logs.

This used to work flawlessly with:
jupyterhub 2.3.1
jupyter_client : 7.3.5
jupyter_core : 4.11.1
jupyter_server : 1.18.1
jupyterlab : 3.4.6

now we have:
jupyterhub 3.0.0
jupyter_client : 7.4.7
jupyter_core : 5.0.0
jupyter_server : 1.23.3
jupyterlab : 3.5.0

Even when changing default paths to have local paths first doesn’t seem to be read by jupyterhub (setting JUPYTER_PREFER_ENV_PATH to 0 through systemd)

Of course we don’t want to open widely the ```
/opt/jupyterhub/etc/jupyter


Any hint to try or debug info to provide would be welcome.
TIA,
Bruno.

tisbam · December 6, 2022, 1:38pm

Hi,

Is it weird the env variable is not taken into account.
To be clear in my case, it still checks the /opt:jupyterhub/etc path before checking the one in homedir. So I still get the permission error, but in the end it finds a path with write access.

bcornec · December 6, 2022, 4:39pm

Here are extracts from our log:

python3[186449]: [I 2022-12-06 10:24:10.545 SingleUserLabApp log:186] 302 GET /user/student401/ -> /user/student401/lab? (@127.0.0.1) 1.13ms
python3[186270]: [I 2022-12-06 10:24:10.546 JupyterHub base:963] User student401 took 3.150 seconds to start
[...]
python3[186270]: [I 2022-12-06 10:24:10.552 JupyterHub users:749] Server student401 is ready
python3[186270]: [I 2022-12-06 10:24:10.553 JupyterHub log:186] 200 GET /hub/api/users/student401/server/progress (student401@82.66.171.3) 1879.29ms
python3[186270]: [I 2022-12-06 10:24:10.614 JupyterHub log:186] 302 GET /hub/spawn-pending/student401 -> /user/student401/ (student401@82.66.171.3) 4.85ms
python3[186449]: [W 2022-12-06 10:24:10.624 SingleUserLabApp log:186] 403 GET /user/student401/terminals/websocket/1 (@82.66.171.3) 34.09ms
python3[186449]: [I 2022-12-06 10:24:10.658 SingleUserLabApp log:186] 302 GET /user/student401/ -> /user/student401/lab? (@82.66.171.3) 1.00ms
[...]
python3[186449]: [W 2022-12-06 10:24:15.167 LabApp] Could not determine jupyterlab build status without nodejs
python3[186449]: [I 2022-12-06 10:24:15.175 SingleUserLabApp log:186] 200 GET /user/student401/lab/api/build?1670322254347 (student401@82.66.171.3) 803.70ms
python3[186449]: [I 2022-12-06 10:24:15.183 SingleUserLabApp log:186] 200 GET /user/student401/lab/api/settings/@jupyterlab/application-extension:context-menu?1670322254880 (student401@82.66.171.3) 15.00ms
python3[186449]: [I 2022-12-06 10:24:15.227 SingleUserLabApp log:186] 200 GET /user/student401/lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1670322255155 (student401@82.66.171.3) 18.82ms
python3[186449]: [W 2022-12-06 10:24:15.645 SingleUserLabApp web:1796] 500 PUT /user/student401/lab/api/workspaces/auto-0?1670322255615 (82.66.171.3): [Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab/workspaces'
python3[186449]: [W 2022-12-06 10:24:15.646 LabApp] wrote error: "[Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab/workspaces'"
python3[186449]: [E 2022-12-06 10:24:15.647 SingleUserLabApp log:178] {
python3[186449]:       "X-Forwarded-Host": "notebooks3.hpedev.io",
python3[186449]:       "X-Forwarded-Proto": "http",
python3[186449]:       "X-Forwarded-Port": "80",
python3[186449]:       "Referer": "http://notebooks3.hpedev.io/user/student401/lab/api/workspaces/auto-0",
python3[186449]:       "X-Real-Ip": "82.66.171.3",
python3[186449]:       "Host": "notebooks3.hpedev.io",
python3[186449]:       "X-Forwarded-For": "82.66.171.3,::ffff:16.31.84.1",
python3[186449]:       "Cache-Control": "no-cache",
python3[186449]:       "Pragma": "no-cache",
python3[186449]:       "Sec-Gpc": "1",
python3[186449]:       "Dnt": "1",
python3[186449]:       "Sec-Fetch-Site": "same-origin",
python3[186449]:       "Sec-Fetch-Mode": "cors",
python3[186449]:       "Sec-Fetch-Dest": "empty",
python3[186449]:       "Cookie": "_xsrf=[secret]; jupyterhub-user-student401=[secret]; jupyterhub-session-id=[secret]",
python3[186449]:       "Content-Length": "405",
python3[186449]:       "Origin": "https://notebooks3.hpedev.io",
python3[186449]:       "X-Xsrftoken": "2|b24c52ef|524c31b63e883ef8830e2be0be0aa2eb|1669226799",
python3[186449]:       "Authorization": "token [secret]",
python3[186449]:       "Content-Type": "text/plain;charset=UTF-8",
python3[186449]:       "Accept-Encoding": "gzip, deflate, br",
python3[186449]:       "Accept-Language": "en-US,fr;q=0.7,en;q=0.3",
python3[186449]:       "Accept": "*/*",
python3[186449]:       "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0",
python3[186449]:       "Connection": "close"
python3[186449]:     }
python3[186449]: [E 2022-12-06 10:24:15.647 SingleUserLabApp log:186] 500 PUT /user/student401/lab/api/workspaces/auto-0?1670322255615 (student401@82.66.171.3) 4.02ms
python3[186449]: [I 2022-12-06 10:24:15.816 SingleUserLabApp log:186] 200 GET /user/student401/api/contents?content=1&1670322255736 (student401@82.66.171.3) 8.93ms
python3[186449]: [W 2022-12-06 10:24:16.303 SingleUserLabApp web:1796] 500 PUT /user/student401/lab/api/workspaces/auto-0?1670322256273 (82.66.171.3): [Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab/workspaces'
python3[186449]: [W 2022-12-06 10:24:16.303 LabApp] wrote error: "[Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab/workspaces'"
python3[186449]: [E 2022-12-06 10:24:16.305 SingleUserLabApp log:178] {
python3[186449]:       "X-Forwarded-Host": "notebooks3.hpedev.io",
python3[186449]:       "X-Forwarded-Proto": "http",
python3[186449]:       "X-Forwarded-Port": "80",
python3[186449]:       "Referer": "http://notebooks3.hpedev.io/user/student401/lab/api/workspaces/auto-0",
python3[186449]:       "X-Real-Ip": "82.66.171.3",
python3[186449]:       "Host": "notebooks3.hpedev.io",
python3[186449]:       "X-Forwarded-For": "82.66.171.3,::ffff:16.31.84.1",
python3[186449]:       "Cache-Control": "no-cache",
python3[186449]:       "Pragma": "no-cache",
python3[186449]:       "Sec-Gpc": "1",
python3[186449]:       "Dnt": "1",
python3[186449]:       "Sec-Fetch-Site": "same-origin",
python3[186449]:       "Sec-Fetch-Mode": "cors",
python3[186449]:       "Sec-Fetch-Dest": "empty",
python3[186449]:       "Cookie": "_xsrf=[secret]; jupyterhub-user-student401=[secret]; jupyterhub-session-id=[secret]",
python3[186449]:       "Content-Length": "450",
python3[186449]:       "Origin": "https://notebooks3.hpedev.io",
python3[186449]:       "X-Xsrftoken": "2|b24c52ef|524c31b63e883ef8830e2be0be0aa2eb|1669226799",
python3[186449]:       "Authorization": "token [secret]",
python3[186449]:       "Content-Type": "text/plain;charset=UTF-8",
python3[186449]:       "Accept-Encoding": "gzip, deflate, br",
python3[186449]:       "Accept-Language": "en-US,fr;q=0.7,en;q=0.3",
python3[186449]:       "Accept": "*/*",
python3[186449]:       "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0",
python3[186449]:       "Connection": "close"
python3[186449]:     }
python3[186449]: [E 2022-12-06 10:24:16.305 SingleUserLabApp log:186] 500 PUT /user/student401/lab/api/workspaces/auto-0?1670322256273 (student401@82.66.171.3) 3.68ms
python3[186449]: [I 2022-12-06 10:24:17.059 SingleUserLabApp log:186] 200 GET /user/student401/api/nbconvert?1670322254579 (student401@82.66.171.3) 2324.50ms

And it seems indeed that the other conf files (we generate one under $HOME/.jupyter called jupyter_notebook_config.py containing:

c = get_config()  # noqa
c.NotebookApp.default_url = 'WKSHP-API101/0-ReadMeFirst.ipynb'

doesn’t open automatically the notebook mentioned (this is what we ultimately try to achieve).
(Path exists)

iscilyas · December 8, 2022, 6:58am

same here. for a while i’ve been noticing that my “settings” (such as theme choice) weren’t being applied but i was too busy to chase it down (and i thought i had misconfigured something). i find it odd that you’re the first to report this issue.

BTW, here’s what I ended up using (based on your hack):

c.Spawner.environment = {
        'JUPYTERLAB_SETTINGS_DIR': lambda spawner: '/home/' + spawner.user.name + '/.jupyter/lab/user-settings',
        'JUPYTERLAB_WORKSPACES_DIR': lambda spawner: '/home/' + spawner.user.name + '/.jupyter/lab/workspaces',
}

The above works fine.

bcornec · December 22, 2022, 1:34am

Seems that our issue is now fixed with jupyterlab 3.5.2 !

minrk · December 22, 2022, 8:13am

This was fixed by this patch (with lots of links to related discussion).

You can also set JUPYTER_PREFER_ENV_PATH=0:

c.Spawner.environment.update({"JUPYTER_PREFER_ENV_PATH": "0"})

to avoid trying to store things in the env, if you need to wait to upgrade.

Topic		Replies	Views
On start, path on /users/ or /home/? JupyterHub jupyterlab	1	399	November 22, 2020
Notebook landing directory vs Root folder JupyterHub how-to , help-wanted	3	442	December 20, 2023
Unable to set default app for user environment in TLJH The Littlest JupyterHub jupyterlab , jupyterhub , help-wanted	0	697	April 25, 2022
Settings not picked up by JupyterLab JupyterHub jupyterlab	7	3246	December 12, 2023
Jupyterhub 2.0.0 c.Spawner.notebook_dir vs $HOME for setting custom home directory JupyterHub help-wanted	4	1406	January 6, 2022

JUPYTERLAB_SETTINGS_DIR defaults to JupyterHub root directory

Related Topics