JUPYTERLAB_SETTINGS_DIR defaults to JupyterHub root directory

tisbam · December 5, 2022, 2:34pm

Hi,

I have setup Jupyterhub on a dedicated server :

venv in /opt/jupyterhub
Gitlab / Local authenticator with home directories automatically created
Traefik proxy
Default to /lab

Is it the standard that the started jupyter server for each user looks for the config in my /opt/jupyterhub/etc ?

To allow user to modify settings on their server I had to do the following “hack” :

c.Spawner.environment = {'JUPYTERLAB_SETTINGS_DIR': lambda spawner: '/home/' + spawner.user.name + '/.jupyter'}

Else I had errors similar to this

Dec 05 14:23:14 jupyterhub jupyterhub[40373]: [W 2022-12-05 14:23:14.332 SingleUserLabApp web:1796] 500 PUT /user/XXX/lab/api/workspaces/default?1670250194370 (10.8.2.1): [Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab'
Dec 05 14:23:14 jupyterhub jupyterhub[40373]: [W 2022-12-05 14:23:14.332 LabApp] wrote error: "[Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab'"

I feel like this is a bit hackish for what I would consider pretty normal usage and I figure I probably have something misconfigured…

Thanks all

bcornec · December 6, 2022, 10:49am

Having the same issue, I tried your hack without any luck. The env variable is correctly setup however we still have the permission denied in the logs.

This used to work flawlessly with:
jupyterhub 2.3.1
jupyter_client : 7.3.5
jupyter_core : 4.11.1
jupyter_server : 1.18.1
jupyterlab : 3.4.6

now we have:
jupyterhub 3.0.0
jupyter_client : 7.4.7
jupyter_core : 5.0.0
jupyter_server : 1.23.3
jupyterlab : 3.5.0

Even when changing default paths to have local paths first doesn’t seem to be read by jupyterhub (setting JUPYTER_PREFER_ENV_PATH to 0 through systemd)

Of course we don’t want to open widely the ```
/opt/jupyterhub/etc/jupyter


Any hint to try or debug info to provide would be welcome.
TIA,
Bruno.

tisbam · December 6, 2022, 1:38pm

Hi,

Is it weird the env variable is not taken into account.
To be clear in my case, it still checks the /opt:jupyterhub/etc path before checking the one in homedir. So I still get the permission error, but in the end it finds a path with write access.

bcornec · December 6, 2022, 4:39pm

Here are extracts from our log:

python3[186449]: [I 2022-12-06 10:24:10.545 SingleUserLabApp log:186] 302 GET /user/student401/ -> /user/student401/lab? (@127.0.0.1) 1.13ms
python3[186270]: [I 2022-12-06 10:24:10.546 JupyterHub base:963] User student401 took 3.150 seconds to start
[...]
python3[186270]: [I 2022-12-06 10:24:10.552 JupyterHub users:749] Server student401 is ready
python3[186270]: [I 2022-12-06 10:24:10.553 JupyterHub log:186] 200 GET /hub/api/users/student401/server/progress (student401@82.66.171.3) 1879.29ms
python3[186270]: [I 2022-12-06 10:24:10.614 JupyterHub log:186] 302 GET /hub/spawn-pending/student401 -> /user/student401/ (student401@82.66.171.3) 4.85ms
python3[186449]: [W 2022-12-06 10:24:10.624 SingleUserLabApp log:186] 403 GET /user/student401/terminals/websocket/1 (@82.66.171.3) 34.09ms
python3[186449]: [I 2022-12-06 10:24:10.658 SingleUserLabApp log:186] 302 GET /user/student401/ -> /user/student401/lab? (@82.66.171.3) 1.00ms
[...]
python3[186449]: [W 2022-12-06 10:24:15.167 LabApp] Could not determine jupyterlab build status without nodejs
python3[186449]: [I 2022-12-06 10:24:15.175 SingleUserLabApp log:186] 200 GET /user/student401/lab/api/build?1670322254347 (student401@82.66.171.3) 803.70ms
python3[186449]: [I 2022-12-06 10:24:15.183 SingleUserLabApp log:186] 200 GET /user/student401/lab/api/settings/@jupyterlab/application-extension:context-menu?1670322254880 (student401@82.66.171.3) 15.00ms
python3[186449]: [I 2022-12-06 10:24:15.227 SingleUserLabApp log:186] 200 GET /user/student401/lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1670322255155 (student401@82.66.171.3) 18.82ms
python3[186449]: [W 2022-12-06 10:24:15.645 SingleUserLabApp web:1796] 500 PUT /user/student401/lab/api/workspaces/auto-0?1670322255615 (82.66.171.3): [Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab/workspaces'
python3[186449]: [W 2022-12-06 10:24:15.646 LabApp] wrote error: "[Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab/workspaces'"
python3[186449]: [E 2022-12-06 10:24:15.647 SingleUserLabApp log:178] {
python3[186449]:       "X-Forwarded-Host": "notebooks3.hpedev.io",
python3[186449]:       "X-Forwarded-Proto": "http",
python3[186449]:       "X-Forwarded-Port": "80",
python3[186449]:       "Referer": "http://notebooks3.hpedev.io/user/student401/lab/api/workspaces/auto-0",
python3[186449]:       "X-Real-Ip": "82.66.171.3",
python3[186449]:       "Host": "notebooks3.hpedev.io",
python3[186449]:       "X-Forwarded-For": "82.66.171.3,::ffff:16.31.84.1",
python3[186449]:       "Cache-Control": "no-cache",
python3[186449]:       "Pragma": "no-cache",
python3[186449]:       "Sec-Gpc": "1",
python3[186449]:       "Dnt": "1",
python3[186449]:       "Sec-Fetch-Site": "same-origin",
python3[186449]:       "Sec-Fetch-Mode": "cors",
python3[186449]:       "Sec-Fetch-Dest": "empty",
python3[186449]:       "Cookie": "_xsrf=[secret]; jupyterhub-user-student401=[secret]; jupyterhub-session-id=[secret]",
python3[186449]:       "Content-Length": "405",
python3[186449]:       "Origin": "https://notebooks3.hpedev.io",
python3[186449]:       "X-Xsrftoken": "2|b24c52ef|524c31b63e883ef8830e2be0be0aa2eb|1669226799",
python3[186449]:       "Authorization": "token [secret]",
python3[186449]:       "Content-Type": "text/plain;charset=UTF-8",
python3[186449]:       "Accept-Encoding": "gzip, deflate, br",
python3[186449]:       "Accept-Language": "en-US,fr;q=0.7,en;q=0.3",
python3[186449]:       "Accept": "*/*",
python3[186449]:       "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0",
python3[186449]:       "Connection": "close"
python3[186449]:     }
python3[186449]: [E 2022-12-06 10:24:15.647 SingleUserLabApp log:186] 500 PUT /user/student401/lab/api/workspaces/auto-0?1670322255615 (student401@82.66.171.3) 4.02ms
python3[186449]: [I 2022-12-06 10:24:15.816 SingleUserLabApp log:186] 200 GET /user/student401/api/contents?content=1&1670322255736 (student401@82.66.171.3) 8.93ms
python3[186449]: [W 2022-12-06 10:24:16.303 SingleUserLabApp web:1796] 500 PUT /user/student401/lab/api/workspaces/auto-0?1670322256273 (82.66.171.3): [Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab/workspaces'
python3[186449]: [W 2022-12-06 10:24:16.303 LabApp] wrote error: "[Errno 13] Permission denied: '/opt/jupyterhub/etc/jupyter/lab/workspaces'"
python3[186449]: [E 2022-12-06 10:24:16.305 SingleUserLabApp log:178] {
python3[186449]:       "X-Forwarded-Host": "notebooks3.hpedev.io",
python3[186449]:       "X-Forwarded-Proto": "http",
python3[186449]:       "X-Forwarded-Port": "80",
python3[186449]:       "Referer": "http://notebooks3.hpedev.io/user/student401/lab/api/workspaces/auto-0",
python3[186449]:       "X-Real-Ip": "82.66.171.3",
python3[186449]:       "Host": "notebooks3.hpedev.io",
python3[186449]:       "X-Forwarded-For": "82.66.171.3,::ffff:16.31.84.1",
python3[186449]:       "Cache-Control": "no-cache",
python3[186449]:       "Pragma": "no-cache",
python3[186449]:       "Sec-Gpc": "1",
python3[186449]:       "Dnt": "1",
python3[186449]:       "Sec-Fetch-Site": "same-origin",
python3[186449]:       "Sec-Fetch-Mode": "cors",
python3[186449]:       "Sec-Fetch-Dest": "empty",
python3[186449]:       "Cookie": "_xsrf=[secret]; jupyterhub-user-student401=[secret]; jupyterhub-session-id=[secret]",
python3[186449]:       "Content-Length": "450",
python3[186449]:       "Origin": "https://notebooks3.hpedev.io",
python3[186449]:       "X-Xsrftoken": "2|b24c52ef|524c31b63e883ef8830e2be0be0aa2eb|1669226799",
python3[186449]:       "Authorization": "token [secret]",
python3[186449]:       "Content-Type": "text/plain;charset=UTF-8",
python3[186449]:       "Accept-Encoding": "gzip, deflate, br",
python3[186449]:       "Accept-Language": "en-US,fr;q=0.7,en;q=0.3",
python3[186449]:       "Accept": "*/*",
python3[186449]:       "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0",
python3[186449]:       "Connection": "close"
python3[186449]:     }
python3[186449]: [E 2022-12-06 10:24:16.305 SingleUserLabApp log:186] 500 PUT /user/student401/lab/api/workspaces/auto-0?1670322256273 (student401@82.66.171.3) 3.68ms
python3[186449]: [I 2022-12-06 10:24:17.059 SingleUserLabApp log:186] 200 GET /user/student401/api/nbconvert?1670322254579 (student401@82.66.171.3) 2324.50ms

And it seems indeed that the other conf files (we generate one under $HOME/.jupyter called jupyter_notebook_config.py containing:

c = get_config()  # noqa
c.NotebookApp.default_url = 'WKSHP-API101/0-ReadMeFirst.ipynb'

doesn’t open automatically the notebook mentioned (this is what we ultimately try to achieve).
(Path exists)

iscilyas · December 8, 2022, 6:58am

same here. for a while i’ve been noticing that my “settings” (such as theme choice) weren’t being applied but i was too busy to chase it down (and i thought i had misconfigured something). i find it odd that you’re the first to report this issue.

BTW, here’s what I ended up using (based on your hack):

c.Spawner.environment = {
        'JUPYTERLAB_SETTINGS_DIR': lambda spawner: '/home/' + spawner.user.name + '/.jupyter/lab/user-settings',
        'JUPYTERLAB_WORKSPACES_DIR': lambda spawner: '/home/' + spawner.user.name + '/.jupyter/lab/workspaces',
}

The above works fine.

bcornec · December 22, 2022, 1:34am

Seems that our issue is now fixed with jupyterlab 3.5.2 !

minrk · December 22, 2022, 8:13am

This was fixed by this patch (with lots of links to related discussion).

You can also set JUPYTER_PREFER_ENV_PATH=0:

c.Spawner.environment.update({"JUPYTER_PREFER_ENV_PATH": "0"})

to avoid trying to store things in the env, if you need to wait to upgrade.

Topic		Replies	Views
Unable to set default app for user environment in TLJH The Littlest JupyterHub jupyterlab , jupyterhub , help-wanted	5	760	December 9, 2024
On start, path on /users/ or /home/? JupyterHub jupyterlab	1	509	November 22, 2020
I can't access to JupyterLab from other users JupyterLab jupyterlab , jupyterhub , how-to , help-wanted	2	380	April 11, 2023
Allow JupyterHub users to install their own JupyterLab extensions JupyterHub jupyterlab , jupyterhub , how-to	5	894	February 5, 2024
How to change RStudio-Server working directory in Jupyterhub? JupyterHub jupyterhub , help-wanted	9	1243	August 3, 2023

JUPYTERLAB_SETTINGS_DIR defaults to JupyterHub root directory

Related topics