Failsafe container?

Roy_Dragseth · October 1, 2024, 8:53am

Sometimes users mess up their setup so jupyterhub throws an internal error on startup. Is it possible to set up a failsafe container image so users can log on an clean up their mess themselves?

manics · October 1, 2024, 11:26am

Containers are ephemeral, so stopping and restarting a user’s server should result in a clean environment. Can you share some examples of the errors you’re seeing?

Roy_Dragseth · October 1, 2024, 1:51pm

Sorry, I was unclear. This happens on our current setup for jupyterhub which is vm-based. When users build their own failed environments on their home directory or mess up their path setups jupyterhub can throw an internal error on startup. Then an admin must log in as root and remove .config/.local/.ipython and so on. I know containers are ephermal, but the user environments survive because their home dirs are on PVCs. I’m trying to be preemtive here before we move our production to kubernetes. It would be nice to be able to provide a failsafe container that would work regardless of what damage a user has done to its environment.

manics · October 1, 2024, 2:58pm

will let you disable the user’s Jupyter configuration file which helps.

Unfortunately disabling the user’s Python directory seems more complicated…

github.com/python/cpython

It should be possible for a Python environment to disable the "user site" functionality without patching the stdlib

opened 05:30AM - 10 Nov 22 UTC

njsmith

type-feature stdlib

The [stdlib module `site.py`](https://github.com/python/cpython/blob/main/Lib/si…te.py) runs at startup, and is responsible for setting up a Python environment's basic search paths. There are various ways to customize this, e.g. you can drop in a `sitecustomize.py` file to add new search paths. One thing `site.py` does by default is to add the "user site path" to `sys.path`, e.g. `$HOME/.local/lib/python$VERSION/site-packages`. This makes sense and is quite handy for global python installs where users want to add a few packages without the fuss of creating a venv: just `pip install --user somepkg`. And this is becoming more important as moves towards doing `--user` installs by default when outside a venv. However, I think most people do *not* expect `pip install --user somepkg` will effectively inject `somepkg` into every venv that user ever creates. The whole point of creating a venv is to make an isolated environment. But, currently, `site.py` *always* adds the user site path to `sys.path` if possible, and there's no way to configure this through an envvar or `sitecustomize.py` – the only way to permanently disable it is to manually patch `site.py` in the stdlib. (Specifically: `site.py` runs `addusersitepackages` before it runs `execsitecustomize`, [see here](https://github.com/python/cpython/blob/87f5180cd79617223ac513e9f45933f774134e32/Lib/site.py#L602-L609).) This leads to surprising behavior, or [fragile hacks like modifying site.py with regexes](https://github.com/njsmith/posy/blob/d7f8c7fac942274e9b41eda1317f663e9dba0018/src/env.rs#L96-L105). There should be some way to create a Python environment where the user site path is disabled, without modifying the stdlib. (And venv should probably use it.) One option would be to move `execsitecustomize` earlier in the setup, before `addusersitepackages`, so it has the option of doing `import site; site.ENABLE_USER_SITE= False`. I think the main downside here is that right now, if someone has a `sitecustomize.py` file in their user site path, it will no longer be executed. This is ... kind of fine? `sitecustomize.py` is supposed to be reserved for the python environment itself; user-specific customization is supposed to go in `usercustomize.py`. So e.g. this kind of setup is already broken if a distro ships a Python with a custom `sitecustomize.py` in it – which Ubuntu, for example, already does! And if it's a major problem we could even detect this case and warn about it. (E.g. after `addusersitepackages` make a second check for the existence of `sitecustomize.py`.) Alternatively we could add some other kind of configuration file, `earlysitecustomize.py` or whatever. As long as it's something that's (a) on disk, so it's persistent for a given environment, and (b) can be dropped into a venv.

Roy_Dragseth · October 2, 2024, 7:16am

Thanks. This is obviously trickier than I thought. Most things are…

minrk · October 2, 2024, 8:08am

You can disable user packages by launching Python in isolated mode:

python3 -I -m jupyterhub.singleuser

or set

PYTHONNOUSERSITE=1
.

I think the linked issue is complicated by the fact that disabling user site requires controlling the command-line or environment variables and they want to be able to make it a property of the env itself. But we can set the env/cli in JupyterHub, so the existing mechanisms work for us.

Roy_Dragseth · October 2, 2024, 6:23pm

So all that would be needed is to add this to profileList?


- display_name: "Failsafe environment"
      description: "Use this if you get internal error on startup"
      kubespawner_override:
        image: jupyter/minimal-notebook:2343e33dec46
        enviroment: 
            PYTHONNOUSERSITE: "1"

Roy_Dragseth · October 3, 2024, 1:08pm

Did a quick test with this and it seems to be the right approach. The PYTHONNOUSERSITE gets propagated to the user environment.

Topic		Replies	Views
Getting occasional errors in my Jupyterhub instances Zero to JupyterHub on Kubernetes	5	240	April 5, 2024
Disable user conda and pip installations Zero to JupyterHub on Kubernetes jupyterhub , how-to , help-wanted	1	30	November 12, 2024
Make a container suitable for JupyterHub Zero to JupyterHub on Kubernetes jupyterhub , help-wanted	3	1067	November 15, 2022
User container fails to start JupyterHub jupyterhub , help-wanted , question	1	851	August 5, 2020
Spawned jupyterlab hangs for some users JupyterHub	2	875	November 6, 2019

Failsafe container?

Related topics