Named Server and unique Workdirectories

Is there somehow possible to define individual working directories for each named server?
(e.g. /home/jupyter-USERNAME/Servername/)

The background is that I would like to use a named server for each project I am working on and only have access to the data for this project.

Currently, all Named Servers use the same directory ((/home/jupyter-USERNAME/) of mine and I’m using subdirectorie, but I still have access to the data from the each projects.

I’m using Little Jupyterhub with the following configuration

c.JupyterHub.spawner_class = 'cdsdashboards.hubextension.spawners.variableusercreating.VariableUserCreatingSpawner'
c.SystemdSpawner.unit_name_template = 'jupyter-{USERNAME}{DASHSERVERNAME}'
c.JupyterHub.allow_named_servers = True
c.CDSDashboardsConfig.builder_class = 'cdsdashboards.builder.processbuilder.ProcessBuilder'
from cdsdashboards.app import CDS_TEMPLATE_PATHS
from cdsdashboards.hubextension import cds_extra_handlers
c.JupyterHub.template_paths = CDS_TEMPLATE_PATHS
c.JupyterHub.extra_handlers = cds_extra_handlers

Are there other ways to ensure data separation between projects? The separation does not have to be perfect, as I use the instance alone.

Since you’re already using a custom spawner I think you could override the working directory in each instance, this is where it’s set (assuming your custon spawner is derived from the default TLJH spawner):

Arguably it’d be nice if those properties could be overriden without overriding the entire start() method- if you think that’d be useful and have please feel free to open a GitHub issue.

There’s also this issue on making it easier to configure SystemdSpawner

but I’m not sure if that would help in this case as you need something dynamic.

Thank you very much.

I had to switch to the other spwaner because the one used by default does not support the named server properly. VariableUserCreatingSpawner was the only one that worked.

I have tried to implement your suggestion in the last few hours. Unfortunately, it does not work.
Working Directory is still “/home/jupyter-xxx/”. Although it is clearly visible in the log files that working_dir is set correctly and the subdirectory is also created.

I would probably have to change systemdspawner directly, but then I would also have to integrate the changes of the VariableUserCreatingSpawner

That’s too much work for now.

just for reference. My current non-working(!) configuration.

#
# https://raw.githubusercontent.com/ideonate/cdsdashboards/master/cdsdashboards/hubextension/spawners/variableusercreating.py
# https://raw.githubusercontent.com/ideonate/cdsdashboards/master/cdsdashboards/hubextension/spawners/variablesystemd.py
# https://raw.githubusercontent.com/jupyterhub/systemdspawner/main/systemdspawner/systemdspawner.py
#
import pwd
import os
from cdsdashboards.hubextension.spawners.variableusercreating import VariableUserCreatingSpawner

class myVariableUserCreatingSpawner(VariableUserCreatingSpawner):

    def start(self):
        try:
                unix_username = self._expand_user_vars(self.username_template)
                pwnam = pwd.getpwnam(unix_username)
        except KeyError:
                self.log.exception(f"No user named {unix_username} found in the system")
                raise
        self.working_dir = pwnam.pw_dir
        self.log.error(f"User-Dir {pwnam.pw_dir} Named Server: {self.name}")
        # If Named Server, use a subdirectory within the home directory
        if self.name: 
            subdir = "".join(ch for ch in self.name if (ch.isalnum() or ch in " _-#"))
            self.working_dir = os.path.join(self.working_dir, subdir) + "/"
            self.log.error("Using " + self.working_dir)
            if os.path.exists(self.working_dir):
                 pass
            else:
                os.mkdir(self.working_dir)            
        return super().start()


c.JupyterHub.spawner_class = myVariableUserCreatingSpawner
c.SystemdSpawner.unit_name_template = 'jupyter-{USERNAME}{DASHSERVERNAME}'
c.JupyterHub.allow_named_servers = True
c.CDSDashboardsConfig.builder_class = 'cdsdashboards.builder.processbuilder.ProcessBuilder'

from cdsdashboards.app import CDS_TEMPLATE_PATHS
from cdsdashboards.hubextension import cds_extra_handlers

c.JupyterHub.template_paths = CDS_TEMPLATE_PATHS
c.JupyterHub.extra_handlers = cds_extra_handlers

That won’t work, because you’re calling super().start() at the end so all your previous configuration is lost.

However I’ve remembered there’s a --ServerApp.root_dir=<existing/directory> argument you can pass to Jupyter-server. The directory must already exist, but you’ve got the code to do that.

self.get_args() is overridable, so you can include --ServerApp.root_dir= dynamically.

This won’t set the starting directory for terminals, only for notebooks.

1 Like

Super. You’ve put me on the right track.

Instead of --ServerApp.root_dir I had to use --ContentsManager.root_dir, but now it works.
Thanks a lot.

#
# https://raw.githubusercontent.com/ideonate/cdsdashboards/master/cdsdashboards/hubextension/spawners/variableusercreating.py
# https://raw.githubusercontent.com/ideonate/cdsdashboards/master/cdsdashboards/hubextension/spawners/variablesystemd.py
# https://raw.githubusercontent.com/jupyterhub/systemdspawner/main/systemdspawner/systemdspawner.py
#
import pwd
import os
from cdsdashboards.hubextension.spawners.variableusercreating import VariableUserCreatingSpawner

class myVariableUserCreatingSpawner(VariableUserCreatingSpawner):
 def get_args(self):
        try:
                unix_username = self._expand_user_vars(self.username_template)
                pwnam = pwd.getpwnam(unix_username)
        except KeyError:
                self.log.exception(f"No user named {unix_username} found in the system")
                raise
        pwnam.pw_name
        wd = pwnam.pw_dir
        # If Named Server, use a subdirectory within the home directory
        if self.name: 
            subdir = "".join(ch for ch in self.name if (ch.isalnum() or ch in " _-#"))
            wd = os.path.join(wd, subdir) + "/"
            self.log.error("Using " + wd)
            if os.path.exists(wd):
                 pass    
            else:
                os.mkdir(wd)                       
                uid = pwd.getpwnam(pwnam.pw_name).pw_uid
                gid = grp.getgrnam(pwnam.pw_name).gr_gid
                os.chown(wd, uid, gid)            
        self.log.error(f"XXX User-Dir {pwnam.pw_dir} Named Server: {self.name} Direcotry {wd} User: { pwnam.pw_name } ")
        args = list(super().get_args())
        #args.append("--ServerApp.root_dir=" + wd)
        args.append("--ContentsManager.root_dir=" + wd)
        return args



c.JupyterHub.spawner_class = myVariableUserCreatingSpawner
c.SystemdSpawner.unit_name_template = 'jupyter-{USERNAME}{DASHSERVERNAME}'
c.JupyterHub.allow_named_servers = True
c.CDSDashboardsConfig.builder_class = 'cdsdashboards.builder.processbuilder.ProcessBuilder'

from cdsdashboards.app import CDS_TEMPLATE_PATHS
from cdsdashboards.hubextension import cds_extra_handlers

c.JupyterHub.template_paths = CDS_TEMPLATE_PATHS
c.JupyterHub.extra_handlers = cds_extra_handlers

1 Like

I would also like to use a separate venv environment for each named server.

Can you think of a simple and automatic solution for my use case?

For separate venv, I’d use a launcher wrapper script instead of directly launching jupyterhub-singleuser. That makes environment setup like this easiest, to me.

It would look like:

c.Spawner.cmd = "/my/singleuser-start.sh"

and

#!/bin/bash -l
# this is /my/singleuser-start.sh

source activate /path/to/some/env

# after whatever setup, it should always end with this
exec jupyterhub-singleuser "$@"

Selecting the env path based on the server name can be in your Spawner or your startup script, whichever makes the most sense for you. Check out the default environment variables that JupyterHub sets (specifically $JUPYTERHUB_USER and $JUPYTERHUB_SERVER_NAME).

2 Likes

EDIT: unfortunately, the approach does not work after all. I had forgotten that the Standard Spawner does not work well with named servers.

thank you both. It looks like I have now realised my idea.

jupyterHub Configuration:

c.JupyterHub.allow_named_servers = True
c.Spawner.cmd = "/usr/local/bin/singleuser-start.sh"
c.MappingKernelManager.cull_busy = False
c.MappingKernelManager.cull_connected = False
c.MappingKernelManager.cull_idle_timeout = 0

Spawner Script

#!/bin/bash -i
cd $HOME
# Init Conda and restart bash
if ! grep "conda initialize" .bashrc;
then
        echo "Conda Init $HOME"
        conda init bash
        exec /bin/bash -i -c /usr/local/bin/singleuser-start.sh "$@"
fi

# Create Data-Directory and venv if necessary
dir=${JUPYTERHUB_CLIENT_ID}
if [ ! -e $dir ]
then
        echo "conda create $dir"
        mkdir "$dir"
        mkdir "$dir/data"
        conda create -y --name "$dir" 
fi
echo "conda activate $dir"
conda activate "$dir"
fulldir="${HOME}/${dir}/data"
exec jupyterhub-singleuser --NotebookApp.notebook_dir="${fulldir}" --ContentsManager.root_dir="${fulldir}" "$@"