Deploying JupyterHub with HTCondor


#1

Hi,

I’m trying to deploy JupyterHub+HTCondor using the BatchSpawner and the only example of such a configuration I found is this one.

I’m stuck on the stage when I managed to make JupyterHub submit the job, but batchspawner-singleuser fails to start on the condor nodes.

The actual command run on the node is:
/bin/sh -c exec' '/usr/share/miniconda3/bin/batchspawner-singleuser' '--ip="0.0.0.0"

The .jupyterhub.condor.err contains the only string:
JUPYTERHUB_API_TOKEN env is required to run jupyterhub-singleuser. Did you launch it manually?

If anybody here has the experience with HTCondor, can you check my configuration and help me debug the problem?

Here is my jupyterhub_config.py:

First, I set up our custom oauth2 using GenericOAuthenticator:


from oauthenticator.generic import LocalGenericOAuthenticator
c.JupyterHub.authenticator_class = LocalGenericOAuthenticator
c.GenericOAuthenticator.oauth_callback_url = 'redacted'
c.GenericOAuthenticator.client_id = 'redacted'
c.GenericOAuthenticator.client_secret = 'redacted'
c.GenericOAuthenticator.login_service = 'redacted'
c.GenericOAuthenticator.userdata_url = 'redacted'
c.GenericOAuthenticator.token_url = 'redacted'
c.GenericOAuthenticator.username_key = 'redacted'
c.GenericOAuthenticator.extra_params = {
    'client_id': 'redacted',
    'client_secret': 'redacted'}

This part seems to work since I can login to the web-interface.

The BatchSpawner part:

c.JupyterHub.spawner_class = 'batchspawner.CondorSpawner'
c.Spawner.http_timeout = 120
c.BatchSpawnerBase.req_nprocs = '1'
c.BatchSpawnerBase.req_runtime = '12:00:00'
c.BatchSpawnerBase.req_memory = '500mb'
c.CondorSpawner.exec_prefix = 'sudo -i -E -u {username}'
c.CondorSpawner.batch_script = '''
Universe  =  vanilla
Requirements = Machine=="jupyter_node_name_redacted"
Executable = /bin/sh
Arguments = \"-c 'export PATH=/usr/share/miniconda3/bin:$PATH; exec {cmd}'\"
ShouldTransferFiles = IF_NEEDED
Remote_Initialdir = /tmp
Output = {homedir}/.jupyterhub.condor.out
Error = {homedir}/.jupyterhub.condor.err
environment = "HOME=/tmp"
GetEnv = True
{options}
Queue
'''

One (maybe important) thing here is that the condor setup I have access to runs all jobs as nobody user, which doesn’t have a home dir. So I make it use /tmp as a home.

And other general settings:

c.JupyterHub.bind_url = 'my_url'
c.JupyterHub.hub_connect_ip = 'my_ip'
c.Authenticator.admin_users = {'admin_user'}
c.LocalAuthenticator.add_user_cmd = ['adduser']
c.LocalAuthenticator.create_system_users = True

Thanks.


#2

Solved it, see the original github issue.