SSHSpawner is not working as expected

JupyterHub
1

Hi,

I am trying to deploy sshspawner in my jupyter hub installation.

I have two servers where one acts as juputerhub and the other is where when the users login, the notebook servers are spanned up.

For authorisation and authentication, we use OAuth and it working fine without any issues.

The problem is:

When a user logs in, he is authenticated and authorised correctly. And a notebook server is spanned up in the remote machine. but the hub seems to be killing it (or tries to kill it). And eventually the user is shown with an error page (in edge it is: the page is not working at the moment).

By the way, I give the spawner command which is a shell script that is a custom one( c.Spawner.cmd = ‘/folder1/folder2/abc.sh‘ ).

I store this script in the remote machine under the given directories as well →folder1/folder2.

Inside this script, I initialize and export the necessary environment variables like: JYPYTERHUB_API_URL, JYPYTERHUB_ACTIVITY_URL, JUPYTERHUB_SERVICE_URL).

I have a python virtual environment in the remote machine. The virtual environment has jupyterhub installed. Post exporting the variables, I activate the virtual environment and the finally invoke the command ‘jupyterhub-singleuser‘.

The skeleton is:

#!bin/bash

export JYPYTERHUB_API_URL=’somevalue’

export JYPYTERHUB_ACTIVITY_URL=’’somevalue’

export JUPYTERHUB_SERVICE_URL=’somevalue’

source bin/activate command goes here…

jupyterhub-singleuser –port 4564 –ip 0.0.0.0

When I examined the log (of the hub server), It has the message of user logging in; his OAuth tokens are generated and the same is recorded.

And then it invokes the command kill -s 0 < /dev/null returned — — 0

Polling returned True ←- this a message logged in the hub log as well)

The remote host has two processes running. One belongs to the shell script ‘'abc.sh’’ and the other for jupyterhub-singleuser command.

What the hub tries to kill is of ‘abc.sh‘ process.

I make sure that the hub server is reachable from the host by invoking curl command (curl http://:8081/hub/api (the same ip what I give in the ENV variable) and it is working

So, I am not sure what is happening here. The command that give (under c.spawner.cmd), should it not have any other logic ? That shell script should always be ‘jupyterhub-singleuser‘ ?

Could someone please help / let me know if additional information is needed.

Thanks in advance.

2

Could you share your JupyterHub config and logs? And which SSH spawner are you using exactly?

3

Thanks for responding back and I really appreciate your time!

Here is my config (notable items):

c.JupyterHub.Spawner_class = ‘SSHSpawner’
c.SSHSpawner.remote_hosts = [‘xxx.com’] ← this is the remote host and is reachable by this hostname.
c.JupyterHub.hub_ip = ‘0.0.0.0’
c.JupyterHub.hub_connect_ip = ‘ip address of the hub’

o.GenericAuthenticator.scope =

c.Spawner.cmd = ‘path/to/a custom shell script’

This shell script is there in the given path in the remote machine. What this does is:

  1. it prints all the environment variables; so that I can verify it has all the required ones with correct values from the host machine.
  2. It alters below ENV vars (by exporting):
    • JUPYTERHUB_API_URL = https: //ip-addr-of-hub-machine:8081/hub/api/users//activity
    • JUPYTERHUB_ACTIVITY_URL = https ://ip-addr-of-hub-machine:8081/hub/api/users//activity
    • JUPYTERHUB_SERVICE_URL = http ://xxx.com:1234/user/ ← same ip as given below)
  3. Activate a python environment where jupyter hub is installed.
  4. Run the single user command as:
    exec jupyter-singleuser --port 1234 --ip 0.0.0.0 --debug

The SSHSpawner version is: 0.0.1a5
Since the SSHSpawner has some issues, I had code the port and explicitly export some ENV variables in the shell script.

In the Hub log:

I see that the Hub/SSHSpawner is able to launch a notebook server in the remote machine.
One line says that: JupyterHub W] 403 GET /hub/api/user (remote machine ip address)

And then these two lines:
command: kill -s 0 < /dev/null returned — — 0
Polling returned True

In the remote side, I see the notebook server running for a brief time.
I see the log file in the remote user home directory.

To examine: I removed the exec command and just ran the rest of the command. What I observed is that the notebook server (in remote machine) kept contacting the hub to update the activity. But got 403 error.

Addition note: My configurable proxy us running in 127.0.0.1 at port 8001.

Could you please let me know if you need any other information. Apologies, I am unable to copy/past the whole files.

4

Are you hardcoding the port like this? You are not supposed to do that. It is JupyterHub’s responsibility to find remote host and port and it launches the single user server on that socket. Why are you overriding the env vars in your custom script? If you provide full path to jupyterhub-singleuser for c.Spawner.cmd, there is no need to explicitly activate the virtual environment.

Also, I recommend you to use NERSC SSHSpawner, maybe it will work out of the box. If not this implementation works currently in production on JupyterHub 5.x but it is more advanced with many features you might not need.

1 Like
5

Thanks for your response.

Actually I want to make one successful notebook server working seamlessly using SSHSpawner. That’s why I make up things like hardcoding the port & environment variables.

Sure, I will take a look at these two spanners.

Though you have mentioned about the second one which works well in JupyterHub 5.X, I would like to check whether it is compatible with JypyterHub 4.1.5 version ?

Thanks

6

It works on JupyterHub 4.x as well

7

Great, is it not yet published in PyPi ?

8

No, that Spawner is very specific to a HPC platform and it is installed directly from the git repo. Maybe it is worth publishing it as the current SSH Spawner is quite outdated. The newer implementation has several features like:

  • Connection based load balancing between SSH hosts
  • Usage of SSH certificates to create a short lived user specific certificate for connections
  • Support for maximum wall time for single user servers

If you are motivated, I am willing to create a new SSH spawner which can be tested on your deployment.

1 Like
9

Sure! that will be great.

Actually my platform is very simple. It has a set few machines that need to be used to spread out the load (notebook servers). It does not have any other software installed for HPC.

For this infrastructure, can the jeanzay spawner be used?

10

No, JeanZaySpawner has been specifically designed for Jean Zay HPC platform. If you have experience with JupyterHub spawners, you can use that package as a reference to develop your own!

Saying so, I am trying to put together a new SSH spawner that should work out of the box on newer versions of JupyterHub. I will try to publish it on GitHub by next week if you are willing to test it!

11

Hi, sorry, I missed your message some how.

I started debugging one of the SSHSpawner packages and finally able to make it work. I needed to make some tweaks in the codebase. At the moment, I am going with that flow. If you publish the new SSHSpawner, I will test that out in the future.

Thanks!

1 Like