Hi,
hopefully this is the correct place for such an outreach. If that's not t…he case, please let me know.
# JupyterHub Outpost
In the past 12 months, we've created a small piece of software, which may add some valuable features to the current JupyterHub environment. I will try to explain the motivation behind it, its features, and its use cases. Please let me know what you think about it.
<details>
<summary>Disclaimer</summary>
This software is the current result of a seven-year journey to make multiple HPC systems and multiple Kubernetes clusters accessible by a central JupyterHub. It is currently used at https://jupyter.jsc.fz-juelich.de (and a few other project-specific JupyterHubs) hosted by the Juelich Supercomputing Centre in Juelich, Germany. <br><br>
Over the years we had different approaches. The JupyterHub Outpost is the latest and most promising result.
</details>
--
## Overview
The JupyterHub Outpost service ([documentation](https://jupyterhub-outpost.readthedocs.io/en/latest/), [source-code](https://github.com/kreuzert/jupyterhub-outpost)) allows JupyterHub to start jupyter servers on multiple, different, remote systems. Therefore, a single JupyterHub can be used as a central access point for a variety of different systems.
## Motivation
The JupyterHub community has created many useful JupyterHub Spawners over the past years, allowing JupyterHub to use the specific features of different systems. For most of these Spawners, JupyterHub has to run locally on the system itself. The JupyterHub Outpost service allows the use of these Spawners on remote systems with no modifications to their code, provided that JupyterHub uses the OutpostSpawner as a mediator.
While Spawners like the SSHSpawner can already spawn single-user servers on remote systems, they are not able to utilize system-specific features like KubeSpawner or BatchSpawner.
## Features
* Use one JupyterHub to offer single-user servers on multiple systems of potentially different types.
* Each (remote) system may use a different JupyterHub Spawner.
* Forward spawn events gathered by the remote Spawner to the user.
* Users may override the configuration of the remote Spawner at runtime (e.g. to select a different Docker Image) if allowed by JupyterHub Outpost administrators.
* Integrated SSH port forwarding solution to reach otherwise isolated remote single-user servers.
* Supports the JupyterHub `internal_ssl` feature.
* One JupyterHub Outpost can be connected to multiple JupyterHubs without the Hubs interfering with each other.
* Configuration of JupyterHub Outpost similar to the JupyterHub configuration.
## Configuration
### JupyterHub configuration
In this example, we will use the z2jh Helm Chart to install a central JupyterHub connected with three JupyterHub Outposts.
To keep things simple for now, all three Outposts will run on the same K8s cluster as JupyterHub. With a slightly different configuration, one could also install them on external K8s clusters.
<details><summary>Pre requirements</summary>
1. Create SSH keypairs to allow a connection between JupyterHub and both remote Outposts.
```
ssh-keygen -f remote-a -t ed25519 -N ''
ssh-keygen -f remote-b -t ed25519 -N ''
kubectl create secret generic --from-file=ssh-privatekey=remote-a --from-file=ssh-publickey=remote-a.pub remote-a-keypair
kubectl create secret generic --from-file=ssh-privatekey=remote-b --from-file=ssh-publickey=remote-b.pub remote-b-keypair
```
2. Create a username + password for the central JupyterHub on each Outpost. This is used for authentication when sending requests to one of the Outposts.
```
CENTRAL_HUB_USERNAME=centralhub # Use the same username for all outposts in this example
CENTRAL_HUB_REMOTE_A_PASSWORD=$(uuidgen)
CENTRAL_HUB_REMOTE_B_PASSWORD=$(uuidgen)
CENTRAL_HUB_LOCAL_PASSWORD=$(uuidgen)
CENTRAL_HUB_REMOTE_A_TOKEN=$(echo -n "${CENTRAL_HUB_USERNAME}:${CENTRAL_HUB_REMOTE_A_PASSWORD}" | base64 -w 0)
CENTRAL_HUB_REMOTE_B_TOKEN=$(echo -n "${CENTRAL_HUB_USERNAME}:${CENTRAL_HUB_REMOTE_B_PASSWORD}" | base64 -w 0)
CENTRAL_HUB_LOCAL_TOKEN=$(echo -n "${CENTRAL_HUB_USERNAME}:${CENTRAL_HUB_LOCAL_PASSWORD}" | base64 -w 0)
kubectl create secret generic --from-literal=SYSTEM_A_AUTHENTICATION=${CENTRAL_HUB_REMOTE_A_TOKEN} --from-literal=SYSTEM_B_AUTHENTICATION=${CENTRAL_HUB_REMOTE_B_TOKEN} --from-literal=SYSTEM_LOCAL_AUTHENTICATION=${CENTRAL_HUB_LOCAL_TOKEN} auth-keys
```
3. We also have to create secrets that will be used by the Outposts for authentication and encryption. In this example this is the same cluster, in the real setup this would be on a different cluster.
```
SECRET_KEY_LOCAL=$(python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')
SECRET_KEY_REMOTE_A=$(python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')
SECRET_KEY_REMOTE_B=$(python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')
# Create a namespace for each outpost
kubectl create namespace remotea
kubectl create namespace remoteb
# used for authentication
kubectl -n remotea create secret generic outpost-remote-a-users --from-literal=usernames=centralhub --from-literal=passwords=${CENTRAL_HUB_REMOTE_A_PASSWORD}
kubectl -n remoteb create secret generic outpost-remote-b-users --from-literal=usernames=centralhub --from-literal=passwords=${CENTRAL_HUB_REMOTE_B_PASSWORD}
kubectl create secret generic outpost-local-users --from-literal=usernames=centralhub --from-literal=passwords=${CENTRAL_HUB_LOCAL_PASSWORD}
# used for encryption
kubectl -n remotea create secret generic outpost-remote-a-cryptkey --from-literal=secret_key=${SECRET_KEY_REMOTE_A}
kubectl -n remoteb create secret generic outpost-remote-b-cryptkey --from-literal=secret_key=${SECRET_KEY_REMOTE_B}
kubectl create secret generic outpost-local-cryptkey --from-literal=secret_key=${SECRET_KEY_LOCAL}
```
</details>
<details><summary>Example JupyterHub installation.</summary>
Notes about this installation: <br>
- uses ingress and DNS name `your.dns.name` <br>
- uses [cert-manager](https://github.com/cert-manager/cert-manager) to get a let's encrypt certificate
- Networkpolicy hub must be disabled, because the configurable-http-proxy will reach the user jupyter-server via `hub:<random_port>` <br>
- Some features of the Outpost must be part of the configuration but are not used in this example (e.g. flavors, list-servers, and sshnoderestart). <br>
<br>
Example z2jh `jupyterhub_values.yaml` file:
```yaml=
### Some common values
cull:
enabled: false
ingress:
annotations:
acme.cert-manager.io/http01-edit-in-place: "false"
cert-manager.io/cluster-issuer: letsencrypt-cluster-issuer
ingressClassName: "nginx"
enabled: true
hosts:
- your.dns.name
tls:
- hosts:
- your.dns.name
secretName: vanilla-tls
prePuller:
hook:
enabled: false
continuous:
enabled: false
scheduling:
userScheduler:
enabled: false
hub:
args:
- -c
- >-
apt update && apt install -y openssh-client && su jovyan && pip install jupyterhub-outpostspawner && jupyterhub -f /usr/local/etc/jupyterhub/jupyterhub_config.py
containerSecurityContext:
runAsUser: 0
command:
- /bin/bash
services:
listservers:
name: list_servers
tunnelrestart:
name: tunnel_restart_nodes
outpostflavors:
name: outpost_flavors
loadRoles:
list_servers:
name: "list_servers-role"
description: "List all running servers. Used in JupyterHub Outpost cleanup cronjob."
scopes:
- "custom:servers:list"
services:
- "list_servers"
restart_forward:
name: "restart_forward-role"
description: "If a JupyterHub Outpost was restarted, all ssh port-forwarding processes may have to be recreated."
scopes:
- "custom:sshnode:restart"
services:
- "tunnel_restart_nodes"
outpost_flavors:
name: "outpost_flavors-role"
description: "Used to update flavors - A feature to allow full control of all resources by the Outpost"
scopes:
- "custom:outpostflavors:set"
services:
- "outpost_flavors"
networkPolicy:
enabled: false
allowNamedServers: true
config:
JupyterHub:
cleanup_proxy: false
default_url: /hub/home
pid_file: /tmp/jupyterhub.pid
tornado_settings:
slow_spawn_timeout: 0
slow_stop_timeout: 0
init_spawners_timeout: 0
extraEnv:
SYSTEM_A_AUTHENTICATION:
valueFrom:
secretKeyRef:
name: auth-keys
key: SYSTEM_A_AUTHENTICATION
SYSTEM_B_AUTHENTICATION:
valueFrom:
secretKeyRef:
name: auth-keys
key: SYSTEM_B_AUTHENTICATION
SYSTEM_LOCAL_AUTHENTICATION:
valueFrom:
secretKeyRef:
name: auth-keys
key: SYSTEM_LOCAL_AUTHENTICATION
extraVolumes:
- name: remote-a-keypair
secret:
secretName: remote-a-keypair
items:
- key: ssh-privatekey
path: ssh-privatekey
mode: 0400
- name: remote-b-keypair
secret:
secretName: remote-b-keypair
items:
- key: ssh-privatekey
path: ssh-privatekey
mode: 0400
extraVolumeMounts:
- name: remote-a-keypair
mountPath: /mnt/remote_a_ssh_key
- name: remote-b-keypair
mountPath: /mnt/remote_b_ssh_key
extraConfig:
customConfig: |-
import os
namespace = os.environ.get("POD_NAMESPACE", "default")
c.JupyterHub.custom_scopes = {
"custom:servers:list": {
"description": "List all running servers. Used in JupyterHub Outpost cleanup cronjob.",
},
"custom:sshnode:restart": {
"description": "If a JupyterHub Outpost was restarted, all ssh port-forwarding process may have to be recreated.",
},
"custom:outpostflavors:set": {
"description": "Used to update flavors - A feature to allow full control of all resources by the Outpost",
},
}
from jupyterhub.auth import DummyAuthenticator
c.JupyterHub.authenticator_class = DummyAuthenticator
from outpostspawner import OutpostSpawner
c.JupyterHub.spawner_class = OutpostSpawner
c.OutpostSpawner.http_timeout = 600
c.OutpostSpawner.start_timeout = 600
c.OutpostSpawner.public_api_url = "https://your.dns.name/hub/api"
c.OutpostSpawner.svc_name_template = f"centralhub-{{servername}}-{{userid}}"
c.OutpostSpawner.options_form = """
Choose a system:
<select name="system">
<option value="Local">Local</option>
<option value="Remote-A">Remote A</option>
<option value="Remote-B">Remote B</option>
</select>
"""
def options_from_form(formdata):
return { "system": formdata["system"][0] }
c.OutpostSpawner.options_from_form = options_from_form
c.OutpostSpawner.consecutive_failure_limit = 0
c.OutpostSpawner.ssh_port = 22
c.OutpostSpawner.ssh_username = "jhuboutpost"
system_options = {
"Remote-A": {
"ssh_during_startup": True,
"ssh_key": "/mnt/remote_a_ssh_key/ssh-privatekey",
"ssh_node": "outpost-remote-a-ssh.remotea.svc",
"request_url": "http://outpost-remote-a.remotea.svc:8080/services",
"auth_env_var": "SYSTEM_A_AUTHENTICATION"
},
"Remote-B": {
"ssh_during_startup": True,
"ssh_key": "/mnt/remote_b_ssh_key/ssh-privatekey",
"ssh_node": "outpost-remote-b-ssh.remoteb.svc",
"request_url": "http://outpost-remote-b.remoteb.svc:8080/services",
"auth_env_var": "SYSTEM_B_AUTHENTICATION"
},
"Local": {
"ssh_during_startup": False,
"request_url": "http://outpost-local.default.svc:8080/services",
"auth_env_var": "SYSTEM_LOCAL_AUTHENTICATION"
},
}
def ssh_during_startup(spawner):
system = spawner.user_options.get("system", "")
if system in system_options.keys():
return system_options[system]["ssh_during_startup"]
else:
raise Exception(f"No configuration for system {system} available")
c.OutpostSpawner.ssh_during_startup = ssh_during_startup
def ssh_key(spawner, port_forward_info):
system = spawner.user_options.get("system", "")
if system in system_options.keys():
return system_options[system]["ssh_key"]
else:
raise Exception(f"No ssh key configured for system {system}")
def ssh_node(spawner, port_forward_info):
system = spawner.user_options.get("system", "")
if system in system_options.keys():
return system_options[system]["ssh_node"]
else:
raise Exception(f"No ssh node configured for system {spawner.user_options.get('system', 'unknown_system')}")
c.OutpostSpawner.ssh_key = ssh_key
c.OutpostSpawner.ssh_node = ssh_node
c.OutpostSpawner.ssh_forward_options = { "StrictHostKeyChecking": "no" }
def request_url(spawner, user_options):
system = user_options.get("system", "")
if system in system_options.keys():
return system_options[system]["request_url"]
else:
raise Exception(f"No request_url configured for {system}")
c.OutpostSpawner.request_url = request_url
def request_headers(spawner, user_options):
system = user_options.get("system", "")
if system in system_options.keys():
auth = os.environ.get(system_options[system]["auth_env_var"], "NoTokenConfigured")
return {
"Content-Type": "application/json",
"Accept": "application/json",
"Authorization": f"Basic {auth}"
}
else:
raise Exception(f"No request headers configured for {system}")
c.OutpostSpawner.request_headers = request_headers
c.OutpostSpawner.request_kwargs = { "request_timeout": 30, "connect_timeout": 10 }
```
`helm upgrade --install -f jupyterhub_values.yaml vanilla https://github.com/jupyterhub/helm-chart/raw/gh-pages/jupyterhub-3.3.7.tgz`
</details>
<details><summary>Example Outpost installation</summary>
Use these three yaml files to install all three outposts:
<details><summary>Local Outpost example</summary>
`local.yaml`:
```yaml=
cryptSecret: outpost-local-cryptkey
outpostUsers: outpost-local-users
fullnameOverride: "outpost-local"
```
```
helm repo add jupyter-jsc https://kaas.pages.jsc.fz-juelich.de/helm-charts/
helm repo update
helm upgrade --install -f local.yaml outpost-local jupyter-jsc/jupyterhub-outpost
```
</details>
<details><summary>Remote A example</summary>
`remote-a.yaml`:
```yaml=
cryptSecret: outpost-remote-a-cryptkey
outpostUsers: outpost-remote-a-users
fullnameOverride: "outpost-remote-a"
sshPublicKeys:
- ... add previously created public key in here ...
```
```
helm repo add jupyter-jsc https://kaas.pages.jsc.fz-juelich.de/helm-charts/
helm repo update
helm -n remotea upgrade --install -f remote-a.yaml outpost-remote-a jupyter-jsc/jupyterhub-outpost
```
</details>
<details><summary>Remote B example</summary>
`remote-b.yaml`:
```yaml=
cryptSecret: outpost-remote-b-cryptkey
outpostUsers: outpost-remote-b-users
fullnameOverride: "outpost-remote-b"
sshPublicKeys:
- ... add previously created public key in here ...
outpostConfig: |
import logging
# Suppress /ping loggings, created by k8s livenessprobe
uvicorn_access = logging.getLogger("uvicorn.access")
class UvicornFilter(logging.Filter):
def filter(self, record):
try:
if "/ping" in record.args:
return False
except:
pass
return True
uvicorn_access.addFilter(UvicornFilter())
from kubespawner import KubeSpawner
c.JupyterHubOutpost.spawner_class = KubeSpawner
c.KubeSpawner.image = "jupyter/scipy-notebook:latest"
```
```
helm repo add jupyter-jsc https://kaas.pages.jsc.fz-juelich.de/helm-charts/
helm repo update
helm -n remoteb upgrade --install -f remote-b.yaml outpost-remote-b jupyter-jsc/jupyterhub-outpost
```
</details>
</details><br>
Using the yaml files and commands above, you will deploy a JupyterHub, which will be able to start jupyter-server for users on two "remote" systems (in this example it's only a different namespace) or on the "local" cluster (the one where JupyterHub itself is running).
## Architecture
The following images will describe the idea behind the JupyterHub Outpost and its architecture.
### Simple setup
<img src="https://jupyterhub-outpost.readthedocs.io/en/latest/_images/architecture-default.png" alt="JupyterHub Outpost Architecture1" width="400">
This image shows the architecture of a JupyterHub Outpost running on a remote Kubernetes cluster ([more](https://jupyterhub-outpost.readthedocs.io/en/latest/architecture.html#default-setup-with-remote-system)):
* JupyterHub was installed via zero2JupyterHub on a k8s cluster. The used Spawner is the JupyterHub Outpostspawner, which was designed to communicate with multiple JupyterHub Outposts.
* On a remote Kubernetes cluster a JupyterHub Outpost was installed via its [Helm Chart](https://artifacthub.io/packages/helm/jupyter-jsc/jupyterhub-outpost)
* The configured Spawner in this scenario is most likely to be the KubeSpawner, but can be anything else, too.
* At the end of the `Spawner.start()` function the final address of the Notebook container is already known (e.g. configuring `c.KubeSpawner.pod_name_template`)
* The JupyterHub Outpost will create a `Kubernetes Service` and a ssh port-forwarding tunnel to the JupyterHub Outpost. Therefore, the configurable-http-proxy can route to the jupyter server as always.
### Local + Remote Resources
One can use as many systems with JupyterHub Outposts as you like. The following image shows an example for one remote JupyterHub Outpost and one installed on the same cluster as the JupyterHub itself. This allows you to start jupyter servers on two different Kubernetes clusters.
<img src="https://jupyterhub-outpost.readthedocs.io/en/latest/_images/architecture-default-plus-local.png" alt="JupyterHub Outpost Architecture1" width="600">
* If the user chooses the `local` cluster (the same as JupyterHub is running at), there's no need to create a port-forwarding tunnel. The configurable-http-proxy can route directly to the jupyter server.
* You can connect as many remote systems to the JupyterHub as you like.
### External Tunneling
In this example we're using an external pod to manage ssh-tunnels. This allows us to avoid a bottleneck with a single JupyterHub pod, which has to manage all tunnels.
<img src="https://jupyterhub-outpost.readthedocs.io/en/latest/_images/architecture-external.png" alt="JupyterHub Outpost architecture3" width="400">
* Jupyter servers from users are still reachable if the JupyterHub pod must be restarted (although the `/api/auth` endpoint is not available during that time)
### Delayed Tunneling
This setup shows the start of a single-user server on an external system where the service address of the single-user server cannot be determined during the start process. This may for example be the case if you have configured a Spawner which spawns single-user server on remote Batch systems. The single-user server has to contact the OutpostSpawner itself, once its location is settled.
<img src="https://jupyterhub-outpost.readthedocs.io/en/latest/_images/architecture-delayed.png" alt="JupyterHub Outpost architecture3" width="400">
* The start process of the jupyter server must send a request to the central JupyterHub, sending its final address. This can be used to create a port-forwarding tunnel to the jupyter server.
* Useful for HPC-Systems, when the jupyter-server may take a long time to start and its final address is not defined at the end of the `Spawner.start()` function (e.g. slurm will decide on which compute node the server will be running)
## Flavors
In this short demo, the `flavors` feature was not used.
The Outpost allows you to offer multiple flavors of your resources, depending on the JupyterHub or even the specific user. While the feature is crucial to keep control of the offered resources, it's not intuitive to configure yet. This will be improved within the next few months.
Example configuration for hub-specific flavors: [here](https://github.com/kreuzert/jupyterhub-outpost/blob/main/project/tests/test_routes/simple_flavors_max_0.py)
Example configuration for user-specific flavors: [here](https://github.com/kreuzert/jupyterhub-outpost/blob/main/project/tests/test_routes/simple_authorization.py)
## Outlook
As mentioned at the top, we're using this piece of software in our current JupyterHub installation with roughly ~350 unique users per [month](https://gitlab.jsc.fz-juelich.de/jupyterjsc/k8s/metrics/-/blob/jupyter.jsc.fz-juelich.de/user_metrics_last_30_days.csv) , to offer them access to six different systems with one central JupyterHub.