Iām not 100% sure I followed your description, but since you are using TLJH with SSHSpawner, and only describe two machines, does that mean you are running notebook servers on the same machine as your ipython cluster? Or is there a third machine?
How can I configure a new ipython profile to control it directly from the IPython Clusters tab
To control a āremoteā cluster, you make two choices:
- where does your controller run, and
- where/how to launch the engines
From your description, Iām assuming you have
- a machine
local, where your notebook server is running
- a machine
remote, where you want your engines to run
- you want the controller on
remote
- you want to start the engine on
remote with mpi
- you want to be able to start/stop the cluster from the clusters tab (i.e. the only inputs are
profile and n)
The first step is to create a profile. Iām going to call it remotempi:
[local]$ ipython profile create --parallel remotempi
This will create ~/.ipython/profile_remotempi/ with various config files (if you leave out --parallel, the only difference is that it wonāt generate empty config files for ipython parallel. These arenāt required.).
The second step is to edit ~/.ipython/profile_remotempi/ipcluster_config.py. Add:
# local:.ipython/profile_remotempi/ipcluster_config.py
c = get_config() # noqa
# tell it to launch the controller via ssh
c.Cluster.controller_launcher_class = "ssh"
# the host to ssh to
c.SSHControllerLauncher.hostname = "remote"
# tell the controller to listen on all ips, so the client can connect to it
# (alternately, use ssh tunneling from the client)
c.Cluster.controller_args = ["--ip=*"]
Next, youāll want to tell it to launch the engines with MPI and ssh. This is more tedious than it should be.
on remote, create a profile with the same name (doesnāt need to match, but fewer things to configure if they do):
[remote]$ ipython profile create --parallel remotempi
And again on remote, you only need one config option, again in ~/.ipython/profile_remotempi/ipcluster_config.py:
# remote:.ipython/profile_remotempi/ipcluster_config.py
c = get_config() # noqa
# launch engines with mpiexec
c.Cluster.engine_launcher_class = "mpi"
Now, back to local, we want to tell it to launch engines on remote by delegating to ipcluster engines, i.e. it will run on local: ssh remote ipcluster engines --profile remotempi, which will amount to mpiexec ipengine on remote.
# back in local:.ipython/profile_remotempi/ipcluster_config.py
c.Cluster.engine_launcher_class = "sshproxy"
c.SSHProxyEngineSetLauncher.hostname = "remote"
At this point, we have:
- a local profile
remotempi
- a remote profile
remotempi
- controller launched on
remote via ssh, that is connectable from local
- starting engines, ultimately via something like
ssh remote mpiexec ipengine
(the ssh launchers take care of things like distributing connection files between the two machines).
how can I connect to running clusters from inside a Notebook?
With the above config, you should be able to start clusters with profile='remotempi':
import ipyparallel as ipp
cluster = ipp.Cluster(profile="remotempi")
rc = cluster.start_and_connect_sync()
Or, if youāve already started a running cluster, load the cluster info from a file and skip the start part:
cluster = ipp.Cluster.from_file(profile="remotempi")
rc = cluster.connect_client_sync()
Iāve put together a demo repo with all of this using docker-compose, which may be useful. It has a lot of boilerplate to set up the machines with ssh, keys, etc. which you probably have already done.