Jupyterhub 3.0.0 with ProfilesSpawner + batchspawner.SlurmSpawner failing to start

JupyterHub
,
1

Hi there,

I’m am really struggling to get this functional. We have an ancient version of jupyterhub (1.4.1) successfully running however trying to modernize it is proving very difficult (for me). I’ve gotten to the stage where jupyterhub is running, it can spawn jobs, which start on our slurm cluster, however jupyterlab never seems to show up in the web interface. If I turn on all the debugging I can see that the spawned job starts and notifies the hub. The hub gets the notification, but then nothing happens until things time out. It looks to me that the port is somehow never being set in batchspawner.py → BatchSpawnerBase → async def start(self): method. I’ve added my own debug code to the final while loop:

    while self.port == 0:
            await gen.sleep(self.startup_poll_interval)
            # Test framework: For testing, mock_port is set because we
            # don't actually run the single-user server yet.
            if hasattr(self, "mock_port"):
                self.port = self.mock_port

and as far as I can tell self.port is always == 0. I’ve seen messages indicating there was an issue with the traitlets package and followups suggesting that it has been fixed, or suggesting that one installs traitlets to a version less than 5, however the latter is not really an option as I have a number of other dependencies that require traitlets > 5.

Has anyone successfully configured Jupyterhub 3.0.0 with a ProfilesSpawner starting a slurm job?

Thanks for any pro tips you may have,
Dane.

2

A colleague has managed to find a fix that allows me to downgrade traitlets. I use pipenv to manage the python environment and that’s where I pin traitlets < 5. Interestingly that wasn’t enough — needed to also pin ipython >= 7. I can’t explain why that works, but there you go. So my Pipfile looks like this:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
traitlets = "<5"
ipython = ">=7"
jupyterhub = "*"
jupyterlab = "*"
jupyterhub-ldapauthenticator = "*"
ipympl = "*"
jupyter-resource-usage = "*"
wrapspawner = {git = "https://github.com/jupyterhub/wrapspawner"}
batchspawner = {git = "https://github.com/jupyterhub/batchspawner"}
jupyterhub-idle-culler = {git = "https://github.com/jupyterhub/jupyterhub-idle-culler"}
pycurl = "*"

[dev-packages]

[requires]
python_version = "3.10"

Now to see if hub works properly. :crossed_fingers:.

3

It doesn’t, unfortunately. The JupyterHub starts and appears to run fine, but then the single-server on the worker side fails with:

…
    from jupyter_server.traittypes import InstanceFromClasses
  File "/opt/jupyterhub/.venv/lib/python3.10/site-packages/jupyter_server/traittypes.py", line 7, in <module>
    from traitlets.utils.descriptions import describe
ModuleNotFoundError: No module named 'traitlets.utils.descriptions'
srun: error: jupyter-016: task 0: Exited with exit code 1
srun: launch/slurm: _step_signal: Terminating StepId=4718822.0
jupyterhub-singleuser ended gracefully

So I think the traitlets thing was a red herring in the end.

4

I feel like I have a similar issue with jupyterhub 4.0.0 and batchspawner. The gridengine job starts, I see some communication with the hub api, but the hub web interface never shows the notebook and eventually times out. One thing that seems like it might not be correct is:

JUPYTERHUB_BASE_URL=/
JUPYTERHUB_SERVICE_PREFIX=/user/orion/
JUPYTERHUB_SERVICE_URL=http://0.0.0.0:0/user/orion/

Have you had any luck resolving this?

5

I managed to resolve this by installing the latest git head of batchspawner. See `SlurmSpawner` behaviors are different if installed through pip compared to installed locally · Issue #266 · jupyterhub/batchspawner · GitHub

6

Yip, sorry I should’ve updated here. We solved it the same way.