Previous spawn for sean.turner failed: 'JUPYTERHUB_SERVICE_PREFIX'

seanturner026 · March 18, 2024, 8:44pm

Hey all,

We’ve been running z2jh helm in our 1.26 EKS Cluster. We’ve upgraded the helm chart from version 2.0.0 to 3.2.1.

This involves upgrading the version of jupyterhub (software) from 3.0.0 to 4.0.2.

Trying to spawn with this upgraded version of software is causing an error. Jupyterhub says spawn failed, and the hub logs are below:

[I 2024-03-18 20:28:36.603 JupyterHub log:191] 302 GET /hub/spawn/sean.turner -> /hub/spawn-pending/sean.turner (sean.turner@::ffff:10.2.67.251) 630.85ms
[I 2024-03-18 20:28:36.616 JupyterHub reflector:282] watching for pods with label selector='component=singleuser-server' in namespace jupyterhub
[I 2024-03-18 20:28:36.617 JupyterHub reflector:282] watching for events with field selector='involvedObject.kind=Pod' in namespace jupyterhub
[I 2024-03-18 20:28:36.619 JupyterHub spawner:2551] Attempting to create pvc jupyterhub-sean-2eturner, with timeout 3
[I 2024-03-18 20:28:36.635 JupyterHub spawner:2567] PVC jupyterhub-sean-2eturner already exists, so did not create new pvc.
[E 2024-03-18 20:28:36.635 JupyterHub user:884] Unhandled error starting sean.turner's server: 'JUPYTERHUB_SERVICE_PREFIX'
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 798, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2718, in _start
        pod = await self.get_pod_manifest()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2014, in get_pod_manifest
        env=self.get_env(),  # Expansion is handled by get_env
            ^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2168, in get_env
        env.update(self._expand_all(self.environment))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1847, in _expand_all
        return {k: self._expand_all(v) for k, v in src.items()}
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1847, in <dictcomp>
        return {k: self._expand_all(v) for k, v in src.items()}
                   ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1849, in _expand_all
        return self._expand_user_properties(src)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1827, in _expand_user_properties
        rendered = template.format(
                   ^^^^^^^^^^^^^^^^
    KeyError: 'JUPYTERHUB_SERVICE_PREFIX'

[E 2024-03-18 20:28:36.855 JupyterHub gen:630] Exception in Future <Task finished name='Task-128' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:981> exception=KeyError('JUPYTERHUB_SERVICE_PREFIX')> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 625, in error_callback
        future.result()
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 902, in spawn
        raise e
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 798, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2718, in _start
        pod = await self.get_pod_manifest()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2014, in get_pod_manifest
        env=self.get_env(),  # Expansion is handled by get_env
            ^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2168, in get_env
        env.update(self._expand_all(self.environment))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1847, in _expand_all
        return {k: self._expand_all(v) for k, v in src.items()}
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1847, in <dictcomp>
        return {k: self._expand_all(v) for k, v in src.items()}
                   ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1849, in _expand_all
        return self._expand_user_properties(src)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1827, in _expand_user_properties
        rendered = template.format(
                   ^^^^^^^^^^^^^^^^
    KeyError: 'JUPYTERHUB_SERVICE_PREFIX'

[E 2024-03-18 20:28:36.880 JupyterHub pages:375] Previous spawn for sean.turner failed: 'JUPYTERHUB_SERVICE_PREFIX'

My understanding is that this prefix is automatically supposed to be configured as /users/{{username}}. However, I’m not able to understand why this is no longer happening.

We have customized our jupyterhub slightly, mainly subclassing the KubeSpawner to add taints and tolerations to the pods spun up so that we have access to GPU, and, adding some environment variables. Most other things are untouched. We also added an HTML template that is used to select what docker image should be deployed.

Any thoughts? I’ve directly tried to configure this on FooSpawner.environment (subclass of KubeSpawner), and also tried to add an init() function that sets self.server = True in order to set the JUPYTERHUB_SERVICE_PREFIX directly. Have also added it to singleUser.extraEnv on the helm chart to try and set the JUPYTERHUB_SERVICE_PREFIX on the template that is rendered in the stack trace, but still no luck.

Any thoughts? Am I thinking about this all wrong? Is there something else that I should be considering?

Thank you very much

mahendrapaipuri · March 19, 2024, 8:14am

Hey,

It is base Spawner that sets this JUPYTER_SERVICE_PREFIX env var in the single user environment. If you are setting env vars in your custom Spawner, I guess maybe you are overriding the existing env vars. Could you share your custom spawner implementation? That gives more leads on why this env var is missing in single user environment

mahendrapaipuri · March 19, 2024, 3:21pm

Cheers for the details. I dont see anything obvious from your custom spawner implementation. You are not really touching spawner.environment attribute which contains the env vars. Maybe try turning debug logs and see if it gives more clues

seanturner026 · March 19, 2024, 4:53pm

Interesting. I made this change:

 c.JupyterHub.spawner_class = BarSpawner
+print("DEBUG ", c.BarSpawner.environment )
 c.BarSpawner.environment = {"JUPYTERHUB_SERVICE_PREFIX": "/users/sean"}
+print("DEBUG ", c.BarSpawner.environment

And I see this in my logs:

DEBUG  <LazyConfigValue {}>
DEBUG  {'JUPYTERHUB_SERVICE_PREFIX': '/users/sean'}

Think this is where the issue would be? Should c.BarSpawner.environment actually have values rather than <LazyConfigValue {}>? Looks like <LazyConfigValue {}> seems to be an empty mapping of some type?

Edit - fixed capitalization

manics · March 19, 2024, 6:12pm

IIRC when Traitlets loads the config file it assumes classnames begin with a capital letter and properties begin with a lowercase letter. Try renaming your class to BarSpawner

seanturner026 · March 19, 2024, 6:26pm

This one actually does have a capitalization similar to BarSpawner. I renamed the Spawner before sharing the code and did not keep the capitalization consistent there. Apologies for the confusion there!

seanturner026 · March 20, 2024, 3:13pm

For reference I deployed the below changes:

c.JupyterHub.spawner_class = BarSpawner
+print("DEBUG ", c.BarSpawner.environment )
 c.BarSpawner.environment = {"JUPYTERHUB_SERVICE_PREFIX": "/users/sean"}
+print("DEBUG ", c.BarSpawner.environment

With the 3.0.0 version of Jupyterhub software and was able to spawn despite seeing the below in the logs:

DEBUG  <LazyConfigValue {}>
DEBUG  {'JUPYTERHUB_SERVICE_PREFIX': '/users/sean'}

Perhaps the LazyConfigValue isn’t empty actually and I need to do a better job revealing it.

I suppose I’ll compare all of the code changes to the spawner between 3.0.0 and whatever the latest 4.x.x version is. I had a look at the changelog ages ago and didn’t really see anything that would break the service prefix but leaving no stone unturned.

edit - for reference, here’s a link that can get to the changes for the spawner.py by doing a ctrl f search.

minrk · March 21, 2024, 10:25am

c.Spawner.environment does start empty. This is only the configured additional environment variables. Perhaps the key is that it’s unlikely to be JupyterHub changes that are causing the problem, but rather kubespawner.

First, you definitely shouldn’t define JUPYTERHUB_SERVICE_PREFIX yourself.

Can you share in your config or custom spawner class where you reference JUPYTERHUB_SERVICE_PREFIX in the environment?

The environment dict variables used are computed at runtime, in Spawner.get_env().

But this KeyError is occurring in _expand_user_properties, which expands templated variables, and does not reference values Spawner.environment.

My guess is that you have an environment variable like:

environment = {"SOME_THING": "https://something/${JUPYTERHUB_SERVICE_PREFIX}"}

This won’t work because the right hand side is interpreted as a Python string template, which you can reproduce with:

In [1]: "https://something/${JUPYTERHUB_SERVICE_PREFIX}".format(username="abc")
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[1], line 1
----> 1 "https://something/${JUPYTERHUB_SERVICE_PREFIX}".format(username="abc")

So if you can share where and how you are populating the environment with something that references {JUPYTERHUB_SERVICE_PREFIX}.

The fix is likely going to be replacing ${JUPYTERHUB_SERVICE_PREFIX} with self.server.prefix. Or if you’ve overridden get_env(), you

e.g. changing

environment["SOME_THING"] = "https://something${JUPYTERHUB_SERVICE_PREFIX}"

to

environment["SOME_THING"] = f"https://something{self.server.prefix}"

(note the f in the second example, evaluating the template immediately).

seanturner026 · March 22, 2024, 6:57pm

Can you share in your config or custom spawner class where you reference JUPYTERHUB_SERVICE_PREFIX in the environment?

Interesting!!!

This actually solved it

We are invoking that variable z2jh the helm chart:

singleuser:
  cmd: jupyterhub-singleuser
  extraEnv:
    # Configure a proxy link so that we can use LocalCluster Dask clusters in JupyterHub.
    # Uses jupyter-server-proxy and optionally dask-labextension.
    # https://docs.dask.org/en/stable/configuration.html#environment-variables
    # https://jobqueue.dask.org/en/latest/interactive.html#configuration
    DASK_DISTRIBUTED__DASHBOARD__LINK: "{JUPYTERHUB_SERVICE_PREFIX}proxy/{port}/status"

The initial error that gets thrown when trying to spawn complains that the Environment variable isn’t set.

stack trace

[E 2024-03-21 17:07:05.053 JupyterHub user:884] Unhandled error starting sean.turner's server: 'JUPYTERHUB_SERVICE_PREFIX'
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 798, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2718, in _start
        pod = await self.get_pod_manifest()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2014, in get_pod_manifest
        env=self.get_env(),  # Expansion is handled by get_env
            ^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2168, in get_env
        env.update(self._expand_all(self.environment))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1847, in _expand_all
        return {k: self._expand_all(v) for k, v in src.items()}
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1847, in <dictcomp>
        return {k: self._expand_all(v) for k, v in src.items()}
                   ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1849, in _expand_all
        return self._expand_user_properties(src)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1827, in _expand_user_properties
        rendered = template.format(
                   ^^^^^^^^^^^^^^^^
    KeyError: 'JUPYTERHUB_SERVICE_PREFIX'

[E 2024-03-21 17:07:05.247 JupyterHub gen:630] Exception in Future <Task finished name='Task-265' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:981> exception=KeyError('JUPYTERHUB_SERVICE_PREFIX')> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 625, in error_callback
        future.result()
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 902, in spawn
        raise e
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 798, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2718, in _start
        pod = await self.get_pod_manifest()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2014, in get_pod_manifest
        env=self.get_env(),  # Expansion is handled by get_env
            ^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2168, in get_env
        env.update(self._expand_all(self.environment))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1847, in _expand_all
        return {k: self._expand_all(v) for k, v in src.items()}
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1847, in <dictcomp>
        return {k: self._expand_all(v) for k, v in src.items()}
                   ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1849, in _expand_all
        return self._expand_user_properties(src)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1827, in _expand_user_properties
        rendered = template.format(
                   ^^^^^^^^^^^^^^^^
    KeyError: 'JUPYTERHUB_SERVICE_PREFIX'

[E 2024-03-21 17:07:05.273 JupyterHub pages:375] Previous spawn for sean.turner failed: 'JUPYTERHUB_SERVICE_PREFIX'

I updated the helm chart value here with double {{}} and I was able to spawn, and, can still access the dashboard

    DASK_DISTRIBUTED__DASHBOARD__LINK: "{{self.server.prefix}}proxy/{{port}}/status"

Thank you all for all of the assistance!!

Topic		Replies	Views
Spawn failed: Could not create PVC claim-jovyan with spawner.start_timeout JupyterHub jupyterhub	1	138	March 17, 2024
Spawn failed: Could not create PVC claim-foobar JupyterHub jupyterhub , help-wanted	11	367	April 23, 2024
Spawn failed: Server at http://some_ip:8888/user/a/ didn't respond in 30 seconds Zero to JupyterHub on Kubernetes jupyterhub	4	330	February 7, 2024
Spawn failed: Could not create pod JupyterHub community , help-wanted	5	1606	May 24, 2022
Jupyerhub spawner failing with an exception: "Initial list of events failed" Zero to JupyterHub on Kubernetes help-wanted	3	679	May 11, 2023

Previous spawn for sean.turner failed: 'JUPYTERHUB_SERVICE_PREFIX'

Related Topics