# KeyError: 'port'

**URL:** <https://discourse.jupyter.org/t/keyerror-port/22303>\
**Category:** BinderHub\
**Tags:** help-wanted\
**Created:** [November 8, 2023, 4:51pm UTC](https://discourse.jupyter.org/t/keyerror-port/22303 "2023-11-08T16:51:43Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nick\_Cote](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nick_cote/32/11307_2.png) [@Nick\_Cote](https://discourse.jupyter.org/u/Nick_Cote)\
**Post date:** [November 8, 2023, 4:51pm UTC](https://discourse.jupyter.org/t/keyerror-port/22303/1 "2023-11-08T16:51:43Z")

</div>

We are doing an initial zero2binderhub deployment on top of a local kubernetes cluster with Harbor as a private registry. We have gotten to the point where Binder builds an image, pushes it to the repository, and goes to start but fails. I can provide more information and logs but was hoping this is something obvious like a missing port definition in a yaml file or something.  
In the Binder UI logs we see :

Found built image, launching…  
Launching server…  
Server requested  
Spawn failed: ‘port’  
Launch attempt 1 failed, retrying…  
Spawn failed: ‘port’  
Launch attempt 2 failed, retrying…  
Spawn failed: ‘port’  
Launch attempt 3 failed, retrying…  
Spawn failed: ‘port’

In the pod logs the related error message is :

Exception in Future \<Task finished name=‘Task-409’ coro=\<BaseHandler.spawn\_single\_user..finish\_user\_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:981\> exception=KeyError(‘port’)\> after timeout  
Traceback (most recent call last):  
File “/usr/local/lib/python3.11/site-packages/tornado/gen.py”, line 625, in error\_callback  
future.result()  
File “/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py”, line 988, in finish\_user\_spawn  
await spawn\_future  
File “/usr/local/lib/python3.11/site-packages/jupyterhub/user.py”, line 902, in spawn  
raise e  
File “/usr/local/lib/python3.11/site-packages/jupyterhub/user.py”, line 798, in spawn  
url = await gen.with\_timeout(timedelta(seconds=spawner.start\_timeout), f)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 2697, in \_start  
pod = await self.get\_pod\_manifest()  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 2003, in get\_pod\_manifest  
env=self.get\_env(), # Expansion is handled by get\_env  
^^^^^^^^^^^^^^  
File “”, line 104, in get\_env  
File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 2161, in get\_env  
env.update(self.\_expand\_all(self.environment))  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1836, in \_expand\_all  
return {k: self.\_expand\_all(v) for k, v in src.items()}  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1836, in   
return {k: self.\_expand\_all(v) for k, v in src.items()}  
^^^^^^^^^^^^^^^^^^^  
File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1838, in \_expand\_all  
return self.\_expand\_user\_properties(src)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1816, in \_expand\_user\_properties  
rendered = template.format(  
^^^^^^^^^^^^^^^^  
KeyError: ‘port’

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [November 8, 2023, 7:41pm UTC](https://discourse.jupyter.org/t/keyerror-port/22303/2 "2023-11-08T19:41:07Z")

</div>

Can you show us your full configuration?

---

<div class="post-metadata">

**Author:** ![Nick\_Cote](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/nick_cote/32/11307_2.png) [@Nick\_Cote](https://discourse.jupyter.org/u/Nick_Cote)\
**Post date:** [November 8, 2023, 8:12pm UTC](https://discourse.jupyter.org/t/keyerror-port/22303/3 "2023-11-08T20:12:35Z")

</div>

Someone else on my team is performing the actual install and I’ve been helping to try and figure out the issue. We are just using a Helm install and the config.yaml file is fairly basic:

> config:  
> BinderHub:  
> use\_registry: true  
> image\_prefix: hub.k8s.xxx.yyy/binder-dev-  
> hub\_url: [http://jhub-test.k8s.hub.k8s.xxx.yyy](http://jhub-test.k8s.hub.k8s.xxx.yyy)  
> DockerRegistry:  
> url: [https://hub.k8s.xxx.yyy](https://hub.k8s.xxx.yyy)  
> token\_url: [https://hub.k8s.xxx.yyy/service/token?service=harbor-registry](https://hub.k8s.xxx.yyy/service/token?service=harbor-registry)
> 
> service:  
> type: ClusterIP  
> ingress:  
> enabled: true  
> hosts:  
> - binder.k8s.xxx.yyy  
> annotations:  
> [kubernetes.io/ingress.class:](http://kubernetes.io/ingress.class:) nginx  
> [kubernetes.io/tls-acme:](http://kubernetes.io/tls-acme:) “true”  
> [cert-manager.io/cluster-issuer:](http://cert-manager.io/cluster-issuer:) incommon  
> tls:  
> - secretName: binder-k8s–tls  
> hosts:  
> - binder.k8s.xxx.yyy  
> #Added dind and imagecleaner to mitigate build issues with BH  
> imageBuilderType: dind

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [November 9, 2023, 5:57pm UTC](https://discourse.jupyter.org/t/keyerror-port/22303/4 "2023-11-09T17:57:21Z")

</div>

Can you turn on debug logging and show us your BinderHub and JupyterHub logs? Which version of the BinderHub helm chart are you using?

---

<div class="post-metadata">

**Author:** ![barkathali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/barkathali/32/11563_2.png) [@barkathali](https://discourse.jupyter.org/u/barkathali)\
**Post date:** [November 18, 2023, 1:08am UTC](https://discourse.jupyter.org/t/keyerror-port/22303/6 "2023-11-18T01:08:50Z")

</div>

@manics We are utilizing the latest version (_version=1.0.0-0.dev.git.3350.he7995d6_) of BinderHub helm chart.

As suggested, debug logging is enabled on JupyterHub.  
For BinderHub, in technical reference ([app — BinderHub documentation](https://binderhub.readthedocs.io/en/latest/reference/app.html#binderhub.app.BinderHub.debug)), there is an option to enable debug logging, can you help how it can utilized in helm chart. I tested with following config, but helm upgrade is failing.

> extraConfig:  
> c.BinderHub.debug = True

Below are the logs from hub pod when we try to build - [binder-examples/conda.git](https://github.com/binder-examples/conda.git)

> [D 2023-11-17 23:27:56.390 JupyterHub reflector:374] pods watcher timeout  
> [D 2023-11-17 23:27:56.391 JupyterHub reflector:289] Connecting pods watcher  
> [D 2023-11-17 23:27:57.087 JupyterHub scopes:877] Checking access to /hub/api/users/binder-examples-conda-eajqhpde via scope admin:users  
> [D 2023-11-17 23:27:57.088 JupyterHub scopes:677] Unrestricted access to /hub/api/users/binder-examples-conda-eajqhpde via admin:users  
> [D 2023-11-17 23:27:57.088 JupyterHub roles:281] Assigning default role to User binder-examples-conda-eajqhpde  
> [I 2023-11-17 23:27:57.090 JupyterHub roles:238] Adding role user for User: binder-examples-conda-eajqhpde  
> [I 2023-11-17 23:27:57.099 JupyterHub log:191] 201 POST /hub/api/users/binder-examples-conda-eajqhpde (binder@xx.xx.xx) 15.64ms  
> [D 2023-11-17 23:27:57.104 JupyterHub scopes:877] Checking access to /hub/api/users/binder-examples-conda-eajqhpde/servers/ via scope servers  
> [D 2023-11-17 23:27:57.104 JupyterHub scopes:677] Unrestricted access to /hub/api/users/binder-examples-conda-eajqhpde/servers/ via servers  
> [D 2023-11-17 23:27:57.105 JupyterHub user:431] Creating \<class ‘BinderSpawner’\> for binder-examples-conda-eajqhpde:  
> [D 2023-11-17 23:27:57.106 JupyterHub base:961] Initiating spawn for binder-examples-conda-eajqhpde  
> [D 2023-11-17 23:27:57.106 JupyterHub base:965] 0/64 concurrent spawns  
> [D 2023-11-17 23:27:57.106 JupyterHub base:970] 0 active servers  
> [I 2023-11-17 23:27:57.119 JupyterHub provider:659] Creating oauth client jupyterhub-user-binder-examples-conda-eajqhpde  
> [D 2023-11-17 23:27:57.132 JupyterHub user:794] Calling Spawner.start for binder-examples-conda-eajqhpde  
> [D 2023-11-17 23:27:57.132 JupyterHub spawner:3199] Applying KubeSpawner override for profile ‘Test Notebook’  
> [D 2023-11-17 23:27:57.132 JupyterHub spawner:3177] … overriding KubeSpawner value cpu\_limit=None  
> [D 2023-11-17 23:27:57.132 JupyterHub spawner:3177] … overriding KubeSpawner value mem\_limit=None  
> [D 2023-11-17 23:27:57.132 JupyterHub spawner:3177] … overriding KubeSpawner value cpu\_guarantee=0.013  
> [D 2023-11-17 23:27:57.132 JupyterHub spawner:3177] … overriding KubeSpawner value mem\_guarantee=0.904G  
> [I 2023-11-17 23:27:57.132 JupyterHub log:191] 202 POST /hub/api/users/binder-examples-conda-eajqhpde/servers/ (binder@xx.xx.xx.xx) 31.34ms  
> [E 2023-11-17 23:27:57.133 JupyterHub user:884] Unhandled error starting binder-examples-conda-eajqhpde’s server: ‘port’  
> Traceback (most recent call last):  
> File “/usr/local/lib/python3.11/site-packages/jupyterhub/user.py”, line 798, in spawn  
> url = await gen.with\_timeout(timedelta(seconds=spawner.start\_timeout), f)  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 2697, in \_start  
> pod = await self.get\_pod\_manifest()  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 2003, in get\_pod\_manifest  
> env=self.get\_env(), # Expansion is handled by get\_env  
> ^^^^^^^^^^^^^^  
> File “”, line 104, in get\_env  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 2161, in get\_env  
> env.update(self.\_expand\_all(self.environment))  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1836, in \_expand\_all  
> return {k: self.\_expand\_all(v) for k, v in src.items()}  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1836, in   
> return {k: self.\_expand\_all(v) for k, v in src.items()}  
> ^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1838, in \_expand\_all  
> return self.\_expand\_user\_properties(src)  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1816, in \_expand\_user\_properties  
> rendered = template.format(  
> ^^^^^^^^^^^^^^^^  
> KeyError: ‘port’
> 
> [D 2023-11-17 23:27:57.133 JupyterHub user:982] Stopping binder-examples-conda-eajqhpde  
> [D 2023-11-17 23:27:57.136 JupyterHub user:1002] Deleting oauth client jupyterhub-user-binder-examples-conda-eajqhpde  
> [D 2023-11-17 23:27:57.140 JupyterHub user:1005] Finished stopping binder-examples-conda-eajqhpde  
> [C 2023-11-17 23:27:57.149 JupyterHub base:1064] Aborting due to 5 consecutive spawn failures  
> [E 2023-11-17 23:27:57.149 JupyterHub gen:630] Exception in Future \<Task finished name=‘Task-2745’ coro=\<BaseHandler.spawn\_single\_user..finish\_user\_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:981\> exception=KeyError(‘port’)\> after timeout  
> Traceback (most recent call last):  
> File “/usr/local/lib/python3.11/site-packages/tornado/gen.py”, line 625, in error\_callback  
> future.result()  
> File “/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py”, line 988, in finish\_user\_spawn  
> await spawn\_future  
> File “/usr/local/lib/python3.11/site-packages/jupyterhub/user.py”, line 902, in spawn  
> raise e  
> File “/usr/local/lib/python3.11/site-packages/jupyterhub/user.py”, line 798, in spawn  
> url = await gen.with\_timeout(timedelta(seconds=spawner.start\_timeout), f)  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 2697, in \_start  
> pod = await self.get\_pod\_manifest()  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 2003, in get\_pod\_manifest  
> env=self.get\_env(), # Expansion is handled by get\_env  
> ^^^^^^^^^^^^^^  
> File “”, line 104, in get\_env  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 2161, in get\_env  
> env.update(self.\_expand\_all(self.environment))  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1836, in \_expand\_all  
> return {k: self.\_expand\_all(v) for k, v in src.items()}  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1836, in   
> return {k: self.\_expand\_all(v) for k, v in src.items()}  
> ^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1838, in \_expand\_all  
> return self.\_expand\_user\_properties(src)  
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1816, in \_expand\_user\_properties  
> rendered = template.format(  
> ^^^^^^^^^^^^^^^^  
> KeyError: ‘port’
> 
> [D 2023-11-17 23:27:57.153 JupyterHub scopes:877] Checking access to /hub/api/users/binder-examples-conda-eajqhpde/server/progress via scope read:servers  
> [D 2023-11-17 23:27:57.153 JupyterHub scopes:677] Unrestricted access to /hub/api/users/binder-examples-conda-eajqhpde/server/progress via read:servers  
> [I 2023-11-17 23:27:57.154 JupyterHub log:191] 200 GET /hub/api/users/binder-examples-conda-eajqhpde/server/progress (binder@xx.xx.xx.xx) 4.14ms  
> [D 2023-11-17 23:27:57.183 JupyterHub reflector:374] events watcher timeout  
> [D 2023-11-17 23:27:57.183 JupyterHub reflector:289] Connecting events watcher  
> [D 2023-11-17 23:27:58.262 JupyterHub log:191] 200 GET /hub/health (@xx.xx.xx.xx) 0.40ms  
> Task was destroyed but it is pending!  
> task: \<Task pending name=‘Task-1437’ coro=\<shared\_client..close\_client\_task() running at /usr/local/lib/python3.11/site-packages/kubespawner/clients.py:58\> wait\_for=\>  
> Exception ignored in: \<coroutine object shared\_client..close\_client\_task at 0x7fc4ca125300\>  
> RuntimeError: coroutine ignored GeneratorExit

Below are the logs from binder pod and UI respectively:

> [I 231117 23:27:57 launcher:197] Creating user binder-examples-conda-eajqhpde for image **redacted** /binderhub/binder-dev-binder-2dexamples-2dconda-8677da:f00a783146e9c6a2ed9726f01fc09fbfbad2f89e  
> [I 231117 23:27:57 launcher:257] Starting server for user binder-examples-conda-eajqhpde with image **redacted** /binderhub/binder-dev-binder-2dexamples-2dconda-8677da:f00a783146e9c6a2ed9726f01fc09fbfbad2f89e  
> [W 231117 23:27:57 web:1869] 500 GET /build/gh/binder-examples/conda.git/HEAD (10.42.2.0): Spawn failed: ‘port’  
> ===

> Found built image, launching…  
> Launching server…  
> Server requested  
> Spawn failed: ‘port’  
> Launch attempt 1 failed, retrying…  
> Spawn failed: ‘port’  
> Launch attempt 2 failed, retrying…  
> Spawn failed: ‘port’  
> Launch attempt 3 failed, retrying…  
> Spawn failed: ‘port’  
> Spawn failed: ‘port’

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [November 23, 2023, 12:07am UTC](https://discourse.jupyter.org/t/keyerror-port/22303/7 "2023-11-23T00:07:10Z")

</div>

Are you sure [KeyError: 'port' - #3 by Nick\_Cote](https://discourse.jupyter.org/t/keyerror-port/22303/3) is your full configuration?  
Your logs contain

> [@barkathali](#):
>
> ```auto
> [D 2023-11-17 23:27:57.132 JupyterHub spawner:3199] Applying KubeSpawner override for profile ‘Test Notebook’
> [D 2023-11-17 23:27:57.132 JupyterHub spawner:3177] … overriding KubeSpawner value cpu_limit=None
> [D 2023-11-17 23:27:57.132 JupyterHub spawner:3177] … overriding KubeSpawner value mem_limit=None
> [D 2023-11-17 23:27:57.132 JupyterHub spawner:3177] … overriding KubeSpawner value cpu_guarantee=0.013
> [D 2023-11-17 23:27:57.132 JupyterHub spawner:3177] … overriding KubeSpawner value mem_guarantee=0.904G
> 
> ```

which isn’t part of the default config.

---

<div class="post-metadata">

**Author:** ![barkathali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/barkathali/32/11563_2.png) [@barkathali](https://discourse.jupyter.org/u/barkathali)\
**Post date:** [November 27, 2023, 5:54pm UTC](https://discourse.jupyter.org/t/keyerror-port/22303/8 "2023-11-27T17:54:50Z")

</div>

@manics Below is the full configuration:

```auto
config:
  BinderHub:
    hub_url: https://jhub- **.k8s.***
    image_prefix: hub. ***.*** /binderhub/binder-dev-
    use_registry: true
  DockerRegistry:
    token_url: https://hub. ***.*** /service/token?service=harbor-registry
    url: https://hub. ***.***
extraPodSpec:
  dnsPolicy: ClusterFirstWithHostNet
  hostNetwork: true
imageBuilderType: dind
ingress:
  annotations:
    cert-manager**/cluster-issuer: incommon
    kubernetes**/ingress.class: nginx
    kubernetes**/tls-acme: "true"
  enabled: true
  hosts:
  - binder. ***.***
  tls:
  - hosts:
    - binder. ***.***
    secretName: binder-***
jupyterhub:
  debug:
    enabled: true
  hub:
    config:
      Authenticator:
        admin_users:
        - XXXXXX
        - XXXXXX
      BinderSpawner:
        auth_enabled: true
      GitHubOAuthenticator:
        allowed_organizations:
        - XXXXX
        client_id: <Redacted>
        client_secret: <Redacted>
        oauth_callback_url: https://jhub- ***.k8s.*** /hub/oauth_callback
        scope:
        - read:org
      JupyterHub:
        authenticator_class: github
      proxy.token: <Redacted>
    services:
      binder:
        apiToken: <Redacted>
  ingress:
    annotations:
      cert-manager**/cluster-issuer: incommon
      nginx.ingress.kubernetes**/proxy-body-size: 600m
      nginx.ingress.kubernetes**/proxy-read-timeout: "1800"
      nginx.ingress.kubernetes**/proxy-send-timeout: "1800"
      nginx.ingress.kubernetes**/rewrite-target: /
      nginx.ingress.kubernetes**/secure-backends: "true"
      nginx.ingress.kubernetes**/ssl-redirect: "true"
      nginx.ingress.kubernetes**/websocket-services: proxy-public
      nginx.org/client-max-body-size: 10m
      nginx.org/websocket-services: proxy-public
    enabled: true
    hosts:
    - jhub- ***.k8s.***
    ingressClassName: nginx
    tls:
    - hosts:
      - jhub- ***.k8s.***
      secretName: https-auto-incommon
  proxy:
    secretToken: <Redacted>
    service:
      type: ClusterIP
  singleuser:
    cmd: jupyterhub-singleuser
    extraEnv:
      DASK_DISTRIBUTED __DASHBOARD__ LINK: $(JUPYTERHUB_SERVICE_PREFIX)proxy/{port}/status
      DASK_GATEWAY __CLUSTER__ OPTIONS__IMAGE: $(JUPYTER_IMAGE_SPEC)
    image:
      name: pangeo/pangeo-notebook
      tag: xxxx.xx.xx
    profileList:
    - default: true
      description: Start a container with the chosen specifications on a node of this
        type
      display_name: Pangeo Notebook
      kubespawner_override:
        cpu_limit: null
        mem_limit: null
      profile_options:
        requests:
          choices:
            mem_1:
              default: true
              display_name: ~1 GB, ~0.125 CPU
              kubespawner_override:
                cpu_guarantee: 0.013
                mem_guarantee: 0.904G
            mem_4:
              display_name: ~4 GB, ~0.5 CPU
              kubespawner_override:
                cpu_guarantee: 0.025
                mem_guarantee: 1.809G
            mem_16:
              display_name: ~16 GB, ~2.0 CPU
              kubespawner_override:
                cpu_guarantee: 0.5
                mem_guarantee: 14.469G
          display_name: Container Selection
      slug: pangeo
    - description: Start a container on a dedicated node with a GPU
      display_name: NVIDIA Tensor A10
      kubespawner_override:
        environment:
          NVIDIA_DRIVER_CAPABILITIES: compute,utility
        extra_pod_config:
          runtimeClassName: nvidia
        extra_resource_limits:
          nvidia.com/gpu: "1"
        mem_guarantee: 14G
        mem_limit: null
      profile_options:
        image:
          choices:
            pytorch:
              default: true
              display_name: Pangeo PyTorch ML Notebook
              kubespawner_override:
                image: pangeo/pytorch-notebook:2023.05.18
              slug: pytorch
            tensorflow:
              display_name: Pangeo Tensorflow ML Notebook
              kubespawner_override:
                image: pangeo/ml-notebook:2023.05.18
              slug: tensorflow
          display_name: Image
      slug: gpu
    - description: 6 GB of memory; up to 4 vCPUS
      display_name: Test Custom
      kubespawner_override:
        cpu_guarantee: 1
        cpu_limit: 4
        mem_guarantee: 2G
        mem_limit: 6G
      profile_options:
        image:
          choices:
            tensorflow:
              display_name: <Redacted>
              kubespawner_override:
                image: <Redacted>
              slug: <redacted>
          display_name: Latest
    storage:
      extraVolumeMounts:
      - mountPath: /home/ **/**
        name: nfs-volume
        readOnly: true
      - mountPath: /**/collections
        name: **-collections
        readOnly: true
      - mountPath: /**/campaign
        name: campaign
        readOnly: true
      extraVolumes:
      - name: nfs-volume
        nfs:
          path: /mnt/**/jupyter/shared
          server: <Redacted>
      - name: **-collections
        nfs:
          path: /gpfs/fs1/collections
          server: <Redacted>
      - name: campaign
        nfs:
          path: /gpfs/csfs1
          server: <Redacted>
registry:
  password: <Redacted>
  url: https://hub.k8s. ***.***
  username: <Redacted>
service:
  type: ClusterIP

```

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 1, 2023, 3:33pm UTC](https://discourse.jupyter.org/t/keyerror-port/22303/9 "2023-12-01T15:33:08Z")

</div>

> [@barkathali](#):
>
> ` DASK_DISTRIBUTED __DASHBOARD__ LINK: $(JUPYTERHUB_SERVICE_PREFIX)proxy/{port}/status`

I think this is the cause of

> [@barkathali](#):
>
> File “/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py”, line 1816, in \_expand\_user\_properties  
> rendered = template.format(  
> ^^^^^^^^^^^^^^^^  
> KeyError: ‘port’

Try using `{{port}}` instead

> <https://github.com/jupyterhub/kubespawner/pull/642>
>
> \### Updated PR summary
> 
> Like for other of KubeSpawner's config options, \`KubeS…pawner.environment\` now supports variable expansion. This means that \`{\` and \`}\` gets reserved for use by variable expansion. To retain existing behavior, replace \`{\` and \`}\` with \`{{\` and \`}}\` respectively in this \`KubeSpawner.environment\`.
> 
> \`{username}\`, \`{userid}\`, \`{servername}\`, \`{hubnamespace}\`, \`{unescaped\_username}\`, and \`{unescaped\_servername}\` will now be expanded if found within strings of the \`KubeSpawner.environment\` config.
> 
> \`username\` and \`servername\` comes escaped to follow the \[DNS label standard\](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-label-names)
> 
> This feature is released as part of KubeSpawner 5.0.0.
> 
> \### Original PR description
> 
> In #644 spawner can add extra services after pod creation. To pass created service name, I've tried to set up notebook environment like this:
> 
> \`\`\`python
> c.KubeSpawner.environment = {"SPARK\_DRIVER\_HOST": "jupyter-{username}-spark-driver--{servername}"}
> \`\`\`
> But unfortunately, environment variables does not support substitutions. Now they are

---

<div class="post-metadata">

**Author:** ![barkathali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/barkathali/32/11563_2.png) [@barkathali](https://discourse.jupyter.org/u/barkathali)\
**Post date:** [December 1, 2023, 4:19pm UTC](https://discourse.jupyter.org/t/keyerror-port/22303/10 "2023-12-01T16:19:13Z")

</div>

thank you @manics !  
Will give it a try and let you know the outcome.

---

<div class="post-metadata">

**Author:** ![barkathali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/barkathali/32/11563_2.png) [@barkathali](https://discourse.jupyter.org/u/barkathali)\
**Post date:** [December 1, 2023, 5:07pm UTC](https://discourse.jupyter.org/t/keyerror-port/22303/11 "2023-12-01T17:07:53Z")

</div>

it worked, thank you @manics !