Error when spawning singleuser pod after config c.Spawner.env_keep on k8s

JupyterHub
,
1

Hi guys,

I had a jupyterhub (chart version 3.3.8), installed via argocd (just like helm), on Rancher Kubernetes Engine. I am trying to set an environment variable to singleuser kernel from k8s secrets.

I have tried setup one or both hub.extraEnv and singleuser.extraEnv with value like:

extraEnv:
    MY_VAR:
      valueFrom:
        secretKeyRef:
          name: MY_SECRET
          key: MY_VAR

But the var do not appear in the kernal.

Later I found out that maybe I need to setup hub.config.Spawner.env_keep to a list if whitelist var, since not all var are passed to singleuser kernel. I have config one or both hub.config.Spawner.env_keep, hub.config.KubeSpawner.env_keep, but this time the kernel cannot spawn with following error log:

Entered start.sh with args: start-notebook.py
2024-08-13T02:24:39.348698662Z Running hooks in: /usr/local/bin/start-notebook.d as uid: 0 gid: 0
2024-08-13T02:24:39.348846340Z Done running hooks in: /usr/local/bin/start-notebook.d
Updated the jovyan user:
2024-08-13T02:24:39.367637966Z - username: jovyan       -> vietvudanh
2024-08-13T02:24:39.367668402Z - home dir: /home/jovyan -> /home/vietvudanh
Update vietvudanh's UID:GID to 1100:100
Attempting to copy /home/jovyan to /home/vietvudanh...
2024-08-13T02:24:39.460988059Z Success!
2024-08-13T02:24:39.461035477Z Changing working directory to /home/vietvudanh/
2024-08-13T02:24:39.466162208Z Running hooks in: /usr/local/bin/before-notebook.d as uid: 0 gid: 0
Sourcing shell script: /usr/local/bin/before-notebook.d/10activate-conda-env.sh
2024-08-13T02:24:39.466915036Z /usr/local/bin/before-notebook.d/10activate-conda-env.sh: line 8: conda: command not found
2024-08-13T02:24:39.467207950Z Sourcing shell script: /usr/local/bin/before-notebook.d/10spark-config.sh
2024-08-13T02:24:39.467366537Z Done running hooks in: /usr/local/bin/before-notebook.d
Running as vietvudanh: start-notebook.py
2024-08-13T02:24:39.475307397Z /usr/bin/env: ‘python’: No such file or directory```

So anyone has experience with this error and help me with this?
2

env_keep controls which environment variable are forwarded from JupyterHub, you shouldn’t need to set it Spawners — JupyterHub documentation

Is the environment variable visible in a JupyterLab terminal?

Can you show us your full Z2JH configuration?

Can you also show us the output of kubectl get pod <podname> for a singleuser server?

3

I had opt to using mount volumes, which worked.
This is current helm values:

hub:
  config:
    Authenticator:
      admin_groups:
        - administrators
      allowed_groups:
        - users
    GenericOAuthenticator:
      client_id: 
      client_secret: 
      oauth_callback_url: 
      authorize_url: 
      token_url: 
      userdata_url:
      scope:
        - openid
        - profile
        - email
        - groups
      login_service: 
      username_claim: preferred_username
      claim_groups_key: groups
      userdata_params:
        state: state
    JupyterHub:
      authenticator_class: generic-oauth
      admin_access: false
    KubeSpawner:
      k8s_api_request_timeout: 10
  extraConfig:
    00-first-config: |
      def notebook_dir_hook(spawner):
        spawner.environment = {'NB_USER':spawner.user.name,'NB_UID':'1100'}

      c.Spawner.pre_spawn_hook = notebook_dir_hook
  db:
    type: postgres
    url: 
proxy:
  service:
    type: ClusterIP
  https:
    enabled: true
    type: offload
singleuser:
  networkPolicy:
    enabled: false
  uid: 0
  image:
    name: <custom image based on pyspark-notebook>
    tag: latest
  cmd:
  memory:
    limit: 4G
    guarantee: 2G
  cpu:
    limit: .5
    guarantee: .5
  defaultUrl: /tree/
  extraEnv:
    JUPYTERHUB_SINGLEUSER_APP: "notebook.notebookapp.NotebookApp"
    JUPYTERHUB_CRYPT_KEY: 
    CHOWN_HOME: "yes"
    CHOWN_HOME_OPTS: -R
    MY_VAR:
      valueFrom:
        secretKeyRef:
          name: MY_SECRET
          key: MY_VAR
  storage:
    homeMountPath: /home/{username}
    extraVolumes:
      - name: jupyterhub-shared
        persistentVolumeClaim:
          claimName: jupyterhub-shared-volume
    extraVolumeMounts:
      - name: jupyterhub-shared
        mountPath: /home/shared

For the pod description, I have checked via kubectl, env name MY_VAR was not set.

4

I’ve just tested this with Z2JH 3.3.8, and it’s working for me

Z2JH singleuser config:

singleuser:
  extraEnv:
    TEST:
      valueFrom:
        secretKeyRef:
          name: test
          key: test

Kubernetes secret:

$ kubectl get secret/test -o yaml

apiVersion: v1
data:
  test: c2VjcmV0
kind: Secret
metadata:
  creationTimestamp: "2024-08-21T10:13:23Z"
  name: test
  namespace: default
  resourceVersion: "88920"
  uid: 780ecef2-ecb1-499c-8731-ef00ee6b6f15
type: Opaque

Jupyter singleuser pod environment variables

$ kubectl get pod/jupyter-admin -o yaml
...
    env:
    ...
    - name: TEST
      valueFrom:
        secretKeyRef:
          key: test
          name: test
...

JupyterLab terminal:

$ echo $TEST
secret