Post-install error: TypeError: 'NoneType' object is not iterable

Hello,

Trying to setup jupyterhub over k3s (over proxmox) using ansible and role GitHub - CyVerse-Ansible/ansible-jupyterhub: This role will install jupyterhub with CyVerse authentication and docker for jupyter notebooks.
From what I can tell, role follows what is described in Installing JupyterHub — Zero to JupyterHub with Kubernetes documentation
Jupyterhub pod in CrashLoopBackOff with error “TypeError: ‘NoneType’ object is not iterable”.

Looking on directions to troubleshoot further.

# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy
# uname -a
Linux nodehostname 5.15.107-1-pve #1 SMP PVE 5.15.107-1 (2023-04-20T10:05Z) x86_64 x86_64 x86_64 GNU/Linux
# k3s --version
k3s version v1.26.4+k3s1 (8d0255af)
go version go1.19.8
# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                      READY   STATUS                  RESTARTS          AGE    IP           NODE       NOMINATED NODE   READINESS GATES
kube-system   coredns-59b4f5bbd5-n76gl                  1/1     Running                 1 (32h ago)       2d2h   10.42.0.13   nodehostname   <none>           <none>
kube-system   local-path-provisioner-76d776f6f9-95psp   1/1     Running                 1 (32h ago)       2d2h   10.42.0.15   nodehostname   <none>           <none>
kube-system   metrics-server-7b67f64457-sd47x           1/1     Running                 1 (32h ago)       47h    10.42.0.14   nodehostname   <none>           <none>
default       continuous-image-puller-qhfb4             1/1     Running                 0                 32h    10.42.0.18   nodehostname   <none>           <none>
default       user-scheduler-7ddf954bc9-tzjkl           1/1     Running                 0                 32h    10.42.0.21   nodehostname   <none>           <none>
default       user-scheduler-7ddf954bc9-6gp4p           1/1     Running                 0                 32h    10.42.0.20   nodehostname   <none>           <none>
kube-system   svclb-proxy-public-207b7caf-bgbvh         1/1     Running                 0                 32h    10.42.0.19   nodehostname   <none>           <none>
default       proxy-55d6685ff7-nw7fr                    1/1     Running                 0                 32h    10.42.0.22   nodehostname   <none>           <none>
falco         falco-nh77w                               0/2     Init:CrashLoopBackOff   335 (4m37s ago)   30h    10.42.0.26   nodehostname   <none>           <none>
default       hub-f5c867d94-d2pfh                       0/1     CrashLoopBackOff        1 (11s ago)       21s    10.42.0.27   nodehostname   <none>           <none>
# kubectl logs hub-f5c867d94-d2pfh
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[E 2023-05-07 23:49:27.665 JupyterHub app:3297]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 3294, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 2773, in initialize
        self.load_config_file(self.config_file)
      File "/usr/local/lib/python3.9/site-packages/traitlets/config/application.py", line 110, in inner
        return method(app, *args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/traitlets/config/application.py", line 885, in load_config_file
        for (config, filename) in self._load_config_files(
      File "/usr/local/lib/python3.9/site-packages/traitlets/config/application.py", line 844, in _load_config_files
        config = loader.load_config()
      File "/usr/local/lib/python3.9/site-packages/traitlets/config/loader.py", line 625, in load_config
        self._read_file_as_dict()
      File "/usr/local/lib/python3.9/site-packages/traitlets/config/loader.py", line 658, in _read_file_as_dict
        exec(compile(f.read(), conf_filename, "exec"), namespace, namespace)
      File "/usr/local/etc/jupyterhub/jupyterhub_config.py", line 474, in <module>
        c[app].update(cfg)
    TypeError: 'NoneType' object is not iterable
# grep -vE '^#|^$' /opt/jupyterhub/config.yaml 
hub:
  config:
    JupyterHub:
      authenticator_class: none
    Authenticator:
  extraConfig:
    cyverse_config.py: |
      c.Spawner.http_timeout  = 600
singleuser:
  defaultUrl: "/lab"
  startTimeout: 600
  image:
    name: jupyter/datascience-notebook
    tag:  "latest"
  memory:
    guarantee: "1G"
  cpu:
    guarantee: 0.5
cull:
  enabled: false
proxy:
  chp:
    resources:
      requests:
        # 0m - 1000m
        cpu: 1000m
        # 100Mi - 600Mi
        memory: 500Mi
prePuller:
  hook:
    enabled: True
debug:
  enabled: true

same with authentication variants

hub:
  config:
    JupyterHub:
      authenticator_class: dummy
    Authenticator:
    DummyAuthenticator:
      password: "12345678"
  extraConfig:
    cyverse_config.py: |
      c.Spawner.http_timeout  = 600
singleuser:
  defaultUrl: "/lab"
  startTimeout: 600
  image:
    name: jupyter/datascience-notebook
    tag:  "latest"
  memory:
    guarantee: "1G"
  cpu:
    guarantee: 0.5
cull:
  enabled: false
proxy:
  chp:
    resources:
      requests:
        cpu: 1000m
        memory: 500Mi
prePuller:
  hook:
    enabled: True

This may be related to
hxxps://discourse.jupyter.org/t/how-am-i-supposed-to-use-existingsecret/5097
hxxps://z2jh.jupyter.org/en/stable/resources/reference.html#hub-existingsecret

Compared
hxxps://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/dev-config.yaml
hxxps://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/HEAD/jupyterhub/values.yaml
hxxps://github.com/CyVerse-Ansible/ansible-jupyterhub/blob/master/templates/config.yaml.j2
but no clear clues at this point.

(hxxps per new user links limit)

Can you try changing cyverse_config.py to something without a dot?

I don’t know what this part of config served. Not found reference to it.
If replacing cyverse_config.py by cyverse_config and redeploy/restart, same error.

Tried a new deployment over k3s over aws.
And I got the same error.

same if I fully remove the extraConfig part and relaunch

sudo kubectl delete pod hub-6f95748f5f-px6m8 --namespace default
sudo helm upgrade --cleanup-on-fail --install jupyterhub jupyterhub/jupyterhub --version=2.0.0 --namespace default --create-namespace --values /opt/jupyterhub/config.yaml --timeout 20m --debug
sudo kubectl get pods --all-namespaces -o wide
sudo kubectl logs hub-59c45b77df-9j25w

Manual install following quickstart Quickstart — JupyterHub documentation is working fine.

Is iterable error due to config.yaml or can it be from somewhere else?

Can you try removing the empty Authenticator: property? Or set it to an empty dictionary {}