Hello,
Trying to setup jupyterhub over k3s (over proxmox) using ansible and role GitHub - CyVerse-Ansible/ansible-jupyterhub: This role will install jupyterhub with CyVerse authentication and docker for jupyter notebooks.
From what I can tell, role follows what is described in Installing JupyterHub — Zero to JupyterHub with Kubernetes documentation
Jupyterhub pod in CrashLoopBackOff with error “TypeError: ‘NoneType’ object is not iterable”.
Looking on directions to troubleshoot further.
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy
# uname -a
Linux nodehostname 5.15.107-1-pve #1 SMP PVE 5.15.107-1 (2023-04-20T10:05Z) x86_64 x86_64 x86_64 GNU/Linux
# k3s --version
k3s version v1.26.4+k3s1 (8d0255af)
go version go1.19.8
# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-59b4f5bbd5-n76gl 1/1 Running 1 (32h ago) 2d2h 10.42.0.13 nodehostname <none> <none>
kube-system local-path-provisioner-76d776f6f9-95psp 1/1 Running 1 (32h ago) 2d2h 10.42.0.15 nodehostname <none> <none>
kube-system metrics-server-7b67f64457-sd47x 1/1 Running 1 (32h ago) 47h 10.42.0.14 nodehostname <none> <none>
default continuous-image-puller-qhfb4 1/1 Running 0 32h 10.42.0.18 nodehostname <none> <none>
default user-scheduler-7ddf954bc9-tzjkl 1/1 Running 0 32h 10.42.0.21 nodehostname <none> <none>
default user-scheduler-7ddf954bc9-6gp4p 1/1 Running 0 32h 10.42.0.20 nodehostname <none> <none>
kube-system svclb-proxy-public-207b7caf-bgbvh 1/1 Running 0 32h 10.42.0.19 nodehostname <none> <none>
default proxy-55d6685ff7-nw7fr 1/1 Running 0 32h 10.42.0.22 nodehostname <none> <none>
falco falco-nh77w 0/2 Init:CrashLoopBackOff 335 (4m37s ago) 30h 10.42.0.26 nodehostname <none> <none>
default hub-f5c867d94-d2pfh 0/1 CrashLoopBackOff 1 (11s ago) 21s 10.42.0.27 nodehostname <none> <none>
# kubectl logs hub-f5c867d94-d2pfh
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[E 2023-05-07 23:49:27.665 JupyterHub app:3297]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 3294, in launch_instance_async
await self.initialize(argv)
File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 2773, in initialize
self.load_config_file(self.config_file)
File "/usr/local/lib/python3.9/site-packages/traitlets/config/application.py", line 110, in inner
return method(app, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/traitlets/config/application.py", line 885, in load_config_file
for (config, filename) in self._load_config_files(
File "/usr/local/lib/python3.9/site-packages/traitlets/config/application.py", line 844, in _load_config_files
config = loader.load_config()
File "/usr/local/lib/python3.9/site-packages/traitlets/config/loader.py", line 625, in load_config
self._read_file_as_dict()
File "/usr/local/lib/python3.9/site-packages/traitlets/config/loader.py", line 658, in _read_file_as_dict
exec(compile(f.read(), conf_filename, "exec"), namespace, namespace)
File "/usr/local/etc/jupyterhub/jupyterhub_config.py", line 474, in <module>
c[app].update(cfg)
TypeError: 'NoneType' object is not iterable
# grep -vE '^#|^$' /opt/jupyterhub/config.yaml
hub:
config:
JupyterHub:
authenticator_class: none
Authenticator:
extraConfig:
cyverse_config.py: |
c.Spawner.http_timeout = 600
singleuser:
defaultUrl: "/lab"
startTimeout: 600
image:
name: jupyter/datascience-notebook
tag: "latest"
memory:
guarantee: "1G"
cpu:
guarantee: 0.5
cull:
enabled: false
proxy:
chp:
resources:
requests:
# 0m - 1000m
cpu: 1000m
# 100Mi - 600Mi
memory: 500Mi
prePuller:
hook:
enabled: True
debug:
enabled: true
same with authentication variants
hub:
config:
JupyterHub:
authenticator_class: dummy
Authenticator:
DummyAuthenticator:
password: "12345678"
extraConfig:
cyverse_config.py: |
c.Spawner.http_timeout = 600
singleuser:
defaultUrl: "/lab"
startTimeout: 600
image:
name: jupyter/datascience-notebook
tag: "latest"
memory:
guarantee: "1G"
cpu:
guarantee: 0.5
cull:
enabled: false
proxy:
chp:
resources:
requests:
cpu: 1000m
memory: 500Mi
prePuller:
hook:
enabled: True
This may be related to
hxxps://discourse.jupyter.org/t/how-am-i-supposed-to-use-existingsecret/5097
hxxps://z2jh.jupyter.org/en/stable/resources/reference.html#hub-existingsecret
Compared
hxxps://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/dev-config.yaml
hxxps://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/HEAD/jupyterhub/values.yaml
hxxps://github.com/CyVerse-Ansible/ansible-jupyterhub/blob/master/templates/config.yaml.j2
but no clear clues at this point.
(hxxps per new user links limit)