Problem using Kubernetes for JupyterHub on a local infrastructure

#1

Hello,
I’m a newbie in Kubernetes, but I installed previously JupyterHub on several servers.
I followed the zero-to-jupyterhub.readthedocs.io documentation until the following state:

  • Kubernetes seems to be running,
  • Helm is installed
  • JupyterHub is installed, but:
  • the hub remains pending.
  • I tried to solve the “pod has unbound immediate PersistentVolumeClaims” problem with the description of a storageclass, without success.
    I realize that I need help in kubernetes management.
    I joined my version under.
    Thanks for your help, or any hint
    Sincerely Yours

Client Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.3”}
Server Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.3”

1 Like

#2

Same here and stucked!

0 Likes

#3

which kubernetes provider? That’s probably the most relevant right now. If you’ve deployed kubernetes yourself, there is a good chance that it doesn’t have dynamic volume provisioning enabled. This is important for deploying jupyterhub with persistent user storage.

0 Likes

#4

hello,
thanks for your answer.
Actually, I tried several ways of installing kubernetes, before leaving this way:

  • I achieved the installation, with one master node and two workers, all in a ready state. But it failed on pod deployment, without any useful messages in logs.

I have currently a working solution based on Docker and Swarm (https://zonca.github.io/2017/10/scalable-jupyterhub-docker-swarm-mode.html). Much more easy to install, with less complex management tools (just docker), and powerfull enough for jupyterhub.
Best regards

1 Like

#5

Same here. I think there’s some information missing in the tutorial for the config.yaml file. Can someone shed some light on this? The pod won’t start with just the secretToken per the instructions at https://z2jh.jupyter.org/en/latest/setup-jupyterhub.html.

0 Likes

#6

I sense a tutorial that tries to emerge there. Maybe you can help it do that. :wink:

0 Likes

#7

For those trying to get Kubernetes and JupyterHub up and running on local infrastructure, we had done some ansible scripts that do just that.

1 Like

#8

On local infrastructure (vms for testing) this is what I’m running into.
I tried to set storage type to none in config.yaml to disable PVC but the problem persists.
Any ideas?

user1@kubernetes-master:`~$ kubectl --namespace=jhub get pod					
NAME                     READY   STATUS    RESTARTS   AGE					
hub-8f5cc6f9f-8km2b      0/1     Pending   0          4m52s					
proxy-6b67b8956f-hxhcn   1/1     Running   0          4m52s					
user1@kubernetes-master:~$					
					
user1@kubernetes-master:~$ kubectl --namespace=jhub describe pod hub-8f5cc6f9f-8km2b					
Name:               hub-8f5cc6f9f-8km2b					
Namespace:          jhub					
Priority:           0					
PriorityClassName:  <none>					
Node:               <none>					
Labels:             app=jupyterhub					
component=hub					
hub.jupyter. org/network-access-proxy-api=true
hub.jupyter. org/network-access-proxy-http=true					
hub.jupyter. org/network-access-singleuser=true					
pod-template-hash=8f5cc6f9f					
release=jhub					
Annotations:        checksum/config-map: 1be5dfcf984d98fd90024277edee399349dd87ec6601703da743691c5737f3d9					
checksum/secret: a6bbce135ab5b011250cbede710e2497cb8db9e989bc1791f6eba875e5399c82					
Status:             Pending					
IP:					
Controlled By:      ReplicaSet/hub-8f5cc6f9f					
Containers:					
hub:					
Image:      jupyterhub/k8s-hub:0.8.2					
Port:       8081/TCP					
Host Port:  0/TCP					
Command:					
jupyterhub					
--config					
/srv/jupyterhub_config.py					
--upgrade-db					
Requests:					
cpu:     200m					
memory:  512Mi					
Environment:					
PYTHONUNBUFFERED:        1					
HELM_RELEASE_NAME:       jhub					
POD_NAMESPACE:           jhub (v1:metadata.namespace)					
CONFIGPROXY_AUTH_TOKEN:  <set to the key 'proxy.token' in secret 'hub-secret'>  Optional: false					
Mounts:					
/etc/jupyterhub/config/ from config (rw)					
/etc/jupyterhub/secret/ from secret (rw)					
/srv/jupyterhub from hub-db-dir (rw)					
/var/run/secrets/kubernetes.io/serviceaccount from hub-token-dlq74 (ro)					
Conditions:					
Type           Status					
PodScheduled   False					
Volumes:					
config:					
Type:      ConfigMap (a volume populated by a ConfigMap)					
Name:      hub-config					
Optional:  false					
secret:					
Type:        Secret (a volume populated by a Secret)					
SecretName:  hub-secret					
Optional:    false					
hub-db-dir:					
Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)					
ClaimName:  hub-db-dir					
ReadOnly:   false					
hub-token-dlq74:					
Type:        Secret (a volume populated by a Secret)					
SecretName:  hub-token-dlq74					
Optional:    false					
QoS Class:       Burstable					
Node-Selectors:  <none>					
Tolerations:     node.kubernetes. io/not-ready:NoExecute for 300s					
node.kubernetes. io/unreachable:NoExecute for 300s					
Events:					
Type     Reason            Age                  From               Message					
----     ------            ----                 ----               -------					
Warning  FailedScheduling  67s (x4 over 5m26s)  default-scheduler  pod has unbound immediate PersistentVolumeClaims					
user1@kubernetes-master:~$					
					
user1@kubernetes-master:~$ kubectl --namespace=jhub get pvc					
NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE					
hub-db-dir   Pending                                                     6m21s					
user1@kubernetes-master:~$					
					
user1@kubernetes-master:~$ kubectl --namespace=jhub describe pvc hub-db-dir					
Name:          hub-db-dir					
Namespace:     jhub					
StorageClass:					
Status:        Pending					
Volume:					
Labels:        app=jupyterhub					
chart=jupyterhub-0.8.2					
component=hub					
heritage=Tiller					
release=jhub					
Annotations:   <none>					
Finalizers:    [kubernetes. io/pvc-protection]					
Capacity:					
Access Modes:					
VolumeMode:    Filesystem					
Events:					
Type       Reason         Age                   From                         Message					
----       ------         ----                  ----                         -------					
Normal     FailedBinding  62s (x26 over 6m50s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set					
Mounted By:  hub-8f5cc6f9f-8km2b					
user1@kubernetes-master:~$
0 Likes

#9

Have you tried setting hub.db.type=sqlite-memory to disable the hub’s persistent volume? https://zero-to-jupyterhub.readthedocs.io/en/latest/reference.html#hub-db
If you still have problems can you paste your full config excluding sensitive info?

0 Likes

#10

That was it.

On local infrastructure, config.yaml looks like this after following the tut at z2jh and your info.
Thanks!

proxy:
  secretToken: "xxxx"

## This portion is missing from the tutorial for anyone trying to setup on bare metal.
hub:
  db:
    type: sqlite-memory

singleuser:
  storage:
    type: sqlite-memory
##
0 Likes