v1.PersistentVolume.ObjectMeta: v1.ObjectMeta.Annotations: ReadString: expects \" or n, but found {

JupyterHub Zero to JupyterHub on Kubernetes
1

Hi,
I posted this on KubeSpawner’s github but I was told to rather move here.

Therefore- I’m experiencing problems with rendering template. I’m copying the content of github issue:

Bug description

I deploy Zero2JupyterHub with KubeSpawner and want to use pre spawn hook to create some PV+PVC. I think I got my code right but the PV creation always fails with this error

Exception in Future <Task finished name='Task-38' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:895> exception=ApiException()> after timeout
18/03/2021 19:28:15     Traceback (most recent call last):
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
18/03/2021 19:28:15         future.result()
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 902, in finish_user_spawn
18/03/2021 19:28:15         await spawn_future
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 707, in spawn
18/03/2021 19:28:15         raise e
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 597, in spawn
18/03/2021 19:28:15         await maybe_future(spawner.run_pre_spawn_hook())
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/spawner.py", line 1012, in run_pre_spawn_hook
18/03/2021 19:28:15         return self.pre_spawn_hook(self)
18/03/2021 19:28:15       File "<string>", line 45, in my_pre_spawn_hook
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api/core_v1_api.py", line 8855, in create_persistent_volume
18/03/2021 19:28:15         return self.create_persistent_volume_with_http_info(body, **kwargs)  # noqa: E501
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api/core_v1_api.py", line 8942, in create_persistent_volume_with_http_info
18/03/2021 19:28:15         return self.api_client.call_api(
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py", line 348, in call_api
18/03/2021 19:28:15         return self.__call_api(resource_path, method,
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py", line 180, in __call_api
18/03/2021 19:28:15         response_data = self.request(
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py", line 391, in request
18/03/2021 19:28:15         return self.rest_client.POST(url,
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/rest.py", line 274, in POST
18/03/2021 19:28:15         return self.request("POST", url,
18/03/2021 19:28:15       File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/rest.py", line 233, in request
18/03/2021 19:28:15         raise ApiException(http_resp=r)
18/03/2021 19:28:15     kubernetes.client.exceptions.ApiException: (400)
18/03/2021 19:28:15     Reason: error
18/03/2021 19:28:15     HTTP response headers: HTTPHeaderDict({'Audit-Id': '91bedb10-4aa4-4646-bf62-6dbf61c75394', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '9c550017-bd97-4766-b8df-a536b4b92a11', 'X-Kubernetes-Pf-Prioritylevel-Uid': '224571b0-0ad6-4802-b62b-be2b33d641cc', 'Date': 'Thu, 18 Mar 2021 18:28:15 GMT', 'Content-Length': '489'})
18/03/2021 19:28:15     HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"PersistentVolume in version \"v1\" cannot be handled as a PersistentVolume: v1.PersistentVolume.ObjectMeta: v1.ObjectMeta.Annotations: ReadString: expects \" or n, but found {, error found in #10 byte of ...|labels\": {\"name\": \"v|..., bigger context ...|ons\": {\"name\": \"viktorias-data-sshfs\", \"labels\": {\"name\": \"viktorias-data-sshfs\"}, \"annotations\": {}|...","reason":"BadRequest","code":400}

I can’t find the root of the problem. Anything similar describing this problem is Helm3: ReadString: expects " or n, but found t · Issue #556 · hashicorp/terraform-provider-helm · GitHub

Expected behaviour

PV (with PVC) are created.

Actual behaviour

Spawning ends with error.

Your personal set up

I deploy ZeroToJupyrtHub v 0.11.1, before adding this part of extraConfig, deployment worked (without creating Pv and PVC of course). Sorry about messy dictionaries but these dictionaries were another problems and I had to make them like this.

  extraConfig:                                                                  
    pre-spawn-hook: |                                                                                                                                                                       
      from kubernetes.client.models import V1PersistentVolumeClaim              
      from kubernetes.client.models import V1PersistentVolumeClaimSpec          
      from kubernetes.client.models import V1ResourceRequirements               
      from kubernetes.client.models import V1LabelSelector                      
      from kubernetes.client.models import V1PersistentVolume                   
      from kubernetes.client.models import V1ObjectMeta                         
      from kubernetes.client.models import V1CSIVolumeSource                    
      from kubernetes.client.models import V1PersistentVolumeSpec               
                                                                                
                                                                                
      def my_pre_spawn_hook(spawner):                                           
            username = spawner.user.name                                        
            home_pvc_name = username + "-claim"                                 
            home_pv_name = username + "-data-sshfs"                             
            share_path = "/nfs4/home/" + username                               
            pk_secret = "jupyterhub-prod-ns/" + username + "-secret"            
            mount_path = "/home/meta/" + username                               
                                                                                
            pv = V1PersistentVolume()                                           
            pv.api_version = "v1"                                               
            pv.kind = "PersistentVolume"                                        
            d0 = dict(name=home_pv_name)                                        
            d1 = dict(name=home_pv_name,labels=d0,annotations={})               
            pv.metadata = V1ObjectMeta(d1)                                      
            d3 = dict(storage="100Gi")                                      
            d4 = dict(server="server-url.cz",port="22",share=share_path,privateKey=pk_secret,user=username,sshOpts="uid=1000,gid=100,allow_other")
            d2 = dict(accessModes=["ReadWriteMany"],capacity=d3,storage_class_name="sshfs",csi=V1CSIVolumeSource(driver="csi-sshfs",volume_attributes=d4))
            pv.spec = V1PersistentVolumeSpec(d2)                                
                                                                                
            pvc = V1PersistentVolumeClaim()                                     
            pvc.api_version = "v1"                                              
            pvc.kind = "PersistentVolumeClaim"                                  
            pvc.metadata = V1ObjectMeta()                                       
            pvc.metadata.name = home_pvc_name                                   
            pvc.metadata.annotations = {}                                       
            pvc.spec = V1PersistentVolumeClaimSpec()                            
            pvc.spec.access_modes = ['ReadWriteMany']                           
            pvc.spec.resources = V1ResourceRequirements()                       
            pvc.spec.resources.requests = {"storage":"100Gi"}                   
            pvc.spec.storage_class_name = "sshfs"                               
            pvc.spec.selector = V1LabelSelector()                               
            pvc.spec.selector.match_labels = {"name":home_pv_name}              
                                                                                
                                                                                
            spawner.api.create_persistent_volume(pv)                            
            spawner.api.create_namespaced_persistent_volume_claim("jupyterhub-prod-ns", pvc)
                                                                                
      c.KubeSpawner.pre_spawn_hook = my_pre_spawn_hook

EDIT
For anyone finding this, the problem is in somehow faulty PersistentVolume combination of dictionaries. I changed to

pv = V1PersistentVolume()                                           
            pv.api_version = "v1"                                               
            pv.kind = "PersistentVolume"                                        
            pv.metadata = V1ObjectMeta()                                        
            pv.metadata.name = home_pv_name                                     
            pv.metadata.labels = {"name":home_pv_name}                          
            pv.spec = V1PersistentVolumeSpec()                                  
            pv.spec.access_modes = ['ReadWriteMany']                            
            pv.spec.capacity = {"storage":"100Gi"}                              
            pv.spec.storage_class_name = "sshfs"                                
                                                                                
            volume_attr = {"server":"server.cz","port":"22","share":share_path,"privateKey":pk_secret,"user":username,"sshOpts":"uid=1000,gid=100,allow_other"}
            pv.spec.csi = V1CSIVolumeSource(driver="csi-sshfs",volume_attributes=volume_attr)

which doesn;t fail anymore on previous error but on missing volume_handle which is a problem of python kubernetes client.

2

I had to do this exact same thing a few months ago. Don’t have a write-up yet, but the code is datahub/common.yaml at 21e4a45c9f694578ec297c2947a0537f3bdcaa5b · berkeley-dsep-infra/datahub · GitHub. Particularly, I’m using the somewhat private spawner._make_create_pvc_request function instead of making the PVC myself. Not the best way to do this, but works very well since we already have PVC creating code.

I hope that helps?

3

The problem was a mess inside those dictionaries, access_mode got twice into the spec and caused bad things. I had to ditch dictionaries and use dot notation, also I’ve added check if pv+pvc already exists. I use spawner.api to call request functions and I somehow like better than spawner._make_create_pvc_request.

I’m including whole pre-spawn hook for anyone who might need it (or just as an inspiration). For me, it was hard finding any example on how to do it :sweat_smile:

extraConfig:                                                                  
    pre-spawn-hook: |                                                           
      from kubernetes.client.models import V1PersistentVolumeClaim              
      from kubernetes.client.models import V1PersistentVolumeClaimSpec          
      from kubernetes.client.models import V1ResourceRequirements               
      from kubernetes.client.models import V1LabelSelector                      
      from kubernetes.client.models import V1PersistentVolume                   
      from kubernetes.client.models import V1ObjectMeta                         
      from kubernetes.client.models import V1CSIPersistentVolumeSource          
      from kubernetes.client.models import V1PersistentVolumeSpec               
      from kubernetes.client.api import CoreV1Api                               
                                                                                
      def create_homes(spawner):                                           
            username = spawner.user.name                                        
            home_pvc_name = username + "-claim"                                 
            home_pv_name = username + "-data-sshfs"                             
            share_path = "/nfs4/home/" + username                               
            pk_secret = "jupyterhub-prod-ns/" + username + "-secret"            
            mount_path = "/home/meta/" + username                               
                                                                                
            v1 = client.CoreV1Api()                                             
            pvs = v1.list_persistent_volume(watch=False)                        
            pv = None                                                           
            for volume in pvs.items:                                            
              if volume.metadata.name == home_pv_name:                          
                pv = volume                                                     
            if not pv:                                                          
              pv = V1PersistentVolume()                                         
              pv.api_version = "v1"                                             
              pv.kind = "PersistentVolume"                                      
              pv.metadata = V1ObjectMeta()                                      
              pv.metadata.name = home_pv_name                                   
              pv.metadata.labels = {"name":home_pv_name}                        
              pv.spec = V1PersistentVolumeSpec()                                
              pv.spec.access_modes = ['ReadWriteMany']                          
              pv.spec.capacity = {"storage":"100Gi"}                            
              pv.spec.storage_class_name = "sshfs"                              
                                                                                
              volume_attr = {"server":"server.cz","port":"22",        
                             "share":share_path,"privateKey":pk_secret,         
                             "user":username,"sshOpts":"uid=1000,gid=100,allow_other"}
              pv.spec.csi = V1CSIPersistentVolumeSource(driver="csi-sshfs",     
                                                        volume_attributes=volume_attr,
                                                        volume_handle="data-id")
              spawner.api.create_persistent_volume(pv)                          

                                                                                
            pvcs = v1.list_persistent_volume_claim_for_all_namespaces(watch=False)
            pvc = None                                                          
            for claim in pvcs.items:                                            
              if claim.metadata.name == home_pvc_name:                          
                pvc = claim                                                     
            if not pvc:                                                         
              pvc = V1PersistentVolumeClaim()                                   
              pvc.api_version = "v1"                                            
              pvc.kind = "PersistentVolumeClaim"                                
              pvc.metadata = V1ObjectMeta()                                     
              pvc.metadata.name = home_pvc_name                                 
              pvc.spec = V1PersistentVolumeClaimSpec()                          
              pvc.spec.access_modes = ['ReadWriteMany']                         
              pvc.spec.resources = V1ResourceRequirements()                     
              pvc.spec.resources.requests = {"storage":"100Gi"}                 
              pvc.spec.storage_class_name = "sshfs"                             
              pvc.spec.selector = V1LabelSelector()                             
              pvc.spec.selector.match_labels = {"name":home_pv_name}            
              spawner.api.create_namespaced_persistent_volume_claim("jupyterhub-prod-ns", pvc)
                                                                                
      c.KubeSpawner.pre_spawn_hook = create_homes