SQL OperationalError with JH default config

JupyterHub Zero to JupyterHub on Kubernetes
1

Hello,

I’m trying to deploy Jupyterhub on OVH k8s cluster but I can’t make it run with default configuration.

config.yaml:

# This file can update the JupyterHub Helm chart's default configuration values.
#
# For reference see the configuration reference and default values, but make
# sure to refer to the Helm chart version of interest to you!
#
# Introduction to YAML:     https://www.youtube.com/watch?v=cdLNKUoMc6c
# Chart config reference:   https://zero-to-jupyterhub.readthedocs.io/en/stable/resources/reference.html
# Chart default values:     https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/0883f9927b35ea2efad490a8e19dda8a4a8e12dc/jupyterhub/values.yaml
# Available chart versions: https://jupyterhub.github.io/helm-chart/
#

$ kubectl get node
NAME                                         STATUS   ROLES    AGE    VERSION
nodepool-813c1e34-xxxx-xxxx-98-node-121a16   Ready    <none>   4h     v1.22.2

$ helm version
version.BuildInfo{Version:"v3.6.3", GitCommit:"d506314abfb5d21419df8c7e7e68012379db2354", GitTreeState:"clean", GoVersion:"go1.16.5"}

helm upgrade --cleanup-on-fail \
  --install skl-jhub jupyterhub/jupyterhub \
  --namespace skl-jhub \
  --create-namespace \
  --version=1.2.0 \
  --values config.yaml

Log output :

$ kubectl logs --namespace skl-jhub hub-87f985879-btlkc
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[I 2021-12-14 12:23:46.895 JupyterHub app:2479] Running JupyterHub version 1.5.0
[I 2021-12-14 12:23:46.895 JupyterHub app:2509] Using Authenticator: jupyterhub.auth.DummyAuthenticator-1.5.0
[I 2021-12-14 12:23:46.895 JupyterHub app:2509] Using Spawner: kubespawner.spawner.KubeSpawner-1.1.0
[I 2021-12-14 12:23:46.896 JupyterHub app:2509] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.5.0
[E 2021-12-14 12:23:46.907 JupyterHub app:2989]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2986, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2521, in initialize
        self.init_db()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 1723, in init_db
        dbutil.upgrade_if_needed(self.db_url, log=self.log)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/dbutil.py", line 112, in upgrade_if_needed
        orm.check_db_revision(engine)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/orm.py", line 771, in check_db_revision
        current_table_names = set(inspect(engine).get_table_names())
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/inspection.py", line 64, in inspect
        ret = reg(subject)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py", line 182, in _engine_insp
        return Inspector._construct(Inspector._init_engine, bind)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py", line 117, in _construct
        init(self, bind)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py", line 128, in _init_engine
        engine.connect().close()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3165, in connect
        return self._connection_cls(self, close_with_result=close_with_result)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 96, in __init__
        else engine.raw_connection()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3244, in raw_connection
        return self._wrap_pool_connect(self.pool.connect, _connection)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3214, in _wrap_pool_connect
        Connection._handle_dbapi_exception_noconnection(
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2068, in _handle_dbapi_exception_noconnection
        util.raise_(
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
        raise exception
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3211, in _wrap_pool_connect
        return fn()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 307, in connect
        return _ConnectionFairy._checkout(self)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 767, in _checkout
        fairy = _ConnectionRecord.checkout(pool)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 425, in checkout
        rec = pool._do_get()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
        return self._create_connection()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
        return _ConnectionRecord(self)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 368, in __init__
        self.__connect()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 611, in __connect
        pool.logger.debug("Error on connect(): %s", e)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
        compat.raise_(
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
        raise exception
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 605, in __connect
        connection = pool._invoke_creator(self)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py", line 578, in connect
        return dialect.connect(*cargs, **cparams)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 584, in connect
        return self.dbapi.connect(*cargs, **cparams)
    sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
    (Background on this error at: http://sqlalche.me/e/14/e3q8)

Edit : Add PVC and PV description below

$ kubectl describe pv
Name:            ovh-managed-kubernetes-8tk3qj-pvc-bacb4137-67a9-437f-a047-f561751920c9
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: cinder.csi.openstack.org
Finalizers:      [kubernetes.io/pv-protection external-attacher/cinder-csi-openstack-org]
StorageClass:    csi-cinder-high-speed
Status:          Bound
Claim:           skl-jhub/hub-db-dir
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        1Gi
Node Affinity:   <none>
Message:         
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            cinder.csi.openstack.org
    FSType:            
    VolumeHandle:      a31dd08e-80a8-4ca8-91ab-7be7295b274e
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1639470543810-8081-cinder.csi.openstack.org
Events:                <none>

$ kubectl describe pvc
Name:          hub-db-dir
Namespace:     skl-jhub
StorageClass:  csi-cinder-high-speed
Status:        Bound
Volume:        ovh-managed-kubernetes-8tk3qj-pvc-bacb4137-67a9-437f-a047-f561751920c9
Labels:        app=jupyterhub
               app.kubernetes.io/managed-by=Helm
               chart=jupyterhub-1.2.0
               component=hub
               heritage=Helm
               release=skl-jhub
Annotations:   meta.helm.sh/release-name: skl-jhub
               meta.helm.sh/release-namespace: skl-jhub
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       hub-55c56748d6-dfljt
Events:
  Type    Reason                 Age   From                                                                                         Message
  ----    ------                 ----  ----                                                                                         -------
  Normal  ExternalProvisioning   26m   persistentvolume-controller                                                                  waiting for a volume to be created, either by external provisioner "cinder.csi.openstack.org" or manually created by system administrator
  Normal  Provisioning           26m   cinder.csi.openstack.org_csi-cinder-controllerplugin-0_f7428e10-cc91-4341-92c5-eaae61679ebc  External provisioner is provisioning volume for claim "skl-jhub/hub-db-dir"
  Normal  ProvisioningSucceeded  26m   cinder.csi.openstack.org_csi-cinder-controllerplugin-0_f7428e10-cc91-4341-92c5-eaae61679ebc  Successfully provisioned volume ovh-managed-kubernetes-8tk3qj-pvc-bacb4137-67a9-437f-a047-f561751920c9

Any ideas why this occur ?

Thanks for your help.

2

It sounds like a problem with your K8S storage. Perhaps it’s not writeable by the JupyterHub user, e.g. maybe it defaults to root-owned, or maybe the storage type is unsuitable (sqlite often has problems with NFS)?

If you quickly want to test Z2JH to rule out other problems you can disable all storage:

hub:
  db:
    type: sqlite-memory

singleuser:
  storage:
    type: none
1 Like
3

Yes you are right, it’s working when running the hub as root:

hub:
  fsGid: 0
  containerSecurityContext:
    runAsUser: 0
    runAsGroup: 0

But then I got this error for singleuser:

[D 2021-12-14 15:27:05.611 SingleUserNotebookApp application:174] Searching ['/home/jovyan', '/home/jovyan/.jupyter', '/home/jovyan/.local/etc/jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2021-12-14 15:27:05.611 SingleUserNotebookApp application:731] Looking for jupyter_config in /etc/jupyter
[D 2021-12-14 15:27:05.611 SingleUserNotebookApp application:731] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2021-12-14 15:27:05.611 SingleUserNotebookApp application:731] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2021-12-14 15:27:05.611 SingleUserNotebookApp application:731] Looking for jupyter_config in /home/jovyan/.local/etc/jupyter
[D 2021-12-14 15:27:05.612 SingleUserNotebookApp application:731] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2021-12-14 15:27:05.612 SingleUserNotebookApp application:731] Looking for jupyter_config in /home/jovyan
[D 2021-12-14 15:27:05.613 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /etc/jupyter
[D 2021-12-14 15:27:05.613 SingleUserNotebookApp application:753] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:753] Loaded config file: /etc/jupyter/jupyter_notebook_config.json
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:753] Loaded config file: /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /home/jovyan/.local/etc/jupyter
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /home/jovyan
[W 2021-12-14 15:27:05.616 SingleUserNotebookApp configurable:193] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py", line 537, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.9/site-packages/jupyter_core/application.py", line 264, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 845, in launch_instance
    app.initialize(argv)
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/mixins.py", line 840, in initialize
    result = super().initialize(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/mixins.py", line 573, in initialize
    return super().initialize(argv)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 88, in inner
    return method(app, *args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/notebook/notebookapp.py", line 2121, in initialize
    self.init_configurables()
  File "/opt/conda/lib/python3.9/site-packages/notebook/notebookapp.py", line 1650, in init_configurables
    connection_dir=self.runtime_dir,
  File "/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py", line 577, in __get__
    return self.get(obj, cls)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py", line 540, in get
    default = obj.trait_defaults(self.name)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py", line 1580, in trait_defaults
    return self._get_trait_default_generator(names[0])(self)
  File "/opt/conda/lib/python3.9/site-packages/jupyter_core/application.py", line 95, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/opt/conda/lib/python3.9/site-packages/jupyter_core/utils/__init__.py", line 11, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/opt/conda/lib/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.local'

So it’s definitely a permissions issue. I’m using default OVH storage class csi-cinder-high-speed I’m new to k8s and I dont know how to setup the correct permissions…

4

Many public cloud providers will setup K8S so that storage volumes respect the fsGroup, which is configured using fsGid in Z2JH (hub.fsGid, singleuser.fsGid).

If these configuration options don’t work for you you’ll need to look at how OVH configure their K8s storage volumes.

5

Yes OVH doesn’t seems to use the fsGroup parameter.
But I’ve found a workaround using initContainers (inspired by this comment) :

hub:
  initContainers:
    - name: permissions-fixer # arbitrary name
      image: alpine # or whatever suits you
      securityContext:
        runAsUser: 0 # run init container as root to fix permission
      volumeMounts: # mount persistent volume
        - name: pvc
          mountPath: /srv/jupyterhub
      command: # fix permissions using chown and chmod
        - sh
        - -c
        - (chown -R 1000:1000 /srv/jupyterhub; chmod -R 0755 /srv/jupyterhub;)
  # the rest of your config ...

singleuser:
  initContainers:
    - name: permissions-fixer
      image: alpine
      securityContext:
        runAsUser: 0
      volumeMounts:  # mount user persistent volume (based on volumeNameTemplate)
        - name: volume-{username}{servername}
          mountPath: /home/jovyan
      command:
        - sh
        - -c
        - (chown 1000:100 /home/jovyan; chmod 0755 /home/jovyan;)
  # the rest of your config ...

With this you’ve got the correct permissions setup and no rooted containers :+1: