# SQL OperationalError with JH default config

**URL:** <https://discourse.jupyter.org/t/sql-operationalerror-with-jh-default-config/12207>\
**Category:** Zero to JupyterHub on Kubernetes\
**Created:** [December 14, 2021, 12:40pm UTC](https://discourse.jupyter.org/t/sql-operationalerror-with-jh-default-config/12207 "2021-12-14T12:40:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![brospars](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/brospars/32/5991_2.png) [@brospars](https://discourse.jupyter.org/u/brospars)\
**Post date:** [December 14, 2021, 12:40pm UTC](https://discourse.jupyter.org/t/sql-operationalerror-with-jh-default-config/12207/1 "2021-12-14T12:40:26Z")

</div>

Hello,

I’m trying to deploy Jupyterhub on OVH k8s cluster but I can’t make it run with default configuration.

`config.yaml`:

```auto
# This file can update the JupyterHub Helm chart's default configuration values.
#
# For reference see the configuration reference and default values, but make
# sure to refer to the Helm chart version of interest to you!
#
# Introduction to YAML: https://www.youtube.com/watch?v=cdLNKUoMc6c
# Chart config reference: https://zero-to-jupyterhub.readthedocs.io/en/stable/resources/reference.html
# Chart default values: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/0883f9927b35ea2efad490a8e19dda8a4a8e12dc/jupyterhub/values.yaml
# Available chart versions: https://jupyterhub.github.io/helm-chart/
#

```

```auto
$ kubectl get node
NAME STATUS ROLES AGE VERSION
nodepool-813c1e34-xxxx-xxxx-98-node-121a16 Ready <none> 4h v1.22.2

```

```auto
$ helm version
version.BuildInfo{Version:"v3.6.3", GitCommit:"d506314abfb5d21419df8c7e7e68012379db2354", GitTreeState:"clean", GoVersion:"go1.16.5"}

```

```bash
helm upgrade --cleanup-on-fail \
  --install skl-jhub jupyterhub/jupyterhub \
  --namespace skl-jhub \
  --create-namespace \
  --version=1.2.0 \
  --values config.yaml

```

Log output :

```auto
$ kubectl logs --namespace skl-jhub hub-87f985879-btlkc
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[I 2021-12-14 12:23:46.895 JupyterHub app:2479] Running JupyterHub version 1.5.0
[I 2021-12-14 12:23:46.895 JupyterHub app:2509] Using Authenticator: jupyterhub.auth.DummyAuthenticator-1.5.0
[I 2021-12-14 12:23:46.895 JupyterHub app:2509] Using Spawner: kubespawner.spawner.KubeSpawner-1.1.0
[I 2021-12-14 12:23:46.896 JupyterHub app:2509] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.5.0
[E 2021-12-14 12:23:46.907 JupyterHub app:2989]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2986, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2521, in initialize
        self.init_db()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 1723, in init_db
        dbutil.upgrade_if_needed(self.db_url, log=self.log)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/dbutil.py", line 112, in upgrade_if_needed
        orm.check_db_revision(engine)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/orm.py", line 771, in check_db_revision
        current_table_names = set(inspect(engine).get_table_names())
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/inspection.py", line 64, in inspect
        ret = reg(subject)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py", line 182, in _engine_insp
        return Inspector._construct(Inspector._init_engine, bind)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py", line 117, in _construct
        init(self, bind)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py", line 128, in _init_engine
        engine.connect().close()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3165, in connect
        return self._connection_cls(self, close_with_result=close_with_result)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 96, in __init__
        else engine.raw_connection()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3244, in raw_connection
        return self._wrap_pool_connect(self.pool.connect, _connection)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3214, in _wrap_pool_connect
        Connection._handle_dbapi_exception_noconnection(
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2068, in _handle_dbapi_exception_noconnection
        util.raise_(
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
        raise exception
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3211, in _wrap_pool_connect
        return fn()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 307, in connect
        return _ConnectionFairy._checkout(self)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 767, in _checkout
        fairy = _ConnectionRecord.checkout(pool)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 425, in checkout
        rec = pool._do_get()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
        return self._create_connection()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
        return _ConnectionRecord(self)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 368, in __init__
        self.__connect()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 611, in __connect
        pool.logger.debug("Error on connect(): %s", e)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
        compat.raise_(
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
        raise exception
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 605, in __connect
        connection = pool._invoke_creator(self)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py", line 578, in connect
        return dialect.connect(*cargs, **cparams)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 584, in connect
        return self.dbapi.connect(*cargs, **cparams)
    sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
    (Background on this error at: http://sqlalche.me/e/14/e3q8)

```

**Edit :** Add PVC and PV description below

```auto
$ kubectl describe pv
Name: ovh-managed-kubernetes-8tk3qj-pvc-bacb4137-67a9-437f-a047-f561751920c9
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: cinder.csi.openstack.org
Finalizers: [kubernetes.io/pv-protection external-attacher/cinder-csi-openstack-org]
StorageClass: csi-cinder-high-speed
Status: Bound
Claim: skl-jhub/hub-db-dir
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 1Gi
Node Affinity: <none>
Message:         
Source:
    Type: CSI (a Container Storage Interface (CSI) volume source)
    Driver: cinder.csi.openstack.org
    FSType:            
    VolumeHandle: a31dd08e-80a8-4ca8-91ab-7be7295b274e
    ReadOnly: false
    VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1639470543810-8081-cinder.csi.openstack.org
Events: <none>

```

```auto
$ kubectl describe pvc
Name: hub-db-dir
Namespace: skl-jhub
StorageClass: csi-cinder-high-speed
Status: Bound
Volume: ovh-managed-kubernetes-8tk3qj-pvc-bacb4137-67a9-437f-a047-f561751920c9
Labels: app=jupyterhub
               app.kubernetes.io/managed-by=Helm
               chart=jupyterhub-1.2.0
               component=hub
               heritage=Helm
               release=skl-jhub
Annotations: meta.helm.sh/release-name: skl-jhub
               meta.helm.sh/release-namespace: skl-jhub
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: hub-55c56748d6-dfljt
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal ExternalProvisioning 26m persistentvolume-controller waiting for a volume to be created, either by external provisioner "cinder.csi.openstack.org" or manually created by system administrator
  Normal Provisioning 26m cinder.csi.openstack.org_csi-cinder-controllerplugin-0_f7428e10-cc91-4341-92c5-eaae61679ebc External provisioner is provisioning volume for claim "skl-jhub/hub-db-dir"
  Normal ProvisioningSucceeded 26m cinder.csi.openstack.org_csi-cinder-controllerplugin-0_f7428e10-cc91-4341-92c5-eaae61679ebc Successfully provisioned volume ovh-managed-kubernetes-8tk3qj-pvc-bacb4137-67a9-437f-a047-f561751920c9

```

Any ideas why this occur ?

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 14, 2021, 3:40pm UTC](https://discourse.jupyter.org/t/sql-operationalerror-with-jh-default-config/12207/2 "2021-12-14T15:40:36Z")

</div>

> [@brospars](#):
>
> ` sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file`

It sounds like a problem with your K8S storage. Perhaps it’s not writeable by the JupyterHub user, e.g. maybe it defaults to root-owned, or maybe the storage type is unsuitable (sqlite often has problems with NFS)?

If you quickly want to test Z2JH to rule out other problems you can disable all storage:

```yaml
hub:
  db:
    type: sqlite-memory

singleuser:
  storage:
    type: none

```

---

<div class="post-metadata">

**Author:** ![brospars](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/brospars/32/5991_2.png) [@brospars](https://discourse.jupyter.org/u/brospars)\
**Post date:** [December 14, 2021, 3:45pm UTC](https://discourse.jupyter.org/t/sql-operationalerror-with-jh-default-config/12207/3 "2021-12-14T15:45:53Z")

</div>

Yes you are right, it’s working when running the hub as root:

```auto
hub:
  fsGid: 0
  containerSecurityContext:
    runAsUser: 0
    runAsGroup: 0

```

But then I got this error for `singleuser`:

```auto
[D 2021-12-14 15:27:05.611 SingleUserNotebookApp application:174] Searching ['/home/jovyan', '/home/jovyan/.jupyter', '/home/jovyan/.local/etc/jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2021-12-14 15:27:05.611 SingleUserNotebookApp application:731] Looking for jupyter_config in /etc/jupyter
[D 2021-12-14 15:27:05.611 SingleUserNotebookApp application:731] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2021-12-14 15:27:05.611 SingleUserNotebookApp application:731] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2021-12-14 15:27:05.611 SingleUserNotebookApp application:731] Looking for jupyter_config in /home/jovyan/.local/etc/jupyter
[D 2021-12-14 15:27:05.612 SingleUserNotebookApp application:731] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2021-12-14 15:27:05.612 SingleUserNotebookApp application:731] Looking for jupyter_config in /home/jovyan
[D 2021-12-14 15:27:05.613 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /etc/jupyter
[D 2021-12-14 15:27:05.613 SingleUserNotebookApp application:753] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:753] Loaded config file: /etc/jupyter/jupyter_notebook_config.json
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:753] Loaded config file: /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /home/jovyan/.local/etc/jupyter
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2021-12-14 15:27:05.614 SingleUserNotebookApp application:731] Looking for jupyter_notebook_config in /home/jovyan
[W 2021-12-14 15:27:05.616 SingleUserNotebookApp configurable:193] Config option `open_browser` not recognized by `SingleUserNotebookApp`. Did you mean `browser`?
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py", line 537, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.9/site-packages/jupyter_core/application.py", line 264, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 845, in launch_instance
    app.initialize(argv)
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/mixins.py", line 840, in initialize
    result = super().initialize(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/mixins.py", line 573, in initialize
    return super().initialize(argv)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 88, in inner
    return method(app, *args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/notebook/notebookapp.py", line 2121, in initialize
    self.init_configurables()
  File "/opt/conda/lib/python3.9/site-packages/notebook/notebookapp.py", line 1650, in init_configurables
    connection_dir=self.runtime_dir,
  File "/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py", line 577, in __get__
    return self.get(obj, cls)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py", line 540, in get
    default = obj.trait_defaults(self.name)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/traitlets.py", line 1580, in trait_defaults
    return self._get_trait_default_generator(names[0])(self)
  File "/opt/conda/lib/python3.9/site-packages/jupyter_core/application.py", line 95, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/opt/conda/lib/python3.9/site-packages/jupyter_core/utils/ __init__.py", line 11, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/opt/conda/lib/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.local'

```

So it’s definitely a permissions issue. I’m using default OVH storage class `csi-cinder-high-speed` I’m new to k8s and I dont know how to setup the correct permissions…

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [December 15, 2021, 1:44pm UTC](https://discourse.jupyter.org/t/sql-operationalerror-with-jh-default-config/12207/4 "2021-12-15T13:44:45Z")

</div>

Many public cloud providers will setup K8S so that storage volumes respect the [`fsGroup`](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod), which is configured using `fsGid` in Z2JH (`hub.fsGid`, `singleuser.fsGid`).

If these configuration options don’t work for you you’ll need to look at how OVH configure their K8s storage volumes.

---

<div class="post-metadata">

**Author:** ![brospars](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/brospars/32/5991_2.png) [@brospars](https://discourse.jupyter.org/u/brospars)\
**Post date:** [December 15, 2021, 2:28pm UTC](https://discourse.jupyter.org/t/sql-operationalerror-with-jh-default-config/12207/5 "2021-12-15T14:28:56Z")

</div>

Yes OVH doesn’t seems to use the `fsGroup` parameter.  
But I’ve found a workaround using `initContainers` (inspired by [this comment](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/421#issuecomment-725429444)) :

```auto
hub:
  initContainers:
    - name: permissions-fixer # arbitrary name
      image: alpine # or whatever suits you
      securityContext:
        runAsUser: 0 # run init container as root to fix permission
      volumeMounts: # mount persistent volume
        - name: pvc
          mountPath: /srv/jupyterhub
      command: # fix permissions using chown and chmod
        - sh
        - -c
        - (chown -R 1000:1000 /srv/jupyterhub; chmod -R 0755 /srv/jupyterhub;)
  # the rest of your config ...

singleuser:
  initContainers:
    - name: permissions-fixer
      image: alpine
      securityContext:
        runAsUser: 0
      volumeMounts: # mount user persistent volume (based on volumeNameTemplate)
        - name: volume-{username}{servername}
          mountPath: /home/jovyan
      command:
        - sh
        - -c
        - (chown 1000:100 /home/jovyan; chmod 0755 /home/jovyan;)
  # the rest of your config ...

```

With this you’ve got the correct permissions setup and no rooted containers 👍