Minikube guide leads to a hub pod crashloop trying to open database file

Hi, I am trying to follow the tutorial for setting up jupyterhub using minikube. However, after installing the helm chart, the hub pod gets a CrashLoopBackOff status.

After running describe, this is what I get:
Readiness probe failed: Get “http://10.244.205.200:8081/hub/health”: dial tcp 10.244.205.200:8081: connect: connection refused

And when I run kubectl logs, I got this;


[I 2024-05-25 10:24:04.883 JupyterHub app:2885] Running JupyterHub version 4.1.5
[I 2024-05-25 10:24:04.883 JupyterHub app:2915] Using Authenticator: jupyterhub.auth.DummyAuthenticator-4.1.5
[I 2024-05-25 10:24:04.883 JupyterHub app:2915] Using Spawner: kubespawner.spawner.KubeSpawner-6.2.0
[I 2024-05-25 10:24:04.883 JupyterHub app:2915] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.1.5
[E 2024-05-25 10:24:04.905 JupyterHub app:3408]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/app.py", line 3405, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/app.py", line 2927, in initialize
        self.init_db()
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/app.py", line 1854, in init_db
        dbutil.upgrade_if_needed(self.db_url, log=self.log)
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/dbutil.py", line 113, in upgrade_if_needed
        orm.check_db_revision(engine)
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/orm.py", line 981, in check_db_revision
        current_table_names = set(inspect(engine).get_table_names())
                                  ^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/inspection.py", line 140, in inspect
        ret = reg(subject)
              ^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 303, in _engine_insp
        return Inspector._construct(Inspector._init_engine, bind)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 236, in _construct
        init(self, bind)
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 247, in _init_engine
        engine.connect().close()
        ^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3280, in connect
        return self._connection_cls(self)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 148, in __init__
        Connection._handle_dbapi_exception_noconnection(
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2444, in _handle_dbapi_exception_noconnection
        raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
        self._dbapi_connection = engine.raw_connection()
                                 ^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3304, in raw_connection
        return self.pool.connect()
               ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 449, in connect
        return _ConnectionFairy._checkout(self)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
        fairy = _ConnectionRecord.checkout(pool)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
        rec = pool._do_get()
              ^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
        with util.safe_reraise():
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
        raise exc_value.with_traceback(exc_tb)
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
        return self._create_connection()
               ^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
        return _ConnectionRecord(self)
               ^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
        self.__connect()
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
        with util.safe_reraise():
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
        raise exc_value.with_traceback(exc_tb)
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
        self.dbapi_connection = connection = pool._invoke_creator(self)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 643, in connect
        return dialect.connect(*cargs, **cparams)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 617, in connect
        return self.loaded_dbapi.connect(*cargs, **cparams)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
    (Background on this error at: https://sqlalche.me/e/20/e3q8)

I followed the instructions, and I left the config.yaml file as it’s shown in the tutorial (just some lines with comments). I am quite new to helm, so I don’t know if I’m missing something crucial

The minikube cluster is hosted on an AWS EC2 instance for which I increased the memory, so it could have enough.

Helm Version: version.BuildInfo{Version:“v3.15.1”, GitCommit:“e211f2aa62992bd72586b395de50979e31231829”, GitTreeState:“clean”, GoVersion:“go1.22.3”}

kubectl version:
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

minikube version
minikube version: v1.33.1
commit: 5883c09216182566a63dff4c326a6fc9ed2982ff

Any help would be much appreciated!

This indicates there’s a problem with the storage volume that Minikube created for the JupyterHub database. e.g. Maybe the premissions are incorrect so JupyterHub can’t write to it, or perhaps it’s not a POSIX filesystem which can cause problems for SQLite? Try searching the Minikube documentation on persistent volumes.

Alternatively you can disable persistent storage completely if this is a test system:

hub:
  db:
    type: sqlite-memory

singleuser:
  storage:
    type: none
2 Likes

Thanks for your response!

I just wanted a test environment to try a few things before delving deeper into implementing JupyterHub with more expensive resources. After disabling the persistent storage, things seem to be working fine

I will be able to test things out finally!

Thanks again :slight_smile: