CrashLoopBackOff due to PermissionError: [Errno 13] Permission denied: '/srv/jupyterhub/jupyterhub_cookie_secret'

Hello all,
I tried to deploy jupyterhub by following this tutorial https://zero-to-jupyterhub.readthedocs.io/_/downloads/en/0.9.0/pdf/.
I created a kubernetes cluster on IBM cloud.
Kubernetes version: 1.18.15_1538

This is my config.yaml

proxy:
  secretToken: "XXXX"
hub:
  uid: 0
  fsGid: 0

I used these commands to deploy jupyterhub to the kubernetes

RELEASE=jhub
NAMESPACE=jhub
helm upgrade --install $RELEASE jupyterhub/jupyterhub --namespace $NAMESPACE --version=0.10.2 --values config.yaml

Now I have CrashLoopBackOff on one of the pods.
When I debug the logs, it says PermissionError: [Errno 13] Permission denied: '/srv/jupyterhub/jupyterhub_cookie_secret'

I checked the instructions on this link before Permission denied writing into /srv/jupyterhub/jupyterhub_cookie_secret · Issue #1049 · jupyterhub/zero-to-jupyterhub-k8s · GitHub

But it did not solve my problem.

Thanks in advance

Instead of

proxy:
  secretToken: "XXXX"
hub:
  uid: 0
  fsGid: 0

Can you try:

proxy:
  secretToken: "XXXX"

And instead of

RELEASE=jhub
NAMESPACE=jhub
helm upgrade --install $RELEASE jupyterhub/jupyterhub --namespace $NAMESPACE --version=0.10.2 --values config.yaml

Can you try:

RELEASE=jhub
NAMESPACE=jhub
helm upgrade --install $RELEASE jupyterhub/jupyterhub --namespace $NAMESPACE --version=0.11.1 --values config.yaml

If you had a clear intention for starting jupyterhub as uid 0 with fsGid 0, please include why you want to do that, because fsGid 0 is probably what caused this issue.

If you want, you can even use a new feature not yet fully released, then you don’t have to specify proxy.secretToken at all, to do that, use the version 0.11.1-n121.h56e5ee17 and let your config be …

# no custom config yet

In the beginning I only had proxy and secretToken fields in config.yaml. I added the other lines after reading the recommendation here Permission denied writing into /srv/jupyterhub/jupyterhub_cookie_secret · Issue #1049 · jupyterhub/zero-to-jupyterhub-k8s · GitHub.
Thanks

1 Like

Let me know what you experience from trying what I suggested!

1 Like

Hi,
I applied all your recommendations.
I still have the same problem
PermissionError: [Errno 13] Permission denied: ‘/srv/jupyterhub/jupyterhub_cookie_secret’

The whole logs

No config at /etc/jupyterhub/config/values.yaml
Loading /etc/jupyterhub/secret/values.yaml
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py”, line 1467, in init_secrets
with open(secret_file, ‘w’) as f:
PermissionError: [Errno 13] Permission denied: ‘/srv/jupyterhub/jupyterhub_cookie_secret’
Thanks
[I 2021-02-05 12:55:24.452 JupyterHub app:2349] Running JupyterHub version 1.3.0
[I 2021-02-05 12:55:24.453 JupyterHub app:2379] Using Authenticator: jupyterhub.auth.DummyAuthenticator-1.3.0
[I 2021-02-05 12:55:24.453 JupyterHub app:2379] Using Spawner: kubespawner.spawner.KubeSpawner-0.15.0
[I 2021-02-05 12:55:24.453 JupyterHub app:2379] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.3.0
[I 2021-02-05 12:55:24.455 JupyterHub app:1465] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
[E 2021-02-05 12:55:24.456 JupyterHub app:2859]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2856, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2389, in initialize
        self.init_secrets()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 1467, in init_secrets
        with open(secret_file, 'w') as f:
    PermissionError: [Errno 13] Permission denied: '/srv/jupyterhub/jupyterhub_cookie_secret'

Thanks

Hi,
Sorry it did not work.
I replied to your previous comment.
Thanks

Also when I tried to upload the latest version jupyterhub-0.11.1-n121.h56e5ee17.tgz I have an error it says
Error: apiVersion ‘v2’ is not valid. The value must be “v1”

I overcomed the error. It is related to helm version.
Now I have a new error with the latest version you recommended.

Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2856, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2391, in initialize
        self.init_db()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 1589, in init_db
        dbutil.upgrade_if_needed(self.db_url, log=self.log)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/dbutil.py", line 115, in upgrade_if_needed
        orm.check_db_revision(engine)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/orm.py", line 771, in check_db_revision
        current_table_names = set(engine.table_names())
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2316, in table_names
        with self._optional_conn_ctx_manager(connection) as conn:
      File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
        return next(self.gen)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2086, in _optional_conn_ctx_manager
        with self._contextual_connect() as conn:
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
        self._wrap_pool_connect(self.pool.connect, None),
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
        Connection._handle_dbapi_exception_noconnection(
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
        util.raise_(
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
        raise exception
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
        return fn()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
        return _ConnectionFairy._checkout(self)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 778, in _checkout
        fairy = _ConnectionRecord.checkout(pool)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 495, in checkout
        rec = pool._do_get()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py", line 241, in _do_get
        return self._create_connection()
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
        return _ConnectionRecord(self)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 440, in __init__
        self.__connect(first_connect_check=True)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 661, in __connect
        pool.logger.debug("Error on connect(): %s", e)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
        compat.raise_(
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
        raise exception
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 656, in __connect
        connection = pool._invoke_creator(self)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect
        return dialect.connect(*cargs, **cparams)
      File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 493, in connect
        return self.dbapi.connect(*cargs, **cparams)
    sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
    (Background on this error at: http://sqlalche.me/e/13/e3q8)

Thanks

Please could you show us your current full config.yaml with secrets redacted?

Can you list your persistent volumes and volume claims (kubectl get pv,pvc in the correct namespace). It’s possible your hub’s persistent volume now has incorrect permissions since you initially tried to use fsGid: 0 and uid: 0. Since it hasn’t been run properly the easiest thing to do would be to delete it so it can be recreated with the correct permissions.

Hi,

$helm upgrade --install $RELEASE jupyterhub-0.11.1-n121.h56e5ee17.tgz --namespace $NAMESPACE --version=0.11.1 --values config.yaml

$cat config.yaml → # no custom config yet

$kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-9ac380d4-9da1-4c99-a017-87edfebac069 20Gi RWO Delete Bound jhub/hub-db-dir ibmc-file-gold 50m

Thanks

Since this is still at the initial test/deployment stage I think the easiest solution is to delete your deployment helm delete ..., ensure your PVC is deleted (if it’s not then delete it manually kubectl delete ...), and deploy again from scratch. If it works then you’ll have a known good baseline that you can start modifying.

The alternative is to obtain a shell inside the hub pod and examine the permissions and ownership of the volume, and then fix them if necessary by running a separate root pod. The first option is definitely easier :smiley:

1 Like

Thank you. I will look into it.

Ah, you would need Helm 3 to use that version.

Yes, later I updated helm. Thanks.

1 Like

Hi,
I tried all.
It is just not working.
I have this error in the logs
File “/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py”, line 493, in connect
return self.dbapi.connect(*cargs, **cparams)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: http://sqlalche.me/e/13/e3q8)

Thanks

This code  in config.yaml which I saw in one of the discussions on discourse.jupyter.org worked for me

## This portion is missing from the tutorial for anyone trying to setup on bare metal.
hub:
  db:
    type: sqlite-memory

singleuser:
  storage:
    type: sqlite-memory
##

That will disable persistent storage for the hub database which means if the hub is restarted you’ll lose all state.

For the singleuser type: sqlite-memory isn’t valid, you can use singleuser.storage.type: none to disable persistent storage:
https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-storage.html#turn-off-per-user-persistent-storage

Persistent storage with Z2JH should work on any correctly configured cluster including baremetal. It sounds like there might be a problem with how your K8S storage is setup. This could be a misconfiguration, a bug, or it could be that your K8S storage has limited features.

Damm even I am having a similar kind of issue, I have searched all over the internet and even have posted on number of threads on different forum, no solution seems to work. I am really frustrated, can anyone of you here help me resolve this issue, I am very much tired now.

Did you try the config which I posted above.
It works for me.