User on hub database are not being persisted after hub pod restart

JupyterHub
,
1

I’m usins a postgres database for Hub component with NativeAuthenticator. The problem is, that for some reasone, if the hub pod needs a restart (after helm upgrade for instance) it seems that the users that is already persisted on database, are not beign recongnize and not being listed on Jupyter Hub UI.

When hub pod restarts, it seems that is recreating all users.

Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[I 2024-08-09 21:25:13.379 JupyterHub app:2885] Running JupyterHub version 4.1.5
[I 2024-08-09 21:25:13.380 JupyterHub app:2915] Using Authenticator: nativeauthenticator.nativeauthenticator.NativeAuthenticator
[I 2024-08-09 21:25:13.380 JupyterHub app:2915] Using Spawner: kubespawner.spawner.KubeSpawner-6.2.0
[I 2024-08-09 21:25:13.380 JupyterHub app:2915] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.1.5
[I 2024-08-09 21:25:13.401 alembic.runtime.migration migration:216] Context impl SQLiteImpl.
[I 2024-08-09 21:25:13.401 alembic.runtime.migration migration:219] Will assume non-transactional DDL.
[I 2024-08-09 21:25:13.412 alembic.runtime.migration migration:622] Running stamp_revision  -> 0eee8c825d24
[I 2024-08-09 21:25:13.579 JupyterHub roles:173] Role jupyterhub-idle-culler added to database
[W 2024-08-09 21:25:13.583 JupyterHub auth:38] 
    The shared database session at Authenticator.db is deprecated, and will be removed.
    Please manage your own database and connections.
    
    Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
    if you have questions or ideas about direct database needs for your Authenticator.
    
[I 2024-08-09 21:25:13.591 JupyterHub roles:239] Adding role admin for User: pmesserschmidt
[I 2024-08-09 21:25:13.598 JupyterHub roles:239] Adding role user for User: pmesserschmidt
[I 2024-08-09 21:25:13.605 JupyterHub roles:239] Adding role admin for User: bpitol
[I 2024-08-09 21:25:13.611 JupyterHub roles:239] Adding role user for User: bpitol
[I 2024-08-09 21:25:13.619 JupyterHub roles:239] Adding role admin for User: rferraz
[I 2024-08-09 21:25:13.625 JupyterHub roles:239] Adding role user for User: rferraz

The users that appears in the log are configure as admin users. The other regular users (not admin) that was created before hub pod restart disapears.

How can I solve this?

This is how the helm chart is configured

hub:
  revisionHistoryLimit:
  config:
    JupyterHub:
      admin_access: true
      authenticator_class: nativeauthenticator.NativeAuthenticator
      db_url: ${DB_CONN_DEV}
    NativeAuthenticator:
      open_signup: false 
      enable_auth_state: true
    Authenticator:
      admin_users:
        - pmesserschmidt 
        - bribeiro 
        - bpitol 
        - tgomes 
        - rferraz

image
image1127×373 34.6 KB

2

I’ve noticed that if the pode hub is restarted after a helm upgrade command, but there’s a user pod running, that user that is not “lost”, but the other users that don’t have a session, are “lost” in the UI ( it appears as “user or password invalid” when trying to loggin, but the user exists in the database")

3

What kind of database is DB_CONN_DEV? If it’s sqlite, where is it mounted, and what persistent volume does it come from?

I believe you need to configure the database connection via the hub.db config, not the passthrough hub.config.....

My guess is that your database is not persisted because it’s pointing to a part of the ephemeral container filesystem, and not a PVC-mounted path that will get mounted again on the next start. As a result, the db is created anew on each launch.

4

Good morning @minrk . I just saw that I forgot to add this section to my topic

  db: 
    type: postgres
    upgrade: false
    pvc:
      annotations: {}
      selector: {}
      accessModes:
        - ReadWriteMany
      storage: 1Gi
      subPath: "hub-dir"
      storageClassName:
    # url:
    # password:

I’m using a external postgre SQL database. I thought I need to add the DB_CONN_DEV in section JupyterHub due to set the database used for the NativeAuthenticator.

5

I found this in docs:

postgres

Use an externaly hosted postgres database.

You have to specify an sqlalchemy connection string for the postgres database you want to connect to in hub.db.url if using this option.

The general format of the connection string is:

postgres+psycopg2://<db-username>:<db-password>@<db-hostname>:<db-port>/<db-name>
The user specified in the connection string must have the rights to create tables in the database specified.

Note that if you use this, you must also set hub.cookieSecret.

I’m not setting a cookieSecret… can it be the reason for the “users” are not beign reconginzed by the jupyterHub UI? I mean, every time the hub restarts, it creates a different cookie so I can’t “decode” the users saved in my database because it was created with a different cookie.

cookieSecret:

6

Just to let you know @minrk , it’s working know. It was a typo in my database connection URL that was forcing the jupyter hub use a different database. Thanks anyway.