Trying to upgrade jupyterhub from helmchart 2.0.0 to 3.1.0

dr.sialenec · November 8, 2023, 9:09am

Hello, I’ve been trying to upgrade helmchart from 2.0.0 to the latest 3.1.0 (speaking as of date 8 Nov. 2023) on k8s (v1.25) AWS cluster. I got this error message when I check the logs of helmrelease

Last Helm logs:

Patch Deployment "hub" in namespace jupyterhub
Patch Deployment "proxy" in namespace jupyterhub
Patch Deployment "user-scheduler" in namespace jupyterhub
Patch Ingress "jupyterhub" in namespace jupyterhub
warning: Upgrade "jupyterhub" failed: cannot patch "jupyterhub-user-scheduler" with kind ClusterRole: clusterroles.rbac.authorization.k8s.io "jupyterhub-user-scheduler" is forbidden: user "system:serviceaccount:jupyterhub:flux-jupyterhub" (groups=["system:serviceaccounts" "system:serviceaccounts:jupyterhub" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["bindings"], Verbs:["create"]}
{APIGroups:[""], Resources:["endpoints"], ResourceNames:["user-scheduler-lock"], Verbs:["get" "update"]}
{APIGroups:[""], Resources:["endpoints"], Verbs:["create"]}
{APIGroups:[""], Resources:["events"], Verbs:["create" "patch" "update"]}
{APIGroups:[""], Resources:["namespaces"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["nodes"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["get" "list" "watch" "get" "list" "patch" "update" "watch"]}
{APIGroups:[""], Resources:["persistentvolumes"], Verbs:["get" "list" "watch" "get" "list" "patch" "update" "watch"]}
{APIGroups:[""], Resources:["pods"], Verbs:["delete" "get" "list" "watch"]}
{APIGroups:[""], Resources:["pods/binding"], Verbs:["create"]}
{APIGroups:[""], Resources:["pods/status"], Verbs:["patch" "update"]}
{APIGroups:[""], Resources:["replicationcontrollers"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["services"], Verbs:["get" "list" "watch"]}
{APIGroups:["apps"], Resources:["replicasets"], Verbs:["get" "list" "watch"]}
{APIGroups:["apps"], Resources:["statefulsets"], Verbs:["get" "list" "watch"]}
{APIGroups:["authentication.k8s.io"], Resources:["tokenreviews"], Verbs:["create"]}
{APIGroups:["authorization.k8s.io"], Resources:["subjectaccessreviews"], Verbs:["create"]}
{APIGroups:["coordination.k8s.io"], Resources:["leases"], ResourceNames:["user-scheduler-lock"], Verbs:["get" "update"]}
{APIGroups:["coordination.k8s.io"], Resources:["leases"], Verbs:["create"]}
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
{APIGroups:["extensions"], Resources:["replicasets"], Verbs:["get" "list" "watch"]}
{APIGroups:["policy"], Resources:["poddisruptionbudgets"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csidrivers"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csinodes"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csistoragecapacities"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["storageclasses"], Verbs:["get" "list" "watch"]} && cannot patch "jupyterhub-user-scheduler" with kind ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io "jupyterhub-user-scheduler" is forbidden: user "system:serviceaccount:jupyterhub:flux-jupyterhub" (groups=["system:serviceaccounts" "system:serviceaccounts:jupyterhub" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["bindings"], Verbs:["create"]}
{APIGroups:[""], Resources:["endpoints"], ResourceNames:["user-scheduler-lock"], Verbs:["get" "update"]}
{APIGroups:[""], Resources:["endpoints"], Verbs:["create"]}
{APIGroups:[""], Resources:["events"], Verbs:["create" "patch" "update"]}
{APIGroups:[""], Resources:["namespaces"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["nodes"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["get" "list" "watch" "get" "list" "patch" "update" "watch"]}
{APIGroups:[""], Resources:["persistentvolumes"], Verbs:["get" "list" "watch" "get" "list" "patch" "update" "watch"]}
{APIGroups:[""], Resources:["pods"], Verbs:["delete" "get" "list" "watch"]}
{APIGroups:[""], Resources:["pods/binding"], Verbs:["create"]}
{APIGroups:[""], Resources:["pods/status"], Verbs:["patch" "update"]}
{APIGroups:[""], Resources:["replicationcontrollers"], Verbs:["get" "list" "watch"]}
{APIGroups:[""], Resources:["services"], Verbs:["get" "list" "watch"]}
{APIGroups:["apps"], Resources:["replicasets"], Verbs:["get" "list" "watch"]}
{APIGroups:["apps"], Resources:["statefulsets"], Verbs:["get" "list" "watch"]}
{APIGroups:["authentication.k8s.io"], Resources:["tokenreviews"], Verbs:["create"]}
{APIGroups:["authorization.k8s.io"], Resources:["subjectaccessreviews"], Verbs:["create"]}
{APIGroups:["coordination.k8s.io"], Resources:["leases"], ResourceNames:["user-scheduler-lock"], Verbs:["get" "update"]}
{APIGroups:["coordination.k8s.io"], Resources:["leases"], Verbs:["create"]}
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
{APIGroups:["extensions"], Resources:["replicasets"], Verbs:["get" "list" "watch"]}
{APIGroups:["policy"], Resources:["poddisruptionbudgets"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csidrivers"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csinodes"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["csistoragecapacities"], Verbs:["get" "list" "watch"]}
{APIGroups:["storage.k8s.io"], Resources:["storageclasses"], Verbs:["get" "list" "watch"]}
    Reason:                        UpgradeFailed
    Status:                        False
    Type:                          Released
  Failures:                        6
  Helm Chart:                      jupyterhub/jupyterhub-jupyterhub
  Last Applied Revision:           2.0.0
  Last Attempted Revision:         3.1.0
  Last Attempted Values Checksum:  4b29d2e274a752ba61102614c7f324e2ef814c65
  Last Release Revision:           19
  Observed Generation:             26
  Upgrade Failures:                1

% k describe clusterrole jupyterhub-user-scheduler
Name:         jupyterhub-user-scheduler
Labels:       app=jupyterhub
              app.kubernetes.io/managed-by=Helm
              chart=jupyterhub-2.0.0
              component=user-scheduler
              helm.toolkit.fluxcd.io/name=jupyterhub
              helm.toolkit.fluxcd.io/namespace=jupyterhub
              heritage=Helm
              release=jupyterhub
Annotations:  meta.helm.sh/release-name: jupyterhub
              meta.helm.sh/release-namespace: jupyterhub
PolicyRule:
  Resources                                  Non-Resource URLs  Resource Names         Verbs
  ---------                                  -----------------  --------------         -----
  events                                     []                 []                     [create patch update]
  events.events.k8s.io                       []                 []                     [create patch update]
  bindings                                   []                 []                     [create]
  endpoints                                  []                 []                     [create]
  pods/binding                               []                 []                     [create]
  tokenreviews.authentication.k8s.io         []                 []                     [create]
  subjectaccessreviews.authorization.k8s.io  []                 []                     [create]
  leases.coordination.k8s.io                 []                 []                     [create]
  pods                                       []                 []                     [delete get list watch]
  persistentvolumeclaims                     []                 []                     [get list watch patch update]
  persistentvolumes                          []                 []                     [get list watch patch update]
  namespaces                                 []                 []                     [get list watch]
  nodes                                      []                 []                     [get list watch]
  replicationcontrollers                     []                 []                     [get list watch]
  services                                   []                 []                     [get list watch]
  replicasets.apps                           []                 []                     [get list watch]
  statefulsets.apps                          []                 []                     [get list watch]
  replicasets.extensions                     []                 []                     [get list watch]
  poddisruptionbudgets.policy                []                 []                     [get list watch]
  csidrivers.storage.k8s.io                  []                 []                     [get list watch]
  csinodes.storage.k8s.io                    []                 []                     [get list watch]
  csistoragecapacities.storage.k8s.io        []                 []                     [get list watch]
  storageclasses.storage.k8s.io              []                 []                     [get list watch]
  endpoints                                  []                 [user-scheduler-lock]  [get update]
  leases.coordination.k8s.io                 []                 [user-scheduler-lock]  [get update]
  pods/status                                []                 []                     [patch update]

Failing pod was only “hub” one. Pod messages:

Events:
  Type     Reason                  Age                 From                     Message
  ----     ------                  ----                ----                     -------
  Normal   Scheduled               57s                 default-scheduler        Successfully assigned jupyterhub/hub-6b48cc8878-vllr4 to ip-xxxxxxx.ec2.internal
  Normal   SuccessfulAttachVolume  47s                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-69b7208b-702c-45f2-a937-adcd565a71b2"
  Normal   Pulling                 46s                 kubelet                  Pulling image "jupyterhub/k8s-hub:3.0.0"
  Normal   Pulled                  38s                 kubelet                  Successfully pulled image "jupyterhub/k8s-hub:3.0.0" in 7.490308961s (7.490318101s including waiting)
  Normal   Created                 26s (x2 over 38s)   kubelet                  Created container hub
  Normal   Started                 26s (x2 over 38s)   kubelet                  Started container hub
  Normal   Pulled                  26s                 kubelet                  Container image "jupyterhub/k8s-hub:3.0.0" already present on machine
  Warning  Unhealthy               17s (x16 over 38s)  kubelet                  Readiness probe failed: Get "http://10.100.10.20:8081/hub/health": dial tcp 10.100.10.20:8081: connect: connection refused
  Warning  BackOff                 7s (x2 over 15s)    kubelet                  Back-off restarting failed container

I tried to update helmchart “only” to the major release version “3.0.0”, but the same behaviour. It looks some more priviledges are needed inside of the kubernetes, but I haven’t found anything mentioned in the release notes. Did I miss something?

Rolled back the helmchart to previous working version (2.0.0) and everything is running fine.

When I investigated the warning message that pointed to clusterrole, it looks a little bit messy for me, and I cannot find out what RBAC permissions it needs.

Thanks for any kind of help.

manics · November 9, 2023, 6:02pm

You can find all the Helm chart templates under

Typically Roles and ClusterRoles are in an rbac.yml file. For example, the roles for the user-scheduler referenced in your error message are in

github.com

jupyterhub/zero-to-jupyterhub-k8s/blob/8ccf2609ba5a42a9430f6940c04b2d8fab9bfc1d/jupyterhub/templates/scheduling/user-scheduler/rbac.yaml

{{- if .Values.scheduling.userScheduler.enabled -}}
{{- if .Values.rbac.create -}}
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: {{ include "jupyterhub.user-scheduler.fullname" . }}
  labels:
    {{- include "jupyterhub.labels" . | nindent 4 }}
rules:
  # Copied from the system:kube-scheduler ClusterRole of the k8s version
  # matching the kube-scheduler binary we use. A modification has been made to
  # resourceName fields to remain relevant for how we have named our resources
  # in this Helm chart.
  #
  # NOTE: These rules have been:
  #       - unchanged between 1.12 and 1.15
  #       - changed in 1.16
  #       - changed in 1.17
  #       - unchanged between 1.18 and 1.20
  #       - changed in 1.21: get/list/watch permission for namespace,

This file has been truncated. show original

dr.sialenec · November 13, 2023, 10:20am

thanks for providing me details about rbac, issue I observed was solved by adding
scheduling:
userScheduler:
enabled: false

but after changing the helm chart version to the latest again the hub pod failed with the logs message:

[I 2023-11-13 09:29:53.199 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2023-11-13 09:29:53.199 JupyterHub app:2889] Using Authenticator: oauthenticator.google.GoogleOAuthenticator-16.1.0
[I 2023-11-13 09:29:53.199 JupyterHub app:2889] Using Spawner: kubespawner.spawner.KubeSpawner-6.1.0
[I 2023-11-13 09:29:53.199 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
Found database schema version 651f5419b74d != 0eee8c825d24. Backup your database and run `jupyterhub upgrade-db` to upgrade to the latest schema.

I have found the thread describing this error - jupyterhub exited with code 1: Found database schema version 4dc2d5a8c53c != 896818069c98 · Issue #2711 · jupyterhub/jupyterhub · GitHub , but it is said also that This method does not work in jupyterhub 3.0.0 (in z2jh 2.0.0).

Our version of jupyterhub was never updated before, and since me as admin has never interact with its UI, have only limited overview of its functions.

Where does those update commands I need to perform? From my perspective, it is not that clear how to perform the update.

Thanks for any help here.

manics · November 13, 2023, 11:41am

If you’re using an external database see

If you’re using the default database (SQLite) upgrades should be automatic- please can you share your old and new Z2JH configurations?

dr.sialenec · November 14, 2023, 1:34pm

I was trying to find it explicitly visible somewhere if SQLite is used. Since we are not specifying it anywhere I assume SQLite is picked up as default chart value.

What I still wonder is…if the documentation is saying that
" If you are using the default database provider (SQLite), then the required db upgrades will be performed automatically when you do a helm upgrade . A backup of the old database is automatically created on the hub volume."

Why the failed pod is saying to backup the database and how to perform jupyterhub upgrade-db while the “hub” pod is not running.

My HelmRelease conf:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: jupyterhub
spec:
  serviceAccountName: <serviceAccountName>
  releaseName: jupyterhub
  chart:
    spec:
      chart: jupyterhub
      sourceRef:
        kind: HelmRepository
        name: jupyterhub
      version: "2.0.0"
  interval: 3m
  install:
    remediation:
      retries: 10
  values:
    #https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/d140f561b2e2808c759401f07ff8cca81a1891c3/jupyterhub/values.yaml
    hub:
      command: ["/bin/sh", "-c"]
      args:
        [
          "yes | pip install oauthenticator[<group>]; jupyterhub --config /usr/local/etc/jupyterhub/jupyterhub_config.py",
        ]
      config:
        GoogleOAuthenticator:
          client_id: <input>
          client_secret: <input>
          oauth_callback_url: https://<host>/hub/oauth_callback
          hosted_domain:
            - <domain>
          login_service: <input>
          gsuite_administrator: { "<input>": "<input>" }
          google_service_account_keys:
            { "<input>": "<input>.json" }
          admin_google_groups: { "<input>": ["jupyterhub-<group>"] }
          allowed_google_groups: { "<input>": ["jupyterhub-<group>"] }
        JupyterHub:
          authenticator_class: google
          admin_access: true
      extraVolumes:
        - name: <jupyterhub_extraVolume>
          # configMap:
          #   name: <jupyterhub_extraVolume>
          secret:
            secretName: <jupyterhub_secretName>
      extraVolumeMounts:
        - mountPath: <mountPath>.json
          subPath: <subPath>.json
          name: <jupyterhub_extraVolumeMount>
      serviceAccount:
        annotations:
          "<eks.amazonaws.com/role-arn>"
    proxy:
      service:
        type: ClusterIP
    ingress:
      enabled: true
      ingressClassName: nginx-internal
      hosts:
        - "<jupyterhub_host>"
      pathType: Prefix
      tls:
        - hosts:
            - <host>
          secretName: <secret_name>
    singleuser:
      serviceAccountName: <service_account>
      memory:
        limit:
        guarantee: 4G
      # extraResource:
      #   limits: {}
      #   guarantees: {}
    scheduling:
      userPlaceholder:
        enabled: false
      userScheduler:
        enabled: false
    prePuller:
      continuous:
        enabled: false

The only difference between old (current) and new is chart version, while doing upgrade. Appreciate your time on this.

manics · November 14, 2023, 1:49pm

You’ve overridden the default args, so the datbase upgrade command isn’t run:

github.com

jupyterhub/zero-to-jupyterhub-k8s/blob/8ccf2609ba5a42a9430f6940c04b2d8fab9bfc1d/jupyterhub/templates/hub/deployment.yaml#L108-L142


      
          {{- /* .Values.hub.args overrides everything the Helm chart otherside would set */}}
          {{- if .Values.hub.args }}
          {{- range .Values.hub.args }}
          - {{ tpl . $ }}
          {{- end }}
          
          {{- /* .Values.hub.args didn't replace the default logic */}}
          {{- else }}
          - jupyterhub
          - --config
          - /usr/local/etc/jupyterhub/jupyterhub_config.py
          {{- if .Values.debug.enabled }}
          - --debug
          {{- end }}
          {{- /* NOTE:
            We want to do automatic upgrades for sqlite-pvc by default, but
            allow users to opt out of that if they want. Users using their own
            db need to 'opt in' Go Templates treat nil and "" and false as
            'false', making this code complex. We can probably make this a
            one-liner, but doing combinations of boolean vars in go templates

This file has been truncated. show original

dr.sialenec · November 14, 2023, 2:17pm

Oh, thanks for finding this. So do I understand it correctly to keep the current setup and be able to run the upgrade, I need to add this in the args section

    db:
      upgrade: true

and revert it afterwards?

Or is there any better way?

dr.sialenec · November 16, 2023, 11:54am

by playing with my helmrelease I removed reference to problematic section with “args” and “cmd” that prevent to upgrade DB during chart upgrade and finally got the new version running. Hub pod runs fine, from UI perspective same fine, but …

reviewing the hub pod logs:

[I 2023-11-16 07:10:04.977 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2023-11-16 07:10:04.977 JupyterHub app:2889] Using Authenticator: oauthenticator.google.GoogleOAuthenticator-16.1.0
[I 2023-11-16 07:10:04.977 JupyterHub app:2889] Using Spawner: kubespawner.spawner.KubeSpawner-6.1.0
[I 2023-11-16 07:10:04.977 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[I 2023-11-16 07:10:05.008 JupyterHub dbutil:130] Upgrading sqlite:///jupyterhub.sqlite
[I 2023-11-16 07:10:05.008 JupyterHub dbutil:99] Backing up jupyterhub.sqlite => jupyterhub.sqlite.2023-11-16-071005
[I 2023-11-16 07:10:05.687 alembic.runtime.migration migration:213] Context impl SQLiteImpl.
[I 2023-11-16 07:10:05.687 alembic.runtime.migration migration:216] Will assume non-transactional DDL.
[I 2023-11-16 07:10:05.693 alembic.runtime.migration migration:619] Running upgrade 651f5419b74d -> 0eee8c825d24, added properties column
[I 2023-11-16 07:10:05.921 JupyterHub roles:183] Role attribute server.scopes has been changed
[I 2023-11-16 07:10:05.932 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[I 2023-11-16 07:10:06.014 JupyterHub reflector:282] watching for pods with label selector='component=singleuser-server' in namespace jupyterhub
[W 2023-11-16 07:10:06.025 JupyterHub _version:68] jupyterhub version 4.0.2 != jupyterhub-singleuser version 3.0.0. This could cause failure to authenticate and result in redirect loops!
[I 2023-11-16 07:10:06.026 JupyterHub app:2928] Initialized 1 spawners in 0.037 seconds
[I 2023-11-16 07:10:06.033 JupyterHub metrics:278] Found 4 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2023-11-16 07:10:06.035 JupyterHub metrics:278] Found 4 active users in the last ActiveUserPeriods.seven_days
[I 2023-11-16 07:10:06.035 JupyterHub metrics:278] Found 4 active users in the last ActiveUserPeriods.thirty_days
[I 2023-11-16 07:10:06.036 JupyterHub app:3142] Not starting proxy
[I 2023-11-16 07:10:06.042 JupyterHub app:3178] Hub API listening on http://:8081/hub/
[I 2023-11-16 07:10:06.042 JupyterHub app:3180] Private Hub API connect url http://hub:8081/hub/
[I 2023-11-16 07:10:06.042 JupyterHub app:3189] Starting managed service jupyterhub-idle-culler
[I 2023-11-16 07:10:06.042 JupyterHub service:385] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2023-11-16 07:10:06.043 JupyterHub service:133] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[I 2023-11-16 07:10:06.047 JupyterHub app:3247] JupyterHub is now running, internal Hub API at http://hub:8081/hub/

There are no issues with authentication but worrying about the message saying:

jupyterhub version 4.0.2 != jupyterhub-singleuser version 3.0.0

Tried to find out some older solutions:

github.com/jupyterhub/zero-to-jupyterhub-k8s

Single-user server warning (question)

opened 01:04PM - 06 Sep 19 UTC

closed 08:34AM - 11 Nov 19 UTC

ablekh

I have just noticed the following warning in a single-user server log output: … ``` [W 2019-09-06 12:58:45.288 SingleUserNotebookApp _version:50] jupyterhub version 1.0.0 != jupyterhub-singleuser version 0.9.2. This could cause failure to authenticate and result in redirect loops! ``` I'm wondering about root cause for this issue, potential adverse effects and corresponding fixes. In this particular instance, the server has launched fine, but ...

but haven’t found yet the correct way to solve this.

Strange is that the pod logs contains the newest version of all packages and helm chart as well, but when logging as admin into jupyterhub UI and running some commands it gives totally different results:

logs right away after upgrade:

logs after about 3 hours later:

Again the same issue with database schema version? I was about to investigate the previous message between jupyterhub and jupyterhub-singleuser version, but in the time of writing this update and checking once again, it changed actually.

What happened with the jupyterhub.proxy.ConfigurableHTTPProxy after I hit jupyterhub upgrade-db ?

Anything else to be taken care of? Thank you for your help.

manics · November 16, 2023, 1:50pm

That means the version of jupyterhub installed in your singleuser image is out of date. In practice it often works because a the protocol between jupyterhub and jupyterhub-singleuser doesn’t always change in a major release, it’s more often a database change that leads to the major release bump.

Are you sure you’re not logging in to an old pod?

You’re using the helm chart, so configurable-http-proxy is run in a separate container. When you ran jupyterhub manually you didn’t pass the config file, so it would’ve used the default config.

dr.sialenec · November 20, 2023, 10:26am

@manics Any proposed solution to this, not to “break” it more than it is now? Thanks for help.

dr.sialenec · November 20, 2023, 10:43am

Apart from this, after leting it run during the weekend and checking pod logs again, it gives and error and warning message for all the users in my values.hub.config.Authenticator.admin_users and allowed_users as well:

$ jupyterhub --show-config
[I 2023-11-20 10:08:12.738 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2023-11-20 10:08:12.738 JupyterHub app:2889] Using Authenticator: jupyterhub.auth.PAMAuthenticator-4.0.2
[I 2023-11-20 10:08:12.738 JupyterHub app:2889] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-4.0.2
[I 2023-11-20 10:08:12.738 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[I 2023-11-20 10:08:12.752 JupyterHub app:1664] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[I 2023-11-20 10:08:12.845 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[E 2023-11-20 10:08:12.849 JupyterHub app:2010] Error adding user <username> already in db
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/app.py", line 2008, in init_users
        await maybe_future(f)
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/auth.py", line 903, in add_user
        raise KeyError(
    KeyError: 'User <username> does not exist on the system. Set LocalAuthenticator.create_system_users=True to automatically create system users from jupyterhub users.'

[W 2023-11-20 10:08:12.851 JupyterHub app:2017]
    You can set
        c.Authenticator.delete_invalid_users = True
    to automatically delete users from the Hub database that no longer pass
    Authenticator validation,
    such as when user accounts are deleted from the external system
    without notifying JupyterHub.

Our helm chart release doesn’t have such options configured, as it was not complaining about those before. What is it like this now?

manics · November 23, 2023, 12:03am

Based on your previous posts it sounds like a combination of the customisations you’ve made to your deployment, and the manual changes you’ve made by shelling into a container and running jupyterhub, may have caused some problems.

Can you try removing all your customisations, and deleting all pods to force them to be cleanly recreated?

dr.sialenec · November 23, 2023, 10:46am

Thank you @manics for your time and effort. I did try to remove helmchart , delete users pods that left. So, uninstalled and let it install again, went without issues…running on latest version…no more errors/warning about schema versions or mismatch jupyterhub vs jupyterhub-singleuser version.

our config at the moment:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: jupyterhub
spec:
  serviceAccountName: flux-jupyterhub
  releaseName: jupyterhub
  chart:
    spec:
      chart: jupyterhub
      sourceRef:
        kind: HelmRepository
        name: jupyterhub
      version: "3.1.0"
  interval: 3m
  install:
    remediation:
      retries: 10
  values:
    hub:
      config:
        Authenticator:
          admin_users:
            - user_name
            - user_name
            - user_name
          allowed_users:
            - user_name
            - user_name
            - user_name
        GoogleOAuthenticator:
          client_id: client_id
          client_secret: client_secret
          oauth_callback_url: https://oauth_callback_url/hub/oauth_callback
          hosted_domain:
            - hosted_domain_name
          login_service: Google account
      serviceAccount:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::role-arn-name
    proxy:
      service:
        type: ClusterIP
    ingress:
      enabled: true
      ingressClassName: ingressClassName
      hosts:
        - "hosts_value"
      pathType: Prefix
      tls:
        - hosts:
            - host
          secretName: secretName
    singleuser:
      serviceAccountName: singleuser
      memory:
        limit:
        guarantee: 4G
    scheduling:
      userPlaceholder:
        enabled: false
      userScheduler:
        enabled: false
    prePuller:
      continuous:
        enabled: false

Looking at GUI, typing jupyterhub in its terminal, visible same error as before related to ConfigurableHTTPProxy

looking at inside of the pod terminal, error related to ConfigurableHTTPProxy and adding user error on top of that:

$ jupyterhub
[I 2023-11-23 09:33:51.445 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2023-11-23 09:33:51.446 JupyterHub app:2889] Using Authenticator: jupyterhub.auth.PAMAuthenticator-4.0.2
[I 2023-11-23 09:33:51.446 JupyterHub app:2889] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-4.0.2
[I 2023-11-23 09:33:51.446 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[I 2023-11-23 09:33:51.459 JupyterHub app:1709] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
[W 2023-11-23 09:33:51.569 JupyterHub app:2154] Deleting role jupyterhub-idle-culler
[I 2023-11-23 09:33:51.616 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[E 2023-11-23 09:33:51.626 JupyterHub app:2010] Error adding user <username> already in db
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/app.py", line 2008, in init_users
        await maybe_future(f)
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/auth.py", line 903, in add_user
        raise KeyError(
    KeyError: 'User <username> does not exist on the system. Set LocalAuthenticator.create_system_users=True to automatically create system users from jupyterhub users.'

[W 2023-11-23 09:33:51.628 JupyterHub app:2017]
    You can set
        c.Authenticator.delete_invalid_users = True
    to automatically delete users from the Hub database that no longer pass
    Authenticator validation,
    such as when user accounts are deleted from the external system
    without notifying JupyterHub.

    [W 2023-11-23 09:33:51.773 JupyterHub app:2581] <username> appears to have stopped while the Hub was down
[I 2023-11-23 09:33:51.786 JupyterHub app:2928] Initialized 1 spawners in 0.033 seconds
[I 2023-11-23 09:33:51.795 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2023-11-23 09:33:51.796 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.seven_days
[I 2023-11-23 09:33:51.797 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.thirty_days
[W 2023-11-23 09:33:51.798 JupyterHub proxy:746] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2023-11-23 09:33:51.798 JupyterHub proxy:750] Starting proxy @ http://:8000
[E 2023-11-23 09:33:51.799 JupyterHub proxy:758] Failed to find proxy ['configurable-http-proxy']
    The proxy can be installed with `npm install -g configurable-http-proxy`.To install `npm`, install nodejs which includes `npm`.If you see an `EACCES` error or permissions error, refer to the `npm` documentation on How To Prevent Permissions Errors.
[C 2023-11-23 09:33:51.799 JupyterHub app:3139] Failed to start proxy
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/app.py", line 3137, in start
        await self.proxy.start()
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/proxy.py", line 754, in start
        self.proxy_process = Popen(
                             ^^^^^^
      File "/usr/local/lib/python3.11/subprocess.py", line 1026, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/usr/local/lib/python3.11/subprocess.py", line 1950, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'configurable-http-proxy'

Error message is for every user defined in the helmrelease. I tried to add as advised by message , but same behavior.

config:
  LocalAuthenticator:
    create_system_users: true

I tried to remove every user manually in GUI and reinstall again, but same behaviour. Conclusion is that everything looks working, just these errors are bothering me to close this topic “upgrade” as successfully solved.

After reinstallation I also suspected that majority of PVC are gone, but some still there, not sure what was the key for deleting them.

% kubectl get pvc
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
claim-admin               Bound    pvc-68c792cc-c642-4843-9a4d-5b859c943fc3   10Gi       RWO            ebs-gp3-delete   415d
claim-user1               Bound    pvc-cba51d26-978d-45c9-8dbd-43fc35e859c9   10Gi       RWO            ebs-gp3-delete   7d23h
claim-user2               Bound    pvc-b4aed399-c3fa-481a-8ab2-5e859fc3c943   10Gi       RWO            ebs-gp3-delete   45m
claim-user3               Bound    pvc-dd816fa5-02bc-4002-87d0-5h43fc3v4e8a   10Gi       RWO            ebs-gp3-delete   50d
hub-db-dir                Bound    pvc-1a9940cd-55dd-408c-96c8-11fc35e84566   1Gi        RWO            ebs-gp3-delete   64m

manics · November 23, 2023, 11:41pm

That’s expected. The container image is configured to run jupyterhub with the correct configuration for Z2JH (e.g. configuration file, working directory). Since you’re running it manually in a terminal most of this config is missing, which is why you’re seeing these errors.

Topic		Replies	Views
Migration from Helm 2 to Helm 3 Zero to JupyterHub on Kubernetes	2	599	March 22, 2022
Any chance to adjust jupyterhub via helm chart values so that it does not try to create any clusterroles? JupyterHub	5	670	October 7, 2021
Installing JupyterHub with Helm 3 - Kubernetes Cluster Roles Zero to JupyterHub on Kubernetes how-to	4	2564	May 26, 2020
New versions for Helm, 2.0.0 issues Zero to JupyterHub on Kubernetes	1	419	May 31, 2023
JupyterHub on K8s: 403 Forbidden for Users Added via Admin Page but not the Helm Chart Config Zero to JupyterHub on Kubernetes jupyterhub , help-wanted	4	325	June 24, 2024

Trying to upgrade jupyterhub from helmchart 2.0.0 to 3.1.0

Related topics