ValueError: services jupyterhub-idle-culler defined in config role definition jupyterhub-idle-culler but not present in database

JupyterHub Zero to JupyterHub on Kubernetes
,
1

I recently tried to upgrade my Z2JH helm chart but got the following error, not sure how to fix this. Didn’t find anything on the internet. Everything is up but the hub pod is failing continuously with this error.

Screenshot 2023-06-04 at 23.56.44
Screenshot 2023-06-04 at 23.56.441920×577 134 KB

Here is the version, I upgraded to
Helm chat version: 2.0.0
JupyterHub: 3.0.0
Jupyterlab: 3.5.0

Other information:
DB: SQLLite (default)
Cull is enabled with the following settings

cull:
  timeout: 10800 # 3 hours
  every: 1800 # 30 minutes

Any help would be much appreciated.

2

@consideRatio any hints on how to solve this?

3

It sounds like you have some custom configuration. Can you show us your full config?

4

Here my custom configurations. If you are asking for something else, then please write back.

custom:
  myHost: "https://host.name"
  cpuGuarantee: 0.2
  cpuLimit: 4
  memGuarantee: '4G'
  memLimit: '6G'
5

Here is a section of config which is using these custom configs

hub:
  config:
   JupyterHub:
     authenticator_class: custom_authenticator_name
  extraConfig:
    spawnerTimeout: |
      c.Spawner.http_timeout = 600
      c.Spawner.start_timeout = 600

    misc: |
      import os
      c.JupyterHub.allow_named_servers = False
      c.JupyterHub.logo_file = os.path.abspath('/etc/jupyterhub/IDALogo.png')
      c.Application.log_level = 'DEBUG'

    aksFix: |
      import os
      os.environ['KUBERNETES_SERVICE_HOST'] = 'kubernetes.default.svc.cluster.local'

    authenticator: |
      import os
      c.Authenticator.delete_invalid_users = True
      c.Authenticator.auto_login = True

    allowHiddenFiles: |
      c.ContentsManager.allow_hidden = True

    jupytext: |
      c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
      c.ContentsManager.default_jupytext_formats = "ipynb,py"
      c.ContentsManager.preferred_jupytext_formats_save = "py:percent"

    cors: |
      import z2jh
      c.Spawner.args.append(f'--NotebookApp.allow_origin={z2jh.get_config("custom.myHost")}')
      c.JupyterHub.tornado_settings = {
          'headers': {
              'Access-Control-Allow-Origin': z2jh.get_config('custom.myHost'),
              'Access-Control-Allow-Methods': 'GET, POST, OPTIONS, PUT, DELETE',
          },
      }
      c.NotebookApp.allow_origin = z2jh.get_config('custom.myHost')

    spawner: |
      import z2jh
      from kubespawner import KubeSpawner

      class CustomFormSpawner(KubeSpawner):
          def _options_form_default(self):
              return f"""
                  <script type="text/javascript">
                      window.location.replace({z2jh.get_config('custom.myHost')})
                  </script>
                  <a href={z2jh.get_config('custom.myHost')}>Landing Page</a>
              """
      c.JupyterHub.spawner_class = CustomFormSpawner
      c.CustomFormSpawner.cpu_guarantee = z2jh.get_config('custom.cpuGuarantee')
      c.CustomFormSpawner.cpu_limit = z2jh.get_config('custom.cpuLimit')
      c.CustomFormSpawner.mem_guarantee = z2jh.get_config('custom.memGuarantee')
      c.CustomFormSpawner.mem_limit = z2jh.get_config('custom.memLimit')
6

Is this your full Z32JH configuration? In your first post you mentioned

but this isn’t in the configuration you’ve shared.

7

No, it is isn’t full. I have other config sections also. Do you need all ?

8

Ideally yes. If you don’t want to share it then could you try deploying Z2JH with the default configuration? Assuming that works then start adding parts of your config until you find the change that breaks your deployment.

9

Here are my configs
config.yaml

proxy:
  service:
    type: LoadBalancer
    annotations: 
      service.beta.kubernetes.io/azure-load-balancer-internal: 'true'
  https:
    enabled: true
    type: manual

singleuser:
  storage:
    type: none
  extraEnv: 
    GRANT_SUDO: "yes"
    NOTEBOOK_ARGS: "--allow-root"
  uid: 0
  cmd: start-singleuser.sh
  defaultUrl: "/lab"

scheduling:
  userScheduler:
    enabled: true
  podPriority:
    enabled: true
    globalDefault: true
    defaultPriority: 10
    userPlaceholderPriority: 0
  userPlaceholder:
    enabled: true

prePuller:
  continuous:
    enabled: true

hub:
  config:
   JupyterHub:
     authenticator_class: custom_authenticator_name
  extraConfig:
    spawnerTimeout: |
      c.Spawner.http_timeout = 600
      c.Spawner.start_timeout = 600

    misc: |
      import os
      c.JupyterHub.allow_named_servers = False
      c.JupyterHub.logo_file = os.path.abspath('/etc/jupyterhub/logo.png')
      c.Application.log_level = 'DEBUG'

    aksFix: |
      import os
      os.environ['KUBERNETES_SERVICE_HOST'] = 'kubernetes.svc.cluster.local'

    authenticator: |
      import os
      c.Authenticator.delete_invalid_users = True
      c.Authenticator.auto_login = True

    allowHiddenFiles: |
      c.ContentsManager.allow_hidden = True

    jupytext: |
      c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
      c.ContentsManager.default_jupytext_formats = "ipynb,py"
      c.ContentsManager.preferred_jupytext_formats_save = "py:percent"

    cors: |
      import z2jh
      c.Spawner.args.append(f'--NotebookApp.allow_origin={z2jh.get_config("custom.myHost")}')
      c.JupyterHub.tornado_settings = {
          'headers': {
              'Access-Control-Allow-Origin': z2jh.get_config('custom.myHost'),
              'Access-Control-Allow-Methods': 'GET, POST, OPTIONS, PUT, DELETE',
          },
      }
      c.NotebookApp.allow_origin = z2jh.get_config('custom.myHost')

    spawner: |
      import z2jh
      from kubespawner import KubeSpawner

      class CustomFormSpawner(KubeSpawner):
          def _options_form_default(self):
              return f"""
                  <script type="text/javascript">
                      window.location.replace({z2jh.get_config('custom.myHost')})
                  </script>
                  <a href={z2jh.get_config('custom.myHost')}>Landing Page</a>
              """
      c.JupyterHub.spawner_class = CustomFormSpawner
      c.CustomFormSpawner.cpu_guarantee = z2jh.get_config('custom.cpuGuarantee')
      c.CustomFormSpawner.cpu_limit = z2jh.get_config('custom.cpuLimit')
      c.CustomFormSpawner.mem_guarantee = z2jh.get_config('custom.memGuarantee')
      c.CustomFormSpawner.mem_limit = z2jh.get_config('custom.memLimit')

params.yaml

proxy:
  service:
    labels:
      environment : dev
      loadBalancerIP: xx.xx.xx.xx
  https:
    hosts:
      - my.custom.host.name
singleuser:
  image:
    name: 'datascience/notebook'
    tag: 'vX'
    pullSecrets:
      - "image-pull-secret"
  cpu:
    limit: 4
    guarantee: 1
  memory:
    limit: 26G
    guarantee: 26G

scheduling:
  podPriority:
    globalDefault: false
  userPlaceholder:
    replicas: 2

custom:
  myHost: "my.custom.host.name"
  cpuGuarantee: 0.2
  cpuLimit: 4
  memGuarantee: '2G'
  memLimit: '5G'

hub:
  config:
    AzureAdOAuthenticator:
      oauth_callback_url: https://my.custom.host.name.callback.url
  image:
    name: 'abc/jpytr-hub'
    tag: 'v200'
    pullSecrets:
      - "image-pull-secret"

imagePullSecret:
  automaticReferenceInjection: true
  registry: 'my.registry'
  enabled: true

cull:
  timeout: 10800
  every: 1800

Secrets.yaml

proxy:
  secretToken: 'my-secret-token'
  https:
    manual:
      key: |
        -----BEGIN RSA PRIVATE KEY-----
        my-private-key
        -----END RSA PRIVATE KEY-----
      cert: |
        -----BEGIN CERTIFICATE-----
        my-private-cert
        -----END CERTIFICATE-----

imagePullSecret:
  username: 'registry_user'
  password: 'registry_password'

hub:
  config:
    Authenticator:
      admin_users:
      - userA
      - userB
    AzureAdOAuthenticator:
      client_id: my-client-id
      client_secret: my-client-secret
      tenant_id: my-tenant-id
    JupyterHub:
      admin_access: true
  extraConfig:
    secret-api-token: |
      c.JupyterHub.services = [
          {
              "name": "service-token",
              "admin": True,
              "api_token": "my-api-token",
          },
      ]
10 
    secret-api-token: |
      c.JupyterHub.services = [
          {
              "name": "service-token",
              "admin": True,
              "api_token": "my-api-token",
          },
      ]

This is the issue, you override existing list of services like this, instead of append to it.

11

Thank you so much for your help.
Appending helped to bring up the hub instance. Much appreciated.

12

I am getting the following error which i am unable to solve

[D 2023-06-08 12:12:17.867 JupyterHub scopes:796] Checking access via scope access:servers                                                  
[D 2023-06-08 12:12:17.867 JupyterHub scopes:630] Client access refused; filters do not match API endpoint /hub/user/behroz-sikander request                                                                                                                                           
[W 2023-06-08 12:12:17.867 JupyterHub web:1796] 404 GET /hub/user/behroz-sikander (::ffff:10.238.18.39): No access to resources or resources not found

Interestingly, when I try to access the following API endpoint
https://my.host.name/hub/api/users
I get the following response which says that I have user and admin role.
As per the documentation, access:servers scope should be already part of the user role. Not sure why I am getting the 404 on the UI or access denied in the logs.

{
    "admin": true,
    "name": "behroz-sikander",
    "kind": "user",
    "server": "/user/behroz-sikander/",
    "roles": [
      "user",
      "admin"
    ],
    "auth_state": null,
    "pending": null,
    "created": "2023-06-08T10:43:42.481575Z",
    "last_activity": "2023-06-08T11:42:38.984000Z",
    "groups": [],
    "servers": {
      "": {
        "name": "",
        "last_activity": "2023-06-08T11:42:38.984000Z",
        "started": "2023-06-08T11:42:29.872127Z",
        "pending": null,
        "ready": true,
        "stopped": false,
        "url": "/user/behroz-sikander/",
        "user_options": {
          "profile_options": {
            "cpu_limit": 1,
            "cpu_guarantee": 1,
            "mem_limit": "4G",
            "mem_guarantee": "4G"
          },
        },
        "progress_url": "/hub/api/users/behroz-sikander/server/progress",
        "state": {
          "pod_name": "jupyter-behroz-2dsikander"
        }
      }
    }
13

Figured out the problem.
We have a custom override of azuread.py file of oauthenticator package. Due to the upgrade, the override failed which lead to default azuread being running. Due to it, the username got changed from firstname-lastname format to lastname, firstname format and all the permissions/roles in jupyter stopped working. After overriding the azuread.py file correctly, it works now.

Here is my custom docker file ontop of k8s-hub:2.0.0, since the python version was upgraded, the COPY below failed. Is there a better way to override the authenticate method of azuread?

FROM jupyterhub/k8s-hub:2.0.0

USER root
RUN rm -f /usr/local/lib/python3.9/site-packages/oauthenticator/azuread.py
COPY azuread.py /usr/local/lib/python3.9/site-packages/oauthenticator/azuread.py