Jupyterhub: Not updating activity

Jupyterhub installed by helm chart version 1.2.0

My Notebook can’t run after start

Notebook Log
Event log Server requested 2022-05-30T09:42:26.671381Z [Normal] Successfully assigned sever-test/jupyter-dtvge45vc to cl1ipvrh40gqcu5gvm6h-itac 2022-05-30T09:42:30Z [Normal] AttachVolume.Attach succeeded for volume "pvc-1b0fc0ab-6c33-437e-9830-88e0441a424f" 2022-05-30T09:42:34Z [Warning] MountVolume.MountDevice failed for volume "pvc-1b0fc0ab-6c33-437e-9830-88e0441a424f" : rpc error: code = Internal desc = Failed to find device path /dev/disk/by-id/virtio-fhm80s2igmt5mkus2cc7. error checking if target ( /dev/disk/by-id/virtio-fhm80s2igmt5mkus2cc7 ) is device: path "/dev/disk/by-id/virtio-fhm80s2igmt5mkus2cc7" does not exist 2022-05-30T09:42:35Z [Normal] Container image "jupyterhub/k8s-network-tools:1.2.0" already present on machine 2022-05-30T09:42:35Z [Normal] Created container block-cloud-metadata 2022-05-30T09:42:36Z [Normal] Started container block-cloud-metadata 2022-05-30T09:42:36Z [Normal] Pulling image "cr.yandex/crp4jk9s67voadrqietj/sever/jhub:master" 2022-05-30T09:42:37Z [Normal] Successfully pulled image "cr.yandex/crp4jk9s67voadrqietj/sever/jhub:master" in 959.131593ms 2022-05-30T09:42:37Z [Normal] Created container notebook 2022-05-30T09:42:37Z [Normal] Started container notebook

Container Log
[I 2022-05-30 12:42:39.316 SingleUserNotebookApp notebookapp:2329] Jupyter Notebook 6.4.10 is running at: [I 2022-05-30 12:42:39.316 SingleUserNotebookApp notebookapp:2329] http://jupyter-dtvge45vc:8888/user/dtvge45vc/ [I 2022-05-30 12:42:39.316 SingleUserNotebookApp notebookapp:2330] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [I 2022-05-30 12:42:39.321 SingleUserNotebookApp mixins:596] Updating Hub with activity every 300 seconds [D 2022-05-30 12:42:39.322 SingleUserNotebookApp mixins:558] Notifying Hub of activity 2022-05-30T09:42:38.460655Z [D 2022-05-30 12:47:10.451 SingleUserNotebookApp mixins:558] Notifying Hub of activity 2022-05-30T09:42:38.460655Z [D 2022-05-30 12:52:39.789 SingleUserNotebookApp mixins:558] Notifying Hub of activity 2022-05-30T09:42:38.460655Z [D 2022-05-30 12:57:12.509 SingleUserNotebookApp mixins:558] Notifying Hub of activity 2022-05-30T09:42:38.460655Z
Hub log
[D 2022-05-30 09:47:02.584 JupyterHub reflector:282] Connecting pods watcher [D 2022-05-30 09:47:03.310 JupyterHub log:189] 200 GET /hub/health (@10.115.158.1) 0.68ms [D 2022-05-30 09:47:05.310 JupyterHub log:189] 200 GET /hub/health (@10.115.158.1) 0.68ms [D 2022-05-30 09:47:07.310 JupyterHub log:189] 200 GET /hub/health (@10.115.158.1) 0.69ms [D 2022-05-30 09:47:09.310 JupyterHub log:189] 200 GET /hub/health (@10.115.158.1) 0.70ms [D 2022-05-30 09:47:10.328 JupyterHub reflector:357] events watcher timeout [D 2022-05-30 09:47:10.329 JupyterHub reflector:282] Connecting events watcher [D 2022-05-30 09:47:10.462 JupyterHub users:783] Not updating activity for <User(dtvge45vc 1/1 running)>: 2022-05-30T09:42:38.460655Z < 2022-05-30T09:45:54.810713Z [D 2022-05-30 09:47:10.462 JupyterHub users:804] Not updating server activity on dtvge45vc/: 2022-05-30T09:42:38.460655Z < 2022-05-30T09:45:54.810713Z [I 2022-05-30 09:47:10.463 JupyterHub log:189] 200 POST /hub/api/users/dtvge45vc/activity (dtvge45vc@10.115.129.152) 13.12ms [D 2022-05-30 09:47:11.310 JupyterHub log:189] 200 GET /hub/health (@10.115.158.1) 0.69ms [D 2022-05-30 09:47:11.557 JupyterHub log:189] 200 GET /hub/health (@10.115.158.1) 0.66ms [D 2022-05-30 09:47:12.472 JupyterHub base:1285] No template for 403 [W 2022-05-30 09:47:12.473 JupyterHub log:189] 403 GET /hub/metrics (@10.115.129.41) 1.44ms [D 2022-05-30 09:47:12.594 JupyterHub reflector:357] pods watcher timeout

Maybe it is important: I have this situation when my colleague use notebook pod with toleration’s configuration.

Hi! Please could you show us your Z2JH configuration with secrets redacted, tell us how your K8S cluster was setup/configured, and show us your logs with debug enabled? Thanks!

For installing I use this command:
helm install jhub --namespace mllab jupyterhub/jupyterhub -f config.yaml

config.yaml:

debug:
  enabled: true

hub:
  extraFiles:
    jupyter_notebook_config.json:
      mountPath: /etc/jupyter/jupyter_notebook_config.json
      data:
        NotebookApp:
          iopub_data_rate_limit = 10000000
    nodegroup_changer_py:
      mountPath: /srv/jupyterhub/nodegroup_changer/nodegroup_changer.py
      stringData: ./nodegroup_changer/nodegroup_changer.py
  extraConfig:
    myConfig: |
      c.Application.log_level = 'DEBUG'
      c.KubeSpawner.uid = 0
      c.KubeSpawner.notebook_dir = '/ipynb'
      c.KubeSpawner.args = ['--allow-root']
      c.KubeSpawner.image_pull_policy = 'Always'
      c.KubeSpawner.start_timeout = 60 * 15
  config:
    Authenticator:
      admin_users:
        - dtvge45vc
      allowed_users:
        - dtvge45vc
    GitHubOAuthenticator:
      client_id: 'XXXXXXXXXXXXXXXX'
      client_secret: 'XXXXXXXXXXXXXXXX'
      oauth_callback_url: 'http://XXXXXXXXXXXXXXXX/hub/oauth_callback'
    JupyterHub:
      admin_access: true
      authenticator_class: github

singleuser:
  extraEnv:
    PYTHONPATH: "/ipynb/private"
  image:
    pullPolicy: Always
  storage:
    capacity: 3Gi
    homeMountPath: /ipynb
    dynamic:
      storageClass: ssd
  serviceAccountName: hub
  profileList:
    - display_name: "Tiny | Minimal Environment"
      description: "CPU: 1, Memory: 4Gb"
      default: true
      kubespawner_override:
        image: XXXXXXXXXXXXXXXX
        component_label: 'Tiny'
        cpu_limit: 1
        mem_guarantee: 4G
        mem_limit: 4G
    - display_name: "Medium | Medium Environment"
      description: "CPU: 8, Memory: 16Gb"
      kubespawner_override:
        component_label: 'Medium'
        image: XXXXXXXXXXXXXXXX
        cpu_limit: 8
        mem_guarantee: 16G
        mem_limit: 16G
    - display_name: "Large | Large Environment"
      description: "CPU: 12, Memory: 48Gb"
      kubespawner_override:
        component_label: 'Large'
        image: XXXXXXXXXXXXXXXX
        cpu_limit: 12
        mem_guarantee: 48G
        mem_limit: 48G
    - display_name: "GPU | gpu Environment"
      description: "CPU: 4, Memory: 86Gb"
      kubespawner_override:
        initContainers:
          - name: init-myservice
            image: busybox:1.28
            command: ['sh', '-c', 'sleep 30']
        image: XXXXXXXXXXXXXXXX
        component_label: 'gpu'
        node_selector:
          nodegroup: gpu
        tolerations:
        - key: "nodegroup"
          operator: "Equal"
          value: "gpu"
          effect: "NoSchedule"
        cpu_limit: 4
        mem_guarantee: 86G
        mem_limit: 86G

cull:
  enabled: false

ingress:
  enabled: true
  annotations: { kubernetes.io/ingress.class: nginx-internal }
  hosts:
    - jhub.XXXXXXXXX
proxy:
  secretToken: "XXXXXXXXXXXXXXXXXXXXXXXXX"
  service:
    type: ClusterIP

imagePullSecret:
  create: true
  registry: XXXXXXX
  username: XXXXXX
  password: XXXXXXXXXXXXX

My logs in first message with debug enable

If this is the log message you are asking about:

there’s nothing wrong. It’s a result of activity coming from multiple sources (e.g. the server itself and the proxy). All that’s happening is the Hub already knows about activity more recent than what’s being given, so it doesn’t need to do anything.

The log message could certainly be clearer, though I admit I’m struggling on what language to use! Anyone have great ideas for the clearest, concise way to say “I am acknowledging that I was notified of an activity update, but I’m not updating the record in the database because the value already stored is more recent than the value I’ve just been notified about.”

OK. What pod log I need to add?

Perhaps something like “Not updating {service activity} that is already tracked in {other service activity}”? Depending on how verbose you want the log message you might be able to add in something about more recent timestamp too.