Strange Glitch - Jupyterhub Terminal repeatedly executes last command

[ramprasd89]

We are running Jupyterhub on AKS using the helm charts from here.
Strangely we can see that the Terminal repeats executing the last entered command without any user intervention. This stops once another command is executed. Please see the screengrab here:

Wonder if this is some sort of glitch or an issue with the configurations. Please let me know if you need additional inputs to troubleshoot this further.

Request your support here - @consideRatio

[ramprasd89]

Hi @manics - Could you please help in this topic?

[manics]

Hi! Please have a read through

Getting good answers to your questions

Eventually you will get stuck when trying to do something. This post is about how to get help with getting unstuck. People love helping others. Below some tips on how to, in my experience, improve your chances of getting a good answer. The answers you get will depend very much on the way you ask your question. Most people like hard problems and thought-provoking questions. So give them an interesting question to chew on, and they will love it. What are the hurdles to getting a good answer? Peo…

Ideally we need enough information to reproduce your problem.

[ramprasd89]

Hi,

Thanks for the reference. Helps me to ask the questions in a right and sensible way.
Please find the config chart we use to reproduce the problem.

# Jupyterhub Dev Config.yaml
imagePullSecrets: 
 - jhub-secret-acr-sp

hub:
  config:
    AzureAdOAuthenticator:
      admin_users: 
        - 
      username_claim: unique_name
      enable_auth_state: true
      client_id: <REDACTED>
      client_secret: <REDACTED>
      oauth_callback_url: <REDACTED>
      tenant_id: <REDACTED>
      scope:
        - openid
        - profile
        - email
    JupyterHub:
      admin_access: true
      authenticator_class: azuread  
  service:
    type: ClusterIP
  baseUrl: /
  extraConfig:
    CustomSpawner: | 
      c.Spawner.cmd = ['start.sh','jupyterhub-singleuser','--allow-root']
#      c.KubeSpawner.args = ['--allow-root']
#      def notebook_dir_hook(spawner):
#        spawner.environment = {'NB_USER':spawner.user.name,'NB_UID':'1000'}
#      c.Spawner.pre_spawn_hook = notebook_dir_hook
  image:
    name: jupyterhub/k8s-hub
    tag: "2.0.0"

# proxy relates to the proxy pod, the proxy-public service, and the autohttps
# pod and proxy-http service.
proxy:
  chp:
    image:
      name: jupyterhub/configurable-http-proxy
      # tag is automatically bumped to new patch versions by the
      # watch-dependencies.yaml workflow.
      #
      tag: "4.5.3" # https://github.com/jupyterhub/configurable-http-proxy/releases
  secretSync:
    image:
      name: jupyterhub/k8s-secret-sync
      tag: "2.0.0"

# singleuser relates to the configuration of KubeSpawner which runs in the hub
# pod, and its spawning of user pods such as jupyter-myusername.
singleuser:
  networkTools:
    image:
      name: jupyterhub/k8s-network-tools
      tag: "2.0.0"
  extraEnv: {}
    # #CHOWN_HOME: 'yes'
    # # GRANT_SUDO: "yes"
  lifecycleHooks:
    postStart:
      exec:
        command:
          - "sh"
          - "-c"
          - >
            cp -r /home/shared/example_notebooks ./;
            cp -rnT /etc/skel/ ./; 
            conda config --add envs_dirs /home/jovyan/conda-envs/; 
  storage:
    type: dynamic 
    extraLabels: {}
    extraVolumes:
      - name: shared-vol
        persistentVolumeClaim:
            claimName: pvc-blob 
    extraVolumeMounts:
      - name: shared-vol
        mountPath: /home/shared
    static:
      pvcName: pvc-jhub-static-azuredisk
      subPath: "{username}"
    capacity: 10Gi
    homeMountPath: /home/jovyan #test this
    dynamic:
      storageClass: #jupyter-azurefile
      pvcNameTemplate: claim-{username}{servername}
      volumeNameTemplate: volume-{username}{servername}
      storageAccessModes: [ReadWriteOnce]
  image:
    name: <REDACTED>/jupyter/minimal-notebook
    tag: "defaultv03"
    pullPolicy:
    pullSecrets: 
     - jhub-secret-acr-sp
  cmd: jupyterhub-singleuser
  profileList:
    - display_name: "DE Environment(default)"
      description: "Default Environment for DE Data Engineering activities"
      default: true
    - display_name: "DE ML Pipeline Environment"
      description: "Environment to run the DE AIStudio ML Pipelines"
      kubespawner_override:
        image: <REDACTED>/jupyter/minimal-notebook:v21
        image_pull_secrets: "jhub-secret-acr-sp"
    - display_name: "DE AI Studio Environment"
      description: "Non ML AI Studio Environment"
      kubespawner_override:
        image: <REDACTED>/jupyter/minimal-notebook:v01
        image_pull_secrets: "jhub-secret-acr-sp"
    - display_name: "Jupyter Minimal Notebook"
      description: "A minimal playground for getting started with Jupyterhub"
      kubespawner_override:
        image: jupyter/minimal-notebook:latest

# scheduling relates to the user-scheduler pods and user-placeholder pods.
scheduling:
  userScheduler:
    image:
      name: k8s.gcr.io/kube-scheduler
      tag: "v1.23.10" # ref: https://github.com/kubernetes/website/blob/c783bb37c972ad7d565c3a6debfb538b12c3843c/content/en/releases/patch-releases.md
  userPlaceholder:
    enabled: true
    image:
      name: k8s.gcr.io/pause
      tag: "3.8"

# prePuller relates to the hook|continuous-image-puller DaemonsSets
prePuller:
  hook:
    # image and the configuration below relates to the hook-image-awaiter Job
    image:
      name: jupyterhub/k8s-image-awaiter
      tag: "2.0.0"
  pause:
    image:
      name: k8s.gcr.io/pause
      tag: "3.8"

# cull relates to the jupyterhub-idle-culler service, responsible for evicting
# inactive singleuser pods.
#
# The configuration below, except for enabled, corresponds to command-line flags
# for jupyterhub-idle-culler as documented here:
# https://github.com/jupyterhub/jupyterhub-idle-culler#as-a-standalone-script
#
# cull:
#   enabled: true
#   users: false # --cull-users
#   adminUsers: true # --cull-admin-users
#   removeNamedServers: false # --remove-named-servers
#   timeout: 3600 # --timeout
#   every: 600 # --cull-every
#   concurrency: 10 # --concurrency
#   maxAge: 0 # --max-age
debug:
  enabled: true

[manics]

I can’t see obvious problems with your configuration, so you might need to start debugging things one step at a time. Things you can try include:

• Deploy the standard Z2JH without customisations
• Deploy JupyterLab directly on your K8S cluster
• Clear your browser cache, try an incognito/private browser window, try a different browser, try a different computer (if it’s an institutionally managed computer try a personal one)
• Check whether there’s any any firewall or network monitoring software any where (your computer, your K8s cluster, your network, proxies on your network, etc)

[ramprasd89]

Hi @manics

Tried the below steps:

• Deploy the standard Z2JH without customisations
  - The glitch is not seen, when performing a plain install without any custom values file.
  (helm upgrade --cleanup-on-fail \ --install <helm-release-name> jupyterhub/jupyterhub \ --namespace <k8s-namespace> \ --create-namespace)
  - The issue starts to replicate when we start using custom values
  (helm upgrade --cleanup-on-fail \ --install <helm-release-name> jupyterhub/jupyterhub \ --namespace <k8s-namespace> \ --create-namespace --values config.yaml)
• Clear your browser cache, try an incognito/private browser window, try a different browser, try a different computer (if it’s an institutionally managed computer try a personal one)
  - Issue persists
• Check whether there’s any any firewall or network monitoring software any where (your computer, your K8s cluster, your network, proxies on your network, etc)
  - We have disabled all firewall/related software that lies in the path of this service, but the issue persist.

[ramprasd89]

Found this in the pod logs, every time the command runs on its own -

[I 2023-01-25 13:05:49.096 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.4) 2.75ms

[ramprasd89]

Complete logs here:

[I 2023-01-25 13:43:04.859 SingleUserLabApp management:365] New terminal with automatic name: 1
[I 2023-01-25 13:43:04.862 SingleUserLabApp log:186] 200 POST /user/admin/api/terminals?1674654184109 (admin@::ffff:10.0.96.5) 620.48ms
[I 2023-01-25 13:43:04.867 SingleUserLabApp log:186] 204 PUT /user/admin/lab/api/workspaces/default?1674654184538 (admin@::ffff:10.0.96.4) 2.60ms
[I 2023-01-25 13:43:05.105 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654184978 (admin@::ffff:10.0.96.4) 2.03ms
[I 2023-01-25 13:43:05.887 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.5) 2.25ms
[I 2023-01-25 13:43:05.959 SingleUserLabApp log:186] 204 PUT /user/admin/lab/api/workspaces/default?1674654185830 (admin@::ffff:10.0.96.4) 2.06ms
[I 2023-01-25 13:43:09.210 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654189081 (admin@::ffff:10.0.96.4) 2.11ms
[I 2023-01-25 13:43:09.212 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654189082 (admin@::ffff:10.0.96.5) 1.55ms
[I 2023-01-25 13:43:12.907 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654192776 (admin@::ffff:10.0.96.5) 2.84ms
[I 2023-01-25 13:43:15.346 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654195216 (admin@::ffff:10.0.96.5) 2.02ms
[I 2023-01-25 13:43:19.452 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654199321 (admin@::ffff:10.0.96.5) 2.98ms
[I 2023-01-25 13:43:19.453 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654199322 (admin@::ffff:10.0.96.4) 2.32ms
[I 2023-01-25 13:43:23.153 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654203024 (admin@::ffff:10.0.96.4) 2.98ms
[I 2023-01-25 13:43:25.593 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654205464 (admin@::ffff:10.0.96.4) 2.89ms
[I 2023-01-25 13:43:29.688 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654209562 (admin@::ffff:10.0.96.4) 1.71ms
[I 2023-01-25 13:43:29.692 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654209562 (admin@::ffff:10.0.96.5) 1.48ms
[I 2023-01-25 13:43:33.404 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654213265 (admin@::ffff:10.0.96.5) 2.54ms
[I 2023-01-25 13:43:35.833 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654215704 (admin@::ffff:10.0.96.5) 1.76ms
[I 2023-01-25 13:43:39.930 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654219801 (admin@::ffff:10.0.96.5) 2.07ms
[I 2023-01-25 13:43:39.932 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654219803 (admin@::ffff:10.0.96.4) 1.41ms
[I 2023-01-25 13:43:43.652 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654223521 (admin@::ffff:10.0.96.4) 2.58ms
[I 2023-01-25 13:43:46.077 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654225945 (admin@::ffff:10.0.96.4) 1.68ms
[I 2023-01-25 13:43:50.170 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654230041 (admin@::ffff:10.0.96.4) 3.06ms
[I 2023-01-25 13:43:50.171 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654230042 (admin@::ffff:10.0.96.5) 2.28ms
[I 2023-01-25 13:43:53.899 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654233769 (admin@::ffff:10.0.96.5) 2.49ms
[I 2023-01-25 13:43:56.320 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654236193 (admin@::ffff:10.0.96.5) 1.76ms
[I 2023-01-25 13:44:00.370 SingleUserLabApp log:186] 200 GET /user/admin/api/kernelspecs?1674654240241 (admin@::ffff:10.0.96.5) 2.73ms
[I 2023-01-25 13:44:00.417 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654240288 (admin@::ffff:10.0.96.4) 1.98ms
[I 2023-01-25 13:44:00.420 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654240289 (admin@::ffff:10.0.96.5) 1.36ms
[I 2023-01-25 13:44:04.148 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654244016 (admin@::ffff:10.0.96.5) 2.86ms
[I 2023-01-25 13:44:06.561 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654246432 (admin@::ffff:10.0.96.5) 1.62ms
[I 2023-01-25 13:44:10.657 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654250528 (admin@::ffff:10.0.96.5) 2.78ms
[I 2023-01-25 13:44:10.658 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654250529 (admin@::ffff:10.0.96.4) 2.19ms
[I 2023-01-25 13:44:14.388 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654254257 (admin@::ffff:10.0.96.4) 2.72ms
[I 2023-01-25 13:44:16.801 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654256673 (admin@::ffff:10.0.96.4) 1.72ms
[I 2023-01-25 13:44:20.901 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654260769 (admin@::ffff:10.0.96.4) 2.77ms
[I 2023-01-25 13:44:20.902 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654260770 (admin@::ffff:10.0.96.5) 2.32ms
[I 2023-01-25 13:44:24.636 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654264505 (admin@::ffff:10.0.96.4) 2.33ms
[I 2023-01-25 13:44:27.042 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654266912 (admin@::ffff:10.0.96.4) 1.69ms
[I 2023-01-25 13:44:31.149 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654271017 (admin@::ffff:10.0.96.4) 3.04ms
[I 2023-01-25 13:44:31.150 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654271018 (admin@::ffff:10.0.96.5) 2.36ms
[I 2023-01-25 13:44:34.881 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654274752 (admin@::ffff:10.0.96.4) 2.77ms
[I 2023-01-25 13:44:36.654 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.4) 1.70ms
[I 2023-01-25 13:44:37.283 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654277152 (admin@::ffff:10.0.96.4) 1.63ms
[I 2023-01-25 13:44:41.394 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654281266 (admin@::ffff:10.0.96.5) 1.93ms
[I 2023-01-25 13:44:41.397 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654281265 (admin@::ffff:10.0.96.4) 1.31ms
[I 2023-01-25 13:44:45.122 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654284992 (admin@::ffff:10.0.96.4) 2.86ms
[I 2023-01-25 13:44:47.533 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654287401 (admin@::ffff:10.0.96.4) 1.72ms
[I 2023-01-25 13:44:51.634 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654291504 (admin@::ffff:10.0.96.4) 3.29ms
[I 2023-01-25 13:44:51.637 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654291505 (admin@::ffff:10.0.96.5) 1.67ms
[I 2023-01-25 13:44:55.362 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654295232 (admin@::ffff:10.0.96.5) 2.51ms
[I 2023-01-25 13:44:57.779 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654297649 (admin@::ffff:10.0.96.5) 1.61ms
[I 2023-01-25 13:45:01.618 SingleUserLabApp log:186] 200 GET /user/admin/api/kernelspecs?1674654301489 (admin@::ffff:10.0.96.5) 2.50ms
[I 2023-01-25 13:45:01.874 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654301745 (admin@::ffff:10.0.96.5) 1.81ms
[I 2023-01-25 13:45:01.879 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654301746 (admin@::ffff:10.0.96.4) 1.65ms
[I 2023-01-25 13:45:05.603 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654305473 (admin@::ffff:10.0.96.4) 2.82ms
[I 2023-01-25 13:45:07.414 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.5) 1.89ms
[I 2023-01-25 13:45:08.018 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654307888 (admin@::ffff:10.0.96.4) 2.15ms
[I 2023-01-25 13:45:12.124 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654311994 (admin@::ffff:10.0.96.5) 1.81ms
[I 2023-01-25 13:45:12.128 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654311993 (admin@::ffff:10.0.96.4) 2.13ms
[I 2023-01-25 13:45:15.844 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654315713 (admin@::ffff:10.0.96.4) 2.49ms
[I 2023-01-25 13:45:18.257 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654318129 (admin@::ffff:10.0.96.4) 1.75ms
[I 2023-01-25 13:45:22.363 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654322234 (admin@::ffff:10.0.96.4) 1.63ms
[I 2023-01-25 13:45:22.369 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654322242 (admin@::ffff:10.0.96.5) 1.31ms
[I 2023-01-25 13:45:26.084 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654325953 (admin@::ffff:10.0.96.5) 2.71ms
[I 2023-01-25 13:45:28.500 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654328369 (admin@::ffff:10.0.96.5) 1.57ms
[I 2023-01-25 13:45:32.610 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654332474 (admin@::ffff:10.0.96.5) 5.99ms
[I 2023-01-25 13:45:32.615 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654332475 (admin@::ffff:10.0.96.4) 7.07ms
[I 2023-01-25 13:45:36.324 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654336192 (admin@::ffff:10.0.96.4) 2.92ms
[I 2023-01-25 13:45:38.739 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654338609 (admin@::ffff:10.0.96.4) 1.97ms
[I 2023-01-25 13:45:39.664 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.4) 1.78ms
[I 2023-01-25 13:45:42.850 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654342721 (admin@::ffff:10.0.96.4) 3.08ms
[I 2023-01-25 13:45:42.851 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654342722 (admin@::ffff:10.0.96.5) 2.19ms
[I 2023-01-25 13:45:46.571 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654346441 (admin@::ffff:10.0.96.4) 2.49ms
[I 2023-01-25 13:45:48.980 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654348849 (admin@::ffff:10.0.96.4) 1.56ms
[I 2023-01-25 13:45:53.090 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654352961 (admin@::ffff:10.0.96.4) 1.86ms
[I 2023-01-25 13:45:53.096 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654352962 (admin@::ffff:10.0.96.5) 1.33ms
[I 2023-01-25 13:45:56.813 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654356680 (admin@::ffff:10.0.96.5) 2.82ms
[I 2023-01-25 13:45:59.227 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654359096 (admin@::ffff:10.0.96.5) 1.73ms
[I 2023-01-25 13:46:02.859 SingleUserLabApp log:186] 200 GET /user/admin/api/kernelspecs?1674654362729 (admin@::ffff:10.0.96.5) 2.58ms
[I 2023-01-25 13:46:03.338 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654363210 (admin@::ffff:10.0.96.5) 1.78ms
[I 2023-01-25 13:46:03.340 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654363211 (admin@::ffff:10.0.96.4) 1.47ms
[I 2023-01-25 13:46:07.059 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654366929 (admin@::ffff:10.0.96.4) 2.81ms
[I 2023-01-25 13:46:09.465 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654369338 (admin@::ffff:10.0.96.4) 1.58ms
[I 2023-01-25 13:46:10.403 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.5) 1.68ms
[I 2023-01-25 13:46:13.577 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654373450 (admin@::ffff:10.0.96.4) 1.73ms
[I 2023-01-25 13:46:13.582 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654373451 (admin@::ffff:10.0.96.5) 1.68ms
[I 2023-01-25 13:46:17.301 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654377170 (admin@::ffff:10.0.96.5) 2.91ms
[I 2023-01-25 13:46:19.706 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654379578 (admin@::ffff:10.0.96.5) 1.71ms
[I 2023-01-25 13:46:23.818 SingleUserLabApp log:186] 200 GET /user/admin/api/sessions?1674654383689 (admin@::ffff:10.0.96.5) 1.70ms
[I 2023-01-25 13:46:23.831 SingleUserLabApp log:186] 200 GET /user/admin/api/kernels?1674654383690 (admin@::ffff:10.0.96.4) 1.39ms
[I 2023-01-25 13:46:27.539 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654387409 (admin@::ffff:10.0.96.4) 2.66ms
[I 2023-01-25 13:46:29.945 SingleUserLabApp log:186] 200 GET /user/admin/api/terminals?1674654389818 (admin@::ffff:10.0.96.4) 1.63ms
[I 2023-01-25 13:47:11.162 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.4) 1.65ms
[I 2023-01-25 13:47:41.906 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.4) 1.94ms
[I 2023-01-25 13:48:12.672 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.5) 22.16ms
[I 2023-01-25 13:48:43.402 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.4) 1.64ms
[I 2023-01-25 13:48:45.962 SingleUserLabApp log:186] 200 GET /user/admin/api/contents?content=1&1674654525315 (admin@::ffff:10.0.96.4) 2.44ms
[I 2023-01-25 13:49:14.192 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.4) 1.79ms
[I 2023-01-25 13:49:44.931 SingleUserLabApp log:186] 101 GET /user/admin/terminals/websocket/1 (admin@::ffff:10.0.96.4) 1.91ms

[manics]

@ramprasd89

Deploy the standard Z2JH without customisations
- The glitch is not seen, when performing a plain install without any custom values file.
(helm upgrade --cleanup-on-fail \ --install <helm-release-name> jupyterhub/jupyterhub \ --namespace <k8s-namespace> \ --create-namespace)
- The issue starts to replicate when we start using custom values

You can probably guess what the next step is then! Start from the plain working config and slowly modify one thing at a time until you identify the change that causes the problem.

[minrk]

this is almost certainly related to the versions of packages in your user image. Most likely relevant packages:

• jupyterlab
• jupyter_server
• jupyter_server_terminals

You should also be able to reproduce this without jupyterhub or kubernetes or anything by running your user image locally in docker, without involving any jupyterhub bits.

One question, though: is it really executing the same command multiple times, or is it showing you the same command and its output multiple times? There’s a difference in where the bug would be. Try picking a command that will produce a different output when run multiple times, e.g. openssl rand -hex 4 or python -c 'import time; print(time.time())'.

[ramprasd89]

Hi,

Thanks for the response.

@manics - The moment I introduce any small config - for e.g to set the proxy.service.type: ClusterIP - I start to see this glitch.

@minrk
Tried executing the openssl rand -hex 4 command and left the terminal idle and see the same output multiple times.

(base) jovyan@jupyter-admin-40test-2ecom:~$ openssl rand -hex 4
d97dcd90
(base) jovyan@jupyter-admin-40test-2ecom:~$ openssl rand -hex 4
d97dcd90
(base) jovyan@jupyter-admin-40test-2ecom:~$ openssl rand -hex 4
d97dcd90

[minrk]

ok, that’s useful info - that it’s a problem in the javascript terminal repeating lines. I’m not sure if this is a problem in the javascript, backend, or syncing between the two, but it narrows it down.

Here’s a question: does it happen only when lines are long, or also when the lines don’t wrap (e.g. export PS1='$ ')?

[ramprasd89]

@minrk

Here’s a question: does it happen only when lines are long, or also when the lines don’t wrap (e.g. export PS1='$ ')?

It happens irrespective of the prompt string or the command length.

[ramprasd89]

Hi Experts -

Any thoughts/possible solution on this problem here?

Regards,
Ram

[deafcrump]

Thanks for the reference. Helps me to ask the questions in a right and sensible way.

[deafcrump]

Thanks for the reference. .

.

[MoKassem]

I’m facing the exact same issue on all these versions of the helm chart

NAME	CHART VERSION	APP VERSION	DESCRIPTION
jupyterhub/jupyterhub	3.0.2	4.0.2	Multi-user Jupyter installation
jupyterhub/jupyterhub	3.0.1	4.0.2	Multi-user Jupyter installation
jupyterhub/jupyterhub	3.0.0	4.0.2	Multi-user Jupyter installation
jupyterhub/jupyterhub	2.0.0	3.0.0	Multi-user Jupyter installation