Spawn failed: Could not create pod

Hi @all,
Since few days now I’m facing this issue (Spawn failed: Could not create pod pod-name/jupyter-user) when launching pods. At time its works but most of the time it doesn’t work and it’s worrying me.
Here is me configuration:

# fullnameOverride and nameOverride distinguishes blank strings, null values,
# and non-blank strings. For more details, see the configuration reference.
fullnameOverride: "jupyter"
nameOverride: "jupyter"

# custom can contain anything you want to pass to the hub pod, as all passed
# Helm template values will be made available there.
custom: {}

# imagePullSecret is configuration to create a k8s Secret that Helm chart's pods
# can get credentials from to pull their images.
imagePullSecret:
  create: false
  automaticReferenceInjection: true
  registry: #xxxxxxxxxxxxxxxxxx
  username: #kubernetes
  password: #xxxxxxxxxxxxx
  email: #xxxxxxxxxxxxx
# imagePullSecrets is configuration to reference the k8s Secret resources the
# Helm chart's pods can get credentials from to pull their images.
imagePullSecrets: []

# hub relates to the hub pod, responsible for running JupyterHub, its configured
# Authenticator class KubeSpawner, and its configured Proxy class
# ConfigurableHTTPProxy. KubeSpawner creates the user pods, and
# ConfigurableHTTPProxy speaks with the actual ConfigurableHTTPProxy server in
# the proxy pod.
hub:
  config:
    Authenticator:
      admin_users:
        - chrys
    JupyterHub:
      admin_access: true
      authenticator_class: dummy
  service:
    type: ClusterIP
    annotations: {}
    ports:
      nodePort:
    extraPorts: []
    loadBalancerIP:
  baseUrl: /
  cookieSecret:
  initContainers: []
  fsGid: 1000
  nodeSelector: {}
  tolerations: []
  concurrentSpawnLimit: 64
  consecutiveFailureLimit: 5
  activeServerLimit:
  deploymentStrategy:
    ## type: Recreate
    ## - sqlite-pvc backed hubs require the Recreate deployment strategy as a
    ##   typical PVC storage can only be bound to one pod at the time.
    ## - JupyterHub isn't designed to support being run in parallell. More work
    ##   needs to be done in JupyterHub itself for a fully highly available (HA)
    ##   deployment of JupyterHub on k8s is to be possible.
    type: Recreate
  db:
    type: sqlite-pvc
    upgrade:
    pvc:
      annotations: {}
      selector: {}
      accessModes:
        - ReadWriteOnce
      storage: 1Gi
      subPath:
      storageClassName:
    url:
    password:
  labels: {}
  annotations: {cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'}
  command: []
  args: []
  extraConfig: {}
  extraFiles: {}
  extraEnv: {}
  extraContainers: []
  extraVolumes: []
  extraVolumeMounts: []
  image:
    name: jupyterhub/k8s-hub
    tag: "1.2.0"
    pullPolicy:
    pullSecrets: []
  resources: {}
  containerSecurityContext:
    runAsUser: 1000
    runAsGroup: 1000
    allowPrivilegeEscalation: false
  lifecycle: {}
  services: {}
  pdb:
    enabled: false
    maxUnavailable:
    minAvailable: 1
  networkPolicy:
    enabled: true
    ingress: []
    ## egress for JupyterHub already includes Kubernetes internal DNS and
    ## access to the proxy, but can be restricted further, but ensure to allow
    ## access to the Kubernetes API server that couldn't be pinned ahead of
    ## time.
    ##
    ## ref: https://stackoverflow.com/a/59016417/2220152
    egress:
      - to:
          - ipBlock:
              cidr: 0.0.0.0/0
    interNamespaceAccessLabels: ignore
    allowedIngressPorts: []
  allowNamedServers: false
  namedServerLimitPerUser:
  authenticatePrometheus:
  redirectToServer:
  shutdownOnLogout:
  templatePaths: []
  templateVars: {}
  livenessProbe:
    # The livenessProbe's aim to give JupyterHub sufficient time to startup but
    # be able to restart if it becomes unresponsive for ~5 min.
    enabled: true
    initialDelaySeconds: 300
    periodSeconds: 10
    failureThreshold: 30
    timeoutSeconds: 3
  readinessProbe:
    # The readinessProbe's aim is to provide a successful startup indication,
    # but following that never become unready before its livenessProbe fail and
    # restarts it if needed. To become unready following startup serves no
    # purpose as there are no other pod to fallback to in our non-HA deployment.
    enabled: true
    initialDelaySeconds: 0
    periodSeconds: 2
    failureThreshold: 1000
    timeoutSeconds: 1
  existingSecret:
  serviceAccount:
    annotations: {}
  extraPodSpec: {}

rbac:
  enabled: true

# proxy relates to the proxy pod, the proxy-public service, and the autohttps
# pod and proxy-http service.
proxy:
  secretToken:
  annotations: {cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'}
  deploymentStrategy:
    ## type: Recreate
    ## - JupyterHub's interaction with the CHP proxy becomes a lot more robust
    ##   with this configuration. To understand this, consider that JupyterHub
    ##   during startup will interact a lot with the k8s service to reach a
    ##   ready proxy pod. If the hub pod during a helm upgrade is restarting
    ##   directly while the proxy pod is making a rolling upgrade, the hub pod
    ##   could end up running a sequence of interactions with the old proxy pod
    ##   and finishing up the sequence of interactions with the new proxy pod.
    ##   As CHP proxy pods carry individual state this is very error prone. One
    ##   outcome when not using Recreate as a strategy has been that user pods
    ##   have been deleted by the hub pod because it considered them unreachable
    ##   as it only configured the old proxy pod but not the new before trying
    ##   to reach them.
    type: Recreate
    ## rollingUpdate:
    ## - WARNING:
    ##   This is required to be set explicitly blank! Without it being
    ##   explicitly blank, k8s will let eventual old values under rollingUpdate
    ##   remain and then the Deployment becomes invalid and a helm upgrade would
    ##   fail with an error like this:
    ##
    ##     UPGRADE FAILED
    ##     Error: Deployment.apps "proxy" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate'
    ##     Error: UPGRADE FAILED: Deployment.apps "proxy" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate'
    rollingUpdate:
  # service relates to the proxy-public service
  service:
    type: ClusterIP
    labels: {}
    annotations: {cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'}
    nodePorts:
      http:
      https:
    disableHttpPort: false
    extraPorts: []
    loadBalancerIP:
    loadBalancerSourceRanges: []
  # chp relates to the proxy pod, which is responsible for routing traffic based
  # on dynamic configuration sent from JupyterHub to CHP's REST API.
  chp:
    containerSecurityContext:
      runAsUser: 65534 # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    image:
      name: jupyterhub/configurable-http-proxy
      tag: 4.5.0 # https://github.com/jupyterhub/configurable-http-proxy/releases
      pullPolicy:
      pullSecrets: []
    extraCommandLineFlags: []
    livenessProbe:
      enabled: true
      initialDelaySeconds: 60
      periodSeconds: 10
    readinessProbe:
      enabled: true
      initialDelaySeconds: 0
      periodSeconds: 2
      failureThreshold: 1000
    resources: {}
    defaultTarget:
    errorTarget:
    extraEnv: {}
    nodeSelector: {}
    tolerations: []
    networkPolicy:
      enabled: true
      ingress: []
      egress:
        - to:
            - ipBlock:
                cidr: 0.0.0.0/0
      interNamespaceAccessLabels: ignore
      allowedIngressPorts: [http, https]
    pdb:
      enabled: false
      maxUnavailable:
      minAvailable: 1
    extraPodSpec: {}
  # traefik relates to the autohttps pod, which is responsible for TLS
  # termination when proxy.https.type=letsencrypt.
  traefik:
    containerSecurityContext:
      runAsUser: 65534 # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    image:
      name: traefik
      tag: v2.4.11 # ref: https://hub.docker.com/_/traefik?tab=tags
      pullPolicy:
      pullSecrets: []
    hsts:
      includeSubdomains: false
      preload: false
      maxAge: 15724800 # About 6 months
    resources: {}
    labels: {}
    extraEnv: {}
    extraVolumes: []
    extraVolumeMounts: []
    extraStaticConfig: {}
    extraDynamicConfig: {}
    nodeSelector: {}
    tolerations: []
    extraPorts: []
    networkPolicy:
      enabled: true
      ingress: []
      egress:
        - to:
            - ipBlock:
                cidr: 0.0.0.0/0
      interNamespaceAccessLabels: ignore
      allowedIngressPorts: [http, https]
    pdb:
      enabled: false
      maxUnavailable:
      minAvailable: 1
    serviceAccount: #
      annotations: {}
    extraPodSpec: {}
  secretSync:
    containerSecurityContext:
      runAsUser: 65534 # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    image:
      name: jupyterhub/k8s-secret-sync
      tag: "1.2.0"
      pullPolicy:
      pullSecrets: []
    resources: {}
  labels: {}
  https:
    enabled: false
    type: letsencrypt
    #type: letsencrypt, manual, offload, secret
    letsencrypt:
      contactEmail:
      # Specify custom server here (https://acme-staging-v02.api.letsencrypt.org/directory) to hit staging LE
      acmeServer: https://acme-v02.api.letsencrypt.org/directory
    manual:
      key:
      cert:
    secret:
      name:
      key: tls.key
      crt: tls.crt
    hosts: []

# singleuser relates to the configuration of KubeSpawner which runs in the hub
# pod, and its spawning of user pods such as jupyter-myusername.
singleuser:
  podNameTemplate:
  extraTolerations: []
  nodeSelector: {}
  extraNodeAffinity:
    required: []
    preferred: []
  extraPodAffinity:
    required: []
    preferred: []
  extraPodAntiAffinity:
    required: []
    preferred: []
  networkTools:
    image:
      name: jupyterhub/k8s-network-tools
      tag: "1.2.0"
      pullPolicy:
      pullSecrets: []
  cloudMetadata:
    # block set to true will append a privileged initContainer using the
    # iptables to block the sensitive metadata server at the provided ip.
    blockWithIptables: true
    ip: 169.254.169.254
  networkPolicy:
    enabled: true
    ingress: []
    egress:
      # Required egress to communicate with the hub and DNS servers will be
      # augmented to these egress rules.
      #
      # This default rule explicitly allows all outbound traffic from singleuser
      # pods, except to a typical IP used to return metadata that can be used by
      # someone with malicious intent.
      - to:
          - ipBlock:
              cidr: 0.0.0.0/0
              except:
                - 169.254.169.254/32
    interNamespaceAccessLabels: ignore
    allowedIngressPorts: []
  events: true
  extraAnnotations: {}
  extraLabels:
    hub.jupyter.org/network-access-hub: "true"
  extraFiles: {}
  extraEnv: 
    JUPYTERHUB_SINGLEUSER_APP: "jupyter_server.serverapp.ServerApp"
  lifecycleHooks: 
    postStart:
      exec:
        command: ["/bin/sh", "-c", "echo Hello from overridden handler > $HOME/message"]
  initContainers: []
  extraContainers: []
  uid: 1000
  fsGid: 100
  serviceAccountName:
  storage:
    type: dynamic
    extraLabels: {}
    extraVolumes: []
    extraVolumeMounts: []
    static:
      pvcName:
      subPath: "{username}"
    capacity: 20Gi
    #homeMountPath: /home/jovyan
    homeMountPath: /home/{username}
    dynamic:
      storageClass:
      pvcNameTemplate: claim-{username}{servername}
      volumeNameTemplate: volume-{username}{servername}
      storageAccessModes: [ReadWriteOnce]
  image:
    name: jupyterhub/k8s-singleuser-sample
    tag: "1.2.0"
    pullPolicy:
    pullSecrets: 
      - "gitlab-docker-registry"
  startTimeout: 300
  cpu:
    limit:
    guarantee: 
  memory:
    limit:
    guarantee: 
  extraResource:
    limits: {}
    guarantees: {}
  cmd: jupyterhub-singleuser
  defaultUrl: "/lab"
  extraPodConfig: {}
  profileList:
    - display_name: "Default environment"
      description: "Image de base de jupyter-lab"
      default: true
    - display_name: "Wiremind mantis"
      description: "Wiremind image"
      kubespawner_override:
        image: docker.wiremind.fr/wiremind/data-science/mantis/cpu-jupyter/master:latest
    # - display_name: "Wiremind custom"
    #   description: "Wiremind custom image"
    #   kubespawner_override:
    #     image: docker.wiremind.fr/wiremind/data-science/mantis/cpu-jupyter/custom:latest

# scheduling relates to the user-scheduler pods and user-placeholder pods.
scheduling:
  userScheduler:
    enabled: true
    replicas: 2                                                                                                               
    logLevel: 4
    # plugins ref: https://kubernetes.io/docs/reference/scheduling/config/#scheduling-plugins-1
    plugins:
      score:
        disabled:
          - name: SelectorSpread
          - name: TaintToleration
          - name: PodTopologySpread
          - name: NodeResourcesBalancedAllocation
          - name: NodeResourcesLeastAllocated
          # Disable plugins to be allowed to enable them again with a different
          # weight and avoid an error.
          - name: NodePreferAvoidPods
          - name: NodeAffinity
          - name: InterPodAffinity
          - name: ImageLocality
        enabled:
          - name: NodePreferAvoidPods
            weight: 161051
          - name: NodeAffinity
            weight: 14631
          - name: InterPodAffinity
            weight: 1331
          - name: NodeResourcesMostAllocated
            weight: 121
          - name: ImageLocality
            weight: 11
    containerSecurityContext:
      runAsUser: 65534 # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    image:
      # IMPORTANT: Bumping the minor version of this binary should go hand in
      #            hand with an inspection of the user-scheduelrs RBAC resources
      #            that we have forked.
      name: k8s.gcr.io/kube-scheduler
      tag: v1.19.13 # ref: https://github.com/kubernetes/website/blob/7a5dc9eb8d2df8c41f52280c3010dbc2bce19523/content/en/releases/patch-releases.md
      pullPolicy:
      pullSecrets: []
    nodeSelector: {}
    tolerations: []
    pdb:
      enabled: true
      maxUnavailable: 1
      minAvailable:
    resources: {}
    serviceAccount:
      annotations: {}
    extraPodSpec: {}
  podPriority:
    enabled: false
    globalDefault: false
    defaultPriority: 0
    userPlaceholderPriority: -10
  userPlaceholder:
    enabled: true
    image:
      name: k8s.gcr.io/pause
      # tag's can be updated by inspecting the output of the command:
      # gcloud container images list-tags k8s.gcr.io/pause --sort-by=~tags
      #
      # If you update this, also update prePuller.pause.image.tag
      tag: "3.5"
      pullPolicy:
      pullSecrets: []
    replicas: 0
    containerSecurityContext:
      runAsUser: 65534 # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    resources: {}
  corePods:
    tolerations:
      - key: hub.jupyter.org/dedicated
        operator: Equal
        value: core
        effect: NoSchedule
      - key: hub.jupyter.org_dedicated
        operator: Equal
        value: core
        effect: NoSchedule
    nodeAffinity:
      matchNodePurpose: prefer
  userPods:
    tolerations:
      - key: hub.jupyter.org/dedicated
        operator: Equal
        value: user
        effect: NoSchedule
      - key: hub.jupyter.org_dedicated
        operator: Equal
        value: user
        effect: NoSchedule
    nodeAffinity:
      matchNodePurpose: prefer

# prePuller relates to the hook|continuous-image-puller DaemonsSets
prePuller:
  annotations: {cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'}
  resources: {}
  containerSecurityContext:
    runAsUser: 65534 # nobody user
    runAsGroup: 65534 # nobody group
    allowPrivilegeEscalation: false
  extraTolerations: []
  # hook relates to the hook-image-awaiter Job and hook-image-puller DaemonSet
  hook:
    enabled: true
    pullOnlyOnChanges: true
    # image and the configuration below relates to the hook-image-awaiter Job
    image:
      name: jupyterhub/k8s-image-awaiter
      tag: "1.2.0"
      pullPolicy:
      pullSecrets: []
    containerSecurityContext:
      runAsUser: 65534 # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    podSchedulingWaitDuration: 10
    nodeSelector: {}
    tolerations: []
    resources: {}
    serviceAccount: 
      annotations: {}
  continuous:
    enabled: true
  pullProfileListImages: true
  extraImages: {}
  pause:
    containerSecurityContext:
      runAsUser: 65534 # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    image:
      name: k8s.gcr.io/pause
      # tag's can be updated by inspecting the output of the command:
      # gcloud container images list-tags k8s.gcr.io/pause --sort-by=~tags
      #
      # If you update this, also update scheduling.userPlaceholder.image.tag
      tag: "3.5"
      pullPolicy:
      pullSecrets: []

ingress:
  enabled: false
  annotations: {cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'}
  hosts: []
  pathSuffix:
  pathType: Prefix
  tls: []

# cull relates to the jupyterhub-idle-culler service, responsible for evicting
# inactive singleuser pods.
#
# The configuration below, except for enabled, corresponds to command-line flags
# for jupyterhub-idle-culler as documented here:
# https://github.com/jupyterhub/jupyterhub-idle-culler#as-a-standalone-script
#
cull:
  enabled: true
  users: false # --cull-users
  removeNamedServers: false # --remove-named-servers
  timeout: 3600 # --timeout
  every: 600 # --cull-every
  concurrency: 10 # --concurrency
  maxAge: 0 # --max-age

debug:
  enabled: false

global:
  safeToShowValues: false

Here is my issue:

Hi! Please could you

  • turn on debug logging and show us your hub and singleuser logs
  • show us the relevant changed parts of your configuration (unfortunately showing the whole file makes it very difficult to see what you’ve customised)

Hi @manics, thanks for responding me,
Here is my singleuser logs:

[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.703 SingleUserNotebookApp manager:345] nbgitpuller | extension was successfully linked.
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.703 SingleUserNotebookApp manager:345] nbclassic | extension was successfully linked.
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.727 SingleUserNotebookApp manager:367] nbclassic | extension was successfully loaded.
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.729 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.729 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.732 SingleUserNotebookApp manager:367] jupyterlab | extension was successfully loaded.
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.732 SingleUserNotebookApp manager:367] nbgitpuller | extension was successfully loaded.
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.733 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.2
[jupyter-chrys notebook]  [D 2022-05-24 06:58:31.741 SingleUserNotebookApp _version:76] jupyterhub and jupyterhub-singleuser both on version 1.4.2
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.741 SingleUserNotebookApp serverapp:2577] Serving notebooks from local directory: /home/jovyan
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.741 SingleUserNotebookApp serverapp:2577] Jupyter Server 1.11.1 is running at:
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.741 SingleUserNotebookApp serverapp:2577] http://jupyter-chrys:8888/user/chrys/lab
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.741 SingleUserNotebookApp serverapp:2577]  or http://127.0.0.1:8888/user/chrys/lab
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.741 SingleUserNotebookApp serverapp:2578] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[jupyter-chrys notebook]  [I 2022-05-24 06:58:31.744 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds
[jupyter-chrys notebook]  [D 2022-05-24 06:58:31.744 SingleUserNotebookApp mixins:518] Notifying Hub of activity 2022-05-24T06:58:31.712299Z
[jupyter-chrys notebook]  [C 2022-05-24 06:58:47.475 SingleUserNotebookApp serverapp:2033] received signal 15, stopping
[jupyter-chrys notebook]  [I 2022-05-24 06:58:47.476 SingleUserNotebookApp serverapp:2368] Shutting down 3 extensions
[jupyter-chrys notebook]  [D 2022-05-24 06:58:47.476 SingleUserNotebookApp manager:372] jupyterlab | extension app "lab" stopping
[jupyter-chrys notebook]  [D 2022-05-24 06:58:47.476 SingleUserNotebookApp manager:374] jupyterlab | extension app "lab" stopped
[jupyter-chrys notebook]  [D 2022-05-24 06:58:47.476 SingleUserNotebookApp manager:372] nbclassic | extension app "notebook" stopping
[jupyter-chrys notebook]  [D 2022-05-24 06:58:47.476 SingleUserNotebookApp manager:374] nbclassic | extension app "notebook" stopped
[jupyter-chrys notebook]  [I 2022-05-24 06:58:47.476 SingleUserNotebookApp serverapp:2342] Shutting down 0 kernels
[jupyter-chrys notebook]  [I 2022-05-24 06:58:47.476 SingleUserNotebookApp serverapp:2359] Shutting down 0 terminals

Here is my hub logs:

[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:51.040 JupyterHub log:189] 200 GET /hub/health (@10.0.38.1) 0.80ms
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:53.040 JupyterHub log:189] 200 GET /hub/health (@10.0.38.1) 1.05ms
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:53.041 JupyterHub log:189] 200 GET /hub/health (@10.0.38.1) 1.36ms
[jupyter-hub-75bddd5768-tn9p4 hub]  [E 2022-05-24 07:01:53.520 JupyterHub user:718] Unhandled error starting chrys's server: Could not create pod mantis-review/jupyter-chrys
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:53.520 JupyterHub user:819] Stopping chrys
[jupyter-hub-75bddd5768-tn9p4 hub]  [W 2022-05-24 07:01:53.521 JupyterHub spawner:2103] Pod mantis-review/jupyter-chrys url changed! jupyter-hub-75bddd5768-tn9p4:51383 -> 10.0.39.112:8888
[jupyter-hub-75bddd5768-tn9p4 hub]  [I 2022-05-24 07:01:53.527 JupyterHub spawner:2620] Deleting pod mantis-review/jupyter-chrys
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:55.040 JupyterHub log:189] 200 GET /hub/health (@10.0.38.1) 0.76ms
[jupyter-hub-75bddd5768-tn9p4 hub]  [E 2022-05-24 07:01:55.550 JupyterHub gen:623] Exception in Future <Future finished exception=ApiException()> after timeout
[jupyter-hub-75bddd5768-tn9p4 hub]      Traceback (most recent call last):
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
[jupyter-hub-75bddd5768-tn9p4 hub]          future.result()
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
[jupyter-hub-75bddd5768-tn9p4 hub]          result = self.fn(*self.args, **self.kwargs)
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/kubespawner/spawner.py", line 2117, in asynchronize
[jupyter-hub-75bddd5768-tn9p4 hub]          return method(*args, **kwargs)
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api/core_v1_api.py", line 7320, in create_namespaced_pod
[jupyter-hub-75bddd5768-tn9p4 hub]          return self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)  # noqa: E501
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api/core_v1_api.py", line 7415, in create_namespaced_pod_with_http_info
[jupyter-hub-75bddd5768-tn9p4 hub]          return self.api_client.call_api(
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py", line 348, in call_api
[jupyter-hub-75bddd5768-tn9p4 hub]          return self.__call_api(resource_path, method,
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py", line 180, in __call_api
[jupyter-hub-75bddd5768-tn9p4 hub]          response_data = self.request(
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/api_client.py", line 391, in request
[jupyter-hub-75bddd5768-tn9p4 hub]          return self.rest_client.POST(url,
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/rest.py", line 274, in POST
[jupyter-hub-75bddd5768-tn9p4 hub]          return self.request("POST", url,
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/kubernetes/client/rest.py", line 233, in request
[jupyter-hub-75bddd5768-tn9p4 hub]          raise ApiException(http_resp=r)
[jupyter-hub-75bddd5768-tn9p4 hub]      kubernetes.client.exceptions.ApiException: (409)
[jupyter-hub-75bddd5768-tn9p4 hub]      Reason: Conflict
[jupyter-hub-75bddd5768-tn9p4 hub]      HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e9299912-1840-41d5-b638-8956dfd7bf78', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '36fb01dc-4a85-4e0e-8b80-f9ecbc144d31', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'c7e670eb-38f1-4eb1-8fc0-dadeebef9de1', 'Date': 'Tue, 24 May 2022 07:01:55 GMT', 'Content-Length': '229'})
[jupyter-hub-75bddd5768-tn9p4 hub]      HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"object is being deleted: pods \"jupyter-chrys\" already exists","reason":"AlreadyExists","details":{"name":"jupyter-chrys","kind":"pods"},"code":409}
[jupyter-hub-75bddd5768-tn9p4 hub]      
[jupyter-hub-75bddd5768-tn9p4 hub]      
[jupyter-hub-75bddd5768-tn9p4 hub]      
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:57.040 JupyterHub log:189] 200 GET /hub/health (@10.0.38.1) 0.74ms
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:58.109 JupyterHub reflector:357] pods watcher timeout
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:58.109 JupyterHub reflector:282] Connecting pods watcher
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:58.632 JupyterHub reflector:357] events watcher timeout
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:58.632 JupyterHub reflector:282] Connecting events watcher
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:01:59.039 JupyterHub log:189] 200 GET /hub/health (@10.0.38.1) 0.75ms
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:02:01.040 JupyterHub log:189] 200 GET /hub/health (@10.0.38.1) 0.87ms
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:02:03.040 JupyterHub log:189] 200 GET /hub/health (@10.0.38.1) 1.11ms
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:02:03.040 JupyterHub log:189] 200 GET /hub/health (@10.0.38.1) 1.08ms
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:02:04.984 JupyterHub user:845] Deleting oauth client jupyterhub-user-chrys
[jupyter-hub-75bddd5768-tn9p4 hub]  [D 2022-05-24 07:02:04.995 JupyterHub user:848] Finished stopping chrys
[jupyter-hub-75bddd5768-tn9p4 hub]  [W 2022-05-24 07:02:05.001 JupyterHub base:976] 2 consecutive spawns failed.  Hub will exit if failure count reaches 5 before succeeding
[jupyter-hub-75bddd5768-tn9p4 hub]  [E 2022-05-24 07:02:05.001 JupyterHub gen:623] Exception in Future <Task finished name='Task-629' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:900> exception=TimeoutError('Could not create pod mantis-review/jupyter-chrys')> after timeout
[jupyter-hub-75bddd5768-tn9p4 hub]      Traceback (most recent call last):
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
[jupyter-hub-75bddd5768-tn9p4 hub]          future.result()
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
[jupyter-hub-75bddd5768-tn9p4 hub]          await spawn_future
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 736, in spawn
[jupyter-hub-75bddd5768-tn9p4 hub]          raise e
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 635, in spawn
[jupyter-hub-75bddd5768-tn9p4 hub]          url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/kubespawner/spawner.py", line 2508, in _start
[jupyter-hub-75bddd5768-tn9p4 hub]          await exponential_backoff(
[jupyter-hub-75bddd5768-tn9p4 hub]        File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 184, in exponential_backoff
[jupyter-hub-75bddd5768-tn9p4 hub]          raise TimeoutError(fail_message)
[jupyter-hub-75bddd5768-tn9p4 hub]      TimeoutError: Could not create pod mantis-review/jupyter-chrys

Here are some part of config customization:

  profileList:
    - display_name: "Default environment"
      description: "Image de base de jupyter-lab"
      default: true
    - display_name: "Wiremind mantis"
      description: "Wiremind image"
      kubespawner_override:
        image: docker.wiremind.fr/wiremind/data-science/mantis/cpu-jupyter/master:latest
  pullSecrets: 
      - "gitlab-docker-registry"
hub:
  config:
    Authenticator:
      admin_users:
        - chrys
    JupyterHub:
      admin_access: true
      authenticator_class: dummy
  storage:
    type: dynamic
    extraLabels: {}
    extraVolumes: []
    extraVolumeMounts: []
    static:
      pvcName:
      subPath: "{username}"
    capacity: 20Gi
    #homeMountPath: /home/jovyan
    homeMountPath: /home/{username}

I haven’t seen that error before. What version of Z2JH are you using?

I use the version 1.2.0 I tried also the version 1.1.4 but the issue was the same

Please if you have time, we can do a call.

1 Like