Custom docker image fails to run "Spawn Failed"

JupyterHub Zero to JupyterHub on Kubernetes
1

I am trying to get a basic custom docker image built with repo2docker to run in a multi environment configuration (also doesn’t work in a single image configuration either). The docker image has jupyterhub installed and it is the same version as the host install.

I’ve tried multiple versions modes of configuring the images under single user, but they all load the example image except for one which returns “spawn failed” after 30s timeout. Can someone help me get this working? Here’s some more information.

su.yaml

singleuser:
  profileList:
    - display_name: "Custom Jupyter Environment"
      description: "Test custom environment"
      kuberspawner_override:
        image:  org_nameinfra/org_name_elg7186:latest
        tag: latest
    - display_name: "Datascience environment"
      description: "Python, R, and Julia."
      kubespawner_override:
        image: jupyter/datascience-notebook:2343e33dec46
    - display_name: "Spark environment"
      description: "The Jupyter Stacks spark image"
      kubespawner_override:
        image: jupyter/all-spark-notebook:2343e33dec46

Event Log

Server requested
2021-08-05T04:02:55.440462Z [Normal] Successfully assigned jhub-staging/jupyter-my_username to aks-nodepool1-12473058-vmss000000
2021-08-05T04:03:29Z [Normal] AttachVolume.Attach succeeded for volume "pvc-e5ea889e-2b16-48d1-a885-07092ba776d9"
2021-08-05T04:03:59Z [Normal] Container image "jupyterhub/k8s-network-tools:0.11.1" already present on machine
2021-08-05T04:03:59Z [Normal] Created container block-cloud-metadata
2021-08-05T04:03:59Z [Normal] Started container block-cloud-metadata
2021-08-05T04:04:00Z [Normal] Pulling image "org_nameinfra/org_name_elg7186:latest"
2021-08-05T04:04:00Z [Normal] Successfully pulled image "org_nameinfra/org_name_elg7186:latest" in 279.796643ms
2021-08-05T04:04:00Z [Normal] Created container notebook
2021-08-05T04:04:00Z [Normal] Started container notebook
2021-08-05T04:04:02Z [Normal] Successfully pulled image "org_nameinfra/org_name_elg7186:latest" in 194.467572ms
Spawn failed: Server at http://10.240.0.5:8888/user/my_username/ didn't respond in 30 seconds

Hub log:

No config at /etc/jupyterhub/config/values.yaml
Loading /etc/jupyterhub/secret/values.yaml
Loading extra config: templates
[I 2021-08-05 04:02:11.510 JupyterHub app:2349] Running JupyterHub version 1.3.0
[I 2021-08-05 04:02:11.511 JupyterHub app:2379] Using Authenticator: oauthenticator.github.GitHubOAuthenticator-0.12.3
[I 2021-08-05 04:02:11.511 JupyterHub app:2379] Using Spawner: kubespawner.spawner.KubeSpawner-0.15.0
[I 2021-08-05 04:02:11.511 JupyterHub app:2379] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.3.0
[I 2021-08-05 04:02:11.512 JupyterHub app:1420] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[W 2021-08-05 04:02:11.547 JupyterHub app:1695] No admin users, admin interface will be unavailable.
[W 2021-08-05 04:02:11.547 JupyterHub app:1696] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-08-05 04:02:11.548 JupyterHub app:1725] Not using allowed_users. Any authenticated user will be allowed.
[I 2021-08-05 04:02:11.670 JupyterHub reflector:201] watching for pods with label selector='component=singleuser-server' in namespace jhub-staging
[I 2021-08-05 04:02:11.757 JupyterHub reflector:201] watching for events with field selector='involvedObject.kind=Pod' in namespace jhub-staging
[W 2021-08-05 04:02:11.774 JupyterHub _version:70] jupyterhub version 1.3.0 != jupyterhub-singleuser version 0.9.4. This could cause failure to authenticate and result in redirect loops!
[I 2021-08-05 04:02:11.774 JupyterHub app:2077] org_name-infra still running
[I 2021-08-05 04:02:11.778 JupyterHub app:2077] my_username still running
[I 2021-08-05 04:02:11.778 JupyterHub app:2416] Initialized 2 spawners in 0.167 seconds
[I 2021-08-05 04:02:11.781 JupyterHub app:2628] Not starting proxy
[I 2021-08-05 04:02:11.790 JupyterHub app:2664] Hub API listening on http://:8081/hub/
[I 2021-08-05 04:02:11.790 JupyterHub app:2666] Private Hub API connect url http://hub:8081/hub/
[I 2021-08-05 04:02:11.790 JupyterHub app:2679] Starting managed service cull-idle
[I 2021-08-05 04:02:11.791 JupyterHub service:339] Starting service 'cull-idle': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2021-08-05 04:02:11.794 JupyterHub service:121] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[I 2021-08-05 04:02:11.824 JupyterHub proxy:319] Checking routes
[I 2021-08-05 04:02:11.824 JupyterHub proxy:399] Adding default route for Hub: / => http://hub:8081
[W 2021-08-05 04:02:11.827 JupyterHub proxy:343] Adding missing route for /user/org_name-infra/ (Server(url=http://10.240.0.12:8888/user/org_name-infra/, bind_url=http://10.240.0.12:8888/user/org_name-infra/))
[W 2021-08-05 04:02:11.827 JupyterHub proxy:343] Adding missing route for /user/my_username/ (Server(url=http://10.240.0.16:8888/user/my_username/, bind_url=http://10.240.0.16:8888/user/my_username/))
[I 2021-08-05 04:02:11.828 JupyterHub proxy:257] Adding user org_name-infra to proxy /user/org_name-infra/ => http://10.240.0.12:8888
[I 2021-08-05 04:02:11.828 JupyterHub proxy:257] Adding user my_username to proxy /user/my_username/ => http://10.240.0.16:8888
[I 2021-08-05 04:02:11.839 JupyterHub app:2739] JupyterHub is now running at http://:8000
[I 2021-08-05 04:02:12.029 JupyterHub log:181] 200 GET /hub/api/users (cull-idle@::1) 26.22ms
[I 2021-08-05 04:02:33.555 JupyterHub log:181] 200 GET /hub/api/authorizations/token/[secret] (my_username@10.240.0.16) 25.93ms
[I 2021-08-05 04:02:35.555 JupyterHub log:181] 200 GET /hub/home (my_username@10.240.0.4) 38.30ms
[I 2021-08-05 04:02:36.708 JupyterHub proxy:281] Removing user my_username from proxy (/user/my_username/)
[I 2021-08-05 04:02:36.711 JupyterHub spawner:1997] Deleting pod jupyter-my_username
[I 2021-08-05 04:02:40.295 JupyterHub log:181] 202 DELETE /hub/api/users/my_username/server (my_username@10.240.0.4) 3.89ms
[I 2021-08-05 04:02:41.184 JupyterHub log:181] 200 GET /hub/error/503?url=%2Fhub%2Fapi%2Fusers%2Fmy_username%2Fserver (@10.240.0.28) 8.74ms
[I 2021-08-05 04:02:41.189 JupyterHub pages:167] Server my_username is already active
[I 2021-08-05 04:02:41.190 JupyterHub log:181] 302 GET /hub/spawn/my_username -> /hub/spawn-pending/my_username (my_username@10.240.0.4) 3.80ms
[I 2021-08-05 04:02:41.237 JupyterHub pages:402] my_username is pending stop
[I 2021-08-05 04:02:41.242 JupyterHub log:181] 200 GET /hub/spawn-pending/my_username (my_username@10.240.0.4) 7.27ms
[I 2021-08-05 04:02:46.359 JupyterHub pages:402] my_username is pending stop
[I 2021-08-05 04:02:46.360 JupyterHub log:181] 200 GET /hub/spawn-pending/my_username (my_username@10.240.0.4) 3.30ms
[I 2021-08-05 04:02:46.618 JupyterHub base:1110] User my_username server took 9.910 seconds to stop
[I 2021-08-05 04:02:46.619 JupyterHub log:181] 204 DELETE /hub/api/users/my_username/server (my_username@10.240.0.4) 9914.83ms
[I 2021-08-05 04:02:51.586 JupyterHub log:181] 200 GET /hub/spawn-pending/my_username (my_username@10.240.0.4) 15.30ms
[I 2021-08-05 04:02:53.234 JupyterHub log:181] 200 GET /hub/spawn/my_username (my_username@10.240.0.4) 8.12ms
[I 2021-08-05 04:02:55.374 JupyterHub spawner:1875] Attempting to create pvc claim-my_username, with timeout 3
[I 2021-08-05 04:02:55.376 JupyterHub log:181] 302 POST /hub/spawn/my_username -> /hub/spawn-pending/my_username (my_username@10.240.0.4) 56.45ms
[I 2021-08-05 04:02:55.400 JupyterHub spawner:1887] PVC claim-my_username already exists, so did not create new pvc.
[I 2021-08-05 04:02:55.403 JupyterHub spawner:1840] Attempting to create pod jupyter-my_username, with timeout 3
[I 2021-08-05 04:02:55.437 JupyterHub pages:402] my_username is pending spawn
[I 2021-08-05 04:02:55.448 JupyterHub log:181] 200 GET /hub/spawn-pending/my_username (my_username@10.240.0.4) 16.62ms
[I 2021-08-05 04:03:11.859 JupyterHub proxy:319] Checking routes
[I 2021-08-05 04:04:11.846 JupyterHub proxy:319] Checking routes
[W 2021-08-05 04:04:30.103 JupyterHub user:738] my_username's server never showed up at http://10.240.0.5:8888/user/my_username/ after 30 seconds. Giving up
[I 2021-08-05 04:04:30.103 JupyterHub spawner:1997] Deleting pod jupyter-my_username
[E 2021-08-05 04:04:30.159 JupyterHub gen:623] Exception in Future <Task finished name='Task-119' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:895> exception=TimeoutError("Server at http://10.240.0.5:8888/user/my_username/ didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
        future.result()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 902, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 719, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 766, in _wait_up
        raise e
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 733, in _wait_up
        resp = await server.wait_up(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 235, in wait_for_http_server
        re = await exponential_backoff(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 183, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://10.240.0.5:8888/user/my_username/ didn't respond in 30 seconds
2

The logs also show a permission denied error as the image is spawing PermissionError: [Errno 13] Permission denied: '/home/org_nameinfra/.local'

This issue has been raised on github here: (repo2docker) Must specify user-id during image build to successfully spawn pods on GKE · Issue #894 · jupyterhub/zero-to-jupyterhub-k8s · GitHub . I have built the image with the user-id flag, and I have built the image on mac and linux platforms, but it hasn’t changed the errors. Also the same errors occur when customizing an existing docker image form the jupyter stacks registry.

Is there a way to delete previously pulled images? During startup the logs will state that the image was pulled in 300-500ms, or it will use existing, but the image is 3GB. When does the image get pulled and where is it stored?

Thanks

tee: /home/org_name/.jupyter-server-log.txt: Permission denied
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/traitlets.py", line 535, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/bin/jupyterhub-singleuser", line 10, in <module>
    sys.exit(main())
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/config/application.py", line 844, in launch_instance
    app.initialize(argv)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/jupyterhub/singleuser/mixins.py", line 554, in initialize
    return super().initialize(argv)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in inner
    return method(app, *args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/notebook/notebookapp.py", line 2082, in initialize
    self.init_configurables()
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/notebook/notebookapp.py", line 1609, in init_configurables
    connection_dir=self.runtime_dir,
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/traitlets.py", line 575, in __get__
    return self.get(obj, cls)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/traitlets.py", line 538, in get
    default = obj.trait_defaults(self.name)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/traitlets.py", line 1578, in trait_defaults
    return self._get_trait_default_generator(names[0])(self)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/jupyter_core/application.py", line 85, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/jupyter_core/utils/__init__.py", line 11, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/srv/conda/envs/notebook/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/srv/conda/envs/notebook/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/srv/conda/envs/notebook/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/srv/conda/envs/notebook/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/org_nameinfra/.local'
3

Hi! Please could you show us your full Z2JH configuration with secrets redacted, and also tell us your version of Z2JH?

Could you also show us the logs from the failed singleuser pod?

Thanks!

4

Thanks manics.

Jupyterhub verison is 1.3.0

Here is the main config merged into a single file.

hub:
  # clone custom JupyterHub templates into a volume
  initContainers:
    - name: git-clone-templates
      image: alpine/git
      args:
        - clone
        - --single-branch
        - --depth=1
        - --
        - https://github.com/github_user/project-homepage.git
        - /srv/repo
      securityContext:
        runAsUser: 1000
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
      volumeMounts:
        - name: custom-templates
          mountPath: /srv/repo
  extraVolumes:
    - name: custom-templates
      emptyDir: {}
  extraVolumeMounts:
    - mountPath: /usr/local/share/jupyterhub/custom_templates
      name: custom-templates
      subPath: "templates"
    - mountPath: /usr/local/share/jupyterhub/static/extra-assets
      name: custom-templates
      subPath: "extra-assets"

  extraConfig:
    templates: |
      c.JupyterHub.template_paths = ['/usr/local/share/jupyterhub/custom_templates/']

hub:
  config:
    GitHubOAuthenticator:
      client_id: <secret>
      client_secret: <secret>
      oauth_callback_url: https://jupyterhub.domain.com/hub/oauth_callback
    #admins not working
    admin:
      users:
        - github_user@domain.com
    JupyterHub:
      authenticator_class: github

proxy:
    https:
      enabled: true 
      hosts:
        - hostname@domain.com
      letsencrypt:
        contactEmail: user@domain.com

singleuser:
  startTimeout: 180
  profileList:
    #- display_name: "Minimal environment"
    #  description: "To avoid too much bells and whistles: Python."
    #  default: true
    - display_name: "Custom Jupyter Environment"
      description: "For use with a course"
      kubespawner_override:
        image:  docker_user/docker_repo
        tag: d33d526 
    - display_name: "Datascience environment"
      description: "Python, R, and Julia."
      kubespawner_override:
        image: jupyter/datascience-notebook:2343e33dec46
    - display_name: "Spark environment"
      description: "The Jupyter Stacks spark image"
      kubespawner_override:
        image: jupyter/all-spark-notebook:2343e33dec46

proxy:
  secretToken: <secret key>

Log from the pod while booting:

tee: /home/user_orginfra/.jupyter-server-log.txt: Permission denied
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/traitlets.py", line 535, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/bin/jupyterhub-singleuser", line 10, in <module>
    sys.exit(main())
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/config/application.py", line 844, in launch_instance
    app.initialize(argv)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/jupyterhub/singleuser/mixins.py", line 554, in initialize
    return super().initialize(argv)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in inner
    return method(app, *args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/notebook/notebookapp.py", line 2082, in initialize
    self.init_configurables()
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/notebook/notebookapp.py", line 1609, in init_configurables
    connection_dir=self.runtime_dir,
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/traitlets.py", line 575, in __get__
    return self.get(obj, cls)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/traitlets.py", line 538, in get
    default = obj.trait_defaults(self.name)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/traitlets/traitlets.py", line 1578, in trait_defaults
    return self._get_trait_default_generator(names[0])(self)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/jupyter_core/application.py", line 85, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/jupyter_core/utils/__init__.py", line 11, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/srv/conda/envs/notebook/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/srv/conda/envs/notebook/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/srv/conda/envs/notebook/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/srv/conda/envs/notebook/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/user_orginfra/.local'

describe from the pod while booting

Name:         jupyter-usename
Namespace:    jhub-staging
Priority:     0
Node:         aks-nodepool1-12473058-vmss000000/10.240.0.4
Start Time:   Sat, 07 Aug 2021 00:42:05 +0000
Labels:       app=jupyterhub
              chart=jupyterhub-0.11.1
              component=singleuser-server
              heritage=jupyterhub
              hub.jupyter.org/network-access-hub=true
              release=jupyterhub
Annotations:  hub.jupyter.org/username: usename
Status:       Running
IP:           10.240.0.26
IPs:
  IP:  10.240.0.26
Init Containers:
  block-cloud-metadata:
    Container ID:  containerd://73d39d31d58224dfa7181959ae9093e06e3283076d46f9621c57e704714cb958
    Image:         jupyterhub/k8s-network-tools:0.11.1
    Image ID:      docker.io/jupyterhub/k8s-network-tools@sha256:e562b1f91469d71e843aeece0500be249344e82154183f082492c7c51a636b52
    Port:          <none>
    Host Port:     <none>
    Command:
      iptables
      -A
      OUTPUT
      -d
      169.254.169.254
      -j
      DROP
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 07 Aug 2021 00:42:23 +0000
      Finished:     Sat, 07 Aug 2021 00:42:23 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:         <none>
Containers:
  notebook:
    Container ID:  containerd://3d9d589a463067edd91f555408dcdc5927adb1996bbd839079ab3126db1be591
    Image:         user_orginfra/user_org_elg7186
    Image ID:      docker.io/user_orginfra/user_org_elg7186@sha256:6955299f680754054c76143773314d61ff02e6cc2fc6ecf09dd7b2ee9d563011
    Port:          8888/TCP
    Host Port:     0/TCP
    Args:
      jupyterhub-singleuser
      --ip=0.0.0.0
      --port=8888
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 07 Aug 2021 00:43:56 +0000
      Finished:     Sat, 07 Aug 2021 00:43:57 +0000
    Ready:          False
    Restart Count:  4
    Requests:
      memory:  1073741824
    Environment:
      JUPYTERHUB_API_TOKEN:           6c9ccd667e97453a9ec70fcb515aaca1
      JPY_API_TOKEN:                  6c9ccd667e97453a9ec70fcb515aaca1
      JUPYTERHUB_ADMIN_ACCESS:        1
      JUPYTERHUB_CLIENT_ID:           jupyterhub-user-usename
      JUPYTERHUB_HOST:                
      JUPYTERHUB_OAUTH_CALLBACK_URL:  /user/usename/oauth_callback
      JUPYTERHUB_USER:                usename
      JUPYTERHUB_SERVER_NAME:         
      JUPYTERHUB_API_URL:             http://hub:8081/hub/api
      JUPYTERHUB_ACTIVITY_URL:        http://hub:8081/hub/api/users/usename/activity
      JUPYTERHUB_BASE_URL:            /
      JUPYTERHUB_SERVICE_PREFIX:      /user/usename/
      MEM_GUARANTEE:                  1073741824
      JUPYTER_IMAGE_SPEC:             user_orginfra/user_org_elg7186
      JUPYTER_IMAGE:                  user_orginfra/user_org_elg7186
    Mounts:
      /home/jovyan from volume-usename (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  volume-usename:
    Type:        PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:   claim-usename
    ReadOnly:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     hub.jupyter.org/dedicated=user:NoSchedule
                 hub.jupyter.org_dedicated=user:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                 From                       Message
  ----     ------                  ----                ----                       -------
  Normal   Scheduled               2m4s                jupyterhub-user-scheduler  Successfully assigned jhub-staging/jupyter-usename to aks-nodepool1-12473058-vmss000000
  Normal   SuccessfulAttachVolume  113s                attachdetach-controller    AttachVolume.Attach succeeded for volume "pvc-a6acbf35-750e-4cf7-bb33-5575bc5a595f"
  Normal   Pulled                  107s                kubelet                    Container image "jupyterhub/k8s-network-tools:0.11.1" already present on machine
  Normal   Created                 107s                kubelet                    Created container block-cloud-metadata
  Normal   Started                 106s                kubelet                    Started container block-cloud-metadata
  Normal   Pulled                  105s                kubelet                    Successfully pulled image "user_orginfra/user_org_elg7186" in 375.612517ms
  Normal   Pulled                  102s                kubelet                    Successfully pulled image "user_orginfra/user_org_elg7186" in 274.096739ms
  Normal   Pulled                  85s                 kubelet                    Successfully pulled image "user_orginfra/user_org_elg7186" in 260.161233ms
  Normal   Pulling                 57s (x4 over 105s)  kubelet                    Pulling image "user_orginfra/user_org_elg7186"
  Normal   Created                 56s (x4 over 104s)  kubelet                    Created container notebook
  Normal   Started                 56s (x4 over 104s)  kubelet                    Started container notebook
  Normal   Pulled                  56s                 kubelet                    Successfully pulled image "user_orginfra/user_org_elg7186" in 276.882771ms
  Warning  BackOff                 26s (x6 over 100s)  kubelet                    Back-off restarting failed container
5

It sounds like the permissions on the /home/user_orginfra/ directory inside your docker image is incorrect. What are they set to currently?

Is there a reason you’re using /home/user_orginfra instead of the default /home/jovyan? Your Z2JH configuration doesn’t mount any volumes there, so nothing will be persistent.

6

Hmm. The user “user_orginfra” is the userid of the oauth (github) user launching the image after logging in. /home/jovyan is the default user the hub pod though. BTW this is running on Azure.