Need help to configure notebook - right error?

JupyterHub
, ,
1

Hi everyone,
I’m stuck on a little problem with my Jupyterhub hosted on Kubernetes and deployed my Helm Chart.
I use JupyterHub 2.0.0 from here: hub.jupyter - helm-chart
And I use jupyter minimal-notebook (2022-11-21) from here : quay.io

Symtpomes
When user try to start it’s server (By going on JupyterHub HUB webpage and click on “Start my server”), server never start and I have CrashLoopBack, when I check container of my pod, I see the Image of Minimal-Notebook in state of Error and the logs says <<PermissionError: [Errno 13] Permission denied: ‘/home/jovyan/.local’>>

What I tried

  1. My error look like this topic : https://github.com/jupyter/docker-stacks/issues/1003
    I tried to start the image with the solution on the topic :
docker run --user root -v new-longhorn-volume:/home/jovyan \
  -e CHOWN_HOME=yes -e CHOWN_HOME_OPTS='-R' \
  -it --rm -p 8888:8888 jupyter/minimal-notebook:2022-11-21

Commit new image named “minimal-notebook-2022-11-21-custom” and import this image on my Harbor Registry, now I use the custom image but the error persist.

  1. I see singleuser.allowPrivilegeEscalation in jupyterhub_config.py
    I added this on my yaml :
singleuser:
  cmd: ["start-singleuser.sh"]
  image:
    name: jupyter/minimal-notebook
    tag: "2022-11-21-custom"
  allowPrivilegeEscalation: true
  1. I tried to put “notebook-minimal” to sleep for debuging, by adding this in my yaml :
singleuser:
  cmd: ["start-singleuser.sh"]
  image:
    name: jupyter/minimal-notebook
    tag: "2022-11-21-custom"
  command:    
      - /bin/bash
      - "-c"
      - "sleep infinity"

But not taken into account, maybe a chart problem on my side (I’m looking into it)
In “values.schema.json” I added this :

    "singleuser": {
      "type": "object",
      "additionalProperties": true,
      "properties": {
        "command": {
          "type": "array"
        },
       }
    },

If someone have an idea or a suggestion…
In advance, I’m sorry for my bad english

2

These are runtime options:
https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#permission-specific-configurations

They may or may not have an effect when embedded in built images so you’ll need to pass them in the spawner environment:

The 2022-11-21 image is very old, can you try a more recent version, e.g. 2024-02-21

3

Hi @manics,
thanks for your feedback ! And sorry for my late reply.
The topic moves forward :slight_smile:

With this solution :

These are runtime options:
Common Features — Docker Stacks documentation

They may or may not have an effect when embedded in built images so you’ll need to pass them in the spawner environment:

I’m able to pass argument in the container.

Now I got this following error :

Warning  Failed                  <invalid> (x3 over <invalid>)  kubelet                    Error: failed to create containerd task: failed to create shim t 
ask: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/026 
9a99d-edce-4fde-8a69-fab2f908290f/volume-subpaths/tmp-notebook-groups/notebook/7" to rootfs at "/etc/group": mount /var/lib/kubelet/pods/0269a99d-edce-4fde-8
a69-fab2f908290f/volume-subpaths/tmp-notebook-groups/notebook/7:/etc/group (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown

Probably a mount problem on my part, I dig on my own. I’ll compile a list of solutions.

About this last point :

The 2022-11-21 image is very old, can you try a more recent version, e.g. 2024-02-21

Unfortunately, my cluster is old and we cannot upgrade now… I tried with the last version of JupyterHub on my Kubernetes version 1.25 and I’ve never been able to start it. We have to upgrade Kubernetes cluster first.

5

Hi there,
topic moves forward :slight_smile: but still have some problem…

So, about this error :

<<PermissionError: [Errno 13] Permission denied: ‘/home/jovyan/.local’>>

@manics I used your solution about kubespawner_override :

singleuser:
  image:
    name: quay.io/jupyter/minimal-notebook
    tag: "2024-02-21"
  profileList:
    - display_name: "abcd"
      description: "Test"
      default: true
      kubespawner_override:
        extraEnv:
          CHOWN_HOME: 'yes'
          CHOWN_HOME_OPTS: '-R' 
          CHOWN_EXTRA: "/home/jovyan/"
          CHOWN_EXTRA_OPTS: '-R'
          GRANT_SUDO: "yes"
        uid: 0
        cmd : ['start-singleuser.sh'] 
        allow_privilege_escalation: true

About the error :

OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting

It was error in my side, I haven’t configured it properly my configmap. Now it’s back.

The container look like start with root user and now everything looking fine :

Entered start.sh with args: start-singleuser.sh                                                                                                               
Running hooks in: /usr/local/bin/start-notebook.d as uid: 0 gid: 2001                                                                                         
Done running hooks in: /usr/local/bin/start-notebook.d                                                                                                        
Updated the jovyan user:                                                                                                                                      
- username: jovyan       -> jdomper                                                                                                                           
- home dir: /home/jovyan -> /home/jdomper                                                                                                                     
Update jdomper's UID:GID to 1139:2001                                                                                                                         
id: ‘jdomper’: no such user                                                                                                                                   
id: ‘jdomper’: no such user                                                                                                                                   
groupadd: cannot open /etc/group                                                                                                                              
Stream closed EOF for jupyterhub/jupyter-jdomper (notebook)

As you can see, container didn’t find the user… Do I create a new topic?
Thanks :).

6

Can you try simplifying your configuration to the minimal required? Get rid of profileList and kubespawner_override, just have a single image and config. This will help with debugging.

Can you also show us your full Z2JH config? Which version of Z2JH are you using?

1 Like