Boil custom packages in Jupyterhub

rdmnf · September 18, 2025, 11:22am

Hi, I’m aiming at pre installing python packages for my JH Kubernetes deployment.

Could you guide me to documentation or some guide on doing it?

Thank you,

Paul2708 · September 19, 2025, 7:00am

Since you are using Kubernetes, you have to build a custom Docker image, most likely based on one of the official Jupyter Docker images. For example:

# Make sure to match your JupyterHub application version
FROM quay.io/jupyter/datascience-notebook:hub-5.2.1

USER root

# Install OS packages, dependencies, packages, ...
# For example:
RUN pip install jupyter-ai[all]

USER jovyan

After building and probaly pushing your image to a public or private Docker image registry (e.g., DockerHub, GitHub CR, …), you have to specify the Docker image in your Helm config:

rdmnf · September 19, 2025, 11:52am

Thank you! I created one and I wrote the name of the image in singleuser.image.name in the value.yaml of the helm chart

FROM quay.io/jupyter/base-notebook:latest

USER root

# Install OS packages, dependencies, packages, ...
# For example:
#RUN pip install jupyter-ai[all]
RUN pip install --no-cache-dir minio pandas

USER jovyan

But the deployment does not succeed; the hook-image-puller pods do not run.

I tried running the image as a single image, it throws the following error:

> k logs jhminio

exec /usr/bin/tini: exec format error

Looking forward for some hints.

Paul2708 · September 19, 2025, 4:15pm

Can you please ensure that you are running the correct image? I just built your image myself, and it’s working just fine.

rdmnf · September 20, 2025, 7:48pm

Thanks! this is how I call this image in the Helm chart:
the image is rdmnf/jh-minio:0.0.1 . the image is public so you can pull it, too.

it should be fine.

these are the status of the pods:

k get pod -n jh-oidc

NAME READY STATUS RESTARTS AGE

hook-image-awaiter-gvr25   1/1     Running                 0             55s

hook-image-puller-2b95d    0/1     Init:Error              3 (39s ago)   57s

hook-image-puller-7bhtt    0/1     Init:Error              3 (42s ago)   57s

hook-image-puller-c6pkg    0/1     Init:Error              3 (39s ago)   57s

hook-image-puller-pzx6j    0/1     Init:CrashLoopBackOff   3 (15s ago)   57s

hook-image-puller-sctkn    0/1     Init:CrashLoopBackOff   3 (18s ago)   57s

hook-image-puller-thzvb    0/1     Init:CrashLoopBackOff   3 (15s ago)   57s

Paul2708 · September 22, 2025, 7:10am

Can you check the error using the following command?

kubectl describe pod hook-image-puller-<id>

Otherwise, you can try to disable the hook image puller and run the image right away.

rdmnf · September 22, 2025, 8:01am

please find the k describe:

Name: hook-image-puller-74t2b

Namespace: jh-oidc

Priority: 0

Service Account: hook-image-puller

Node: fr-tlswrk01/192.168.120.184

Start Time: Mon, 22 Sep 2025 10:00:25 +0200

Labels: app=jupyterhub
              app.kubernetes.io/component=hook-image-puller

              app.kubernetes.io/instance=jupyterhub

              app.kubernetes.io/name=jupyterhub

              component=hook-image-puller

              controller-revision-hash=866fbfc774

              pod-template-generation=1

              release=jupyterhub
Annotations:

Status: Pending

IP: 10.244.5.250

IPs:

IP: 10.244.5.250

Controlled By: DaemonSet/hook-image-puller

Init Containers:

image-pull-metadata-block:
Container ID:    containerd://128de4141a97d8c7ba848b0b62df2750b6960995d158c4f94d310305d0e32f70

Image:           quay.io/jupyterhub/k8s-network-tools:4.1.0

Image ID:        quay.io/jupyterhub/k8s-network-tools@sha256:6e2f52383d13905345a3a46ecbd0680e685b1eebe881e30bb170aa480115e906

Port:            <none>

Host Port:       <none>

SeccompProfile:  RuntimeDefault

Command:

  /bin/sh

  -c

  echo "Pulling complete"

State:          Terminated

  Reason:       Completed

  Exit Code:    0

  Started:      Mon, 22 Sep 2025 10:00:26 +0200

  Finished:     Mon, 22 Sep 2025 10:00:26 +0200

Ready:          True

Restart Count:  0

Environment:    <none>

Mounts:         <none>
image-pull-singleuser:
Container ID:    containerd://28dfd6b5125c9fe337aafe03dfc133d0e38f78abd11552e3b220251388b7a175

Image:           rdmnf/jh-minio:0.0.1

Image ID:        docker.io/rdmnf/jh-minio@sha256:41a7f52e27bb30ad093140f85f81f901263b9edd7aa739aa5da48549bbf2185d

Port:            <none>

Host Port:       <none>

SeccompProfile:  RuntimeDefault

Command:

  /bin/sh

  -c

  echo "Pulling complete"

State:          Waiting

  Reason:       CrashLoopBackOff

Last State:     Terminated

  Reason:       Error

  Exit Code:    255

  Started:      Mon, 22 Sep 2025 10:00:44 +0200

  Finished:     Mon, 22 Sep 2025 10:00:44 +0200

Ready:          False

Restart Count:  2

Environment:    <none>

Mounts:         <none>
Containers:

pause:
Container ID:    

Image:           registry.k8s.io/pause:3.10

Image ID:        

Port:            <none>

Host Port:       <none>

SeccompProfile:  RuntimeDefault

State:           Waiting

  Reason:        PodInitializing

Ready:           False

Restart Count:   0

Environment:     <none>

Mounts:          <none>
Conditions:

Type Status

PodReadyToStartContainers True

Initialized False

Ready False

ContainersReady False

PodScheduled True

Volumes:

QoS Class: BestEffort

Node-Selectors:

Tolerations: hub.jupyter.org/dedicated=user:NoSchedule
                          hub.jupyter.org_dedicated=user:NoSchedule

                          node.kubernetes.io/disk-pressure:NoSchedule op=Exists

                          node.kubernetes.io/memory-pressure:NoSchedule op=Exists

                          node.kubernetes.io/not-ready:NoExecute op=Exists

                          node.kubernetes.io/pid-pressure:NoSchedule op=Exists

                          node.kubernetes.io/unreachable:NoExecute op=Exists

                          node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:

Type Reason Age From Message

Normal Scheduled 33s default-scheduler Successfully assigned jh-oidc/hook-image-puller-74t2b to fr-tlswrk01

Normal Pulled 33s kubelet Container image “Quay” already present on machine

Normal Created 33s kubelet Created container image-pull-metadata-block

Normal Started 33s kubelet Started container image-pull-metadata-block

Normal Pulled 15s (x3 over 32s) kubelet Container image “rdmnf/jh-minio:0.0.1” already present on machine

Normal Created 15s (x3 over 32s) kubelet Created container image-pull-singleuser

Normal Started 15s (x3 over 32s) kubelet Started container image-pull-singleuser

Warning BackOff 1s (x4 over 30s) kubelet Back-off restarting failed container image-pull-singleuser in pod hook-image-puller-74t2b_jh-oidc(57974205-aad5-4b1d-b8cf-2aa47b2caf43)

rdmnf · September 22, 2025, 8:14am

When I disable prePuller.hook.enable: false the following is the status of the pods, I cannot run it.

> k get pod -n jh-oidc
NAME                                 READY   STATUS                  RESTARTS      AGE
continuous-image-puller-7d2lh        0/1     Init:Error              3 (37s ago)   53s
continuous-image-puller-gdcg5        0/1     Init:Error              3 (38s ago)   53s
continuous-image-puller-lsvpc        0/1     Init:Error              3 (35s ago)   53s
continuous-image-puller-rm7r6        0/1     Init:Error              3 (35s ago)   53s
continuous-image-puller-tqjpf        0/1     Init:CrashLoopBackOff   3 (8s ago)    53s
continuous-image-puller-v2nxx        0/1     Init:Error              3 (38s ago)   53s
hub-696b88f57d-hzg9p                 1/1     Running                 0             53s
jupyter-XXXXXXXXXXXX---b8c2cb02   0/1     Error                   2 (16s ago)   21s
proxy-997b454f8-c7d5b                1/1     Running                 0             53s
user-scheduler-dcdb7c49b-5dw29       1/1     Running                 0             53s
user-scheduler-dcdb7c49b-dvlz7       1/1     Running                 0             53s

and the k describe of continuous-image-puller-XXXX:

Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 False 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:                      <none>
QoS Class:                    BestEffort
Node-Selectors:               <none>
Tolerations:                  hub.jupyter.org/dedicated=user:NoSchedule
                              hub.jupyter.org_dedicated=user:NoSchedule
                              node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                              node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                              node.kubernetes.io/not-ready:NoExecute op=Exists
                              node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                              node.kubernetes.io/unreachable:NoExecute op=Exists
                              node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  16m                  default-scheduler  Successfully assigned jh-oidc/continuous-image-puller-7d2lh to fr-tlswrk01
  Normal   Pulled     16m                  kubelet            Container image "quay.io/jupyterhub/k8s-network-tools:4.1.0" already present on machine
  Normal   Created    16m                  kubelet            Created container image-pull-metadata-block
  Normal   Started    16m                  kubelet            Started container image-pull-metadata-block
  Normal   Pulled     15m (x5 over 16m)    kubelet            Container image "rdmnf/jh-minio:0.0.1" already present on machine
  Normal   Created    15m (x5 over 16m)    kubelet            Created container image-pull-singleuser
  Normal   Started    15m (x5 over 16m)    kubelet            Started container image-pull-singleuser
  Warning  BackOff    106s (x72 over 16m)  kubelet            Back-off restarting failed container image-pull-singleuser in pod continuous-image-puller-7d2lh_jh-oidc(093ffe4d-5e12-45bb-aae2-bd6c8b0dbc23)

Paul2708 · September 22, 2025, 10:02am

Sorry for going back and forth, but can you provide the logs of the pod running the Docker image, namely jupyter-XXXXXXXXXXXX---b8c2cb02. It should state why it cannot start the JupyterLab environment.

rdmnf · September 22, 2025, 11:01am

no worries, sure:> k logs -n jh-oidc jupyter-XXXXXl—b8c2cb02

Defaulted container “notebook” out of: notebook, block-cloud-metadata (init)

exec /usr/bin/tini: exec format error

Paul2708 · September 22, 2025, 2:47pm

So there is a mismatch between the architecture used to build the image and the architecture on which the Docker container is running.
If I’m trying to run the image, I get the following warning:

WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64/v3), and no specific platform was requested

So you either build the image on the same CPU architecture it will be running on, or you have to target the platform explicitly, using:

rdmnf · September 23, 2025, 10:45am

Great! thanks! I can run it now.
the icons are not loaded correctly though, what could be the issue?

the base image is datascience notebook

rdmnf · September 24, 2025, 11:37am

It’s fine now after few hours!

Topic		Replies	Views
Error when starting Jupyterhub with image from my public docker registry Zero to JupyterHub on Kubernetes jupyterhub , help-wanted	1	1721	December 11, 2021
The base image for zero-to-jupyterhub-k8s JupyterHub jupyterhub , how-to , help-wanted	10	134	December 24, 2025
Kubernetes cannot create container for centos7 based jupyterhub notebook Zero to JupyterHub on Kubernetes jupyterhub , how-to , help-wanted	5	891	October 27, 2021
Kubespawner_override images Zero to JupyterHub on Kubernetes question	3	1143	December 28, 2021
Singleuser won't use any images except the default Zero to JupyterHub on Kubernetes help-wanted	0	366	June 15, 2021

Boil custom packages in Jupyterhub

Related topics