Spawn single-user notebook on Kubernetes cluster using kubespawner

JupyterHub Zero to JupyterHub on Kubernetes
,
1

Hi Folks!
I am trying to spawn single-user notebook on Kubernetes cluster using kubespawner . I am able to login using LDAP authenticator and I can see corresponding Kubernetes instance has been created in Kubernetes cluster. But service is not able to respond.
I tried with increasing http_timeout and start_timeout but no luck yet and hoping to get some advice what might have gone wrong.

I have seen similar post and tried with all the suggestion, but nothing seems to be working so far.

Any help would be greatly appreciated. #NewtoCommunity

Environment:
Jupyterhub : 2.2.2
jupyterhub-kubespawner : 4.2.0

Config:
import logging

c.JupyterHub.log_level = logging.DEBUG
c.KubeSpawner.debug = True
c.SwarmSpawner.debug = True
c.NotebookApp.allow_root=True

############################################################################################################

kubespawner.KubeSpawner

############################################################################################################
import os
import socket

c.JupyterHub.spawner_class = ‘kubespawner.KubeSpawner’
c.JupyterHub.ip = str(‘0.0.0.0’)
c.JupyterHub.hub_ip = str(‘127.0.0.1’)
c.JupyterHub.cleanup_servers = False
c.KubeSpawner.start_timeout = int(3600)
c.Spawner.http_timeout = int(60)
c.JupyterHub.tornado_settings = {‘slow_spawn_timeout’: 30}

if os.environ.get(“CI”):
c.JupyterHub.hub_connect_ip = “127.0.0.1”
else:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect((“8.8.8.8”, 80))
host_ip = s.getsockname()[0]
s.close()
c.JupyterHub.hub_connect_ip = host_ip

c.KubeSpawner.image = str(xxxxxxx:5000/xxxxxxx/bdp-jupyter-rc:7’)
c.KubeSpawner.storage_pvc_ensure = False
c.JupyterHub.allow_named_servers = True
c.KubeSpawner.profile_list = [
{
‘display_name’: ‘Sandbox’,
‘default’: True,
‘kubespawner_override’: {
‘image’: xxxxxxxxx/bdp-jupyter-rc:7’,
‘cpu_limit’: 1,
‘mem_limit’: ‘512M’,
}
}
]

Error :

image
image2518×920 186 KB

2

I think the property your looking for is the singleuser.startTimeout

I’ve attached details below on how I got mine working in k8.

I installed this to k8 using the following chart (below)
Notes: I use keycloak as my authenticator… and using the lab teradata/jupyterlab-extensions
I use a separate config map to load the connections… thats specific to the TD image.
I’m also using Traefik as my ingress controller so I have the path https:///jhub so I dont need the loadbalancer. I create the ingressRoute seperatly as I enforce the TLS through it.
I also change the default home directory through use of the cmds.

export NAMESPACE=jhub
export RELASENAME=jupyterhub-dev
export VERSION=1.2.0

kubectl config set-context --current --namespace=$NAMESPACE

helm upgrade --cleanup-on-fail --install $RELASENAME jupyterhub/jupyterhub --namespace $NAMESPACE --create-namespace --version=$VERSION --values jhub_config.yaml

############Chart
# This file can update the JupyterHub Helm chart's default configuration values.
#
# For reference see the configuration reference and default values, but make
# sure to refer to the Helm chart version of interest to you!
#
# Introduction to YAML:     https://www.youtube.com/watch?v=cdLNKUoMc6c
# Chart config reference:   https://zero-to-jupyterhub.readthedocs.io/en/stable/resources/reference.html
# Chart default values:     https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/81aa3eb42a34cce00d134d88383f2212d041f6a8/jupyterhub/values.yaml
# Available chart versions: https://jupyterhub.github.io/helm-chart/
#
singleuser:
  cmd: ["jupyterhub-singleuser","--notebook-dir=/home"]
  startTimeout: 120
  defaultUrl: "/lab"
  image:
    name: teradata/jupyterlab-extensions
    tag: latest
  storage:
    homeMountPath: /home/jovyan/data
    capacity: 5Gi
    dynamic:
      storageClass: pmax-storage-policy
    extraVolumes:
      - name: connections
        configMap:
          name: connections
          items:
            - key: jhub_td_connections.yaml
              path: tdconfig.yaml
    extraVolumeMounts:
      - name: connections
        mountPath: /tmp/tdconfig
  extraEnv:
    accept_license: "Y"
    TD_CONFIG: /tmp/tdconfig/tdconfig.yaml
hub:
  db:
    pvc:
      storageClassName: pmax-storage-policy
  baseUrl: /jhub
  config:
    GenericOAuthenticator:
      client_id: <myclientId>
      client_secret: <Myclientsecret>
      oauth_callback_url: https://<mydomain>/jhub/hub/oauth_callback
      authorize_url: https://<mydomain>/keycloak/realms/<myrealm>/protocol/openid-connect/auth
      token_url: https://<mydomain>/keycloak/realms/<myrealm>/protocol/openid-connect/token
      userdata_url: https://<mydomain>/keycloak/realms/<myrealm>/protocol/openid-connect/userinfo
      login_service: keycloak
      username_key: preferred_username
      userdata_params:
        state: state
    JupyterHub:
      authenticator_class: generic-oauth

proxy:
  service:
    type: ClusterIP
3

Thank you @chrispward,
I am trying with helm chart now , as looks like that is preferred way.

While trying to set up custom service account for my singleuser , it is fetching default value which doenst have role define. Do i need to configure any other place then in Signleuser as below ?

config.yaml
singleuser:
serviceAccountName: “dev-sa”

Error :
HTTP response body: b’{“kind”:“Status”,“apiVersion”:“v1”,“metadata”:{},“status”:“Failure”,“message”:“pods is forbidden: User \“system:serviceaccount:xxxxx-dev1:default \” cannot list resource \“pods\” in API group \”" in the namespace \“xxxxx-dev1\”",“reason”:“Forbidden”,“details”:{“kind”:“pods”},“code”:403}\n’

4

I didnt create/use a service account.
Not sure if the charts auto create one. I think they do if you enable it.
If not, you’d have to manually create one 1st.
Looks like a priv issue (403) so the service account needs more privs.
My advice… install it with the bare min 1st and then modify you values from there. k8 can be hard to trace so get it working, then make a small alteration and update etc.
ie helm upgrade blah blah… etc.

1 Like