Creating Ingress rules dynamically for single-user server

JupyterHub
, ,
#1

Hi Folks !
Part of kubespawner , my service and pods are created dynamically in Kubernetes cluster for single-user server. But I am not able to connect to this pod from outside within same namespace.
Do we need to create separate ingress rules tie to these services ?
Is there is a way to create ingress dynamically as a part of Spawn same as pods and service for single-user server ?
Thank you and appreciated with any feedback :slight_smile:

#2

Hi! You should be able to connect to JupyterLab (or notebook) running in the pods through the JupyterHub interface. What are you trying to do?

#3

Hi @manics , Thanks for the feedback.
I am trying to use kubespawner to start the single user server. Pods are created in kubernetes, but somehow not able to communicate with error :
Spawn failed: Server at http://xxxxxxx-dev1.svc.cluster.local:8888/user/xxxxx/ didn’t respond in 60 seconds.
Just to rule out, i am trying to curl using service DNS but not able to connect with “connection refused”.
Now this make sense , why my Spawn didnt respond and failed .

Any take on this , really appreciated with any help :slight_smile:

#4

It sounds like you’re using a non-standard configuration, since services aren’t created by default. Please could you show us your full configuration, and tell us how your Kubernetes cluster was setup? Ideally enough information for someone to reproduce your problem if they wanted to.

#5

Sure @manics
I am starting my jupyterhub server with below config file

Environment:
Jupyterhub : 2.3.1
jupyterhub-kubespawner : 4.2.0
jupyterlab : 3.4.5
Notebook : 6.4.12
ipynb : 0.5.1

##########################################
import os
import sys

c = get_config()

c.LDAPAuthenticator.use_ssl = False
c.JupyterHub.shutdown_on_logout = True

c.Spawner.env_keep = [‘PATH’, ‘PYTHONPATH’, ‘PYSPARK_PYTHON’,‘SPARK_HOME’,‘HADOOP_HOME’,‘HADOOP_CONF_DIR’]

import logging

c.JupyterHub.log_level = logging.DEBUG
c.KubeSpawner.debug = True
c.SwarmSpawner.debug = True
c.LocalProcessSpawner.debug = True

c.NotebookApp.allow_root=True

############################################################################################################

kubespawner.KubeSpawner

############################################################################################################
import os
import socket

#c.JupyterHub.bind_url = ‘http://127.0.0.1:8080/’

c.Authenticator.admin_users = {‘xxx’}
c.JupyterHub.admin_access = True
c.JupyterHub.allow_named_servers = True
c.JupyterHub.cleanup_servers = False
c.KubeSpawner.services_enabled = True
c.JupyterHub.spawner_class = ‘kubespawner.KubeSpawner’
c.JupyterHub.authenticator_class = ‘dummy’

c.kubespawner.cmd = [‘jupyter notebook’]
c.KubeSpawner.default_url = ‘/notebook’

c.KubeSpawner.port= int(8888)
c.KubeSpawner.storage_pvc_ensure = True

c.JupyterHub.ip = str(‘0.0.0.0’)
c.JupyterHub.hub_ip = str(‘127.0.0.1’)

c.KubeSpawner.start_timeout = int(3600)
c.Spawner.http_timeout = int(60)
c.JupyterHub.tornado_settings = {‘slow_spawn_timeout’: 30}

c.KubeSpawner.storage_pvc_ensure = False

if os.environ.get(“CI”):
c.JupyterHub.hub_connect_ip = “127.0.0.1”
else:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect((“8.8.8.8”, 80))
host_ip = s.getsockname()[0]
s.close()

c.JupyterHub.hub_connect_ip = host_ip

c.KubeSpawner.image = str(‘xxxxxxxxxx’)
c.KubeSpawner.service_account = str(‘dev-xxxx’)
c.KubeSpawner.namespace = str(‘xxxxxxxxxx’)
c.KubeSpawner.image_pull_secrets = [‘xxxxxxxxxx’]
c.KubeSpawner.image_pull_policy = str(‘IfNotPresent’)

c.KubeSpawner.profile_list = [
{
‘display_name’: ‘BDP Sandbox’,
‘default’: True,
‘kubespawner_override’: {
‘image’: ‘xxxxxxxxxx’,
‘image_pull_secrets’: [‘xxxxxxxxxx’],
‘namespace’: ‘xxxxxxxxxx’,
‘cpu_limit’: 2,
‘cpu_guarantee’ : 1,
‘mem_limit’: ‘512M’,
}
}
]

#6

I think it’ll be easier if you start by deploying JupyterHub using the official Helm chart:

Once it’s working with a default configuration you can then modify it to fit your needs.

#7

Thanks @manics , helm is not yet supported in our organization. Jupyterhub is running fine without KubeSpawner, but we want to spin single-user server

#8

Hi @manics , i am trying to install jupyterhub with helm Chart as you suggested.
All the pods and services are up and running with default installations following “zero to jupyterhub” instructions.
But , while trying to login it is given me resources forbidden error as below. As i can see it is trying to use “default” service account.
I have configured to use custom service account with all the roles , but it is still fetching SA as “default”.

config.yaml
singleuser:
serviceAccountName: “dev-sa”

Error :
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({‘Audit-Id’: ‘3adec320-bbc9-4691-912e-ff2457654072’, ‘Cache-Control’: ‘no-cache, private’, ‘Content-Type’: ‘application/json’, ‘X-Content-Type-Options’: ‘nosniff’, ‘X-Kubernetes-Pf-Flowschema-Uid’: ‘0ab620e2-e89c-4db4-80a3-3788f991216b’, ‘X-Kubernetes-Pf-Prioritylevel-Uid’: ‘168a6768-cb6f-486e-b0a0-29df80a36208’, ‘Date’: ‘Mon, 05 Sep 2022 18:10:42 GMT’, ‘Content-Length’: ‘328’})
HTTP response body: b’{“kind”:“Status”,“apiVersion”:“v1”,“metadata”:{},“status”:“Failure”,“message”:“pods is forbidden: User \“system:serviceaccount:xxxxx-dev1:default\” cannot list resource \“pods\” in API group \”\" in the namespace \“xxxxx-dev1\”",“reason”:“Forbidden”,“details”:{“kind”:“pods”},“code”:403}\n’

Any take on this , what might have been missing ?

#9

I think it’s the service account for the hub pod that needs these permissions. You’ve configured it to luanch singleuser pods with that service account.

I believe the config should be:

hub:
  serviceAccount:
    create: false
    name: "dev-sa"

to tell the hub to use an existing service account with the name dev-sa