Hi Folks !
Part of kubespawner , my service and pods are created dynamically in Kubernetes cluster for single-user server. But I am not able to connect to this pod from outside within same namespace.
Do we need to create separate ingress rules tie to these services ?
Is there is a way to create ingress dynamically as a part of Spawn same as pods and service for single-user server ?
Thank you and appreciated with any feedback
Hi! You should be able to connect to JupyterLab (or notebook) running in the pods through the JupyterHub interface. What are you trying to do?
Hi @manics , Thanks for the feedback.
I am trying to use kubespawner to start the single user server. Pods are created in kubernetes, but somehow not able to communicate with error :
Spawn failed: Server at http://xxxxxxx-dev1.svc.cluster.local:8888/user/xxxxx/ didnât respond in 60 seconds.
Just to rule out, i am trying to curl using service DNS but not able to connect with âconnection refusedâ.
Now this make sense , why my Spawn didnt respond and failed .
Any take on this , really appreciated with any help
It sounds like youâre using a non-standard configuration, since services arenât created by default. Please could you show us your full configuration, and tell us how your Kubernetes cluster was setup? Ideally enough information for someone to reproduce your problem if they wanted to.
Sure @manics
I am starting my jupyterhub server with below config file
Environment:
Jupyterhub : 2.3.1
jupyterhub-kubespawner : 4.2.0
jupyterlab : 3.4.5
Notebook : 6.4.12
ipynb : 0.5.1
##########################################
import os
import sys
c = get_config()
c.LDAPAuthenticator.use_ssl = False
c.JupyterHub.shutdown_on_logout = True
c.Spawner.env_keep = [âPATHâ, âPYTHONPATHâ, âPYSPARK_PYTHONâ,âSPARK_HOMEâ,âHADOOP_HOMEâ,âHADOOP_CONF_DIRâ]
import logging
c.JupyterHub.log_level = logging.DEBUG
c.KubeSpawner.debug = True
c.SwarmSpawner.debug = True
c.LocalProcessSpawner.debug = True
c.NotebookApp.allow_root=True
############################################################################################################
kubespawner.KubeSpawner
############################################################################################################
import os
import socket
#c.JupyterHub.bind_url = âhttp://127.0.0.1:8080/â
c.Authenticator.admin_users = {âxxxâ}
c.JupyterHub.admin_access = True
c.JupyterHub.allow_named_servers = True
c.JupyterHub.cleanup_servers = False
c.KubeSpawner.services_enabled = True
c.JupyterHub.spawner_class = âkubespawner.KubeSpawnerâ
c.JupyterHub.authenticator_class = âdummyâ
c.kubespawner.cmd = [âjupyter notebookâ]
c.KubeSpawner.default_url = â/notebookâ
c.KubeSpawner.port= int(8888)
c.KubeSpawner.storage_pvc_ensure = True
c.JupyterHub.ip = str(â0.0.0.0â)
c.JupyterHub.hub_ip = str(â127.0.0.1â)
c.KubeSpawner.start_timeout = int(3600)
c.Spawner.http_timeout = int(60)
c.JupyterHub.tornado_settings = {âslow_spawn_timeoutâ: 30}
c.KubeSpawner.storage_pvc_ensure = False
if os.environ.get(âCIâ):
c.JupyterHub.hub_connect_ip = â127.0.0.1â
else:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect((â8.8.8.8â, 80))
host_ip = s.getsockname()[0]
s.close()
c.JupyterHub.hub_connect_ip = host_ip
c.KubeSpawner.image = str(âxxxxxxxxxxâ)
c.KubeSpawner.service_account = str(âdev-xxxxâ)
c.KubeSpawner.namespace = str(âxxxxxxxxxxâ)
c.KubeSpawner.image_pull_secrets = [âxxxxxxxxxxâ]
c.KubeSpawner.image_pull_policy = str(âIfNotPresentâ)
c.KubeSpawner.profile_list = [
{
âdisplay_nameâ: âBDP Sandboxâ,
âdefaultâ: True,
âkubespawner_overrideâ: {
âimageâ: âxxxxxxxxxxâ,
âimage_pull_secretsâ: [âxxxxxxxxxxâ],
ânamespaceâ: âxxxxxxxxxxâ,
âcpu_limitâ: 2,
âcpu_guaranteeâ : 1,
âmem_limitâ: â512Mâ,
}
}
]
I think itâll be easier if you start by deploying JupyterHub using the official Helm chart:
Once itâs working with a default configuration you can then modify it to fit your needs.
Thanks @manics , helm is not yet supported in our organization. Jupyterhub is running fine without KubeSpawner, but we want to spin single-user server
Hi @manics , i am trying to install jupyterhub with helm Chart as you suggested.
All the pods and services are up and running with default installations following âzero to jupyterhubâ instructions.
But , while trying to login it is given me resources forbidden error as below. As i can see it is trying to use âdefaultâ service account.
I have configured to use custom service account with all the roles , but it is still fetching SA as âdefaultâ.
config.yaml
singleuser:
serviceAccountName: âdev-saâ
Error :
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({âAudit-Idâ: â3adec320-bbc9-4691-912e-ff2457654072â, âCache-Controlâ: âno-cache, privateâ, âContent-Typeâ: âapplication/jsonâ, âX-Content-Type-Optionsâ: ânosniffâ, âX-Kubernetes-Pf-Flowschema-Uidâ: â0ab620e2-e89c-4db4-80a3-3788f991216bâ, âX-Kubernetes-Pf-Prioritylevel-Uidâ: â168a6768-cb6f-486e-b0a0-29df80a36208â, âDateâ: âMon, 05 Sep 2022 18:10:42 GMTâ, âContent-Lengthâ: â328â})
HTTP response body: bâ{âkindâ:âStatusâ,âapiVersionâ:âv1â,âmetadataâ:{},âstatusâ:âFailureâ,âmessageâ:âpods is forbidden: User \âsystem:serviceaccount:xxxxx-dev1:default\â cannot list resource \âpods\â in API group \â\" in the namespace \âxxxxx-dev1\â",âreasonâ:âForbiddenâ,âdetailsâ:{âkindâ:âpodsâ},âcodeâ:403}\nâ
Any take on this , what might have been missing ?
I think itâs the service account for the hub pod that needs these permissions. Youâve configured it to luanch singleuser pods with that service account.
I believe the config should be:
hub:
serviceAccount:
create: false
name: "dev-sa"
to tell the hub to use an existing service account with the name dev-sa