Hi Team,
After deploying the juypterhub solution from https://zero-to-jupyterhub.readthedocs.io/en/latest/.
We are facing the permission issue.
config.yaml
singleuser:
defaultUrl: “/lab”
image:
name: jupyter/base-notebook
tag: latest
storage:
type: “static”
static:
pvcName: “nfs-home-pvc”
subPath: ‘{username}’
extraEnv:
NB_USER: ${USER}
CHOWN_HOME: ‘yes’
CHOWN_HOME_OPTS: ‘-R’
cmd: “start-singleuser.sh”
Need your Guidance.
Could you give us more details please? What permission issue have you encountered?
HI Manics,
Please find the problem overview:-
Expected Behaviour:-
uid=1001(user1) gid=1001(user1) groups=1001(user1)
uid=1002(user2) gid=1002(user2) groups=1002(user2)
Current Behaviour:-
uid=1000(jovyan) gid=0(root) groups=0(root),100(users)
Overview:
Inside the Juypterlab the permission of the users is not as per the Linux environment.
Looking forward to your response.
Thank You
Atul Yadav
This is the relevant part of the base-notebook startup script:
You’ll need to run the singleuser server as root (UID 0) by setting singleuser.uid: 0 in your configuration so that the starting user has administrative rights to modify the default user.
Hi Manics,
Please provide the config.yaml file as still i am getting the error while launching the pod.
Thank You
Atul Yadav
I don’t have a config file with the required configuration right now. Can you please paste your current configuration here? Use triple backticks before and after so that it’ll be formatted as code.
Hi Manics,
Please find the config file output.
secretToken: 97141abb55ea5321867979cb57bb2e6a86a2f4d6bb166fca45aedb07c212c42d
service:
type: NodePort
nodePorts:
http: 30080
https: 30443
networkPolicy:
enabled: true
hub:
cookieSecret: 1470700e01f77171c2c67b12130c25081dfbdf2697af8c2f2bd05621b31100bf
db:
type: sqlite-memory
prePuller:
continuous:
enabled: true
singleuser:
defaultUrl: "/lab"
storage:
type: "static"
static:
pvcName: "nfs-home-pvc"
subPath: '{username}'
extraEnv:
GRANT_SUDO: "yes"
uid: 0
fsGid: 0
cmd: "start-notebook.sh"You’ll need to pass the username and uid into the spawner as the environment variables NB_USER and NB_UID, for example using a prespawn hook. This is an old bit of config I’ve used in the past with the LDAPAuthenticator though I haven’t tested it recently:
def pre_spawn_start(self, user, spawner):
auth_state = yield user.get_auth_state()
self.log.debug('pre_spawn_start auth_state:%s' % auth_state)
if not auth_state:
return
# setup environment
spawner.environment['NB_UID'] = str(
auth_state['uidNumber'][0])
spawner.environment['NB_USER'] = auth_state['uid'][0]
You’ll need to adjust for work with whatever authenticator you’re using. Even if you’re using the LDAPAuthenticator you’d most likely still need to modify it to match your LDAP server config.
HI Manics,
Thanks for your guidance, but error behaviour is changed now.
linux ENV behaviour
user2@server11:~$ id
uid=1001(user2) gid=1001(user2) groups=1001(user2)
Jupyterhub Terminal
Why the secondary group information is coming “100”.
Thank You
Atul Yadav
100 is the default group of the user in the Docker image. The startup script does not remove the existing group, it only adds to it:
Hi Manics,
As per your input, using ldap server for authenticating.
ubuntu client authentication is working fine, but juypterhub login stuck at" Attempting to bind user1 with uid=user1,ou=people,dc=local".
please advice us to isolate the problem.
Thank You
Atul Yadav
Kubernetes master is running at https://192.168.126.221:6443
KubeDNS is running at https://192.168.126.221:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://192.168.126.221:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
server11 Ready master 78m v1.15.12 192.168.126.221 <none> Ubuntu 18.04.4 LTS 4.15.0-91-generic docker://18.9.7
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
NAMESPACE NAME READY STATUS RESTARTS AGE
jhub continuous-image-puller-9762g 1/1 Running 0 16m
jhub hub-7c658466b9-l8pm4 1/1 Running 0 16m
jhub proxy-6dd8854ff4-s7zxt 1/1 Running 1 43m
jhub user-scheduler-6bd845b669-mv4wh 1/1 Running 2 63m
jhub user-scheduler-6bd845b669-zgcnk 1/1 Running 2 63m
kube-system calico-kube-controllers-75dbcbbf8b-zqvld 1/1 Running 4 77m
kube-system calico-node-htzls 1/1 Running 4 77m
kube-system coredns-5d4dd4b4db-7j42f 1/1 Running 4 77m
kube-system coredns-5d4dd4b4db-kgwqc 1/1 Running 4 77m
kube-system etcd-server11 1/1 Running 4 76m
kube-system kube-apiserver-server11 1/1 Running 5 76m
kube-system kube-controller-manager-server11 1/1 Running 4 76m
kube-system kube-proxy-lf95z 1/1 Running 4 77m
kube-system kube-scheduler-server11 1/1 Running 4 77m
kube-system metrics-server-5fccfd96d4-lvvcz 1/1 Running 0 45s
kube-system tiller-deploy-5cccc67b55-6rjdz 1/1 Running 4 76m
REPOSITORY TAG IMAGE ID CREATED SIZE
jupyter/base-notebook latest 31c0bb3c5c56 19 hours ago 650MB
gcr.io/kubernetes-helm/tiller v2.16.8 c3607a0e6c78 5 days ago 88MB
calico/node v3.9.6 15f795b449d2 2 weeks ago 195MB
calico/pod2daemon-flexvol v3.9.6 63fbf227cf10 2 weeks ago 9.78MB
calico/cni v3.9.6 0ce7550069ed 2 weeks ago 167MB
calico/kube-controllers v3.9.6 081a5bf738ad 2 weeks ago 56MB
jupyter/datascience-notebook 76402a27fd13 e5d2b03a401a 2 weeks ago 4.65GB
tensorflow/tensorflow 2.1.1-gpu-jupyter a1b9e806848a 3 weeks ago 3.78GB
k8s.gcr.io/kube-controller-manager v1.15.12 7b4d4985877a 5 weeks ago 159MB
k8s.gcr.io/kube-apiserver v1.15.12 c81971987f04 5 weeks ago 207MB
k8s.gcr.io/kube-proxy v1.15.12 00206e1127f2 5 weeks ago 82.5MB
k8s.gcr.io/kube-scheduler v1.15.12 196d53938faa 5 weeks ago 81.2MB
jupyterhub/k8s-image-awaiter 0.9.0 7b10713c8231 2 months ago 4.15MB
jupyterhub/k8s-network-tools 0.9.0 00c5e7b520c2 2 months ago 5.62MB
jupyterhub/k8s-hub 0.9.0 2f82dac5c3ab 2 months ago 632MB
jupyterhub/configurable-http-proxy 4.2.1 3379bc93b897 3 months ago 138MB
gcr.io/google_containers/kube-scheduler-amd64 v1.13.12 54f3185a42a5 8 months ago 79.6MB
k8s.gcr.io/metrics-server-amd64 v0.3.6 9dd718864ce6 8 months ago 39.9MB
k8s.gcr.io/coredns 1.3.1 eb516548c180 17 months ago 40.3MB
jupyter/all-spark-notebook 2343e33dec46 708afc986779 18 months ago 6.32GB
jupyter/datascience-notebook 2343e33dec46 60b2ad31b985 18 months ago 6.33GB
jupyter/base-notebook 2343e33dec46 8253370cbe2d 18 months ago 738MB
k8s.gcr.io/etcd 3.3.10 2c4adeb21b4f 18 months ago 258MB
gcr.io/google_containers/pause 3.1 da86e6ba6ca1 2 years ago 742kB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB
root@server11:~/locuz# kubectl --namespace=jhub describe pod hub-7c658466b9-l8pm4
Name: hub-7c658466b9-l8pm4
Namespace: jhub
Priority: 0
Node: server11/192.168.126.221
Start Time: Tue, 16 Jun 2020 13:14:21 +0530
Labels: app=jupyterhub
component=hub
hub.jupyter.org/network-access-proxy-api=true
hub.jupyter.org/network-access-proxy-http=true
hub.jupyter.org/network-access-singleuser=true
pod-template-hash=7c658466b9
release=jhub
Annotations: checksum/config-map: 959726a148b08eb6e19e367350da5a6f05ed2208bcdc8d5eea672c2858455d8c
checksum/secret: 42e7d091ebcdeb82427cc8cacc42d6f086bc4aac9555220bc34b1c65ca6e9790
cni.projectcalico.org/podIP: 192.168.119.44/32
Status: Running
IP: 192.168.119.44
Controlled By: ReplicaSet/hub-7c658466b9
Containers:
hub:
Container ID: docker://b37ad130f50fdb25c4c45ecbe8882e86e8b98534638d111fa17cb5ac752939df
Image: jupyterhub/k8s-hub:0.9.0
Image ID: docker-pullable://jupyterhub/k8s-hub@sha256:ec78bdae0fed0c49474950a1fe4fec3b00448599d4a35e51a93b10a7f342846b
Port: 8081/TCP
Host Port: 0/TCP
Command:
jupyterhub
--config
/etc/jupyterhub/jupyterhub_config.py
--debug
State: Running
Started: Tue, 16 Jun 2020 13:14:23 +0530
Ready: False
Restart Count: 0
Requests:
cpu: 200m
memory: 512Mi
Readiness: http-get http://:hub/hub/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
PYTHONUNBUFFERED: 1
HELM_RELEASE_NAME: jhub
JPY_COOKIE_SECRET: <set to the key 'hub.cookie-secret' in secret 'hub-secret'> Optional: false
POD_NAMESPACE: jhub (v1:metadata.namespace)
CONFIGPROXY_AUTH_TOKEN: <set to the key 'proxy.token' in secret 'hub-secret'> Optional: false
Mounts:
/etc/jupyterhub/config/ from config (rw)
/etc/jupyterhub/cull_idle_servers.py from config (rw,path="cull_idle_servers.py")
/etc/jupyterhub/jupyterhub_config.py from config (rw,path="jupyterhub_config.py")
/etc/jupyterhub/secret/ from secret (rw)
/etc/jupyterhub/z2jh.py from config (rw,path="z2jh.py")
/var/run/secrets/kubernetes.io/serviceaccount from hub-token-jwrk6 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: hub-config
Optional: false
secret:
Type: Secret (a volume populated by a Secret)
SecretName: hub-secret
Optional: false
hub-token-jwrk6:
Type: Secret (a volume populated by a Secret)
SecretName: hub-token-jwrk6
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m default-scheduler Successfully assigned jhub/hub-7c658466b9-l8pm4 to server11
Normal Pulled 2m59s kubelet, server11 Container image "jupyterhub/k8s-hub:0.9.0" already present on machine
Normal Created 2m59s kubelet, server11 Created container hub
Normal Started 2m58s kubelet, server11 Started container hub
Warning Unhealthy 4s (x8 over 74s) kubelet, server11 Readiness probe failed: Get http://192.168.119.44:8081/hub/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
root@server11:~/locuz# kubectl --namespace=jhub log hub-7c658466b9-l8pm4
[D 2020-06-16 07:44:24.080 JupyterHub application:556] Looking for /etc/jupyterhub/jupyterhub_config in /srv/jupyterhub
Loading /etc/jupyterhub/config/values.yaml
Loading /etc/jupyterhub/secret/values.yaml
Loading extra config: ldapConfig
[D 2020-06-16 07:44:24.640 JupyterHub application:578] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2020-06-16 07:44:24.686 JupyterHub app:2240] Running JupyterHub version 1.1.0
[I 2020-06-16 07:44:24.687 JupyterHub app:2271] Using Authenticator: builtins.LDAPAuthenticatorInfoUID
[I 2020-06-16 07:44:24.687 JupyterHub app:2271] Using Spawner: kubespawner.spawner.KubeSpawner
[I 2020-06-16 07:44:24.687 JupyterHub app:2271] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.1.0
[I 2020-06-16 07:44:24.689 JupyterHub app:1345] Loading cookie_secret from env[JPY_COOKIE_SECRET]
[D 2020-06-16 07:44:24.689 JupyterHub app:1516] Connecting to db: sqlite://
[D 2020-06-16 07:44:24.709 JupyterHub orm:718] Stamping empty database with alembic revision 4dc2d5a8c53c
[I 2020-06-16 07:44:24.721 alembic.runtime.migration migration:154] Context impl SQLiteImpl.
[I 2020-06-16 07:44:24.721 alembic.runtime.migration migration:161] Will assume non-transactional DDL.
[I 2020-06-16 07:44:24.728 alembic.runtime.migration migration:513] Running stamp_revision -> 4dc2d5a8c53c
[D 2020-06-16 07:44:24.729 alembic.runtime.migration migration:723] new branch insert 4dc2d5a8c53c
[W 2020-06-16 07:44:24.741 JupyterHub app:1579] JupyterHub.hub_connect_port is deprecated as of 0.9. Use JupyterHub.hub_connect_url to fully specify the URL for connecting to the Hub.
[W 2020-06-16 07:44:24.743 JupyterHub app:1624] No admin users, admin interface will be unavailable.
[W 2020-06-16 07:44:24.743 JupyterHub app:1626] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2020-06-16 07:44:24.743 JupyterHub app:1655] Not using whitelist. Any authenticated user will be allowed.
10
[D 2020-06-16 07:44:24.790 JupyterHub app:1914] Initializing spawners
[D 2020-06-16 07:44:24.792 JupyterHub app:2038] Loaded users:
[I 2020-06-16 07:44:24.792 JupyterHub app:2311] Initialized 0 spawners in 0.003 seconds
[I 2020-06-16 07:44:24.794 JupyterHub app:2520] Not starting proxy
[D 2020-06-16 07:44:24.795 JupyterHub proxy:766] Proxy: Fetching GET http://10.102.23.122:8001/api/routes
[I 2020-06-16 07:44:24.798 JupyterHub app:2556] Hub API listening on http://0.0.0.0:8081/hub/
[I 2020-06-16 07:44:24.798 JupyterHub app:2558] Private Hub API connect url http://10.108.134.38:8081/hub/
[I 2020-06-16 07:44:24.798 JupyterHub app:2571] Starting managed service cull-idle
[I 2020-06-16 07:44:24.799 JupyterHub service:335] Starting service 'cull-idle': ['python3', '/etc/jupyterhub/cull_idle_servers.py', '--url=http://127.0.0.1:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2020-06-16 07:44:24.801 JupyterHub service:121] Spawning python3 /etc/jupyterhub/cull_idle_servers.py --url=http://127.0.0.1:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[D 2020-06-16 07:44:24.806 JupyterHub spawner:1113] Polling subprocess every 30s
[D 2020-06-16 07:44:24.808 JupyterHub proxy:315] Fetching routes to check
[D 2020-06-16 07:44:24.808 JupyterHub proxy:766] Proxy: Fetching GET http://10.102.23.122:8001/api/routes
[I 2020-06-16 07:44:24.810 JupyterHub proxy:320] Checking routes
[I 2020-06-16 07:44:24.811 JupyterHub app:2631] JupyterHub is now running at http://10.105.25.144:443/
[D 2020-06-16 07:44:24.811 JupyterHub app:2233] It took 0.735 seconds for the Hub to start
[D 2020-06-16 07:44:25.008 JupyterHub base:285] Recording first activity for <APIToken('e76e...', service='cull-idle')>
[I 2020-06-16 07:44:25.017 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 15.34ms
[I 2020-06-16 07:44:26.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.83ms
[I 2020-06-16 07:44:36.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.76ms
[I 2020-06-16 07:44:46.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.72ms
[I 2020-06-16 07:44:56.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.68ms
[I 2020-06-16 07:45:06.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.67ms
[I 2020-06-16 07:45:16.136 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 1.08ms
[D 2020-06-16 07:45:24.812 JupyterHub proxy:766] Proxy: Fetching GET http://10.102.23.122:8001/api/routes
[I 2020-06-16 07:45:24.815 JupyterHub proxy:320] Checking routes
[I 2020-06-16 07:45:26.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.69ms
[I 2020-06-16 07:45:36.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.64ms
[I 2020-06-16 07:45:46.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.72ms
[I 2020-06-16 07:45:56.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.95ms
[I 2020-06-16 07:45:57.147 JupyterHub log:174] 302 GET /hub/home -> /hub/login?next=%2Fhub%2Fhome (@192.168.126.221) 1.15ms
[I 2020-06-16 07:45:57.191 JupyterHub log:174] 200 GET /hub/login?next=%2Fhub%2Fhome (@192.168.126.221) 29.45ms
[D 2020-06-16 07:45:58.380 JupyterHub <string>:342] Attempting to bind user1 with uid=user1,ou=people,dc=local
[E 2020-06-16 07:48:09.271 JupyterHub web:1792] Uncaught exception POST /hub/login?next=%2Fhub%2Fhome (192.168.126.221)
HTTPServerRequest(protocol='http', host='192.168.126.221:30080', method='POST', uri='/hub/login?next=%2Fhub%2Fhome', version='HTTP/1.1', remote_ip='192.168.126.221')
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1703, in _execute
result = await result
File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/login.py", line 144, in post
user = await self.login_user(data)
File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 699, in login_user
authenticated = await self.authenticate(data)
File "/usr/local/lib/python3.6/dist-packages/jupyterhub/auth.py", line 383, in get_authenticated_user
authenticated = await maybe_future(self.authenticate(handler, data))
File "<string>", line 345, in authenticate
File "<string>", line 291, in get_connection
File "/usr/local/lib/python3.6/dist-packages/ldap3/core/connection.py", line 355, in __init__
self.do_auto_bind()
File "/usr/local/lib/python3.6/dist-packages/ldap3/core/connection.py", line 370, in do_auto_bind
self.open(read_server_info=False)
File "/usr/local/lib/python3.6/dist-packages/ldap3/strategy/sync.py", line 56, in open
BaseStrategy.open(self, reset_usage, read_server_info)
File "/usr/local/lib/python3.6/dist-packages/ldap3/strategy/base.py", line 139, in open
raise exception_history[0][0]
ldap3.core.exceptions.LDAPSocketOpenError: socket connection error while opening: [Errno 110] Connection timed out
[D 2020-06-16 07:48:09.273 JupyterHub base:1197] No template for 500
[E 2020-06-16 07:48:09.278 JupyterHub log:166] {
"X-Forwarded-Host": "192.168.126.221:30080",
"X-Forwarded-Proto": "http",
"X-Forwarded-Port": "30080",
"X-Forwarded-For": "192.168.126.221",
"Upgrade-Insecure-Requests": "1",
"Referer": "http://192.168.126.221:30080/hub/login?next=%2Fhub%2Fhome",
"Connection": "close",
"Origin": "http://192.168.126.221:30080",
"Content-Length": "31",
"Content-Type": "application/x-www-form-urlencoded",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US,en;q=0.5",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0",
"Host": "192.168.126.221:30080"
}
[E 2020-06-16 07:48:09.278 JupyterHub log:174] 500 POST /hub/login?next=%2Fhub%2Fhome (@192.168.126.221) 130898.53ms
[D 2020-06-16 07:48:09.285 JupyterHub proxy:766] Proxy: Fetching GET http://10.102.23.122:8001/api/routes
[I 2020-06-16 07:48:09.286 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.55ms
[I 2020-06-16 07:48:09.287 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.11ms
[I 2020-06-16 07:48:09.287 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.13ms
[I 2020-06-16 07:48:09.288 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.18ms
[I 2020-06-16 07:48:09.288 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.33ms
[I 2020-06-16 07:48:09.289 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.50ms
[I 2020-06-16 07:48:09.289 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.63ms
[I 2020-06-16 07:48:09.290 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.70ms
[I 2020-06-16 07:48:09.290 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.83ms
[I 2020-06-16 07:48:09.291 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.82ms
[I 2020-06-16 07:48:09.291 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.75ms
[I 2020-06-16 07:48:09.292 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.76ms
[I 2020-06-16 07:48:09.292 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.77ms
[I 2020-06-16 07:48:09.293 JupyterHub proxy:320] Checking routes
[I 2020-06-16 07:48:16.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.60ms
[D 2020-06-16 07:48:24.812 JupyterHub proxy:766] Proxy: Fetching GET http://10.102.23.122:8001/api/routes
[I 2020-06-16 07:48:24.814 JupyterHub proxy:320] Checking routes
[I 2020-06-16 07:48:26.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.80ms
[I 2020-06-16 07:48:36.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.67ms
[I 2020-06-16 07:48:46.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.67ms
[I 2020-06-16 07:48:56.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.67ms
[I 2020-06-16 07:49:06.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.76ms
[I 2020-06-16 07:49:16.134 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.60ms
[D 2020-06-16 07:49:24.812 JupyterHub proxy:766] Proxy: Fetching GET http://10.102.23.122:8001/api/routes
[I 2020-06-16 07:49:24.814 JupyterHub proxy:320] Checking routes
[I 2020-06-16 07:49:26.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.64ms
[I 2020-06-16 07:49:36.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.64ms
[I 2020-06-16 07:49:46.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.63ms
[I 2020-06-16 07:49:56.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.83ms
[I 2020-06-16 07:50:06.136 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.91ms
[D 2020-06-16 07:50:13.305 JupyterHub <string>:342] Attempting to bind user1 with uid=user1,ou=people,dc=local
[E 2020-06-16 07:52:23.226 JupyterHub web:1792] Uncaught exception POST /hub/login?next=%2Fhub%2Fhome (192.168.126.221)
HTTPServerRequest(protocol='http', host='192.168.126.221:30080', method='POST', uri='/hub/login?next=%2Fhub%2Fhome', version='HTTP/1.1', remote_ip='192.168.126.221')
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1703, in _execute
result = await result
File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/login.py", line 144, in post
user = await self.login_user(data)
File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 699, in login_user
authenticated = await self.authenticate(data)
File "/usr/local/lib/python3.6/dist-packages/jupyterhub/auth.py", line 383, in get_authenticated_user
authenticated = await maybe_future(self.authenticate(handler, data))
File "<string>", line 345, in authenticate
File "<string>", line 291, in get_connection
File "/usr/local/lib/python3.6/dist-packages/ldap3/core/connection.py", line 355, in __init__
self.do_auto_bind()
File "/usr/local/lib/python3.6/dist-packages/ldap3/core/connection.py", line 370, in do_auto_bind
self.open(read_server_info=False)
File "/usr/local/lib/python3.6/dist-packages/ldap3/strategy/sync.py", line 56, in open
BaseStrategy.open(self, reset_usage, read_server_info)
File "/usr/local/lib/python3.6/dist-packages/ldap3/strategy/base.py", line 139, in open
raise exception_history[0][0]
ldap3.core.exceptions.LDAPSocketOpenError: socket connection error while opening: [Errno 110] Connection timed out
[D 2020-06-16 07:52:23.227 JupyterHub base:1197] No template for 500
[E 2020-06-16 07:52:23.229 JupyterHub log:166] {
"X-Forwarded-Host": "192.168.126.221:30080",
"X-Forwarded-Proto": "http",
"X-Forwarded-Port": "30080",
"X-Forwarded-For": "192.168.126.221",
"Upgrade-Insecure-Requests": "1",
"Connection": "close",
"Origin": "http://192.168.126.221:30080",
"Content-Length": "31",
"Content-Type": "application/x-www-form-urlencoded",
"Referer": "http://192.168.126.221:30080/hub/login?next=%2Fhub%2Fhome",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US,en;q=0.5",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0",
"Host": "192.168.126.221:30080"
}
[E 2020-06-16 07:52:23.229 JupyterHub log:174] 500 POST /hub/login?next=%2Fhub%2Fhome (@192.168.126.221) 129925.33ms
[D 2020-06-16 07:52:23.236 JupyterHub proxy:766] Proxy: Fetching GET http://10.102.23.122:8001/api/routes
[I 2020-06-16 07:52:23.237 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.29ms
[I 2020-06-16 07:52:23.238 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.63ms
[I 2020-06-16 07:52:23.238 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.83ms
[I 2020-06-16 07:52:23.239 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.93ms
[I 2020-06-16 07:52:23.239 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 7.03ms
[I 2020-06-16 07:52:23.240 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.81ms
[I 2020-06-16 07:52:23.240 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.83ms
[I 2020-06-16 07:52:23.241 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.84ms
[I 2020-06-16 07:52:23.241 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.85ms
[I 2020-06-16 07:52:23.241 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.89ms
[I 2020-06-16 07:52:23.242 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.84ms
[I 2020-06-16 07:52:23.242 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.80ms
[I 2020-06-16 07:52:23.242 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 6.79ms
[I 2020-06-16 07:52:23.244 JupyterHub proxy:320] Checking routes
[D 2020-06-16 07:52:24.812 JupyterHub proxy:766] Proxy: Fetching GET http://10.102.23.122:8001/api/routes
[I 2020-06-16 07:52:24.813 JupyterHub proxy:320] Checking routes
[I 2020-06-16 07:52:26.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.78ms
[I 2020-06-16 07:52:36.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.70ms
[I 2020-06-16 07:52:46.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.96ms
[I 2020-06-16 07:52:56.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.74ms
[I 2020-06-16 07:53:06.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.67ms
[I 2020-06-16 07:53:16.136 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.87ms
[D 2020-06-16 07:53:24.811 JupyterHub proxy:766] Proxy: Fetching GET http://10.102.23.122:8001/api/routes
[I 2020-06-16 07:53:24.813 JupyterHub proxy:320] Checking routes
[I 2020-06-16 07:53:26.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.66ms
[I 2020-06-16 07:53:36.135 JupyterHub log:174] 200 GET /hub/health (@192.168.126.221) 0.71ms
root@server11:~/locuz#That error message suggests JupyterHub is unable to connect to your LDAP server- you’ll have to figure out why.
Thank you for your support and guidance.
After resolving LDAP Server connectivity error, the permission issue is popping up
hub.yaml as per the below link
wget https://gist.githubusercontent.com/manics/c4bcf53a210d444db9e64db7673e8580/raw/d07ed5f30c4dbc3f1a632899f2fe58b98799b674/zero-to-jupyterhub-config.yml
Thank You
Atul
Only update the hub section for “c.KubeSpawner.cmd”
jupyterhub/k8s-singleuser-sample 0.9.0 854de563978d 2 months ago 676MB
jupyterhub/k8s-image-awaiter 0.9.0 7b10713c8231 2 months ago 4.15MB
jupyterhub/k8s-network-tools 0.9.0 00c5e7b520c2 2 months ago 5.62MB
jupyterhub/k8s-hub 0.9.0 2f82dac5c3ab 2 months ago 632MB
#Jupyterhub Master And LDAP Client
user5@server11:~$ id
uid=1004(user5) gid=1004(user5) groups=1004(user5)
user5@server11:~$
#Test Case 1: c.KubeSpawner.cmd = ['/usr/local/bin/start.sh', 'jupyterhub-singleuser']
Set username to: user5
Setting CWD to /home/user5/
Set user user5 UID:GID to: 1004:100
Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --NotebookApp.default_url=/lab --debug
[D 2020-06-17 10:56:40.970 SingleUserNotebookApp application:178] Searching ['/home/user5', '/home/user5/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2020-06-17 10:56:40.971 SingleUserNotebookApp application:556] Looking for jupyter_config in /etc/jupyter
[D 2020-06-17 10:56:40.971 SingleUserNotebookApp application:556] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2020-06-17 10:56:40.971 SingleUserNotebookApp application:556] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2020-06-17 10:56:40.971 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/user5/.jupyter
[D 2020-06-17 10:56:40.971 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/user5
[D 2020-06-17 10:56:40.972 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /etc/jupyter
[D 2020-06-17 10:56:40.973 SingleUserNotebookApp application:578] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2020-06-17 10:56:40.973 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2020-06-17 10:56:40.973 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2020-06-17 10:56:40.973 SingleUserNotebookApp application:578] Loaded config file: /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2020-06-17 10:56:40.973 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/user5/.jupyter
[D 2020-06-17 10:56:40.973 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/user5
[W 2020-06-17 10:56:40.975 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`. Did you mean `browser`?
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'runtime_dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/jupyterhub-singleuser", line 10, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 660, in main
return SingleUserNotebookApp.launch_instance(argv)
File "/opt/conda/lib/python3.7/site-packages/jupyter_core/application.py", line 268, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 663, in launch_instance
app.initialize(argv)
File "/opt/conda/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 558, in initialize
return super().initialize(argv)
File "</opt/conda/lib/python3.7/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize
File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/notebook/notebookapp.py", line 1766, in initialize
self.init_configurables()
File "/opt/conda/lib/python3.7/site-packages/notebook/notebookapp.py", line 1380, in init_configurables
connection_dir=self.runtime_dir,
File "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/opt/conda/lib/python3.7/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
ensure_dir_exists(rd, mode=0o700)
File "/opt/conda/lib/python3.7/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
os.makedirs(path, mode=mode)
File "/opt/conda/lib/python3.7/os.py", line 211, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/opt/conda/lib/python3.7/os.py", line 211, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/opt/conda/lib/python3.7/os.py", line 211, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/opt/conda/lib/python3.7/os.py", line 221, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/user5/.local'
#Test Case 2: c.KubeSpawner.cmd = ['/usr/local/bin/start.sh', 'jupyter-labhub']
Set username to: user5
Setting CWD to /home/user5/
Set user user5 UID:GID to: 1004:100
Executing the command: jupyter-labhub --ip=0.0.0.0 --port=8888 --NotebookApp.default_url=/lab --debug
[D 2020-06-17 10:59:45.246 SingleUserLabApp application:178] Searching ['/home/user5', '/home/user5/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2020-06-17 10:59:45.247 SingleUserLabApp application:556] Looking for jupyter_config in /etc/jupyter
[D 2020-06-17 10:59:45.247 SingleUserLabApp application:556] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2020-06-17 10:59:45.247 SingleUserLabApp application:556] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2020-06-17 10:59:45.247 SingleUserLabApp application:556] Looking for jupyter_config in /home/user5/.jupyter
[D 2020-06-17 10:59:45.247 SingleUserLabApp application:556] Looking for jupyter_config in /home/user5
[D 2020-06-17 10:59:45.248 SingleUserLabApp application:556] Looking for jupyter_notebook_config in /etc/jupyter
[D 2020-06-17 10:59:45.248 SingleUserLabApp application:578] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2020-06-17 10:59:45.248 SingleUserLabApp application:556] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2020-06-17 10:59:45.249 SingleUserLabApp application:556] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2020-06-17 10:59:45.249 SingleUserLabApp application:578] Loaded config file: /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2020-06-17 10:59:45.249 SingleUserLabApp application:556] Looking for jupyter_notebook_config in /home/user5/.jupyter
[D 2020-06-17 10:59:45.249 SingleUserLabApp application:556] Looking for jupyter_notebook_config in /home/user5
[W 2020-06-17 10:59:45.251 SingleUserLabApp configurable:168] Config option `open_browser` not recognized by `SingleUserLabApp`. Did you mean `browser`?
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'runtime_dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/jupyter-labhub", line 10, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.7/site-packages/jupyterlab/labhubapp.py", line 42, in main
return SingleUserLabApp.launch_instance(argv)
File "/opt/conda/lib/python3.7/site-packages/jupyter_core/application.py", line 268, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 663, in launch_instance
app.initialize(argv)
File "/opt/conda/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 558, in initialize
return super().initialize(argv)
File "</opt/conda/lib/python3.7/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize
File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/notebook/notebookapp.py", line 1766, in initialize
self.init_configurables()
File "/opt/conda/lib/python3.7/site-packages/notebook/notebookapp.py", line 1380, in init_configurables
connection_dir=self.runtime_dir,
File "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/opt/conda/lib/python3.7/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
ensure_dir_exists(rd, mode=0o700)
File "/opt/conda/lib/python3.7/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
os.makedirs(path, mode=mode)
File "/opt/conda/lib/python3.7/os.py", line 211, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/opt/conda/lib/python3.7/os.py", line 211, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/opt/conda/lib/python3.7/os.py", line 211, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/opt/conda/lib/python3.7/os.py", line 221, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/user5/.local'
#Describe OutPut
Name: jupyter-user5
Namespace: jhub
Priority: 0
Node: server11/192.168.126.221
Start Time: Wed, 17 Jun 2020 16:53:33 +0530
Labels: app=jupyterhub
chart=jupyterhub-0.9.0
component=singleuser-server
heritage=jupyterhub
hub.jupyter.org/network-access-hub=true
release=jhub
Annotations: cni.projectcalico.org/podIP: 172.31.0.182/32
hub.jupyter.org/username: user5
Status: Running
IP: 172.31.0.182
Init Containers:
block-cloud-metadata:
Container ID: docker://e90e6c230368570dbc5dc24e374d0581632329950f7908eaf555ff6365cdfc23
Image: jupyterhub/k8s-network-tools:0.9.0
Image ID: docker-pullable://jupyterhub/k8s-network-tools@sha256:120056e52fef309132697d405683a91ce6e2e484b67b7fd4bd7ca5e7d1937a34
Port: <none>
Host Port: <none>
Command:
iptables
-A
OUTPUT
-d
169.254.169.254
-j
DROP
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 17 Jun 2020 16:53:34 +0530
Finished: Wed, 17 Jun 2020 16:53:34 +0530
Ready: True
Restart Count: 0
Environment: <none>
Mounts: <none>
Containers:
notebook:
Container ID: docker://6b27dde8ff4abbf817c056e06c8e81b40bd61011055963fa8cb1eea147512952
Image: jupyterhub/k8s-singleuser-sample:0.9.0
Image ID: docker-pullable://jupyterhub/k8s-singleuser-sample@sha256:a500eeb50bd20c29478cfaa6cc3699232bf42534d673ff7474ebc3cab37d02fb
Port: 8888/TCP
Host Port: 0/TCP
Args:
/usr/local/bin/start.sh
jupyterhub-singleuser
--ip=0.0.0.0
--port=8888
--debug
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 17 Jun 2020 16:53:37 +0530
Finished: Wed, 17 Jun 2020 16:53:38 +0530
Ready: False
Restart Count: 1
Environment:
NB_UID: 1004
NB_GID: 1004
NB_USER: user5
NB_GROUP: user5
JUPYTERHUB_API_TOKEN: 59a1b0ef5930412a8677c7b184af2e67
JPY_API_TOKEN: 59a1b0ef5930412a8677c7b184af2e67
JUPYTERHUB_ADMIN_ACCESS: 1
JUPYTERHUB_CLIENT_ID: jupyterhub-user-user5
JUPYTERHUB_HOST:
JUPYTERHUB_OAUTH_CALLBACK_URL: /user/user5/oauth_callback
JUPYTERHUB_USER: user5
JUPYTERHUB_SERVER_NAME:
JUPYTERHUB_API_URL: http://10.111.186.45:8081/hub/api
JUPYTERHUB_ACTIVITY_URL: http://10.111.186.45:8081/hub/api/users/user5/activity
JUPYTERHUB_BASE_URL: /
JUPYTERHUB_SERVICE_PREFIX: /user/user5/
JUPYTER_IMAGE_SPEC: jupyterhub/k8s-singleuser-sample:0.9.0
JUPYTER_IMAGE: jupyterhub/k8s-singleuser-sample:0.9.0
Mounts:
/home/user5 from nfs-home-pvc (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
nfs-home-pvc:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: nfs-home-pvc
ReadOnly: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: hub.jupyter.org/dedicated=user:NoSchedule
hub.jupyter.org_dedicated=user:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9s jhub-user-scheduler Successfully assigned jhub/jupyter-user5 to server11
Normal Pulled 8s kubelet, server11 Container image "jupyterhub/k8s-network-tools:0.9.0" already present on machine
Normal Created 8s kubelet, server11 Created container block-cloud-metadata
Normal Started 8s kubelet, server11 Started container block-cloud-metadata
Normal Pulled 5s (x2 over 7s) kubelet, server11 Container image "jupyterhub/k8s-singleuser-sample:0.9.0" already present on machine
Normal Created 5s (x2 over 7s) kubelet, server11 Created container notebook
Normal Started 5s (x2 over 7s) kubelet, server11 Started container notebook
Warning BackOff 2s (x2 over 3s) kubelet, server11 Back-off restarting failed container