Restricted PSP: z2jh fails in local gitlab oauth workflow

JupyterHub Zero to JupyterHub on Kubernetes
,
1

Hello Everyone,

So, I have managed to get z2jh working with a local GitLab using a permissive PSP and a custom JupyterHub image that runs as root.

Here is the environment I used to get it working:

PSP used:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: jupyterhub
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default'
    apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  'runtime/default'
    apparmor.security.beta.kubernetes.io/defaultProfileName:  'runtime/default'
spec:
  privileged: false
  # Required to prevent escalations to root.
#  allowPrivilegeEscalation: false
  # This is redundant with non-root + disallow privilege escalation,
  # but we can provide it for defense in depth.
  allowedCapabilities:
    - NET_ADMIN
#    - SETGID
  # Allow core volume types.
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    # Assume that persistentVolumes set up by the cluster admin are safe to use.
    - 'persistentVolumeClaim'
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    # Require the container to run without root privileges.
    rule: 'RunAsAny'
  seLinux:
    # This policy assumes the nodes are using AppArmor rather than SELinux.
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  fsGroup:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  readOnlyRootFilesystem: false

ClusterRoleBindings

I followed this up with a ClusterRoleBinding, as a normal RoleBinding restricting it to the jupyter ns did not work.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp:jupyterhub
rules:
- apiGroups:
  - policy
  resources:
  - podsecuritypolicies
  resourceNames:
  - jupyterhub # the psp we are giving access to
  verbs:
  - use
---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp:jupyterhub
subjects:
- kind: ServiceAccount
  name: default
  namespace: jupyter
- kind: ServiceAccount
  name: hook-image-awaiter
  namespace: jupyter
- kind: ServiceAccount
  name: hub
  namespace: jupyter
- kind: ServiceAccount
  name: user-scheduler
  namespace: jupyter
roleRef:
  kind: ClusterRole
  name: psp:jupyterhub
  apiGroup: rbac.authorization.k8s.io

Additional settings

I also had to allow the hub image to run as the root user. This was necessary cause I could not find a work around to not allowing the c.LocalAuthenticator.create_system_users: True

Failures

Any attempts on reducing these permission or privileges causes the singleuser nb to fail.

As of now, the entire setup does not seem very secure. Here are the two questions I have:

  1. I was curious as to how is this handled when you use the GitHub or other supported OAuth modules? Could not find any documentation on this.
    a. https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1312
  2. Can we somehow get around all this root and privileged containers in z2jh? Especially for on-premise deployments?
    a. https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1491

Any pointers or references would be appreciated. Cheers!

2

Hi, welcome to the forum!

Can you let us know what you’ve tried from that issue, and what did or didn’t work? If you could show us your Z2JH config that would also be helpful.

3

Hi Simon,

Thank you!

So, I gave it another shot and after carefully reading through and understanding the various components, I got it to work (mostly) without any issues.

Since it is a local installation, I enabled network policy and the cloudMetadata option. This removed the init container for singleuser images as mentioned on the GitHub issue.

I disabled the hooks as I have set the imagePullPolicy to Always, as it makes sense to do this at a cluster level to ensure patched images are used. The hooks seem redundant in this scenario, correct?

Also, I disabled the LocalGitLabOAuthenticatior after reading about it and enabled the GitLabOAuthenticator, this removed the need for running Hub as a root user as the users are no longer added to the local system.

What did not work out-of-the-box was the user-placeholder pod. Since the image used is the gcr.io/google_containers/pause:3.1 image, it runs as root by default. Adding a SecurityContext block to run the image as a non-root user did the trick. Can that be added in the conf file here? I don’t think anyone would need to override this value, so this could be hardcoded? You could also change the image to a custom image, but I think it would make sense to just add this as a default value to avoid a few hassles.

Here is the final z2js config that worked with the restricted psp:

auth:
  admin:
    users:
      - nerdsec

proxy:
  secretToken: "lolbins"
  service:
    loadBalancerIP: 10.x.x.x
  https:
    enabled: true
    type: secret
    secret:
      name: z2jh

singleuser:
  cloudMetadata:
    enabled: true
    ip: 169.254.169.254
  networkPolicy:
    enabled: true
    egress:
    # Required egress is handled by other rules so it's safe to modify this
      - to:
          - ipBlock:
              cidr: 0.0.0.0/0
              except:
                - 169.254.169.254/32
  image:
    name: "custom/jupyter/singleuser"
    tag: "0.9.0"
  imagePullSecret:
    enabled: true
    registry: "megusta"
    username: "xxx"
    password: "xxx"
  extraEnv:
    EDITOR: "nano"
  defaultUrl: "/lab"
  memory:
    limit: 4G
  cpu:
    limit: 2
  storage:
    capacity: 2Gi
  extraSecurityContext:
    runAsUser: 1000
    runAsGroup: 1000
    allowPrivilegeEscalation: false


hub:
  image:
    name: "custom/jupyter/hub"
    tag: "0.9.0"
    pullSecrets:
      - secret
  extraEnv:
    GITLAB_HOST: "https://me.gusta.com"
  extraConfig:
    myconfig: |
      c.JupyterHub.authenticator_class = 'oauthenticator.gitlab.LocalGitLabOAuthenticator'
      c.GitLabOAuthenticator.oauth_callback_url = "https://me.gusta.com/hub/oauth_callback"
      c.GitLabOAuthenticator.client_id = "lolBins"
      c.GitLabOAuthenticator.client_secret = "lolBins"

cull:
  enabled: true
  timeout: 3600
  every: 300

prePuller:
  hook:
    enabled: false
  continuous:
    enabled: false

scheduling:
  userScheduler:
    enabled: true
    replicas: 2
    logLevel: 4
  userPlaceholder:
    enabled: true
    replicas: 2
  podPriority:
    enabled: true

Is it okay if I add this to the docs? There seems to be no section on PSP in https://zero-to-jupyterhub.readthedocs.io.

Cheers!

4

That makes sense, I can’t think of any reason the pause container has to run as root. Please feel free to open a PR!

Yes, I think that would be helpful. Please open it as a separate PR and cross-reference this thread as there may be some discussions about it whereas the pause PR should be straightforward. Thanks for contributing! :grinning:

5

Sure! Will do! :slight_smile:

1 Like