Customizable DNS Port

JupyterHub Zero to JupyterHub on Kubernetes
1

Hey there,

I am running JupyterHub on OpenShift, which requires some adjustments, but it works without modifying the JupyterHub Helm chart.

As pointed out by @Will_Holtam, OpenShift’s local DNS server runs on port 5353 instead of port 53.
The current workaround is to use kustomize to patch the ports.

The ports are specified in a named template.
I’m not that familiar with Helm but is it currently possible to set the ports via Helm values? And if not, do you think it should be made customizable since propably all OpenShift deployments have to patch the ports outside JupyterHub’s Helm chart?

Best regards
Paul

2

You shouldn’t need to patch the chart, you can add an additional *.networkPolicy.egress to allow the extra port:

What would be useful to know is if there’s anything required for OpenShift that can’t be configured using the existing values.yaml.

3

You are absolutely right; it works without issues.

I’m already documenting my journey on deploying JupyterHub on an OpenShift cluster. I’m happy to share my findings once the deployment is completed, which should be in the next couple of weeks.

4

That’d be great! Kubernetes on Red Hat OpenShift — Zero to JupyterHub with Kubernetes documentation links to GitHub - gembaadvantage/z2jh-openshift: A kustomization of the z2jh project so that it works with OpenShift. which has several kustomize patches, but it’s not clear to me whether all of those changes can also be made using the existing values.yaml parameters.

5

Sorry for taking so long to get back to you.

To deploy JupyterHub using Z2JH on OpenShift, the following Kubernetes manifests and configurations are necessary.

Create Service Account for KubeSpawner

The JupyterHub uses the spawner KubeSpawner to run singleuser pods for users. Due to OpenShift’s Security Context Constraints (SCC), OpenShift runs pods with random user IDs (UID). Since the singleuser pod expects to run as user jovyan, a new service account is necessary that has permission to run pods with any UID.

Thus, the spawner requires a service account, which can be created with the following manifest. It creates a new service account called kubespawner with the permission anyuid.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubespawner
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubespawner
rules:
  - verbs:
      - use
    apiGroups:
      - security.openshift.io
    resources:
      - securitycontextconstraints
    resourceNames:
      - anyuid
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubespawner
subjects:
  - kind: ServiceAccount
    name: kubespawner
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubespawner

Setup Helm Values

The following config.yaml sets the Helm values of JupyterHub’s Helm chart. It only contains the OpenShift-related configurations and is therefore not a minimal POC.

# Default values (ensure that the correct tag is selected):
# https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/2421cc409465aea9f3e35e20f6bf4f3bc4e60064/jupyterhub/values.yaml
# https://z2jh.jupyter.org/en/latest/resources/reference.html

hub:
  # Since OpenShift runs with random user and group IDs, the IDs in <...>.containerSecurityContext are set to null so that OpenShift can set the values.
  containerSecurityContext:
    runAsGroup: null
    runAsUser: null
  config:
    KubeSpawner:
      # ServiceAccount "kubespawner" has permission to run images with any UID
      # and can set the filesystem group ID to be able to create files in the volumes.
      service_account: kubespawner
      fs_gid: 100
      # Since the KubeSpawner uses a ServiceAccount, the token is mounted by default
      automount_service_account_token: False
  podSecurityContext:
    fsGroup: null

prePuller:
  containerSecurityContext:
    runAsGroup: null
    runAsUser: null
  hook:
    containerSecurityContext:
      runAsGroup: null
      runAsUser: null
  pause:
    containerSecurityContext:
      runAsGroup: null
      runAsUser: null

proxy:
  chp:
    containerSecurityContext:
      runAsGroup: null
      runAsUser: null
  secretSync:
    containerSecurityContext:
      runAsGroup: null
      runAsUser: null
  traefik:
    containerSecurityContext:
      runAsGroup: null
      runAsUser: null

scheduling:
  userPlaceholder:
    containerSecurityContext:
      runAsGroup: null
      runAsUser: null
  userScheduler:
    containerSecurityContext:
      runAsGroup: null
      runAsUser: null

singleuser:
  fsGid: null
  # Since OpenShift is not a cloud provider like AWS or Azure, the metadata block (singleuser.cloudMetadata.blockWithIptables) is disabled. This comes in handy because this option results in an init container that requires elevated privileges, which is by default disallowed by OpenShift.
  cloudMetadata:
    blockWithIptables: false
  networkPolicy:
    egress:
      # OpenShift's local DNS server runs on port 5353, not the default port 53. Thus, the singleuser pod cannot reach the hub pod. Therefore, a new egress rule in singleuser.networkPolicy.egress is created by copying the default policy for port 53 and adopting it to port 5353.
      - ports:
          - protocol: UDP
            port: 5353
          - protocol: TCP
            port: 5353
        to:
          # Copied values from generated NetworkPolicy,
          # not sure where the IP addresses come from...
          - ipBlock:
              cidr: <...>
          - namespaceSelector:
              matchLabels:
                kubernetes.io/metadata.name: kube-system
          - ipBlock:
              cidr: <...>
          - ipBlock:
              cidr: <...>
          - ipBlock:
              cidr: <...>

I hope this excerpt may help others in the future as well.

6

Thanks, hopefully that helps others!

Communication between the singleuser pods and the hub/proxy is managed by pod labels (matchLabels):

so the ipBlocks are to support non-JupyterHub traffic. For example, by default we block egress from singleuser pods to the IPv4 Private Address Space but allow egress to public IPs, but this is configurable by the admin.