How am I supposed to use `existingSecret`

JupyterHub Zero to JupyterHub on Kubernetes
, ,
1

Hi!

I am deploying JupyterHub on a Kubernetes cluster. The authentication is done using Azure AD.
The config.yaml is version controlled in Git, and I would like to get rid of any references to the AzureAD client secrets. That is

auth:
  azuread:
    callbackUrl: ...
    clientId: ...
    clientSecret: ...
    tenantId: ...

type: azuread

I have tried to navigate previous posts on this issue, to no avail. I know of the concept of existingSecret, however there are no references to how exactly you are supposed to use this.

Could anyone provide some insight on how I can expose the AzureAD authentication secrets as a kubernetes secret, to avoid having to commit this to VCS?

Cheers in advance :smile:

2

I don’t know about existingSecret, but I stored my secrets in an Azure Key Vault, then used an Azure Pipeline to populate my config file and upgrade the cluster when I pushed to the default branch: https://www.github.com/alan-turing-institute/hub23-deploy/tree/main/.az-pipelines%2Fcd-pipeline.yml (This is for a BinderHub, but it’s dependent on the z2jh-k8s chart).

More documentation here: https://alan-turing-institute.github.io/hub23-deploy

1 Like
3

Thanks for the tips! I’ll take a look.

I’d still love some advice on the existingSecret, which was introduced in Z2JH 0.9, if I recall correctly.

4

The configuration reference includes a cautionary note.

NOTE: if you choose to manage the secret yourself, you are in charge of ensuring the secret having the proper contents.

We merged this PR, but my personal opinion is that it was introducing something that we can’t really maintain and will lead too quite a bit of complexity for users to use it.

I want to make it sustainable to use, but in my mind, it currently isn’t. One need to make sure the content is updated at all time with a lot of configuration that change between versions and your own configuration in general. Doing that in turn requires a lot of what I consider unsustainable maintenance such as helm template <...> --show-only templates/hub/secret.yaml --set hub.existingSecret="" which you then manually apply in your k8s cluster to some k8s secret, and then you update your hub.existingSecret to reference that again.

Making this sustainable is something reflected in this issue:

1 Like
5

I’ve now opened an PR to make use of hub.existingSecret more sustainable, see:

6

The problem with this solution is that on Kubernetes the secret will be put in plain text in a config map whereas it really should end up in a Kubernetes secret.

7

Stil not sure how I can use this functionality for auth.azuread.clientSecret as in the original question. Could you add an example, please?

8

Note that any configuration under auth is deprecated in 0.11.1, but if used you will get an error that should be somewhat informative on how to configure.

There should not be any issues with using hub.existingSecret, or hub.config.<SomeAuthenticatorClass>.<some_config_option> to configure your authenticator class in the latest versions of the z2jh helm chart (0.11.1-n361.h166e2157 currently) at this point.

Documentation about authentication:

https://zero-to-jupyterhub.readthedocs.io/en/latest/administrator/authentication.html

Documentation about hub.config:

https://zero-to-jupyterhub.readthedocs.io/en/latest/resources/reference.html#hub-config

Documentation about hub.existingSecret:

https://zero-to-jupyterhub.readthedocs.io/en/latest/resources/reference.html#hub-existingsecret

9

Thank you that was very helpful. I am still using 0.9 for now but shall upgrade soon to new version.

10

One year anniversary update.

Thanks for all the replies, and sorry for my lack of engagement. I have revisited the issue, and after some hacking around I finally managed to get it to work.

In the following I will outline how to set up authentication (AAD in this case) in a VCS-friendly way (using existingSecret). This has also been updated to the new version of Z2JH (using helm chart version 1.1.0 at the time of writing).

Restating the problem

When setting up authentication we need to specify the following in the values.yaml-file:

# in values.yaml
hub:
  config:
     AzureAdOAuthenticator:
       client_id: <my secret client_id>
       client_secret: <my secret client secret>
       tenant_id: <my tenant id>
       oauth_callback_url: <callback url>
    JupyterHub:
       authenticator_class: azuread

Some of these values are things you do not want to commit to VCS’ like GitHub.

The Solution

To solve this problem, we can use the existingSecret keyword in values.yaml and let Jupyter Hub pull configuration settings from an already existing K8s secret. To me the documentation is still lacking in providing a clear example on how to do this, especially on how exactly the secret should be set up.

After some trial and error, I realised that the existing secret has to mirror exactly the values.yaml-syntax. And that the config residing in the secret, and the config residing in the original values.yaml-file will be merged.

Thus, for me, the solution was to create a my_secret_aad_config.yaml containing the authentication settings from above:

# in my_secret_aad_config.yaml
hub:
  config:
     AzureAdOAuthenticator:
       client_id: <my secret client_id>
       client_secret: <my secret client secret>
       tenant_id: <my tenant id>
       oauth_callback_url: <callback url>
    JupyterHub:
       authenticator_class: azuread

Then creating a kubernetes secret from this file:

kubectl create secret generic secret-aad-config --from-file=values.yaml=my_secret_aad_config.yaml -n <namespace jupyterhub deployed to>

(I then delete the file my_secret_aad_config.yaml as this is no longer needed).

Then, in values.yaml (which is the config commited to GitHub), we remove the auth-config, and reference the existing secret as such:

# in values.yaml
hub:
  existingSecret: secret-aad-config # name of the k8s secret
  # Removed the config set in the secret

Verify the solution

I was struggling to figure out how to verify that the hub properly picks up on the existing secret. However, when the hub-pod is being deployed, the first few lines of logs are:

Loading /usr/local/etc/jupyterhub/secret/values.yaml                                                                                                                                                             
Loading /usr/local/etc/jupyterhub/existing-secret/values.yaml

Documentation

I had to spend a fair bit of time figuring this out, and I think that the documentation could greatly benefit from a walk-through example of how to use this feature.

4 Likes
11

You have done an amazing job clarifying this! Want to copy paste this into an issue at the zero-to-juoyterhub-k8s repo?

It is a concrete action point on what can help the docs evolve making it suitable as an issue in tve repo!

Thanks for being so thorough!

12

Thanks!

I have submitted an issue.

1 Like