Kubernetes python module not found in custom jupyterhub container for custom authentication


We are looking into allowing our customers to test out the software we distribute through Jupyterhub hosted in Azure (kubernetes). Generally, I am very happy with the guide “zero to kubernetes” which I used to set up most of the infrastructure, without much prior experience working with kubernetes, network, azure etc, so kudos for that!. However, I am struggling to set up authentication.

Our current wish is to use Azure ADB2C. I can see that OAuthenticator had support for this at some point in late 2019, but it was removed on the grounds that you could use the Azure AD Authenticator for this. However, when I tried the Azure AD auth I end up setting up the auth to use “regular AAD”. By looking into the endpoints used by the ADB2C tenant I see that this makes sense as Azure have recently moved B2C tenants to new endpoints and the “old” ones now refer to “regular AAD”. In the AAD authenticator code the old style endpoints are hard-coded so this cannot currently be used for new B2C tenants, and not for any B2C tenants some time after 2021.01.14 (see OAuthenticator github issue). Based on replies from the jupyterhub team in topics/issues regarding specific/custom authenticators, I decided to create my own B2C authenticator in which I simply adapt the code mentioned above by updating the endpoints.

My approach is to extend the jupyterhub hub-image by installing my own authenticator and deploy this through specifying this hub-image in the chart config file: “hub.image.name/tag” as well as the authenticator: “auth.custom.className”, inspired by the topic with the title “Adding authenticators, spawners or other custom pieces in the Z2JH setup” (I was only allowed two links). However, after running helm upgrade, the hub pod gets stuck in CrashLoopBack. When I print the logs, I see that it fails with the error “ModuleNotFoundError: No module named ‘kubernetes’” (python error) during the loading of the config file. My derived Dockerfile is shown below where the copy step copies in the custom python package:

FROM jupyterhub/jupyterhub:1.1.0
RUN mkdir -p /srv/auth/
WORKDIR /srv/auth/
COPY . .
RUN python3 -m pip install --no-cache .
WORKDIR /srv/jupyterhub/

I interpret this as if my added steps somehow invalidates the python environment, but I have no idea how or why. Does anyone have any ideas what may cause this issue? And please let me know if you see any flaws in my approach or know of a better way!

Thank you :slight_smile:

Hi! Please could you show us your Z2JH config file, with secrets redacted? Thanks!

It is fairly simple at the moment so I don’t know if you will be able to figure something out by looking at it. Perhaps I have misunderstood how you package/refer to custom authenticators?

Some additional notes: The Azadb2cAuthentication is, as mentioned, just a copy of the azuread.py which I put into the dummyauthenticator package template and replaced the “login.microsoftonline.com” part of the authorize and token urls with “B2CAPPNAME.b2clogin.com”

Thanks for the reply!

I’ve just realised you’re using the standard JupyterHub Docker image, which doesn’t contain the required additions to run as part of Z2JH.

The Z2JH image is https://hub.docker.com/r/jupyterhub/k8s-hub/tags
You could try extending that instead? Note the paths are different, see https://github.com/jupyterhub/zero-to-jupyterhub-k8s/tree/master/images/hub

1 Like

Doh, that makes a lot of sense. After extending the k8s-hub image I no longer get the missing kubernetes module error. Sadly, it did not get me all the way as I now have an issue with loading in my custom authenticator. Hopefully, I’ll manage to figure how to fix this myself after inspecting the images some more and diving into the forum. Otherwise, I’ll likely create a new topic when I feel sufficiently stuck again :sweat_smile:

Thank you very much for your assistance!

1 Like