Hello together,
I have extended the Jupyterhub hub using a Dockerfile. I am attaching the code below.
For a while I had no problems with it. Then I had to reset and rebuild the Kubernetes cluster. The jhub_remote_user_authenticator used to be recognized, but now I get the following error message. (See below)
I changed jhub_remote_user_authenticator a bit, why it is copied locally and then installed.
I hope you guys can help me further. Thanks a lot in advance.
kubectl -n jhub logs hub-65f9bb8d6-4cll5
[D 2022-10-05 08:26:23.362 JupyterHub application:837] Looking for /usr/local/etc/jupyterhub/jupyterhub_config in /srv/jupyterhub
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: volume.py
[D 2022-10-05 08:26:23.641 JupyterHub application:858] Loaded config file: /usr/local/etc/jupyterhub/jupyterhub_config.py
[C 2022-10-05 08:26:23.693 JupyterHub application:112] Bad config encountered during initialization: The 'authenticator_class' trait of <jupyterhub.app.JupyterHub object at 0x7f8551791910> instance must be a type, but 'jhub_remote_user_authenticator.remote_user_auth.RemoteUserAuthenticator' could not be imported
[D 2022-10-05 08:26:23.693 JupyterHub application:113] Config at the time: {'JupyterHub': {'config_file': '/usr/local/etc/jupyterhub/jupyterhub_config.py', 'log_level': 10, 'upgrade_db': True}}
[D 2022-10-05 08:26:23.693 JupyterHub application:963] Exiting application: jupyterhub
Dockerfile
ARG JUPYTERHUB_K8S_VERSION=1.1.3-n721.h0acbcf43
ARG BASE_IMAGE=jupyterhub/k8s-hub:${JUPYTERHUB_K8S_VERSION}
FROM ${BASE_IMAGE}
USER root
RUN mkdir -p /jhub_remote_user_authenticator
COPY jhub_remote_user_authenticator/ /jhub_remote_user_authenticator/
RUN pip3 install --upgrade --force-reinstall /jhub_remote_user_authenticator
RUN mkdir -p /templates
COPY templates/* /templates/
USER ${NB_USER}
Can you tell us exactly what’s changed between the two deployments? For example are you using the same container image or did you rebuild it? If you rebuilt it are you using the same versions of all components including jhub_remote_user_authenticator or have you updated something?
No, I haven’t actually changed anything, at least I wouldn’t know. I use the same yaml files with the same versions. The only thing that changed is that I don’t build the docker image locally anymore, I build it on gitlab now.
It sounds like jhub_remote_user_authenticator wasn’t installed properly. You could try running the Docker container manually, and check whether it’s importable.
I checked everything again and remembered that I had to change setup.py once so that I can build the Docker image locally and use jhub_remote_user_authenticator. I undid the change and the authenticator is recognised.
But now I have a different error message.
Should I create a new thread for this one because I’m stuck there too?
[D 2022-10-08 08:22:38.496 JupyterHub application:837] Looking for /usr/local/etc/jupyterhub/jupyterhub_config in /srv/jupyterhub
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: volume.py
[D 2022-10-08 08:22:38.788 JupyterHub application:858] Loaded config file: /usr/local/etc/jupyterhub/jupyterhub_config.py
[I 2022-10-08 08:22:38.847 JupyterHub app:2771] Running JupyterHub version 2.3.1
[I 2022-10-08 08:22:38.847 JupyterHub app:2801] Using Authenticator: jhub_remote_user_authenticator.remote_user_auth.RemoteUserAuthenticator
[I 2022-10-08 08:22:38.847 JupyterHub app:2801] Using Spawner: kubespawner.spawner.KubeSpawner-4.1.0
[I 2022-10-08 08:22:38.847 JupyterHub app:2801] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.3.1
[D 2022-10-08 08:22:38.848 JupyterHub app:1775] Connecting to db: sqlite:///jupyterhub.sqlite
[I 2022-10-08 08:22:38.861 JupyterHub dbutil:130] Upgrading sqlite:///jupyterhub.sqlite
[I 2022-10-08 08:22:38.861 JupyterHub dbutil:99] Backing up jupyterhub.sqlite => jupyterhub.sqlite.2022-10-08-082238
[I 2022-10-08 08:22:39.250 alembic.runtime.migration migration:204] Context impl SQLiteImpl.
[I 2022-10-08 08:22:39.251 alembic.runtime.migration migration:207] Will assume non-transactional DDL.
[E 2022-10-08 08:22:39.254 alembic.util.messaging messaging:65] Can't locate revision identified by '651f5419b74d'
FAILED: Can't locate revision identified by '651f5419b74d'
[E 2022-10-08 08:22:39.303 JupyterHub app:3297]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 3294, in launch_instance_async
await self.initialize(argv)
File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 2813, in initialize
self.init_db()
File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 1777, in init_db
dbutil.upgrade_if_needed(self.db_url, log=self.log)
File "/usr/local/lib/python3.9/site-packages/jupyterhub/dbutil.py", line 135, in upgrade_if_needed
upgrade(db_url)
File "/usr/local/lib/python3.9/site-packages/jupyterhub/dbutil.py", line 84, in upgrade
check_call(['alembic', '-c', alembic_ini, 'upgrade', revision])
File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['alembic', '-c', '/tmp/tmpl6mcn2a1/alembic.ini', 'upgrade', 'head']' returned non-zero exit status 255.
[D 2022-10-08 08:22:39.304 JupyterHub application:961] Exiting application: jupyterhub
Sounds like something has gone wrong with your database. Can you try running your server without an existing database (so a new one is created), if that works then we can try and work out what’s gone wrong during the upgrade.
RUN pip3 install --upgrade --force-reinstall /jhub_remote_user_authenticator
I suspect this caused jupyterhub to be upgraded because it was a dependency somewhere, and then you had issues related to the switch to a new JupyterHub version.