Invalid credentials to notify hub of activity


I have a problem to notify the Hub of activity from a single user server (log below) It might be related to this issue but I am not sure.

On the user side, the hub seems to be working, they can spawn new servers, kernels therein etc. Some user report that sometimes their progress is not saved correctly (which I assume is related to the issue above).

Our setting is a JupyterHub spawned from a conda environment
with minimal packages (mamba create --name jhub jupyterhub sudospawner), while the single-user server is spawned from a production environment (mamba create --name singleuserenv jupyterhub-singleuser jupyterlab production_packages). I am using the sudospawner with the following sudospawner-singleuser file

#!/bin/bash -l
set -e

# Activate the notebook environment
module load singleuserenv

exec python "$(dirname "$0")/jupyterhub-singleuser" $@

The resulting log is full of:

[W 2024-06-04 09:36:40.422 JupyterHub web:1873] 403 POST /hub/api/users/nsiemer/activity ( Missing or invalid credentials.
[W 2024-06-04 09:36:40.425 JupyterHub log:192] 403 POST /hub/api/users/nsiemer/activity (@ 8.90ms

Does anyone know how to fix this?

If you delete module load singleuserenv and instead run /full/path/to/bin/jupyterhub-singleuser to you see a difference in behaviour?