update: we turned on more verbose logs and discovered we needed CA certificates among other things installed. The following was run
sudo apt-get update
sudo apt-get install -y ca-certificates
sudo mkdir -p /etc/pki/tls/certs
sudo ln -s /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt
pip install --upgrade certifi
sudo systemctl restart jupyterhub
Thanks folks for contributing, we’re super grateful for quick eyes on this one. Cheers