How can Prometheus scrape metrics from JupyterHub?

Prometheus metrics were exposed via a /hub/metrics endpoint in 0.9.0. Then, the metrics endpoint was updated to require authentication in 1.0.0. A prometheus server needs to be able to make requests to this endpoint and it expects to get a metrics response. Currently, it is getting a 403 forbidden error.

My question is: how are people getting around this? I know there is a setting to simply disable this authentication behavior, but I am interested in preserving the authentication requirement. From what I can tell there is no way to simply allow login to JupyterHub via HTTP Basic Authentication or via a Bearer token, which are the two authentication methods primarily supported by Prometheus. So, there is no way for the Prometheus server to access this data, which was meant exclusively for it.

1 Like

This assumption turned out to be incorrect. When I had been testing with an example token, I made a mistake that prevented the token auth from working properly. Though it is not explicitly documented, you can use a token to authenticate and hit any of the internal pages/resources (not just the API routes). So, it can be configured as follows.

First, create a token for some dedicated user. I created a prometheus user in JupyterHub for this.

jupyterhub token prometheus >/etc/prometheus/secrets/jupyterhub

Then, add a section to your scrape config for the JupyterHub job to use this token:

- job_name: 'jupyterhub-exporter'
  bearer_token_file: /etc/prometheus/secrets/jupyterhub
  - targets: [''] # Or whatever <host>
1 Like

You could also do without the token providing that you disable prometheus authentication in file:
c.JupyterHub.authenticate_prometheus = False and then your scrape_configs will look like:

  - job_name: 'jupyterhub'
    metrics_path: '/hub/metrics'
      - targets:
        - '{{ jupyterhub_host }}:{{ jupyterhub_port }}'
1 Like

Yes, that is a possibility, however, you have to be OK with exposing your Prometheus metrics to any WAN your JupyterHub is connected to (in our case, the public Internet). This wasn’t acceptable for us.