How can Prometheus scrape metrics from JupyterHub?

JupyterHub
1

Prometheus metrics were exposed via a /hub/metrics endpoint in 0.9.0. Then, the metrics endpoint was updated to require authentication in 1.0.0. A prometheus server needs to be able to make requests to this endpoint and it expects to get a metrics response. Currently, it is getting a 403 forbidden error.

My question is: how are people getting around this? I know there is a setting to simply disable this authentication behavior, but I am interested in preserving the authentication requirement. From what I can tell there is no way to simply allow login to JupyterHub via HTTP Basic Authentication or via a Bearer token, which are the two authentication methods primarily supported by Prometheus. So, there is no way for the Prometheus server to access this data, which was meant exclusively for it.

1 Like
2

This assumption turned out to be incorrect. When I had been testing with an example token, I made a mistake that prevented the token auth from working properly. Though it is not explicitly documented, you can use a token to authenticate and hit any of the internal pages/resources (not just the API routes). So, it can be configured as follows.

First, create a token for some dedicated user. I created a prometheus user in JupyterHub for this.

jupyterhub token prometheus >/etc/prometheus/secrets/jupyterhub

Then, add a section to your scrape config for the JupyterHub job to use this token:

scrape_configs:
- job_name: 'jupyterhub-exporter'
  bearer_token_file: /etc/prometheus/secrets/jupyterhub
  static_configs:
  - targets: ['10.10.0.2:8075'] # Or whatever <host>
1 Like
3

You could also do without the token providing that you disable prometheus authentication in jupyterhub_config.py file:
c.JupyterHub.authenticate_prometheus = False and then your scrape_configs will look like:

  - job_name: 'jupyterhub'
    metrics_path: '/hub/metrics'
    static_configs:
      - targets:
        - '{{ jupyterhub_host }}:{{ jupyterhub_port }}'
1 Like
4

Yes, that is a possibility, however, you have to be OK with exposing your Prometheus metrics to any WAN your JupyterHub is connected to (in our case, the public Internet). This wasn’t acceptable for us.

5

I was not able to get the jupyterhub token prometheus method to work, as it complained that the resulting token didn’t have the read:metrics scope. This is with JupyterHub 3.1.1, which has RBAC features.

See Prometheus Config for Jupyterhub via Helm for the method that worked for me. I created a JupyterHub service with a predefined API key, and a role with the required scope attached to that service.

First define a random API key to use:

export JUPYTERHUB_METRICS_API_KEY=$(openssl rand -hex 16)

Then in jupyterhub_config.py

    c.JupyterHub.services = [
        {
            "name": "service-prometheus",
            "api_token": os.environ["JUPYTERHUB_METRICS_API_KEY"]
        },
    ]

    # Add a service role to scrape prometheus metrics
    c.JupyterHub.load_roles = [
        {
            "name": "service-metrics-role",
            "description": "access metrics",
            "scopes": [
                "read:metrics",
            ],
            "services": [
                "service-prometheus",
            ],
        }
    ]

You should then be able to access the metrics with

curl -H "Authorization: Bearer $JUPYTERHUB_METRICS_API_KEY" http://<jupyterhub-server>/hub/metrics
2 Likes
6

Thank you, @ianhinder, works fine for us!