Prometheus Config for Jupyterhub via Helm

JupyterHub
,
1

Hello everyone,

I am looking for an example of how the Prometheus config file for Jupyterhub could look like. I have installed Jupyterhub and Prometheus via Helm. The config should therefore match the helm values.
The instructions from Jupyterhub Monitoring are unfortunately incomplete and my research has not found a suitable config.

I hope you can help me with this.
Thank you very much.

2

An example scrape config:

scrape_configs:
  - authorization: 
      credentials: <api_token>  # API token registered with hub that allows to access metrics endpoint
    job_name: jupyterhub
    metrics_path: /metrics
    scheme: https  # http if there is no internal TLS
    static_configs:
    - targets:
      - <hub_ip>:<hub_port>
   # If using internal TLS
    tls_config:
      ca_file: /path/to/internal-ssl/hub-ca/hub-ca.crt
      cert_file: /path/to/hub-internal.crt
      insecure_skip_verify: true
      key_file: /path/to/hub-internal.key

In JupyterHub config you need to setup a service as follows:

# Create a service to scrape prometheus metrics
c.JupyterHub.services = [
    {
        "name": "service-prometheus",
        "api_token": <api_token>,
    },
]

# Add a service role to scrape prometheus metrics
c.JupyterHub.load_roles += [
    {
        "name": "service-metrics-role",
        "description": "access metrics",
        "scopes": [
            "read:metrics",
        ],
        "services": [
            "service-prometheus",
        ],
    }
]

Hope that helps!

1 Like
3

Thank you, but I have the following problem and therefore cannot test the config.

I use the following command:

 helm upgrade --install --cleanup-on-fail --create-namespace --namespace prometheus prometheus prometheus-community/prometheus -f prometheus-config.yaml

However, the pods are not restarted and the new values from the config are therefore not adopted.

4

We dont use kubernetes and so, I dont know really have operational experience of it.

5

I was able to fix the problem and Prometheus successfully scrapes the metrics.
I was able to use the JupyterHub config from @mahendrapaipuri.
I had to adjust the Prometheus config slightly.

prometheus-config.yaml

extraScrapeConfigs: |
  - job_name: jupyterhub
    authorization:
      credentials: "<api_token>"
    metrics_path: /metrics
    scheme: http
    static_configs:
    - targets:
      - <hub_ip>:<hub_port> # See endpoint from the Hub-Service

In order for the deployment to pass the new config to the pods, the following commands must be executed:

helm upgrade --install --cleanup-on-fail --create-namespace --namespace prometheus prometheus prometheus-community/prometheus -f prometheus-config.yaml

kubectl rollout restart deployment prometheus-server -n prometheus