How to set up Prometheus metrics with (The Littlest) JupyterHub?

JupyterHub
1

I have followed the instructions to install The Littlest JupyterHub on my custom server, including SSL, and everything is working correctly.

In addition, I have installed Prometheus in said server, and I successfully added it as a data source in an external Grafana server using http://sub.domain.space:9090.

Now, I was trying to leverage the JupyterHub Prometheus metrics that are advertised in the documentation, but so far I’ve been unsuccessful. The docs only give this hint:

Prometheus is a separate open source tool that can be configured to repeatedly poll JupyterHub’s /metrics endpoint to parse and save its current state.

I inferred that I needed to add a new job to /etc/prometheus/prometheus.yml, and this is what I tried:

scrape_configs:
  ...
  - job_name: jupyterhub

    scheme: https
    static_configs:
      - targets: ['localhost']

However, I still don’t see the jupyterhub_* metrics showing up. I must be doing something wrong, but I’m not sure how to proceed.

  • The scheme: https and targets: ['localhost'] is clearly wrong - the machine has a SSL certificate for the domain I pointed to it, and therefore I don’t think it’s going to work. But how or where do I see Prometheus spitting errors because of it?
  • I could point the target to the domain, like targets: ['sub.domain.space'], but I’ll still get authentication errors. This post by @jason_anderson suggests a solution, but when I try jupyterhub token $whatever, I get an error:
root@jupyterhub:~# /opt/tljh/hub/bin/jupyterhub token root
[W 2022-11-19 15:34:01.462 JupyterHub app:1828] No admin users, admin interface will be unavailable.
[W 2022-11-19 15:34:01.462 JupyterHub app:1829] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2022-11-19 15:34:01.462 JupyterHub app:1858] Not using allowed_users. Any authenticated user will be allowed.
No such user: root

The No admin users does not make much sense to me, since I do have an admin user:

image
image1140×202 19.7 KB

Any pointers?

2

Quick response but do post back if it doesn’t help. Try taking a look at the setup within mybinder.org-deploy repo in jupyterhub org on github. We use Prometheus for metrics there. Also the SRE guide for mybinder may have tips.

1 Like
3

Thanks a lot @willingc! I see there’s mybinder.org-deploy/metrics.md at 3189394e1feda728f0c288cc31c3376b0e6e7693 · jupyterhub/mybinder.org-deploy · GitHub with some information. I don’t see the Prometheus configuration though, possibly because it’s encrypted: Adding a 'remote write' on prometheus for ovh configura… by mael-le-gal · Pull Request #1093 · jupyterhub/mybinder.org-deploy · GitHub

I am digging a bit more. After changing adding --log.level=debug to the prometheus config, now I’m seeing this in the logs:

Nov 20 22:42:16 jupyterhub prometheus[9882]: ts=2022-11-20T22:42:16.169Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=jupyterhub target=https://localhost:80/metrics msg="Scrape failed" err="Get \"https://localhost:80/metrics\": http: server gave HTTP response to HTTPS client"
Nov 20 22:42:21 jupyterhub prometheus[9882]: ts=2022-11-20T22:42:21.314Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=node target=http://localhost:9100/metrics msg="Scrape failed" err="Get \"http://localhost:9100/metrics\": dial tcp [::1]:9100: connect: connection refused"

so at least now I can see whether it’s working or not. Will continue debugging and will post a solution here if I find one.

4

In the end I managed to make it work by

  • Using the actual domain in the prometheus.yml rather than localhost so I could properly use scheme: https :heavy_check_mark:
  • Adding a bearer_token obtained from JupyterHub (since the /opt/tljh/hub/bin/jupyterhub token CLI still didn’t work) :heavy_check_mark:

I suppose in production-grade deployments people run Prometheus in a separate machine rather than in the same machine (making the localhost problem moot), but apart from that happy that this is working for me :+1: