Accessing single-user JupyterLabs API through JupyterHub API

I’m trying to create a flow that I can see which kernels are taking up the most CPU/RAM/Resources on my server. On the original Jupyter Notebook, I would do this through api/sessions and then do some awk/ps/bash magic to link those sessions/kernels to processes etc.

Though with JupyterLab, I have a service config like in Using JupyterHub’s REST API — JupyterHub 2.3.1 documentation with this as my config:

c.JupyterHub.services = [
    {
        "name": "service-admin",
        "api_token": "my-token",
    },
]

c.JupyterHub.load_roles = [
    {
        "name": "service-role",
        "scopes": [
            # specify the permissions the token should have
            "admin:users",
            "admin:servers",
            "proxy"
        ],
        "services": [
            # assign the service the above permissions
            "service-admin",
        ],
    }
]

and this works fine for looking at endpoints found here JupyterHub REST API — JupyterHub 2.3.1 documentation, but I want to access each lab server’s API endpoint so I can grab everyone’s sessions/kernels.

There’s a blurb in the API docs saying

The same API token can also authorize access to the Jupyter Notebook REST API provided by notebook servers managed by JupyterHub if it has the necessary access:users:servers scope:

but that link is broken and any number of Authorization tags/guessing via the proxy URLs result in 404 pages on my end. The closest I’ve gotten to any kind of result is going to /user/{my user}/api/status and getting this:

{
    "message": "Forbidden",
    "reason": null
}

Instead of a 404 result. What am I missing? Am I able to get a flow where I can access each lab’s api/sessions?

There’s an example in the docs:

Or you can make a request using curl:

$ curl -H "Authorization: token $JUPYTERHUB_API_TOKEN" "http://localhost:8000${JUPYTERHUB_SERVICE_PREFIX}api/status"

{"connections": 0, "kernels": 0, "last_activity": "2022-07-14T21:36:09.196342Z", "started": "2022-07-14T20:59:32.957825Z"}

I’ve attempted that method and can grab the users, but same result when trying to get the status of any given server within the hub trying to get api/status

import requests

baseURL = 'MY BASE URL'

session = requests.Session()
session.headers = {'Authorization': 'token MY TOKEN'}
session.verify = False

users = session.get(baseURL+'/hub/api/users')

for u in users.json():
	if 'server' in u and u['server'] is not None:
		url = baseURL+'/user/'+u['name']+'/api/status'
		print(url)
		s = session.get(url)
		print(s.text)
		quit()

Always returns HTTP 403, is there something in the config services/load_roles I’m missing maybe?

You need to grant your service the access:servers scope to have permission to talk to the single-user server API. admin:servers does not include this, it must be added explicitly.

That was absolutely it, I didn’t figure admin wouldn’t have assumed access.

Thanks so much!