Jupyterhub-singleuser consuming significant and constant CPU

JupyterHub
1

I’m using python 3.11 + jupyterhub 4.0.2 + jupyterlab 4.0.5 and am currently having a problem with jupyterhub-singleuser consuming constant and high CPU even when the user is not connected to the web console. I am launching it like so via jupyterhub_config.py

c.Spawner.cmd = ['bash', '-l', '-c', 'exec jupyterhub-singleuser "$@"', '--']

My reproducing steps are:

  1. Log into jupyterhub via web browser
  2. Exit jupyterhub/lab by going to File → Logout
  3. Observe on the server that jupyterhub-singleuser is running and chewing lots of CPU
akrherz  3566392 20.4  0.0 5029420 241284 ?      Ssl  10:35   1:54 /opt/miniconda3/envs/prod/bin/python3.11 /opt/miniconda3/envs/prod/bin/jupyterhub-singleuser --NotebookApp.allow_origin=jupyterhub.fqdn.redacted

Any ideas?

2

CPU usage never goes down? Are you testing this locally on your workstation using LocalProcessSpawner? Do you see anything strange in logs? Could you share them?

3

Correct, the above now looks like this without me having any interaction with the server

akrherz  3566392 11.4  0.0 5029420 244008 ?      Ssl  10:35  16:58 /opt/miniconda3/envs/prod/bin/python3.11 /opt/miniconda3/envs/prod/bin/jupyterhub-singleuser --NotebookApp.allow_origin=...

I have not tried such a thing, this is happening on the server that my colleagues use.

Nothing logged since I logged out of the session.

4

How is your JupyterHub deployed? Even if you have logged out of JupyterLab, the backend server keeps running until you terminate the single user server. I was talking about the logs of that single user server.

5

Thank you for the response. I don’t know where the logs for the single user server are located? I only see the logs for the running of jupyterhub -f jupyterhub_config.py. Here’s my config

from oauthenticator.generic import GenericOAuthenticator

origin = 'jupyterhub...'
c.Spawner.args = [f'--NotebookApp.allow_origin={origin}']
c.JupyterHub.tornado_settings = {
    'headers': {
        'Access-Control-Allow-Origin': origin,
    },
}
c.Spawner.default_url = "/user/{username}/lab"
c.SingleUserNotebookApp.default_url = "/user/{username}/lab"
c.Spawner.ip = '127.0.0.1'
c.Spawner.cmd = ['bash', '-l', '-c', 'exec jupyterhub-singleuser "$@"', '--']

c.JupyterHub.authenticator_class = GenericOAuthenticator

c.GenericOAuthenticator.client_id = ...
c.GenericOAuthenticator.client_secret = ...

c.GenericOAuthenticator.allow_all = True
c.GenericOAuthenticator.oauth_callback_url = 'https://.../hub/oauth_callback'
c.GenericOAuthenticator.token_url = 'https://iastate.../oauth2/v1/token'
c.GenericOAuthenticator.userdata_url = 'https://iastate.../oauth2/v1/userinfo'
c.GenericOAuthenticator.username_key = "email"
c.GenericOAuthenticator.scope = ['openid', 'profile', 'email', 'address', 'phone', 'offline_access']
c.GenericOAuthenticator.login_service = "Okta@ISU"

c.NotebookApp.disable_check_xsrf = True
6

In that case your single user server logs will be logged along with JupyterHub logs. In any case, if you dont see anything strange in the logs, I assume it must be down to either polling or PeriodicCallbacks that backend jupyter_server uses. I am not quite sure though but this is my guess!

7

hooking up py-spy in the user environment may also help identify the culprit.

8

Thank you, I did not know about this fancy tool. Here’s the top output over about 1 minute after I exit jupyterhub

NotebookApp.allow_origin=jupyterhub....edu' (python v3.11.5)
Total Samples 6578
GIL: 8.00%, Active: 45.00%, Threads: 4

  %Own   %Total  OwnTime  TotalTime  Function (filename)                                                                
 25.00%  45.00%   13.84s    25.60s   ppid_map (psutil/_pslinux.py)
 18.00%  18.00%   10.44s    10.44s   open_binary (psutil/_common.py)
  1.00%   2.00%    1.17s     1.35s   pids (psutil/_pslinux.py)
  1.00%   1.00%   0.180s    0.180s   <listcomp> (psutil/_pslinux.py)
  0.00%  45.00%   0.100s    25.70s   children (psutil/__init__.py)
  0.00%   0.00%   0.030s    0.040s   virtual_memory (psutil/_pslinux.py)
  0.00%  45.00%   0.030s    25.85s   get_metric_values (jupyter_resource_usage/metrics.py)
  0.00%   0.00%   0.020s    0.020s   _worker (concurrent/futures/thread.py)
  0.00%   0.00%   0.020s    0.020s   set (prometheus_client/values.py)
  0.00%   0.00%   0.020s    0.040s   memory_info (psutil/_pslinux.py)
  0.00%  45.00%   0.020s    25.72s   wrapper (psutil/__init__.py)
  0.00%  45.00%   0.010s    25.86s   metrics (jupyter_resource_usage/metrics.py)
  0.00%   0.00%   0.010s    0.010s   _remove (_weakrefset.py)
  0.00%   0.00%   0.010s    0.020s   _schedule_next (tornado/ioloop.py)
  0.00%   0.00%   0.010s    0.050s   system_metric (jupyter_resource_usage/metrics.py)
  0.00%   0.00%   0.010s    0.010s   __init__ (asyncio/events.py)
  0.00%   0.00%   0.010s    0.010s   usage_percent (psutil/_common.py)
  0.00%  45.00%   0.010s    25.93s   _run_once (asyncio/base_events.py)
  0.00%   0.00%   0.010s    0.010s   _call_soon (asyncio/base_events.py)
  0.00%   0.00%   0.000s    0.010s   _init (psutil/__init__.py)
  0.00%   0.00%   0.000s    0.010s   add_timeout (tornado/ioloop.py)
  0.00%   0.00%   0.000s    0.040s   <listcomp> (jupyter_resource_usage/metrics.py)
  0.00%   0.00%   0.000s    0.020s   _bootstrap_inner (threading.py)
  0.00%   0.00%   0.000s    0.010s   __init__ (psutil/__init__.py)
9

It looks like jupyter_resource_usage could be the culprit. Try uninstalling it from your singleuser environment and see if that significantly reduces your CPU usage?

10

Oh my, thank you. I found jupyter-resource-usage=1.0.1 installed via conda-forge. I removed it and now jupyterhub-singleuser appears to be completely idle.

11

If you figure out the root cause for why jupyter-resource-usage uses so much CPU you could open an issue: Issues · jupyter-server/jupyter-resource-usage · GitHub

12

If I am right, that must be Prometheus metrics endpoint that is consuming the CPU. @akrherz Could you try with --ResourceUseDisplay.enable_prometheus_metrics=False in c.Spawner.args?

13

That seems to be working as well. I reinstalled the extension, set your suggested flag to False and observed no CPU usage. Setting it back to True caused the CPU usage to return.

14

@jtp What is the purpose of PrometheusHandle in jupyter-resource-usage? It is not very clear from me from the current code base.

Do you think it makes sense to set the default for enable_prometheus_metrics to False?

15

Not entirely sure, but it’s probably there so the metrics can be collected and displayed somewhere else alongside other metrics, for example in a Grafana dashboard.

I guess? Maybe worth discussing in an issue?

16

Not entirely sure, but it’s probably there so the metrics can be collected and displayed somewhere else alongside other metrics, for example in a Grafana dashboard.

That is what I thought but I do not see this handler registered with any endpoint.

Fair enough! I will open an issue.