I have a Jupyterhub install on Mint 20.3 started with jupyterhub-systemdspawner. It is running on a 48 cores server.
c.Spawner.cpu_limit is set to 4.
Reading the documentation, i thought that a user notebook would be able to use up to 4 cores @ 100% (400%). But in reality, a notebook is able to use 48 cores @ 2.83% (400%).
c.Spawner.cpu.limit is supposed to be the maximum number of cpu-cores a single-user notebook server is allowed to use.
How can I really limit a notebook to a maximum number of cpu-cores instead of using all cpu-cores at a lower rate?
Python code used for testing:
from multiprocessing import Pool
import psutil
import time
def f(x):
set_time = 1
timeout = time.time() + 60*float(set_time) # X minutes from now
while True:
if time.time() > timeout:
break
if name == ‘main’:
# Test on for core/cpu
# processes = 4
# Test on all core/cpu
processes = psutil.cpu_count()
print (‘utilizing %d cores\n’ % processes)
pool = Pool(processes)
pool.map(f, range(processes))
It only works if your spawner supports it. It looks like systemdspawner does support it, I wonder if your system doesn’t support it, like if cgroups isn’t supported. (I’m not sure what systemdspawner uses to limit cpu usage).
You could try c.SystemdSpawner.cpu_limit = 4.0 and see if that helps.
@markperri is right. It sounds like you want CPU pinning (or CPU affinity) which isn’t supported. It’d be very difficult to do, how would you know which CPUs to assign to each user with multiple users, and how would you handle other processes running on the system?
I want to limit the number of cores used by a user. I think the documentation is misleading. c.Spawner.cpu_limit = 4 should let you use up to 4 cores at 100% for a total of 400%.
Instead, it seems to take 400% of the total number of core. On a 10 cores system, a user will be able to use 10 core @ 40% for a total of 400% instead of just 4 cores.
c.Spawner.cpu_limit = 4.0 does not change the behaviour.
Really all you care about is that in top they show 400%, right? If you have 12 cores then 3 users can get their full computational power. If you really want cpu pinning then you should use something like slurmspawner.