Hi @consideRatio, Thank you very much for these replies - everything is working now Actually, I’ve been Googling around this topic for a few days, and several of your previous posts have helped me enormously, so thanks for those too
I finally managed to get everything going by following the link in your first post above. I think my current solution may be unnecessarily complicated (the suggestion in your second post seems cleaner), but I think I’ll leave things as they are for now and revisit when I’ve gained more familiarity with Google Cloud and Kubernetes.
Regarding auto-scaling based on CPU usage, I take your point as to why this is difficult. I was pursuing this idea before I discovered the
userProfiles option, which I think will be fine for my use case. Most of my users have low resource requirements, and those wanting more will almost always know about it before they start their session. With my current setup, I can make a range of options available and users can choose. If they decide partway through that they need more power, they can simply log-out and then sign back in on a more powerful machine. I didn’t realise this was possible at first, which is why I was looking for some way of up-scaling to more powerful machines (not just more nodes) as resources become limited.
In case anyone else is having similar issues, below is an extract from my current configuration. However, please note the caveat that the solution from @consideRatio, above, is probably a better/simpler way to achieve the same thing.
Right, I’m off to play with my new Hub
First, create the cluster and a
default pool with the
gcloud container clusters create my-cluster-name \
Then create the “standard”
user pool, as per the tutorial
gcloud container node-pools create user-pool \
Next, create a pool of more powerful machines for users needing more CPUs. Note that I’ve included the default JupyterHub
taint, plus a new
taint defining this particular group of users (
gcloud container node-pools create user-pool-hi-cpu \
config.yaml, I have the following
# ... Other singleuser config.
- display_name: "Standard (default)"
<4 CPUs, <12 GB RAM, no GPU. Resources allocated dynamically. Use this unless you have special requirements.
- display_name: "High CPU (new node)"
Access to 16 CPUs, 12 GB RAM, no GPU. For CPU-heavy processing
- effect: NoSchedule
- effect: NoSchedule
Finally, I also have this in
config.yaml to ensure that “user” and and “core” pods get allocated correctly