I had to burn our TLJH instance recently, because it was exploited for Bitcoin mining by this attack.
We use JupyterHub for pvlib python tutorials at various conferences and workshops big and small, and it has been used by NREL researchers with interns and other labs as training material.
Before I build a new instance, I want to make sure it won’t get taken down by another mining exploit/attack. I understand no solution is full-proof, but this is a volunteer service partially funded by NumFOCUS and IEEE, so we don’t have a lot of extra time or money.
One solution that’s been suggested is to issue pre-generated username/passwords instead of letting anyone log in. There are some complications if attendees are not known in advance or physically present, so any suggestions are welcome.
I am also thinking that our next instance should use JupyterHub instead of TLJH. Are there more security features? Any other suggestions? I don’t want to have to shut down the tutorial in between events because it limits usage by smaller ad hoc trainings, but that might be the only option.
The easiest solution is indeed giving access to only trusted users. One way to control this is to have github auth and add all the prospective users in the allowed_users list.
I am also thinking that our next instance should use JupyterHub instead of TLJH.
TLJH is an all inclusive distribution of JupyterHub, the other is Z2JH (kubernetes based). I would suggest to not try to roll you own JupyterHub and keep TLJH with secure authentication
I guess one of the best approaches is to limit the access to only trusted users and to limit the time people are allowed on the JupyterHub. The less time they are granted, the less they can tinker around. That would require to log off people after workshops and frequently create and disable accounts. In addition, you should definitively think about spawners and the isolation they bring with them. For one instance, e.g., I use docker spawners to isolate people from each other. Others might say it is not safe enough because people can get out of their containers. Then you might also think about sshspawner plus virtual machines? However, the safest way is not to allow untrusted people to work on a machine you are responsible for.
I agree with everyone else, the best thing is to only grant access to those who require it.
Failing that, if access is time limited you could use the DummyAuthenticator with a global password- this will allow any username as long as the shared password is entered
You’ll need to make sure everyone uses a unique username e.g. their email. You can change this password after the event to limit abuse.
If you don’t want to manage allowed users individually, several of the JupyterHub OAuthenticator plugins support allowed users via groups. However, I’m not as familiar with TLJH configuration, so I don’t know how easy it is to use OAuthenticator with TLJH.
You might consider using some IaC setup that would allow you to bring up an instance when needed for a conference or workshop, then destroy it soon after.
I do something similar for courses at a university, using terraform to create the instance and everything else needed on Google Cloud. Makes it very easy to create and destroy instances on the spot, and the less time an unnecessary instance is up and running, the less exploitable it is. Also saves some $$$.
Can you tell me more about this? Or how I can find more information? I’ve never heard of an “IaC” setup but Googling I came up with this “Infrastructure as Code” medium article about using Terraform on Google Cloud. How hard is this to create? Is this a good reference? Any others you recommend? Looks like Terraform has a freemium service. I’ll give it a try. Thanks!
terraform is open source, although it’s possible to pay for a cloud version. There’s certainly a learning curve, but their site has quite a bit of tutorials, helpful forums, etc. Major cloud providers tend to provide documentation and code snippets on creating resources with terraform too, for instance this Google Cloud intro.
If you manage many TLJHs or related resources, it could be worth the investment.
Another alternative you can find at Marvin Kastner / minimal-jupyterhub-docker-config · GitLab where I presented a configuration working with Docker in 2019. For reproducibility, some images and libraries were fixed to specific versions in those days, so you might want to re-adjust the setup before deployment. If you are fluent in Linux server administration and docker administration, there is little to know in addition.
