Adventures with Servers attacked by Botnets
I have a toy VPS on Digital Ocean. It runs the backend of a very simple prototype for a demo that I had to run almost a year ago. The backend has a signup and login system but fortunately we didn’t use any sensitive data for the demo, just fake email addresses and a bunch of data about businesses like addresses and phone numbers, all publicly available stuff. At the time we were in a rush so I set up the machine without any particular protection. There was no firewall set up, just a robust password for a user with sudo privileges, which I used to login via ssh. That was it.
Almost a month ago, during a demo, we noticed that the machine was much slower than usual to return results. It had to serve at most 10-12 users at a time, in practice a very undemanding task, so there was obviously something wrong. I logged in and noticed … a file I didn’t upload. Holy crap! It was a perl script, running an IRC bot. Fortunately it was located in home, so I noticed it immediately. I secured immediately the machine and then I checked all my other servers for suspect activities. Thanks to whoever convinced me to install audit a long time ago. Fortunately only the toy machine was “drilled”.
I spent some time investigating how the perl script was uploaded but I couldn’t find the vulnerability. While investigating I discovered a massive amount of tentatives to connect via ssh on each of my servers. If you have a server check out the authentication logs. On Debian/Ubuntu they are in /var/log/auth.log. My logs were full of
Dec 1 06:50:58 server_name sshd[23306]: Failed password for root from 43.229.53.66 port 55069 ssh2
I had a firewall active, limiting access to ssh connection on port 22. I was using ufw to configure iptables and the limit condition is pretty strict by default. It temporarily bans IPs that make six tentatives within the time span of 30 seconds. I “felt safe”, but I couldn’t be more wrong.
The botnet attacking my servers (and likely yours) is probably made of a gazillion of machines. They randomly attack at intervals, sometimes regularly. They try to login using common user names like “bob”, “jesse” or known UNIX usernames like “root” or “ftp”. They are clearly tuned to escape some of the common default rules of firewalls. For example in my logs I noticed a pretty “insisting” machine, with IP 43.229.53.66, that attempted a login every minute for two hours, then paused for one hour, and then restarted. It’s a legit behavior according to the default rules I had set. Something similar happened on EVERY server I had.
So I rolled up my sleeves and hardened my machines. At first I tried fail2ban, which can happily live together with ufw. It worked, in that I noticed fewer tentatives. While I was at it I went all out and put in place the following changes:
- ssh login only via Key-Based Authentication
- changed ssh port
- disabled root login via ssh
- disabled password authentication
All of these (and more) are explained in this well done guide. They are all pretty easy to put in place. The boring part was that I had to do it for each one of my servers and the virtual machines that I use to test on my Mac. I also had to update all my provisioning scripts to match the new changes. It took a while but it was worth it. My logs are now cleaner and I can sleep a bit better.
In case you wonder the great Prompt app supports key-based authentication so you can ssh into your servers also from iPhone and iPad. “Look ma, no password to type!” :)
I’d have liked to report the attacking IP addresses to DigitalOcean. I reported a bunch via this form and I got a reply that I should submit one report per IP, plus evidence. It seems a bit of an overkill. I think I have around 300 IPs to report. Maybe I should build a bot that extracts IPs from my logs and submits reports to DigitalOcean :)
UPDATE 2015-12-11: Bitninja looks a cool service to detect this kind of attacks.