Well, crap. A couple days ago, I noted that the St. John’s server was acting very slow. I waited until the evening to check on what was happening, and around that time saw a huge number of email bounce messages from various email providers like AT&T and Cox. Something was wrong.
I quickly found out that St. John’s was being the source of thousands of spam messages, headed all over the globe. I killed the mail server program Postfix, and the spam stopped, and the system sped up significantly.
I spent a couple days on and off trying to find where the spam was coming from. I did network sniffing at both the external and internal network cards, but all I found was the normal traffic I would expect (i.e. nothing was feeding the server from either the big bad Internet, or from inside the building).
It quickly got to the point where no effective email service was available due to our being put on a couple blocklists. And the CPU on the server, which is also a router to get people in the building out on to the Internet, was being eaten by the bot which was clearly running inside the server.
Now, there are many thousands of Windows malware, including virii, bots, and the like. There are only a couple that affect Linux boxes.
I had been working with our ISP (Cox) on this. I had one hint from them, that we had the Alureon (AKA TDSS) virus. They also gave me an IP address for the virus command server (a computer in Russia). I blocked that IP address for both sides of our connection using IPTables. But Alureon is a Windows virus, not Linux. I download a tool to check and literally hit every machine in the building, nothing. So that left a couple laptops. But I don’t think that this was a valid hint, as the spam kept coming even when I pulled the RJ-45 out of the building network connector.
One of the blocklists told me that I had a Grum botnet client. Again, it’s a Windows based bot, so who knows.
Finally, I gave up. I had read over and over that rootkits on Linux were nearly impossible to find and eradicate. I shut down every service on the computer, pulled the config files and logs off, and then wiped the machine and reinstalled Ubuntu.
I went back and got the basic machine running, created users, changed every password. I ensured that I had a good firewall running, but setting up IPTables to let only a certain number of ports through, and zorching everything else.
Next I got the email back going. When I installed, I had specified a mail server (Postfix) and a LAMP setup. While those were helpfully running after install, I shut them down (except the web server).
I had installed several packages from the Ubuntu software center in the week or so leading up to this, so naturally I wondered if that was the attack vector. I have not reinstalled those packages.
I spent time last evening and today working to get us off the various blocklists, and that seems to be going OK. When I get some time, I am going to try to look through the logs and determine where the attack came from.
I’ve always had a great deal of faith in Linux (in fact, I recently switched my laptop to Linux only and have been very happy with it), but this incident has me a little paranoid. One thing I will do in the next day or so, when the system is quiet, is to clone the drive so that I can restore it quickly if this happens again. I will also do some research to see if I can find out what I might have missed while setting things up and running them. I also need to get the extra stuff going I had before.