Well, crap. A couple days ago, I noted that the St. John’s server was acting very slow. I waited until the evening to check on what was happening, and around that time saw a huge number of email bounce messages from various email providers like AT&T and Cox. Something was wrong.

I quickly found out that St. John’s was being the source of thousands of spam messages, headed all over the globe.  I killed the mail server program Postfix, and the spam stopped, and the system sped up significantly.

I spent a couple days on and off trying to find where the spam was coming from.  I did network sniffing at both the external and internal network cards, but all I found was the normal traffic I would expect (i.e. nothing was feeding the server from either the big bad Internet, or from inside the building).

It quickly got to the point where no effective email service was available due to our being put on a couple blocklists.  And the CPU on the server, which is also a router to get people in the building out on to the Internet, was being eaten by the bot which was clearly running inside the server.

Now, there are many thousands of Windows malware, including virii, bots, and the like.  There are only a couple that affect Linux boxes.

I had been working with our ISP (Cox) on this.  I had one hint from them, that we had the Alureon (AKA TDSS) virus.  They also gave me an IP address for the virus command server (a computer in Russia).  I blocked that IP address for both sides of our connection using IPTables.  But Alureon is a Windows virus, not Linux.  I download a tool to check and literally hit every machine in the building, nothing.  So that left a couple laptops.  But I don’t think that this was a valid hint, as the spam kept coming even when I pulled the RJ-45 out of the building network connector.

One of the blocklists told me that I had a Grum botnet client.  Again, it’s a Windows based bot, so who knows.

Finally, I gave up.  I had read over and over that rootkits on Linux were nearly impossible to find and eradicate.  I shut down every service on the computer, pulled the config files and logs off, and then wiped the machine and reinstalled Ubuntu.

I went back and got the basic machine running, created users, changed every password.  I ensured that I had a good firewall running, but setting up IPTables to let only a certain number of ports through, and zorching everything else.

Next I got the email back going.  When I installed, I had specified a mail server (Postfix) and a LAMP setup.  While those were helpfully running after install, I shut them down (except the web server).

I had installed several packages from the Ubuntu software center in the week or so leading up to this, so naturally I wondered if that was the attack vector.  I have not reinstalled those packages.

I spent time last evening and today working to get us off the various blocklists, and that seems to be going OK.  When I get some time, I am going to try to look through the logs and determine where the attack came from.

I’ve always had a great deal of faith in Linux (in fact, I recently switched my laptop to Linux only and have been very happy with it), but this incident has me a little paranoid.  One thing I will do in the next day or so, when the system is quiet, is to clone the drive so that I can restore it quickly if this happens again.  I will also do some research to see if I can find out what I might have missed while setting things up and running them.  I also need to get the extra stuff going I had before.

2 Responses to “Hijacked!”

  1. Raegan Rethard Says:

    Yes, but considering what we COULD have had to deal with over the years with a Windows server, having only one Linux virus in what, fifteen years? I think are pretty good odds.

  2. Bill Hensley Says:

    A followup to this post. I had the same thing happen again a couple months ago, but this time I found out what the problem was. I went back through the logs from the previous issue and verified that it was the same cause.

    The problem: a spammer broke one of my user’s passwords, and was sending the spam out pretending to be her.

    In both cases, I had created the users with a username, and the password the same as the username. In both cases, the users did not follow the procedure to change their password.

    I figured this out by trapping a couple of the spam messages SMTP transactions, and seeing the username, and then seeing the password in the data stream. I immediately changed her password to something more random, and the spam stopped immediately.

    After that, when I issue a new user account, I immediately contact the user and have them run the password changer, or walk down to their room and run it with them.

    Live and learn. PBKaC.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: