Success Means Security: How to protect your most profitable web sites from distributed denial of service attacks (an open source approach)

One of the side effects of success is that you need to worry about security. You see how celebrities have to walk around with bodyguards, their homes have state of the art alarm systems, and the paparazzi is still always looking for a chink in their armor.

The same thing happens online. The more successful you are, the tighter the security of your online assets has to be. (You don't want your competitor ending up with your customer list, do you?)

It is always a good idea to install firewalls, intrusion detection systems, and to have an experienced system administrator or a competent hosting provider that regularly patches your servers with the latest service packs and security updates. Having your site simply defaced is the “best-case scenario” of what could happen when your site's security is breached. It is incredible how hackers break into sites, steal customer information worth thousands of dollars and sell it for a few hundred bucks. Personally, I place decoys in my customer databases so that I can tell when/if this valuable information has been stolen.

Unfortunately there is a type of security attack that is extremely difficult to fight: a distributed denial of service (DDOS). On the up side, you know you are doing really well when hackers try this on you. 😉

What is a Distributed Denial of Service (DDOS) attack?

This is not a trivial attack by some script kiddie. DDOS is a serious and well-planned attack, usually with the purpose extorting ransom. The idea behind it is to exhaust the target server or network’s limited resources. Small attacks try to exhaust the server's CPU or memory by sending a lot of fake requests. The main challenge blocking the attacks is being able to detect legitimate traffic from harmful traffic.

When it is a single script kiddie running a script to send overwhelming requests to your server, it’s enough to block his or her IP. A distributed attack is a whole different ballgame. Distributed attacks scale to a proportion that, no matter what you do, once your bandwidth is exhausted and your system is unable to sustain the number of invalid packets, your network will crumble. SCO was a very notorious case of just this type of massive attack.

DDOS attackers start by spreading viruses or hacking many vulnerable systems, usually at universities where there are high bandwidth connections to the Internet. They install zombies or slaves that are commanded remotely, waiting for instructions. Once they have a sizeable number of infected systems, they assume control from a single master server, which is also a compromised/hacked system. They simply instruct the slaves to attack the IP address or range of IPs of the targeted victim.

A well-known program by security experts that is used for this purpose is the Trinoo or Trin00 program/trojan. A more complex type of DDOS attack involves IP spoofing, altering the source IP of the packets before they leave the attacker's network so that the victim has a really hard time filtering out the bad traffic. IP spoofing (faking IP addresses) is easier said than done, though. Most modern routers come with ingress/egress filtering rules, and TCP sequence numbers are stronger than in the old days. If you lost me there, don't worry, I will come back to this topic in a later post.

Don’t negotiate with terrorists

In October last year we experienced our first DDOS attack. After bringing down our primary and secondary server for several hours, we got an e-mail offering help. The e-mail read something along the lines of: “I'm chinese hacker. I stop attack for $10k”

I have to think that there are site owners that do send the $10,000 dollars. Otherwise, why would he ask for such a sum of money in the first place? But although we lost a lot of money having the server down, paying the hacker would have meant his inevitable return each month. You simply don’t negotiate with terrorists. I can't imagine what amount of brainwashing it would take for me to accept such a “deal.” But what were my alternatives?

I knew some larger affiliate networks that had faced similar problems and they went to a particular company that specializes in protecting companies from these types of attacks. We went knocking on their door, paid more than the hacker was asking plus a big monthly check, and we were supposedly safe after 48 hours of attack.

The attacker adjusted and, despite our expensive security, we were brought down again—a few more times. I commissioned my top systems guy to study some open source solutions and we built a solid framework after about a month. It is the same framework we still use today, and we haven't had any more successful attacks. Let me share it here so you know where to look it up if you ever need it.

Evasive maneuvers with mod_evasive

I am very thankful to the writers of this piece of art—mod_evasive has saved my company a lot of money during the year we have used it. It has been a blessing.

Put simply, it is a rate-limiting Apache module. If it receives more than a specified number of simultaneous requests, it will block the rest. You can return a forbidden status (HTTP 403) or you can setup iptables filtering rules to block the IP address completely. In our experience, there have been almost no false positives and the software scales as long as the server has the CPU and memory to handle the incoming requests.

The default values are reasonably good, but you can tweak them for maximum benefit.

<IfModule mod_evasive20.c>
DOSHashTableSize 3097
DOSPageCount 5
DOSSiteCount 100
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 600
</IfModule>

An extra layer of protection

The module alone is probably good for most setups, but in our case we wanted to be a little extra cautious. We installed the evasive module in a front-end server that is the only IP visible to all users. That server then passes the filtered requests back to the real servers via another great module: mod_proxy. This type of setup is called a reverse proxy server. There are many advantages to this approach:

1. You can have multiple, real servers and your setup can intelligently proxy the request to the available one.

2. You can cache the dynamic responses, reducing the CPU/memory load on the real servers.

3. The attackers don't know the real servers IPs. They might be able to successfully bring down the reverse_proxies+mod_evasive servers, but they need extra work to find the real ones. You can repl
   ace the
   proxies with new ones faster than you can real servers, if such a scenario ever happens. It hasn't happened to us.

I hope I didn't get too technical and you were able to follow most of what I discussed here. It is definitely interesting stuff. I can't say you will enjoy a DDOS if it happens to you, but think about it this way: you’re big enough to be attracting attacks. You’re a celebrity in your own right now. 🙂