DDoS (Distributed Denial of Service) is a type of attacks to interrupt the continuous healthy response of a server, it’s considered one of the hardest attacks to mitigate and has harvested so many sites. To understand how deep, bad and well hidden this type of attacks is, imagine the following scenario.

A tale of two banks

Imagine two banks: Bank Scam and Bank Scan. Bank Scan is our good guy, a bank where employees are trying their best to be the best, an honest bank of a very well-built reputation. Bank Scam on the other hand is the shady bank that’s doing everything wrong and more.

Bank Scan has just set up a new branch in an area of Bank Scam authority. Bank Scam didn’t like the idea and decided to use malicious ways to slow down Bank Scan and to reduce productivity. So, each day Bank Scam hired 500 persons to go to Bank Scan and enter the bank as ordinary customers, take a number and wait for their turn in the customer service lounge. On the first day of this plan, Bank Scan branch manager was very happy, he got 500 new potential customers which is awesome.  The branch manager got a few complaints from customers that they waited and waited the whole day from 9 to 9 without getting served. “No problem at all, I still have beaten the record”, the branch manager thought to himself. The second day, Bank Scam sent another “different” 500 person to Bank Scan early in the morning so they can take all the seats in the bank and forbid legitimate customers from even getting seated in the bank. The branch manager was even happier thinking to himself “Wow! 500 again! two days in a row!” without having any idea that someone is acting in a bad faith and all of this fake traffic is getting the branch closer to bankruptcy. Again, on this day, Bank Scan branch manager got a lot of complaints from customers that the bank was even more crowded than the first day, the manager told them how sorry they were and that they would work on solutions.

The same scenario happened each and every single day for a whole month. The branch manager, without involving the HQ, took a very brave decision to: 1. Expand the branch, so that all customers can find a seat and 2. Hire more customer service agents to cover a higher number of customers. A very bad move due to lack of inspection. The next day Bank Scam noticed the changes and in return sent 1000 fake customers instead of 500.

The branch manager took a 3rd step before giving up, 3. Hire a new security staff. After hiring the new security team, they couldn’t identify fake customers. So, this didn’t solve the problem at all.

Finally, the branch manager decided that the matter got out of his hands, a whole month without closing a single deal with a client. It’s time to elevate this to HQ. When Bank Scan HQ got the report, it rang a bell. They saw this behavior before, and decided to take some actions.

Before diving into what they would do, let’s revisit the case. Each day Bank Scan (our good guy) gets 500 visitors very early in the morning and all of these 500 visitors are paid and hired by the competition to waste bank resources and employees’ time. Not only that, but this forbids real customers from getting the needed service. This prevented the bank from closing a single deal in long time and converted the branch to an unproductive unit where real customers can’t get served at all. All the solutions weren’t successful and made things worse.

To translate this analogy into technical terms, DDoS attack is using multiple nodes to send simultaneous traffic to a specific target/server to cause interruptions in operation and consume resources and eventually convert the server into an unresponsive piece of garbage so that real visitors can’t make use of the service, hence the name, denial of service.

DDoS Consequences

We had a very recent case where our customer had a brutal DDoS attack, solving the problem helped reducing the server specs from EC2 m4.4xlarge to m4.xlarge, which reduced the annual hosting costs by almost $4000, so, in short, even if DDoS didn’t beat the server to its knees, it costs you resources, and the consequences are devastating.

DDoS Solution

You can try scaling resources or using multiple nodes and load balancers to mitigate DDoS. This may or may not work based on the volume of the attack. So we won’t consider this a solution. Also, you can always optimize your server workers or your code (In the bank analogy, the customer service employees) to provide faster responses and serve more requests in shorter duration. But sometimes, this might not be feasible.

All solutions will revolve around behavior analysis, back to our bank analogy, what if all fake customers are coming from the same street/block? say same words? do same requests? wear hats? wear yellow? or don’t have a head? Huh, you didn’t see that coming? Well, don’t have a head is what we call a headless browser, coming from the same street/block is when all nodes in DDoS attack are coming from the same country or IP subnet. Saying the same words or doing same requests is another way to mark bad requests. Wearing hat translates to using specific proxies and wearing yellow is using specific user agents. All of these behaviors can be spotted in the logs and can be used to block bad requests based on this mutual behavior.

Anyways, all these behavior analysis solutions can be applied to the target directly or to an intermediate channel which then sends the request to the target after making sure the request is legit. You can use CSF (ConfigServer Firewall) to apply these rules like blocking countries, blocking IP subnets and much more on the server itself.

However, Internal behavior analysis, where the target itself does the checks, doesn’t always work, mainly because the process might be resource exhausting and results might be non-deterministic, you might end with blocking legit users or allowing enough bad requests that can kill your server. No single formula to generalize.

The solution is external behavior analysis services, you might have missed one huge detail, when Bank Scam sends fake customers to the bank early in the morning, and next day sends another batch of fake customers. Where did the first batch goes? Good guess, they went to Hotel Spam which wants to attack Hotel Star. In short, fake customers are hired to attack multiple servers but the same fake customer rarely visit the same node twice. However, the intermediate channel handles a lot of servers, so it can easily spot a fake customer even if the customer didn’t visit the same node twice. So if both Bank Scan and Hotel Star hired the same behavior analyst (DDos Prevention service) both entities will be able to benefit from each other knowledge and will eventually, with the help of the behavior analyst, mark and block fake customers/bad requests.

I’ve tried the two big players in this field: CloudFlare and Sucuri. When it comes to DDoS mitigation, Sucuri does the better job. I might write a separate post to compare the two. Anyways, back to your DDoS problem, try out these solutions in this order to fix the DDoS problem:

1. Optimize your server configurations and code, you will need less resources which will help mitigating DDoS.
2. If all requests are coming from the same IP subnet: Block/challenge the subnet.
3. If all requests are coming from a country which is not the country of your main audience: Block/Challenge the country.
4. If all requests have the same user-agent: Challenge the agent (Never block a family of user agents, unless it’s a fake user agent of course)
5. If all requests use some kind of proxy: Block/Challenge the proxy.
6. If you don’t have enough time or all of these solutions didn’t work: Hire Sucuri or Cloudflare.

Do you still have any problem? Do you have other possible solutions? Let us know in the comments section below.