On March 2, we posted a blog about a series of multi-gigabyte attacks generated by vulnerable Memcached servers logged by our mitigation platform. Since then, many questions have been raised, including the nature of this threat and more importantly, Nexusguard’s assessment of, and our ability to defend against this threat. 

While there are plenty of other authoritative studies and research to answer these questions, this post explains and elaborates our observations and mitigation approach.

What was the discovery?

After examining the log of a series of volumetric UDP attacks that first emerged on February 28 GMT, our researchers confirmed that part of them was reflected from Memcached servers deployed by thousands of organizations ranging from universities and government agencies to leading ISPs, hosting providers and domain registrars. The UDP protocol was exploited owing to its design limitation that did not help with the problem of IP spoofing.

How prevalent are Memcached servers?

Mainly deployed by network operators, service providers, internet portals and large organizations running heavy-traffic network services, Memcached server is a general-purpose distributed memory caching system used to speed up dynamic database-driven websites by caching data and objects in RAM. This effectively reduces the number of times an external data source, such as a database or API, must be read.

Why are Memcached servers vulnerable?

Memcached servers are meant to be deployed in house, not to serve the internet, and therefore they are supposed to be placed in a secure zone hidden from the internet. When best practices are followed, it is unlikely that these servers will be exploited to reflect amplification DDoS attacks. However, if they are exposed to the internet, be it knowingly or carelessly, cybercrooks can identify accessible Memcached servers through a simple scanning.

What are the attack signatures? Are they evolving?

The primary signature is the abuse of UDP port 11211. By sending spoofed requests to this port, the attacker could trick unprotected Memcached servers into sending (reflecting) amplified responses, vastly larger than the original request, to the victim IP address. The result is pipe saturation and service degradation.

According to our initial findings on the attack we saw on February 29, attack traffic was evenly distributed across multiple ports on the victim server, with each UDP stream lasting for 48-60 seconds. This pattern suggested to us that Memcached-powered attacks were still in its fledgling stage.

After the Memcached vulnerability was publicized by the media and gained widespread attention in the past week, it became evident that more vulnerable Memcached servers have been identified and are now being exploited by a greater number of attackers. From our observations, there are now at least four times more source IPs and source ports, in the attack botnet, and that number is still growing; while UDP stream duration now appears to be more wide-ranging.

How does Nexusguard mitigate against Memcached attacks?

While it is true that the amplification efficiency, at 51,000 times, is unprecedented, so is the attack size they can potentially create, the signatures of Memcached attacks are relatively easy to identify. In other words, to our trained eyes, we are able to detect and proactively block attack traffic upstream before they can do any lasting damage.

In fact, after the Memcached vulnerability was disclosed, ISPs, network operators and service providers alike (including our partners) are taking preventive measures to mend their fences, including rate limiting UDP traffic from source port 11211 (ingress and egress) to mask Memcached exposed to the internet (iptables on UNIX works).

Meanwhile, our research team is undertaking a scanning project to identify publicly accessible devices that have the Memcached UDP service open to the internet. Below is a summary of the top countries/regions with most Memcached servers (UDP protocol) accessible as of March 8.

Preemptive measures taken by Nexusguard to proactively mitigate Memcached attacks

1. We have informed our Service Provider partners to harden security on their Memcached servers by rate-limiting UDP service on port 11211 from the internet. This way, we can eliminate most Memcached attack traffic upstream before they can reach our scrubbing network and our client networks downstream.
   
   
2. To protect the network infrastructure of our clients using our Origin Protection solution, we have blacklisted the source IP addresses we identified as suspicious of sending out malicious requests to Memcached servers. Together with our other IP reputation engines, we will continue to monitor and blacklist IPs by checking their activities against our attack signature database, including those that characterize Memcached attacks.
   
   
3.  Since we already have taken the above steps to proactively block Memcached attacks, residual attack traffic i.e. junk UDP traffic that slips through the two lines of defence would be controlled and managed by our SOC, and automatically scrubbed by and dispersed across our global scrubbing network.

What is important for enterprises to learn from this is the importance of ensuring that all facets of their internet facing business be protected. This means not just protecting domains and websites, but to make sure that their underlying network infrastructure, as well as critical services such as DNS, be protected.

If you are our client, be reassured that you are well protected from Memcached attacks. It is no exaggeration that a gigantic DDoS attack in excess of 1Tbps is powerful enough to take down a regional internet backbone or an ISP. Having said that, due to the fact that signatures of Memcached-fueled amplification attacks can easily be recognized and that the Memcached vulnerability can be patched quickly by following standard best practices. Memcached attacks, as daunting as it may sound, are actually manageable. 

Top 20 countries/regions with most vulnerable Memcached servers (UDP) exploited to reflect amplification attacks (Nexusguard data, compiled based on actual attacks logged and mitigated, as of March 8)

Related blog post:

Mitigating The Record-Setting Memcached DDoS Attacks