20 Jan 22
Ransomware. Significado y cómo identificarlo
Actualmente se habla mucho del ransomware. En este artículo queremos contarte qué es y cómo identificarlo. Su nombre deriva de […]
Mission: web security
One of the key aspects of Transparent Edge’s next-generation CDN, it’s how serious we are when it comes to web application security. In this new post of the CDN invisibles , (meaning, those who integrate the support team at the only Spanish CDN), we want to tell you how we recently discovered someone probing the vulnerabilities of one of our clients. We can’t tell you which one, but we can assure you that you know their website.
It all started during the early morning review of our monitoring systems, we noticed that the previous night there was a significant increase in the MISS (cache misses) traffic associated with this client. This MISS traffic was the result of a number of requests for resources that, at the time, weren’t cached in our platform. What could be the reason for this?
The principle of Ockham’s razor states that, all things being equal, the simplest explanation is usually the correct one. Our first hypothesis, based on past experiences, was that there was a content invalidation through regular expressions, colloquially referred to as “banning”.
Our platform allows clients to manually invalidate content, when said content becomes obsolete. For instance, if a news website needs to display a new homepage. This way, interacting both through our panel or our API, it’s possible to invalidate content that the client needs at a specific time.
Simply put, we recognize two types of invalidations: purging, that affects individual resources, and banning, that operates through regular expressions and removes all objects from the cache that match said expressions.
As you may have deduced, banning can be dangerous. If we don’t properly control it, we may be shooting ourselves in the foot, quickly purging out of the cache a large number of objects.
Obviously, the subsequent requests for these purged objects will result as a MISS, and they will have to be retrieved from the client’s backend platform.
We started doubting this initial hypothesis when we checked the data. In our database, we thoroughly register the operations of our clients, so it was as simple as checking the history of invalidations carried out by said client. There we saw that the invalidations that took place during the time of the MISS traffic peak were completely normal. Also, the trend of said MISS traffic was interesting: it had grown exponentially and then just as quickly, decreased. Actually, by studying the graphics of our monitoring systems, we saw how this unusual behavior only lasted four minutes: from 22.22 to 22.26. So, if this traffic hadn’t occurred due to an invalidation, what was the reason behind it?
At this point, and with web security in mind, the next step was to get our hands dirty and analyze the raw data extracted from the log files, where we have the precise records of all the traffic processed by each and every one of the nodes that constitute our CDN (content delivery network) platform .
And here’s the surprise: between 22.22h and 22.26h, there were approximately 860.000 requests were recorded from the homepage, but each one slightly modified to change the cache key, forcing a MISS and going directly towards the client’s backend.
Studying these 860.000 requests we’ve observed that there had been a total of 2.400 requests based on different IP addresses and from random countries, such as Indonesia, Russia, Brazil, the USA, China, Colombia, India, Argentina, Mexico, Ukraine, just to name a few.
The most curious situation was when we analyzed those requests thoroughly: despite the high total, they weren’t evenly distributed. On the contrary, we saw IP addresses that had done only one or two requests. The alarming part was when we noticed the trend: the number of requests increased progressively into a mathematical sequence, reaching 10.000.
Whoever was responsible, clearly wasn’t conducting an attack: the number of requests never got to the point of compromising our platform and honestly, nobody would launch a 4-minute attack. Actually, the indication in our monitoring that made us suspicious was the increase in MISS traffic, while all other indicators (CPU usage, load, memory consumption, bandwidth usage, etc.), were never altered in any way.
The fact that an attack wasn’t taking place, doesn’t mean that it wasn’t dangerous, they were testing the waters. Whoever was responsible, presumably what they pretended to find out was how far they could go before the IPs were blocked. This is why it only acted for 4 minutes and started scaling the requests linearly: they didn’t want to make noise, and go unnoticed, but they were. Because one of our missions as a CDN is web security and we have our monitoring set up to prevent such issues.
We don’t know who did it and we probably won’t find out. What we do know is that if they decide to pay us a new visit with all their artillery, this time, we will be waiting for them. Sometimes Ockham gets it wrong, but these are the mistakes that make our job worthwhile.
Alberto Suárez López, Systems Administrator at Transparent Edge
More Asturian than cider, Albert studied Technical Management in Computer Science and completed a Postgraduate degree in Web Engineering, at the University of Oviedo. Since then, he’s been facing every possible UNIX infrastructure you can imagine and has conquered them all, thanks to his deep knowledge, that makes him a versatile weapon against any technical challenge.
Compartir:
