All news
Bot yet: how “forgotten” automation is dangerous for business
Some time ago, while linking DDoS and bot protection for a large e-commerce company, we discovered significant automated activity in the subnet of a major credit bureau. It turned out that our client was actually being attacked by a bank, sending them numerous requests to the service on a regular basis. The high load on the client's servers was noticeable during peak hours. This did not result in unavailability or noticeable performance degradation of the client's websites and mobile applications due to the large amount of spare capacity.
As further joint analysis with the customer showed, the cause of this "malicious" activity was not hackers, saboteurs-employees of the same bank or competitors, but a manifestation of elementary carelessness and disorder in the creation and support of automation. And periodic attacks are not attacks at all, but simply greedy and incredibly persistent automation, which was forgotten a year and a half ago. It cannot be said that we were very surprised. Every third company connecting a solution for protection against bots or DDoS attacks at the application level encounters one or another volume of traffic of a similar nature.
In fact, one of the stages of connecting resources to protect against a wide range of illegal actions is “training” in solutions for working with client traffic. Experts analyze the traffic received over several days and divide it into groups. The “Browser” traffic group is generally the most understandable and predictable. Using a standard browser (Yandex.Browser, Chrome, Firefox, etc.), the user sends a request to the client service. The groups of client mobile applications are known to the company itself, and anti-bots are trained on their traffic, so they are well differentiated.
Things get a little more complicated when you use automated actions. This can be legal or illegal. An example of clearly legitimate automation is search engine queries. But there are other automations that, in the simplest case, look like constant or periodic traffic from a known hosting subnet sending a lot of similar queries. In fact, large and medium-sized services usually have useful automation features as well. When bots request information for different purposes and tasks, as part of partnerships with third-party services, and even within different services of the same group of companies.
Basically, when preparing traffic for a new service, our specialists participate in the process of auditing all existing automation. The automation parameters (called "Traffic Source Subnet", "Agent Type" and "Load Characterization Method") are highlighted. Then the customer engineer should say: "This is our automation, let's skip it" or "I will delete it". In fact, in every third case, the customer's IT representative, in addition to expectations, answers the following: "Yes, this is our Zabbix" and "This is definitely a bot". A long time ago, we were using an old, slow version of the API, which added stress to us. " Or even like this: "It looks like it was configured by a team that left long ago, and it is not clear why it works and what is the benefit. Now we will just mark it as suspicious and conduct a subsequent check. We will decide what to do with it." And in almost every large company we encounter situations where engineers say: “Yes, these are our partners, but for some reason we haven’t agreed with them: “I’ll go with you to solve this.”
Forgotten botnets are not as harmless as they seem at first glance. Often, those requests that never load the service with thousands of users survive, but now they dramatically slow down the work of systems that remember millions of clients, products, transactions and other objects. And this automation consumes computing resources for years. In some cases, once useful automation that has become irrelevant but continues to work can account for up to 30% of the total traffic, which already creates a significant load on the servers. And the larger the service, the longer it lives and the more it cooperates with the services of other teams and companies, the more likely it is that you will encounter “forgotten” automation.
Experience shows that if the service was launched 3-4 years ago, has tens of thousands of unique users per week and cooperates with partner services, then such unexplained automation is almost always encountered, and not a single instance exists. And if your IT team has changed more than once, you may not be able to determine when and by whom the decision was made to establish interaction with the same partner, as in the case of e-commerce clients. This is a bank, but after a 3-week process, the connection was disconnected, but this did not affect anything. The load was simply reduced.
The essence of the emergence of "forgotten" automation is the lack of accounting and blurring of responsibility boundaries in combination with the rational principle of "why break something that works". For now, you can get around this problem by buying additional capacity. This is usually easier and cheaper. But even in this case, if you do not regularly clean your services and do not allocate resources for audit and inventory, your technical debt burden may be unacceptable. As practice shows, after internal or external circumstances restore order to resources, the service begins to "breathe" more freely. Monitoring errors are eliminated, the load on the server is reduced, and the response time is improved.
The editor's opinions may not coincide with the author's point of view.