Table of contents of the article:
When it comes to web security, one of the most common mistakes is designing defenses by looking at the system solely from the developer's perspective. You check the code, install a firewall, add a CAPTCHA, and assume the problem is solved.
In reality, bot builders don't look at the application the way a developer, system administrator, or end user does. They aren't interested in the page's graphical appearance, the button layout, or the quality of the browsing experience. Instead, they look at HTTP requests, endpoints, parameters, cookies, tokens, headers, and the responses returned by the server.
It is precisely from this difference in perspective that one of the most important lessons we have learned working every day with Linux infrastructures, web applications, managed services and publicly exposed systems arises: to defend an application from bots, you must first understand how a bot is built.
This doesn't mean developing tools designed to hack other people's systems. It means studying, within authorized and controlled environments, how automated software interacts with a site, maintains a session, replicates a request, and attempts to circumvent protection.
A public application is never completely closed
Every application accessible over the Internet must communicate with the user's device. The browser sends requests to the server, receives responses, downloads resources, transmits data, and calls APIs. Everything that needs to reach the client can be observed by the client itself.
We can minimize the information exposed, apply rigorous controls, and make the process extremely complex, but we cannot turn public communication into something completely invisible.
A sufficiently motivated user can analyze the traffic generated by the browser and identify:
- the addresses of the endpoints used by the application;
- the parameters sent in the requests;
- the HTTP headers required by the backend;
- cookies necessary to maintain the session;
- the structure of JSON responses;
- the order in which operations are performed;
- any tokens used by the frontend.
Once this information is understood, the graphical interface can become almost irrelevant. The bot doesn't necessarily need to open the page as a human would: it can attempt to communicate directly with the backend.
We can't completely prevent a public system from being monitored. However, we can make its automation more difficult, unstable, and expensive.
The problem is not just page scraping
When you hear the word scraping, many people think of a program that downloads the HTML content of certain pages to extract titles, prices, or descriptions. This is just one possible form of automation.
A bot can be designed to register accounts, test credentials, fill out forms, query APIs, reserve resources, check availability, send spam, download files, or replicate some of the functionality offered by an application.
From an infrastructure perspective, some automations produce very noticeable loads. They generate thousands of requests in close proximity, originate from the same IP address, and follow easily recognizable patterns.
More advanced automations behave differently. They distribute requests across multiple addresses, respect variable pauses, store cookies, execute JavaScript, and use real or headless browsers. They can even reproduce seemingly believable navigation sequences.
For this reason, simply counting requests from an IP address is not enough . The IP address is a useful signal, but it doesn't necessarily represent the identity of a visitor. Multiple legitimate users can share the same address, while a single automated operator can use hundreds of them.
CAPTCHA is useful, but it is not a complete solution
When a form is abused, the first reaction is often to insert a CAPTCHA. This is understandable: introducing an additional verification can stop many basic scripts and quickly reduce spam and automated requests.
The problem arises when the CAPTCHA is considered an absolute guarantee.
This type of verification is just one step in the process. If the application poorly checks the result on the backend, accepts reusable tokens, or secures the form but not the endpoint receiving the data, the bot can attempt to bypass the interface and communicate directly with the server.
Additionally, an automation can use a full browser, wait for the verification to pass, and then proceed with the requests. In some scenarios, a human can even complete the check, while the rest of the operations are performed automatically.
This doesn't make the CAPTCHA useless. It just means it needs to be part of a broader strategy, in which the backend verifies the token, its expiration date, the action for which it was issued, and its consistency with the current session.
Client-side protection that is not validated properly on the server-side is, at best, a temporary obstacle.
The first line of defense must be built on the perimeter
In the infrastructures we manage, some of the protection is applied before the request reaches the application itself. Services such as Web Application Firewall, rate limiting systems, reputation analysis, and bot management tools help reduce unwanted traffic right at the edge.
For example, we can apply different limits to login, registration, search, or password recovery endpoints. We can introduce additional checks when a source exceeds a certain frequency, exhibits anomalous characteristics, or repeatedly attempts the same operation.
It's important to avoid overly generic configurations. A valid threshold for a login page might be completely inappropriate for an API used by a mobile application or for a page that loads numerous resources in parallel.
Protection must therefore be contextual . It's not just how many requests are sent that matter, but also which resource is requested, in what sequence, with which credentials, and with what result.
The perimeter can significantly slow down a bot, but it shouldn't be the only security layer. If the application accepts any formally valid request, a sufficiently sophisticated automation system could continue to operate at a slower speed.
Protection must continue within the application
A more robust defense requires the backend to not only check for the presence of the expected parameters, but also check the context in which those parameters were generated.
In a sensitive flow, we can use signed tokens, unpredictable random values, and identifiers that are only valid for a limited time. These elements can be associated with the session, the user, the requested action, or a specific stage in the process.
When the server receives the request, it should at least verify that:
- the token was actually generated by the backend;
- has not expired;
- has not already been used;
- is connected to the correct session;
- is valid for the requested operation;
- the sequence of operations is consistent.
This way, it's not enough to simply copy a single request observed in the browser. The bot must reconstruct the entire process, maintain proper state, and obtain new values on each attempt.
You can make some form elements dynamic, insert certain fields only after a JavaScript check has been performed, or periodically modify the request structure. These techniques can increase the cost of automation, but they shouldn't replace server-side validation.
JavaScript obfuscation alone isn't a definitive security measure. The browser-facing code must still be downloaded and executed. It can be made more difficult to analyze, but it can't be considered secret.
The final decision must always belong to the server.
Building a test bot changes the way you design your defense
When analyzing the security of an application flow, we also try to consider it from an automation perspective. Within systems we own or for which we have received explicit authorization, we can build test clients that simulate different levels of complexity.
The first layer can be a simple HTTP client that sends requests without running JavaScript. The second layer can handle cookies, redirects, tokens, and sessions. A subsequent layer can use an automated browser and reproduce a more realistic browsing experience.
This approach allows us to answer very concrete questions:
- Can the endpoint be called without visiting the page?
- Can a token be reused multiple times?
- Is the verification actually associated with the session?
- Is the sequence of operations checked?
- Does the system distinguish a single error from hundreds of attempts?
- Can a limitation be circumvented by simply changing the IP address?
Without this type of testing, you risk implementing protections that are theoretically correct but ineffective in practice. Seeing your system running through an unofficial client immediately highlights what information is actually needed and which checks can be bypassed.
There is no single signal capable of identifying all bots
An unusual user agent can be suspicious, but it can be changed. A bad IP address is a strong indication, but it can be replaced. The absence of JavaScript can reveal a basic script, but automated browsers can execute it.
Even seemingly human behavior isn't a guarantee. Random pauses, mouse movements, and click sequences can be simulated.
An effective strategy therefore combines multiple pieces of information: request rate, session continuity, header consistency, source reputation, JavaScript check results, errors generated, navigation paths, and historical behavior.
A single signal can produce false positives. Combining multiple signals allows you to assign a risk level and choose a proportionate response.
A moderately suspicious request may receive additional scrutiny. A highly anomalous request may be slowed or blocked. A recognized and useful crawler, such as a search engine, may be allowed.
The goal is not to block all automation, but to distinguish the useful from the harmful.
Blocking too much can be as harmful as blocking too little
An overly aggressive anti-bot configuration can prevent access by real users, break legitimate integrations, block tracking systems, or compromise your site's indexing.
For this reason, each rule must be observed after activation. It's important to monitor generated events, blocked requests, completed checks, and any issues reported by users.
Thresholds must be adjusted to actual traffic. An e-commerce site, a publishing portal, a management system, and a public API have completely different behaviors. Copying the same configuration from one project to another can lead to unpredictable results.
Anti-bot security isn't a one-time switch. It's a process of continuous observation and correction.
The realistic goal is to increase the cost of the attack
No protection can guarantee that a public application will never be automated. A party with sufficient time, expertise, and resources can continue to study the system and adapt their tools.
This doesn't mean that defenses are useless. On the contrary, their value lies in altering the economic balance of the operation.
A bot that's simple to develop, fast, and stable can be cost-effective. However, if it requires full-fledged browsers, proxies, frequent updates, complex session management, and constant manual intervention, its cost will quickly escalate.
Each additional layer can reduce the speed of automation, increase errors, and require constant maintenance. In many cases, it's enough to increase the cost of abuse beyond the benefit.
A good anti-bot strategy doesn't promise the impossible: it turns a simple, convenient operation into a slow, fragile, and expensive one.
Our approach as a Managed Server
At Managed Server Srl, we address these issues starting with the infrastructure, but not stopping there. Linux servers, reverse proxies, CDNs, firewalls, and perimeter protection systems are essential, but they must communicate with the application logic.
A rate limiting rule alone can't determine the commercial value of a transaction. An application, however, can distinguish a search from a purchase, a failed attempt from a successful login, or a newly created account from a long-standing customer.
To do this, we work on multiple levels: log analysis, web server configuration, endpoint protection, session management, anomaly observation and coordination with developers, as well as the implementation of proprietary WAFs and behavioral analysis.
The starting point remains the same: understanding how the system could be automated . Only then can we decide which steps to protect, which signals to collect, and where to introduce friction without penalizing legitimate users.
Defending yourself means changing your perspective
Those who design an application naturally tend to follow its intended path. Those who build a bot, on the other hand, look for the shortest, most repeatable, and least expensive route.
Learning to think this way allows you to identify overly permissive endpoints, reusable tokens, controls present only in the frontend, and flows that the backend does not verify correctly.
Offensive knowledge, exercised ethically and appropriately, improves defensive capability. Not because it allows for the creation of an inviolable barrier, but because it helps avoid purely decorative protections.
On the web, everything transmitted can be observed and, with enough effort, imitated. Our task is not to magically hide what needs to be communicated, but to build systems in which every request is evaluated, contextualized, and verified.
This is the difference between adding a CAPTCHA and designing a true anti-bot strategy. And it's also why, before attempting to stop an automation, you need to learn to understand it.
