A web developer managing a site with roughly 50,000 statically generated public pages posted a plea to Hacker News this week, describing a bot traffic problem that has spiraled beyond anything they have dealt with before. The post struck a nerve, drawing attention to an escalating arms race between site operators and AI scrapers that shows no signs of slowing down.
Facebook alone requested 3 million pages in 48 hours
The numbers are staggering. Facebook's crawler requested the same set of pages more than 3 million times in a two-day window. That is not normal bot behavior, and the developer says they have deployed every tool at their disposal, Vercel and Cloudflare among them, to block the traffic. None of it has worked. The scrapers adapt faster than the defenses.
The pattern is not unique to one site. As AI companies race to feed their models with web content, the volume of scraping traffic has increased dramatically across the internet. What used to be manageable, occasional bots indexing pages for search engines has become a flood of automated requests that consume bandwidth, inflate hosting bills, and degrade performance for actual human users.
Residential proxies make blocking nearly impossible
The most troubling development is the sophistication of the scraping infrastructure. AI scrapers are routing their requests through residential proxy networks, which means the traffic originates from IP addresses belonging to real home internet connections. This makes traditional IP-based blocking effectively useless, because blocking a residential IP means blocking a real person's connection.
When the developer put their entire application behind an authentication gate using Cloudflare Turnstile, the bots adapted again. They began creating accounts to gain access to content that was meant exclusively for human readers. The scrapers are not just fetching pages anymore. They are simulating user behavior to bypass access controls.
Turnstile is not holding up
Cloudflare Turnstile was designed as a less intrusive alternative to CAPTCHA, using behavioral analysis to distinguish humans from bots. In practice, it appears to be failing against determined AI scraping operations. The developer plans to switch to hCaptcha, which is generally considered more difficult for automated systems to solve, but acknowledged the move feels like a last resort.
The broader problem is that each defensive measure triggers a countermeasure. Block an IP range and the scraper moves to residential proxies. Add a CAPTCHA and the scraper trains a solver or pays for human CAPTCHA-solving services. Require account creation and the scraper automates signups. The cycle has no natural endpoint.
Hosting costs are spiking across the board
Beyond the security implications, there is a financial dimension that hits smaller operators hardest. Hosting costs for the site in question have increased significantly, driven entirely by bot traffic. Static site hosting platforms charge based on bandwidth and serverless function invocations, both of which scale with request volume. When a single crawler makes millions of requests, the bill reflects that activity regardless of whether a human ever sees the content.
This creates a perverse dynamic where running a public website becomes more expensive not because more people are reading it, but because more machines are scraping it. The developer described the situation as depressing, and the sentiment is widespread among site operators who are watching their infrastructure costs climb for traffic that provides no value.
The trust problem extends beyond traffic
The developer made a point that goes deeper than bot traffic: they can no longer trust anything they read or see on the web. This is a symptom of the same underlying issue. When AI companies scrape content at scale, they ingest it into models that reproduce it in new contexts, often without attribution or accuracy guarantees. The original source loses control over how its content is used, and readers lose the ability to verify provenance.
For a developer running 50,000 pages of presumably original content, the experience is especially frustrating. The content was created for human consumption. It is being consumed primarily by machines, and those machines are consuming it aggressively enough to cause real financial and operational harm.
This is not a niche problem
The Hacker News discussion drew responses from other developers dealing with identical issues. Some reported similar spikes from AI-focused crawlers. Others described abandoning public-facing content entirely, moving everything behind authentication as the only reliable way to control access. A few expressed concern that the web as a platform for open publishing is becoming unsustainable for independent operators.
The AI industry's demand for training data is driving this behavior, and there is currently no regulatory framework that addresses the problem. Robots.txt, the longstanding convention for signaling which pages should not be crawled, is treated as a suggestion rather than a rule by most AI scraping operations. The technical community has been discussing alternatives, including pay-per-crawl models and content provenance standards, but none have reached widespread adoption.
For now, developers are left building increasingly elaborate defenses around their content, fighting an arms race against well-funded operations with every incentive to keep scraping. The developer's question to Hacker News was straightforward: is anyone else struggling with this? The answer, based on the responses, is a resounding yes.