- AI bots and scrapers could soon overtake human users on the Internet
- Publishers struggle with falling click-through rates
- robots.txt instructions are widely ignored
New data claims that AI bots are rapidly taking over web traffic, with Tollbit information showing that there was one new AI bot visit for every 31 human visits in the final months of 2025, up from 1:200 in early 2025.
At the same time, and as humans interact more directly with AI, human visits decreased by approximately 5% between the third and fourth quarters of 2025.
And it’s clear that the use of human AI is responsible for this change: while training crawls decreased by around 15% between the second and fourth quarters of 2025, RAG bots increased by around 33% and AI search indexers by around 59%.
The way we access the Internet is fundamentally changing
OpenAI leads the way in scraping: its RAG bot “ChatGPT-User” was about five times more active than Meta’s second most active bot, and about 16 times higher than Perplexity’s agent.
In terms of user behavior, a separate survey cited in the report found that more than a third (37%) of active AI users now initiate searches with artificial intelligence (like ChatGPT or Gemini) rather than traditional search.
Even more alarming, the analysis found that the robots.txt file, a set of instructions telling automated bots what parts of the page they can and cannot crawl, was ignored about 30% of the time on average, and up to 42% of the time by the ChatGPT user.
Since the robots.txt file was inapplicable and completely dependent on goodwill, the instructions were deemed effectively obsolete.
And that’s not the only bad news for publishers, who are struggling on the traffic front. Sites without direct AI licensing agreements saw their click-through rates (CTR) drop approximately 3x between Q2 and Q4 2025. Even those with AI licensing agreements are not immune, with CTR rates also falling.
Faced with all this, Tollbit calls on regulators to intervene to establish acceptable AI models and protect intellectual property.
But with AI going nowhere, it’s clear that the Internet is undergoing a major change and website owners should prepare for AI bots to become their primary readers.
Follow TechRadar on Google News And add us as your favorite source to get our news, reviews and expert opinions in your feeds. Make sure to click the Follow button!
And of course you can too follow TechRadar on TikTok for news, reviews, unboxings in video form and receive regular updates from us on WhatsApp Also.
