One of the fastest-growing uses of the Internet is coming from AI scrapers. These are companies that crawl the web and scrape content to sell to AI companies to train models or sell to others looking for specific kinds of content. The scraping is done by bots that visit a website and copy the content. This is all machine-to-machine traffic with no human involvement. This traffic doesn’t reach to homes or businesses but instead impacts the Internet by increasing the traffic between data centers on backbone fiber routes and at Internet hubs.
According to a report from Tollbit, at the beginning of 2025, there was one visit by an AI bot to a website for every 200 human visits. By the end of the year, that ratio dropped to one AI bot visit for every 31 human visits.
Scraping websites for content is a controversial activity since it often involves grabbing copyrighted materials from news sites and other sources. A lot of companies have tried to make it hard to scrape the content from their site, but even the most sophisticated tech companies are finding the practice difficult to stop or block. The scraping bots are often successfully snagging content that is behind paywalls.
The practice is playing havoc with the traditional web compensation model of paying for click-throughs to websites. The click-throughs generated to websites are dropping as people use AI sites to answer questions instead of visiting the raw source of information. As an increasing percentage of searches are done through AI, the traditional compensation model will become obsolete – which is bad news for content providers. This also spells long-term trouble for search engines like Google that make money by directing users to websites.
Content providers claim that the AI bots are bypassing the robot.txt web protocol that provides a directory to web scrawlers to define the parts of a website that are open to scraping and areas that are copyrighted and off limits.
The AI bots are becoming increasingly able to mimic human users to avoid capthca and other tools used to frustrate bots. There is a technology battle being waged as content providers try new ways to frustrate bots and bots find new ways to bypass any blocking tools.
There are some well-known lawsuits from content providers to try to stop scrapers. In October 2025, Reddit sued Perplexity and related businesses. The Reddit suit claims that bots are illegally taking copyrighted materials. The Reddit suit also attacks the tactic of bots that mimic human users to grab content.
Google filed a suit in December 2025 against SerpApi for grabbing copyrighted materials, and also complains that the bot company uses illegal tactics such as armies of bots to overwhelm servers and fake and constantly changing user names to frustrate security measures.
There are a lot of consequences from this growing traffic for the companies that control the web. AI bot traffic is growing at a rate of over 25% annually while human visits to websites have started to decline. People don’t feel a need to visit websites when they can get questions answered by an AI search engine. The companies funding the basic Internet infrastructure are getting no compensation for the giant use of web resources.
There is no question that this is going to change the way that content providers get compensated. A lot of content providers sell access to articles behind paywalls, and that will fail as a revenue model if their content is made widely available through AI by scrapers.






