The secret flaw in the internet of agents
A new report shows AI search engines are quietly sidestepping robots.txt, and the media could end up paying dearly.
What will happen when agents are in the hands of everyone? For starters, it seems clear people won't visit websites in anywhere near the numbers they used to, letting bots do the hard work of mining information from the web and bringing it directly to them. That's not me saying that—it's TollBit, whose State of the Bots report for Q2 2025 showed just how strange things are getting out there.
I went deep on the report with CEO Toshit Panigrahi in the most recent Media Copilot podcast, but today I'm zeroing in on one of its revelations: that several of the major AI companies are now bypassing one of the internet's age-old standards for managing bot traffic. It feels to me like a turning point where they moved from liberally interpreting the mechanisms for protecting content to brazenly flouting them.
More on that in a minute, but quick reminder that The Media Copilot's AI Quick Start class for journalists and PR professionals is coming up soon, on July 17. I'll also soon have much more to share about the fall cohorts for my six-week AI training programs that I'm doing in partnership with The Upgrade, but you can preview what I've got in store here (journalists) and here (PR/comms).
A MESSAGE FROM THE MEDIA COPILOT
AI is officially a daily reality for media pros. Over 60% of people in journalism and communications now tap AI for all kinds of tasks, letting them research, brainstorm, draft, and pitch faster. But most are still just dabbling, dropping quick questions into ChatGPT and moving on.
This one-hour AI power session is your chance to level up. No shallow tool tour — just clear, practical ways to turn AI into a serious advantage for journalism and PR.
You’ll get the strategy and confidence to use AI well — without losing your edge or your voice.
You’ll learn:
Where AI is really transforming media right now
The AI mindset top pros use to get ahead of the curve
How to put AI to work — from smarter pitching to sharper stories
Details:
🗓️ July 17, 1 p.m. ET
💸 $49 — a smart spend to stay relevant
🎁 Includes slides, pro-level prompts, and a quickstart AI guide
AI agents are rewriting how web traffic works, and the media isn’t ready
Imagine you owned a bookstore. A big chunk of your income relies on customers walking in and buying books, so you design your business around that behavior. You might stock cheap “impulse buy” items near the register or add a coffee bar as an extra draw. You might even cut deals with publishers to showcase popular bestsellers in prime spots to help move more copies.
Now imagine that one day a robot walks in to buy a book on behalf of someone. It walks right by the coffee stand, ignores the tchotchkes by the register, skips the fancy displays — it just picks up the book it came for, pays, and leaves. The next day, four robots come in, then 12 the day after. Soon, robots outnumber your human customers, which are dwindling every day. Sales from non-book items evaporate, publishers stop bothering with those displays, and the coffee goes cold. Revenue tanks.
In response, maybe you start charging robots an entry fee — and if they refuse, you bar them from entry. But then one day a robot that looks exactly like a person comes in — to the point that you can't tell the difference? Then what?
This isn’t just a thought experiment; it’s a good analogy of what’s happening to publishers right now. Bot traffic to media sites has surged over the past three months, according to fresh data from TollBit’s first-ever State of the Bots report for Q1 2025. What’s worse, top AI search engines are choosing to disregard long-established standards for blocking bots, in some cases arguing that when a search “agent” acts for a human, it should be treated like a human.
Robots rising
TollBit’s report reveals how quickly the search landscape is shifting. In recent months, AI firms have either rolled out new search tools or greatly increased their search activity. Bot scraping for retrieval-augmented generation, AKA “RAG," which is distinct from training data, jumped 49% over the previous quarter. Anthropic's Claude notably added web search to its toolset, and ChatGPT, still by far the most popular chatbot, had a big spike in users. On top of that, the deep research tools from all the major players really took hold.
Keep reading with a 7-day free trial
Subscribe to The Media Copilot to keep reading this post and get 7 days of free access to the full post archives.