Bots vs. publishers: The coming clash over internet economics

Why Cloudflare is calling out Perplexity’s tactics, and how it affects not just the media, but the entire web economy.

Returning from vacation this week, I wondered how things would look now that GPT-5 has been out for almost two weeks. Given all the hype prior to the launch, it felt like it was going to be a major pivot point in AI. And it still might turn out to be that, but mostly I'm struck by how the conversation has been mostly about all the backpedaling the company has been forced to do, and less about GPT-5's abilities, performance, and impact.

In other words, I don't think I missed much, and as far as the media is concerned, the biggest news this summer isn't GPT-5; It's the ongoing standoff over AI bots. Cloudflare reframed the issue earlier this summer when it took a clear stance by blocking bots by default on its network, and now it's escalated the conflict by calling out Perplexity directly. What happens next could reshape the economics of the AI internet, with big consequences for publishers.

More on that in a minute, but there's something else happening this summer: PR agencies are getting serious about AI. On the consulting side of my business, I've had more inbound interest in the past month than I've had in the first six months of 2025. That might be because The Media Copilot, along with my partners at The Upgrade, now offer comprehensive AI training packages for PR and comms teams, but I also think that the advancements of models like GPT-5 have enabled so much that you couldn't do, even in 2024.

If you'd like to learn more about what you can do with AI as a PR professional or a journalist, you're in luck: The Media Copilot is offering courses for both, each starting the week of Sept. 8. These aren't just a bunch of demos—the whole idea is, over the course of six weeks, to give you hands-on experience using AI tools in media-specific use cases to transform the way you work. Scroll down just a bit to learn more and grab some discount codes.

Also, my monthly class on AI for beginners, AI Quick Start, is happening this Thursday at 1 p.m. ET. Like everything I do, the class is tailored to professionals in media, PR, and communications, and this will be the first time I'll share my findings on agents, including Perplexity's Comet browser and ChatGPT Agent. Sign up here for just $49.

Finally, a quick reminder: No Thursday newsletter this week. But you can follow me on LinkedIn for more frequent commentary on what's happening in AI and media.

---

LEVEL UP YOUR AI SKILLS THIS FALL

You hear it everywhere now: AI is changing workplaces across industries. Jobs are being redefined. If you’re still monkeying around with some favorite prompts in ChatGPT, chances are you’re behind. And if you’re leading a team, you can’t put off AI training anymore.

That’s why The Media Copilot has partnered with The Upgrade to offer two tightly focused AI training courses—one for journalists, one for PR pros—to show how you can go beyond the prompt and transform the way you work. Kicking off in September, these six-week programs focus on media workflows and real productivity gains, with 1:1 coaching and a final project to cement your skills.

💥 SAVE 15% at checkout with these codes:

🔹 AI for Journalists: Use code AIJO15OFF-SEP (checkout link)
🔹 AI for PR & Comms: Use code AIPR15OFF-FL1 (checkout link)

AI is changing the rules. Make sure you know how to play the game.

---

How does the web economy change when robots to most of the reading? (Credit: Midjourney)

Who controls access in the age of AI agents?

A war is brewing in the world of AI agents. After announcing in July that it would begin blocking AI crawlers by default, Cloudflare then directly accused Perplexity of skirting web standards to scrape content. In a pointed blog post, Cloudflare detailed how Perplexity allegedly circumvents these blocks, including the use of third-party crawlers to access restricted websites.

Perplexity didn’t waste time issuing a rebuttal. In its own post, the company claimed Cloudflare was overstating its reliance on third-party tools. More significantly, it argued that Cloudflare fundamentally misunderstood (and thus misrepresented) how its bots function. Because Perplexity's agents respond to specific user requests—rather than performing broad web crawls—the company insists those bots should be given the same level of access a human would have.

This dispute cuts straight to the core of how the AI-powered internet will operate. Finding common ground here is crucial, not just for bots and developers, but for media companies and the entire landscape of information access. Importantly, Perplexity did not deny that its bots bypass the Robots Exclusion Protocol (robots.txt), but rather said ignoring it was justified. The logic: If you wouldn't deny the content to a person, you should also provide it to a bot acting on behalf of that person.

When bots wear human masks

The main point of contention is over a specific kind of bot. User-agent bots don't perform the same actions as training bots or search crawlers. They’re not hoovering up data to feed large models or index the web. These bots are task-oriented—they fetch information in real time, tailored to a specific prompt. Ask a chatbot for your local hair salon’s business hours, and it will dispatch a bot to retrieve them on the spot. According to Perplexity, that data isn’t saved in a big central database.

From the user’s perspective, though, the distinction is murky. When you pose a question to a chatbot, you can’t easily tell whether the answer comes from training data, a search index, or live agent action. You just expect it to work—and to pull in the best info available. Increasingly, that means bots querying external sources in real time, a trend that’s rapidly driving up agent activity online.