Scrape no more
The tide is turning against AI scraping, which could enable publishers to finally make peace with the bots scooping up their content.
I’m living the agent life. As I write this, I’ve just told Perplexity’s Comet Assistant to begin research on my next column. Earlier, it helped fill an online shopping cart with groceries for the week, help plan my attendance at Condé Nast’s Beyond the Prompt event in New York City, and scan some court documents in The New York Times's lawsuit against the OpenAI. I don’t yet have access to ChatGPT Agent, but I’m excited to see how it compares.
All of which is to say my personal AI activity shot up over the past week. As these agents catch on—and I believe they will—that’s going to mean more and more web traffic will be comprised of bots, often acting on behalf of individuals. That’s going to have big consequences for how the web evolves, which makes the framework for how we deal with bots extremely important.
Cloudflare has had a lot to say about that recently. The company, which manages a large portion of internet traffic, switched to a “block bots by default” stance for the sites on its network, which I previously wrote about. This week I’m diving deeper into the issue, first with today’s column and then again this Friday when I interview Cloudflare Chief Strategy Officer Stephanie Cohen for The Media Copilot podcast.
More on the new era of AI scraping in a minute, but first a reminder that there are only 10 days left to grab a 25% discount a spot in one of my fall AI transformation courses, which begin in September. There’s one for journalists and one for PR professionals, and they both give extensive hands-on training on how to apply AI tools (including browser agents) to your work. More details and discount codes below, but hurry—the prices go up on Aug. 1.
A MESSAGE FROM THE MEDIA COPILOT AND THE UPGRADE
Most people are still playing with AI at the surface—tweets, outlines, press releases. But the real value comes when you go deeper.
We’ve built two hands-on, live courses—one for journalists, one for PR pros—to show you how. These six-week programs focus on strategy, workflows, and real productivity gains, with 1:1 coaching and a final project to cement your skills.
Kicking off in September, both courses are six weeks and 100% live. And both come with 1:1 coaching and a capstone project guaranteed to level-up your personal workflows.
💥 Act fast: Get 25% off either course if you enroll by July 31.
🔹 Journalists: Use code AIJO25OFF-SEP (checkout link)
🔹 PR & Comms: Use code AIPR25OFF-FL1 (checkout link)
This is your AI upgrade. Don’t miss it.
The days of free AI scraping may be ending
Ever since AI chatbots burst onto the scene, it feels as if the media has been stuck on the losing side of a slow-motion war of attrition. As chatbots skyrocket in popularity, every answer they serve up instead of a search result translates into fewer clicks for the sites supplying that information. Even Google—with a business model built on monetizing search—keeps plunging deeper into AI, and the data shows bot traffic is rising fast.
However, actions have a habit of inspiring reactions. Lawsuits are mounting as more media companies take on the AI giants over copyright, which may yet prove decisive—recent rulings notwithstanding. And publishers are shoring up their website defenses against AI crawlers, blocking more of them than ever.
Now, the conflict may have reached a turning point: Cloudflare, one of the web’s backbone infrastructure providers, has stepped into the fray. In a headline-grabbing announcement, the company said it would block AI scrapers by default on the sites it serves. Site operators on Cloudflare's network must now opt in if they want AI bots indexing their pages; otherwise, those crawlers will simply be turned away.
Cloud fair
Cloudflare handles roughly 20% of global web traffic, giving the move real weight. There's also clear upside: alongside the policy shift, Cloudflare is unveiling a marketplace for bot traffic. Rather than slamming the door entirely on bots, site owners can levy a fee through the new Pay Per Crawl program—a micropayment system where bots can pay for the content they scrape. A handful of startups, including TollBit and ScalePost, already play in this space, but Cloudflare's scale could make it their most formidable rival overnight.
