The bot blockade
Cloudflare might kickstart a new era where bots have to pay to play. Plus: AI takes on newsroom bias, fake experts at Elle, and more
Just when you thought the AI news cycle might slow down for the summer, on July 1 Cloudflare dropped this nugget: it would be switching to an opt-in policy for AI scraping, meaning customers using its infrastructure would start blocking bots by default. That has big implications, which I get into below, but first a quick note that this week's podcast with Hugging Face's Florent Daudens won't publish until Monday on account of the July 4 holiday here in the States. It's going to be a great day for a family barbecue (my plans), but if you're one of those people who likes to use the day to map out the rest of your month, I've got an AI class coming up in a couple of weeks you might be interested in… 👇
STOP DABBLING. START DOMINATING.
AI isn’t a fad. For most journalists now, it’s simply the reality. Yet many reporters are still stuck dabbling: using ChatGPT for story ideas or a few headlines and calling it a day.
It’s time to level up.
Use the summer to get an unfair advantage: a one-hour power session on how to actually use AI like a pro. Research smarter. Pitch sharper. Write faster—without losing your voice to the algorithm.
Here’s what you’ll walk away with:
The real state of AI in media—no hype, just how it’s changing the game.
A simple way to approach prompting to get what you actually want.
Pro-level use cases: Find better sources. Focus your stories. Protect your data.
📅 Live: July 17, 1 p.m. ET
💸 $49: The cheapest upskill you’ll make all year.
🎁 Extras: Slides, prompt templates, and a quickstart AI guide to keep you sharp.
👉 Don’t just survive AI. Use it to do better journalism.
Save your spot now—before someone else does your job faster.
Cloudflare puts AI scrapers on notice
Last week AI startup TollBit painted a grim picture of the world of AI scraping in its latest State of the Bots report, citing data that shows AI crawlers are harvesting content at unprecedented levels, often ignoring protections. This week, one of the biggest infrastructure players of the internet is doing something about it.
Cloudflare, which provides internet services that help websites and applications operate securely, just announced it would block AI crawlers by default. That means, if you're a website operator, you'll now need to specifically opt in to having your website scraped by AI. If you don't, the bots get blocked.
Considering Cloudflare manages about 20% of all internet traffic, the potential implications are significant. And so is the business opportunity: With the announcement, Cloudflare is also launching a new marketplace for bot traffic. Instead of blocking AI bots outright, site owners will be able to charge them a fee to access content. The Pay Per Crawl program is essentially a micropayment system for content similar to TollBit's, but given Cloudflare's scale, it may have just become the biggest player in that space.
Another important detail: Cloudflare's blocking system can discern between AI crawling for training, inference (AKA "RAG"), and regular ol' search, so it asks bots to identify what they are so website operators can pick and choose which ones they let through. Curious what this means for Google bots: Will Cloudflare block the Google bot—which famously is the same for both AI and search—by default? It seems unlikely.
The move isn't out of nowhere. Cloudflare launched a one-click option to block AI crawlers last year; this is a logical, if significant next step. It was also deeply planned: The press release includes several quotes from media executives like Neil Vogel of Dotdash Meredith, Mark Howard at TIME, and Bill Gross at ProRata.ai, among many others. Cloudflare CEO Matthew Prince gave full-throated support to content creators and an open web in his statement: "Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it."
If the internet is going to move to a new economy of information, it's going to take more than lawsuits and outdated technical standards that rely on the honor system of the Robots Exclusion Protocol. And there is power in the default; when faced with a choice, most people don't make one, which means bot blocking among Cloudflare's users can only increase from here, potentially giving publishers a little more breathing room before AI subsumes the web. Cloudflare alone can't stop the winds of change. But maybe it can give those trapped in the gale something to hang onto.
The Chatbox
All the AI news that matters to media*
AI hunts for bias at Law360
Law360 isn't the first publication to leverage AI to help correct for media bias, but it might be the first to do so with its editorial team kicking and screaming. After the publication began requiring that all articles be run through its bias indicator, the editorial union denounced the move and called for management to return to making its use voluntary, according to Nieman Lab. Weeks earlier a company executive accused the newsroom of liberal bias, and afterward Editor-in-Chief Anne Urda reportedly told staffers that the decision to implement the bias indicator was made "above her."
Staffers criticized many of the indicator's outputs, and even though they weren't required to accept any suggestions from the tool, they still feared disciplinary action if they didn't "go through the motions" of using it. True, you might blame the editorial team for not giving the indicator a fair shot. You might also blame management for turning to a barely tested AI tool to solve its bias problem when a more collaborative and constructive approach was called for.
AI is winning the battle for attention
For media, the hits may keep on coming, but at least we have data on how many. A new report from Similarweb, reported in TechCrunch, shows that news searches that resulted in zero click-throughs to websites increased from 56% to 69% since May of last year. The report looks at traffic to news sites specifically, and it's mostly bad news: Even though referral traffic to news sites from ChatGPT has increased 25x in one year, it's still minuscule compared to search, sending only 25 million clicks. The biggest topics for chatbots? Stocks and finance, which Similarweb theorizes represents more "explanatory" use of AI. With more going on in the chat, don't expect the numbers for publishers to get any better.
The Guardian's archive meets Claude
A new tool from Nieman Lab's Joshua Benton shows how publishers might strategically connect their archives to AI systems. His Guardian MCP Server, which plugs into 2 million Guardian articles via Anthropic's Model Context Protocol, lets anyone analyze Guardian stories in sophisticated ways—tracking coverage trends, finding similar articles, and even studying individual reporters' beats. Benton's tool shows what you can do with AI and a little technical knowhow, and although The Guardian's 16-year-old open API policy enabled this experiment, he says it's a proof-of-concept for how a publisher could connect its archives to an AI interface, externally or internally. (AI-assisted)
When AI ‘journalists’ become fake experts
It's a tale as old as… well, ChatGPT: AI content masquerading as legitimate journalism when publishers prioritize cost-cutting over credibility. Only this time it involved some big names: the Belgian editions of Elle and Marie Claire. As reported by The Times of London, Ventures Media's decision to create hundreds of articles under fake bylines—complete with AI-generated headshots and fictional biographies—goes way beyond just AI assistance, replacing human expertise entirely. While the publisher claims this was merely a "test," the scale suggests otherwise: One fictitious writer produced 403 articles in a single year. Whatever the savings were for using AI instead of real writers, it's certainly not worth the erosion of trust. (AI-assisted)
Book publishers face AI ultimatum from authors
Authors are turning up the heat on publishers with a new open letter signed by over 70 writers, including Dennis Lehane and Lauren Groff, demanding the "big five" publishing houses promise never to release AI-generated books, NPR reports. The petition, which garnered 1,100+ signatures in 24 hours, arrives as recent federal court rulings have given AI companies broader rights to train on copyrighted works under fair use doctrine. This puts publishers squarely in the crosshairs as what organizer Riley Redgate calls "the next—and possibly last—line of defense" against AI replacing human writers. The letter targets specific concerns beyond copyright, including AI narration threatening authors' audiobook income and the proliferation of fake AI books on Amazon masquerading under real authors' names. With only Simon & Schuster responding so far with vague assurances about protecting intellectual property rights, the campaign highlights how creative industries are shifting from purely legal battles to direct pressure on the gatekeepers who control distribution and audience access. (AI-assisted)
*Some items are AI-assisted. For more on what this means, see this note.