Not every bot is the enemy
Blocking every bot feels safe. It rarely pays off. Plus: newsroom cuts top 2,300, Meta’s crawler math, and Germany’s new AI rules.
Not every bot hitting your server is a threat. This week’s column makes the case for sorting AI crawlers by function instead of blocking them all on reflex, since some of the ones publishers kick out first are quietly building the authority that gets a source cited in AI answers.
Also in this issue: The AI effect on the media ecosystem is getting real. Google search traffic to UK publishers is on pace to halve within two years, and Meta’s crawlers logged 9.1 billion hits last quarter without sending a single reader back.
AI training built for your team
The Media Copilot also runs hands-on, custom AI training for media, PR, and communications teams, built around your actual tools and workflows instead of generic slides.
The focus is work transformation, not just skill building: We run workshops that teach people how to build the muscle memory of working alongside AI, treating it like an eager assistant, not just a tool set. We unleash the potential of AI on your work, and show you how to become the CEO of your job, with AI agents multiplying your ability to get things done.
Every program is custom to the client. A typical program includes:
2–6 live sessions shaped around your team’s goals
Assessment, training, and custom AI tool-building, not a single workshop
Built on your real documents, clients, and workflows
Our fall calendar is open, but we take on a limited number of team partnerships each quarter, If you’re considering upskilling your team with AI, be sure to get in touch as soon as possible 👇
The bots publishers should be letting through the door
Most media companies are now pretty aggressive when it comes to blocking AI bots from their content. Reports say most major publications almost universally block crawlers from major AI companies like OpenAI and Anthropic, and many small to midsize publishers mirror that, configuring their robots exclusion protocol settings, or robots.txt, to keep them out.
It’s a defensible choice. Sometimes it’s the right one. But it also crudely compresses a complex issue into a binary: Block or don’t block. And the reasoning is often just as straightforward, pointing to the fact that most AI platforms don’t give back significant human traffic. Meanwhile, the crawl volume has become punishing enough that infrastructure cost is a factor. From an ROI standpoint, it’s an easy call.
There are exceptions. Google is the big one, because presence in AI Overviews is currently welded into Search itself, so slamming the door on Googlebot isn’t really an option for most media companies (although the balance of that equation is starting to shift). And if you’ve cut a licensing deal with a specific platform, its crawler obviously gets a pass.
The bigger problem with a pure block-or-allow framing is that it treats direct traffic and direct revenue as the only things worth measuring. That’s too narrow a window to really get a full picture of the opportunity AI presents. An AI-forward strategy has to be more thorough: sort the bots by function, understand how each one connects your work to an audience, treat that audience as real even when it’s not a human clicking through, and shape how your content actually appears once it lands inside someone else’s answer.
Sort the bots before you swing the axe
Different bots are doing different jobs, and lumping them together is where most publisher policies go wrong. There are several types, but broadly three matter most to publishers: training bots, search bots, and retrieval bots. At least those are the ones that operate inside the “legitimate” ecosystem, declared, documented, and mostly respectful of robots.txt. Set the unauthorized scrapers aside for a minute.
Read the rest at mediacopilot.ai
Graduate from AI tools to AI agents
AI Quick Start is a one-hour live workshop for journalists, PR pros, and communicators who are ready to move past using AI as turbocharged Google and start putting it to work in real, practical ways.
In just 60 minutes, you’ll learn how to:
Sharpen drafts, edits, and ideas with a clear, repeatable prompting framework
Use AI responsibly without putting sources, clients, or credibility at risk
Build a reusable AI assistant that sounds like you, not a generic chatbot
You’ll also get an introduction to agents, using AI to take on bigger tasks by operating apps just like you can. Once you go agent, you never go back.
📅 Next class: Friday, Sept. 11 at 1 p.m. ET.
💳 Price: $49.
📦 Includes: prompts, deck, recording, and more.
The class works as a standalone skills boost or as an on-ramp to AI for Media, and your seat is just $49.
The Chatbox
All the AI news that matters to media*
Google traffic to UK publishers is collapsing
Organic Google Search referrals to eight major UK publishing groups dropped 7.1% between the first and second quarters of 2025, according to Press Gazette. Hold that pace and Google’s pageview volume to these publishers is on track to halve by the third quarter of 2027. The analysis, built by DJB Strategies from 10.8 billion pageviews and an Ipsos survey of consumer AI habits, lands in an awkward quarter for the idea that search and AI can coexist peacefully: Google reported Search revenue grew 12% in the same period, powered partly by an expanding AI Overviews footprint pushing toward zero-click answers. The study’s authors don’t expect the trend to slow down. (AI-assisted)
Beehiiv wants to be your whole publishing stack
Beehiiv used its Summer Release Event on July 16 to announce it’s no longer just a newsletter tool, unveiling four new products in one sitting: Community, a built-in discussion board; Copilot, an AI agent for audience and business operations; programmatic advertising; and a rebuilt visual editor. Together they mark the platform’s broadest push yet to become an all-in-one publishing system instead of just a place to send emails. The move tracks a broader pattern in digital publishing, where newsletter platforms increasingly compete to own more of a publisher’s daily workflow, from audience engagement and monetization to AI-assisted operations, all inside one login. For independent publishers already juggling five vendors, that consolidation pitch is getting harder to ignore. (AI-assisted)
2026 is on pace to be journalism’s worst year
Newsroom layoffs in the U.S. and U.K. topped 2,300 jobs in the first half of 2026 alone, according to Press Gazette’s running tally, a pace that would mark a 34% jump over all of 2025’s losses if it continues. Public broadcasters and wire services accounted for the majority of cuts, alongside newspapers, magazines and digital publishers. Press Gazette counted at least 3,434 journalism jobs lost in 2025 and 3,875 in 2024, so 2026 is on track to be the worst year yet, even before the second half is counted. Publishers increasingly frame the cuts as reorganizations rather than cost-cutting, but the underlying pressure is consistent: audiences are migrating to video, social platforms and AI-powered search and chat, and newsroom budgets are following them there. (AI-assisted)
Meta’s crawler math is brutal for publishers
Meta’s crawlers hit publisher websites 9.1 billion times in the second quarter of 2026 and sent almost nobody back in return, according to DataDome’s Q2 2026 AI Traffic Report. Meta accounted for most of the growth in a network that logged 17.7 billion AI agent requests overall between April and June, up 45% from the first quarter. The company runs two crawlers with different jobs: Meta-ExternalAgent scrapes sites to train its models without sending traffic or payment back, while Meta-WebIndexer indexes pages so Meta AI can answer real-time questions, more like a conventional search crawler. ExternalAgent still led for the quarter, at 5.3 billion requests to WebIndexer’s 3.75 billion, but WebIndexer’s monthly volume passed ExternalAgent’s in June for the first time, a sign Meta is investing as heavily in answering questions as it is in training on the answers. (AI-assisted)
Why ‘liquid content’ should worry every newsroom
What happens when AI doesn’t summarize the news, but reformats it entirely around what one reader wants? Nic Newman, a senior research associate at the Reuters Institute, showed a room at London’s Media Voices Publisher Summit exactly that on July 10, according to Press Gazette. Asked about the recent U.K. heatwaves, Anthropic’s Claude answered in Q&A format, built filterable graphs comparing air conditioning uptake by country, and pulled in Met Office warnings, all from the same information base news outlets already use. Newman calls this “liquid content,” AI’s ability to serve any shape of answer a reader wants, and says it’s a bigger threat to news media than any single chatbot summary. He opened with a deliberately provocative slide claiming AI can do what journalists do, quicker and better. The point wasn’t that it’s true. It’s that publishers need to plan as if audiences believe it. (AI-assisted)
Fiction Feast drops writers, gains an AI byline
Freelance writers for Fiction Feast, Bauer Media’s monthly short story insert for Take a Break, were told their services are no longer needed. In their place: stories that appear to be produced with AI tools rather than commissioned from human authors, Press Gazette first reported. Bauer Media has not responded to requests for comment on what happened to the freelancers or how the new stories are produced. Press Gazette’s Dominic Ponsford floated a financial motive: costs are tight, and cutting freelance commissions may be a way to keep the title running while keeping remaining named human writers employed elsewhere at the company. Ponsford frames that as a theory, not a confirmed fact, since Bauer hasn’t explained itself. For the freelancers who built Fiction Feast’s back catalog, the silence is its own kind of answer. (AI-assisted)
Germany puts AI Overviews under media law
Germany has become the first country to formally classify AI search features like Google’s AI Overviews and Perplexity’s answers as media products, meaning they now fall under national media law, according to Reuters. The case was made by Germany’s Commission for Licensing and Supervision, which represents the country’s 14 state media authorities: AI summaries and chatbot answers don’t just aggregate existing information, the commission argues, they generate new output shaped by the provider’s own system, exactly the kind of activity media law is built to govern. The ruling follows a separate case out of Munich in June, where a court found Google could be held directly liable for inaccurate information served up by AI Overviews. Together, the two moves give German publishers a legal foothold that publishers elsewhere still don’t have. (AI-assisted)
AP backs a new standard for AI content tracking
Blocking an AI crawler tells you nothing about what happens after a crawler you didn’t block gets through, which is the gap the Standards for Publisher Usage Rights coalition was built to close. The Associated Press just joined as the group’s first U.S. founding member, bringing one of the world’s largest news licensing organizations into a publisher-led push for usage-based licensing instead of opaque scraping. Founded in March 2026, the coalition already counts the BBC, the Financial Times, The Guardian, Sky, The Times of London and the European group MediaHaus among its founders, and now lists 30 publisher members and six affiliates with AP’s addition. The pitch is straightforward: publishers don’t just need an off switch for AI crawlers, they need a meter that shows what happens after access is granted. (AI-assisted)
A new AI tool tries to keep readers on-site
One in 10 people worldwide now use AI tools like ChatGPT, Perplexity or Google’s AI Overviews for news at least weekly, per the Reuters Institute, and Arc XP wants publishers to capture that behavior on their own turf. The Washington Post’s media operating system launched Ask The News on July 13, a conversational AI search tool publishers can embed directly on their sites. It answers using the publisher’s own reporting, with attribution and editorial safeguards built in, and is designed to decline a question outright rather than hallucinate an answer when the source reporting doesn’t support one. The tool’s Subscription Gateway also reworks the paywall itself, limiting AI-generated answers instead of articles, so readers hit a subscribe prompt only after the tool has already shown them some value. (AI-assisted)
*AI-assisted news items are drafted with AI and then carefully edited by Media Copilot editors.