Publishers want to charge bots. First, they have to find them

A new report from TollBit shows how bot traffic is growing—and why it’s getting harder to track.

There’s kind of an issue with all these companies who are building systems that are supposed to monetize traffic from AI bots. Well, two: The idea presupposes that:

1. Tracking bots is straightforward, and

2. A website owner can reliably tell them what to do

A new report from TollBit suggests neither of those are safe assumptions, with user agents clouding what counts as a person, and some AI services applying that ambiguity to scrape at an industrial scale.

More on what all that means for publishers in today’s column. But the question of how you deal with bots—or use them yourself—ties directly into your organization’s AI policy, which is exactly what I’m going to be talking about this Thursday, Sept. 25 at 1 p.m. ET on a LinkedIn Live with the Public Relations Society of America (PRSA). My colleague Melissa Flynn and I are going to unpack what makes a good AI policy and answer the key question: What do you do when it fails? Hope you can join.

Finally, if you haven’t signed up for my monthly class on AI for beginners, you’ve still got time. The next AI Quick Start is happening this Friday, Sept. 25 at 1 p.m. ET. The class is continually updated with guidance on the latest tools—like how journalists and PR professionals can get more done with user agents like Perplexity Comet and ChatGPT Agent. Scroll down for how to register 👇

---

Learn Practical Ways of Using AI in Journalism and PR

AI isn’t a fad. For most journalists and PR professionals now, it’s simply the reality. Yet many reporters and executives are still stuck dabbling: using ChatGPT for story ideas or a few headlines and calling it a day.

It’s time to upgrade.

Coming up, The Media Copilot is offering a one-hour power session on how to actually use AI for real media work. Research smarter. Pitch sharper. Write faster—without losing your voice to the algorithm.

Here’s what you’ll walk away with:

• The real state of AI in media—no hype, just how it’s changing the game.

• A simple way to approach prompting to get what you actually want.

• Pro-level use cases: Find better sources. Focus your stories and campaigns. Protect your data.

📅 Live: Sept. 21, 1 p.m. ET

💸 $49: The cheapest upskill you’ll make all year.

🎁 Extras: Slides, prompt templates, and a quickstart AI guide to keep you sharp.

👉 Don’t just survive AI. Use it to do better work.

---

Robots are mixing in with the humans visiting your site. Are you sure you know which is which? (Credit: Midjourney)

The AI bot conundrum: how to reliably get them to pay

We all know AI is eating the internet, with bots harvesting content and not offering anything in return. This is, of course, the fuel behind the mounting legal battles between media organizations and major AI labs. But in the meantime, the big question remains: What can be done about these bots?

Blocking them is one approach, but how effective is that, really? Which kinds of content are most vulnerable to being scraped and replaced by AI-generated answers? And is it even possible to make these bots pay?

A helpful starting point is the latest State of the Bots report from AI startup TollBit. For publishers under pressure, it offers some hard numbers on how pervasive AI has become in the media ecosystem and how fast it's expanding. While the proliferation of AI bots is certainly alarming for those in the content business, it may also represent an opportunity.

Bots, masked and unmasked

In hopes of maximizing that opportunity, TollBit isn’t just presenting charts and graphs with its latest report. It’s also taking a position. The company argues that AI bots crawling the web should at minimum identify themselves to the sites they visit and scrape. The company is openly calling for regulation to make that happen, something CEO Toshit Panagrahi told me back in June after a previous State of the Bots report revealed that bots from Perplexity, Meta, and Google were ignoring the Robots Exclusion Protocol, the standard used by websites to control bot access.

How AI scrapers are breaking the internet’s honor code, with TollBit’s Toshit Panigrahi

There's nuance here. I won’t rehash the entire backstory, but in short: some AI bots act on behalf of users (as opposed to bots for training or indexing), and those are a bot subspecies generally called user agents. That affords them a certain status, at least according to AI companies: Because they function as human stand-ins, they should therefore be treated like human visitors, not bots. So they don’t identify themselves as bots.

As bots that behave like humans and identify as humans grow, future analytics might show a rebound in human visitors masking an expanding bot audience (Credit: TollBit)

This setup creates a major problem since it becomes extremely difficult to distinguish between actual human traffic (i.e. real people clicking and viewing pages) and robotic activity. That murkiness compromises reliable analytics. Because of this ambiguity, TollBit predicts that the amount of “human” traffic to websites will soon bounce back as user agents become more widespread, but only because tracking tools won’t be able to tell the difference.