Today your site will have refused a few thousand machine requests. Refusals don’t tend to get looked at, which is fair enough, because nothing actually happened. But some of those requests were worth money, and there’s no way of telling which.
Three kinds of machine visit
Machines don’t all want the same thing.
A training crawl takes your content to build or improve a model. It happens at scale, and it’s permanent - once your archive is inside a trained model, there’s no mechanism to get it back out again.
A retrieval fetch happens when someone asks a question and a system goes off looking for a current answer. It can run thousands of times a day - roughly ten times the server requests of a training crawler - and you’re paying for that traffic whether or not anyone is paying you.
An agentic visit is a machine acting for a named person, mid-task. Booking something, comparing something, doing the reading on their behalf. It’s the most interesting of the three commercially, and probably the least well understood.
Most publishers see one undifferentiated lump of bot traffic and can’t separate any of it out. Which is a problem, because the right answer is different for each of the three, and that’s where it starts to cost money.
Refuse, give away, or sell
When a machine asks you for something, there are only really three things you can do. You refuse it, you hand it over for nothing, or you sell it.
Almost every publisher setup can manage the first two. Hardly any can do the third, which is probably why the conversation keeps collapsing back into blocking. If selling isn’t on the table, the only decision left is where to draw the line between refusing and giving away, and that’s a pretty miserable call to have to keep making every time the technology shifts.
It also explains why licensing deals look the way they do. If you can’t transact at the moment of the request, the only thing left to sell is a bulk arrangement, agreed in advance and priced without much sight of what’s being taken. PPA’s research puts the average of those at under 2.15% of a publisher’s annual revenue, across 73 publicly known deals.
Under 2.15% of annual revenue - the average multi-year AI licensing deal, across 73 publicly known deals. Source: PPA.
What selling access needs
Four things need to be in place.
1. A product that comes apart. If the only thing you sell is a subscription, that’s the only thing anyone can buy. But describe it as components instead - an archive, a live feed, a calculated metric, a set of profiles, a video library, one specific field — and each of those becomes something that can carry its own terms and its own price. Publishers generally have a lot more saleable surface than they can currently sell.
2. Rules that tell askers apart. Anonymous visitor, logged-in reader, paying subscriber, search crawler, training crawler, agent acting for a named person. That’s six different commercial relationships, and most systems treat them as two: allowed, or not. Being able to give different things to different askers is really the whole game, and it depends on being able to tell the traffic apart in the first place.
3. The ability to change those rules without a release. This one matters more than it sounds like it should. If adjusting what a given type of user can see needs an engineering ticket and a deploy window, then in practice your access policy gets set once a quarter by whoever happens to have capacity. Commercial teams don’t experiment with pricing under those conditions, or with much else.
4. A way to take payment at the moment of the request. When something arrives asking for what it isn’t entitled to, you could just answer with a price. The mechanics already exist and they’re fairly unglamorous: the request comes in, the answer is payment required, a payment link gets generated, the person the agent is working for approves it the way they’d approve anything else they buy online, and the request completes. It’s what Courtney demoed live on the morning, and James reckons it may have been the first agentic transaction of its kind made in the UK, because the payment rails for it are only just arriving here.
A lot of the talk about machine payments has attached itself to stablecoins and wallets and protocols that no finance director is signing off on any time soon. None of that is necessary, though. A human approving a payment they can see, on rails they already use, will get through an internal review that the other version won’t.
With the third option available, you’re working out what content is worth rather than who to let in.
Why an agent isn’t a crawler
A crawler is just extracting. It has no budget, no principal, nothing to offer you. An agent has been asked by a specific person to do a specific job, usually with money attached and often in a hurry, so it’s a lot closer to a customer at a counter than to something hoovering up an archive.
It’s already showing up in the data, too. Individual subscriber accounts are starting to appear reading thousands of pages a day, which is a person’s agent at work. That points at the practical version of all this: agentic usage layered on top of the human subscription, rather than a separate machine business.
Most businesses are built to serve humans who navigate, which means the market you can actually serve is smaller than the market that’s out there, and the gap gets wider every month. A growing share of real demand now turns up in a form your systems can only turn away.
Five things worth checking
Roughly in order of how quickly you’ll be able to answer them.
- Can you tell training crawls, retrieval fetches and agent visits apart in your own traffic today? If not, what would it take?
- How many separately saleable components does your product contain, as opposed to how many things you currently sell?
- If you wanted to change what a logged-in non-subscriber can see, how long would that take and who would have to do it?
- What happens right now when a request arrives for something the asker isn’t entitled to? Is the answer a refusal, or is it a price?
- And the one that keeps coming up. When somebody in your business finally decides what machines can and can’t have, who is that person - and is it the same person who’s accountable for the revenue it affects?
Talk it through
MonetizationOS works on exactly this: the layer that decides who gets access to what, for humans and machines alike, plus the commercial tooling around it.
If you’d like help answering the five questions against your own setup, talk to Scott Howland - Head of Global Revenue, [email protected]
‍
We originally wrote this for attendees of the PPA’s September event: The Future of Access - an event focused on machine access, which we partnered on. It applies to any media business with a subscription or paywall strategy. Thank you to the PPA for hosting, and to everyone who attended the event.
