Attribution vs Incrementality in Agentic Commerce
Attribution counts orders an assistant touched. Incrementality measures the orders that would not have happened without it. Why the difference decides whether your AI investment is real.
Attribution counts every order an AI-powered shopping assistant touched; incrementality counts only the orders that would not have happened without it. Attribution flatters the vendor, because it credits sales the shopper would have made anyway, while incrementality — measured against a randomized holdout — isolates the revenue the assistant actually caused. If you are deciding whether an assistant earns its budget, incrementality is the only number that answers the question.
Two words that sound alike and mean the opposite
Attribution and incrementality get used interchangeably in vendor decks. They are not the same thing, and the gap between them is where budgets quietly leak.
Attribution answers: how many orders did the assistant touch? A shopper opens the chat, asks a question, checks out later — attribution credits that order to the assistant. The rule can be strict (bought in the same session) or loose (interacted anywhere in a multi-day journey). Either way, it is a counting exercise.
Incrementality answers a harder question: how many of those orders would not have happened otherwise? Some shoppers who talk to the assistant were always going to buy; they would have checked out with or without a conversation. Attribution counts them. Incrementality does not.
Here is the same distinction side by side:
- What it counts — Attribution: every order the assistant touched. Incrementality: only orders that would not have happened without it.
- How it is computed — Attribution: a join between sessions and orders. Incrementality: a comparison against a randomized holdout.
- Which way it leans — Attribution: upward, because it absorbs demand that already existed. Incrementality: neutral, because it removes what would have happened anyway.
- What it proves — Attribution: activity, and a fair billing base. Incrementality: causation, and real ROI.
- Can it flatter you — Attribution: almost always. Incrementality: never — it stays honest even when the answer is uncomfortable.
Why attribution flatters everyone
Attribution is popular because it is easy and generous. It needs no experiment and no control group — just a join between sessions and orders — and it sweeps up demand that already existed.
Picture a bestseller. Shoppers arrive intending to buy it, and many open the assistant on the way — to check sizing, ask about shipping, confirm a detail. Attribution credits the assistant with every one of those orders. But the assistant did not create the demand; it stood next to it. Remove the assistant and most of those orders still land.
That is why an attribution dashboard almost always shows a large, reassuring number — one that should make a CFO suspicious rather than satisfied. A metric that only rises and can never be wrong is not measuring impact; it is measuring traffic. Attributed revenue still has a legitimate use as a billing base, a fair way to size activity and price a service. But a billing base is not proof of lift, and confusing the two is the central sleight of hand in agentic-commerce ROI claims.
The ladder: tracked, attributed, assisted, incremental
It helps to see these as rungs on a ladder, each a stronger or weaker claim than the last.
- Tracked. You recorded that an interaction happened and an order followed. Raw logging — necessary, but it claims nothing.
- Attributed. You assign credit for the order to the assistant under a rule you chose. The number depends entirely on the rule.
- Assisted. You loosen the rule further — the assistant "helped" somewhere in the journey. The window widens, the number grows, the claim softens.
- Incremental. You subtract what would have happened anyway. The number shrinks, and for the first time it is honest.
Each step from tracked to assisted makes the headline bigger and the evidence weaker; only incrementality reverses the direction and answers the question a merchant is actually paying to answer.
How a holdout isolates the truth
The clean way to measure incrementality is an experiment. Split traffic at random: most shoppers see the AI-powered shopping assistant, a held-out slice never does. Because assignment is random, the two groups are statistically identical in intent, source, and seasonality, so any durable difference in conversion, order value, or revenue per visitor between them is the assistant's incremental effect — not a story, a measurement.
This is the difference between "shoppers who used the assistant converted better" — which may just mean motivated buyers use the assistant — and "the group with access converted better than the group without," which is causation. The holdout removes selection bias, because the control group is chosen by a coin flip, not by shopper behaviour.
A trustworthy holdout has two properties: it is randomized, so the groups are comparable, and it is persistent and always-on, so you measure a steady-state effect across a full seasonal cycle rather than a one-week launch spike.
Why merchants and CFOs should insist on it
A finance team does not fund activity; it funds outcomes. "The assistant touched a lot of orders" is an activity statement. "The assistant caused revenue that otherwise would not exist" is an outcome statement — and only the second survives a budget review. A partner willing to be measured against a holdout is confident the lift is real; one who offers only an attribution dashboard is asking you to take generosity on faith.
This is where visibility and revenue part ways. GEO work can make your catalogue eligible to surface in AI answers — eligibility and readiness, not guaranteed placement, and never control over what a third-party model says. GEO gets you seen. Querytail helps you sell. Being seen is worth nothing if you cannot prove it turned into buying, and proving that needs a holdout, not a mention count. See how the assistant fits the wider stack on the product page.
Querytail's native holdout
Querytail runs the holdout as a native, always-on part of the platform, not a bolt-on you configure and defend. A randomized share of traffic is held back automatically, and the console reports incremental conversion, order value, and revenue per visitor against that control — measured on your own data, on a methodology you can inspect. Attributed revenue is still reported and can serve as a billing base, but it is never presented as proof. Proof is the holdout.
The market backdrop makes this urgent. As assistants like Amazon's Rufus — renamed Alexa for Shopping in May 2026 — normalise shopping through conversation, every merchant will be handed an attribution dashboard by someone. The ones who ask "where is the holdout?" are the ones who will actually know what their assistant is worth.
FAQ
So is attributed revenue useless?
No. It is a reasonable way to bill and to size activity, but it is not evidence of incremental lift. Use it for pricing conversations, and use incrementality for value decisions.
Doesn't a holdout mean hiding the assistant from paying customers?
It means a small, randomized slice of traffic does not see it, so you can measure the rest with confidence — a tiny cost next to scaling an investment you cannot prove.
Want to see the native holdout report incremental revenue on your own traffic? Book a walkthrough, or, if you would rather measure it on your own catalogue, join the Design Partner program.