How to Actually Prove Agentic-Commerce ROI
Attribution dashboards claim revenue that would have happened anyway. Learn how a native holdout proves the incremental ROI of an AI-powered shopping assistant, not just attributed revenue.
Most agentic-commerce ROI figures are inflated. Attribution dashboards claim orders that would have happened anyway, so the revenue they report looks impressive and proves almost nothing. The honest way to prove ROI is a native, always-on holdout: a randomized group of shoppers who never meet the assistant, measured against those who do, so you can see the incremental add-to-cart, conversion, and average order value the assistant actually creates.
That is the difference between counting and proving. Here is how to tell them apart, and a practical framework you can apply to any AI-powered shopping assistant, including ours.
The four rungs of the revenue ladder
Every claim about AI-driven revenue sits on one of four rungs. Moving up the ladder means moving from activity toward causation.
- Tracked. The assistant did something measurable: a shopper opened it, asked a question, saw a recommendation. This is engagement, not revenue.
- Attributed. A rule assigns an order to the assistant because a touch happened in the same session or journey. This is where most dashboards stop, and where most overcounting begins.
- Assisted. A tighter version of attribution: the shopper meaningfully interacted with the assistant before buying. Useful for billing and reporting, still not proof.
- Incremental. The order happened because of the assistant and would not have happened otherwise. This is the only rung that answers your CFO's real question.
Attribution and assisted revenue live on rungs two and three. They tell you what the assistant touched. They cannot tell you what it changed.
Why attribution overcounts
Consider a shopper who already has a product in their cart, opens the assistant to confirm a shipping detail, then checks out. Attribution records a win. But that order was going to happen. The assistant was present, not decisive.
Multiply that across a catalog and you get a number that is technically true and directionally useless. Attribution rewards the assistant for being in the room. It cannot subtract the orders you would have won anyway, because it has nothing to subtract them against.
That "against" is the whole game. Without a control group there is no counterfactual, and without a counterfactual there is no proof.
Assisted Net Revenue is a billing base, not evidence
We report Assisted Net Revenue (ANR): the net revenue from orders where a shopper genuinely engaged the assistant before purchasing, returns and cancellations removed. It is a clean, auditable base for pricing and reporting, calculated the same way every period.
But ANR is a billing base, not a proof of incrementality. It sits on the "assisted" rung. It describes association, not causation. Any vendor who hands you an assisted-revenue figure and calls it your ROI is asking you to trust a correlation.
We do not attribute revenue to ourselves. We prove it, with a holdout.
The holdout: how proof actually works
A holdout is a randomized share of your shoppers who never see the assistant. They browse the same catalog, the same prices, the same site. The only difference is that the AI-powered shopping assistant is switched off for them.
Because assignment is random and persistent, the two groups are statistically identical in everything except exposure. So when the exposed group adds to cart more often, converts at a higher rate, or spends more per order, the gap is causal. That gap is your incremental lift, the number worth putting in a board deck.
A credible holdout has four properties:
- Native. Built into the deployment, running on your own traffic and your own orders, not modeled after the fact.
- Always-on. Permanent, not a one-week test. Seasonality, promotions, and traffic mix shift constantly; a standing control group absorbs that noise.
- Randomized and persistent. A shopper assigned to the holdout stays there, so there is no self-selection and no leakage.
- Measured on outcomes that matter. Incremental add-to-cart, conversion rate, and AOV, compared group against group, not session against session.
The merchant does not have to trust the vendor. You read your own data, from your own store, with a method you can audit.
A practical framework you can apply
You do not need a data-science team to hold a vendor to this standard:
- Define the four rungs out loud. Ask the vendor to label every number they show you as tracked, attributed, assisted, or incremental. Watch how quickly the impressive figures move down the ladder.
- Separate the billing base from the proof. Accept an assisted metric like ANR for what it is, a way to price the service, and insist on a holdout for the ROI question.
- Demand a native, always-on control group. If the assistant cannot be withheld from a randomized share of live traffic, the vendor cannot prove incrementality, only assert it.
- Pick outcomes before launch. Agree on incremental add-to-cart, conversion, and AOV as the scorecard. Decide the measurement window up front so nobody cherry-picks a good week.
- Let it run through a full cycle. Give the holdout enough time to smooth out promotions and seasonality before you read the verdict.
This framework is deliberately number-free. The point is not to promise a lift figure but to ensure that whatever figure you see was earned against a control group, not assembled from touches.
Where this fits with the rest of the stack
Proof depends on the foundations underneath it. Our Certified Catalog keeps the assistant grounded in your own certified product data, so the recommendations behind your lift rest on a verified source rather than guesswork. The AI-Powered Shopping Assistant is the surface shoppers actually engage. GEO-Ready Discovery prepares your catalog to be eligible and readable when AI answer engines look for products. GEO is about eligibility and readiness, not guaranteed placement; no one, us included, controls how a third-party engine ranks or mentions you.
That distinction matters right now. As shoppers increasingly ask assistants like Amazon's Rufus (renamed Alexa for Shopping in May 2026) what to buy, the temptation to claim credit for every AI-adjacent order will only grow. Readiness gets you into those conversations; the holdout is what proves they paid off. See how these pieces connect on our product overview.
Frequently asked questions
Does a holdout mean losing sales to the control group?
A small, randomized share of traffic runs without the assistant. That is the cost of knowing the truth, and the slice can be sized to keep the trade-off modest. Measuring real incrementality once beats guessing about it forever.
Isn't assisted revenue good enough for reporting?
For reporting and billing, yes. Assisted Net Revenue is a clean, consistent base. Just do not confuse it with proof. The moment someone asks "would these orders have happened anyway?", only the holdout can answer.
What if a vendor cannot offer a holdout?
Then they can show you attribution, but not incrementality. You are free to buy on attribution; just know which rung of the ladder you are standing on, and price the risk accordingly.
Want to see honest measurement in practice? Book a walkthrough and we will show you the holdout, the ladder, and how ANR is calculated. If you would rather prove it on your own traffic, the Design Partner program builds a holdout into your deployment from day one.