Why We Run a Permanent Holdout on Every Deployment
Every Querytail deployment ships with an always-on control group. Here is why a permanent holdout, not a one-off test, is the only honest way to prove incremental revenue.
Querytail runs a permanent, always-on holdout on every deployment because a one-off A/B test answers the question once and then quietly goes stale. A small, randomly selected group of your shoppers never sees the AI-powered shopping assistant, giving you a live control group that keeps proving incremental revenue as seasonality, catalog, and traffic change. We do not attribute revenue to ourselves. We prove it, with a holdout.
Why a one-off test goes stale
An A/B test is a snapshot. You split traffic, run it for a few weeks, read the result, and ship. That is fine for a button color. It is not fine for a system whose impact moves with your business.
The moment the test ends, the number starts aging. Your spring assortment gives way to autumn. A hero product sells out. Your paid-traffic mix shifts. A competitor changes price. Each of these moves the baseline the assistant is measured against, and a test you ran in March cannot see any of it. By the time your CFO asks whether the assistant is still working, the honest answer from a one-off test is "it was, once."
A permanent holdout never has that problem. Because the control group is always on, the comparison is always current. The lift you report this month is measured against this month's shoppers, this month's catalog, and this month's traffic.
How the permanent holdout works
The mechanism is deliberately simple, because simple is auditable.
A small, randomly selected slice of your visitors is assigned to a control group and never sees the assistant. Everyone else gets the full experience. Assignment is random and sticky: a shopper placed in the holdout stays there for the measurement window, so you are always comparing like with like rather than chasing a moving target.
We then compare the two groups on the metrics that matter, including conversion, average order value, revenue per visitor, and return rate, and the gap between them is the incremental contribution of the assistant. Not the revenue it touched. The revenue that would not have happened without it.
Two things make that number trustworthy:
- It reconciles to your own data. The holdout reads against the merchant's own order data, the same orders that reach your finance system, not a parallel figure only we can see.
- It is grounded in certified product truth. The experience the treated group receives is built only on your approved product data, so the lift you measure comes from an assistant you can stand behind.
You keep checkout. You keep the data. You keep your brand voice. The holdout sits underneath all of it as a measurement instrument, not a rebrand of your storefront.
Assisted revenue is not incremental revenue
Most vendors report assisted revenue: every order where a shopper happened to touch the assistant. It is a big, flattering number, and much of it is not caused by the product at all. Plenty of those shoppers would have bought anyway.
A permanent holdout lets you separate the two and keep them separate over time. Assisted revenue tells you how much activity flowed through the assistant. The holdout tells you how much revenue exists because of it. Watching both, month over month, is how you catch the moment assisted revenue keeps climbing while incremental revenue flattens, the signal that you have started paying for orders you were already going to win.
That discipline is the whole point of our promise: Proven incremental revenue, not attributed revenue.
What a permanent holdout gives you
- A live number, not a stale one — lift measured against current seasonality, catalog, and traffic, every month.
- A vendor you can audit — results that reconcile to your own order data, not a dashboard the vendor alone controls.
- Assisted separated from incremental — so you always know which part of the revenue is genuinely caused by the assistant.
- Protection against drift — price changes, seasonal swings, and traffic-mix shifts are absorbed by the control group instead of quietly inflating your result.
- A CFO-ready answer — "is it still working?" has a current, defensible answer at any time, not only in the weeks after launch.
Why this matters now
AI-powered shopping assistants are becoming a normal part of how people buy, on merchant sites and on large platforms alike. Amazon's shopping assistant, Rufus, was renamed Alexa for Shopping in May 2026. As the category fills up, "our AI drove revenue" will be claimed everywhere. A permanent holdout is how you tell a real number from a marketing one, including from us. You can see how measurement fits the rest of the platform on the product page.
FAQ
Does the holdout mean some shoppers get a worse experience?
No. The control group sees your standard site, the same experience every shopper had before you deployed. Nothing is degraded; a small group simply does not receive the assistant, so there is always a clean baseline to measure against.
Can we change the size of the holdout, or turn it off?
Yes. The control group is configurable, and you can adjust it as your confidence grows. We recommend keeping a permanent holdout precisely so the proof never goes stale, but the choice, and the data, stay yours.
Isn't a permanent holdout just an A/B test?
It is the same statistical idea, kept always on. An A/B test has a start and an end, so its answer expires the day it stops. A permanent holdout keeps the result current for as long as the assistant is live.
Want to see the holdout running on your own catalog? Book a walkthrough, or work with us directly through the Design Partner program.