Skip to main content
Back to blog

Strategy

Does Your AI Advisor Generate Additional Sales?

Querytail··Updated ·4 min read

Updated October 11, 2026 to reflect Querytail's current offer and measurement approach.

A dashboard can show that an AI advisor touched an order. It cannot, by itself, show that the order would not have happened otherwise. To evaluate return on investment, start with a credible comparison, then compare the measured value with the full cost of the service.

A ladder of evidence

  • Tracked: a customer opened the advisor, asked a question, or saw a recommendation.
  • Attributed: a defined rule links an order to an earlier advisor interaction.
  • Assisted: a meaningful advisor interaction occurred during the purchase journey.
  • Incremental: an experiment estimates the change in outcomes caused by access to the advisor.

The first three measures describe activity and association. The last asks a causal question. None of the first three is, by itself, a return-on-investment figure.

Why a control group matters

Customers who choose to use an advisor may differ from those who do not. They may be closer to buying, have a more complex question, or spend more time on the site. Comparing these two self-selected groups can make an advisor look effective even when it changed little.

A randomized holdout compares visitors assigned access to the advisor with visitors assigned to the standard storefront. Both groups should be defined before analysis, and the comparison should include all eligible visitors assigned to each group.

A valid reading needs more than a randomized setting. Check the actual allocation, control-group exposure, missing events, order matching, sample size, and uncertainty. If one of those checks fails, the responsible conclusion is that the effect has not been established.

A worked test with two possible readings

All figures in this example are invented to explain the method. They are not Querytail results, customer data, forecasts or a claim of improved conversion. The example tests purchase conversion, not ROI.

Suppose 40,000 eligible visitors on the same storefront are assigned before any advisor interaction, 20,000 to the standard experience and 20,000 to an experience with the advisor. Define a purchase within an agreed observation window as the primary outcome before looking at the data. Keep each visitor in their assigned group, reconcile orders, verify that the control group did not see the advisor and check the actual split and missing events. Visitors who never open the advisor still belong in the access group.

  • A positive reading in this fictional test: 800 of 20,000 control visitors purchase (4.0%), versus 960 of 20,000 visitors with access (4.8%). The estimated difference is +0.8 percentage point. A simple approximate 95% interval runs from +0.4 to +1.2 percentage points. If the design and data checks pass and the analysis method was agreed in advance, this supports a positive conversion effect for that cohort and period, not a general revenue or ROI claim.
  • An inconclusive reading of the same fictional setup: 800 of 20,000 control visitors purchase (4.0%), versus 840 of 20,000 visitors with access (4.2%). The point estimate is +0.2 percentage point, but the approximate 95% interval runs from −0.19 to +0.59 percentage point. Because it includes zero, the effect is unresolved. The positive point estimate must not be marketed as a proven lift.

These intervals use a simplified comparison of two proportions and do not replace a predeclared analysis suited to real traffic. If the allocation is invalid, control visitors saw the advisor, or orders cannot be reconciled, neither reading is usable regardless of its apparent size. A conversion effect also needs margins, returns and full costs before it can become an ROI assessment. The ROI worksheet shows that next step.

Build the ROI calculation

Agree on the commercial outcome before reading the data. Revenue per eligible visitor can be useful because it combines purchase frequency and order value. Depending on the merchant's economics, gross margin, returns, and support workload may also matter.

Estimate incremental value from a valid comparison, then subtract all actual costs over the same period, including setup and ongoing operations. Keep the observation period explicit. Do not add an attributed-revenue figure to an experimental estimate: that would count the same activity twice.

The analysis may show a positive, negative, or inconclusive result. A decision can still be made under uncertainty, but the uncertainty must be visible.

Where Querytail fits

Product Engine organizes merchant-controlled product knowledge. Brand Advisor uses that knowledge to guide customers on the merchant's site. A valid comparison with an unexposed group can help assess the effect of that guidance.

An assisted-order report can be useful for understanding journeys, but it cannot show how many purchases the guidance caused. Keep that distinction clear when assessing ROI.

Questions for any vendor

  • What is the primary outcome, and was it chosen before analysis?
  • Who was eligible for the comparison?
  • Was the assignment valid and the control group unexposed?
  • How were orders matched and exclusions handled?
  • What is the uncertainty around the effect?
  • Which costs are included in the ROI calculation?

Can a positive point estimate prove ROI? No. First establish that the experiment is valid, then consider uncertainty and costs.

Can I use attribution if an experiment is not possible? Yes, for operational understanding. Label it as attribution, not causal proof.

For a measurement plan grounded in your store and catalog, request a demo.

For brands and specialty retailers whose customers need advice before they buy

Help your customers choose with confidence.

Share your website and we'll tailor the conversation to your products and the questions your customers ask.

Request a demo