How to Run an AI Sales Agent Proof of Concept on Shopify: Success Criteria, Holdouts and Mobile Rules

The short answer. A proof of concept (POC) is a time-limited test that answers one written question before you commit. Write the question, the primary measure and the decision rule before launch. Randomly hold back part of your traffic so you have a comparison group (in effect an A/B test of the agent on versus off), and judge the result on all sessions you assigned, not just the shoppers who chatted. Check the experience on real phones, size the test so it can detect the change you care about, and read the conversations as well as the numbers. A POC that cannot end in "stop" is a demo.

Disclosure: Rep AI sells an AI sales agent for Shopify. The method in this playbook applies to any AI sales agent or shopping assistant. Two Rep AI customer examples appear near the end, labeled and linked, to show how to read a vendor-published number. The customer results come from Rep AI's own published case studies. They are not results of any test described here.

Last reviewed 2026-10-16. This guide describes a method. It does not promise any result, and each customer figure is one store's result as Rep AI's case study reports it.

What a POC is, and what it is not

A trial lets you try a product. A proof of concept answers a business question: should we roll this out on our store? A rollout is what you do after the answer is yes. In a POC you agree in writing, before the test starts, what you are asking, how you will measure it, who decides and what counts as success. That protects you from a test that ends with numbers and no decision.

Rep AI offers a 14-day free trial with no credit card required. A trial is useful for setup and for checking the agent's answers. A revenue question needs a comparison group and enough traffic, and may need longer, so plan the POC window separately.

Step 1: Write the question and the decision

Write one question and the decision that follows each answer.

• Question: "For eligible sessions on our product pages, does showing the sales agent raise orders per session compared with not showing it?"
• If yes by the agreed margin: roll out to the pages tested, then extend.
• If no difference: change the setup and retest, or stop.
• If worse on any guardrail: pause and investigate.

Keep sales and support in separate POCs: they need different measures and often different owners. This guide focuses on the sales agent and notes where a support test differs.

Step 2: Set success criteria before launch

Choose the measures and thresholds with finance and the store owner. Write them down and date the document. A threshold set after you have seen the results is not a criterion.

Three rules for the table:

1. One primary measure. If you have five, you can find one that looks good.
2. A margin, not a direction. "Higher" is not a threshold. Agree with finance the smallest difference that would justify the cost and effort.
3. Define terms once. What is an eligible session, what is an attributed order and how long is the attribution window. For support, define "resolved": for example no human touch and no reopen within a window you choose, such as 7 days. Use identical definitions for both groups.

For turning results into money, use the method in How to Calculate the ROI of an AI Sales Agent on Shopify: Attribution, Costs and Honest Math. This guide stops at whether the difference is real.

Step 3: Design the holdout

A holdout is a group of shoppers who do not see the AI agent while the rest do. It is the clearest way to separate the agent's effect from seasonality, promotions, ad campaigns and traffic mix, because both groups live through the same weeks. To set it up:

• Assign by visitor, not by page view. A shopper who sees the agent on one visit and not the next contaminates both groups. Ask your vendor whether it can split visitors, or use a theme-level toggle or an A/B testing tool, and check what keeps each assignment fixed. Stored identifiers can expire, so a test that runs for months can lose its groups.
• Randomize. Let a tool or a random function assign visitors, not country, device or traffic source.
• Choose the split. An even split gives a clear reading sooner. A smaller holdout is possible if you worry about lost sales, but it takes longer.
• Define eligibility. Decide which pages and sessions can show the agent, and apply the same rule to the holdout. Keep promotions, pages and widgets identical, so the agent is the one difference.
• Check the holdout stays clean. Test in a clean browser that the control group sees no launcher and no proactive message.
• Exclude internal and bot traffic from both groups, then check balance before you read results: compare device, new versus returning and traffic source. A large gap suggests the assignment is not random.

Analyze everyone you assigned. Count all sessions assigned to each group, whether or not the shopper chatted. Comparing shoppers who chatted with shoppers who did not is misleading, because the shoppers who start a conversation may differ from those who do not, in ways unrelated to the agent.

If you cannot randomize

For a support POC, you can route a random share of eligible contacts through the agent and the rest through your current process. If your vendor supports it, a shadow review is another option: the agent drafts answers, your team scores them, and no shopper sees them yet.

Step 4: Size the test and set the duration

Small effects on rare events need a lot of traffic. A rough planning formula for sessions needed per group is:

n ≈ 16 × p × (1 − p) ÷ d²

where p is your baseline conversion rate and d is the smallest absolute difference you want to detect. It assumes 80% power and a 5% two-sided significance level, and it applies only to a conversion rate (an order or no order per session). It does not apply to revenue per eligible session or average order value, which vary in size and need an analyst. An analyst can refine it for your case.

Invented example, for method only. A store has 40,000 eligible sessions a month and a baseline conversion rate of 2%. To detect a 10% relative change (2.0% to 2.2%, so d = 0.002), each group needs about 16 × 0.02 × 0.98 ÷ 0.002², or roughly 78,000 sessions. That is about 156,000 in total, close to four months at 40,000 sessions a month. To detect a 20% relative change (d = 0.004), each group needs about 19,600 sessions, close to one month.

What follows from this:

• Decide the smallest change that matters, and size the test to detect it.
• If traffic is limited, choose a measure that happens more often, such as add-to-cart rate, test for a larger effect, or run longer.
• Run whole weeks, including weekends, and cover any promotion in both groups.
• Set the end date in advance and do not stop early because one group looks ahead.

Step 5: Run the mobile checks

Your own analytics show the share of shoppers on phones. The small-screen experience decides whether the test measures the agent or a layout problem, so check real devices before launch, not just a desktop emulator.

• Launcher placement. Does it cover the add-to-cart button, a sticky bar, a size chart, a cookie banner or another app's widget?
• The open chat and the keyboard. Can the shopper close the chat with one tap? Does the input stay visible when the keyboard opens, and does the screen jump?
• Proactive messages. Do they cover the product image, the price or the buy button, and can a shopper dismiss them?
• Checkout. Check whether chat can run in checkout on your Shopify plan, and decide deliberately whether it should.
• Browsers and speed. Test Safari on iPhone, Chrome on Android and the in-app browsers that social platforms open. Compare page load with and without the widget on a mid-range phone and a slow connection, and treat page speed as a guardrail.
• Accessibility basics. Tap targets large enough, readable contrast, and a working keyboard and screen-reader path through the chat.

Split results by device in advance and check that both groups have a similar device mix.

Step 6: Run the test

• Freeze the setup. Do not change prompts, policies, triggers or pages mid-test. If you must, log the date and the change, along with promotions, outages and site changes.
• Read conversations each week. Sample transcripts and score them for accuracy and tone. Note wrong answers and questions your pages should have answered.
• Agree stop rules. For example, pause if the agent gives a wrong policy answer that affects a shopper, and write down who decides.

Step 7: Read the result and decide

1. Apply the rule you wrote. Compare the primary measure against the agreed margin.
2. Respect uncertainty. A difference inside normal variation is not a win or a loss. Ask your analyst whether the gap is distinguishable from chance for your sample.
3. Check the guardrails. A gain with worse returns or complaints may not be a gain.
4. Read the segments you named in advance, not one that happens to look good.
5. Read the transcripts again: they explain the movement.
6. Write a decision memo: the question, the setup, the result, the guardrails, the decision and who approved it.

Two published examples, and how to read them

Vendors publish customer numbers, and Rep AI does too. Treat them as prompts for what to measure, not as a forecast. Both examples are one store's results, as Rep AI's own case study on that store reports them.

Name the denominator. HigherDOSE sells infrared saunas and recovery tools. Rep AI's case study on HigherDOSE, on a page dated June 17, 2026, reports that in the most recent 30-day period (dates not given), 45% of website visitors engaged with the chat experience, 2.5% of those interactions triggered a sales action, and 6% of those actions converted into purchases. Each figure has a different denominator: the 6% is a share of actions, not of visitors. The case study states the starting problem as no visibility into how chat influenced revenue. In your plan, name the denominator of every figure before launch, as in Step 2.

Read the conversations. Proof Wallets sells minimalist wallets. Rep AI's case study on Proof Wallets describes early support overload and a view that hiring did not fit how the team wanted to operate. Dana Peters, Co-founder at Proof Wallets, says in it: "I'm always wary of software metrics because they're usually inflated. So I went through the conversations myself—and I concluded that the numbers were real". The same case study reports that "12.7% of Proof's total store revenue is powered by Rep AI." The page states no time window. That is the case study's own attribution of revenue. It does not tell you what a holdout on your store would show; Steps 3 and 4 do.

A one-page POC plan

FAQ

How do I A/B test an AI chatbot on Shopify? Randomly assign visitors to see the agent or not, keep the assignment fixed across visits, keep everything else identical, and compare the same measure for all assigned sessions in both groups. Do not compare shoppers who chatted with shoppers who did not.

How long should an AI agent pilot run? Long enough to reach the sample size your primary measure needs, in whole weeks, covering a normal buying cycle. Work it out as in Step 4.

What success criteria should I set for an AI agent proof of concept? One primary measure with an agreed margin, a few secondary measures, guardrails that must not get worse, and a transcript review for quality. Set them before launch and write them down.

How do I design a holdout test for an AI agent? Assign by visitor, randomly and persistently, define eligible sessions the same way for both groups, check that the control sees no agent, and analyze everyone assigned.

Can I run the POC during a free trial? A free trial suits setup and reviewing the agent's answers. A revenue question may need more traffic and time than a trial gives, so plan the test window against your sample size.

Where can I see the agent before setting up a test? You can try Rep AI's simulator at /simulator.