Run a sort order A/B test experiment

Sort order experiments let you A/B test a sort order against one or more challengers on the same catalog surface. Layers assigns each shopper’s browsing session to a single group and keeps it there for the rest of that session. It then measures conversion, add-to-cart, and revenue lift so you can pick a winner based on real traffic instead of preview snapshots. Use an experiment when you want to:

• Compare a new sort order against the live one before promoting it store-wide.
• Test different weighted group tunings or soft boost settings on real shoppers.
• Roll out a change gradually, for example by sending 10% of traffic to a challenger first.
• Compare more than two variants at once with a multi-group split.

​

Prerequisites

• A published sort order to use as the control. The control is the sort order shoppers see today.
• One or more published challenger sort orders. Create them under Sort Orders first. See Create a sort order and Duplicate a sort order.
• The Sort Orders — Edit permission to create or change experiments, and Sort Orders — View to read results.

​

Create an experiment

1. Go to Sorting → Experiments and click Create.
2. Give the experiment a name your team will recognize.
3. Fill in each step:
   • Step 1 — What you’re testing: select the sort order under test (control), the challenger sort order, and which surfaces run the test.
   • Step 2 — Who gets split: leave it open to all eligible shoppers, or narrow the audience with contextual conditions.
   • Step 3 — Traffic split: set the share of traffic each group receives.
   • Step 4 — Schedule: set an optional start and end date.
4. Review the summary on the right, then Save the experiment as a draft.
5. When you’re ready, change the status to Running to start collecting data.

​

What you’re testing

Select the sort order under test. Only requests that already resolve to this sort order enter the experiment. Shoppers who would see a different sort order aren’t affected. Then select where it runs:

• Search — results pages for shopper queries.
• Collections — collection and category browse pages.

You can select one or both. If you select Collections, you can further scope the test to specific collection handles instead of every collection that uses the sort order.

​

Who gets split

By default the experiment splits every eligible shopper who lands on the target sort order. Click Narrow the audience to add contextual conditions, for example only shoppers on mobile, only a specific locale, or only signed-in customers. Layers evaluates conditions per request. Sessions that don’t match see the control sort order and stay out of measurement. Assignment is deterministic and based on the shopper’s browsing session. Layers combines the session identity with the experiment ID to place the session in a group, so the shopper sees the same variant across page loads, searches, and collection pages for the rest of that session. A session typically ends after about 30 minutes of inactivity. A shopper who comes back later starts a new session and can be placed in a different group. If Layers can’t read a session identity for a request, it falls back to the shopper’s device identity. Layers records results against the same session it used for assignment, so what a shopper saw and what they did are counted together.

​

Traffic split

Choose how to divide traffic:

• Two groups: drag the slider to set the control’s share (5–95% in 5% steps). The challenger gets the rest.
• Multiple challengers: add more groups and set an integer weight for each. Weights must sum to 100. Click Even split to distribute traffic equally.

The split bar at the top of the step previews the share each group receives and which sort order it’s mapped to.

​

Schedule

The schedule controls when a Running experiment can serve challenger variants. Leave Start and End blank to keep it open until you stop the experiment manually, or set dates to limit when it runs. Statuses:

• Draft — saved but not serving traffic.
• Running — actively assigning shoppers and collecting data.
• Paused — stopped serving; shoppers go back to the control sort order. Results collected so far are kept.
• Completed — finished. No new assignments; results remain available.
• Archived — hidden from the default list.

​

Read the results

Open an experiment and switch to the Results tab. Compare groups by revenue per session, sessions, conversion rate, click-through rate, purchase rate, click-to-order rate, add-to-cart rate, average order value, and clicks. Click-through rate measures the share of tracked searches with a click; purchase rate measures the share with an order; click-to-order rate measures how often a clicked search leads to an order. Each group is labeled with its sort order, and Layers identifies the control group explicitly. Sessions counts the distinct sessions exposed to each group. Conversion rate and add-to-cart rate are the share of those sessions with at least one attributed order or add to cart. Revenue per session and clicks per session divide each group’s attributed totals by its sessions. Click-through rate, purchase rate, and click-to-order rate are still measured per tracked search or browse request. Layers leaves out sessions that saw more than one group, along with their orders and revenue, so one shopper’s activity isn’t credited to two variants. Experiment tracking is still early-stage. Attribution covers sessions Layers can see, so totals can differ from Shopify or other analytics. Cross-check close results before acting. The metrics table compares each challenger with the control. Layers tests conversion rate, click-through rate, purchase rate, click-to-order rate, and add-to-cart rate for statistical significance. Every group needs at least 100 sessions before Layers reports confidence; until then, the confidence column reads Not tested. Revenue per session and average order value are not significance-tested. Clicks are a count, and clicks per session shows lift only, without a significance test. The cumulative revenue-per-session chart shows how revenue per session changes over the run, not conversion lift. The control line is dashed and muted; challenger lines use distinct colors. Use the legend to show or hide groups. The date selector filters the chart points shown, but does not change the totals or recalculate the cumulative baseline. Hover over a point to see that day’s revenue per session.

​

Explore experiment data with LayersQL

For deeper analysis of experiment events, see the LayersQL documentation.

​

Common tasks

• Pause an experiment: from the list or detail view, switch the status to Paused. Shoppers return to the control sort order immediately; you can resume later without losing data.
• Test for one audience or keep app traffic out: see Test sort orders by audience.
• Duplicate an experiment: use Duplicate on the row actions menu to clone the setup as a new draft. Use it to iterate on a follow-up test with the same audience or split.
• Bulk delete drafts: multi-select rows on the list and select Delete to remove drafts you don’t need.
• Promote a winner: once you’ve picked a variant, apply its configuration to the live sort order in the Sort Order editor and complete the experiment.