Skip to main content
Sort order experiments are in Beta. The feature is available in the dashboard under Sorting and reuses your existing Sort Orders permissions.
Sort order experiments let you A/B test a sort order against one or more challengers on the same catalog surface. Layers assigns each shopper to a single group and keeps them there for the life of the test. It then measures conversion, add-to-cart, and revenue lift so you can pick a winner based on real traffic instead of preview snapshots. Use an experiment when you want to:
  • Compare a new sort order against the live one before promoting it store-wide.
  • Test different weighted group tunings or soft boost settings on real shoppers.
  • Roll out a change gradually, for example by sending 10% of traffic to a challenger first.
  • Compare more than two variants at once with a multi-group split.

​
Prerequisites

  • A published sort order to use as the control. This is the sort order shoppers see today.
  • One or more published challenger sort orders. Create them under Sort Orders first. See Create a sort order and Duplicate a sort order.
  • The Sort Orders — Edit permission to create or change experiments, and Sort Orders — View to read results.

​
Create an experiment

  1. Go to Sorting → Experiments and click Create.
  2. Give the experiment a name your team will recognize.
  3. Fill in each step:
    • Step 1 — What you’re testing: pick the sort order under test (control), the challenger sort order, and which surfaces run the test.
    • Step 2 — Who gets split: leave it open to all eligible shoppers, or narrow the audience with contextual conditions.
    • Step 3 — Traffic split: choose the share of traffic each group receives.
    • Step 4 — Schedule: set an optional start and end date.
  4. Review the summary on the right, then Save the experiment as a draft.
  5. When you’re ready, change the status to Running to start collecting data.

​
What you’re testing

Pick the sort order under test. Only requests that already resolve to this sort order enter the experiment. Shoppers who would see a different sort order aren’t affected. Then pick where it runs:
  • Search — results pages for shopper queries.
  • Collections — collection and category browse pages.
You can select one or both. If you select Collections, you can further scope the test to specific collection handles instead of every collection that uses the sort order.

​
Who gets split

By default the experiment splits every eligible shopper who lands on the target sort order. Click Narrow the audience to add contextual conditions, for example only shoppers on mobile, only a specific locale, or only signed-in customers. Layers evaluates conditions per request. Sessions that don’t match see the control sort order and stay out of measurement. Assignment is deterministic and sticky. Layers hashes the shopper’s device or session identity with the experiment ID to place them in a group. The same shopper then sees the same variant for the life of the test, including across page loads and return visits on the same device.

​
Traffic split

Choose how to divide traffic:
  • Two groups: drag the slider to set the control’s share (5–95% in 5% steps). The challenger gets the rest.
  • Multiple challengers: add more groups and set an integer weight for each. Weights must sum to 100. Click Even split to distribute traffic equally.
The split bar at the top of the step previews the share each group receives and which sort order it’s mapped to.

​
Schedule

Leave the schedule open-ended to run until you stop the experiment manually, or set a Start and End date. Layers only serves the challenger variants while the experiment status is Running and the current time is inside the schedule window. Statuses:
  • Draft — saved but not serving traffic.
  • Running — actively assigning shoppers and collecting data.
  • Paused — stopped serving; shoppers go back to the control sort order. Results collected so far are kept.
  • Completed — finished. No new assignments; results remain available.
  • Archived — hidden from the default list.

​
Read the results

Open an experiment and switch to the Results tab. Layers reads events straight from your analytics data and shows per-group totals for:
  • Requests
  • Add-to-cart rate
  • Conversion rate
  • Revenue and revenue per session
Each challenger row also shows lift vs. control and a confidence value from a two-proportion z-test with Wilson confidence intervals. A challenger needs at least 100 sessions per group before Layers reports significance. Until then, the confidence column reads Not tested. The Cumulative lift chart plots how each challenger’s conversion lift has trended over the run so you can spot whether an early lead is stabilizing or eroding.
Let the experiment run long enough to clear the 100-session gate on every group. Also cover at least one full business cycle for your store (typically a week). Calling a test on a single-day spike often flips once weekend traffic arrives.

​
Query experiment data with LayersQL

While an experiment runs, Layers stamps every browse and search event with the experiment ID and the assigned group. You can query these fields directly in LayersQL using the experiment_id and experiment_group dimensions on the collections, search-text, search-image, and search-similar datasets. For example, to compare conversion by group for a running experiment:
Replace the experiment_id value with the ID shown in the experiment’s URL.

​
Common tasks

  • Pause an experiment: from the list or detail view, switch the status to Paused. Shoppers return to the control sort order immediately; you can resume later without losing data.
  • Duplicate an experiment: use Duplicate on the row actions menu to clone the setup as a new draft. Handy for iterating on a follow-up test with the same audience or split.
  • Bulk delete drafts: multi-select rows on the list and choose Delete to remove drafts you don’t need.
  • Promote a winner: once you’ve picked a variant, apply its configuration to the live sort order in the Sort Order editor and complete the experiment.