The secret sauce to growing search KPIs: A/B testing

Jon Silvers10 Apr 2026

Problem

E-commerce teams tune search ranking, personalization and merchandising rules by intuition. The settings interact, and traffic is uneven, so a manual check does not show what a change does to the KPIs. A change that raises conversion rate (CVR) can still lower revenue, and a change that moves the clicked position can mean opposite things.

The article offers three test designs. Each has a hypothesis, two variants and a rule for reading the result.

Example from the article: five $100 headphone sales are worth more than 20 $2 sticker sales, although they are fewer conversions.

Solution in brief

Run each ranking question as an A/B test with two variants and read it against a pair of KPIs chosen in advance. The pair makes the result interpretable: revenue per search next to conversion, add-to-cart next to average click position, click-through rate next to revenue per session. Test the whole ranking configuration, not one parameter.

Questions this case answers
  • How to compare ranking configurations in an online experiment when the primary metric (conversion) and the business metric (revenue) can disagree
  • How to read a secondary behavioral metric correctly when its meaning depends on the direction of the primary metric
  • How to check that a search ranking change improves revenue and not only conversion rate
  • How to find out whether heavy personalization in search results hurts product discovery

Tools and methods

  • A/B test of a search configuration — compares two full ranking setups on live traffic.
  • Revenue per Search (RPS) with confidence intervals — shows whether a revenue lift is significant or too volatile.
  • NeuralSearch (Algolia) — ranking model that can be optimized for conversion or for revenue.
  • Personalization weight — low versus heavy weight of user affinity in ranking.
  • Custom Ranking and Dynamic Re-Ranking (Algolia) — manual business-rule ordering versus ordering by recent clicks and conversions.
Continue reading — it's free

Google or a one-time email code. No password.