Profit Prophet: Fortune-Telling on Cushions, or How to Evaluate the Effectiveness of a New Search System

Ekaterina Ramirez25 May 2024

Problem

A search vendor must convince a prospective e-commerce client that its search will bring more profit than the client's current one. An A/B test is not available at the sales stage: it costs both sides time, the client may choose a competitor, and sharing profit data before a contract is odd. The vendor has only a limited log of the client's sessions, an incomplete catalog and its own result lists for the same queries.

Example from the article: Constructor pitched a fictional gift store, "Radost i tochka": +200 conversions from ranking alone, about 1,000,000 of revenue per month with recall included.

Solution in brief

Compare the client's and the new system's result lists offline, query by query. Each product that changes position gains or loses conversions according to how many conversions the old and new position normally get across the whole store. Products shown by only one system get separate rules, and the total is priced with average order value.

Questions this case answers
  • How to estimate the revenue uplift of a new ranking before any experiment, using only logs of the current system and an offline comparison of two result lists
  • How to score a ranking change for items that only one of the two systems shows, when their position effect on outcomes is unknown
  • How to estimate the profit uplift of a new search ranking without an A/B test
  • How to compare two search result lists offline using the current system's click and conversion logs

Tools and methods

  • Position-share conversion model — share of all conversions that each result position receives.
  • Precision and recall estimates — gains from reordering shared products and from products only one system shows.
  • Linear regression — forecasts conversions for positions missing in the client's logs.
  • Smoothing and caps — logarithmic smoothing and per-position limits against outliers.
  • GPT — planned for product relevance labels.
Continue reading — it's free

Google or a one-time email code. No password.