Representative Sampling: A Statistical Method to Derive Data Insights from Noisy Data Distributions

Neil Palleti25 Mar 2022

Problem

A retailer builds one profile per customer from in-store, online and third-party data. When the data is sparse, one shopper splits into several profiles, so averages such as visits per customer come out skewed. Manual cleaning is impossible: at this scale a labeler cannot tell whether a profile is complete. Random samples of an impure set stay impure, however many times you draw them.

Example from the article: a customer pays with three credit cards, 10 store visits per card. The pipeline sees three shoppers with 10 visits each instead of one with 30, which pushes the visits distribution to the right.

Solution in brief

Treat each verified global distribution as a constraint and score every profile by how well it fits. Draw a weighted random sample by these scores. Complex metrics computed on the sample can then be scaled to the whole base.

Questions this case answers
  • How to draw a subset from a noisy, partly duplicated population so that its distributions match independently verified population statistics
  • How to extrapolate complex metrics from a clean sample to the full population when the full data cannot be trusted directly
  • How to pick a trustworthy subset of customer profiles when some profiles are fragmented and the full set is noisy?
  • How to build a sample whose distributions match verified global statistics, such as visits per customer or channel mix?

Tools and methods

  • Constraint distributions — verified global statistics (visits per year, channel mix) used as targets
  • Per-constraint scoring with frequency normalization — gives each profile a sub-score for each constraint, corrected for the noisy population
  • Weighted random sampling without replacement — draws the sample in proportion to the aggregate scores
  • KL divergence or mean comparison — checks how close the sample is to the targets
Continue reading — it's free

Google or a one-time email code. No password.