Imputing Data for the Zestimate

Angela Burden3 Nov 2023

Problem

Zillow collects home data from many providers, and each leaves different gaps and uses different formats. The Zestimate (Zillow's ML home valuation) needs every feature, and wrong or empty inputs lower its accuracy. Mean, median or one-column-at-a-time fills ignore how attributes interact, and those interactions differ by region. About 5% of homes have no public record information, so they miss important features and gain the most from a better fill.

Example from the article: a home with known square footage and no bedroom count, or a 2000 sq ft home with 20 bathrooms.

Solution in brief

Train a denoising autoencoder on cleaned home attributes. During training, part of the input is corrupted and the network learns to restore the original values. At scoring time the missing or implausible cells are filled from the reconstruction. Zillow chose it over median, kNN, random forest and a variational autoencoder, then checked it against the Zestimate error on sale pairs.

Questions this case answers
  • How to impute missing and implausible values in high-dimensional tabular data from several providers when attributes interact non-linearly and missingness differs by region
  • How to evaluate an imputation model by its effect on a downstream predictor rather than by reconstruction error alone
  • How to fill missing and implausible property attributes before they reach a valuation model?
  • How to choose between median, kNN, random forest and autoencoder imputation for mixed tabular data?

Tools and methods

  • Denoising autoencoder — reconstructs all features at once and fills missing or implausible values
  • Median, kNN, random forest, variational autoencoder — baselines and rival imputers
  • Quantile transform, embeddings, regional aggregates — input preparation for mixed data
  • H3 index and census regions — source of regional surrogate features
  • PyTorch, Kubeflow Pipelines, Metaflow — training and the production workflow
Continue reading — it's free

Google or a one-time email code. No password.