Problem
Zillow has tens of millions of monthly visitors with very different goals: some browse for fun, some check a home value, some are about to make an offer. These users need different features, so the team must find near-term buyers among all users. The real intent signals (working with an agent, making an offer, closing) mostly happen outside Zillow, and the outcome is known only about 60 days later from public records.
Example from the article: a daily batch job scores every recently active registered user, and the score can decide who sees local-agent upsell messaging and who sees more home recommendations.
Solution in brief
Train a binary classifier on behavioral features from the last days and weeks. Positive labels come from completed transactions in public records, matched to registered users. Cut 1,500 features to 50, train gradient boosted trees on a downsampled negative class, and judge quality by how positive labels spread across score deciles, not by AUC.
Questions this case answers
- How to predict that a user will complete an offline, delayed outcome from product behavior when only a subset of true positives can be labeled
- How to validate a classifier whose negative class contains hidden positives, so precision, recall and AUC are unreliable
- How to find the users who are about to buy when the purchase happens outside our product?
- How to build training labels for a propensity model when the target event is only visible in public records weeks later?
Tools and methods
- Propensity model — binary classifier of likelihood to transact soon.
- Windowed behavioral features — event counts over 1, 3 and 7 days, by platform.
- Preference-variance features — narrowing of location preferences.
- Three-step feature selection — univariate filter, correlation pruning, random forest importance.
- Gradient boosting decision trees — the production model, over 1,000 trees.
- Decile analysis — quality check when AUC is distorted.
Google or a one-time email code. No password.