Problem
A classifieds marketplace earns its value when a buyer and a seller close a deal, but most deals happen offline and cannot be tracked. Counting buyer-to-seller contacts is a poor substitute: a buyer can message ten sellers to compare replies and discounts, and ten contacts do not mean ten deals. Avito also has six verticals with different business models, so the metric must be comparable across all of them. Business, analysts and data scientists each want a different definition of the target event.
Example from the article: a phone buyer writes to ten sellers to see who answers faster or gives a discount. The contact count grows, but no value is created.
Solution in brief
Measure the missing funnel step between contact and deal: the agreement between seller and buyer. Detect it with an ML model trained on chats, using two opposite classes ("agreement" and "no agreement"). Get the labeling instruction by labeling real data with overlap and fixing the instruction until only random disagreements remain.
Questions this case answers
- How to define a north-star metric when the core value event happens offline and cannot be tracked directly
- How to get a labeling instruction for a fuzzy binary concept that several teams accept, and detect that concept in conversations
- How to define a North Star metric when the core value event (a deal) is not directly observable
- How to get a labeling instruction that business, analysts and data scientists all accept
Tools and methods
- Binary labeling ("looks like an agreement" or not) — keeps the target simple for annotators and the product team.
- Instruction calibration on real data — iterate labeling with overlap and review disagreements.
- Golden set and annotator validation — train and check annotators.
- BERT, TF-IDF with logistic regression — models trained on the labeled data.
- Validation on real deals — checks that the model tracks deals.
Google or a one-time email code. No password.