Problem
realestate.com.au lets users save search filters and receive daily notifications of new matching listings. The site has 12.7 million monthly users, 4.6 million saved searches and hundreds of new listings per hour at weekday peaks. A typical weekday generates millions of notifications. A conventional design copies the filter logic of the live search into a second system, and the two can drift apart.
Example from the article: a team was adding a filter that excludes listings going to auction. With a conventional matcher the exclusion would have to be built twice, once in listing search and once in saved-search matching.
Solution in brief
Store each saved search as the same Elasticsearch query that the live search would run, in an index with a percolator field. When a listing is created, send it as a document in a percolate query and get back every saved search it matches. One query generator then serves both live search and alerts.
Questions this case answers
- How to match each newly created item against millions of stored user queries without reimplementing the query logic that the live search already has
- How to notify users about new listings that match their saved searches when there are millions of saved searches?
- How to keep saved-search matching consistent with the live search without writing the filter logic twice?
- How to support new search filters in saved-search alerts without extra development?
Tools and methods
- Elasticsearch percolator — stores queries in an index and finds the stored queries that match a given document
- Listing Search API — returns the exact query it would run for a search
- Postgres tables (Match Store) — hold the matches until the 4pm collation
- Saved Search Feeder and Matcher — services that store queries and percolate new listings
Google or a one-time email code. No password.
