Leaves to Neurons: Using Deep-Cross Networks for Ranking Search Results

Tim Huang19 May 2023

Problem

Thumbtack ranks professionals for consumers searching across very different job categories, from house cleaning to landscaping. For years gradient boosted decision trees (GBDT) were the production ranker, and plain feed-forward neural networks (DNN) never beat them offline. A neural ranker would also open the way to embeddings and text inputs. The team's hypothesis: trees learn feature interactions on their own, while a DNN needs them engineered by hand.

Example from the article: certain job type and zip code combinations convert worse by nature, so the ranker has to model that pair, not each feature alone.

Solution in brief

Thumbtack trained a Deep-Cross Network (DCN-V2) on the same features as the GBDT. Cross layers multiply the input by itself, which gives the network explicit feature crosses. After tuning embedding regularization, the model lowered log loss by 0.59% against a retrained GBDT and improved conversion in an A/B test on native app users.

Questions this case answers
  • How to move a tabular ranking model from gradient-boosted trees to a neural network without losing quality when feature interactions drive relevance
  • How to replace a gradient-boosted search ranker with a neural network that matches or beats it?
  • How to make a neural ranker learn feature interactions that trees capture on their own?
  • How to diagnose and control overfitting of a neural ranker on high-cardinality categorical features?

Tools and methods

  • DCN-V2 — neural ranker whose cross layers build feature interactions explicitly
  • GBDT — baseline ranker, retrained on the new dataset for a fair comparison
  • Embedding and dense-layer regularization — L2 on embeddings, cross and dense layers, dropout on dense layers
  • Permutation importance and SHAP — find which features the network overfits and interacts on
Continue reading — it's free

Google or a one-time email code. No password.