Text Deduplication: A Review

Lavrenty Grigorev27 May 2023

Problem

A team ranks a news stream with an ML model for site users or internal business clients, and the output still contains near-copies of the same item. Users waste time, someone cleans results by hand, and the business pays for labor. The same issue appears in datasets and in text corpora for language models. Exact duplicates are easy to drop, but near-duplicates differ by small edits, and where "small" ends is a business decision.

Example from the article: a bank's NLP team deduplicates a news feed after relevance ranking, so that each item reaches the user once.

Solution in brief

Treat near-duplicate search as its own task, not as semantic textual similarity (STS). For huge corpora, hash texts with MinHash LSH and accept some noise. Where quality matters, use LSH only to get candidate pairs and rerank them with a model trained on labeled pairs.

Questions this case answers
  • How to find and remove near-duplicate texts at scale without comparing all pairs
  • How to evaluate a near-duplicate detector when only some pairs are labeled
  • How to find and remove near-duplicate texts at scale
  • How to deduplicate a training corpus or a train/validation split

Tools and methods

  • Jaccard index — word overlap between two texts; exact but needs all pairs.
  • Shingling + MinHash LSH — cheap candidate generation that scales on CPU.
  • KNRM, DSSM, cross-encoder — pair models for reranking LSH candidates.
Continue reading — it's free

Google or a one-time email code. No password.