Problem
Whatnot, a live shopping marketplace, saw more scam attempts against users, especially new ones who do not know the platform rules. A scam starts with a harmless direct message, such as a question about a listing or a giveaway win. The scammer builds trust, then tries to move the chat off the platform. A message-level model has low precision, because each message alone looks innocent. A rule engine over scalar values cannot read context.
Example from the article: a "buyer" tells a new seller that a payment was declined but money was debited, sends a screenshot and asks the seller to email the support address shown in it.
Solution in brief
Score the whole conversation with an LLM, only for accounts that pass cheap signal filters. The LLM returns JSON with a scam likelihood and an explanation. The existing rule engine combines the likelihood with account signals and decides whether to restrict the account and alert the ops team.
Questions this case answers
- How to detect adversarial multi-turn behavior when no single event is a reliable signal and the tactics keep changing
- How to turn free-text model judgments into auditable enforcement decisions with a human fallback
- How to detect scam conversations in a marketplace when a single message looks harmless?
- How to combine an LLM's judgment with account signals so that enforcement stays controllable?
Tools and methods
- LLM conversation scoring (OpenAI API) — scam likelihood from 0 to 1 plus reasoning
- Rule engine with threshold rules — LLM score plus account age, message frequency, order history
- Signal pre-filter — selects conversations for the LLM
- OCR on attachments — adds image text to the LLM input
- Human review — ops checks flagged and uncertain cases
- LangChain, Apache Kafka — LLM orchestration, delivery of access changes
Google or a one-time email code. No password.