Problem
KRAFTON (PUBG) receives community feedback about cheating in many languages, full of slang and sarcasm. An hourly LLM summary names the same issue differently each time, so the trend dashboard splits one problem into many topics. Separately, a research cheat-detection model written as local pandas code is too slow for production: while detection lags, cheaters keep ruining matches.
Example from the article: one hour the LLM said "wall tracking", another hour "ESP suspicion", later "cheating frustration".
Solution in brief
Keep a topic dictionary: vector search proposes similar existing topics, an LLM checks whether they are really the same issue, and only truly new issues are added. Score each pipeline step with LLM judges from three providers to pick a cheaper model per step. Move the detection model from hourly batch to Structured Streaming, then fix cost, consistency and scale.
Questions this case answers
- How to turn noisy multilingual user feedback into a stable set of recurring issue topics by matching new mentions to a growing dictionary with an LLM check
- How to move a research detection model from batch code into a low-latency streaming pipeline with a small team while controlling its cost and memory
- How to turn noisy multilingual player feedback into a stable set of issue topics with an LLM
- How to evaluate an LLM pipeline step by step and choose a cheaper model for each step
Tools and methods
- Topic dictionary with vector search and LLM validation — keeps topic names stable.
- LLM-as-a-judge ensemble — three judge models from different providers score each step from 0 to 5.
- Databricks
ai_queryand Vector Search — LLM calls in SQL and vector search on existing Delta tables. - Structured Streaming with
applyInPandas— reuses validated pandas feature code in a near-real-time job. - Online Feature Store, MLflow, Model Serving endpoints — key lookups instead of full joins, API-based inference.
- Dispatcher and worker streaming jobs — scale-out for higher volume.
Google or a one-time email code. No password.