Problem
Teams score free text with an LLM judge or with embedding cosine similarity and then trust the number. Three Kaggle competitions show how these scores break. Participants fooled a panel of three small LLM judges, found a leak in published metadata, and gamed a similarity metric with one special token. The same risk applies to any LLM-based evaluation, leaderboard or reward signal: the metric stops measuring quality.
Example from the article: in the "LLMs - You Can't Please Them All" competition (1693 teams), the speaker reached the maximum of the first metric on day one with a short injected instruction asking the judge to output 9.
Solution in brief
Treat every automatic metric as an attack surface. Check what the judge or encoder accepts, what metadata leaks the answer, and whether the metric moves when meaningless tokens change. A system prompt alone does not protect an LLM judge.
Questions this case answers
- How to make an LLM-as-a-judge scoring system resistant to adversarial text in the answers and how to detect answer leaks through metadata before an evaluation
- How to check that a cosine-similarity metric on sentence embeddings cannot be gamed by meaningless tokens
- How to make an LLM-as-a-judge scoring system harder to hack with adversarial text
- How to find and prevent test-set leaks through metadata before running an evaluation or competition
Tools and methods
- LLM-as-a-judge panel — three different LLMs (Phi, Qwen, Gemma) grade each essay 0 to 9
- Prompt injection — text that tells the judge to ignore the task and output a chosen grade
- Token-count matching with tiktoken — links hidden test rows to public metadata
- Cosine similarity on sentence-t5-base embeddings — metric of LLM Prompt Recovery, exploitable through the end-of-sequence token
Google or a one-time email code. No password.