Problem
Teams that ship an LLM feature usually watch availability, latency and error rate. These can stay green while quality falls: a few lines changed in a prompt shift the answers, hallucinations rise at normal latency, and a minor model update changes the answer style. Without a quality indicator there is no service-level objective (SLO) to alert on and no way to tell whether a prompt or model change made the service worse. Generic metrics such as relevancy also miss what users value in a specific feature.
Example from the article: a toy service that summarizes Grimm's fairy tales (Little Red Riding Hood) in at most 500 characters, simple enough for children, with GPT-4o as both summarizer and judge.
Solution in brief
Score each quality dimension with a G-Eval judge: fixed evaluation steps, a rubric with five score bands, the source text as reference, and temperature 0. Run the judge many times on the same input to see its noise before fixing a target. Then add task-specific metrics for the real audience and keep a human reviewing the judge.
Questions this case answers
- How to turn the quality of free-text model output into a stable numeric service-level indicator when infrastructure metrics cannot see quality regressions
- How to make an LLM judge repeatable enough to compare prompt or model versions
- How to define a quality SLI and SLO for an LLM feature when availability and latency look healthy?
- How to score summaries from an LLM service automatically with an LLM judge and keep the score stable?
Tools and methods
- G-Eval (DeepEval) — LLM judge that returns a 0–1 score and a reason from steps and a rubric
- GPT-4o — generates the summary and acts as the judge
- LLM-as-a-judge — automated scoring of free-text output
- Human review — verifies and corrects the judge's final verdicts
Google or a one-time email code. No password.
