SRE2.0: No LLM Metrics, No Future: Why SRE Must Grasp LLM Evaluation Now

Takahiro Sato16 Jun 2025

Problem

Teams that ship an LLM feature usually watch availability, latency and error rate. These can stay green while quality falls: a few lines changed in a prompt shift the answers, hallucinations rise at normal latency, and a minor model update changes the answer style. Without a quality indicator there is no service-level objective (SLO) to alert on and no way to tell whether a prompt or model change made the service worse. Generic metrics such as relevancy also miss what users value in a specific feature.

Example from the article: a toy service that summarizes Grimm's fairy tales (Little Red Riding Hood) in at most 500 characters, simple enough for children, with GPT-4o as both summarizer and judge.

Solution in brief

Score each quality dimension with a G-Eval judge: fixed evaluation steps, a rubric with five score bands, the source text as reference, and temperature 0. Run the judge many times on the same input to see its noise before fixing a target. Then add task-specific metrics for the real audience and keep a human reviewing the judge.

Questions this case answers
  • How to turn the quality of free-text model output into a stable numeric service-level indicator when infrastructure metrics cannot see quality regressions
  • How to make an LLM judge repeatable enough to compare prompt or model versions
  • How to define a quality SLI and SLO for an LLM feature when availability and latency look healthy?
  • How to score summaries from an LLM service automatically with an LLM judge and keep the score stable?

Tools and methods

  • G-Eval (DeepEval) — LLM judge that returns a 0–1 score and a reason from steps and a rubric
  • GPT-4o — generates the summary and acts as the judge
  • LLM-as-a-judge — automated scoring of free-text output
  • Human review — verifies and corrects the judge's final verdicts
Continue reading — it's free

Google or a one-time email code. No password.