Problem
A team builds a RAG system for a domain such as law or HR and needs domain experts to judge its answers. Langfuse and LangSmith show a trace as a tree of technical steps, and an expert who has never used them cannot tell where an answer went wrong. Custom chain-of-thought output makes it worse: a JSON with 11 fields, among them reasoning and query_expansion, is a wall of text. Debugging turns into long arguments with experts and the improvement cycle slows down.
Example from the article: in the Enterprise RAG Challenge 2.0 (100 annual-report PDFs, 100 questions), one question went through 10 to 14 pipeline steps, and the author needed to see the whole path at once.
Solution in brief
Build a small web page over the raw JSONL logs that shows one question on one screen: the prompt and response of each step, and a map of document pages used, analyzed or skipped. The expert reviews this page instead of the observability tool. It complements Langfuse and takes 3 to 5 hours to assemble from ready components.
Questions this case answers
- How to debug a multi-stage question-answering pipeline together with domain experts who cannot read technical traces
- How to find which stage of a multi-stage pipeline produced a wrong answer and how to evaluate each stage separately instead of only the final output
- How to debug a multi-step RAG pipeline together with domain experts who cannot read Langfuse or LangSmith traces
- How to find out at which stage of a RAG pipeline a wrong answer was produced
Tools and methods
- Streamlit — the tracer UI (Gradio or another framework works too).
- Langfuse, LangSmith, Opik, MLflow — standard tracing tools that the tracer complements.
- Stage-wise RAG evaluation — unit tests for retrieval plus model and human evaluation.
- Pydantic structured output with
reasoningfields — the step logs the tracer displays.
Google or a one-time email code. No password.