From Chaos to Control: Building a Streamlit Tracer for Debugging RAG Systems

Vlad Drobotukhin24 May 2025

Problem

A team builds a RAG system for a domain such as law or HR and needs domain experts to judge its answers. Langfuse and LangSmith show a trace as a tree of technical steps, and an expert who has never used them cannot tell where an answer went wrong. Custom chain-of-thought output makes it worse: a JSON with 11 fields, among them reasoning and query_expansion, is a wall of text. Debugging turns into long arguments with experts and the improvement cycle slows down.

Example from the article: in the Enterprise RAG Challenge 2.0 (100 annual-report PDFs, 100 questions), one question went through 10 to 14 pipeline steps, and the author needed to see the whole path at once.

Solution in brief

Build a small web page over the raw JSONL logs that shows one question on one screen: the prompt and response of each step, and a map of document pages used, analyzed or skipped. The expert reviews this page instead of the observability tool. It complements Langfuse and takes 3 to 5 hours to assemble from ready components.

Questions this case answers
  • How to debug a multi-stage question-answering pipeline together with domain experts who cannot read technical traces
  • How to find which stage of a multi-stage pipeline produced a wrong answer and how to evaluate each stage separately instead of only the final output
  • How to debug a multi-step RAG pipeline together with domain experts who cannot read Langfuse or LangSmith traces
  • How to find out at which stage of a RAG pipeline a wrong answer was produced

Tools and methods

  • Streamlit — the tracer UI (Gradio or another framework works too).
  • Langfuse, LangSmith, Opik, MLflow — standard tracing tools that the tracer complements.
  • Stage-wise RAG evaluation — unit tests for retrieval plus model and human evaluation.
  • Pydantic structured output with reasoning fields — the step logs the tracer displays.
Continue reading — it's free

Google or a one-time email code. No password.