Developing a RAG System on the History of Bashkortostan Using a Benchmarking Methodology

Ruslan Akhmetvaleev23 May 2026

Problem

The AI Laboratory of the Institute of History, Language and Literature (Ufa Federal Research Centre) helps historians work with scattered sources on Bashkortostan. Historians verify every fact against quotations, many sources sit in closed archives the LLM never saw, and one place or person has different names and assessments in different periods. A general LLM invents dates and people and gives no references. The team also had to show that a RAG beats the bare LLM, with no ready benchmark.

Example from the article: asked which division Salavat Yulaev commanded in the Great Patriotic War, a model should notice that he lived 100 or 200 years earlier. A cavalry division carried his name, so confusion is likely.

Solution in brief

Load verified sources into one corpus, index it by paragraph in PostgreSQL with pgvector, and answer through retrieval, reranking and generation, with page-level references. Measure the gain as coverage on a benchmark of six question groups, mostly generated by an LLM from the corpus and filtered by hand.

Questions this case answers
  • How to build a question-answering system over a small closed corpus that the LLM has not seen, with page-level source references
  • How to measure how much retrieval adds over an LLM-only answer on a niche domain without a ready benchmark
  • How to build a RAG system over a closed specialised document corpus that the LLM did not see in pretraining
  • How to measure how much a RAG adds over an LLM-only answer on a niche domain

Tools and methods

  • RAG with hybrid search and a reranker — retrieves relevant paragraphs.
  • PostgreSQL + pgvector — stores chunk vectors; enough for a small corpus.
  • Gemma — small LLM chosen for consumer-GPU deployment.
  • LLM metadata extraction — abstract, authors, editors.
  • Six-group coverage benchmark — factual, chronological, terminological, historiographical, cause-and-effect, hallucination check.
  • Largest remainder method — allocates chronological questions across eras.
Continue reading — it's free

Google or a one-time email code. No password.