Why Reports Mislead, or How to Measure the Effect of an LLM Feature

Nikita Bukhanchenko23 May 2026

Problem

A team that pretrains a large LLM (GigaChat Ultra at Sber) has many hypotheses: data, losses, pipeline and architecture changes. Training is expensive, so each feature gets only a few runs. The usual tech-report comparison trains the baseline and the feature model once and calls the higher benchmark score a win. But the score depends on the random seed, so the gap can be smaller than the natural spread, and the team may ship a lucky run.

Example from the article: the real effect of a feature may be +0.7 pp while seed noise is ±1.0 pp; on the slide MMLU is 82.1 ± 0.8 for the baseline and 82.8 ± 1.2 for the feature.

Solution in brief

Treat a benchmark score as a sample from a distribution: train the baseline and the feature on several seeds and compare the two small samples. Use tests that work with fewer than 10 observations, cut metric variance with fixed data order and paired seeds, and check training stability and parameter parity before trusting a gain.

Questions this case answers
  • How to tell a real effect of a model change from random training-seed noise when only a few training runs per variant are affordable
  • How to compare two groups of fewer than ten observations each with a permutation test or a Bayesian bootstrap instead of asymptotic tests
  • How to tell a real effect of a new feature from random seed noise when only a few training runs are affordable
  • How to compare two groups of fewer than 10 observations each when the t-test and the classic bootstrap do not work

Tools and methods

  • Multi-seed runs — several trainings per setup to measure baseline noise
  • Permutation test — pool the samples, reshuffle into two groups many times
  • Bayesian bootstrapped A/B — resample with random weights
  • Metric variance reduction — fixed data order, smoothing, pass@k, paired seeds
  • Stability monitoring — activations, weights, max_vio
Continue reading — it's free

Google or a one-time email code. No password.