Increment vs Experiment: How to Reduce Uncertainty When Developing ML Models

Andrey Tyunyatkin24 May 2025

Problem

In classical software an increment is a shipped feature. In ML the increment is an experiment: until training finishes, the team knows neither the final quality nor the cost of the next step. ML leads cannot promise deadlines, struggle to rank ideas, and lose track of the notebooks, models and datasets that pile up. The cost is missed schedules and irreproducible results.

Example from the article: a model reaches AUC 0.78. Is that enough, or is another 2% worth a couple more weeks of modeling?

Solution in brief

Process: describe each idea as a growth point card, score it with a custom formula, and track it on a CRISP-DM Gantt chart; when late, lower requirements. Artifacts: give every artifact type a registry (checklists, research monorepo, experiment tracker, data catalog, feature store).

Questions this case answers
  • How to plan deadlines and prioritize ideas when the outcome of each modeling experiment is unknown until it finishes
  • How to organize research code, datasets and experiment artifacts into registries so that a modeling team can reproduce results
  • How to plan and track deadlines for ML projects when the outcome of each experiment is unknown
  • How to prioritize a backlog of ML ideas with a custom scoring formula

Tools and methods

  • ML growth point card — one task card in a tracker: feature–target statement, goal, metrics, stakeholders, product vision
  • Custom prioritization formula + planning poker — benefit divided by cost, parameters scored 1 to 10 by stakeholders
  • Gantt chart on CRISP-DM stages — goal, key steps, definition of done and definition of ready per stage
  • Research monorepo — dev containers, naming standards, CI linting and SAST, merge requests for every experiment
  • ClearML (or Weights & Biases, MLflow) — hyperparameter logging, tags, model registry, experiment comparison
  • OpenMetadata — data catalog with data quality checks, versioned metadata, lineage
  • MVP feature store — Airflow, a data catalog and a database
Continue reading — it's free

Google or a one-time email code. No password.