How Developer Productivity Changed with AI4SDLC Tools

Alexey Antonov23 May 2026

Problem

A bank invests heavily in AI tools for developers and has to show whether they pay off. Industry numbers come from surveys of self-assessment: DORA reports more than 80% of users noticing improvement, Anthropic claims +50% for its own developers. An A/B test is not possible, because tool usage and teams are very heterogeneous and usage is a continuous quantity rather than on/off. A before/after comparison is mathematically unjustified.

Example from the article: T-Bank evaluated three tools of its AI4SDLC platform (Copilot autocomplete, Developer Assistant coding agent, IDE chat) on real git and telemetry data and did not reproduce the +50% claim.

Solution in brief

Define productivity through delivery and developer metrics computed from git and CI, then estimate the effect of each tool's usage intensity on them by double machine learning. A backdoor causal graph decides which covariates (grade mix, team size, other AI tools, season, CI time) must be controlled.

Questions this case answers
  • How to measure the causal effect of a continuous usage intensity of a tool on a team output metric when a randomized experiment is impossible
  • How to define and clean a delivery-speed metric from version control and CI event logs
  • How to measure the causal effect of AI coding tools on developer productivity without an A/B test
  • How to define and clean a delivery metric such as lead time for changes from git and CI data

Tools and methods

  • Backdoor criterion — chooses the covariates to control for
  • Double machine learning (DML) — estimates the average treatment effect (ATE) and the effect within a subgroup (CATE)
  • Lead time for changes (LTFC) — team-level delivery metric
  • Git/GitLab data, IDE telemetry, CI/CD pipeline logs — inputs
Continue reading — it's free

Google or a one-time email code. No password.