Problem
A team that ships an AI assistant for experts has to show that it saves time without making the experts' work worse. In radiology a chest CT study has 400–800 slices, a radiologist reads about 20 studies a day at 5–20 minutes each, and scans grow faster than radiologists. Published evaluations mostly measure time alone, their designs differ (some compare the same doctors before and after), and a faster read may hide a worse report.
Example from the article: the only published CT study found by the speaker showed 22.1% less reading time with AI (328 ± 122 s vs 421 ± 175 s, 3 radiologists) and said nothing about accuracy.
Solution in brief
Run a randomized reader study. 40 experts in four groups read the same 200 cases. In the two experimental groups, each expert reads half of the assigned cases with the tool and half without, in random order, so every expert is their own control. Two extra control groups separate the effect of the tool from the effect of the study setting and of knowing what the tool detects.
Questions this case answers
- How to measure whether an AI assistant saves expert time without lowering decision quality, using a randomized within-expert design with control groups
- How to separate the effect of a tool from the effect of being informed about it and of an artificial study setting in an expert experiment
- How to measure the effect of an AI assistant on expert speed and accuracy with a controlled reader study
- How to design a cross-over experiment where every expert is their own control
Tools and methods
- Cross-over reader study — each expert reads cases both with and without the tool
- Control and informed control groups — read without the tool; the informed group is told which findings the tool detects
- Per-finding quality metrics — sensitivity, precision, specificity, F1
- Automatic timing — reading time logged per case
Google or a one-time email code. No password.