How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals

What the paper explores

Wu and colleagues reviewed public FDA summaries for medical AI devices and conducted a case study of pneumothorax detection across hospital datasets. Their analysis found gaps in public reporting of evaluation settings and showed that good performance within one site could coexist with weaker results at another.

How to interpret the evidence

Public summaries do not contain every detail available during regulatory review. The work is a historical analysis, and the cross-site example concerns selected datasets and models.

Questions for MedTech teams

Specify independent evaluation settings, relevant subgroups and clinical workflow questions. Evidence of generalizability should go beyond a single performance number on one dataset.