A study of 950 AI medical devices found that lack of clinical validation and public company status were linked to higher odds of early recalls.

Original Title: Early Recalls and Clinical Validation Gaps in Artificial Intelligence-Enabled Medical Devices

Journal: JAMA health forum

DOI: 10.1001/jamahealthforum.2025.3172

AI Medical Device Recalls and Validation Gaps

Overview

Artificial intelligence-enabled medical devices (AIMDs) are increasingly common in clinical practice, yet many receive US Food and Drug Administration (FDA) clearance through an accelerated pathway that does not require prospective human testing. This raises concerns about their performance and safety after entering the market. This study investigated the frequency of recalls among AIMDs and examined whether recalls were associated with two key factors: the lack of premarket clinical validation and the type of manufacturer (publicly traded vs. privately held). Researchers analyzed 950 FDA-cleared AIMDs and found that while recalls were relatively uncommon, affecting 6.3% of devices, they were significantly linked to these pre-market and commercial characteristics.

Novelty

The study’s primary contribution is its quantitative analysis connecting post-market safety events to specific pre-market conditions. It demonstrates that devices from publicly traded companies had 5.9 times higher odds of being recalled, while those without any reported clinical validation had 2.8 times higher odds. This highlights a potential gap in the regulatory process for these advanced technologies. Furthermore, the research reveals that these issues often appear early in a device’s lifecycle; 43.4% of all recalls occurred within the first 12 months of the device receiving clearance. This finding suggests that the current 510(k) clearance pathway may not be fully adequate for catching performance failures in AI technologies before they are widely adopted in clinical settings.

My Perspective

The findings point to a potential conflict between commercial pressures and patient safety. For publicly traded companies, the need to meet investor expectations may encourage faster product launches, possibly at the expense of comprehensive pre-market testing. This effectively creates a situation where AIMDs are tested in real-world clinical environments rather than in controlled trials. The 510(k) pathway is intended for devices that are substantially equivalent to existing products, but its application to complex, adaptive AI algorithms may be problematic. The opaque nature of some AI models means their failure modes can be difficult to predict, which makes robust, prospective clinical validation before deployment even more essential than for traditional medical devices.

Potential Clinical / Research Applications

For healthcare organizations, this research provides valuable information for procurement decisions. Clinicians and administrators could prioritize AIMDs that are supported by transparent and prospective clinical trial data, particularly when considering tools for critical diagnostic tasks. For the research community, this study opens pathways to investigate the specific types of algorithmic or software failures that lead to recalls. Future studies could explore whether certain classes of AI models are more susceptible to post-market problems. There is also an opportunity to develop improved pre-market evaluation frameworks tailored to AIMDs, which could better predict real-world performance and enhance patient safety.

Important writing guidelines

This cross-sectional study analyzed publicly available FDA databases on AIMD clearance and recalls. The authors used Kaplan-Meier analysis to assess recall-free survival over time and multivariable logistic regression to identify factors associated with recall events. Among 950 AIMDs, 60 devices (6.3%) were involved in 182 recalls. The analysis showed that a lack of clinical validation (Odds Ratio 2.8) and being manufactured by a publicly traded company (Odds Ratio 5.9) were independently associated with a higher likelihood of recall. Study limitations include its reliance on publicly available reports and the exclusion of software updates that were not formally classified as recalls by the FDA.