Bias in AI: Medical AI products may be forming and amplifying healthcare bias against demographic subgroups.
The recent flood of Artificial Intelligence (AI) models and content has undoubtedly brought along an unwelcome bug from the human-made content it scrapes from: bias. AI has been noted to form, copy, and amplify harmful stereotypes of already marginalized groups. This phenomenon has far reaching effects as AI becomes a normal part of everyday life. Most worrying is that the prolific bias already circling healthcare settings may now be exacerbated by AI-driven decisions. These can have a direct impact on patient care and outcomes, underscoring the need for careful evaluation of potential biases.
Therefore, a research group led by Dr. Shannon L. Walston at Osaka Metropolitan University’s Graduate School of Medicine conducted a scoping review to identify studies validating commercially available radiology AI products and to note trends when reporting on sex, age, and ethnic demographic subgroups. The team collected 545 studies on 252 products with reported demographic subgroup data. Trends were mapped using a regression analysis.
Of the 545, only 77 studies validating 52 products were found to include demographic details and subgroup analysis results. When the Wilson Confidence Interval formula, which is used to calculate proportion, was applied to studies validating AI for tuberculosis detection, the researchers found that 67% of the reported datasets were at risk of being underpowered for sex subgroup analysis. This revealed that, despite the demand for improved demographics reporting to address the dangers of biased medical AI, performance reporting for demographic subgroups has not become more common.
This finding exposes the need for effective, transparent reporting to confirm the safe and unbiased performance of these medical AI products across all patient subgroups.
“This scoping review quantifies how fragmented the commercial validation landscape is, showing that reporting for both the demographics and per-subgroup performance is inadequate for estimating subgroup bias. This systemic problem requires effort from all stakeholders, from researchers to regulatory agencies, encouraging thorough reporting and commercial product validation to support physician and patient trust in medical AI products," stated Dr. Walston.
The findings were published in European Radiology.
Conflict of interest
The authors of this manuscript declare relationships with the following companies: Medical AI Promotion Institute, Inc.
###
About OMU
Established in Osaka as one of the largest public universities in Japan, Osaka Metropolitan University is committed to shaping the future of society through the “Convergence of Knowledge” and the promotion of world-class research. For more research news, visit https://www.omu.ac.jp/en/ and follow us on social media: X, Instagram, LinkedIn.
