Five leading AI models showed biases towards a limited number of products
AI is biased towards well-known or highly visible products: When asked to recommend drug products, LLMs tended to favor well-known brands and products with greater online visibility, even when alternative products were available.
Osaka Metropolitan University
Large language models (LLMs), such as ChatGPT and Claude, are increasingly being used to provide health information to the public. Although this is beneficial for improving access to treatment information, their limitations were revealed by a new study that found LLMs were biased in their treatment choices and using these outputs in practice could drive up treatment costs.
A research team led by Professor Daiju Ueda at Osaka Metropolitan University’s Graduate School of Medicine in collaboration with The University of Texas’ MD Anderson Cancer Centeraskedfive LLMs(GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Flash, Llama 4, and MedGemma) torecommend up to three FDA-approved drug products for a range of simulated scenarios.
Although an average of 5.5 FDA-approved products were available for each specified active ingredient, a single product accounted for an average of 97.5% of first-choice recommendations across the tested scenarios and models.
“The recommendations were mostlyfor brand-name drugs, thosehighly visible in Google searches, and treatmentswith dedicated Wikipedia pages,” Professor Ueda said.
The LLMs also tended not to recommend products with later FDA approval and those more commonly prescribed in actual physician practice. The study also identified recommendations that were inconsistent with regulatory information, including discontinued products.
The researchers then evaluated how much this could potentially cost patients using US data.Theylookedatascenariowhere physicians’ prescribing behavior was performed in accordance with the LLMs’ recommendations.
“We found thatannual acquisition costs would increase by approximately US$55.6 million,” Professor Ueda summarized.“This represents a 1,019% increase in the cost of purchasing these prescriptions.”
When instructed to prioritize lower-cost or generic products, the models shifted their recommendations accordingly; however, recommendations remained concentrated on a limited set of products.
“Our findings highlight the need to evaluate healthcare AI for biases in product selection and the handling of cost and regulatory information,” Professor Ueda said. “Combining AI systems with trusted drug information sources and human oversight may help support more appropriate medication choices.”
Competing interests statement
The authors declare no competing interests.
###
About OMU
Established in Osaka as one of the largest public universities in Japan, Osaka Metropolitan University is committed to shaping the future of society through the “Convergence of Knowledge” and the promotion of world-class research. For more research news, visit https://www.omu.ac.jp/en/ and follow us on social media: X, Instagram, LinkedIn.
Advertisement
