(Credit: Cavan Images / Raffi Maghdessian / Cavan via Getty Images)
AI tools can make it harder for radiologists to accurately read images, such as X-rays and CT scans, according to a new study from Harvard Medical School published today.
The results highlight the need to better understand how radiologists interact with AI tools, noting that previous studies have focused on analyzing "groups of clinicians as a whole, overlooking the variation" between individuals.
The team measured the performance of 140 radiologists with and without AI assistance on 15 chest X-ray diagnosis tasks. It considered factors such as the radiologists' seniority, expertise, and years of experience. The radiologists were also trained on how to use the assistive AI system before the experiment.
Surprisingly, the study found that none of these factors can reliably predict whether a clinician's accuracy increases with the use of AI-assisted diagnostic tools. They also did not always improve the performance for lower-performing radiologists. Perhaps more disconcerting, the conclusions drawn by radiologists after using AI tools had a direct impact on the prescribed treatments for the 324 patients in the study.
“We find that different radiologists, indeed, react differently to AI assistance — some are helped while others are hurt by it,” says co-senior author Pranav Rajpurkar, assistant professor of biomedical informatics at Harvard Medical School.
While most of the study seemed to challenge prevailing wisdom, it affirmed one thing: If the AI tool itself is often inaccurate, it lowers the performance of the radiologist. However, the study stopped short of providing an explanation for why AI tools lead to such a high variation in performance, choosing instead to focus on the fact that these variations exist and are unpredictable based on "conventional" experience factors.
The results should not discourage the adoption of AI tools in medicine, the researchers say, but they do underscore the need to better understand how humans and AI interact. Greater understanding in this area could lead to more effective tools that "minimize harm" and improve patient outcomes.
“Clinicians have different levels of expertise, experience, and decision-making styles, so ensuring that AI reflects this diversity is critical for targeted implementation,” says co-author Feiyang “Kathy” Yu.