Physicians Outperform Online Symptom Checkers in Diagnosis Accuracy

Though symptom checking apps and websites have been touted by some as the future of clinical diagnostics, a recent study has found that they are far less accurate than actual physicians at identifying the correct diagnosis when given a clinical vignette. The results of the study, published in a research letter in JAMA, compared the performance of physicians and online symptom checkers in diagnosing hypothetical clinical situations.

Lead author Ateev Mehrotra, MD, MPH, of Harvard Medical School, and colleagues had previously assessed the accuracy of 23 web- and app-based symptom checkers in a 2015 BMJ study, finding that the computer algorithms identified the correct diagnosis just 34% of the time. The 45 clinical vignettes used to test these symptom checkers provided the patient’s medical history and symptom presentation, but did not include physical examination or test findings.

In this study, the researchers distributed the same 45 clinical vignettes to physicians participating in Human Dx, an online platform where doctors generate differential diagnoses based on such vignettes. The doctors did not know which cases were part of this research study.

The 234 physicians who assessed the study vignettes determined the correct diagnosis first 72.1% of the time, which was more than double the accuracy rate of the symptom checkers. They were also significantly more likely (84.3% vs 51.2%) to have listed the correct condition in their top 3 diagnoses. Diagnosis accuracy was similar whether the physician was an attending, a fellow or resident, or an intern.

Accuracy also varied based on the acuity and prevalence of the conditions described in the vignettes. Physicians had an accuracy rate of 79.1% when diagnosing high-acuity situations for which urgent care would be required, compared to the 24.3% of symptom checkers that listed the correct diagnosis for these emergent conditions. In addition, 75.5% of physicians correctly identified an uncommon condition (defined as diagnoses that accounted for less than 0.3% of ambulatory visits in the US in 2009 to 2010) as opposed to 28.1% of symptom checking apps or websites.

The study authors acknowledged that although the symptom checkers had much lower accuracy rates, future research might examine whether these technologies could help support the diagnosis of a physician instead of replacing it.