Evaluating Diagnostic Decision Support Systems for Patients Requiring Urgent Primary or Emergency Care or with Stroke

By rigorously evaluating a popular patient-facing symptom checker app, understanding around this app’s safety, accuracy, and ability to prompt patients to seek care will be improved.

Project Details -
Ongoing

Grant Number

R01 HS029513
Funding Mechanism(s)

AHRQ Health Services Research Projects (R01)
AHRQ Funded Amount

$1,812,513
Principal Investigator(s)

Fraser, Hamish S.F.
Organization

Brown University
Location

Providence

Rhode Island
Project Dates

07/01/2023 - 05/31/2028
Technology

Clinical Decision Support System

Machine Learning

Mobile App
Care Setting

Emergency Department
Medical Condition

Stroke
Population

Adults
Type of Care

Acute Care
Health Care Theme

Clinical Decision Making

Health Literacy

Patient Engagement

An individual’s proactiveness in seeking care plays a critical role in receiving accurate and timely diagnoses and treatment. The failure to seek timely care can lead to poor outcomes. An individual’s lack of recognition of the symptoms they are having, or their hesitancy to seek care, can directly impact morbidity and mortality. For instance, in the case of stroke, an individual optimally needs to receive treatment within 4 hours of onset to benefit from treatment options that can reverse the damage. Despite public education campaigns to heighten awareness of the symptoms of stroke, the proportion of patients seeking care in the 4-hour window remains low.

Symptom checkers (SCs) that give diagnostic decision support are available via mobile apps or the internet. These tools have experienced large growth, with over 100 available apps that have been used tens of millions of times. While research on SCs has been limited and of small sample size, it has shown that their use can impact users’ behaviors around seeking care. Although early research has been positive, the studies lack rigorous evaluation of these apps, including research on safety, effectiveness, and impact on decisions. This research will evaluate the safety, usability, and diagnostic and triage accuracy of one popular SC and the impact of its use on patient’s decisions to seek care.

The specific aims of the research are as follows:

Evaluate the diagnosis and triage accuracy, usability, safety, and effect on patient decision making of the Ada Health SC app in emergency or urgent care.
Study the diagnosis and triage accuracy of existing algorithms and create new algorithms with datasets from patients presenting with transient ischemic attack (TIA) or stroke.

This work will build on previous research by the team of a popular SC app (Ada Health), that studied 241 patients seen in urgent primary care or the emergency department at Rhode Island Hospital. In this current research project, 700 individuals will be recruited from those same areas, including those that may be having a possible TIA or stroke. The patients will be assessed as to the urgency of care they intend to seek both before using the Ada Health SC app and afterward. Participants will be asked to complete a questionnaire on app usability. The accuracy of the app’s diagnosis and triage abilities will be assessed by comparing the diagnostic and triage outputs of the app to the assessment of the physician who evaluated the patient. In addition, the app’s results will be compared with and critiqued by physicians who did not participate in the care of the patient.

With an aim of improving early diagnosis and risk stratification, the research team will create and validate new machine learning algorithms with two unique data sets of 2,300 patients with possible TIA or stroke seen at the same clinical locations as the study. Outputs will be compared with existing SCs, algorithms, and guidelines on diagnosis and management of TIAs and stroke. A large global data set of 158,000 patients who used the app and who had possible TIA or stroke will be analyzed to evaluate the impact of age, sex, race, ethnicity, country, and socioeconomic group on their symptoms, comparing this to the study data set results.

This research will enhance the body of research on SCs to more rigorously understand their safety, accuracy, and ability to direct patients to seek care, as well as the possible impact on health service use.

Comparison of diagnostic and triage accuracy of Ada Health and WebMD symptom checkers, ChatGPT, and physicians for patients in an emergency department: Clinical data analysis study.

Citation: Fraser H, Crossland D, Bacher I, Ranney M, Madsen T, Hilliard R. Comparison of diagnostic and triage accuracy of Ada Health and WebMD symptom checkers, ChatGPT, and physicians for patients in an emergency department: Clinical data analysis study. JMIR Mhealth Uhealth. 2023 Oct 3;11:e49995. doi: 10.2196/49995. PMID: 37788063; PMCID: PMC10582809.

Link: https://pubmed.ncbi.nlm.nih.gov/37788063/

Principal Investigator: Fraser, Hamish S.F.

Project Name: Evaluating Diagnostic Decision Support Systems for Patients Requiring Urgent Primary or Emergency Care or with Stroke

Document Type: Journal Publication

Research Method: Quantitative, Retrospective

Technology: Mobile App

Studying the Accuracy of Symptom-Checker App in Diagnosing Strokes in a Real-World Setting

Theme:

Engaging and Empowering Patients

Subtheme:

Technology Solutions to Engage Patients and Their Families in Care

Rigorously evaluating a popular, patient-facing symptom-checker app will improve understanding around app safety, accuracy, and ability to prompt patients to seek care.

Millions use healthcare apps for diagnosis, but little research proves their accuracy in practice

Many Americans rely on healthcare apps like WebMD or SymptomMD to identify initial diagnoses of their symptoms. While this technology increases our access to healthcare information, these symptom-checker apps are unregulated and offer no safeguards or assurances that diagnostic information is accurate. Misdiagnosis of serious conditions can lead to poor medical outcomes or even death, especially if patients stay home rather than seek emergency care.

These symptom-checker apps have not been rigorously evaluated; past studies reviewed symptom-checker performance against common and typical disease presentations compiled by clinicians rather than directly from patient input. These descriptions are edited for clarity by clinicians but may not accurately reflect how patients describe their symptoms.

Symptom-checking apps will be tested in a real-world setting

Dr. Hamish Fraser, an associate professor of medical science and health systems policy and practice at Brown University’s Center for Biomedical Informatics, wants to address this gap in the research. He and a team are studying how accurately the popular symptom-checker app Ada Health can diagnose urgent or emergency complaints including stroke, based on patient input in real-world settings.

The researchers chose to focus on a stroke diagnosis because the symptoms often are confusing and timing is critical for seeking treatment—the golden window for effective stroke treatment is within 4 hours after it begins. A misdiagnosis that delays treatment can be catastrophic.

“It’s absolutely critical that we use (the apps) in real patients in real-world situations, exactly as the real world operates, because the situation can be very, very different from a lab test.” – Dr. Hamish Fraser

To study the performance of Ada Health’s symptom-checker app, the team is comparing app derived results among patients in the emergency department (ED) or at an urgent primary care visit against actual diagnoses.

Real patients, including those with a possible stroke, will enter into the app the symptoms for which they are seeking treatment to attain the app’s “diagnosis” and treatment recommendations. The patient-reported information also will be evaluated by physicians who are not treating the patient to establish a third-party clinical diagnosis. These two diagnoses and triage recommendations will be compared with the actual diagnoses and triage outputs provided by the treating clinicians. The clinical data collected in the study will also be used to test the WebMD symptom checker and different versions of the ChatGPT language models.

Using machine learning may improve accuracy in assessing symptoms

In addition, Dr. Fraser and the research team will couple data from 2,300 patients who have presented with symptoms of stroke at the ED with machine learning techniques to create new algorithms to improve early stroke diagnosis and risk stratification. Outputs will be compared with existing symptom checkers, algorithms, and guidelines on diagnosis and management of stroke.

These research results will improve understanding of symptom-checker safety, helping patients, particularly high-risk ones, to better recognize urgent symptoms and seek care, as well as enhance health service use for a broad range of patients, including those with stroke. The results also could have broader policy implications, including better regulation of new large language model apps and other technological tools. It’s important for researchers to measure the impact of these apps on health to provide guidance to policymakers.

This Research is Featured in