Friedman CP et al. 1999 "Enhancement of clinicians' diagnostic reasoning by computer-based consultation - a multisite study of 2 systems."

Reference
Friedman CP, Elstein AS, Wolf FM, et al. Enhancement of clinicians' diagnostic reasoning by computer-based consultation - a multisite study of 2 systems. JAMA 1999;282(19):1851-1856.
Abstract
"Context: Computer-based diagnostic decision support systems (DSSs) were developed to improve health care quality by providing accurate, useful, and timely diagnostic information to clinicians. However, most studies have emphasized the accuracy of the computer system alone, without placing clinicians in the role of direct users. Objective: To explore the extent to which consultations with DSSs improve clinicians' diagnostic hypotheses in a set of diagnostically challenging cases. Design: Partially randomized controlled trial conducted in a laboratory setting, using a prospective balanced experimental design in 1995-1998. Setting: Three academic medical centers, none of which were involved in the development of the DSSs. Participants: A total of 216 physicians: 72 at each site, including 24 internal medicine faculty members, 24 senior residents, and 24 fourth-year medical students. One physician's data were lost to analysis. Intervention: Two DSSs, ILIAD (version 4.2) and Quick Medical Reference (QMR; version 3.7.1), were used by participants for diagnostic evaluation of a total of 36 cases based on actual patients. After training, each subject evaluated 9 of the 36 cases, first without and then using a DSS, and suggested an ordered list of diagnostic hypotheses after each evaluation. Main Outcome Measure: Diagnostic accuracy, measured as the presence of the correct diagnosis on the hypothesis list and also using a derived diagnostic quality score, before and after consultation with the DSSs. Results: Correct diagnoses appeared in subjects' hypothesis lists for 39.5% of cases prior to consultation and 45.4% of cases after consultation. Subjects' mean diagnostic quality scores increased from 5.7 (95% confidence interval [CI], 5.5-5.9) to 6.1 (95% CI, 5.9-6.3) (effect size: Cohen d = 0.32; 95% CI, 0.23-0.4; P, .001). Larger increases (P = .048) were observed for students than for residents and faculty. Effect size varied significantly (P,.02) by DSS (Cohen d = 0.20; 95% CI, 0.08-0.32 for ILIAD vs. Cohen d = 0.45; 95% CI, 0.31-0.59 for QMR).
Conclusions: Our study supports the idea that "hands-on" use of diagnostic DSSs can influence diagnostic reasoning of clinicians. The larger effect for students suggests a possible educational role for these systems."
Objective
"To explore the extent to which consultations with [decision support systems] improve clinicians' diagnostic hypotheses in a set of diagnostically challenging cases."
Type Clinic
Primary care
Size
Large
Geography
Urban
Other Information
The study was located at the University of Illinois at Chicago, the University of Michigan at Ann Arbor, and the University of North Carolina at Chapel Hill.
Type of Health IT
Decision support system
Type of Health IT Functions
In the first type of DSS, "users enter clinical findings about a case using an interface that allows both free-text and menu-based entry. The system generates a rank-ordered list of diagnostic hypotheses, each with estimated probability. [The first DSS] can suggest next steps in a work-up that would clarify the differential. Users can also browse [the first DSS]'s representation of each disease. The version of [the first DSS] used in this study contained explicit representations of 920 diseases. [In the second DSS, d]isease representations ... are not statistical; relationships between findings and diseases are expressed on heuristic 5-point scales. The [second DSS] can be used in a case analysis mode to generate a ranked list of potential diagnoses for an entered set of case findings. The system offers several special functions, such as comparison and contrast of pairs of diseases, designed to help clinicians refine their diagnoses. The version of [the second DSS] used in this study contained explicit representations of 623 diseases."
Workflow-Related Findings
"A correct diagnosis appeared in subjects' hypothesis lists for 764 cases (39.5%) before DSS consultation, increasing to 879 cases (45.4%) after consultation... Positive consultations, where the correct diagnosis was present after consultation but not before, were observed for 232 cases (12.0%); negative consultations, where the correct diagnosis was present before consultation but not after, were observed in 117 cases (6.0%). The overall consultation effect (net gain) is 115 cases (5.9%)...The largest consultation effects were observed for the students, with smaller effects for residents and faculty. Larger consultation effects were observed in subjects using [the second DSS]."
Study Design
Pre-postintervention (no control group)
Study Participants
A total of 216 subjects participated. At each site, and for each of two selected DSSs, the authors selected 12 faculty physicians (general internists); 12 internal medicine residents; and 12 fourth-year medical students.