Bindels R et al. 2003 "The efficacy of an automated feedback system for general practitioners."

Bindels R, Hasman A, Kester AD, et al. The efficacy of an automated feedback system for general practitioners. Inform Prim Care 2003(11):69-74.
"OBJECTIVE: An automated feedback system that produces comments about the non-adherence of general practitioners (GPs) to accepted practice guidelines for ordering diagnostic tests was developed. Before implementing the automated feedback system in daily practice, we assessed the potential effect of the system on the test ordering behaviour of GPs. DESIGN: We used a randomised controlled trial with balanced block design. SETTING: Five times six participant groups of GPs in a computer laboratory setting. INTERVENTION: The GPs reviewed a random sample of 30 request forms they filled in earlier that year. If deemed necessary, they could make changes in the tests requested. Next, the system displayed critical comments about their non-adherence to the guidelines as apparent from the (updated) request forms. SUBJECTS: Twenty-four randomly selected GPs participated. MAIN OUTCOME MEASURES: The number of requested diagnostic tests (17% with 95% confidence interval [CI]: 12-22%) and the fraction of tests ordered that were not in accordance with the practice guidelines (39% with 95% CI: 28-51%) decreased due to the comments of the automated feedback system. The GPs accepted 362 (50%) of the 729 reminders. IMPLICATIONS: Although our experiment cannot predict the size of the actual effect of the automated feedback system in daily practice, the observed effect may be seen as the maximum achievable."

"To assess the efficacy of an automated feedback system, [where] efficacy is defined as the percentage of decisions made that are in line with relevant practice guidelines."

Type Clinic
Primary care
Small and/or medium
Other Information
The study took place in the Maastricht region of the Netherlands.
Type of Health IT
Decision support system
Type of Health IT Functions
The system displays "a recommendation ... [that] describes the discrepancies between the physician's actions and the guidelines, offering recommendations for improvement... The...system consists of five parts: a knowledge base, an order entry system, a module that provides reactive support (i.e. the recommendations), a module that provides passive support and a database. The knowledge base in which the recommendations are stored now contains 150 rules (recommendations) derived from accepted national and regional guidelines about various medical problems. To use the ... system in daily practice, the general practitioner (GP) must enter relevant medical patient data (signs, symptoms, working hypotheses, and the reason for request) and the tests to be ordered into an electronic order entry form ... Next, the reactive support module of the ... system reads the patient data on the electronic request form and checks whether any of the rules in the knowledge base will trigger. If a rule triggers, the corresponding recommendation is generated immediately and presented by means of a pop-up window,overlaying the interface of the electronic request form. The recommendation window contains critical comments about the requested tests as well as a link to the text of the practice guideline for more explanation (passive support module). Finally, the GP then decides to accept or reject the recommendation."
Workflow-Related Findings
"The mean number of test requests decreased due to the whole intervention. The mean percentage decrease over the two GP groups was 30%."
"There was a strong decrease in the proportion of tests not in accordance with the practice guidelines (43% over the two GP groups...for the total intervention)."
"Due to comments of the...system 457 tests were removed (leucocyte count 66 times, packed cell volume (PCV) 39 times and differential count 34 times) and 46 tests were added (fasting glucose 29 times and alanine aminotransferase (ALAT) 10 times). The comments of the...system resulted in a mean decrease in the number of tests ordered in the intervention groups of 17%."
"Of the 729 presented recommendations, 362 (50%) were accepted and 367 (50%) were ignored. The comments of the system resulted in a mean decrease in the proportion of tests not in accordance with guidelines in the intervention groups of 39%."
"There was a significant decrease in the proportion of inappropriately requested tests in the intervention group compared to the control groups (P<0.001)."
Use statistics showed that over half of the GPs (13/24) rarely followed the system recommendations and 11 of 24 made changes based on the advice of the system. "A possible explanation in that GPs in the Maastricht region have received feedback on their test ordering behaviour since 1985. Their test ordering behaviour is already largely in line with the guidelines."
Study Design
Randomized controlled trial (RCT)
Study Participants
Of the 90 GPs who were randomly selected, 24 agreed to participate in the study.