Evaluating the Impact of an ACPOE/CDS System on Outcomes - 2009

Principal Investigator
Funding Mechanism
RFA: HS04-011: Transforming Health Care Quality through Information Technology (THQIT) – Implementation Grants
Grant Number
UC1 HS 015319
Project Period
09/04 – 09/07
AHRQ Funding Amount
Summary Status as of
September 2007, Conclusion of Grant

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Implementation and Use

Summary: Computerized provider order entry (CPOE) has emerged as a viable option for reducing errors while increasing efficiency in clinical settings; however, these systems have not yet been perfected, and most research evaluating these systems has been conducted in the inpatient setting—primarily in academic medical centers. The majority of health care in the United States is delivered in community-based ambulatory settings, where the use of ambulatory (A)CPOE systems is low and research questions about how these systems impact medication safety, their time-intensity, and the perceptions and attitudes of clinicians and staff using them remain largely unexplored.

The project implemented an ACPOE system for medications (called the electronic [e]-prescribing system) with basic clinical decision support (CDS) alerts and captured lessons learned. It evaluated the time-intensity of e-prescribing, and the human factors associated therewith, specifically, the perceptions, experiences and attitudes of clinicians and staff toward e-prescribing. It also evaluated the impact of the system on medication errors and downstream preventable adverse drug events (ADEs).

The e-prescribing system was implemented at all 16 sites (60 clinics) of The Everett Clinic – the largest independent medical group in Washington Sate. The Everett Clinic is a vertically integrated, multidisciplinary physician group practice that provides comprehensive, community-wide health care to 275,000 patients in the north Puget Sound area.

The time-intensity of e-prescribing was measured in a direct observation time-motion study. The experiences with and perceptions of clinicians and staff using the system were captured by conducting focus groups. The attitudes of these same users were captured via a survey instrument based on the theoretical model called the “Information Technology Adoption Model”. The impact of the e-prescribing system on medication errors and preventable ADEs was estimated by evaluating 5,000 prescriptions before and 5,000 prescriptions after e-prescribing implementation. The characteristics and severity of medication errors were described, and the relationship between these and downstream ADEs was estimated. The time-motion study, focus groups, and survey administration took place at three primary care clinics; the medication safety evaluation was conducted using prescriptions written in all clinics.

Specific Aims

  • Implement the e-prescribing system in all practice sites within the integrated health care delivery system, documenting lessons learned and strategies used that enabled successful implementation. (Achieved)
  • Evaluate the time-intensity of e-prescribing on clinicians and staff. (Achieved)
  • Evaluate the human factors aspects of implementation by conducting focus groups to capture information about the experiences and perceptions of clinicians and staff with the e-prescribing system; and by administering a survey instrument to these same users, capture information about their attitudes toward e-prescribing implementation. (Achieved)
  • Evaluate the impact of the e-prescribing system on medication safety, specifically, medication errors and associated ADEs. (Achieved)

Impact and Findings:

Lessons Learned

Implementation steps, strategies used, and lessons learned that enabled successful adoption included staged roll-out, adequate technical support, one-on-one training, and just-in-time training. Clinician involvement contributed to iterative improvement. Chances of successful implementation increase with visionary, stable, and supportive clinic leadership. Workflow redesign was recognized as an essential component of implementation and was undertaken as a part of the process.

Time-Intensity of e-Prescribing

During an interim stage of hardware implementation (prescribers using laptop computers), prescribers at the e-prescribing sites spent significantly less time on writing tasks (-3.0 minutes/hour), but this time savings was offset by increased computer tasks (3.9 minutes/hour). After adjusting for site, prescriber, and prescription type, e-prescribing tasks took marginally longer than hand-written tasks (12 seconds). Nurses spent 5.4 minutes longer per hour performing computer tasks than their counterparts at the paper-based site; however, when computing and writing tasks were combined, the difference between sites was not statistically different. At all three sites, nurses spent only 1.1 minutes/hour on prescription-related tasks. At the e-prescribing sites, medical assistants spent a non-significantly greater amount of time conducting computer-related tasks (3.4 minutes/hour), and conducting prescription-related tasks (0.6 minutes/hour). The estimates of prescriber time were re-calculated after the permanent hardware configuration had been adopted – e-prescribing using a desktop computer in the examination room. These estimates revealed that e-prescribing using a computer in an examination room takes 69 seconds, 25 seconds longer than to hand-write and 24 seconds longer than to e-prescribe from a prescriber’s personal office. This calculates to 20 additional seconds per patient and 6 additional minutes per provider, per day. These findings revealed that hardware configurations have a significant impact on workflow.

Human Factors

From the focus group analysis, ten themes emerged that describe perceptions of e-prescribing implementation: 1) improved availability of clinical information resulted in prescribing efficiencies and more coordinated care; 2) improved documentation resulted in safer care; 3) efficiencies were gained by using fewer paper charts; 4) organizational support facilitated adoption; 5) transition required time and resulted in workload shift to staff; 6) hardware configurations and network stability were important in facilitating workflow; 7) e-prescribing was time-neutral or time-saving; 8) changes in patient interactions enhanced patient care but required education; 9) pharmacy communications were enhanced but required education; 10) positive attitudes facilitated adoption.

Results of the survey work revealed improvements in scores on a 5-point Likert scale, when comparing before to after e-prescribing implementation in the domains of "intent to use technology" and for "perceived usefulness". For prescribers, significant associations were found between computer use at home for professional use and each domain score; and between computer knowledge and three of four domains.

Medication Errors

Results of the medication error and ADE study revealed that the frequency of errors declined from 18 percent to 8 percent comparing handwritten to e-prescriptions, a reduction in adjusted odds of 70 percent (OR: 0.3; 95% CI 0.23 to 0.40). The largest reductions were seen in adjusted odds of errors of illegibility (97 percent), use of inappropriate abbreviations (94 percent), and missing information (85 percent). There was a 57 percent reduction in adjusted odds of errors that did not cause harm (potential ADEs) (OR 0.43; 95% CI 0.38 to 0.49). The reduction in the number of errors that caused harm (preventable ADEs) was not statistically significant, perhaps due to few errors in this category.


This work describes the successful implementation and impact of an ACPOE system in an independent medical group. ACPOE use was associated with a reduction in medication errors and potential ADEs, thereby contributing to improved medication safety. Use of the system was largely time neutral for prescribers and staff. Perceptions of users were largely positive, and this contributed to successful adoption. Attitudes toward adoption improved as implementation progressed.

The effectiveness of electronic health records, CPOE, and e-prescribing systems is critically dependent upon the interrelationships between humans, the tools they use, and the environment in which they live and work (i.e., the human factors aspects). Many factors influence the use of CPOE systems, including personality, prior computer experience, attitudes, interest, and enthusiasm. The results of the study provide evidence that implementing an e-prescribing system in a community-based, ambulatory setting can be achieved successfully and can have a positive impact on clinic efficiency and on medication safety.

Selected Outputs

Devine EB, Hollingworth W, Hansen RN, et al. Electronic prescribing at the point of care: A time-motion study in the primary care setting. Health Serv Res 2010 Feb;45(1):152-71.

Devine EB, Hansen RN, Wilson-Norton JL, et al. The impact of computerized provider order entry on medication errors in a multispecialty group practice. J Am Med Inform Assoc 2010;17:78-84.

Devine EB, Wilson-Norton JL, Lawless NM, et al. Implementing an ambulatory e-prescribing system: Strategies employed and lessons learned to minimize unintended consequences. In: Henriksen K, Battles JB, Keyes MA, Grady ML, editors. Advances in patient safety: New directions and alternative approaches. Vol. 4. Technology and Medication Safety. AHRQ Publication No. 08-0034-4. Rockville, MD: Agency for Healthcare Research and Quality; August 2008.

Hollingworth W, Devine EB, Hansen RN, et al. The impact of e-prescribing on prescriber and staff time in ambulatory care clinics: A time-motion study. J Am Med Inform Assoc. 2007:14:722-30.

Devine EB, Wilson-Norton JL, Lawless NM, et al. Characterization of prescribing errors in an internal medicine clinic. Am J Health-Syst Pharm. 2007; 64:1062-70.

Devine EB, Wilson-Norton JL, Lawless NM, et al. Preparing for ambulatory computerized prescriber order entry by evaluating pre-implementation medication errors. In: Henriksen K, Battles JB, Marks ES, et al., eds. Advances in patient safety: from research to implementation. Vol. 2, Concepts and methodology. Rockville (MD): Agency for Healthcare Research and Quality; Feb 2005. AHRQ Publication No. 05-0021-2.

Grantee’s Most Recent Self-Reported Quarterly Status (as of October 2010): This project is complete. The project team has published multiple manuscripts and a revision of the eighth and final manuscript is currently under review. They have also presented in numerous venues, locally, regionally and nationally, including sharing their work at four Annual AHRQ Meetings (2005, 2006, 2007 and 2009), creating a podcast with AHRQ, and providing a copy of our survey instrument, Information Technology in Primary Care Practice, to the AHRQ National Resource Center for Health Information Technology.

Milestones: Progress is completely on track.

Budget: Spending roughly on target.