Evaluating the Impact of an ACPOE/CDS System on Outcomes (Washington)

Project Final Report (PDF, 806.9 KB) Disclaimer

Evaluating the Impact of an ACPOE/CDS System on Outcomes - 2009

Summary Highlights

  • Principal Investigator: 
  • Funding Mechanism: 
    RFA: HS04-011: Transforming Health Care Quality through Information Technology (THQIT) – Implementation Grants
  • Grant Number: 
    UC1 HS 015319
  • Project Period: 
    09/04 – 09/07
  • AHRQ Funding Amount: 
  • PDF Version: 
    (PDF, 63.69 KB)

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Implementation and Use

Summary: Computerized provider order entry (CPOE) has emerged as a viable option for reducing errors while increasing efficiency in clinical settings; however, these systems have not yet been perfected, and most research evaluating these systems has been conducted in the inpatient setting—primarily in academic medical centers. The majority of health care in the United States is delivered in community-based ambulatory settings, where the use of ambulatory (A)CPOE systems is low and research questions about how these systems impact medication safety, their time-intensity, and the perceptions and attitudes of clinicians and staff using them remain largely unexplored.

The project implemented an ACPOE system for medications (called the electronic [e]-prescribing system) with basic clinical decision support (CDS) alerts and captured lessons learned. It evaluated the time-intensity of e-prescribing, and the human factors associated therewith, specifically, the perceptions, experiences and attitudes of clinicians and staff toward e-prescribing. It also evaluated the impact of the system on medication errors and downstream preventable adverse drug events (ADEs).

The e-prescribing system was implemented at all 16 sites (60 clinics) of The Everett Clinic – the largest independent medical group in Washington Sate. The Everett Clinic is a vertically integrated, multidisciplinary physician group practice that provides comprehensive, community-wide health care to 275,000 patients in the north Puget Sound area.

The time-intensity of e-prescribing was measured in a direct observation time-motion study. The experiences with and perceptions of clinicians and staff using the system were captured by conducting focus groups. The attitudes of these same users were captured via a survey instrument based on the theoretical model called the “Information Technology Adoption Model”. The impact of the e-prescribing system on medication errors and preventable ADEs was estimated by evaluating 5,000 prescriptions before and 5,000 prescriptions after e-prescribing implementation. The characteristics and severity of medication errors were described, and the relationship between these and downstream ADEs was estimated. The time-motion study, focus groups, and survey administration took place at three primary care clinics; the medication safety evaluation was conducted using prescriptions written in all clinics.

Specific Aims

  • Implement the e-prescribing system in all practice sites within the integrated health care delivery system, documenting lessons learned and strategies used that enabled successful implementation. (Achieved)
  • Evaluate the time-intensity of e-prescribing on clinicians and staff. (Achieved)
  • Evaluate the human factors aspects of implementation by conducting focus groups to capture information about the experiences and perceptions of clinicians and staff with the e-prescribing system; and by administering a survey instrument to these same users, capture information about their attitudes toward e-prescribing implementation. (Achieved)
  • Evaluate the impact of the e-prescribing system on medication safety, specifically, medication errors and associated ADEs. (Achieved)

Impact and Findings:

Lessons Learned

Implementation steps, strategies used, and lessons learned that enabled successful adoption included staged roll-out, adequate technical support, one-on-one training, and just-in-time training. Clinician involvement contributed to iterative improvement. Chances of successful implementation increase with visionary, stable, and supportive clinic leadership. Workflow redesign was recognized as an essential component of implementation and was undertaken as a part of the process.

Time-Intensity of e-Prescribing

During an interim stage of hardware implementation (prescribers using laptop computers), prescribers at the e-prescribing sites spent significantly less time on writing tasks (-3.0 minutes/hour), but this time savings was offset by increased computer tasks (3.9 minutes/hour). After adjusting for site, prescriber, and prescription type, e-prescribing tasks took marginally longer than hand-written tasks (12 seconds). Nurses spent 5.4 minutes longer per hour performing computer tasks than their counterparts at the paper-based site; however, when computing and writing tasks were combined, the difference between sites was not statistically different. At all three sites, nurses spent only 1.1 minutes/hour on prescription-related tasks. At the e-prescribing sites, medical assistants spent a non-significantly greater amount of time conducting computer-related tasks (3.4 minutes/hour), and conducting prescription-related tasks (0.6 minutes/hour). The estimates of prescriber time were re-calculated after the permanent hardware configuration had been adopted – e-prescribing using a desktop computer in the examination room. These estimates revealed that e-prescribing using a computer in an examination room takes 69 seconds, 25 seconds longer than to hand-write and 24 seconds longer than to e-prescribe from a prescriber’s personal office. This calculates to 20 additional seconds per patient and 6 additional minutes per provider, per day. These findings revealed that hardware configurations have a significant impact on workflow.

Human Factors

From the focus group analysis, ten themes emerged that describe perceptions of e-prescribing implementation: 1) improved availability of clinical information resulted in prescribing efficiencies and more coordinated care; 2) improved documentation resulted in safer care; 3) efficiencies were gained by using fewer paper charts; 4) organizational support facilitated adoption; 5) transition required time and resulted in workload shift to staff; 6) hardware configurations and network stability were important in facilitating workflow; 7) e-prescribing was time-neutral or time-saving; 8) changes in patient interactions enhanced patient care but required education; 9) pharmacy communications were enhanced but required education; 10) positive attitudes facilitated adoption.

Results of the survey work revealed improvements in scores on a 5-point Likert scale, when comparing before to after e-prescribing implementation in the domains of "intent to use technology" and for "perceived usefulness". For prescribers, significant associations were found between computer use at home for professional use and each domain score; and between computer knowledge and three of four domains.

Medication Errors

Results of the medication error and ADE study revealed that the frequency of errors declined from 18 percent to 8 percent comparing handwritten to e-prescriptions, a reduction in adjusted odds of 70 percent (OR: 0.3; 95% CI 0.23 to 0.40). The largest reductions were seen in adjusted odds of errors of illegibility (97 percent), use of inappropriate abbreviations (94 percent), and missing information (85 percent). There was a 57 percent reduction in adjusted odds of errors that did not cause harm (potential ADEs) (OR 0.43; 95% CI 0.38 to 0.49). The reduction in the number of errors that caused harm (preventable ADEs) was not statistically significant, perhaps due to few errors in this category.


This work describes the successful implementation and impact of an ACPOE system in an independent medical group. ACPOE use was associated with a reduction in medication errors and potential ADEs, thereby contributing to improved medication safety. Use of the system was largely time neutral for prescribers and staff. Perceptions of users were largely positive, and this contributed to successful adoption. Attitudes toward adoption improved as implementation progressed.

The effectiveness of electronic health records, CPOE, and e-prescribing systems is critically dependent upon the interrelationships between humans, the tools they use, and the environment in which they live and work (i.e., the human factors aspects). Many factors influence the use of CPOE systems, including personality, prior computer experience, attitudes, interest, and enthusiasm. The results of the study provide evidence that implementing an e-prescribing system in a community-based, ambulatory setting can be achieved successfully and can have a positive impact on clinic efficiency and on medication safety.

Selected Outputs

Devine EB, Hollingworth W, Hansen RN, et al. Electronic prescribing at the point of care: A time-motion study in the primary care setting. Health Serv Res 2010 Feb;45(1):152-71.

Devine EB, Hansen RN, Wilson-Norton JL, et al. The impact of computerized provider order entry on medication errors in a multispecialty group practice. J Am Med Inform Assoc 2010;17:78-84.

Devine EB, Wilson-Norton JL, Lawless NM, et al. Implementing an ambulatory e-prescribing system: Strategies employed and lessons learned to minimize unintended consequences. In: Henriksen K, Battles JB, Keyes MA, Grady ML, editors. Advances in patient safety: New directions and alternative approaches. Vol. 4. Technology and Medication Safety. AHRQ Publication No. 08-0034-4. Rockville, MD: Agency for Healthcare Research and Quality; August 2008.

Hollingworth W, Devine EB, Hansen RN, et al. The impact of e-prescribing on prescriber and staff time in ambulatory care clinics: A time-motion study. J Am Med Inform Assoc. 2007:14:722-30.

Devine EB, Wilson-Norton JL, Lawless NM, et al. Characterization of prescribing errors in an internal medicine clinic. Am J Health-Syst Pharm. 2007; 64:1062-70.

Devine EB, Wilson-Norton JL, Lawless NM, et al. Preparing for ambulatory computerized prescriber order entry by evaluating pre-implementation medication errors. In: Henriksen K, Battles JB, Marks ES, et al., eds. Advances in patient safety: from research to implementation. Vol. 2, Concepts and methodology. Rockville (MD): Agency for Healthcare Research and Quality; Feb 2005. AHRQ Publication No. 05-0021-2.

Grantee’s Most Recent Self-Reported Quarterly Status (as of October 2010): This project is complete. The project team has published multiple manuscripts and a revision of the eighth and final manuscript is currently under review. They have also presented in numerous venues, locally, regionally and nationally, including sharing their work at four Annual AHRQ Meetings (2005, 2006, 2007 and 2009), creating a podcast with AHRQ, and providing a copy of our survey instrument, Information Technology in Primary Care Practice, to the AHRQ National Resource Center for Health Information Technology.

Milestones: Progress is completely on track.

Budget: Spending roughly on target.

Evaluating the Impact of an ACPOE/CDS System on Outcomes - Final Report

Sullivan S. Evaluating the Impact of an ACPOE/CDS System on Outcomes - Final Report. (Prepared by University of Washington under Grant No. UC1 HS015319). Rockville, MD: Agency for Healthcare Research and Quality, 2008. (PDF, 806.9 KB)

The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
Principal Investigator: 
Document Type: 
This project does not have any related event.
This project does not have any related resource.

Information Technology in Primary Care Practice

This is a questionnaire designed to be completed by nurses and physicians in an ambulatory setting. The tool includes questions to assess user's perceptions of electronic health records.

Year of Survey: 
Survey Link: 
Information Technology in Primary Care Practice (PDF, 82.65 KB) (Persons using assistive technology may not be able to fully access information in this report. For assistance, please contact Corey Mackison)
Document Type: 
Research Method: 
Care Setting: 
Copyright Status: 
Permission has been obtained from the survey developers for unrestricted use of this survey; it may be modified or used as is without additional permission from the authors.
This project does not have any related project spotlight.
This project does not have any related survey.

Washington's Everett Clinic: E-Prescribing in a Fast-Changing Health IT Environment

Sean Sullivan, Ph.D.

When it comes to implementing an e-prescribingsystem, project leaders at the Everett Clinic in Washington state have some advice:  Pay close attention to the people who will use your system. Start simple and slow, be prepared to change gears if it's the right thing to do, and be mindful of how the new technology affects the ways in which people do their work.

That advice comes from experience.

The Everett Clinic is a community-based, physician-owned integrated health system in the north Puget Sound area. With 16 locations, 250 physicians in 40 diverse specialties, and 1,300 staff members, the Everett Clinic serves about 225,000 patients, who make approximately 700,000 visits per year.

The Everett Clinic is serious about health information technology (IT): The clinic owns its own IT subsidiary and began developing it own electronic medical record (EMR) system in 1995. Its physician-owners decided to tackle e-prescribing in 2003. To date, the Everett Clinic has implemented its own e-prescribing system among all of its ambulatory care centers.

E-prescribing is widely seen as a way to reduce medication errors. To evaluate the impact of e-prescribing on medication safety, the Everett Clinic established a partnership with investigators from the University of Washington Pharmaceutical Research and Policy Program. With support from the Agency for Healthcare Research and Quality (AHRQ), project leaders are currently assessing the impact of that implementation on medication safety and staff workflow, while documenting important lessons learned from the project. In addition, the clinic plans to add a clinical decision support component next fall, following a switch from the clinic's home-grown EMR system to the commercial EPIC system.

But even in the midst of all this continuing activity, project leaders say they have learned some important lessons, including the following:

  • An iterative approach is crucial. "Don't hit the physicians with everything atonce," says Jennifer Wilson-Norton, R.Ph., M.B.A., director of pharmacy at the Everett Clinic and implementation leader for the project.
  • "Have a dedicated implementation team and a designated evaluation team. That way everyone contributes from their field of expertise," say Beth Devine, Pharm.D., M.B.A., and Sean Sullivan, Ph.D., faculty at the University of Washington and lead evaluators for the project.
  • Prepare your users well for implementation and provide them with plenty of technical support. Everett's pharmacists made a point of being available for training, in groups and one on one, and for trouble-shooting and "just-in-time" support.
  • Be flexible. "You can't be locked in to a technology if it doesn't work, or if you see a better alternative," Wilson-Norton says.
  • Remember to be realistic about your time frame. "You still have to keep your core business going," Wilson-Norton points out. "You have to be careful how you balance urgency, because everything is urgent."

Wilson-Norton acknowledges that the mission to transform the clinic has gone through many phases, starting with "building our own system and now transitioning to a commercial-based system. This is part of being an engaged organization."

For the e-prescribing component, Wilson-Norton notes that project leaders decided to begin with "a very basic" implementation that focused on refill orders to make the adjustment easier for physicians and other staff users.

There's a lot to get used to. For example, physicians have to be more precise in the e-prescribing world than they were in the paper world, where misspelled drug names are often corrected by a pharmacist. In the e-prescribing world, misspelled drug names simply aren't recognized. 

The implementation team has been making adjustments in workflow as well. As the team began the process of installing computers in exam rooms, the issue of security versus ease of access arose. To expect busy doctors to manually log on and log off every time they entered and left an exam room was unrealistic. "That's just not a sustainable business model," Wilson-Norton says.  But the terminals had to be secured. The project team came up with a solution in the form of a card that recognizes and authorizes individual users, who have different levels of access to information. This both ensures security and permits the right individuals to access the right information.

The EPIC implementation this fall will bring another wave of change. Wilson-Norton says that the clinic's home-grown EMR system would be more expensive to maintain and more difficult to update and enhance in the long run; the switch to EPIC will improve care and be more cost-effective in the long run. That was not the case when the clinic decided to develop its own system years ago, but since then technology prices have come down, making a commercial system much more affordable.

But a new EMR system also means a new e-prescribing system -- EPIC's. Wilson-Norton says that some aspects of Everett's current e-prescribing system are more user-friendly. For example, Everett's system gives physicians a drop-down menu with a range of packaging options for a specific drug. But when using EPIC's e-prescribing system, doctors will have to enter the correct packaging size themselves.

But with EPIC in place, the clinic will be able to move on from a basic e-prescribing system to one that includes the more advanced features of clinical decision support. "Therein lies greater potential to prevent errors and improve medication safety," says Devine.  She explains that the project team will probably take a staggered approach to clinical decision support implementation -- again, to ease the transition and to avoid "alert fatigue," which happens when physicians who are flooded withcomputer alerts choose to ignore them.

As part of the AHRQ project, evaluators at Everett and University of Washington are also assessing the impact of e-prescribing technology on workflow through a time-motion study. Here again, the project team had to adapt when confronted with an unforeseen twist. The original plan called for laptop computers with wireless communications for every physician. But it turned out that the wireless communications technology was not reliable enough, so the laptops were abandoned in favor of the exam room desktop computers.

Although the time-motion study is still underway, the project team already has made some important workflow modifications. For example, in the newly outfitted exam rooms, a medical assistant takes each patient's vital signs and enters the data into the computer so that all the information is available when the doctor comes in to see the patient. According to Wilson-Norton, nurses now spend more of their time in front of the computer, communicating with doctors electronically. 

This project does not have any related emerging lesson.

Project Details - Ended


The project consisted of two aims: 1) Implement an ambulatory, computerized prescriber order entry (ACPOE, e-prescribing) system with basic clinical decision support (CDS) alerts, capture lessons learned, and evaluate workflow/ workload and human factors impacts on prescribers and staff; and 2) Evaluate the impact of the system on medication safety - medication errors and adverse drug events. The Everett Clinic is a community-based, multispecialty health system. The e-prescribing system was implemented at all sixteen sites (60 clinics). Comparing pre- to post-implementation, the workflow/workload and human factors evaluations took place at three primary care clinics; the medication safety evaluation at all clinics. Roll-out was staggered. Throughout implementation, iterative improvements were made and lessons learned were captured. Workload/workflow was evaluated by a time-motion study; human factors impacts were captured from focus groups and a survey instrument. Ten thousand prescriptions were evaluated to identify and characterize prescribing-related errors. All clinics are now using the e-prescribing system. Staggered roll-out, iterative improvements, individual training, and real-time availability of technical assistance enabled successful adoption. Use of the system was time-neutral for prescribers. End-user feedback was positive. E-prescribing resulted in a reduction of (potential) medication errors from 28 to 9 percent.