Evaluating Electronic Health Record Data for Use in Diabetes Quality Reporting - 2012

Principal Investigator
Funding Mechanism
PAR: HS09-212: AHRQ Grants for Health Services Research Dissertation Program (R36)
Grant Number
R36 HS 020165
Project Period
March 2011 – June 2012
AHRQ Funding Amount

Summary: The Centers for Medicare and Medicaid Services (CMS) has started to permit the use of electronic health record (EHR) data for reporting quality measures for performance measurement. While research exists on the challenges of using claims and registry data for various pay-for-performance programs, there is little guidance on how to use EHR data for quality reporting. Few studies have explored the validity of using EHR data to identify a target population, or examined how the use of EHR data can impact quality measures specifically. In addition, there is little information on the accuracy and consistency of EHR data, specifically on how physicians use the EHR to store information about patient diagnoses.

Despite this lack of information, pay-for-performance programs are expanding their use of EHR data, which may be a particular concern given that existing quality measures are often perceived by physicians to not measure quality accurately. Before using this data for quality reporting, it is critical to understand how physicians use the EHR and what motivates their choices of where and how they document diagnosis and treatment.

In this Health Services Research Dissertation Grant, Dr. Hirsch used a mixed-method approach to investigate the validity of using EHR data for diabetes performance measures. The first component consisted of one-on-one telephone interviews with primary care clinicians employed by Geisinger Health System, an integrated health care system in Pennsylvania that began implementing its EHR in outpatient clinics in 1996. The purpose of these key informant interviews was to gain an understanding of how clinicians enter data in the EHR when diagnosing and treating patients with diabetes and what may motivate their documentation behavior such as workflow or billing processes. The second component of the study consisted of an extraction of EHR data to identify primary care patients with diabetes using eight different EHR-based methods of identification. Dr. Hirsch assessed the validity of the methods by comparing them to the gold standard of a manual medical record review, and then determined whether the method of identifying patients with diabetes could impact performance measurement scores.

Specific Aims:

  • Evaluate the validity of different EHR criteria in identifying primary care patients with diabetes. (Achieved)
  • Determine the impact of utilizing different EHR criteria on existing quality measures for diabetes (Achieved)
  • Learn where in the EHR physicians document diagnoses of diabetes and identify motivators of documentation behaviors. (Achieved)

2012 Activities: The majority of the work for this project was completed in 2011. The focus of activity in 2012 was data analysis and dissertation development.

Impact and Findings: Dr. Hirsch identified four themes from the interviews with clinicians. First, clinicians identified two data fields – the problem list diagnosis field and the encounter diagnosis field – as the locations in the EHR where they most frequently document a diagnosis of diabetes. Second, clinicians endorsed the use of a problem list diagnosis for identifying patients with diabetes for quality measures, while they expressed concern that depending on an encounter, diagnosis could result in over- or under-identifying patients. Third, while clinicians were not specifically asked about organizational influences on their documentation behavior, it emerged that organizational factors have an influence on how diagnosis data are entered into the EHR including workflow, internal quality performance programs, and leadership pressure. Fourth, clinicians expressed concerns about the unintended consequences of using EHR data for performance measurement and quality reporting purposes including negative impact on care processes, insurance coverage issues, and unnecessary patient anxiety.

For the second component of the study, the EHR-based methods for identifying patients with diabetes had high specificity (greater than 99 percent) and moderate-to-high sensitivity (65 to 100 percent). The method of identifying patients with diabetes did not have an impact on the performance measures. However, the EHR criteria used in each of the definitions selectively identified patients who had better quality performance scores.

Target Population: Chronic Care*, Diabetes

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation

*This target population is one of AHRQ’s priority populations.