Developing and Using Valid Clinical Quality Metrics for Health Information Technology (Health IT) with Health Information Exchange (HIE) - 2011

Summary: Traditional metrics for measuring quality of care in ambulatory settings have been largely designed to measure ambulatory care in isolation, independent of interactions with other health care providers and settings. Innovations in health care driven by the implementation of health information technology (IT) with health information exchange (HIE) require revised sets of quality metrics to assess the impact these interventions promise. For example, new metrics are needed to capture the effects of data sharing between generalists and specialists in the ambulatory setting and of sharing data across transitions between inpatient and outpatient settings. Further, new quality metrics are needed to capitalize on the rich clinical data that could be extracted from electronic health records (EHRs) and other electronic sources.

This project developed a set of quality metrics, including existing and newly-developed metrics, that could potentially capture the effects of health IT with HIE and be retrieved electronically. This process was accomplished through the contributions of the Health Information Technology Evaluation Collaborative, a multi-institutional academic collaborative established to evaluate health IT and HIE initiatives in New York State, with additional input from the New York State Department of Health and four regional health information organizations that are implementing health IT with HIE in the ambulatory setting. The quality metric set was validated by a national expert panel with expertise in health IT, quality measurement, health care policy, and health economics.

Dr. Kaushal's team tested the accuracy of electronic retrieval of the data for the metric set compared to the standard manual chart review. This work was done in collaboration with a network of federallyqualified health centers. The metric set was then used to evaluate the effects on quality of using health IT with HIE, specifically EHRs and electronic portals. To do so, the team prospectively followed all eligible patients from selected federally-qualified health centers over 1 year to determine if quality improves using health IT with HIE.

Specific Aims:

• Develop a modified set of quality metrics that can be retrieved electronically and is sensitive to the types of improvements in quality that health IT with HIE may contribute in an ambulatory care setting. (Achieved)
• Validate the modified quality metric set. (Achieved)
• Test the reliability of electronic retrieval of the modified quality metric set. (Achieved)
• Use the modified quality metric set to evaluate the long-term effects of using health IT with HIE on improving health care quality. (Achieved)

2011 Activities: In the first years of the project, existing metrics were iteratively rated and refined over time and then validated by an expert panel. Next, the reliability of the metrics was assessed by comparing electronic reporting to manual review. Performance on the metrics over time was also measured. Two no-cost extensions, totaling a year-and-a-half, (1-year and 6-months) were necessary to conclude testing of the electronic reporting and quality improvement. Additionally in 2011, the team focused on manuscript development and the dissemination of study results. As of March 2011, this project has been completed.

Impact and Findings: Seventeen metric sets for measuring ambulatory care quality were identified through a literature review. The metric sets contained a total of 1,064 individual metrics. Of these, the team excluded 122 duplicates; 84 metrics not relevant to the ambulatory care setting; 136 not relevant to adult primary care; 189 consisting of provider, practice, or health plan characteristics; and 23 on patient or provider satisfaction. The remaining 510 metrics underwent a rating process, in which the scores from raters were averaged to create a summary score. A 36-member national expert panel was convened to validate the final metric set. The metrics were assessed according to feasibility of delivering data electronically to the physician at the point of care, potential impact on medical decision making, clinical importance, feasibility of reporting data electronically, and a global rating. The final metric set included 18 selected from metrics already endorsed by national organizations and 14 de novo metrics to address targeted care coordination more explicitly than the existing metrics. The process of developing and validating the metrics raised five issues that are highly relevant to the current national discussion on EHRs and quality: 1) data structure; 2) EHR usability and workflow; 3) community integration; 4) vendor maturity and priorities; and 5) quality metric specification.

The EHR was then evaluated for its use as an electronic documentation and reporting tool as well as for its potential to improve care over time. Twelve metrics were electronically obtained and manually extracted from the EHR. Using the manually extracted data as the standard, the reliability of electronic reporting was high overall. However, there was substantial variation in accuracy across the metrics. Quality improved significantly over time.

The development and validation of this metric set predated and informed the measures for the Centers for Medicare and Medicaid Services EHR Meaningful Use. Of the 18 existing metrics, 15 are included in Stage 1 Meaningful Use in identical or similar forms. None of the metrics developed de novo are reflected in Stage 1 Meaningful Use, in part because they are novel metrics that do not yet have accompanying specifications. The existing metrics in the study were aligned with and supported the conceptual basis of Meaningful Use. The metrics developed for this study could not be easily reported by most vendor EHRs. This observation highlights larger policy ramifications as community providers strive to demonstrate Meaningful Use.

Target Population: General

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management. 

Business Goal: Knowledge Creation