Developing and Using Valid Clinical Quality Metrics for Health Information Technology (Health IT) with Health Information Exchange (HIE)

This project sought to identify existing, and create new, clinical quality of care metrics able to be reliably retrieved from electronic systems, such as electronic health records (EHRs) and health information exchanges (HIEs). Current national metrics were not developed from the vantage that data could be retrieved electronically. The ability to do so has taken on new importance given the Centers for Medicare & Medicaid Service's Meaningful Use Initiatives. Metrics which can be derived from EHRs and HIEs must be identified and developed, and their validity established. If electronic reporting of quality measures is not valid and reliable, not only will incorrect data be reported, but financial incentives and penalties could be given to the wrong providers.

The main objectives of the project were to:

• Develop a modified set of quality metrics that can be retrieved electronically and is sensitive to the types of improvements in quality that health information technology with HIE may contribute in an ambulatory care setting.
• Validate the modified quality metric set.
• Test the reliability of electronic retrieval of the modified quality metric set.
• Use the modified quality metric set to evaluate the long-term effects of using health information technology with HIE on improving health care quality.

The team successfully developed a quality metric set which captures data from EHRs with HIEs. The final set contains 18 existing and 14 new metrics. A 36 member national expert panel validated the metric set. The metrics were then implemented in a federally qualified health center in New York and tested to determine the reliability of the metrics. Of the final set, 11, all of which are included in meaningful use, were assessed for reliability of electronic reporting.

With the exception of three metrics which were eventually excluded from analysis, the overall reliability of electronic reporting was high, although reliability across metrics was found to vary considerably. Those metrics which utilized laboratory results, such as for diabetes, had higher reliability than those that utilized medications, such as for asthma. This is reflective of the fact that laboratory tests are more standardized, especially in the context of HIEs; whereas medication information is found in several places in an EHR and is captured with both structured and unstructured (i.e. free text) formats. Text-based reports of testing, such as colorectal cancer screening, were found to have lower reliability.

In addition to demonstrating the reliability of retrieval of these metrics, the project team also noted that provider quality of care as demonstrated by these measures improved over time with use of an EHR.