Automating Assessment of Asthma Care Quality - 2010

Target Population: Adults, Asthma, Chronic Care*, Pediatric*, Teenagers

Summary: This project aimed to develop, validate, apply, and evaluate a scalable method for routine and comprehensive measurement of outpatient asthma care quality (ACQ). This project leveraged health information technology (IT) to assess and improve quality of care for the insured, indigent, uninsured, and underinsured populations of the Pacific Northwest. To accomplish this task, the project employed MediClass (a medical classifier), which is a proven natural language processing technology for extracting care quality data from both coded data and free-text clinical notes in the electronic medical record (EMR). The project performed retrospective analysis of EMR data from two distinct health systems: a mid-sized health maintenance organization, Kaiser Permanente Northwest (KPNW); and a consortium of federally qualified health center clinics for whom an EMR is delivered and managed by one member organization, Our Community Health Information Networks (OCHIN, Inc.). Data was extracted from Kaiser Permanente’s Certification Commission for Health Information Technology-certified Epic-based EMR, HealthConnect; and OCHIN’s Epic-based EMR, EpicCare. Since these EMR applications reside in separate health systems, the implementation of the products generated differences in the data that were accommodated when these data were interpreted for quality assessments. This project leveraged MediClass to implement methods for collecting and transforming data into common formats for quality assessment across multiple data capture, representation, and storage processes.

The starting study participants included patients older than 12 years who were identified to have asthma by a single visit diagnosis code. Subsequently, a modified Healthcare Effectiveness Data and Information Set method that included text processing of clinician notes was applied in order to qualify patients with persistent asthma for inclusion in assessments of care delivered. The study population was drawn from approximately 24 months of OCHIN data and about 120 months of KPNW data.

Most quality measures were associated with several unique target concepts identified by MediClass processing and validation of these “target concepts” was based on chart reviews at the encounter level. Validation of the ACQ measures was based on chart reviews at the patient level. Sensitivity, specificity, false positive and negative rates, and 95-percent confidence intervals (CIs) were computed for each measure based on the review of 900 patient charts. After the automated measurement method was refined and applied to the target population for each quality measure, the proportions and 95-percent CIs were computed for patients receiving the indicated care measures. In addition, patient-level summary ACQ scores were analyzed. These measures were reported on the entire target population and by age, severity, and health care system subgroups. The availability of several years of EMR data at KPNW enabled the project to evaluate the relationship of the automated ACQ measures with health outcomes. Logistic regression was employed to model the association between the automated ACQ summary score and the primary outcome measure.

Specific Aims:

• Refine ACQ measures from the RAND Quality Assessment Tools Project for use as a quality measure set to evaluate ambulatory asthma care performance. (Achieved)
• Develop and validate an automated (generalizable and scalable) method for applying the above care quality measures using comprehensive EMR data. (Achieved)
• Apply the automated method developed above to assess ambulatory ACQ in two distinct health plans representing diverse patient populations and care practices. (Achieved)
• Evaluate the association between automated measures of adherence to recommended asthma care processes and measures of clinical outcomes using KPNW data only. (Achieved)

2010 Activities: The process for development and refinement of the ACQ measure set was completed. Subsequently the study team obtained the necessary data components including access to medical records onsite at OCHIN to review medical records and the technology used to process medical records, both text and coded fields; and a limited data set from the EMR was shared with the KPNW Center for Health Research. The study team pulled and formulated data for outcomes in the immediate years following the end of the measurement period in KPNW. They also included a pull of these same data in the year prior to qualification for the ACQ measures to correct outcomes analysis for patient baseline severity. Data were analyzed and incorporated into the project final report.

An extensive chart review of stratified random samples of persistent asthma patients and exacerbation events at each site was completed. The chart review process involved a sample of roughly 450 patients at each site. Chart reviews were utilized to collect the data necessary to evaluate criteria for assessing performance on each quality measure. A 10 percent quality assurance sample was conducted to provide secondary review and to resolve discrepancies.

The measurement method utilized a three-year observation period beginning in 2001 at KPNW only. The study staff also pulled KPNW data for a second three-year observation period beginning in 2006. This second data pull provided an observation period that was contemporaneous with the OCHIN data used and allowed comparisons of measurements across the two health systems.

Grantee’s Most Recent Self-Reported Quarterly Status (as of September 2010): As the project came to a close, the primary focus was on the analysis of data and development of manuscripts. The project was able to meet all of its aims and project spending was roughly on target.

Impact and Findings: Most ACQ measures performed well in the KPNW system. Overall accuracy, as measured by chart review, ranged from 63 percent to 100 percent and averaged 88 percent across all measures. Sensitivity was 60 percent or greater for 16 of the 18 implemented measures. Fifteen measures had specificity of 60 percent or higher. There were two measures for which specificity was over 90 percent but which had poor sensitivity.

Overall, the automated ACQ measures did not perform as well in the OCHIN system as they did in the KPNW system. Mean overall accuracy was 85 percent and ranged from 72 to 99 percent. Among the 11 routine care measures, eight had specificities over 80 percent and five had sensitivities over 80 percent. Three measures had specificities of 50 percent or lower while another five had sensitivities of 50 percent or lower. Of the seven exacerbation-related measures, five were evaluable at OCHIN and had a mean overall accuracy of 70 percent with a range of 36 to 96 percent. Sensitivity tended to be relatively low, ranging from 5.3 to 58.1 percent, while specificities were generally high, with a minimum of 67 percent with four measures greater than 90 percent.

Potential explanations for the discrepancy in performance of the automated measures between OCHIN and KPNW include the possibility that chart reviewers may have had differential access to sections of the medical record retrieved by the automated method across sites, and that there may be much greater variability in how and where OCHIN providers document visits for many of the text-based measures. Additional effort may be needed in the specifications of the automated method to capture variations across sites.

Based on chart review results, delivery of routine non-exacerbation care is similar between the two sites. Of these measures, both organizations performed well (better than 90 percent) in providing prescriptions for beta-2 agonists and assuring that their persistent asthma patients were not taking non-cardioselective beta-blockers. Providers in both organizations preformed moderately well (60 to 80 percent) in providing anti-inflammatory controllers and querying patients about tobacco use. Flu vaccination was documented in about 40 percent of patients. The remaining measures were only present between 10 to 30 percent of the time.

Among the exacerbation-related care measures there was similar agreement across sites in the chart review results. Both organizations performed well on the review of current medications and the performance of a chest exam. Performance was poor on the remaining exacerbation-related measures.

The study staff also compared prevalences of the ACQ recommended care as determined by the automated method and by chart review. At KPNW they found generally good agreement between the two methods with a few exceptions. Among the exacerbation-related measures, chart reviewers found more chest exams and current medications reviews while the automated method found more evidence for 4-week followup contacts post hospital discharge. Data for the theophylline measure was too sparse to evaluate adequately. Only six measures were performed 80 percent or more of the time according to either the chart review or the automated method. These results confirm that there are significant gaps between recommended care and real world practice.

A similar analysis was performed comparing chart-review prevalences with those of the automated method at the OCHIN site for each care measure. Among the routine non-exacerbation care measures, chart reviewers found more evidence of flu vaccinations and metered dose inhaler instructions while the automated system detected more instances of smoking status queries and tobacco counseling or referral. Among the measurements of exacerbation-related care, prevalences were higher based on chart review for all measures except the theophylline lab measure, for which the data were to sparse to evaluate. Care gaps in the OCHIN system appear to be very similar to those in the KPNW system, with only five measures performing 80 percent or more of the time according to either the chart review or the automated system.

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation

*AHRQ Priority Population.