Automating Assessment of Asthma Care Quality (Oregon)

Project Final Report (PDF, 969.36 KB) Disclaimer

Automating Assessment of Asthma Care Quality - 2010

Summary Highlights

  • Principal Investigator: 
  • Funding Mechanism: 
    RFA HS07-002: Ambulatory and Safety Quality Program: Enabling Quality Measurement through Health Information Technology (EQM)
  • Grant Number: 
    R18 HS 017022
  • Project Period: 
    September 2007 – September 2010, Including No-Cost Extension
  • AHRQ Funding Amount: 
  • PDF Version: 
    (PDF, 345.79 KB)

Target Population: Adults, Asthma, Chronic Care*, Pediatric*, Teenagers

Summary: This project aimed to develop, validate, apply, and evaluate a scalable method for routine and comprehensive measurement of outpatient asthma care quality (ACQ). This project leveraged health information technology (IT) to assess and improve quality of care for the insured, indigent, uninsured, and underinsured populations of the Pacific Northwest. To accomplish this task, the project employed MediClass (a medical classifier), which is a proven natural language processing technology for extracting care quality data from both coded data and free-text clinical notes in the electronic medical record (EMR). The project performed retrospective analysis of EMR data from two distinct health systems: a mid-sized health maintenance organization, Kaiser Permanente Northwest (KPNW); and a consortium of federally qualified health center clinics for whom an EMR is delivered and managed by one member organization, Our Community Health Information Networks (OCHIN, Inc.). Data was extracted from Kaiser Permanente’s Certification Commission for Health Information Technology-certified Epic-based EMR, HealthConnect; and OCHIN’s Epic-based EMR, EpicCare. Since these EMR applications reside in separate health systems, the implementation of the products generated differences in the data that were accommodated when these data were interpreted for quality assessments. This project leveraged MediClass to implement methods for collecting and transforming data into common formats for quality assessment across multiple data capture, representation, and storage processes.

The starting study participants included patients older than 12 years who were identified to have asthma by a single visit diagnosis code. Subsequently, a modified Healthcare Effectiveness Data and Information Set method that included text processing of clinician notes was applied in order to qualify patients with persistent asthma for inclusion in assessments of care delivered. The study population was drawn from approximately 24 months of OCHIN data and about 120 months of KPNW data.

Most quality measures were associated with several unique target concepts identified by MediClass processing and validation of these “target concepts” was based on chart reviews at the encounter level. Validation of the ACQ measures was based on chart reviews at the patient level. Sensitivity, specificity, false positive and negative rates, and 95-percent confidence intervals (CIs) were computed for each measure based on the review of 900 patient charts. After the automated measurement method was refined and applied to the target population for each quality measure, the proportions and 95-percent CIs were computed for patients receiving the indicated care measures. In addition, patient-level summary ACQ scores were analyzed. These measures were reported on the entire target population and by age, severity, and health care system subgroups. The availability of several years of EMR data at KPNW enabled the project to evaluate the relationship of the automated ACQ measures with health outcomes. Logistic regression was employed to model the association between the automated ACQ summary score and the primary outcome measure.

Specific Aims:

  • Refine ACQ measures from the RAND Quality Assessment Tools Project for use as a quality measure set to evaluate ambulatory asthma care performance. (Achieved)
  • Develop and validate an automated (generalizable and scalable) method for applying the above care quality measures using comprehensive EMR data. (Achieved)
  • Apply the automated method developed above to assess ambulatory ACQ in two distinct health plans representing diverse patient populations and care practices. (Achieved)
  • Evaluate the association between automated measures of adherence to recommended asthma care processes and measures of clinical outcomes using KPNW data only. (Achieved)

2010 Activities: The process for development and refinement of the ACQ measure set was completed. Subsequently the study team obtained the necessary data components including access to medical records onsite at OCHIN to review medical records and the technology used to process medical records, both text and coded fields; and a limited data set from the EMR was shared with the KPNW Center for Health Research. The study team pulled and formulated data for outcomes in the immediate years following the end of the measurement period in KPNW. They also included a pull of these same data in the year prior to qualification for the ACQ measures to correct outcomes analysis for patient baseline severity. Data were analyzed and incorporated into the project final report.

An extensive chart review of stratified random samples of persistent asthma patients and exacerbation events at each site was completed. The chart review process involved a sample of roughly 450 patients at each site. Chart reviews were utilized to collect the data necessary to evaluate criteria for assessing performance on each quality measure. A 10 percent quality assurance sample was conducted to provide secondary review and to resolve discrepancies.

The measurement method utilized a three-year observation period beginning in 2001 at KPNW only. The study staff also pulled KPNW data for a second three-year observation period beginning in 2006. This second data pull provided an observation period that was contemporaneous with the OCHIN data used and allowed comparisons of measurements across the two health systems.

Grantee’s Most Recent Self-Reported Quarterly Status (as of September 2010): As the project came to a close, the primary focus was on the analysis of data and development of manuscripts. The project was able to meet all of its aims and project spending was roughly on target.

Impact and Findings: Most ACQ measures performed well in the KPNW system. Overall accuracy, as measured by chart review, ranged from 63 percent to 100 percent and averaged 88 percent across all measures. Sensitivity was 60 percent or greater for 16 of the 18 implemented measures. Fifteen measures had specificity of 60 percent or higher. There were two measures for which specificity was over 90 percent but which had poor sensitivity.

Overall, the automated ACQ measures did not perform as well in the OCHIN system as they did in the KPNW system. Mean overall accuracy was 85 percent and ranged from 72 to 99 percent. Among the 11 routine care measures, eight had specificities over 80 percent and five had sensitivities over 80 percent. Three measures had specificities of 50 percent or lower while another five had sensitivities of 50 percent or lower. Of the seven exacerbation-related measures, five were evaluable at OCHIN and had a mean overall accuracy of 70 percent with a range of 36 to 96 percent. Sensitivity tended to be relatively low, ranging from 5.3 to 58.1 percent, while specificities were generally high, with a minimum of 67 percent with four measures greater than 90 percent.

Potential explanations for the discrepancy in performance of the automated measures between OCHIN and KPNW include the possibility that chart reviewers may have had differential access to sections of the medical record retrieved by the automated method across sites, and that there may be much greater variability in how and where OCHIN providers document visits for many of the text-based measures. Additional effort may be needed in the specifications of the automated method to capture variations across sites.

Based on chart review results, delivery of routine non-exacerbation care is similar between the two sites. Of these measures, both organizations performed well (better than 90 percent) in providing prescriptions for beta-2 agonists and assuring that their persistent asthma patients were not taking non-cardioselective beta-blockers. Providers in both organizations preformed moderately well (60 to 80 percent) in providing anti-inflammatory controllers and querying patients about tobacco use. Flu vaccination was documented in about 40 percent of patients. The remaining measures were only present between 10 to 30 percent of the time.

Among the exacerbation-related care measures there was similar agreement across sites in the chart review results. Both organizations performed well on the review of current medications and the performance of a chest exam. Performance was poor on the remaining exacerbation-related measures.

The study staff also compared prevalences of the ACQ recommended care as determined by the automated method and by chart review. At KPNW they found generally good agreement between the two methods with a few exceptions. Among the exacerbation-related measures, chart reviewers found more chest exams and current medications reviews while the automated method found more evidence for 4-week followup contacts post hospital discharge. Data for the theophylline measure was too sparse to evaluate adequately. Only six measures were performed 80 percent or more of the time according to either the chart review or the automated method. These results confirm that there are significant gaps between recommended care and real world practice.

A similar analysis was performed comparing chart-review prevalences with those of the automated method at the OCHIN site for each care measure. Among the routine non-exacerbation care measures, chart reviewers found more evidence of flu vaccinations and metered dose inhaler instructions while the automated system detected more instances of smoking status queries and tobacco counseling or referral. Among the measurements of exacerbation-related care, prevalences were higher based on chart review for all measures except the theophylline lab measure, for which the data were to sparse to evaluate. Care gaps in the OCHIN system appear to be very similar to those in the KPNW system, with only five measures performing 80 percent or more of the time according to either the chart review or the automated system.

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation

*AHRQ Priority Population.

Automating Assessment of Asthma Care Quality - Final Report

Hazlehurst B, McBurnie M, Mularski R, et al. Automating Assessment of Asthma Care Quality - Final Report. (Prepared by the Kaiser Permanente Center for Health Research, Portland, Oregon under Grant No. R18 HS017022). Rockville, MD: Agency for Healthcare Research and Quality, 2010. (PDF, 969.36 KB)

The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
Principal Investigator: 
Document Type: 
Medical Condition: 
This project does not have any related resource.
This project does not have any related survey.
This project does not have any related project spotlight.
This project does not have any related survey.
This project does not have any related story.
This project does not have any related emerging lesson.


Each year, thousands of preventable deaths are attributed to asthma, with resultant large economic burden. It has been estimated that half of asthma-related medical costs are attributed to hospitalizations, even though asthma is able to be managed through ambulatory and self-care. Opportunities to improve asthma care are dependent on the ability to assess the care that is delivered to these patients. However quality measurement is dependent on extensive clinical chart review that is unscalable unless electronically automated.

This project developed, implemented, and evaluated the viability of an automated method using natural language processing to conduct routine, comprehensive assessment of the quality of outpatient asthma care across different care systems. The research involved retrospective analysis of electronic medical record (EMR) data from two health systems: a mid-sized health maintenance organization (HMO) and a consortium of public health clinics and care organizations located primarily in the Pacific Northwest.

The specific aims of this project were as follows:

  • Refine asthma care quality measures from the RAND Quality Assessment Tools Project for use as a quality measure set to evaluate ambulatory asthma care performance. 
  • Develop and validate a generalizable and scalable automated method for applying the measures identified in aim 1, using comprehensive EMR data. 
  • Apply the method developed in aim 2 to assess ambulatory asthma care quality in two distinct health plans representing diverse patient populations and care practices. 
  • Evaluate the association between automated measures of adherence to recommended asthma care processes and measures of clinical outcomes. 

This project successfully developed a comprehensive set of 22 measures for assessing the quality of outpatient asthma care and operationalized 18 of them. The measures performed well, although unwanted measurement variation across health systems remained in some cases. The project team found that guideline-recommended care is associated with a reduction in the frequency and severity of asthma exacerbations.