Download the 2019 Year in Review Report (PDF, 6.02 MB)

PRO Highlight: AHRQ Advances the Use of Patient-Reported Outcomes to Improve Care

PROs include information about the status of a patient’s health condition that comes directly from the patient, without interpretation by a clinician or others. These data can yield insights into health status, function, symptom burden, adherence, health behaviors, and quality of life. PROs also help to improve patient-centered care by informing clinical decision making and supporting patient self-management, care planning, goal setting, and goal attainment.

Despite these benefits, PRO data are not routinely collected or used in clinical practice. Providers face barriers from measurement selection to data integration into clinical decision making6 . Until recently, there were no standards for collecting and integrating PRO data into health IT systems, which limited providers’ ability to use PRO data or easily share it across health systems for research or quality improvement purposes. The limited use of PRO data is also a result of the challenges of collecting and using the information in a way that does not disrupt the workflow of providers, along with the complications of integrating PRO data into EHRs. Data-element and data-capture standards could enable PRO assessments to be conducted and easily shared across clinical or research systems, regardless of the EHR or health IT solution.

To address this problem, AHRQ partnered with the Office of the National Coordinator for Health Information Technology (ONC) on a project that aimed to advance the collection and integration of standardized PRO data in a manner that supports interoperable sharing of information. ONC developed the PRO FHIR Implementation Guide (IG) that guides developers in implementing FHIR for the exchange of PRO data as well as the capabilities required for a successful integration of PROs in patient records. AHRQ hosted a challenge competition to develop applications incorporating the PRO FHIR IG. AHRQ also worked with MedStar Health to modify an existing app and to pilot test the modified app and the winning app from the challenge competition. The challenges and lessons learned are described below.

AHRQ Step Up App Challenge: Advancing Care by Crowdsourcing FHIR®-enabled Digital Patient Self-Assessment Tools

The goal of the AHRQ-funded Step Up App Challenge was to develop a user-friendly app capable of collecting standardized PRO data in a variety of ambulatory settings, including primary and specialty care. This effort is part of AHRQ’s support for HHS Secretary Alex Azar’s priority related to increasing value in healthcare by empowering patients with data and information to help them take an active role in making decisions about their care.

AHRQ challenged teams to design, develop, and pilot a user-friendly app to simplify the process of collecting, interpreting, aggregating, and sharing PRO data related to physical function outcomes. Participants were asked to incorporate the PRO FHIR IG in their app designs. The use of FHIR standards would enable the integration of PRO data into EHRs and across providers.

AHRQ partnered with the marketing firm Sensis to design the competition strategy. The competition was structured using a multi-stage incentive model (proposal, development, and pilot testing) with increasing capital prizes for awardees.

Phase 1 Proposal: Activating the Software Development Community

The $250,000 challenge was announced at the 2018 Blue Button 2.0 conference. The challenge elicited 54 proposal submissions. From those submissions, 10 teams were chosen and each was awarded $12,500 to develop functioning prototypes for Phase 2.

Phase 2 Development: Prototyping FHIR-Enabled Mobile Apps Leveraging a PROMIS API

In addition to being the grand prize winner in the Challenge Competition, the PRISM app also won second place in the FHIR® App Showcase in the 2019 AMIA Annual Symposium.

In Phase 2, AHRQ provided the winning teams with the time, resources, and information to develop a patient-facing mobile app compliant with the PRO FHIR IG to administer the Patient-Reported Outcomes Measurement Information System (PROMIS®) physical functioning (PF) measures. The teams were required to design mobile apps that engage patients in collecting PRO data, including leveraging computerized adaptive testing so that PRO questions would be tailored to the patient’s health status. In early 2019, three teams were selected as winners. The grand prize winner, the PROMIS Reporting and Insight System from Minnesota (PRISM) app, advanced to the final challenge phase. PRISM was developed by a multidisciplinary team comprising experts from the University of Minnesota’s Carlson School of Management and the Institute for Health Informatics, Fairview Health System/HealthEast Kidney Stone Institute, and PerkHealth, a Minnesota-based startup mobile app development company.

More information about the winning teams is available at

Phase 3 Pilot Testing: Deploying the Mobile App in Nine MedStar Health Clinics

In Phase 3 of the challenge, the PRISM team collaborated with MedStar Health System to pilot the app in nine clinics that use three different EHR systems. The pilot provided the PRISM team with hands-on implementation experience from backend technical integration to trouble-shooting in clinics. The code used to develop the PRISM app is publicly available at

“The challenge competition was a successful public-private partnership. It helped create a competitive spirit and sense of urgency to get an initial application designed well enough to win at each stage of the contest. The final result is released as open source. This is a way to give back to the community.”
- Steve Johnson, PRISM Developer

App Modification and Followup Pilot Tests

Through a separate contract, MedStar Health modified an existing app, Outcomes Based Electronic Research Database (OBERD) using the PRO FHIR IG. OBERD is a web-based app used in MedStar Health orthopedic practices to collect PRO PF data. MedStar conducted pilot tests of both the OBERD and PRISM apps in 18 primary and specialty care practices in the MedStar system and the Capital Area Primary Care Research Network. While both apps collected PF data, the OBERD app was used on a tablet while the PRISM app was designed for use on a patient’s mobile device. Participating sites were of various size and geographic locations, with distinct workflows and three different EHRs.

The findings from the pilot tests highlighted important barriers and facilitators for PRO implementation, some of which align with the existing literature, and several that generate important considerations for the potential scalability of collecting technology-enabled PRO data. Notably, the pilots demonstrated that for a technical implementation of this scale, the PRO app itself was rarely the most significant challenge. The most notable hurdle was the interplay of multiple socio-technical factors that impacted implementation. These barriers included technical and workflow integration, patient engagement, and perceived clinical utility.

AHRQ-Funded Pilots Contribute to the Knowledge Base on Integrating Patient-Reported Outcomes in Care

Collectively, this AHRQ-funded work revealed the following key lessons learned:

  • The mobile app can be used by patients taking surveys. For users with a range of technical abilities, the apps presented a good user experience.
  • For many practices, technical assistance or additional staffing are critical components to ensure adoption. It is essential to address staff workload so that clinic staff have the appropriate resources to perform survey collection tasks.
  • No amount of planning can replace testing in the real world, which is critical to successful implementation. Given the complexity of healthcare contexts, all of the planning and preparation rarely prepares an organization for the realities of local implementation. Many realities are simply impossible to anticipate.
  • An abstraction layer between the app and the EHR is critical for successful Integration of PRO data in the EHR. The FHIR server functioned as an adapter and bridge between the apps and the technical and version issues encountered during the integration with different EHRs. A standards-based, loosely coupled architecture enables rapid reuse of existing apps and allows developers to focus on tool building.
  • Institutional policies impact success. One of the most significant barriers encountered was a delay in implementation due to complex institutional policies and regulations regarding approval and clearance to integrate PRO technology with the health institution’s native information systems architecture. Such policies and regulations can be highly variable by institution and health system, but will need to be part of any implementation timeline. Technical teams should allow significant time to navigate these processes and to meet institutional requirements, particularly when vendors are working with a system for the first time.

This AHRQ-funded research is an important step toward testing the application of the PRO FHIR IG, as well as a demonstration of the factors critical to the successful adoption, potential scaling, and sustained use of the technology in ambulatory care settings. Findings from the pilot highlight critical points at which the coordination of human and technical processes is crucial to ensure the successful use of PRO data. A successful implementation of a PRO data collection tool does not necessarily guarantee or imply long-term adoption or meaningful data use. Successful use of PRO data is complex and tends to be context-dependent and strongly coupled to the existing relationships between patients and their providers.