Evaluation of the Use of AHRQ and Other Quality Indicators

In this chapter we describe the methods used in this study. There are three main components to our approach: an environmental scan to identify and catalog users of the AHRQ QIs and developers or vendors of similar projects; a series of interviews with individuals and organizations who use AHRQ or other QIs to understand how and why QIs are used, and what improvements, if any, users might like to see; and case studies to illustrate how QIs are used in two particular geographic areas.

2.1. Environmental Scan

We conducted an environmental scan to identify four types of organizations: (1) users of the AHRQ QIs; (2) "non-users," i.e., organizations that are using an alternative to the AHRQ QIs; (3) developers of similar sets of quality indicators; and (4) vendors of quality measurement products that may or may not include AHRQ QIs. The following six types of sources were used:

• Databases of published literature.
• Conference presentation abstracts.
• World Wide Web search engines.
• Reviews of quality measurement activities conducted by third parties.
• AHRQ materials.
• Query of RAND Health research staff.

The specific sources and the search terms are listed in Tables 2.1 and 2.2.

Table 2.1. Sources Queried in Environmental Scan

Table 2.2. Search Queries Used in Environmental Scan

Each organization identified in the search was entered into a database together with the following information:

• Type of organization (e.g., hospital).
• List of quality indicators used.
• Primary use of quality indicators (research, public reporting, pay-for-performance, quality improvement, vendor, developer).
• Description of other uses of quality indicators.
• Citation.
• Contact information.
• Method of identification.

The environmental scan was used to understand the range of ways in which the AHRQ QIs are being used and to select interview candidates.

Return to Contents

2.2. Interviews

2.2.1. Selection of interviewees

In order to select interview candidates, we first consulted with five members of the AHRQ QI team, which includes both AHRQ staff and contractors. We used information gained from these discussions, together with the results of our environmental scan, to compile a list of potential interviewees. We used the following method to select interviewees from this complete list:

1. We cross-tabulated the environmental scan results by type of organization (hospital association, state government, etc.) and primary indicator use (pay-for-performance, public reporting, etc.) to create a table representing the universe of potential interviewees.
2. We selected the number of interviewees to be chosen in each type-use cell in order to distribute interviews across cells and ensure multiple interviews per cell where possible.
3. Two of the researchers (PH and SM) reviewed the list of organizations independently and chose interviewees for each cell based on the criteria: (a) maximize the variety of uses and types of organizations within each cell; (b) maximize the estimated impact of quality indicator activity. The reviewers compared notes and reconciled differences to come to the final list of suggested interviewees to forward to AHRQ.
4. We met with AHRQ staff to discuss the suggested interviewees and incorporated AHRQ staff comments to arrive at the final list.

We then began the process of identifying the most appropriate respondent(s) for each organization. Identification of the most knowledgeable respondent was a multi-stage process. First, we identified an initial contact through available published materials or a telephone call to the organization. We then requested that the individual we contacted forward our interview request to the most appropriate individual(s). Prior to the interview, interviewees were given a fact sheet on the project, information about how the data would be used and assurances about confidentiality, and a list of sample questions similar to those asked during the interviews. All procedures, including verbal consent procedures, were approved in advance by the RAND Human Subjects Protection Committee, RAND's Institutional Review Board.

2.2.2. Interview procedures

We created interview guides based on input of AHRQ QI team staff, our own knowledge of the AHRQ QI program, and the results of the environmental scan. Separate guides were created for users of AHRQ QIs, developers of similar products, and vendors of quality measurement products. The interviews covered three main topics:

1. How AHRQ QIs (and other quality indicators) have been used.
2. Experiences (including impact of use and lessons learned) from quality indicator use.
3. Suggestions for future priorities for the AHRQ QI program.

Semi-structured interviews were conducted by telephone by one researcher accompanied by a note-taker. Some interviews were also recorded using a digital voice recorder. Interviewees were guaranteed that none of their responses would be reported in an identifiable format. During the interviews, we made note of any supporting documents mentioned by the interviewees and asked for a copy of the documents at the close of the interview. Interview notes were completed and edited in a timely fashion and data from the interviews and supporting materials were coded into the dimensions in the interview guide and entered into a database for analysis. Quotes published in this report have been reconstructed from interview notes or recordings and thus in some cases may differ slightly from the exact wording used by the interviewee.

2.2.3. International Users

We took advantage of the (non-project related) participation of one member of the team (SM) in an OECD Expert Group meeting on patient safety. The meeting took place on June 29 and 30, 2006, in Dublin, Ireland. At the meeting, we interviewed five researchers who had used the AHQI QIs in countries other than the United States.

Return to Contents

2.3. Case Studies

Based on the results of the first round of interviews, we identified two geographic areas in which to conduct in-depth case studies on the use of AHRQ QI indicators for public reporting. The case study is a research strategy that is used in many settings (e.g., policy, political science, and public administration research; organizational and management studies; planning research; etc.). It is an empirical inquiry in which multiple sources of evidence are used to provide data from which judgments can be made about the usefulness of an approach to a problem and the generalizability of the findings to other sponsoring organizations and markets. We selected two geographic areas (Boston, MA and Dallas-Fort Worth, TX) and identified multiple organizations within each of these markets to provide a full picture of the various perspectives on a common application of AHRQ QIs.

These case studies were designed to refine our notions of the critical variables present in AHRQ QI initiatives; obtain evidence regarding how successful organizations were in developing and implementing QI initiatives; and to document the barriers and facilitators to using AHRQ QIs in real-world health care settings. We first identified organizations for case study interviews by (1) analyzing the results of the environmental scan to identify users of the AHRQ QIs in the two geographic areas; (2) asking for nominations from representatives of organizations who had already been asked to participate; (3) taking suggestions from AHRQ staff. Detailed notes were taken for each case study. These notes as well as media and policy reports on health policy issues in the geographic areas were used as the basis for the case study analysis.

Return to Contents
Proceed to Next Section