Evaluation of the Use of AHRQ and Other Quality Indicators

Our environmental scan revealed strong demand for hospital care quality indicators. Demand for indicators for research and quality monitoring is strong and has a relatively long history. Demand is higher and increasing rapidly for quality indicators that can be used for other, newer purposes. These purposes include public reporting to inform consumers' choice of providers and otherwise drive provider improvement; pay-for-performance to reward high-quality providers; the development of tiered insurance products; and using quality indicators to select a network of providers.

This demand has led to a proliferation of quality indicators. In addition to AHRQ, the market leaders in developing hospital quality indicators are the Centers for Medicare and Medicaid Services (CMS), the Joint Commission on Accreditation of Healthcare Organizations (JCAHO), the Hospital Quality Alliance (HQA—a collaboration between CMS, JCAHO, and several other organizations), and the Leapfrog Group. In this section, we discuss these and other developers and vendors of quality indicators, and how the quality indicators developed by each of these agencies/organizations compares to the AHRQ QIs. Our environmental scan identified two main categories of players in the market for quality indicators. The first type, "developers," includes organizations that develop, support, and distribute quality indicators. The second type, "vendors," includes organizations that develop and/or sell quality measurement products to providers, insurers, and others. Their products often include the AHRQ QIs (or variants thereof), indicators from other developers, and/or indicators developed by the vendors themselves.

3.1. Developers

The environmental scan identified 12 organizations that have developed indicators that are similar in some way to the AHRQ QIs. The organizations that have developed indicators that are widely used and focused on hospitals are summarized in Table 3.1 and described below.

Although there are similarities between these indicators and those developed by AHRQ, none of the indicators developed by organizations other than AHRQ were comparable to the AHRQ QIs on all of their major characteristics: based on administrative data, outcome-focused, hospital-focused, based on transparent methodology, and available for public use.

Table 3.1. Developers of Quality Indicators and Comparison with AHRQ QIs

Source: RAND analysis of environmental scan results.
Note: Indicators were judged to be a "national standard" if they were described that way by any of the study's interviewees.

JCAHO/CMS/HQA. Both JCAHO and CMS have developed quality indicators of hospital care for common conditions. CMS's measures were originally used for quality improvement initiatives conducted by Medicare Quality Improvement Organizations (QIOs). JCAHO's Core Measures have been used as part of the JCAHO hospital accreditation process since 2002. They cover five clinical areas: (1) acute myocardial infarction, (2) heart failure, (3) pneumonia, (4) surgical infection prevention, and (5) pregnancy and related conditions. JCAHO-accredited hospitals choose 3 of these 5 areas for reporting, depending on the services they provide. JCAHO publishes the results of the measures publicly on the Web.²¹

Since the measures had significant overlap, CMS and JCAHO agreed in 2004 to align specifications for overlapping measures and to maintain them as a shared set of measures. A subset of the joint CMS-JCAHO measures was later selected by the HQA, a public-private partnership for measuring and reporting hospital quality. Their Hospital Quality Measures are now publicly reported on the Web for both accredited and non-accredited hospitals.²² They are also used in other CMS activities such as the Premier pay-for-performance demonstration project.²³

Like the AHRQ QIs, the CMS/JCAHO/HQA measures are widely used and viewed as a national standard.^d A key difference between those measures and the AHRQ QIs is that they are largely based on clinical data collected from medical records rather than administrative data. JCAHO has estimated that collection of the clinical data for the Core Measures takes an average of 22-27 minutes per case for acute myocardial infarction, heart failure, and pneumonia.²⁴

A second key difference is that the CMS/JCAHO/HQA measures are process indicators while the AHRQ QIs are outcome indicators. Another difference is that, while the AHRQ QIs reflect a broad range of conditions, the CMS/JCAHO/HQA measures currently reflect only five conditions; however, JCAHO and CMS are currently developing indicators in additional clinical areas.

The method used by JCAHO to implement its Core Measures is also different from that used for the AHRQ QIs. Hospitals pay vendors to measure the JCAHO Core Measures on their behalf using standardized specifications. Hospitals have made a wide variety of arrangements with vendors for Core Measure collection and reporting, according to their specific needs and characteristics. All vendors of the JCAHO Core Measures must undergo a certification process through which JCAHO ensures that they have appropriately implemented the measures.

Due to these differences, the CMS/JCAHO/HQA measures and the AHRQ QIs can be considered complementary in some respects. A number of the users of the AHRQ QIs interviewed (11 of 36) also use the JCAHO/CMS/HQA measures.

The only way in which the CMS/JCAHO/HQA measures and the AHRQ QIs could be considered competitors is as a function of limited hospital resources for quality measurement. Hospitals are required to report the JCAHO Core Measures for accreditation and may have limited resources for other quality measurement activities, including the AHRQ QIs. One interviewee told us:

AHRQ could do a lot of terrific things with the AHRQ QIs, but facilities are trying to meet requirements right now and don't have time and resources to work with other quality indicators to the exclusion of what they might like to do. Hospitals are doing only what they have to do—either by mandate or by the market.^e

Leapfrog. The Leapfrog Group has developed a set of quality indicators that are widely used and considered to be a national standard. The indicators are intended to improve value in health care purchasing. Provider performance on the indicators is presented in a public report on Leapfrog's Web site. In addition to developing and marketing its own quality indicators, Leapfrog operates a pay-for-performance program, the Leapfrog Hospital Rewards Program, which uses JCAHO Core Measures and an efficiency measure in addition to the Leapfrog indicators. The program is implemented through vendors, who pay Leapfrog for every participating hospital, and then charge hospitals accordingly.

Unlike the AHRQ QIs, most of the Leapfrog indicators are not outcome-focused and require primary data collection. The indicators are organized into four content areas called "Leaps": (1) computerized physician order entry, (2) intensive care unit staffing, (3) high-risk treatments, and (4) safe practices. Data are collected through a survey of hospitals. Leaps 1, 2, and 4 are structure and process indicators, such as use of a computerized physician order entry system or staffing hospital intensive care units with intensivists (physicians who specialize in critical care medicine). Leap 3 (high-risk treatments) overlaps considerably with the AHRQ IQIs. It measures procedure volume and risk-adjusted mortality for selected conditions. Leapfrog is currently standardizing its specifications to those used in the AHRQ IQIs in order to minimize the reporting burden for hospitals.

Institute for Healthcare Improvement (IHI). The IHI measures overall hospital mortality as part of its activities to improve hospital quality. This measurement activity is conducted in conjunction with the implementation of a specific set of interventions that are intended to improve quality in participating hospitals. The indicator used is similar to the AHRQ IQIs in that it is based on risk-adjusted mortality associated with hospital stays and is based on the analysis of administrative data. Unlike the AHRQ IQIs, however, the IHI measures the mortality rate for all conditions. Hospital- and area-level characteristics are used in regression models to control for patient risk. This measurement approach originated in the United Kingdom and has also been applied to hospitals in many countries other than the United States.²⁵

States. We also interviewed representatives from California and Pennsylvania, two states that have developed their own methodologies for measuring quality using administrative data. These states developed their own measurement approaches largely because their public reporting efforts predate the development of the AHRQ QIs. Both states also use data elements that are unavailable in the hospital administrative data collected in most other states. These features include a flag to indicate conditions that were present on hospital admission (California) and detailed data on severity of illness (Pennsylvania). Other states, such as New York, have also developed their own measurement approaches which may predate the AHRQ QIs or use data elements not available in other states.

Vendors. We interviewed several vendors who, in addition to implementing existing measures from other developers in their measurement tools, have also developed proprietary indicators. Some of these indicators are similar to the AHRQ QIs in that they are based on administrative data and are outcomes indicators. The key difference is the definitions and specifications of most vendors' indicators are proprietary. The vendors' indicators have also not always been subjected to validation of the same rigor as the AHRQ QIs. In the next subsection, we discuss the vendors identified by the environmental scan in more detail.

Return to Contents

3.2. Vendors

The environmental scan identified 12 vendors of quality measurement products that were determined to include the AHRQ QIs.^fThese vendors are listed in Table 3.2.

Typically, the AHRQ QIs are included in software tools that are marketed to hospitals for quality improvement or to insurers or business groups for hospital profiling. The vendors' products offer additional functionality to the basic AHRQ QI software. For example, the vendors' measurement tools often include non-quality indicators that inform hospital administration, such as financial performance indicators. The tools are often designed to offer users a variety of reporting options. These measurement tools may be particularly useful for hospitals that do not have the in-house expertise or staff time to construct indicators of quality and other aspects of care from raw data. Similar tools are used by insurance companies and other organizations.

Table 3.2. Vendors of Quality Measurement Products That Include the AHRQ QIs

Source: RAND analysis of environmental scan results.

As mentioned above, many of these tools include proprietary quality indicators developed by the vendors themselves. In addition, many of the vendors are licensed to implement the JCAHO Core Measures, and many also produce indicators from other developers, such as Leapfrog.

Some users of the AHRQ QIs whom we interviewed use vendors for their measurement activities and expressed a high degree of satisfaction with the vendors' services. On the other hand, some users expressed a concern that the AHRQ QIs as implemented by some vendors may differ in key respects from the official AHRQ QI specifications, and that the proprietary nature of the tools makes these differences non-transparent. One hospital association captured this sentiment:

The AHRQ QIs are standardized measures, risk-adjusted, and not in a "black box" so we can get the numerator and denominator and make them accessible to hospitals. The industry is sick and tired of vendors and consulting firms creating black boxes.

Another interviewee sounded similar themes:

The problem is that if there's any "black box" methodology, [users] won't touch it—it's politically dead, even if there is an underlying valid scientific process. Hospitals want to check their own numbers. [The vendors'] offers sound nice. The problem is, a hospital can't replicate the findings or understand differences in methodology/calculations. [Users] like transparency, a tool that is open, where everyone can see what is happening, hospitals can replicate the results, then everyone can talk about the differences. It democratizes quality reporting.

Return to Contents

3.3. AHRQ's Position in the Market for Quality Indicators

While the quality indicators developed by organizations other than AHRQ share certain characteristics with the AHRQ QI program, there are no other sources of indicators that are viewed as a national standard and are also publicly available, transparent, hospital-focused, outcome-focused, and based on administrative data. Many of our interviewees stressed that the AHRQ QIs fill an important void in this respect. A representative of an integrated delivery system described the process of searching for quality indicators that could be readily used for monitoring quality and guiding quality improvement activities:

When we started looking for indicators, we really struggled to find valid quality measures based on readily available data and with benchmark potential. Without manually auditing patient charts, and coming up with numerator and denominator definitions on our own, there was no way we could do it by ourselves. AHRQ offered the set of measures prescribed for our needs.

A representative of a state doing public reporting told us:

If we didn't have the AHRQ QIs, it would be difficult as a state to come up with our own indicators and there are not many other choices that are based on administrative data. Until electronic medical records are commonplace (5-10 years at least), we need to deal with using administrative data.

An insurance company representative highlighted the importance of AHRQ's role in the quality indicator market, stating that more marketing of the QIs is needed:

AHRQ is doing something that no one else is doing. We have to have a national standard, something used across the country for comparison. [Does AHRQ] realize they're one of the only good options out there? They should really pick up the outreach so that others will pick up using the QIs.

3.3.1. Overview of users and uses of the AHRQ QI

AHRQ's unique position in the market for quality indicators has led to a wide proliferation of uses for the AHRQ QIs. Our environmental scan of users of the AHRQ QIs identified 114 users of the indicators and a range of different purposes, including public reporting, quality improvement/benchmarking, pay-for-performance, and research. Table 3.3 summarizes the number of users of the AHRQ QIs by type of organization and purpose of use.

Table 3.3. Users of the AHRQ Qis

Source: RAND analysis of environmental scan results.

The most common uses of the AHRQ QIs include:

• Research. We identified 43 organizations that use AHRQ QIs for research. For example, Leslie Greenwald and colleagues used the AHRQ QIs to compare the quality of care provided in physician-owned specialty hospitals and competitor hospitals.²⁶
• Quality improvement. We identified 23 organizations that use the AHRQ QIs as part of a quality improvement activity, including reports benchmarking performance against peers; however, these organizations do not release the quality information into the public domain.^g
• Public reporting. We identified 20 organizations using the AHRQ QIs for public reporting. We classified an activity as "public reporting" if a publicly available report was published that compares AHRQ QI results between hospitals or geographic areas such as counties. The organizations using the AHRQ QIs for public reporting, with Web links to the reports, are listed in Table 3.4.
• Pay-for-Performance. We identified 4 organizations that are using the AHRQ QIs in pay-for-performance programs. Three were health plans and one was a CMS demonstration project.

Table 3.4. Users of the AHRQ Qis

Source: RAND analysis of environmental scan results.
Note: "Public reporting" was defined as a publicly available report that compares AHRQ QI results between hospitals or geographic areas such as counties. Not all of the public reports identified in this table are intended to influence consumers' choice of provider.
"One-time reports" are published comparisons that are not labeled as an ongoing activity.
"Periodic reports" are published comparisons, updated periodically, that are in static format (e.g., documents available as .pdf files online).
"Interactive tools" are online comparisons that allow users to create customized reports (e.g., selection of providers or indicators of interest).

3.3.2. Uses of Specific AHRQ QIs

We asked users of the AHRQ QIs, and vendors of quality measurement packages including the AHRQ QIs, which specific QIs they were using. Among organizations we interviewed, the PSIs and IQIs were used more frequently than the PQIs. Of the 42 organizations, 33 were using the PSIs, 30 were using the IQIs, and 17 were using the PQIs.

Within the PSI and IQI sets, some indicators were used more frequently than others. Users of the PQIs, on the other hand, were more likely to use every PQI. There were no meaningful differences in the frequency of use of particular PQIs (data not shown).

3.3.2.1. Use of IQIs

Figure 3.1 shows the frequency of use of each IQI by the users and vendors we interviewed. The IQIs that reflect mortality rates for medical conditions were used most frequently, particularly:

• IQI 16—congestive heart failure mortality (23 users).
• IQI 17—stroke mortality (23 users).
• IQI 20—pneumonia mortality (22 users).

The rates of mortality for certain medical procedures were also commonly used, particularly:

• IQI 12—coronary artery bypass graft mortality (23 users).
• IQI 13—craniotomy mortality (19 users).
• IQI 11—abdominal aortic aneurysm repair mortality (18 users).
• IQI 14 —hip replacement mortality (18 users), and
• IQI 30—percutaneous transluminal coronary angioplasty mortality (18 users).

The procedure volume indicators were used less frequently, and the procedure utilization rates, both hospital- and area-level, were used least frequently.

Figure 3.1. Frequency of Use of Specific AHRQ IQIs among 42 Users of the AHRQ QIs

Figure 3.1. depicts five categories of quality indicators measured against frequency of use in a columnar graph. The data represented by this graph is contained in the following table.

Source: RAND analysis of environmental scan results.

3.3.2.2. Use of PSIs

Figure 3.2 shows similar counts of the frequency of use of each PSI. The area-level PSIs were used less frequently than the hospital-level PSIs. Among the hospital-level indicators, there was considerable variation in frequency of use between the indicators. The most frequently used PSIs were PSI 12—postoperative pulmonary embolism (PE) or deep vein thrombosis (DVT) (28 users), PSI 8—postoperative hip fracture (26 users), and PSI 13—postoperative sepsis (26 users). The least frequently used hospital-level PSIs were PSIs 18, 19, and 20—obstetric trauma with instrument, without instrument, and during cesarean delivery (15, 16, and 15 users, respectively).

Figure 3.2. Frequency of Use of Specific AHRQ PSIs among 42 Users of the AHRQ QIs

Figure 3.2. depicts two categories of Patient Safety Indicators (PSI) measured against frequency of use in a columnar graph. The data represented by this graph is contained in the following table.

Source: RAND analysis of environmental scan results.

3.3.3. International uses of QIs

Measuring quality of care has become a policy priority in many countries outside of the United States, and numerous countries and international organizations are in the process of instituting requirements for data collection and reporting of quality indicators.²⁷ The AHRQ QIs are an attractive option for international users, since many countries already require hospitals to report the required administrative data.

Perhaps the most visible international endeavor is the Organization for Economic Cooperation and Development's (OECD) Health Care Quality Indicators (HCQI) Project. The OECD is an intergovernmental economic research institution headquartered in Paris, France, with a membership of 30 countries that share a commitment to democratic government and the market economy. One of its widely used products is OECD Health Data, which provides internationally comparable information on infrastructure, cost and utilization at the health system level,²⁸ but so far no information on quality of care. In an attempt to bridge this gap, in 2003 the OECD brought 23 of its member countries together with international organizations such as the World Health Organization (WHO) and the European Commission (EC), expert organizations such as the International Society of Quality in Healthcare (ISQua) and the European Society for Quality in Healthcare (ESQH), and several universities.²⁹ The goal of the meeting was to work on the development and implementation of quality indicators at the international level.

The project initiated its work with two major activities. The first was an effort to introduce a pilot set of quality indicators that can be reported by a substantial portion of the OECD countries. This activity recently led to the 2006 release of an initial list of indicators and corresponding data.³⁰ The second activity was to identify additional quality indicators for five priority areas: cardiac care, diabetes mellitus, mental health, patient safety, primary care/prevention/health promotion. Through an expert panel process, 86 indicators were selected for the five areas and the OECD is currently investigating the availability and validity of required data.³¹ Several AHRQ PSIs were selected for the patient safety area³² and an indicator similar to the PQIs was selected for the primary care area.³³

Researchers from several countries have tried to run the PSIs against national discharge data both as part of their participation in the HCQI Project and also for other projects. This has been attempted in Canada, Germany, Italy, and Sweden. In addition, a group in Belgium successfully constructed some of the HCUP indicators, the predecessors of the AHRQ QIs, from national data.³⁴

At this point, results from those projects are largely unpublished in English-language journals. But during a recent OECD meeting in Dublin, Ireland, experts from 15 countries discussed issues around the adoption of the AHRQ PSIs in countries other than the United States. Researchers from several countries had cross-walked the AHRQ PSI specifications, which are based on the U.S.-specific ICD-9-CM diagnosis codes, to ICD-10 diagnosis codes, which most countries are now using.

This conversion was found to be unproblematic, in particular because only a limited number of diagnosis codes had to be cross-walked to construct the indicators. A greater issue turned out to be the conversion of procedure codes. The AHRQ definitions are based on the ICD-9 procedure classification, whereas other countries use national procedure classification systems. Similarly, other countries use different DRG groupings than those used in the United States. Substantial work on mapping the different coding systems used in the U.S. and in other countries is needed.

In countries that have tested the AHRQ PSIs, the average rates were reported to be similar to those observed in the United States. Countries that do not yet have DRG-based prospective payment systems saw much lower rates, possibly resulting from less thorough coding of secondary diagnoses in the absence of implications for payment.

Our interviews show that there is interest in using the AHRQ QIs internationally and sufficient data and technical capability to implement them. This makes it likely that some AHRQ QIs will be adopted by the OECD HCQI Project for international comparisons of quality of care and patient safety. Furthermore, as several international organizations are striving to align their measurement and reporting activities,^h selected AHRQ QIs could become part of an international measurement standard.

3.3.4. "Non-users" of QIs

We identified and interviewed representatives of five organizations that are currently using quality indicators other than the AHRQ QIs but that are similar to the AHRQ QIs. Our goal was to understand why these organizations were not using the AHRQ QIs, and specifically whether this decision was based on an evaluation of the merits of the QIs. Three of the organizations were using quality indicators that they had developed themselves and which predated the AHRQ QIs. They did not voice any strong objections to the AHRQ QIs but preferred their own indicators due to various methodological factors and the fact that their indicators were better tailored for their specific needs. The other two organizations had elected not to use the AHRQ QIs because they were not already in use by the hospitals that would be participating in the organizations' quality measurement activities. The JCAHO Core Measures were chosen instead because they were already being collected by hospitals.

3.3.5. Examples of uses of QIs

In order to illustrate how the AHRQ QIs are being used, we have chosen examples of specific uses for quality monitoring, public reporting, and pay-for-performance.

3.3.5.1. Example of AHRQ QI use for quality improvement

Figure 3.3 was drawn from a report provided to hospitals by a hospital association we interviewed. Reports such as the one we reviewed are sent to hospital CEOs quarterly. The reports include all AHRQ IQIs (shown in the example) as well as all AHRQ PSIs. The report also includes indicators from JCAHO and Leapfrog (not shown). Hospitals are presented with targets based on benchmarks calculated by the hospital association. The hospital association works with hospitals to help them explain why they do not meet targets in areas of poor performance.

Figure 3.3. Sample AHRQ QI Report Used by a Hospital Association for Quality Improvement

Figure 3.3. depicts a sample AHRQ QI report in a tabular format. The data represented by this graph is contained in the following table:

Notes:
A = No Data Available;
B = Substantially Worse than Target;
C = Worse than Target;
D = Better than Target;
E = Meets Target;
* Interpret with caution (less than 25 cases)

3.3.5.2. Example of AHRQ QI use for pay-for-performance

The State of Florida uses the AHRQ QIs as part of a public reporting tool aimed to help consumers choose a hospital. Figure 3.4 captures a segment of a Web page comparing hospitals in Broward County on one of the AHRQ IQIs, postoperative hip fracture (IQI 19). Users can click on a hospital to get more detailed information on quality as well as the hospital's characteristics (teaching status, non-profit status, etc.) and location.

Figure 3.4. Sample AHRQ QI Report Used by the State of Florida for Public Reporting

Figure 3.4. depicts a table showing sample AHRQ QI Report used by the State of Florida. The data represented by this graph is contained in the following table.

All copyrights in and to 3M™ APR™ DRGs Classification System and 3M™ APR™ Software are owned by 3M. All rights reserved.

What the Complication/infection rate means:

The percentage rates reported on this page reflect each hospital's unique population, and should not be compared between hospitals. Instead, it is strongly recommended consumers compare each hospital on the basis of whether their rates are "as expected", "lower than expected", or "higher than expected".

The percentage rate is:

• Lower than Expected—Fewer complications/infections than expected given how sick patients were
• As Expected—Expected number of complications/infections given how sick patients were
• Higher than Expected—More complications/infections than expected given how sick patients were

"X" is less than 30 cases; "Too few cases" is less than 5 cases.

3.3.5.3. Example of AHRQ QI use for public reporting

Figure 3.5 is drawn from a report provided to hospitals by an insurer we interviewed. The example extracts one AHRQ PSI (PSI 12), postoperative pulmonary embolism (PE) or deep vein thrombosis (DVT). The report allows hospitals to compare their performance to that of their peers. Good performers earn an incentive payment.

Figure 3.5. Sample AHRQ QI Report Used by an Insurance Company for Pay-for-Performance

Figure 3.5. depicts a sample table of an AHRQ QI report used by an insurance company to report pay-for-performance data. The data contained in this graph is in the following table.

Notes: Confidence Interval Designations
A = Significant Difference at 80%
B = Significant Difference at 90%
C = Significant Difference at 95%
D = No Significant Difference vs. Cohort Average

^d. Indicators were judged to be a "national standard" if they were described that way by any of the study's interviewees.

^e. This and all quotes appearing in this report are reconstructions based on interview notes or recordings.

^f. We attempted to determine whether vendors' proprietary products included the AHRQ QIs, but since limited information is available from some vendors, some mistaken attribution is possible. There are also other vendors with similar quality measurement products that do not include the AHRQ QIs, but they were not included in our study.

^g. Due to the methods used to identify users, the scan is likely to have significantly undercounted the number of organizations (especially hospitals and hospital associations) using the AHRQ QIs for internal quality improvement activities, since this type of use rarely results in publicly available information that could be used to identify the user in an environmental scan.

^h. For example, the European Commission has recently ceded its activities in quality indicator development to the OECD to avoid duplication and is funding part of the HCQI Project.

Return to Contents
Proceed to Next Section