Evaluation of ARRA Comparative Effectiveness Research Dissemination Contract Efforts

Patients and their health care providers have many options when deciding on a treatment plan. Sorting through large volumes of information is difficult and time consuming for physicians and patients alike. This has created a need for synthesized research conducted and compiled by objective experts. Recognizing this vacuum, the Agency for Healthcare Research and Quality (AHRQ) has taken a leading role in developing and widely disseminating comparative effectiveness research (CER), a type of patient-centered outcomes research (PCOR),¹ and sharing it with decisionmakers, including clinicians, health care system administrators, business purchasers, and consumers.

The goal of the evaluation contract is to assess secular trends in consumer and clinician awareness of CER and specific CER topics. Specifically, the data collection effort will ascertain: 1) if and how levels of awareness, understanding, use, and perceived benefits of CER are changing and 2) trends in awareness of AHRQ's EHC Program. Using a computer-assisted telephone interviewing (CATI) survey and a mail survey, respectively, IMPAQ/Battelle (the "IMPAQ team") assessed levels of awareness and use of CER among consumers and clinicians at two points in time. This report focuses on the second data collection phase (wave 2) and presents a longitudinal analysis of the wave 1 and wave 2 consumer and clinician survey data.

Wave 2 Consumer Survey Findings

• Although nearly 65 percent of respondents were aware of the concept of comparing treatment choices and 45 percent had heard of research that can help compare treatment options, only 11 percent indicated that they had heard of the research referred to "by a specific name." Of that group, only two respondents knew it as "comparative effectiveness research," and two respondents identified this research as "patient-centered outcomes research," "PCOR," or "shared decisionmaking."
• When given a definition, 21 percent of respondents indicated that they were aware of research that can help compare treatment options. Respondents who were between 45 and 64 years old (compared to those 18 to 44 years old and 65 and older), Black (compared to White and other races), female, and not enrolled in Medicare or Medicaid (compared to enrollees) were statistically more likely to be aware of the concept of such research.
• Print media, such as newspapers, journals, and magazines, served as the most common source of information on CER, followed by Web sites, television/radio, and health care providers. Of those learning about CER from their clinicians, 51 percent (21 respondents) indicated that their provider had initiated discussions about CER. Health care providers and Web sites were the most common preferred methods to obtain medical information.
• Of the 598 respondents indicating awareness of CER (unaided or aided) and/or the EHC Program, just over half currently use research to help make medical decisions; one-fifth have used it in the past. Respondents who were between 18 and 44 years old (compared to aged 45 and older respondents), Black (compared to White and other races), and female were statistically more likely to use such research. Medicare beneficiaries were statistically more likely than their non-Medicare enrollee counterparts to use CER; however, Medicaid beneficiaries were statistically less likely than non-enrollees to report use of CER.
• A majority of respondents who indicated awareness of CER (unaided or aided) and/or the EHC Program also reported positive perceived benefits of CER. An 88 percent majority reported that evaluating treatment options provides information to help "make good medical health care choices," and an 88 percent majority reported that evaluating treatment options "allows patients and doctors to make choices based on the needs of individual patients."
• A minority (11 percent) of consumers had heard of AHRQ prior to the survey. Very few surveyed consumers had heard of the EHC Program (seven percent) or visited its Web site (three respondents).
• A majority of respondents were interested in evaluating treatment options before making medical decisions (74 percent), while fewer were interested in learning more about evaluating treatment options for specific conditions (51 percent) and learning about the EHC Program (58 percent).

Longitudinal Consumer Survey Findings

The longitudinal analysis comparing wave 1 to wave 2 suggests increases in awareness of and interest in CER, AHRQ, and the EHC Program among consumers. Although changes in consumers' awareness of CER did not reach statistical significance, unaided awareness increased from 61 percent in wave 1 to 65 percent in wave 2, while aided awareness increased from 18 percent in wave 1 to 21 percent in wave 2. Awareness of research on the evaluation of treatment options for specific medical conditions rose from 73 percent in wave 1 to 86 percent in wave 2. As with awareness of CER generally, the increase was not statistically significant.

Consumers reported an increase in awareness of AHRQ and the EHC Program from wave 1 to wave 2. Four percent of consumers indicated awareness of AHRQ in wave 1 and 11 percent reported awareness in wave 2; this change was statistically significant. The percent of consumers reporting awareness of the EHC Program grew from four percent in wave 1 to seven percent in wave 2, but this increase did not reach statistical significance.

In addition to awareness, the team observed a statistically significant increase in consumers' interest in learning more about CER. Consumers indicating interest in learning more about evaluating treatment options for specific medical conditions grew from 37 percent in wave 1 to 51 percent in wave 2. Although the increase did not reach statistical significance, consumers' interest in evaluating treatment options to prepare for medical decisions increased from 69 percent in wave 1 to 74 percent in wave 2. Consumers also reported an increase in their intent to use CER to prepare for medical decisions, but the increase did not achieve statistical significance.

Interest in learning about the EHC Program increased over time from 44 percent in wave 1 to 58 percent in wave 2, and the increase was statistically significant. Consumers also reported an increase in intention to use AHRQ's products or other studies before a medical visit to inform decisionmaking from 39 percent in wave 1 to 45 percent in wave 2, although that increase was not statistically significant.

Wave 2 Clinician Survey Findings

• One-fifth (20 percent) of clinician respondents indicated awareness of CER when prompted with its name. A larger portion of clinicians indicated that they were more familiar with the term "PCOR" (49 percent) or "evidence-based medicine" (94 percent) than CER.
• The most common sources of exposure to CER were an article in a medical or science journal (24 percent); conference or professional meetings (17 percent); a continuing education course (11 percent); and colleagues (9 percent).
• Over one-third (38 percent) of the clinicians were aware of AHRQ. The majority of clinicians (88 percent) were unaware of the EHC Program. However, 17 percent of respondents reported that they had heard of the EHC Program's Web site.
• Awareness of EHC Program products was generally high among respondents who indicated awareness of the EHC Program; at least 40 percent of respondents had heard of each of the ten EHC Program products that the questionnaire listed, although fewer clinicians reported having ever read or used them.
• Slightly over half (54 percent) indicated that they were interested in learning more about CER. Similarly, over half (57 percent) indicated that they were interested in learning more about the EHC Program.
• The majority of clinicians indicated that they share educational materials with some (38 percent), most (39 percent), or all (12 percent) of their patients. The most common reasons for why they do not discuss treatment options included: patients' existing awareness of treatment options (18 percent); lack of time (16 percent); concern that patients will expect the clinician to know the "best" treatment (14 percent); patients being overwhelmed by the amount of information (14 percent); and concern that patients will have difficulty understanding the treatment options (14 percent).
• Slightly over half (56 percent) of the clinicians reported that they had seen, read, or heard messaging that encourages patients to explore and compare their treatment options with their doctors. Among those who reported exposure to such information, just over one-third (39 percent) reported having seen these messages in the last month.
• Approximately half of the clinicians surveyed indicated that they are likely to use EHC Program consumer (47 percent) and clinician summaries (50 percent) in the next year, while approximately ten percent were unlikely to use either product.

Longitudinal Clinician Survey Findings

The longitudinal analysis comparing wave 1 to wave 2 suggests increases in awareness of and interest in CER, AHRQ, and the EHC Program among clinicians. Although changes in consumers' awareness of CER did not reach statistical significance, aided awareness increased from 18 percent in wave 1 to 20 percent in wave 2.

Clinicians reported statistically significant increases in both awareness of AHRQ and the EHC Program from wave 1 to wave 2. Thirty-three percent of clinicians indicated awareness of AHRQ in wave 1 and 38 percent reported awareness in wave 2. Likewise, the percent of clinicians reporting awareness of the EHC Program grew from eight percent in wave 1 to 12 percent in wave 2.

While awareness of CER, AHRQ, and the EHC Program increased, clinicians' knowledge and understanding decreased, although the changes were not statistically significant. Clinicians reported a slight decrease in the average CER Knowledge Score from 5.37 (out of 11) in wave 1 to 5.22 in wave 2. Clinicians' EHC Program Knowledge Score also declined from 6.39 (out of 11) in wave 2 to 6.32 in wave 2.

Like awareness, the team observed increases in clinicians' interest in learning more about CER and the EHC Program, although the changes were not statistically significant. Scores indicating clinicians' interest in learning more about CER increased slightly from 3.57 (out of 4) in wave 1 to 3.58 in wave 2. Clinicians also reported an increase in their interest in learning more about the EHC Program. In wave 1, clinicians reported a score of 3.61 (out of 4); in wave 2, the score grew slightly to 3.63.

Use of the EHC Program Web site among clinicians increased from wave 1 to wave 2 and the change was statistically significant. Twenty-eight percent of clinicians reported that they had previously visited the EHC Program Web site in wave 1; 41 percent reported visiting the Web site in wave 2. Clinicians also reported a slight increase in their intention to use EHC Program clinician products in the near future, although the change was not statistically significant. Fifty percent of clinicians reported intention to use EHC Program clinician products in wave 1; this figure increased by 0.1 percentage point in wave 2.

¹ In the consumer questionnaire, rather than using the terms "CER" or "PCOR," the survey instruments refer to the "concept of evaluating treatment options." In the clinician questionnaire, the terms "comparative effectiveness research" and "patient-centered outcomes research" were used, but their abbreviations were not. For expediency, in this report we refer simply to "CER."

Return to Contents

Proceed to Next Section