AHRQ’s Vanderbilt Report
(March 29, 2013)
While the National Center for Health Statistics has at least recognized that the determining whether health and healthcare disparities have increased or decreased will commonly turn on whether one examines relative differences in favorable outcomes or relative differences in adverse outcomes, the Agency for Healthcare Research and Quality has yet to produce anything reflecting an awareness that different measures may yield different conclusions. The dearth of attention given to such issues by AHRQ is perhaps best reflected by the fact in a 2003 commentary on the Journal of the American Medical Association, AHRQ’s officials discussed as article that relied on absolute differences between rates as a measures of disparities as showing the way that improvements in healthcare tend to reduce disparities. In the National Healthcare Disparities Report, however, AHRQ relies on the larger of the relative differences (in the favorable or the adverse outcome), which relative difference tends to change in the opposite direction of absolute differences as rates of appropriate healthcare increase. So researchers who rely on absolute differences between rates tend to reach opposite conclusions from those AHQR would reach.[i] Nevertheless AHRQ funds a great deal of healthcare disparities research where researchers rely on absolute differences between rates, without either the researchers’ or the AHRQ funding officials’ being aware that the researchers will tend to reach opposite conclusions from those AHRQ would reach. See Comment on Aaron and Clancy JAMA 2003, Measurement Problems in the National Healthcare Disparities Report (APHA 2007), and pages 19-21 and 29-32 of the Harvard University Measurement Letter.
In 2007 AHRQ entered into a contract with Institute for medicine and Public Health of the Vanderbilt University Medical Center (No, 290-2207-10065), broadly described as “Evidence-Based Practices Centers III,” which ultimately yielded a 475-page report, issued in August 2012, denominated “Evidence Report/Technology Assessment Number 2008,” and styled “3. Quality Improvement Interventions to Address Healthcare Disparities, Closing the Quality Gap: Revisiting the State of the Science.” The structured abstract of the report states as its objective: “The report evaluates the effectiveness of quality improvements (QI) strategies in reducing disparities in health and health care.” The report, which cites 4258 references, provides a great deal of information on findings of studies the way improvements in healthcare affected health and healthcare disparities.
The report, which was peer reviewed,[ii] shows no recognition whatever of the way the various measures may be affected by the prevalence of an outcome or even that various measures could yield different conclusions as to changes in disparities. In discussing various findings it makes no mention of the measure the researchers employed. Consequently, the extensive report provides no useful information about the way healthcare improvements affect health and healthcare disparities and may well provide a great deal of misleading information..
That is not any more reflection on Vanderbilt Medical Center than it is on the state of healthcare disparities research generally. As reflected in the Harvard University Measurement Letter, Harvard could have produced a similar report.
Nevertheless the report constitutes a great waste of resources. While the waste of resources involves with this contract is of the same nature of all other AHRQ funded research that fails to consider the ways measure tend to changes as healthcare improves, the scope of the research seemed to provide reason to learn how much the research cost. I sought such information in Freedom of Information Act request dated February 9, 2013. By letter of March 25, 2013, AHRQ responded with a copy of the contract. But AHRQ had redacted the minimum and maximum total amounts of the contract and the range of costs for task orders, citing FOIA exemption (b) (4) and noting that the provision “permits the withholding of commercial or financial information that was obtained from a person outside the government and that is privileged or confidential. The AHRQ response did, however, direct me to the Federal procurement Data System website (www.fpds.gov), which seems to indicate that the total amount of the contract was $5,970,037.00. I have not yet determined whether the information meets my needs or whether it will be necessary to appeal AHRQ’s decision to appeal the redactions.
[i] In 2003 AHRQ may not have decided on a method by which to measure disparities. But by 2013 AHRQ had still failed to reflect an awareness of that different measures might yield different results, much less that they would tend systematically to do so.
[ii] The eight peer reviewers included the six technical experts on the report. That would not seem a very good practice. In any case, the works of none of the reviewers/consultants reflects an understanding of the way standard measures of differences between outcome rates tend to be affected by the prevalence of an outcome. That, of course, is the state of research in the area save for the works discussed in Section E.7 of the Measuring Health Disparities page.