National Healthcare Disparities Report Measurement Issues
(Nov. 5, 2013; rev. Mar. 15, 2014, and June 11, 2020)
Material added June 11, 2020: The paragraph after the three asterisks was written in 2013 and last updated in March of 2014. At some point I may present a completer treatment of the measurement problems in the National Healthcare Disparities Reports (NHDR) (since 2013, National Health Quality and Disparities Reports (NHQDR)), possibly in conjunction with a follow-up to my July 1, 2015 letter to Agency for Healthcare Research and Quality (AHRQ) discussed below. Suffice it to say that reports and most of what Agency for Healthcare Research and Quality does with regard to the monitoring of health and healthcare disparities yields virtually nothing of value and a great deal that is misleading as a result of a failure to understand how measures of disparities tend to be affected by the prevalence of an outcome. The same may be said about the work of virtually all institutions studying health and healthcare disparities. But, given its role and putative expertise and its apparent continuing obliviousness to so many issues that its leadership actually examined, AHRQ can be regarded as epitomizing innumeracy of the health and healthcare disparities industry.
For the present, I merely add following references and points that pertain to the brief material after the asterisks:
(a) “Race and Mortality Revisited,” Society (July/Aug. 2014). This item discusses AHRQ in several places with regard to the way that the agency’s activities regarding the NHDR and otherwise are undermined by the failure to recognize the way that measures tend to be affected by the prevalence of an outcomes (including the failure to understand that it is even possible for relative differences in a favorable outcome and relative differences in the corresponding adverse outcome to change in opposite directions much less that the National Center for Health Statistics (NCHS) recognized that this tends to occur systematically). It also discusses the AHRQ’s funding of research when neither the researchers nor that AHRQ personnel monitoring it are aware that it is possible for different measures to yield different conclusions about such things as whether improvements in care increase or decrease demographic differences in healthcare outcomes. See especially the discussion of the Vanderbilt study at page 333 (also the subject of the AHRQ’s Vanderbilt Study subpage of the Measuring Health Disparities page) where over 4000 studies were examined to determine whether improvements in care reduced disparities without consideration of the measures used in the studies that were examined.
(b) Letter to Agency for Healthcare Research and Quality (July 1, 2015). This letter to the AHRQ director discusses, inter alia, the way that the 2012 NHDR reported as some of some of the largest reductions in disparities (based on the comparative size of absolute changes in rates of advantaged and disadvantaged group) situations where the AHRQ/NHDR would also regard the disparities to be much larger at the end of the period than at the beginning of the period (based on the comparative size of the relative differences between adverse outcomes at the end of the period and the beginning of the period). As discussed below, the NHDR continues the approach that leads to this absurdity, while AHRQ leadership and AHRQ staff and expert consultants involved in preparing the NHDR are unlikely aware that such an anomaly is even possible (even if they might have been aware of the issue in 2015).
(c) “The Mismeasure of Health Disparities,” Journal of Public Health Management and Practice (July/Aug. 2016). This article discusses that by reversing its recommendation that healthcare disparities be measured in terms of relative differences in adverse outcomes like nonreceipt of cancer screening and nonreceipt of immunization – and now would measure disparities in terms of the corresponding favorable outcomes – the National Center for Health Statistics repudiated a decade of National Healthcare Disparities Reports. See discussion below of the AHRQ response to that reversal.
One especially misleading aspect of NHDRs with respect to the discussion of favorable outcomes even though the NHDR is measuring relative differences in the adverse outcome involves the use of bar charts. Such charts ordinarily provide a way of graphically the size of illustrating relative differences (as well as absolute differences). But the NHDRs have used bar charts showing favorable outcomes even though the reports are measuring the disparity in terms of the relative difference in the adverse outcome. Thus, the bar chart would graphically illustrate the relative difference in the adverse outcome that the NHDR is actually measuring only if turned upside down.
The 2018 NHQDR nicely illustrate this anomaly in the several situations where it presents bar charts for (a) being uninsured all year and (b) having insurance. These indicators are very close to being simply an adverse outcome (i.e., (a)) and the corresponding favorable outcomes (i.e., (b)). And usually the figures will sum to within a percentage point or two of 100%. Other issues aside, even if there is value in reporting information on matters that are so similar, it is certainly a waste of space in the report to present charts on both indicators. Ironically, the reasoning that results in the presentation of information on matters that are so similar is that one is cast in adverse terms and the other is cast in favorable terms, even though both are actually measured in adverse terms. It could be compared to presenting information a disparity in something described as receipt of mammography and a disparity described in something described as nonreceipt of mammography, even though both are actually measured in terms of relative differences in nonreceipt of mammography.
But the 2018 NHQDR does this two times. And in doing so, as in Figures 29 and 30 (at 29-30) for American Indian/Alaska Native–white comparisons, it presents the bar chart for the adverse outcome for (a) and the favorable outcome for (b), thus leading readers to believe that both charts are providing useful, but different information. In fact, for all practical purposes, Figure 30 is the same as the Figure 29 turned upside down. But Figure 30 is misleading because it gives a false impression with regard to the size of the relative difference that the NHDR is actually measuring. The same points apply to Figure 31 and 32 (at 31).
While the July 1, 2015 letter to the director of AHRQ received some attention at the agency (as reflected in the July 13, 2015 response from AHRQ Medical Office Ernest Moy and the email exchange of which it was a part), in the 2018 NHQDR, AHRQ was continuing to measure health and healthcare disparities in terms of relative differences in adverse outcome while identifying changes in disparities on the basis of the comparative size of absolute changes in the advantaged and disadvantaged groups’ rates (which would be the same thing as measuring disparities in terms of absolute differences between rates). This is the reason why the 2012 report presented as some of the largest reductions in disparities over a particular period situations where AHRQ would regard the disparities to be much larger at end of the period than the beginning of the period.
Given the rate ranges at issue, as health and healthcare improve absolute differences tend to show opposite patterns of changes from relative differences in adverse outcomes. An example may be found in the reportage of the disparity between high income and poor people with regard to what is termed “Children who had their height and weight measured by a health provider within the past 2 years,” but which involves a comparison of the rates at which the groups failed to have height and weight measured. Appendix A (at A.3-4) shows a relative difference adverse to the poor of 137.5%, which is based on 2016 nonreceipt rates of 9.5% for the poor 4.4% for high income. The 137.5% relative difference is up from 108% in 2002, when the nonreceipt rates were 17.5% and 8.4%. This could also be cast as a smaller relative decreases in the outcome for poor people than the highest income people (as reflected in the smaller yearly decrease for the poor (4.52% at A.4-20) than high income people (5.11%, at A.4-36). The relative difference in the favorable outcome, however, was decreasing, as would also be indicated by larger relative increase in the favorable outcome for poor people than for highest income people. And the absolute difference was decreasing.
But whereas the Appendix reports the relative changes of each group’s rates for the adverse outcome –which, again, was larger for the highest income group than for poor persons, thus increasing the relative difference between the groups’ rates – the Appendix examines the absolute changes for each group’s rate for purposes of identifying directions of changes. And, thus, it finds a larger annual absolute change for the poor group (0.56 percentage points) than the high income group (0.3 percentage points). The only reason the report does not regard this to be a decrease in disparity (even though the relative difference in the adverse outcome is larger at the end of the period than at the beginning of the period) is that yearly difference between the two values is less than 1 percentage point.
The data in the underlying the report show many departures from the patterns I describe, in substantial part, it would seem, because disparities are in fact decreasing. But that does not excuse the continuing failure of AHRQ to understand that the measurement issue even exists.
A few other aspects of the 2018 report mention. The manner by which the report limits the identification of changes in disparities to situations where the differences between absolute change values are greater than 1.0 would be problematic even if the changes in absolute differences provided a sound means of determining whether disparities were increasing or decreasing. For one thing, as the number of years examined increases, there are fewer and fewer situations where the yearly 1.0 unit difference threshold is met. For example, consider the following situation: There occurs a dramatic reduction in disparity between 2001 and 2005, resulting in the reduction of what had been a very larger disparity in 2001 to very small disparity in 2005, and assume that, in examining the situation, the NHDR regarded there to have been a dramatic reduction in disparity as of 2005. But then there is little change between 2005 and 2017. As a consequence, the difference in yearly change values, now spread over 16 years, would no longer meet the 1.0 threshold. Thus, the NHDR would treat the matter as if there were no change in disparity between 2001 and 2017. In fact, however, there was a dramatic change since 2001. Such fact is not altered because there has been no change since 2005.
In addition, whereas in older reports that compared yearly annuals changes in the manner that led to the anomalies discussed with regard to the 2012 report, the difference between the absolute changes had to exceed 1 percentage point (though confusingly presented as 1%). In the 2018 report the absolute differences are cast in terms of the units by which a matter is discussed. Yet, solely as a matter of convention and principally for purposes of avoiding the interpretation of numerous zeros before a value in a percent (e.g., 0.0007%), rare outcomes are cast in terms of numbers of events per 1,000, per 10,000, per 100,000, or per million, while outcome that occur in more than 1% of cases are more commonly cast in terms of percents (i.e., numbers of events per 100). If the height and weight check rates were cast in term of events per 1000 (95 and 44 per thousand for failure to check height and weight in 2016), the difference in yearly changes would have met the threshold for regarding the disparity to be decreasing (even though, again, the report would have regarded the disparity to be larger at the end of the period than at the beginning of the period).
Thus, it should not be surprising that three of the four situations where the NHDR identifies changes in the black white disparities in Figure 64 (the improvements identified at A.3-80) involve situations where the indicator was measured in terms of events per 100,000 or per million. Further, the single situation where a change was reported when rates were presented in terms of percentages, “Hospital patients with heart attack given fibrinolytic medication within 30 minutes of arrival” – and where the appendix show 1.786 yearly percentage point decline in the adverse outcome rate for whites compared with a 3.585 yearly percentage point decline for blacks between 2005 and 2013 – the underlying data do not support the values shown in the appendix. The black adverse outcome rate declined from 72.3% to 48.6% over the period (or 2.963 percentage points per year) while the white adverse outcome rate declined from 61.3% to 44.1% (or 2.15 percentage points per year), which would seem not meet the threshold of a 1.0 percentage point difference between the two yearly declines.
This is a draft of an explanation that in 2010 the National Healthcare Disparities Report (NHDR) changed its method of analyzing health disparities by adopting a regression approach that determined directions of changes in health and healthcare disparities on the basis of which of two groups being compared experienced the larger change in its outcome rates. See page 6 of the 2010 NHDR. The change in approach was apparently intended to be consistent with the approach to measuring disparities in Healthy People 2010/2010, which measures all health and healthcare disparities in terms of relative differences in outcome rates. And the new NHDR approach would have been consistent with the Health People approach if the changes to be compared were relative changes in adverse outcomes. But apparently the changes in rates were measured in terms of absolute (percentage) differences. This would not be consistent with Healthy People 2010. In fact, as explained on many of pages of this site and perhaps most comprehensively in the October 9, 2012 Harvard University Measurement Letter, in the common situation where a relatively common favorable healthcare outcome increase in overall prevalence (and where relative differences in favorable outcomes would tend to decrease while relative differences in adverse outcomes would tend to increase, as explained on many pages of this site), absolute differences would tend to change in the same direction as the relative difference in the favorable outcome and the opposite direction of the relative difference in the adverse outcome. Some examples from the 2012 NHDR are presented in Table 5 of my 2013 Federal Committee on Statistical Methodology Research Conference presentation “Measuring Health and Healthcare Disparities.” The table shows situations the 2012 highlights as some of the fastest improving healthcare disparities where the National Center for Health Statistics (and Health People 2020) would have found increasing disparities. This matter is also the subject of the conference paper (at 27-28).