Prefatory note added July 30, 2014: This subpage, which contains material also appearing as Section E.7 of the Measuring Health Disparities page, describes the scholarly responses to (treatments of) my descriptions of the patterns by which measures of differences between outcome rates tend to be systematically affected by the prevalence of an outcome. The material is included here to facilitate reference to it in “Race and Mortality Revisited,” Society (July/August 2014). The items discussed below were principally directed to my “Can We Actually Measure Health Disparities?,” Chance (Spring 2006) (guest editorial), “Measuring Health Disparities,” J Public Health Manag Pract (May/June 2006) (letter), “Race and Mortality,” Society (Jan./Feb. 2000) (reprinted in Current, Feb. 2000), “Divining Difference,” Chance (Spring 1994), and “The Perils of Provocative Statistics,” Public Interest (Winter 1991). Because the material below was pulled from the Measuring Health Disparities page, it continues to contain some references to other materials on that page, including parenthetical references in terms of the numbering system of that page.
***
The references listed in Section A to D of the Measuring Health Disparities page take for granted that the views they express as to the correlations between prevalence of an outcome and binary measures of difference are correct. I expect that there will eventually be universal acceptance, if not of the precise views I have been expressing since 1987, at least of the fact that the standard measures of differences between outcome rates are in some manner affected by overall prevalence and, hence, that none of those measures can provide useful information regarding the comparative situation of two groups without taking overall prevalence into account. But even in the countries where issues of the relationships between overall prevalence of an outcome and various measures of experiencing or avoiding it have lately been given a good deal more attention than in the United States, it may be some time before there will be widespread, much less universal, understanding of these issues.
Set out below is the discussion of the scholarly work that has in responded somewhat directly to the issues I have raised about the way standard measures of differences between outcome rates tend to be affected by the prevalence of an outcome.
(a) National Center for Health Statistics: In a 2005 monograph “Methodological Issues in Measuring Health Disparities (item a1 below), citing “Race and Mortality” (Society 2000), the National Center for Health Statistics observed: “Conclusions about changes in disparity over time also depend on whether an indicator is expressed in terms of favorable or adverse events.” The reliance on “Race and Mortality” (and the examples the monograph used) would seem to reflect recognition that relative differences in favorable and adverse events tend to change in opposite directions because of the pattern whereby the rarer an outcome the greater tends to be the relative difference in experiencing it and the smaller tends to be the relative difference in avoiding it. But the NCHS decision to deal with the matter simply by recommending that all health and healthcare disparities be measured in terms of relative differences in adverse outcomes is utterly inconsistent the point of “Race and Mortality” that, because the two relative differences tend to be affected by the prevalence of an outcome changes, neither relative difference is a useful indicator of whether a health disparity in increasing or decreasing without taking implications of overall prevalence into account (a point I repeatedly made to the lead author of the monograph, Kenneth Keppel, as NCHS was developing that position and as Keppel and Jeffrey Pearcy were drafting an article as a more direct response to “Race and Mortality” and “Divining Difference” (Chance 1994).
When Keppel and Pearcy published their article (item a2 below), it appeared to regard the patterns I described as if they were phenomena I had observed regarding particular outcomes rather than as systematic patterns stemming from properties of the underlying risk distributions. See my Comment on Keppel and Pearcy (item D.6 above). Keppel and Pearcy’s reply to this comment (item a3 below) is somewhat cryptic (though it does reflect some appreciation that the crucial issue involves what is occurring with the underlying distributions, something that, as discussed below, so far only Carr-Hill/Chalmers-Dixon, Gallestey, Gansky, and Lambert/Subramanian seem to have recognized). On the panel at the 7th International Conference on Health Policy Statistics at which I presented reference B13 (ICHPS 2008), Keppel (in presenting item a4 below) expressed the view that the points I have expressed in B13 and elsewhere, while correct with respect to cross-sectional data, are not correct with respect to longitudinal data, stating that patterns observed over time are not the same as those observed when a cutoff is raised or lowered. But, while patterns observed over time will rarely if ever be precisely those illustrated by the lowering or raising of a cutoff, they will almost invariably exhibit similar tendencies. And it makes no sense to make comparisons over time without consideration of such tendencies.
Keppel and Pearcy addressed this matter further in a 2009 Chance article, simply observing:
“James Scanlan, in his 1994 CHANCE article "Divining Difference," pointed out that the relative difference in rates of survival between black and white infants decreases as the relative difference in rates of infant mortality increases. When disparities are measured in HP2010, indicators are usually expressed in terms of adverse events so meaningful comparisons can be made across indicators.”
But Keppel and Pearcy offered no justification for the approach and ignored entirely the criticisms I have repeatedly made of the NCHS approach, including in the 2006 Chance editorial “Can We Actually Measure Health Disparities.”
Kenneth Keppel also circulated within NCHS a document titled “Response to Jim Scanlan,” which he shared with me by email of May 22, 2008. My May 22, 2008 email responding to Keppel may be found here. The internal document appears to challenge my contention that the prevalence of an outcome affects relative difference in experiencing it, asserting that changes in relative differences over time, rather than being a function of changes in prevalence, are in fact functions of the comparative proportionate changes in each group’s rate. But I had made clear both in “Divining Difference” and “Race and Mortality,” and it should be evident in any case, that (a) a certain pattern of proportionate changes in different baseline rates as prevalence change (i.e., the group with the lower baseline rate for the outcome will tend to experience a larger proportionate change in the outcome while the other group will tend to experience a larger proportionate change in the opposite outcome) and (b) the patterns I describe whereby the two relative differences tend to change in opposite directions as the prevalence of an outcome changes are simply corollaries to one another. Keppel’s positing a situation where the patterns of proportionate changes operate contrary to the patterns I described, with corresponding changes in relative differences contrary to the patterns I have described, is simply a demonstration that the patterns I describe do not have to occur. But that they do not have to occur – and indeed that many times they in fact do not occur – is something I have pointed out numerous times. That such patterns may not be observed in a particular situation does not provide a reason to ignore the implications of the patterns or, in situations where they are observed, provide a basis to maintain that some meaningful change in the comparative situation of two groups has occurred and that it has occurred in the direction indicated by the particular relative difference the observer happens to prefer.
If NCHS is to provide useful guidance in this area, it must directly address these issues. Specifically, it must address whether relative differences in adverse and favorable outcomes (as well as other measures) tend to be affected by the overall prevalence of an outcome and, if so, whether those measures can provide useful information about whether disparities are changing over time or are otherwise larger in one setting than another if prevalence is not in some manner taken into account. And the longer it takes for NCHS to address the issues, the greater will be the amount of flawed research for which NCHS will be in part responsible.
(Subsequent to the last material modification of the consensus materials, I further criticized the NCHS failure to responsibly address health and healthcare disparities measurement issues in quite a few places, including an October 9, 2012 letter to Harvard University (at __), a Federal Committee on Statistical Methodology 2013 Research Conference paper “Measuring Health and Healthcare Disparities” (especially at 11-12 and Sections C.1.a-c, at 26-28), and my July/Augst 2014 Society article “Race and Mortality Revisited” (especially the section titled “Response of the National Center for Health Statistics to “Race and Mortality”)).
(b) Carr-Hill and Chalmers-Dixon. In the 2005 Southeast Public Health Observatory Handbook of Health Inequalities Measurement, relying on my 2001 presentation on these issues in Oslo (B1), Carr-Hill and Chalmers-Dixon (item b below) (at 171-72), explicitly accepted the reasoning of that presentation with regard to relative differences, including a recognition of the way the statistical patterns are functions of the properties of the underlying distributions. They noted:
“[I]f inequalities in health were to be measured in terms of the numbers and proportion who survive rather than the numbers and rates of death, the picture is very different (Part (b) of Table 11.4). The point is that as a negative outcome becomes more rare, it is more and more likely to occur disproportionately among the less advantaged groups. Conversely, as a valued outcome becomes relatively rarer, it is likely to be concentrated among the elite. This is a simple consequence of the statistical distributions [citing B1], rather than another example of inequalities.”
But the lengthy document, which discusses a variety of health disparities issues and measurement techniques, seems not to recognize the implications of such acceptance as to other measures it discusses. In my view, the acceptance calls into question much of the reasoning in the remainder of the document. For the patterns described here affect each of the measures discussed in the Carr-Hill/Chalmers-Dixon document.
(c) Houweling et al. A 2007 article by Houweling et al. (item c below) is the article mentioned in Section E.2 as one co-authored by two of the authors of the 1997 Lancet article that calls the conclusions of the Lancet article into question. See also B51a (Comment on Mackenbach Lancet 1997). The Houweling article is in part a response to Race and Mortality (A10) and questions Race and Mortality for overstating the force of the tendencies it describes. The Houweling article ignores entirely Race and Mortality’s discussion as to why certain patterns will tend to be observed and why in some cases they will tend not to be observed – in particular the explanation that differences between rates will be a function of (a) the prevalence of the outcome and (b) the size of the difference between the underlying distributions. But the Houweling article nevertheless finds systematic correlations between the prevalence of an outcome and relative differences in experiencing it and avoiding it that are the same as those described in Race and Mortality. Houweling et al. were unaware of the 2006 Chance editorial, Can We Actually Measure Health Disparities? (A12) and other treatments of absolute differences between 2005 and 2007 (see Section E.3 of MHD), but their study found systematic correlations between absolute differences and the prevalence of an outcome according to the same reverse U-shaped pattern illustrated or described in the references listed in Section E.3. However, the Houweling article suggested that the odds ratio would avoid the problems arising from the correlations it describes. In my view, as discussed in the Section E.3 references, the odds ratio does not avoid such problems, because differences measured in odds ratios tend also to be correlated with the prevalence of an outcome (according to a pattern that is the opposite of that exhibited by absolute differences). Also, in my view, the explanations Houweling et al. offer for the observed patterns are less sound than those described in Race and Mortality and many other places listed on this page. Nevertheless, as with my own treatments of these issues, the Houweling article raises questions as to the validity of the overwhelming majority of health disparities research to date.
Because the Houweling authors included two of the most prominent authorities on health inequalities measurement (Johann P. Mackenbach and Anton E. Kunst), the article is potentially of great significance. But those authors have gone on to do subsequent work in ways that, according to their own reasoning in Houweling et al., is flawed for failure to consider the implications of overall prevalence. See items D70 (Comment on Mackenbach NEJM 2008) and D113 (Comment on Mackenbach BMJ 2011).
(d) Eikemo et al. Whereas the Houweling article responded to Race and Mortality without the authors’ evidencing an awareness of the 2006 Chance editorial (A12), a 2009 article by Eikemo et al. (item c2 below) in the same journal responded specifically to the Chance editorial, while also discussing the Houweling article. Like Houweling et al., Eikemo et al. found correlations between relative differences and prevalence of an outcome and concluded that overall prevalence must be taken into account in interpreting relative differences. Like the Houweling article, however, the Eikemo article offered little guidance on how to do that. Also like the Houweling article, the Eikemo article failed to consider the reasons for the observed patterns that had been explained at some length in Race and Mortality and varied other places.
Item D72 is a comment on Eikemo et al. and somewhat on Houweling et al. It was submitted to the International Journal for Equity in Health at the beginning of October 2009. The journal originally was to publish it upon receiving comments from Eikemo et al. and Houweling et al. but decided it could only publish a much shorter version (and is awaiting my submission of the shorter version). The comment stresses the critical point that, as noted above, observed patterns are functions of (a) the prevalence of the outcome and (b) the differences in the underlying distributions, as had been made clear enough in Race and Mortality. Yet, in contrast to Carr-Hill and Chalmers-Dixon and Gallestey, neither Houweling et al. nor Eikemo et al. seem to recognize these forces. See also D79 (Comment on Huijts and Eikemo EJPH 2009), which discusses an article by Huijts and Eikemo (item c3 below) that alludes to the patterns I have described as a potential explanation for variations in health inequalities across countries.
(e) Bauld et al. A 2008 article by Bauld et al. (item d below) devotes several paragraphs to discussion of my treatments of the pattern whereby the rarer an outcome, the greater tends to be the relative difference in experiencing it and the smaller tends to be the relative difference in rates of avoiding it. They caution that, assuming the correctness of my reasoning, “[i]f governments fail to take account of ‘Scanlan’s rule,’ they run the risk of guaranteeing failure, largely for conceptual and methodological reasons rather than social welfare reasons.” But the authors do not make clear whether they wholly accept the reasoning or not, and it is not clear whether they think that my discussions of these patterns are based solely on observations without regard to the identification of the factors underlying the patterns. Further, they find among “real-world examples that resonate with Scanlan’s arguments” a study by Gisselmann finding that part of the reason for increasing social inequalities in birth outcomes in Sweden “’is likely to be found in the decline in the proportion of women with low education.’” But the pattern identified by Gisselmann raises a compositional issue akin to that addressed in introductory section of D43 (Comment on Boström). (Other issues raised in the Gisselmann article are discussed at pages 13-14 of B7 (BSPS 2006).) The compositional issue addressed by Gisselmann involves a matter quite different from the implications of what Bauld et al. term “Scanlan’s rule,” which implications are present without regard to compositional changes. Finally, the authors conclude by recommending that governmental entities in the United Kingdom continue to establish and monitor health inequalities reductions goals. But if one accepts my reasoning, there is no purpose in creating or monitoring goals until one has effectively addressed the measurement issue.
(f) David Mechanic. In several articles since 2002 (items e1 to e3 below), David Mechanic has discussed or cited A3 (Public Interest 1991) or A12 (Chance 2006). Referencing illustrations of large relative differences where outcomes are rare, and increasing relative differences corresponding with declining absolute differences, he has observed that the former have been overemphasized and the latter provide more valuable information as to the public importance of a disparity. On the basis of what he has written, however, I cannot say whether he fully agrees with my reasoning regarding the failure of either measure to provide useful information as to whether a disparity is increasing or decreasing in a meaningful sense.
(g) Jorge Bacallao Gallestey. In two 2007 articles (items f1, f2) Jorge Bacallao Gallestey cited A12 (Chance 2006) and, like Carr-Hill and Chalmers-Dixon, recognized the distributional basis for the patterns whereby relative differences in one outcome and its opposite would change systematically in opposite directions as the outcome changed in prevalence. See especially Table 2 in item f1. But, as was also the case with Carr-Hill and Chalmers-Dixon, other parts of these articles seem not to reflect recognition of the implications of that recognition as to other measures the articles describe.
(h) Stuart Gansky. In a 2009 presentation at the Joint Statistical Meetings (JSM) (item g below) Stuart Gansky cited A12 (Chance 2006) for the proposition that a useful measure of health disparities must not change when there occurs a simple overall change in prevalence. He then presented results of study of how health disparities indexes “are affected by underlying factors conditional on prevalence difference (i.e. probit model),” stating (in the abstract)
[Health Disparities Indexes] were estimated for the California Oral Health Needs Assessment 2004, a complex survey, to assess associations with untreated caries. Absolute measure, slope index of inequality, relative index of inequality (RII-mean), and absolute concentration index (ACI) were prevalence invariant with constant prevalence difference in a probit simulation model; relative measures depended on prevalence.
I am unfamiliar with the detail of the analysis. But inasmuch as a probit model appears to achieve the same results as the approach described on the Solutions sub-page of the Measuring Health Disparities (see introductory note to the Solutions sub-page and January 9, 2010 Follow-up Comment on Morita) it may well satisfy the criterion of remaining invariant as overall prevalence changes. Gansky and his colleagues may soon publish a paper based on the JSM presentation and that should shed further light on his technique.
(i) Cristina Masseria. In a 2009 article (item h below) on health disparities measurement issues, Christina Masseria relied on A12 (Chance 2006) for the proposition that “caution is needed when interpreting the results of relative inequalities since they increase (decrease) as a consequence of a decrease (increase) in the overall level of mortality (survival).” That seems almost to capture the point, though failing to recognize that the pattern is a tendency that must be interpreted with a regard for the factors underlying it. In any case, she goes on to state: “Houweling et al. [item c1 here] have shown that low levels of mortality can be achieved together with low levels of health disparities. Indeed, a recent US study shows that there is no clear relationship between level of health and relative inequalities.” Thus, in my view, Masseria, like Houweling et al. and Eikemo et al. (and Huijts and Eikemo and Remes et al. (see below)), fails to recognize the forces that are invariably at work, as discussed above with regard to the Houweling and Eikemo articles, and that the function of health disparities research is to identify patterns of health disparities that are not functions of overall prevalence.
(j) Remes et al. In a 2009 article (item i below), Remes et al., citing Race and Mortality, state:
Scanlan [] argues that all measures of differences between binary outcomes are affected by the overall prevalence of an outcome and changes in it. For example, declining mortality tends to be accompanied by increasing relative differentials. Departures from the expected patterns of change are still possible, as demonstrated by Houweling et al [item c1] in a cross-national comparison study.
While not necessarily quarreling with Race and Mortality, this article is similar to several of the other above-discussed items in failing to reflect a recognition of the forces underlying the observed patterns that Race and Mortality described, as reflected by the discussion throughout the remainder of the article. See D13 (Comment on Edwards), D2 (Comment on Shaw), D39 (Comment on Singh) responding, respectively, to references 12, 13, and 14 of Remes et al.
(k) Peter J. Lambert and Subbu Sabramanian. In a 2014 article (item k.1) Peter J. Lambert and Subbu Sabramanian discuss the pattern of relative differences termed “heuristic rule X in “Can We Actually Measure Health Disparities?” The authors “explore the issue analytically, providing a criterion from stochastic ordering theory under which one of two groups is considered disadvantaged and the other advantaged, and showing that Scanlan’s heuristic obtains as a rigorous finding in such cases.” Item k.2 is a subsequent working paper that is a pretty strong affirmation of the existence of the pattern whereby the two relative differences tend to change in opposite directions as the prevalence of an outcome changes and the importance of understanding it.
(() PRISMA-Equity Bellagio Group. In 2012 the PRISMA-Equity Bellagio Group published guidelines on reporting of systematic review with a focus on health equity in PloS Medicine which editors provided a comment. I posted a comment on the editorial noting, inter alia, that, while the authors stressed the importance of reporting both relative and absolute differences in outcome rates, they showed no awareness that there existed two relative differences or that the two relative differences to change in opposite direction as the prevalence of an outcome changes. Pursuant to the editors’ request, the authors of the guidelines responded to my comment on the editorial. The response appeared to reflect recognition of the importance of the issues I raised and stated that such issues will be addressed in a forthcoming “Elaboration and Explanation document of the PRISMA-Equity 2012.”
Items by authors referenced in Section E.7 are:
a.1. Keppel K., Pamuk E., Lynch J., et al. 2005. Methodological issues in measuring health disparities. Vital Health Stat 2005;2 (141): http://www.cdc.gov/nchs/data/series/sr_02/sr02_141.pdf
a.2. Keppel KG, Pearcy JN. Measuring relative disparities in terms of adverse events. J Public Health Manag Pract 2005;11(6):479–483
a.3. Keppel K.G., Pearcy J.N. 2006. Response to Scanlan concerning: measuring health disparities in terms of adverse events. J Public Health Manag Pract 12(3):295:
http://www.nursingcenter.com/library/JournalArticle.asp?Article_ID=641472
a.4. Keppel KG. Measuring Disparities in Health People 2010, presented at the 7th International presented at the 7th International Conference on Health Policy Statistics, Philadelphia, PA, Jan. 17-18, 2008 (invited session).
a.5. Keppel KG, Pearcy JN. Healthy People 2010: Measuring Disparities in Health. Chance 2009;22(1)_____.
b. Carr-Hill R, Chalmers-Dixon P. The Public Health Observatory Handbook of Health Inequalities Measurement. Oxford: SEPHO; 2005: http://www.sepho.org.uk/extras/rch_handbook.aspx
c.1. Houweling TAJ, Kunst AE, Huisman M, Mackenbach JP. Using relative and absolute measures for monitoring health inequalities: experiences from cross-national analyses on maternal and child health. International Journal for Equity in Health 2007;6:15: http://www.equityhealthj.com/content/6/1/15
c.2. Eikemo TA, Skalicka V, Avendano M. Variations in health inequalities: are they a mathematical artifact? International Journal for Equity in Health 2009;8:32: http://www.equityhealthj.com/content/pdf/1475-9276-8-32.pdf
c.3. Huijts T, Eikemo TA. Causality, social selectivity or artefacts? Why socioeconomic inequalities in health are not smallest in the Nordic countries. Eur J Pub Health 2009;19:452-53
d. Bauld L, Day P, Judge K. Off target: A critical review of setting goals for reducing health inequalities in the United Kingdom. Int J Health Serv 2008;38(3):439-454: http://baywood.metapress.com/app/home/contribution.asp?referrer=parent&backto=issue,4,11;journal,7,157;linkingpublicationresults,1:300313,1; http://www.britannica.com/bps/additionalcontent/18/33140632/Off-Target-A-Critical-Review-of-Setting-Goals-for-Reducing-Health-Inequalities-in-the-United-Kingdom
e.1. Mechanic D. Disadvantage, inequality and social policy. Health Affairs 2002;21(2):48-59;
e.2 Mechanic D. Who shall lead: Is there a future for population health? J Health Politics, Policy and Law 2003;28(2):421-442.
e.3 Mechanic D. Population health challenges for science and society. Milbank Quarterly 2007;85(3):553-559.
f.1. Gallestey JB. Two controversial topics in the context of health inequality measurement. Rev. Cuban Public Health 2007;33(3): http://bvs.sld.cu/revistas/spu/vol33_3_07/spu18207.htm
f.2. Gallestey JB. Indicadores basados en la noción de entropía para la medición de las desigualdades sociales en salud Indicators based on the notion of entropy for the measurement of social inequalities in health. Rev. Cuban Public Health 2007; 33 (4):http://bvs.sld.cu/revistas/spu/vol33_4_07/spu07407.html
g. Gansky S. A Simulation Study of Health Disparity Indexes: How Do They Depend on Prevalence? presented at 2009 Joint Statistical Meetings of the American Statistical Association, International Biometric Society, Institute for Mathematical Statistics, and Canadian Statistical Society, Washington, DC, Aug. 1-6, 2009: http://www.amstat.org/meetings/jsm/2009/onlineprogram/index.cfm?fuseaction=abstract_details&abstractid=304602
h. Masseria C. Health inequality: Why is it important and can we actually measure it? Eurohealth 2009;15(3):4-6: http://www.euro.who.int/Document/Obs/Eurohealth15_3.pdf
i. Remes H, Martikainen P, Valkonen T. Mortality inequalities by parental education among children and young adults in Finland 1990e2004. J Epidemiol Community Health 2010;64:136-141.
k.1. Lambert PJ, Subramanian S. 2014. Disparities in Socio-Economic outcomes: Some positive propositions and their normative implications. Soc Choice Welf, DOI 10.1007/s00355-01400794-
7. http://link.springer.com/article/10.1007%2Fs00355-014-0794-y#page-1
(an earlier version of this item is available open access at http://www.ecineq.org/milano/WP/ECINEQ2012-281.pdf)
k.2. Lambert P, Subramanian S. 2014. Group inequalities and “Scanlan’s Rule”: Two apparent conundrums and how we might address them. Working Paper 84/2014, Madras School of Economics.
l. Welch v. et al. Reply to Mr. Scanlan. PLoS Med Jan. 10, 2013: http://www.plosmedicine.org/annotation/listThread.action?root=60047