James P. Scanlan, Attorney at Law

Illogical Premises

(May 6, 2010; rev. Jan. 9, 2012)

This page is related to the Subgroup Effects, Inevitability of Interaction, and Explanatory Theories sub-pages of the Scanlan’s Rule page of jpscanlan.com.

The Measuring Health Disparities (MHD) and Scanlan’s Rule pages of jpscanlan.com and the materials they make available explain that the overwhelming majority of efforts to appraise the size of differences between outcome rates in the law and the social and medical sciences are fundamentally flawed for failure to recognize that, for reasons inherent in normal risk distributions, standard measures of differences between outcome rates are tend to be systematically affected by the overall prevalence of an outcome.  The most notable of the patterns of correlations between standard measures of differences between rates and the prevalence of an outcome is that whereby the rarer an outcome the greater tends to the relative difference in experiencing it and the smaller tends to be the relative difference in avoiding it.  Absolute differences and odds ratios tend also to be affected by the overall prevalence of an outcome, though in more complicated ways.  Roughly, as uncommon outcomes become more common absolute differences tend to increase; as common outcomes become even more common absolute differences tend to decrease.  Differences measured by odds ratios tend to change in the opposite direction of absolute differences between rates.  Also, at least outside a range defined by a 50% rate for either group, absolute differences tend to track the smaller of the two relative differences, while differences measured by the odds ratio tend to track the larger of the two relative differences.  See Section E.7 of MHD regarding the extent of scholarly agreement with these views.  See also the Truncation Issues sub-page of the Scanlan’s Rule page regarding variations within subpopulations that are truncated portions of a larger population. 

Putting the above patterns aside, however, there exists a substantial body of research that is based on an illogical premise.  Most research relies on the relative difference between outcome rates as a measure of association.  And those who employ such measure frequently study some variation on the issue of whether two groups have been differentially affected by some factor.  The premise for such analyses is that even when two groups have different base rates of experiencing some outcome it is somehow normal that a factor that causes those rates to change would cause the rates to change in the same proportionate amount.  When two groups with different base rates of experiencing an outcome undergo the same proportionate change in those rates, the relative difference between the rates at which the two groups experience the outcome is the same after the change as before the change.  Thus, sometimes the above-described premise as to what is normal is framed in terms of the supposition that the relative difference between the two rates would remain the same. 

The premise is probably most explicit in the study of subgroup effects (also termed effect heterogeneity or modification) where the failure to observe equivalent relative risk reductions (or increases) across different baseline rates is considered such an effect.  The most important implication of the premise involves the application of a relative risk reduction observed as to one baseline rate to different baseline rates for purposes of deriving the clinically relevant absolute risk reduction and number needed to treat in order to prevent a single event.  See the Subgroup Effects sub-page.  But the explicitness of the premise that proportional changes are somehow normal, and the implications of the reliance on that premise, will vary from setting to setting and study to study, and there may be some cases where the premise arguably is not present at all.  But there can be little doubt that research aimed at identifying circumstances where proportional changes do not occur is conducted with the belief that there is something meaningful to be learned when a factor leads to different proportional changes in the baseline rates of two groups. 

But the premise that, absent a meaningful differential effect, a factor will cause equal proportionate changes in two different base rates is an illogical one for the simple reason that a factor cannot cause equal proportionate changes in two different base rates of experiencing an outcome while at the same time causing equal proportionate changes in the rates of avoiding the outcome (i.e., experiencing the opposite outcome).  That is, if Group A has a base rate of 5% and Group B has a base rate of 10%, a factor that reduces the two rates by equal proportionate amounts, say 20% (from 5% to 4% and from 10% to 8%) would necessarily increase the opposite outcome by two different proportionate amounts (95% increased to 96%, a 1.05% increase; 90% to 92%, a 2.2% increase).  And since there is no more reason to expect that two group will experience equal proportionate changes in one outcome than there is to expect them to experience equal proportionate changes in the opposite outcome, there is no reason to regard it as somehow normal that the two groups will experiences equal proportionate changes in either outcome.  See the Inevitability of Interaction sub-page regarding why as a subgroup effect, or interaction, as the concept is currently understood, must always exist (as to one outcome or the other) when subgroups have different baseline rates for some outcome. [i] 

As it happens, for reasons discussed in the first paragraph and the sources it references, not only is there no reason to expect two groups to undergo equal proportionate changes either in experiencing an outcome or in avoiding the outcome, there exists a tendency for the group that undergoes the larger proportionate increase in one outcome to undergo the smaller proportionate decrease in the opposite outcome.[ii]  But, as suggested at the outset, this item is principally devoted to the illogical nature of the premise that equal proportionate changes in any outcome is somehow normal and the attaching of significance to departures from such pattern. 

Various types of research involve examining departures from proportionate changes in outcome rates, or variations on that theme, and which, for reasons just noted, may be deemed to be based on illogical premises.  These include:

(1) Studies of whether health or healthcare disparities, measured in terms of relative differences in adverse or favorable outcomes rates, have increased or decreased (or, a matter involving the same issues from a different perspective, whether different groups have undergone different proportionate changes in their rates of experiencing some adverse or favorable outcome).

(2) Studies of whether therapeutic interventions (or exacerbating factors) have the same proportionate effect on different groups or different outcomes – e.g., whether an intervention that reduces rates of experiencing some adverse outcome achieves equal proportionate reductions among women as among men, among minorities as among whites, among high-risk and low-risk groups; whether smoking increases some outcome more among one group than another or increases the rate of experiencing one condition more than another.   See generally the Subgroup Effects sub-page.

(3) Studies of whether health disparities are greater within one subpopulation than another (with subpopulations being defined by education, SES, or age group) or in one country than another. 

(4) Studies of whether health disparities are greater as to one outcome than another.

(5) Studies of heterogeneity in self-rated health among different groups (e.g., whether poor self-rated health increases mortality more among one SES group than another; whether some factor increases poor self-rated health more among one SES group than another).  See Reporting Heterogeneity sub-page of MHD

(6) Studies of whether racial groups experience the same improvements in rates of meeting some academic achievement or proficiency standard.

(7) Studies of whether following a change in the overall prevalence of an outcome a group that had comprised a disproportionate part of the population experiencing the outcome comprises an even more (or less) disproportionate part (e.g., whether following a decline in poverty it has become more (or less) feminized).  See note ii.

The list could be much longer, as there are countless types of studies that involve the same premise. 

A question may arise as to why the premise discussed is deemed illogical while research that fails to recognize the patterns described in the first paragraph (or fails to recognize at least that the underlying distributions, whatever their actual shapes, are influencing the patterns of measures of difference as overall prevalence changes over time) is deemed merely to involve flawed reasoning.  There is probably no firm distinction between what is illogical and what is simply flawed reasoning.  But I here term illogical the premise that, absent the presence or occurrence of something meaningful, different base rates would undergo equal proportionate changes because of the fact that it is impossible for the two groups to undergo equal proportionate changes for rates of experiencing an outcome and equal proportionate changes in rates of failing to experience the outcome (along with the absence of any reason to expect equal proportionate changes in one outcome than in its opposite). 

That said, consider research that relies on absolute differences between rates as a measure of disparity, of which there is much in the healthcare disparities area.  In appraising the size of disparities before and after a general increase in some outcome rate, and concluding that the disparity was larger or smaller after the change (or concluding that a disparity is otherwise larger in one setting than another), researchers commonly rely on the premise that, absent the occurrence of something meaningful, two group with different base rates would undergo the same percentage point changes in their rates.[iii]  Whether or not one agrees that there exist reasons to expect the patterns of absolute difference changes discussed in the first paragraph, it would seem that thoughtful observers would recognize that absolute differences naturally will tend to be small both when an outcome is rare and when it is nearly universal and, between those points, will tend in some manner to grow larger and then smaller.  Even so, however, I would not be inclined to regard as necessarily illogical the premise that (again, absent the occurrence of something meaningful) two groups with different base rates would undergo the same percentage point changes, because, at least until the point where equal percentage point changes would cause one group’s rate to reach 100%, undergoing the same percentage point changes would not be impossible. 

Consider next the odds ratio.  There is least reason of all to regard it as illogical to expect that, absent the occurrence of something meaningful, two groups with different base rates would undergo equal proportionate changes in their odds of experiencing or avoiding an outcome.[iv]  A group's odds of experiencing or avoiding an outcome is a combination of the rate of experiencing the outcome and the rate of avoiding the outcome, which rates must change in opposite directions.  For that reason, there may even be a basis to expect that the odds ratio will be unaffected by changes in the overall prevalence of an outcome, as I first explored in note 1 of The Perils of Provocative Statistics (Public Interest, 1991).  And there exists a body of authority to the effect that odds ratios are unaffected by the overall prevalence of an outcome.[v]  But, as shown in the 1991 article and many places since then, differences measured by odds ratio tend also to be affected by the overall prevalence of an outcome (and to be so affected in the opposite manner of the absolute difference).  Thus, the odds ratio does not provide an effective means of identifying meaningful differences in associations or meaningful changes in differences between outcome rates.  But the expectation that, absent the occurrence of something meaningful, two groups would undergo the same proportionate changes to their odds of experiencing an outcome does not involve an element of impossibility or even any other obvious reason why the expectation is unsound.  See Table 3 of the Subgroup Effects sub-page regarding the way that the applications of reduction in odds  observed as to one baseline rate to different baseline rates yields an estimate of absolute risk reduction that is closer to a statistically sound estimate than estimates based on an assumption of a constant risk ratio either for experiencing the outcome or for avoiding the outcome.