James P. Scanlan, Attorney at Law

Spurious Contradictions

(Nov. 8, 2013)

This page is related to many pages on this site that discuss the way researchers employ one measure of a difference between outcome rates without recognizing that other measures would tend to yield an opposite conclusion about such things as (a) whether a disparity is increasing or decreasing over time or which subgroup is most affected by a factor or (b) that one or more other measures in fact yielded a different conclusion in the situation being examined. This page is somewhat related to the Statistical Significance Vig subpage of the Vignettes page of the site, which discusses the way observers mistakenly regard the nonsignificant effect in the same direction as a significant effect as conflicting with the significant effect. 

A good example of the implications of the failure to consider that other measures could or would yield contrary conclusions may be found in Section C of the Harvard University Measurement Letter, which discusses the fact that a study that would have found that an incentive program reduced racial disparities in CABG rates if had relied on relative differences in receipt of CABG as a measure of disparity, but that instead relied on absolute differences and hence found an increase in disparity.  Table 7 and 8 of my 2013 Federal Committee on Statistical Methodology Research Conference presentation illustrates the contrast. 

The following example, which relates to two studies of changes in disparities in relatively uncommon medical procedures and diagnostics among Medicare recipients during periods when such outcomes were generally increasing, presents a comparable contrast.  But it is the manner in which the seeming differences in the findings of the studies would be subsequently appraised that is of special interest. 

The Escarce and McGuire APHA 2004 Study. 

A study published in American Journal of Public Health in 2004, Escarce and McGuire,[i] examined changes in racial in procedures/diagnostics between 1986 and 1997.  Relying on relative differences between rates of receiving such procedures or diagnostics a measure of disparity, a common enough approach at the time, the authors found that disparities generally decreased. 

Table 1 is based on the 12 situations addressed in Escarce and McGuire where there occurred the common pattern whereby rates of the advantaged and disadvantaged group were both increasing and where a disparity adverse to the disadvantaged group existed at both the beginning and the end of the period examined, limited to the situations where the authors found a statistically significant change.  The table presents counts for the combinations of changes in directions of the two relative differences and the absolute difference.

Table 1.  Counts of Combinations of Directions of Changes of Racial Disparities According to Various Measures Based on Escarce and McGuire APHA 2004.

The table shows that in 9 of the 12 cases, each of the three measures changed in the way prevalence-related forces commonly operate.  That is, as an outcome increased in overall prevalence (in this case the favorable outcome), (a) the relative difference between rates of experiencing the outcome decreased and relative difference in failing to experience the outcome increased (a pattern that exists when an outcomes increase regardless of the rate ranges at issue) and (b) the absolute difference increased (a pattern that exists when rates are well below 50% both before and after the general increase in outcome rates).  See Introduction to the Scanlan’s Rule.  In 3 cases, all three measures increased, an indicator, not only that the strength of the forces causing the rates to differ increased, but that such forces increased sufficiently to overcome the tendency for the relative difference in the outcome that was increasing (i.e., the favorable outcome) to decrease.  The study, however, reflects no consideration of the prevalence-related patterns or of the many instances where, had the authors relied on absolute differences, as a measure of disparity, they would have found increasing disparities. 

The Jha et al. 2005 NEJM Study. 

In a 2005 study published in the New England Journal of Medicine, Jha et al.[ii] examined changes in disparities in relatively uncommon medical procedures and diagnostics among Medicare beneficiaries during a period between 1992 and 2001 when rates were generally increasing.  Table 2, which is based on the 8 situations meeting the specifications underlying Table 1, presents the same types of counts as in that table.  Like Table 1, Table 2 shows a trend whereby in most cases the measures changed in a manner consistent with the prevalence-related patterns described above.  That is, the relative difference in the increasing (favorable) outcome decreased in 5 of 8 cases, the relative difference in the decreasing (adverse) outcome increased in 7 of 8 cases, and the absolute difference increased in 5 of 8 cases.  Had Jha et al. relied on relative differences in the favorable outcome as Escarce and McGuire had done, they would have decreasing disparities in 5 of 8 cases.

Table 2.  Counts of Directions of Changes of Racial Disparities According to Various Measures Based on Jha et al. NEJM 2005. [ref b4618a2]

Instead, however, Jha et al. measured disparities in terms of the absolute difference between rates – a measure that would have caused Escarce and McGuire to find increases in all 12 cases covered in Table 1 – and found that disparities increased in 7 of the 8 cases covered in Table 2.  The authors did not discuss why they relied on absolute differences or discuss the possibility that other measures might yield different conclusions.[iii]

The 2008 Le Cook et al. MCRF Study’s Commentary on Perceived Contrasts Between the Findings of the Escarce and McGuire Study and the Jha et al. Study.  In an 2008 article in Medical Care Research and Review titled “Measuring Trends in Racial/

Ethnic Health Care Disparities,” Le Cook et al.[iv] (a group of authors that included one of the authors of Escarce and McGuire), while addressing a number of seemingly sophisticated measurement issue, discussed what the authors perceived to be contrasting findings of Escarce and McGuire and Jha et al. in the following terms:

“The most basic factors affecting health status are age and gender, and we consider studies about trends in disparities that adjust for age and gender, but not SES or other factors, in keeping with the spirit of the IOM definition. Escarce and McGuire (2004) replicated methodology in Escarce, Epstein, Colby, and Schwartz (1993) to compare Black and White rates of procedure use in Medicare between 1986 and 1997. Identical clinical algorithms and definitions were applied to 5% of the national Part B data from the traditional (fee-for-service) Medicare program for the two time periods. Rates were adjusted by age and gender. Black–White differences persisted over this time period, but the differences narrowed. For all but 4 of the 30 procedures and tests measured, there was a statistically significant decrease in the Black–White disparity.

“Jha, Fisher, Li, Orav, and Epstein (2005) also used Part B data from traditional Medicare to measure differences between Blacks and Whites, adjusted for age and gender, for nine major surgical procedures. Between 1992 and 2001, the difference increased significantly for five out of the nine procedures and narrowed significantly for only one procedure. The methods and data in this recent study were the same as Escarce and McGuire, except for the partial overlap in time periods. Assembly of a longer time series in Medicare would be necessary to reconcile the apparent differences in the findings of the two studies.”

Thus, the authors imply that the reasons for the seemingly contradictory results of Escarce and McGuire and Jha et al. involved the time periods examined and even suggest it might be useful to conduct similar analyses over a longer period.  And though they state that the “methods … were same” in the two studies, indicating that they examined some aspects of the methods, the authors show no recognition that the apparent differences in the findings of the two studies are functions of the different measures the studies employed.

The illustrations in Tables 1 and 2 are focused on the specific situations that are most illustrative of the measurement issues.  For example, I excluded from Table 1 all situations where a disparity at the beginning of the period was eliminated at the end of the period, in which case all measures the disparities were eliminated, in which case all measures of disparity will necessarily have decreased (including the absolute difference).  Those examining the complete results of the study, and unaware of the measurement issues involve, would not be in a position to focus the inquiry on the situations most illustrative of measurement issues and hence to recognize, for example, with respect to the matters considered in the two tables, had both studies used either the measurement approach of the first study, or the measurement approach of the second study, the results would have been almost identical.

The manner in which Cook et al. appraised the findings of the two studies is nevertheless usefully illustrative of the fog in which health and healthcare disparities is conducted, the more so if it is an unremarkable example than it were a remarkable example. .