CASE STUDY
(April 4, 2009; rev. June 12, 2013)
This page has now been augment by a treatment of a nuance of the matter treated here in the Case Study II subpage of the Scanlan’s Rule page.
Set out below are three related issues concerning the measurement of disparities between the rates at which groups experience or fail to experience an outcome.
ISSUE 1:
In Table 1 below the first two columns of data show the rates at which an advantaged group (AG) and a disadvantaged group (DG) experience a favorable outcome in four settings. AGFR stands for the advantaged group’s favorable outcome rate and DGFR stands for the disadvantaged group’s favorable outcome rate. The favorable outcomes can be envisioned as selection for hire among employment applicants, the granting of a mortgage application, the receipt of some medical procedure or the experiencing of some favorable health outcome, the passage of a physical ability or paper-and-pencil test, or myriad other things. The four settings designated A through D can be regarded as reflecting different pairs of groups seeking the favorable outcome, decision-making entities such as employers, banks, or hospitals, different collections of such entities, different geographical regions, different types of outcomes, or the same pairs of groups, entities, regions, or types of outcomes at different points in time.
|
Table 1
|
|
Settings
|
AGFR
|
DGFR
|
RelDiffFav
|
RelDiffAdv
|
AbsDiff
|
OR
|
|
A
|
20.0%
|
9.0%
|
55.0%
|
13.8%
|
0.11
|
2.53
|
|
B
|
40.1%
|
22.7%
|
43.5%
|
29.2%
|
0.17
|
2.29
|
|
C
|
59.9%
|
40.5%
|
32.3%
|
48.2%
|
0.19
|
2.19
|
|
D
|
90.0%
|
78.2%
|
13.1%
|
117.1%
|
0.12
|
2.50
|
The column RelDiffFav contains the relative difference in experiencing the favorable outcome. It is derived from the ratio [DGFR]/[AGFR], which would be commonly referred to as the risk ratio. Sometimes observers call the risk ratio the relative difference. But technically the relative difference is 1-([DGFR]/[AGFR]), a distinction that can be pertinent to what is termed Approach 5 below. Thus, for example, where DG’s rate of experiencing a favorable outcome is 60% and AG’s rate of experiencing the outcome is 80%, the risk ratio would be .75 and the relative difference would be 25% – i.e., DG is 25% less likely to experience the favorable outcome than AG.
The column RelDiffAdv contains the relative difference in experiencing the adverse outcome (in other words, failing to experience the favorable outcome). It is derived from the ratio (1-[DGFR])/(1-[AGFR]), which would be commonly referred to as the risk ratio with regard to the adverse outcome. Here, too, some might refer to the risk ratio as the relative difference. But technically the relative difference in this situation is ((1-[DGFR])/(1-[AGFR]))-1. Thus, for example, where DG’s rate of experiencing the adverse outcome is 40% and AG’s rate is 20%, the risk ratio is 2.0 and the relative difference would be 100% – i.e., DG is 100% more likely to experience the unfavorable outcome than AG.
Use of the disadvantaged group’s rate as the numerator in both risk ratios is the most common method for calculating relative differences. But practices vary across settings and within settings. The Equal Employment Opportunity Commission’s 80% rule for determining whether a selection device warrants scrutiny as to its impact uses the disadvantaged group’s rate in the numerator in determining whether that group’s rate is less than 80% of the rate of the advantaged group. One the other hand, in appraising relative changes of selection, many use the advantaged group’s rate as the numerator, as in Baldus & Cole’s Statistical Proof of Discrimination (1980).
In any case, with regard to either or both outcomes, one might reverse the numerators and denominators. In such case, the risk ratio would be the reciprocal of the original risk ratio. That is, for example, in the case of the favorable outcome, the risk ratio of .75 (60/80) would become a risk ratio 1.33 (80/60); and .75 * 1.33 equals 1.0. In that sense, the ratios are equivalent regardless of which group’s rate is used as the numerator. Nevertheless, the relative difference would not be the same in the two situations, since the relative difference for the favorable outcome would be 25% (i.e., DG’s rate is 25% less than AG’s rate) under the original method where DG’s rate was used as the numerator and 33% (i.e., AG’s rate is 33% greater than DG’s rate) with AG’s rate used as the numerator. It should be recognized that the choice of numerator determines whether a risk ratio will be less than or greater than 1.0, and that in the former case the relative difference cannot be greater than 100% while in the latter case it can be limitless. That may have some bearing on how disparity is intuitively regarded. Nevertheless, the choice of numerator is not usually of consequence to issues such as those addressed here. I merely note the distinction for clarity and because it has potential significance with regard to the rankings in Approach 5 below.
The column AbsDiff contains the absolute difference between rates, in terms of percentage points. Some refer to this difference as a “percent difference.” That can create great confusion, for example, when one group’s rate is 10% and the other group’s rate is 20%. Thus, “percentage point difference” better distinguishes the measure from a relative difference. The absolute difference is the same regardless of whether one examines the favorable or the adverse outcome.
The column OR contains the odds ratio in terms of the ratio of DG’s odds of experiencing the adverse outcome (i.e., (1-[DGFR])/[DGFR])) to AG’s odds of experiencing the adverse outcome (i.e., (1-[AGFR])/[AGFR]). Depending on which group’s rate is used as the numerator in the odds ratio, and on whether one is examining the ratio of the two groups’ odds of experiencing the favorable outcome or the adverse outcome, the odds ratio may be greater than one or less than one. In the latter case, the larger the odds ratio, the smaller the difference. So it is sometimes important to refer to the size of the “difference measured by the odds ratio” rather than simply the size of the odds ratio. It is sometimes stated that the odds ratios are equivalent whether one examines the favorable or the adverse outcome. This is true in the sense that of four possible odds ratios, each will be the same as one of the others and the reciprocal of two of the others. But, as with the choice of numerator for the relative difference, the choice of numerator in the odds ratio, and whether the odds used in the ratio are the odds of the favorable or the unfavorable outcome, affects the size of the difference measured by the odds ratio. However, this distinction is not germane to the discussion here.
With the above description of fields provided as background, set out below is a description of five different approaches to comparing the size of the differences between rates in settings A through D set out in Table A.
Approach 1 – relative differences in favorable outcomes: In many circumstances, differences between rates are measured in terms of relative differences in experiencing the favorable outcome. That is the most common approach to measuring the size of disparities in employment selection rates or test passage rates. It used to be the quite predominant approach to measuring healthcare disparities with regard to receipt of treatments/procedures like immunization and mammography. While probably less predominant than it used to be, it is still the most common approach, notwithstanding the recommendation of the National Center for Health Statistics (NCHS) (discussed with respect to Approach 2) that all disparities be measured in terms of relative differences in adverse outcome rates. According to Approach 1, a ranking of the four settings in terms of the largest to smallest disparity would be: A, B, C, D.
Approach 2 – relative differences in adverse outcomes: With regard to health issues such as whether a person is sick or healthy or whether a person dies or lives, it has long been the common practice to measure the size of a disparity in terms of relative differences in adverse outcome rates. In a number of papers between 2004 and 2006, NCHS statisticians recommended that both health and healthcare disparities be measured in terms of relative differences in adverse outcomes (i.e., with regard to the latter, for example, relative differences in failure to receive immunization or mammography rather than relative differences in receipt of such treatments/procedures. Such approach underlies the measure of health and healthcare disparities with regard the evaluation of progress toward meeting the goals of Healthy People 2010. There are also some employment situations where disparities between racial, ethnic, age, or gender groups might be measured in terms of relative differences in adverse outcomes, for example, with regard to termination rates or rates or disqualification for failure to satisfy some prerequisite. Also, disparities with regard to the securing of mortgage, school discipline, and traffic stops are generally measured in terms of relative differences in adverse outcome rates. According to Approach 2, a ranking of the four settings in terms of the largest to the smallest disparity would be: D, C, B, A.
Approach 3 – absolute differences between rates: Some researchers measure disparities in terms of absolute differences between rates. This seems to be increasingly the case with respect to studies of disparities in healthcare. According to Approach 3, a ranking of the four settings in terms of the largest to the smallest disparity would be: C,B,D,A.
Approach 4 – odds ratios: Many researchers rely on odds ratios to measure disparities. Some do so because of what they perceive to be its qualities as a measure. But there appears to be an increasing use of the odds ratio simply because the odds ratio is conveniently generated by logistic regression analyses. According to Approach 4, a ranking of the four settings in terms of the largest to the smallest disparity would be: A, D, B, C.
A summary of the above is set out below:
Approach 1 A, B, C, D.
Approach 2 D, C, B, A.
Approach 3 C, B, D, A.
Approach 4 A, D, B, C.
Is one approach superior to another, say, for purposes of evaluating which setting is most likely to involve discrimination by decision-makers or for any other purpose? If so, why? Is any approach of value?
ISSUE 2:
Table 2 set outs information similar to that set out in Table 1, limited to three settings.
|
Table 2
|
|
Setting
|
AGFR
|
DGFR
|
RelDiffFav
|
RelDiffAdv
|
AbsDiff
|
OR
|
|
A
|
25.1%
|
11.1%
|
55.8%
|
18.7%
|
0.14
|
2.68
|
|
B
|
59.9%
|
40.5%
|
32.3%
|
48.2%
|
0.19
|
2.19
|
|
C
|
95.0%
|
88.3%
|
7.0%
|
131.7%
|
0.07
|
2.49
|
Based on the insight you derive from the analysis of Issue 1, can you rank the three disparities by largest to smallest? What is your basis for the ranking?
ISSUE 3:
In some employment contexts, disparities are not analyzed on the basis of differences in selection or non-selection rates, since such information is sometimes not available. Instead, the disparities are analyzed based on the difference between the proportion a group comprises of the pool of persons eligible for selection and the proportion the group comprises of those. In various other settings, disparities might also be cast in terms of a comparison between the proportion a group comprises of those who could experience some favorable or adverse outcome and the proportion the group comprises of those who experience the outcome. Table 3 set out the proportions a disadvantaged group comprises of persons in a pool eligible for selection for some favorable outcome and those who are selected for the outcome.
|
Table 3
|
|
Setting
|
DG Proportion of Pool
|
DG Proportion of Selections
|
|
A
|
20.00%
|
10.00%
|
|
B
|
3.00%
|
1.00%
|
Which of the four measures of differences can you derive from these rates and what are such differences? On the basis of such information, can you form an opinion as to which disparity is larger? If so, which disparity do you consider to be larger and why do you think it is larger?
Answers to these questions may be found on the Case Study Answers subpage of the Scanlan’s Rule page.