James P. Scanlan, Attorney at Law

IDEA Data Center Disproportionality Guide

(Aug. 11, 2014; rev. Nov. 16, 2014)

Prefatory note 2, added November 25, 2014 (rev. Dec. 7, 2014):  On November 25, 2014, an Addendum was added to this page to expand on the subject of Table 2 below, that is, the way that, for a given strength of the forces causing outcome rates of advantaged and disadvantaged groups to differ, the proportion the disadvantaged group comprises of the pool will affect relative and absolute differences between that proportion and the proportion of persons experiencing an outcome.  The measurement issues addressed on this page are also addressed in a November 17, 2014 amicus curiae brief in Texas Department of Housing and Community Development, et al. v.  The Inclusive Communities Project, Inc., Supreme Court No. 13-1731, and a August 11, 2013 letter to the IDEA Data Center. 

Prefatory note:  This subpage, which addresses the IDEA Data Center (IDC) guide for identifying “significant disproportionality” under the Individuals with Disabilities Education Act,  could be a subpage to the Educational Disparities page because it involves the quantification of differences in educational outcomes apart from discipline.  But it probably fits better here, as a subpage to the Discipline Disparities page, because it involves at least one disciplinary outcome and because it involves a situation where the focus is on putatively adverse outcomes in circumstances where observers commonly quantify disparities in terms of relative differences without recognizing that generally reducing the adverse outcome tends to increase relative differences between rates of experiencing the outcome (as discussed, for example, in the “Lending and Discipline Disparities” section of “Race and Mortality Revisited,” Society (July/Aug. 2014), “Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies, ” Amstat News  (Dec. 2012); “The Paradox of Lowering Standards,” Baltimore Sun (Aug. 5, 2013); “Things government doesn’t know about racial disparities,” The Hill (Jan. 28, 2014)).  The subpage does, however, involve certain absolute difference issues.  The creation of this subpage was prompted by the Department of Education’s seeking public comment on whether it should establish a standard approach to identifying significant disproportionality under Section 618 (d) of the Individuals with Disabilities Education Act, as the Government Accountability Office had urged it to do in a February 2013 report titled “Individuals with Disabilities Education Act: Standards Needed to Improve Identification of Racial and Ethnic Overrepresentation in Special Education (GAO-13-137) .”  The request was posted on June 24, 2014.  After an extension from the original closing date of July 21, 2014, the comment period was extended to June 28, 2014.  Apparently 95 comments were received and they are available here. My comment is available here.

The crucial point of the discussion below of the patterns by which measures change as the prevalence of an outcome changes is that the measures discussed cannot effectively quantify the strength of an association.  That the measures discussed here involve situations where a common response to observed patterns of relative differences in adverse outcomes (or measures that are functions of those relative differences) include things that result in general reductions in the frequency of the outcome, which then tends to increase the disproportionality as it is being measured, highlights the failure of understanding in the area.

The discussion, however, will also show some problems with the measures discussed in the IDC guide even apart from those related to the ways standard measures of differences between outcome rates tend to be affected by the frequency of an outcome.

This page is closely related to the Disabilities – Public Law 104-446 subpage of the Discipline Disparities page.  That subpage addresses the congressional mandate that where “significant discrepancies” are found in the suspensions of students with disabilities schools must seek approaches to discipline of the type that generally reduce suspension rates.  Congress is unaware, however, that general reductions in suspensions tend to increase relative differences in discipline rates.  See also "Race and Mortality Revisited" at 15.

Some parts of this page will be difficult for many to understand.  In terms of difficulty, I compare it to the Comment on Baicker Health Affairs 2005 and the Offense Type Issues subpage of the Discipline Disparities page.  But one ought not to attempt to interpret data concerning significant disproportionality under the IDEA without at least understanding the patterns by which standard measures of significant disproportionality are affected by the prevalence of an outcome and the implications of such patterns with respect to the soundness of such measures.       

***

In October 2011 the Data Accountability Center (DAC) issued a technical assistance guide titled  “Methods for Assessing Racial/Ethnic Disproportionality in Special Education” pursuant to Department of Education, Office of Special Education Programs Grant No. H373Y070002.  The document provided guidance on identifying racial/ethnic disproportionality with respect (1) the identification of children as children with disabilities, including identification of children with particular disabilities; (2) the placement of children in particular educational environments; and (3) the incidence, duration, and type of disciplinary actions, including suspensions/expulsions among students with disabilities. The document was superseded in May 2014 by a document of the same name produced by The IDEA Data Center (IDC) (apparently a successor entity to DAC) under U.S. Department of Education, Office of Special Education Programs Grant No. H373Y130002.  Both documents listed the same four authors.   

I have not carefully compared the two documents concerning the nature of the changes between the 2011 and 2014 versions.  But the documents appear to be essentially the same with respect to the subjects addressed here, which involve the guidance provided for calculating:

            (a) relative (percentage) differences between rates of experiencing the aforementioned outcomes;

            (b) absolute (percentage point) differences between rates of experiencing those outcomes;

            (c) relative differences between the proportion a group comprises of the population potentially experiencing an outcome (the pool) and the proportion the group comprises of persons experiencing the outcome; and

            (d) absolute differences between the proportion a group comprises of pool and the proportion the group comprises of persons experiencing the outcome. 

By way of summary of what I will illustrate in Table 1 several paragraphs below, as the relevant outcome generally decreases in overall frequency, (a), (c), and (d) will tend to increase, while (b) will tend to decrease (and the opposite will tend to occur when the outcomes increase in overall frequency).  Similarly, (a), (c), and (d) will tend to be larger, while (b) will tend to be smaller, in places where the outcomes are less common than in places where the outcomes are more common.  The described patterns for (a), (c) and (d) will tend to occur regardless of the rate ranges at issue.  The described pattern for (b) will tend to exist in the case of the rate ranges at issue for the particular outcomes at issue, but would not tend to exist where the outcome rates are very high. 

The principal problem with the IDC guide that is pertinent to the main measurement issues addressed on the Discipline Disparities page and its subpages is the failure to recognize the patterns by which (a) measures of differences between outcome rates and (b) measures of differences between the proportion a group comprises of the pool and the proportion it comprises of the population experiencing the outcome tend to be systematically affected by the frequency of an outcome.  The pattern is easier to describe with respect to the measures of differences between outcome rates (though I will below treat similar issues with respect to the representational comparisons).  The rarer the outcome the greater tends to be the relative difference in experiencing the outcome and the smaller tends to be the relative difference in avoiding the outcome.  Thus, general reductions in the adverse outcomes at issue will tend to increase relative differences between rates of experiencing the outcomes while reducing relative differences between rates of avoiding the outcomes.[i]  Since the outcome rates at issue are well below 50 percent for all groups being compared, general reductions in the outcome will tend to decrease absolute differences.  See the discussion of interpretations of patterns of changes of demographic differences in falling below the basic level in the Education Trust GC Study and the McKinsey Achievement Gap Study subpages of the Educational Disparities page.

Similarly, in areas or among subpopulations where the adverse outcomes are less common, relative differences in the adverse outcome will tend to be larger, while relative differences in the corresponding favorable outcome will tend to be smaller, than in areas or among subpopulations where the adverse outcomes are more common.  Given the rate ranges at issues for the outcomes at issue, lower overall rates tend to be associated with smaller absolute differences than higher overall rates.  Thus, there will be a tendency for relative and absolute differences to yield different results concerning changes over time in circumstances where frequency generally changes over time and different results as to the comparative size of differences from setting to setting where the settings examined involve different overall rates.

The main problems with the analyses of rate ratios and rate differences of the type found in the guide are covered in many places, including the recent “Race and Mortality Revisited,” Society (July/Aug. 2014). 

But I note that the analyses of rate ratios and rate differences in the IDC guide have the additional problem that they involve comparisons of a subject group’s rate with the rate of all other persons. Almost invariably, it is better to compare the subject group’s rate with the white rate (or the rate of the largest advantaged group) rather than the rate for all other persons, among other reasons, because comparisons of a group’s rate with the rate for all other persons allow the racial/ethnic composition of the others category to influence the size of the difference (a matter of consequence even when the difference is appraised with a sound measure).  The approach in the guide can also lead, for example, to a situation where the Hispanic adverse outcome rate will be lower than the rate for all others even when the Hispanic outcome rate is greater than the white rate.

The guide, in its Section 6, presents a method for adjusting for the composition of the others category.  I have not examined the method closely enough to form a view on its general soundness.

A similar problem exists in the guidance for comparing the proportion a group comprises of the potentially affected pool with the proportion it comprises of persons experiencing the outcome.  But there also exists an additional problem that would exist even if the pool were comprised solely of the subject disadvantaged group and a single advantaged group.  For, with regard to any given pair of rates at which the disadvantaged group and the advantaged group experience an outcome, both the relative and absolute differences between the proportion the group comprises of the pool and the proportion it comprises of persons experiencing the outcome are affected by the proportion the disadvantaged comprises of the pool.  The larger that proportion is, the smaller will tend to be the relative difference between the proportion the group comprises of the pool and the proportion it comprises of persons experiencing the outcome.  But the proportion the subject group comprises of the pool affects the absolute difference between the proportion the group comprises of the pool and the proportion it comprises of persons experiencing the outcome in a more complicated way.  Roughly (and, again, for any given pair of outcome rates  for the advantaged and disadvantaged groups), as the proportion the disadvantaged group comprises of the pool increases up to the point slightly above 40 percent, the absolute differences between the proportion the group comprises of the pool and the proportion it comprises of persons experiencing the outcome tends to increase; as the proportion the group comprises of the pool increases beyond the point slightly below 40 percent, the absolute differences between the proportion the group comprises of the pool and the proportion it comprises of persons experiencing the outcome tends to decrease.  I will illustrate this pattern in Table 2 below.

The implications of this problem are ultimately academic, however.  That is, whether a group’s outcome rate is contrasted with the white rate or the rate for all others can be a matter of consequence for an analysis of group differences, because the outcome rates can provide a sound basis for appraising the strength of the forces causing the outcome rates to differ (as in the case of the EES figure discussed in “Race and Mortality Revisited”).   But while representational figures can enable one to derive the relative difference between the subject group’s rate of experiencing the outcome and either (a) the rate for all other groups combined or (b) the rate of whites, one cannot derive from the compositional information the actual rates at which the various groups experience an outcome. And one needs the actual rates in order to appraise the strength of the forces causing the outcome rates to differ.  This is explained in greater detail in Section C (at 23-26) the September 2013 University of Kansas Faculty Workshop paper titled “The Mismeasure of Discrimination.” [ii]  

Returning to the main point of this subpage (which involves the effects of prevalence of an outcome on measures in the IDC guide), both the relative and absolute difference between the proportion a group comprises of the pool and the proportion it comprises of persons experiencing an outcome tend to be systematically affected by the prevalence of an outcome.  I have explained in various places that a corollary to the pattern by which the rarer an outcome the greater tends to be the relative difference in experiencing it and the smaller tends to be the relative difference in avoiding it is a pattern whereby the rarer an outcome the greater tends to be the proportion the more susceptible group comprises of persons experiencing the outcome (and the proportion it comprises of those failing to experience the outcome).  The pattern can be inferred from the Tables 1 and 2 in "Race and Mortality Revisited" and is shown explicitly in Table 1 of “Divining Difference,” Chance (Spring 1994) and Table 1 of “Can We Actually Measure Health Disparities?,” Chance (Spring 2006). 

Thus, both relative and absolute differences between the between the proportion the more susceptible (disadvantaged) group comprises of the pool and the proportion it comprises of persons experiencing the outcome increase as the outcome becomes less common (while both relative and absolute differences between the proportion the group comprises the pool and the proportion it comprises of persons failing to experience the outcome decrease as the prevalence of an outcome increases).   

Some may have initial difficulty understanding the pattern whereby relative and absolute differences between the proportion a group comprises of the pool and the proportion it comprises of persons experiencing the outcome both change in the same direction as the prevalence of an outcome changes, given that relative and absolute differences between rates of experiencing the outcome commonly change in opposite directions as the prevalence of the outcome changes in the rate ranges at issue here.  But one should keep in mind that the absolute difference between the proportion a group comprises of the pool and the proportion it comprises of persons experiencing the outcome is different from the absolute difference between the outcome rates of the advantaged and disadvantaged groups. 

The pattern by which the absolute difference between rates tends to change as the prevalence of an outcome changes varies depending on the rate ranges issue, and the absolute difference can tend to change in either the direction or the opposite direction of the relative difference between rates that one happens to be examining.  (As noted in various places, while both relative differences and the absolute difference can all change in the same direction as the prevalence of an outcome changes, when all do not change in the same direction, the absolute difference will change in the same direction as the smaller of the two relative differences.)   But the absolute difference between the proportion a group comprises of the pool and the proportion it comprises of persons experiencing the outcome will always change in the same direction as the relative difference between those proportions.  That occurs because the proportion the group comprises of the pool remains unchanged as the prevalence of an outcome changes.  Given that fact, the larger (or smaller) the proportion the group comprises of persons experiencing the outcome, the larger (or smaller) will be both relative and absolute difference between the proportion the group comprises of the pool and the proportion it comprises of persons experiencing the outcome. 

Illustration of the Effects of Prevalence on the Four Measures in the IDC Guide

Table 1 below is based on the data in Table 1  of the 2006 British Society for Population Studies (BSPS) paper “The Misinterpretation of Health Inequalities in the United Kingdom,” which shows the effects of serially lowering a cutoff from the point where almost everyone fails to a point where almost everyone passes where the mean scores of the advantaged and disadvantaged groups differ by half a standard deviation.  (Those specifications also underlie Tables 1 and 5 of "Race and Mortality Revisited.")  The point of the illustration in the BSPS paper and the recent article is that, though there is no real change in the strength of the forces causing rates to differ when there occurs a change in overall prevalence akin to that effected by changing a test cutoff, standard measures give various, contrasting, false impressions of changes in the strength of those forces.  Table 1 below treats the matter as if there existed only two groups, the advantaged group (AG) and the disadvantaged group (DG), thereby obviating certain issues about other groups discussed above.

The proportion the disadvantaged group comprises of those potentially experiencing the outcome is irrelevant to the illustration of the directions of the patterns by which the four measures discussed at the outset change as the prevalence of an outcome changes.  But using some representational figure in the illustration will make it easier to follow.  For Table 1 I arbitrarily use representational figures of 20 percent and 70 percent – that is the disadvantaged group comprises 20 percent or 70 percent of the pool.  (Different representational figures will yield different values for the relative and absolute differences between the proportion the group comprises of the pool and the proportion it comprises of persons experiencing the outcome; but the patterns of directions of changes as the frequency of an outcome changes would remain the same.)

The column headings in the table should be self-explanatory.  But I have added parenthetical letters a though d to correspond to above letter identifications of the measures addressed in the IDC guide.  I place the field showing the proportion DG comprises of persons experiencing the adverse outcome after (a) and (b) but before (c) and (d) because the proportion is pertinent to the latter measures but not to the former measures.  While the BSPS table covers the entire range of outcome values (and thus covers situations where the absolute difference between rates changes in the same direction as the relative difference as the prevalence of an outcome changes), Table 1 is limited to outcome rates no higher than 20 percent for the advantaged group (which thus limits the situation to one where the relative and absolute differences tend to change in opposite directions as the prevalence of an outcome changes). 

There is a small inconsistency in the manner of presentation in that (a) is presented in terms of a rate ratio while (c) is presented in terms of a relative difference (which is the rate ratio minus 1 as discussed, for example, in note 1 of "Race and Mortality Revisited").  I use that approach because that it the way the measures are presented in the IDC guide.

Table 1.  Illustration of the effect of reducing the frequency of an adverse outcome on four measures of disproportionality where the disadvantaged group comprises 20 percent and 70 percent of the pool (ref b5526 b1)

DG Prop Pool	AG Adv Rate	DG Adv Rate	(a) Adv Rate Ratio	(b) Abs Df Btw Rates (perc points)	DG Prop of Adv	(c) Rel Df Bwt DG Prop Adv and Pool	(d) Abs Df Btw DG Prop Adv and Pool (perc points)
20.00%	20.00%	36.69%	1.83	16.69	31.44%	57.22%	11.44
20.00%	10.00%	21.77%	2.18	11.77	35.24%	76.21%	15.24
20.00%	5.00%	12.71%	2.54	7.71	38.86%	94.32%	18.86
20.00%	3.00%	8.38%	2.79	5.38	41.12%	105.58%	21.12
20.00%	1.00%	3.44%	3.44	2.44	46.22%	131.11%	26.22
70.00%	20.00%	36.69%	1.83	16.69	81.06%	15.80%	11.06
70.00%	10.00%	21.77%	2.18	11.77	83.55%	19.36%	13.55
70.00%	5.00%	12.71%	2.54	7.71	85.58%	22.25%	15.58
70.00%	3.00%	8.38%	2.79	5.38	86.70%	23.85%	16.70
70.00%	1.00%	3.44%	3.44	2.44	88.92%	27.02%	18.92

 

The table illustrates the patterns described at the outset whereby as the frequency of the outcome decreases, measures (a), (c) and (d) increase, while measure (b) decreases.  Not shown in the table is that as the adverse outcome decreases, relative and absolute differences between rates of experiencing the favorable outcome would decrease, as would both the relative and absolute differences between the proportion DG comprises of the pool and the proportion it comprises of persons experiencing the favorable outcome.  I may eventually create a table illustrating that pattern.  But one can infer it from the figures in the aforementioned tables in the two Chance  articles.[iii]

Illustrations of the Effects of the Proportion the Subject Group Comprises of the Pool on Measures (c) and (d) of the IDC Guide

The following illustration is subsidiary to the main point of this page and it should be regarded with recognition that measures of differences between proportion a group comprises of the pool and the proportion the group comprises of persons experiencing the outcome are unsound for prevalence-related reasons reflected in Table 1 and discussed in the Section C of the Kansas Law paper.  The illustration should thus be regarded as a further reason why an unsound measure is unsound.

Table 2 is based on a situation where the advantaged group’s adverse outcome rate is 20 percent (and there exists the same half standard deviation difference between means of the underlying distributions as in the situation underlying Table 1).  Table 2 illustrates the points made above about the ways the proportion the subject group comprises of the pool affects, for any given pair of rates, both relative and absolute difference between that proportion the disadvantaged group comprises of the pool and the proportion it comprises of persons experiencing the outcome

Table 2.  Illustration of the effect, for given pairs of advantaged and disadvantaged group rates of experiencing an outcome, of the proportion the disadvantaged group comprises of the pool on relative and absolute differences between that proportion and the proportion the group comprise of persons experiencing the outcome (ref b5526b1)

 

DG Prop Pool	AG Adv Rate	DG Adv Rate	(a) Adv Rate Ratio	(b) Abs Df Btw Rates (perc points)	DG Prop of Adv	(c) Rel Df Bwt DG Prop Adv and Pool	(d) Abs Df Btw DG Prop Adv and Pool (perc points)
10.00%	20.00%	36.69%	1.83	16.69	16.93%	69.33%	6.93
20.00%	20.00%	36.69%	1.83	16.69	31.44%	57.22%	11.44
30.00%	20.00%	36.69%	1.83	16.69	44.02%	46.72%	14.02
40.00%	20.00%	36.69%	1.83	16.69	55.02%	37.54%	15.02
41.00%	20.00%	36.69%	1.83	16.69	56.04%	36.69%	15.04
42.00%	20.00%	36.69%	1.83	16.69	57.05%	35.84%	15.05
43.00%	20.00%	36.69%	1.83	16.69	58.05%	35.01%	15.05
44.00%	20.00%	36.69%	1.83	16.69	59.04%	34.18%	15.04
45.00%	20.00%	36.69%	1.83	16.69	60.02%	33.37%	15.02
46.00%	20.00%	36.69%	1.83	16.69	60.98%	32.57%	14.98
47.00%	20.00%	36.69%	1.83	16.69	61.93%	31.77%	14.93
48.00%	20.00%	36.69%	1.83	16.69	62.87%	30.99%	14.87
49.00%	20.00%	36.69%	1.83	16.69	63.80%	30.21%	14.80
50.00%	20.00%	36.69%	1.83	16.69	64.72%	29.44%	14.72
60.00%	20.00%	36.69%	1.83	16.69	73.35%	22.24%	13.35
70.00%	20.00%	36.69%	1.83	16.69	81.06%	15.80%	11.06
80.00%	20.00%	36.69%	1.83	16.69	88.01%	10.01%	8.01
90.00%	20.00%	36.69%	1.83	16.69	94.29%	4.77%	4.29

 

The table shows that, as discussed earlier, as the proportion the group comprises of the pool increases, the relative difference between that proportion and the proportion the group comprises of persons experiencing the outcome decreases.  But the absolute difference between those proportions increases to the point where the proportion the group comprises of the pool slightly exceeds 40 percent and decrease as the proportion the group comprise of the pool continues to increase.[iv]

Illustrations of the Effects on a Sound Measure of Association of Comparing a Disadvantaged Group’s Rate with an Overall Rate Rather than the Advantaged Group’s Rate. 

The following is illustrations are peripheral to main subject of this subpage but generally pertinent to the underlying measurement issues.

One reason never to compare a disadvantaged group rate with an overall rate rather than the advantaged group rate is that doing so will invariably allow the disadvantaged group’s rate to affect the measure of difference.  Whatever the measure used, the greater the proportion the disadvantaged group comprises of the population, the smaller the difference.  That applies to the sound measure of association that I commonly term the EES, for estimated effect size, and that is used throughout "Race and Mortality Revisited."  I illustrate that pattern in Table 3 (though it may be too obvious a matter to warrant illustration).  But a further effect on a sound measure of association of comparing disadvantaged group’s rate with the overall rate rather the advantaged group’s rate is that doing so undermines the sound measure of association (in a manner apart from the effect just mentioned).  I illustrate that in Table 4. 

Like Tables 1 and 2, Table 3 is based on BPSP Table 1, where means of the advantaged and disadvantaged group differ by half a standard deviation.  When AG’s rate is compared with DG’s rate, the EES value is .50.  But the table shows, for the same prevalence level used in Table 2, the different EES values derived from comparing the DG rate with the overall rate.  Both values are smaller than .5 and are smaller where DG comprises 50 percent of the pool than where DG comprise 20 percent of the pool. 

Table 3.  EES values based on comparison of DG’s adverse outcome rate with an overall adverse outcome rate at level of prevalence benchmarked by 20 percent adverse outcome rate for AG where DG comprises 20 percent and 50 percent of the population.  [ref b5607 a 2]

 

DG Prop Popl	AG Adv Rate	DG Adv Rate	Overall Adv Rate	EES
20.00%	20.00%	36.69%	23.34%	0.38
50.00%	20.00%	36.69%	28.35%	0.22

 

Table 4 now illustrates that comparing the subject group’s rate with an overall rate renders the EES an unsound measure of association.  The EES is a sound measure of association (assuming that the underlying distributions are normal) because it does not change as the overall prevalence of an outcome changes in a manner akin to the effected by lowering a test cutoff.  Table 4, however, shows that EES values based on a comparison of DG’s rate with an overall rate (rather than AG’s rate) change as the prevalence of an outcome changes.  I illustrate the effect in Table 4 using selected levels of overall prevalence for simplicity, except that I include all levels benchmarked by adverse outcome rates for AG of 20 percent and less. 

All EES values decline as the prevalence of the outcome declines.  But substantial EES differences are only found between the high prevalence levels and low prevalence levels.  The changes are quite minor in the prevalence ranges of the outcomes at issue in the IDC disproportionality guide.  But that merely means that comparing a disadvantaged group’s rate with an overall rate may make little difference in that context.  The fact would remain that it never makes sense to compare a disadvantaged group’s rate with an overall rate rather than the advantaged group’s rate. 

By way of clarification as the difference in the illustrations in Tables 3 and 4.  Table 3 show that comparing DG’s rate against an overall rate yields a smaller EES than would comparing DG’s rate with AG’s rate.  Table 4 show that reduction in the EES differs by level of prevalence (hence undermining the EES as a measure of difference).

Table 4.  EES values based on comparison of DG’s adverse outcome rate with an overall adverse outcome rate at various levels of prevalence where DG comprises 20 percent and 50 percent of the population. 

 

DG Prop Pop	AG Adv Rate	DG Adv Rate	Overall Adv Rate	EES
20.00%	99.00%	99.76%	99.15%	0.42
20.00%	90.00%	96.25%	91.25%	0.41
20.00%	70.00%	84.61%	72.92%	0.40
20.00%	40.00%	59.48%	43.90%	0.38
20.00%	20.00%	36.69%	23.34%	0.38
20.00%	10.00%	21.77%	12.35%	0.37
20.00%	5.00%	12.71%	6.54%	0.36
20.00%	3.00%	8.38%	4.08%	0.35
20.00%	1.00%	3.44%	1.49%	0.35
50.00%	99.00%	99.76%	99.38%	0.31
50.00%	90.00%	96.25%	93.12%	0.28
50.00%	70.00%	84.61%	77.31%	0.26
50.00%	40.00%	59.48%	49.74%	0.24
50.00%	20.00%	36.69%	28.35%	0.22
50.00%	10.00%	21.77%	15.88%	0.21
50.00%	5.00%	12.71%	8.86%	0.20
50.00%	3.00%	8.38%	5.69%	0.19
50.00%	1.00%	3.44%	2.22%	0.19

 

 

ADDENDUM – EFFECTS OF THE PROPORTION A GROUP COMPRISES OF THE POOL ON MEASURES OF DIFFERENCES BETWEEN THAT PROPORTION AND THE PROPORTION THE GROUP COMPRISE OF PERSONS EXPERIENCING AN ADVERSE OR FAVORABLE OUTCOME

(Nov. 25, 2014)

This Addendum expands on the point of Table 2 of the IDEA Data Center Disproportionality Guide subpage of the Discipline Disparities page regarding the way that, for any given EES or pair of outcome  rates, the proportion the disadvantaged group (DG) comprises of the pool will affect relative and absolute differences between that proportion and the proportion DG comprises of persons experiencing an adverse outcome (which here will be designated RDP, for relative difference between proportions, and ADP, for absolute difference between proportions).  The discussion below, however, addresses this issue with respect to both the adverse and favorable outcomes, hence, with respect to perceptions about both DG overrepresentation among persons experiencing an adverse outcome and DG underrepresentation among persons experiencing the corresponding favorable outcome. 

Among the factors undermining relative and absolute difference between the proportion a group comprise of the pool and the proportion it comprises of persons experiencing an outcome as means of quantifying the strength of the forces causing those proportions to differ is that, for any given pair of outcome rates and any given EES, the proportion DG comprises of the pool will tend to affect both RDP and ADP.  The discussion of this problem with these measures, however, should not be read to suggest that the measures might otherwise be sound.  For, as discussed on the main subpage and other places (including in Argument Section A.2 of the November 2017 amicus curiae brief in Texas Department of Housing and Community Development, et al. v.  The Inclusive Communities Project, Inc., Supreme Court No. 13-1731, such measures are fatally undermined by the fact that they tend to be affected by the frequency of an outcome.

To facilitate the reader’s understanding of the key points of this Addendum, the matter is cast in terms of the relationship between the proportion DG comprises of the pool and the proportions it comprises of persons experiencing the adverse outcome or the corresponding favorable outcome.  But the described patterns could be more abstractly cast in terms of the relationship between the proportion a particular group comprises of the pool and (a) the proportion it comprises of persons experiencing the outcome to which the group is more susceptible than non-members of the group and (b) the proportion it comprises of persons experiencing the opposite outcome.[v]

A.  Effects for any Given EES of the Proportion DG Comprises of the Pool on Relative Differences Between that Proportion and the Proportion DG Comprises of Persons Experiencing Adverse and Favorable Outcomes

The pattern of the effect of the proportion DG comprises of the pool on RDP for any given EES is comparatively simple.  For any given EES, and regardless of whether the favorable or adverse outcome is examined, the higher the proportion DG comprises of the pool, the smaller will be the RDP.  Thus, observers monitoring disproportionality in terms of the RDP will find that disproportionality to decrease when the proportion DG comprises of the pool increases and vice versa.  Similarly, given the exact same outcome rates for DG and AG (the advantaged group) in two different jurisdictions, observers relying on the RDP for either the adverse or favorable outcome will find a smaller disparity in the jurisdiction with where DG comprises a higher proportion of the pool.[vi] 

Addendum Table A illustrates this pattern, where RDP-Adv and RDP-Fav represent the relative difference between the proportions DG comprises of the pool and the proportion it comprises of persons experiencing the adverse and the favorable outcomes, for particular pairs of adverse outcome rates for AG and DG reflecting an EES of .5. 

Addendum Table A.  Illustration of the effect, for given EES, of the proportion DG comprises of the pool on RDP-Adv and RDP-Fav [refb5825e2] 

 

EES	Ag Adv Rate	DG Adv Rate	DG Prop Pool	DG Prop of Adv	DG Prop of Fav	RDP-Adv	RDP-Fav
0.5	20.00%	36.69%	20.00%	31.40%	16.52%	56.98%	17.38%
0.5	20.00%	36.69%	40.00%	54.96%	34.55%	37.40%	13.63%
0.5	20.00%	36.69%	60.00%	73.30%	54.29%	22.17%	9.52%
0.5	20.00%	36.69%	80.00%	87.98%	76.00%	9.98%	5.00%
0.5	40.00%	59.48%	20.00%	27.04%	14.47%	35.19%	27.65%
0.5	40.00%	59.48%	40.00%	49.70%	31.09%	24.26%	22.28%
0.5	40.00%	59.48%	60.00%	68.98%	50.37%	14.96%	16.04%
0.5	40.00%	59.48%	80.00%	85.57%	73.02%	6.96%	8.72%

 

B.  Effects for Any Given EES of the Proportion DG Comprises of the Pool on Absolute Differences Between that Proportion and the Proportion DG Comprises of Persons Experiencing the Adverse and Favorable Outcomes

The effects, for any given EES, of the proportion DG comprises of the pool on the ADP for the adverse outcome (ADP-Adv) and favorable outcome (ADP-Fav) are more complicated than the effect for on RDP.  In general, as the proportion DG comprises of the pool increases from a very low point, both the ADP-Adv and the ADP-Fav will tend to increase for a time and thereafter decrease.   But the point of change of direction (that is, the maximum ADP) will be different for the ADP-Adv and ADP-Fav and will be affected by other factors, in complex ways. 

For any given EES, the proportions DG comprises of the pool at the point of the maximum ADP-Fav will always be greater than the proportion DG comprises of the pool at maximum ADP-Adv, as illustrated in Appendix Table B.   Thus, for any given EES, as the proportion DG comprises of the pool changes, there can be situations where observers measuring disparities in terms of the ADP-Adv and observers measuring disparities in terms of ADP-Fav, will reach opposite conclusions depending on whether they examine the proportion DG comprises of persons experiencing the favorable outcome of the proportion DG comprise of persons experiencing the adverse outcome. 

For example, as shown in the first two rows of the Appendix Table B, without any change in the adverse outcome rates of AG and DG, as the proportion DG comprises of the pool increases beyond 45%, those examining the overrepresentation of DG among persons experiencing the adverse outcome (ADP-Adv) would find the overrepresentation to be decreasing, while those examining the underrepresentation of DG among those experiencing the favorable outcome (ADP-Fav) would find the underrepresentation to continue to increase up until the point where DG comprises 52% of the pool.

Addendum Table B.  General illustration of the patterns by which the effect of increases in the proportion DG comprises of the pool on ADP-Adv and ADP-Fav changes from one of increasing those measures to one of decreasing those measures  [refb5825d6B] 

 

Row No	EES	AG Adv Rate	DG Adv Rate	DG Prop Pool	DG Prop Adv	DG Prop Fav	ADP-A	ADP-Fav	PntMaxADP-Adv	PntMaxADP-Fav
1	0.3	20.00%	29.46%	45.00%	55.00%	41.92%	9.60	3.08	X
2	0.3	20.00%	29.46%	52.00%	48.00%	48.87%	9.42	3.13		X
3	0.3	40.00%	51.60%	47.00%	53.00%	41.76%	6.28	5.24	X
5	0.3	40.00%	51.60%	53.00%	47.00%	47.69%	6.18	5.31		X
5	0.3	60.00%	70.88%	48.00%	52.00%	40.11%	4.22	7.89	X
6	0.3	60.00%	70.88%	54.00%	46.00%	46.00%	4.16	8.00		X
7	0.5	20.00%	36.69%	43.00%	57.00%	37.40%	15.00	5.60	X
8	0.5	20.00%	36.69%	53.00%	47.00%	47.17%	14.36	5.83		X
9	0.5	40.00%	59.48%	45.00%	55.00%	35.64%	9.81	9.36	X
10	0.5	40.00%	59.48%	55.00%	45.00%	45.27%	9.43	9.73		X
11	0.5	60.00%	77.34%	47.00%	53.00%	33.37%	6.39	13.63	X
12	0.5	60.00%	77.34%	57.00%	43.00%	42.81%	6.13	14.19		X
13	0.7	20.00%	44.43%	40.00%	60.00%	31.66%	19.64	8.34	X
14	0.7	20.00%	44.43%	55.00%	45.00%	45.93%	18.04	9.07		X
15	0.7	40.00%	67.00%	44.00%	56.00%	30.22%	12.75	13.78	X
16	0.7	40.00%	67.00%	57.00%	43.00%	42.22%	11.88	14.78		X
17	0.7	60.00%	82.90%	46.00%	54.00%	26.64%	8.12	19.36	X
18	0.7	60.00%	82.90%	60.00%	40.00%	39.00%	7.50	21.00		X

 

Appendix Table B also reflects the following patterns respecting the proportion DG comprises of the pool at the point of the maximum values of ADP-Adv and ADP-Fav ­– that is, the points at which further increases in the proportion DG comprises of the pool will cease to cause the measure value to increase and begin to cause it to decrease.  

For any given EES, the greater the frequency of an adverse outcome, the greater will be the proportion DG comprises of the pool at the maximum values of both ADP-Adv and ADP-Fav.  A simplified illustration of this pattern, drawn from Appendix Table B, may be found in Appendix Table C.

Addendum Table C.   Illustration of the patterns by which the effect of increases in the proportion DG comprises of the pool on ADP-Adv and ADP-Fav changes from one of increasing those measures to one of decreasing those measures, with focus on differing frequencies for a particular EES 

 

RowNo	EES	AG Adv Rate	DG Adv Rate	DG Prop Pool	DG Prop Adv	DG Prop Fav	ADP-A	ADP-Fav	PntMaxADP-Adv	PntMaxADP-Fav
1	0.5	20.00%	36.69%	43.00%	57%	37.40%	15.00	5.60	X
2	0.5	20.00%	36.69%	53.00%	47%	47.17%	14.36	5.83		X
3	0.5	40.00%	59.48%	45.00%	55%	35.64%	9.81	9.36	X
4	0.5	40.00%	59.48%	55.00%	45%	45.27%	9.43	9.73		X
5	0.5	60.00%	77.34%	47.00%	53%	33.37%	6.39	13.63	X
6	0.5	60.00%	77.34%	57.00%	43%	42.81%	6.13	14.19		X

 

For any given frequency of an adverse outcome, as benchmarked by the AG rate, the larger the EES, (a) the smaller will be proportion DG comprises of the pool at the maximum ADP-Adv and (b) the larger will be the proportion DG comprises of the pool at the maximum ADP-Fav.  Appendix Table D provides a simplified illustration of these patterns.  The odd numbered rows show the former pattern and the even numbered rows show the pattern.  That is, for example, with the frequency benchmarked at the AG adverse outcome rate of 20%, an increase in the EES from .3 to .5 caused the DG proportion of the pool at the point of maximum ADP-Adv to decrease from 45% to 43% (rows 1 and 3), while causing the DG proportion of the pool at the point of maximum ADP-Adv to increase from 52% to 53% (rows 2 and 4). 

Addendum Table B.  Illustration of the patterns by which the effect of increases in the proportion DG comprises of the pool on ADP-Adv and ADP-Fav changes from one of increasing those measures to one of decreasing those measures, with focus on differing EESs for particular frequencies

 

RowNo	AG Adv Rate	EES	DG Adv Rate	DG Prop Pool	DG Prop Adv	DG Prop Fav	ADP-A	ADP-Fav	PntMaxADP-Adv	PntMaxADP-Fav
1	20.00%	0.3	29.46%	45.00%	55.00%	41.92%	9.60	3.08	X
2	20.00%	0.3	29.46%	52.00%	48.00%	48.87%	9.42	3.13		X
3	20.00%	0.5	36.69%	43.00%	57.00%	37.40%	15.00	5.60	X
4	20.00%	0.5	36.69%	53.00%	47.00%	47.17%	14.36	5.83		X
5	20.00%	0.7	44.43%	40.00%	60.00%	31.66%	19.64	8.34	X
6	20.00%	0.7	44.43%	55.00%	45.00%	45.93%	18.04	9.07		X
1	60.00%	0.3	70.88%	48.00%	52.00%	40.11%	4.22	7.89	X
2	60.00%	0.3	70.88%	54.00%	46.00%	46.00%	4.16	8.00		X
3	60.00%	0.5	77.34%	47.00%	53.00%	33.37%	6.39	13.63	X
4	60.00%	0.5	77.34%	57.00%	43.00%	42.81%	6.13	14.19		X
5	60.00%	0.7	82.90%	46.00%	54.00%	26.64%	8.12	19.36	X
6	60.00%	0.7	82.90%	60.00%	40.00%	39.00%	7.50	21.00		X

 

Thus, appraisals of disparities based on comparisons of the proportion a group comprises of a pool and the proportion it comprises of persons experiencing either an adverse outcome or the corresponding favorable outcome – either for purposes of determining whether the forces causing the differences are substantial enough to warrant attention or for purposes of determining whether the forces are increasing or decreasing over time or are otherwise larger in one setting than another or under one group of procedures than another – have the potential to be affected in very complicated ways by a variety factors having nothing to do with the strength of those forces.  The complications would exist both with respect to efforts to appraise disparities in settings where disadvantaged groups comprise different proportion of the pools and with respect to situations where the proportion the disadvantaged group comprises of the pool changes over time.