James P. Scanlan, Attorney at Law

Educational Disparities/Achievement Disparities

(Apr., 2011, rev. May 17, 2014)

This page discusses the way that perceptions about disparities in educational are generally flawed as a result of the failure to recognize the way standard measures of differences in outcome rates tend to be affected by the prevalence of an outcome.   The page has six subpages.  The Disparities by Subject subpage examines observed patterns of changes in various measures of disparities for different subjects (where differing proficiency rate rages have implications regarding the way general changes may affect absolute differences between rates).  The Harvard CRP NCLB Study subpage discusses a 2006 Harvard’s Civil Rights Project study that compared patterns of proficiency disparities under state tests and under National Assessment of Educational Progress tests, while relying on relative differences in proficiency rates, without recognizing the pattern by which tests with generally high pass rates would tend to show smaller relative differences in pass rates, but larger relative differences in failure rates, than tests with lower pass rates.  The New York Proficiency Rate Disparities subpage discusses a 2013 NYCAN study of changes in absolute differences between proficiency rates of demographic groups in New York State during a period of substantial decreases in proficiency rates without consideration of the patterns by which absolute differences tend to changes when proficiency rates generally decline.

The Education Trust HA Study subpage discusses a 2014 Education Trust study that examined demographic differences in achieving certain levels of academic success in terms of absolute differences between rates without consideration of the implications of demographic differences in rates of being among high achievers.  The Education Trust GC Study subpage discusses a 2013 Education Trust study that examined demographic differences in absolute changes in rates of (a) falling below the basic reading level and (b) reaching the advanced reading level, during a period of general improvements in proficiency, without recognizing that the rate ranges were such that disadvantaged would tend to experience larger absolute decreases in rates of falling below the basic level, but smaller absolute increase in rates of reaching the advanced level, than advantaged groups.  The Annie E. Casey 2014 Proficiency Disparities Study subpage discusses a 2014 Annie E. Casey Foundation 2014 study of demographic differences in proficiency rates that, at least in part, relied on absolute differences between rates without recognizing the way absolute differences tend to change as proficiency rates generally improve. 

This page and its subpages are closely related to the Discipline Disparities and it subpage, which, among other things, addresses the way observers misinterpret racial differences in discipline rates as a result of the failure to recognize the way standard measures of differences in outcome rates tend to be affected by the prevalence of an outcome.

***

The Measuring Health Disparities, Scanlan’s Rule, Mortality and Survival, Measures of Association, Lending Disparities, and Discipline Disparities pages of this site (and their sub-pages) address the failure in the law and the social and medical sciences to recognize the way that, for reasons related to the shapes of normal distributions of factors associated with experiencing an outcome, standard measures of differences between outcome rates tend to be affected by the overall prevalence of an outcome.  Most notably, as an outcome increases in overall prevalence, relative differences in experiencing it tend to decrease while relative differences in failing to experience it tend to increase.  Thus, for example, if a test cutoff is lowered (or overall test performance improves) relative differences in pass rates tend to decrease while relative differences in failure rates tend to increase. Absolute differences and odds ratios tend also to be affected by the overall prevalence of an outcome, though in a more complicated way.  Roughly, as uncommon outcomes (less than 50% for both groups) become more common, absolute differences tend to increase; as common outcomes (more than 50% for both groups) become even more common, absolute differences tend to decrease.  The matter is a bit more complicated where the rate is above 50% for one group and below 50% for the other, as explained in the introductory section to the Scanlan’s Rule page.  Differences measured by odds ratios tend to change in the opposite direction of absolute differences. 

These patterns, as illustrated with test score data, may be found in Figure 5 of the 2009 Royal Statistical Society presentation.

As suggested by the reference to test outcomes these patterns are also present in data on demographic differences in educational achievement.  Indeed, the force of the patterns may be the strongest in the educational setting.  And it should be obvious that as rates of achieving proficiency generally improve, relative differences in proficiency rates will tend to decrease while relative differences in rates of failing to achieve proficiency will tend to increase.  It should also be obvious that improvements in proficiency rates where current rates are quite low will tend to increase absolute differences between rates while improvements where rates are already high will tend to reduce absolute differences between proficiency rates.   But, as in other contexts, the role of these patterns is universally ignored in analyses of educational disparities, even when the disparities involve such things as math proficiency or statistical literacy. This is a draft of a page addressing some of this misunderstanding.

Some points related to the educational setting have been touched upon in prior articles or other materials. 

• “Race and Mortality”  (Society, Jan/Feb 2000) briefly discussed how supporters of affirmative action in college admissions have pointed to the small relative differences between minority and white graduations rates at elite universities, while opponents have pointed to large relative differences in rates of failing to graduate at those universities. Neither side recognized that both patterns are to expected given that graduation rates are generally high at elite universities

• “An Issue of Numbers” (National Law Journal, Mar. 5, 1990) and “The Perils of Provocative Statistics” (Public Interest, Winter 1991) discuss the perception that high black representation among persons disqualified from competing in intercollegiate sports by NCAA academic eligibility standards is a function of the stringency of those standard.  In fact, the lower the standard, the greater will be the black representation among those disqualified.

• Slide 13 of  the presentation Measurement Problems in the National Healthcare Disparities  (American Public Health Association 2007) used data from a chart styled “Remarkable Results” in a November 4, 2007 Washington Post article[i], which discussed perceived substantial decreases in achievement gaps at a school in Maryland, to illustrate that the National Center for Health Statistics, relying on relative differences in adverse outcomes, would find the gaps to have increased. 

• The Discipline Disparities pages discusses the perception that large relative differences in public school discipline rates are the consequence of stringent discipline policies.  In fact, the more stringent the policy, the smaller will tend to be relative differences in discipline rates.

In the tables below, I illustrate the relevant issues with data from the above-referenced Remarkable Results table in a November 4 2007 Washington Post article and data from an April 5, 2011 Washington Post article[ii] that, in finding larger improvements in rates of scoring at advanced levels in math among Asians than whites, relied on percentage point changes as the indicator of the size of changes in rates of scoring at those levels.  With such data, I also present information concerning on standard measures of differences between rates or changes in rates, as well as the only actually useful measure of difference between outcome rates or changes in rates – that described on the Solutions sub-page of the Measuring Health Disparities page (and which I usually term “EES” and which, as noted there, is also known as the probit).  The Solutions/Probit method derives from a pair of rates the difference, in terms of the percentage of a standard deviation, between hypothesized underlying distributions.  It should be recognized, however, that in the educational context, the actual underlying data generally are available to enable one to directly determine whether differences between means have changed.

There two ways one can examine changes in group differences over time when overall rates are changing.  One can compare the sizes of the disparity at each point in time (which is the way the matter is characterized in the third paragraph of this item).  Or one can compare the size of the changes experienced by each group.  Because the Remarkable Results table presented data for a number of groups, the latter approach is less complicated and is primarily used here with regard to those data.[iii] 

A useful feature of the Solutions/Probit approach is that difference between the disparity at two points in time will be the same as the difference between the changes experienced by the groups being compared between the two points in time (as explained in the Subgroups Effects sub-page of the Scanlan’s Rule page).[iv]  To illustrate that point parts of the data from the articles will be analyzed two ways.

Table 1 presents data from the Remarkable Results table in the 2007 Post article, showing (a) the percentage increase in proficiency rates, (b) the percentage decrease in non-proficiency rates; (c) the percentage point changes in proficiency rates; (d) the estimated effect size derived from the earlier rate and later rate.[v] 

 

Table 1 – Changes in Proficiency Rates from Remarkable Results Table (Washington Post, Nov.4, 2007)
Group	Subject	2003 Prof Rate	2007 Prof Rate	IncProf%	DecNonProf%	AbsProfInc	EES
All	Reading	65%	85%	30.77%	57.14%	20	0.65
All	Math	60%	88%	46.67%	70%	28	0.93
Asian	Reading	81%	94%	16.05%	68.42%	13	0.68
Asian	Math	67%	94%	40.30%	81.82%	27	1.11
African American	Reading	64%	72%	12.50%	22.22%	8	0.23
African American	Math	53%	81%	52.83%	59.57%	28	0.8
White	Reading	81%	90%	11.11%	47.37%	9	0.41
White	Math	81%	97%	19.75%	84.21%	16	1.00
Hispanic	Reading	45%	84%	86.67%	70.91%	39	1.19
Hispanic	Math	45%	83%	84.44%	69.09%	38	1.07
Free/Red Meals	Reading	43%	80%	86.05%	64.91%	37	1.02
Free/Red Meals	Math	41%	83%	102.44%	71.19%	42	1.18
Special Education	Reading	33%	77%	133.33%	65.67%	44	1.18
Special Education	Math	33%	81%	145.45%	71.64%	48	1.31
Lim Engl Prof	Reading	8%	83%	937.50%	81.52%	75	2.26
Lim Engl Prof	Math	8%	75%	837.50%	72.83%	67	2.08

 

Table 2 limits the showing to the African American and white changes in math.  One observes a larger percentage increase in proficiency rates for African Americans, a larger percentage decrease in non-proficiency for whites, and a larger percentage point increase for African Americans.  Each change is in the direction that would be expected where the only forces operating are the above-described distributional (prevalence-related) patterns..  The EES figure, however, reveals that the change was greater for whites than African Americans (1.0 standard deviations for whites compared with .8 standard deviations for African Americans).  Thus the disparity increased. 

 

Table 2 – Changes in Proficiency Rates from Remarkable Results Table, (data from Washington Post, Nov.4, 2007) White and African American (Math Only) [ref zz2817]
Group	Subject	2003 Prof Rate	2007 Prof Rate	IncProf%	DecNonPrf%	AbsProfInc	EES
African American	Math	53.00%	81.00%	52.83%	59.57%	28.00	0.8
White	Math	81.00%	97.00%	19.75%	84.21%	16.00	1

 

Creation of the Discipline Disparities page in April 2012, which page makes reference to the figures in Table 2, caused me to recognize a need also to present the information in Table 2 in terms of the differences between African Americans and whites at two points in time (something I had previously done only for the  Asian/White data in the April 5, 2011 Washington Post article, see Tables 3 and 4 below).  Thus, Table 2a shows the differences between African Americans and whites at two points in time.  From this perspective, too, each change is in the direction that would be expected when the only forces operating are the above-described distributional patterns.  (See note iii as to why this would necessarily be the case).   Properly measured, however, the disparity increased.

Table 2a – Changes in Differences between Proficiency Rates from Remarkable Results Table, (data from Washington Post, Nov.4, 2007) White and African American (Math Only) [ref b2817]
Yr	White	African American	FavRatio	AdvRatio	Abs	EES
2003	81.00%	53.00%	1.53	2.47	28	0.8
2007	97.00%	81.00%	1.20	6.33	16	1

 

(A shortcoming of Tables 2a as an example is that the reader may be inclined to think there is something significant in the fact that, while the pairs of EES figures and the pairs of absolute difference figures mean different things in the two tables, those pairs are identical in the two tables.  But that those pairs of figures are identical in the two tables is simply a coincidence.  As noted elsewhere on this page, that the arithmetic difference between the figures in each pair of figures is the same – i.e., 12 percentage points in the case of the pairs of absolute differences  and .20 standard deviations in the case of the pairs of EESs – is not a coincidence.)

Table 3 is based on data presented in the April 5, 2011 Washington Post article. Like Tables 1 and 2, Table 3 shows (a) the percentage increase in scoring at the advanced level, (b) the percentage decrease in failure to score at the advanced level; and (3) the percentage point changes in scoring at the advanced level; (d) the estimated effect size.

Table 3 – Changes in Asian and White Rates of Advanced Level Scores in Virginia and Maryland (data from Washington Post, Apr. 6, 2011)
State	Group	2006 Adv Level	2009 Adv level	AdvLevInc%	NonAdvDec%	AbsProfInc	EES
VA	Asian	59.00%	76.00%	28.81%	41.46%	17.00	0.49
VA	White	43.00%	58.00%	34.88%	26.32%	15.00	0.40
MD	Asian	40.00%	58.00%	45.00%	30.00%	18.00	0.47
MD	White	35.00%	48.00%	37.14%	20.00%	13.00	0.34

 

In Maryland the Asian improvement was larger according to measures (a), (b), and (c).  When that occurs, the group with the larger increase by those measures will invariably have the larger increase according to (d).  In Virginia, however, whites had a larger percentage increase in scoring at the advanced level, while Asians had the larger percentage decrease in failing to reach the advanced level.  Thus, in the Virginia situation, one can determine which group had the larger change only by reference to the EES.  While Asians experienced a larger change in both states (according to the EES), the Asian-white difference was larger in Maryland than Virginia (.13 standard deviations compared with .9 standard deviations), which is consistent with the fact that in Maryland the actual difference in changes was large enough to cause the Asian advantage to appear in all measures. 

Table 4 analyzes the data in terms of the disparities at two points it time. Thus, it presents (a) the ratio of the Asian to white favorable outcome rate; (b) the ratio of the white to Asian adverse outcome rate; (c) the absolute differences between rates; (d) the disparity as measured by the EES. 

 

Table 4 – Changes in Asian-White Disparities in Rates of Advanced Level Scores in Virginia and Maryland (data from Washington Post, Apr. 6, 2011)
State	Yr	Asian Adv Level Rate	White Adv Level Rate	FavRatio	AdvRatio	AbsDf	EES
VA	2006	59.00%	43.00%	1.37	1.39	16	0.43
VA	2009	76.00%	58.00%	1.31	1.75	18	0.51
MD	2006	40.00%	35.00%	1.14	1.08	5	0.14
MD	2009	58.00%	48.00%	1.21	1.24	10	0.27

 

 As discussed above, the difference between the EES scores reflecting the group changes over time will equal the difference between the EES scores reflecting the disparity at each point in time.  It may be noted that in Virginia while the Asians change reflected by the EES was .09 larger than that for whites, the difference in disparity changed by only .08.  The difference between the .08 and .09 figures is simply a result of rounding/inexactness of the procedure as implemented by the Solutions database.