McKinsey Achievement Gap Study
(June 28, 2014)
Prefatory note: Like the main Educational Disparities page of jpscanlan.com and its subpages, this subpage involves a study that examined differences in educational outcome rates without recognizing the way that the measure employed tends to be affected by the overall prevalence of an outcome. To understand this page, the reader should have a general understanding of the patterns by which standard measures of differences between outcome rates tend to change as the prevalence of an outcome changes (as explained on many pages of this site). Specifically, the rarer an outcome, the greater tends to be the relative difference in experiencing it and the smaller tends to be the relative difference in experiencing the opposite outcome. Further, when an outcome rate changes in overall prevalence the group with the lower baseline rate for the outcome will tend to experience a larger proportionate change it is rate of experiencing the outcome, while the other group will tend to experience a larger proportionate change in its rate of experiencing the opposite outcome. Thus, for an obvious example, when a test score is lowered thereby reducing failure rates, relative differences in failure rates tend to increase while relative differences in pass rates tend to decrease; further, the group with the lower failure rate will tend to experience a larger proportionate decrease in its failure rate, while the other group will tend to experience a larger proportionate increase in its pass rate. Absolute (percentage point) differences will also change as the prevalence of an outcome changes. Roughly, as uncommon outcomes (less than 50 percent for both groups being compared) become more common, absolute differences between rates tend to increase; as common outcomes (greater than 50 percent for both groups being compared) become even more common, absolute differences tend to decrease. Where the rate of either outcome is less than 50 percent for one group and more than 50 percent for the other group – or crosses the 50 percent point for either group – the prevalence-related pattern is difficult to predict. See the introduction to the Scanlan’s Rule page of jpscanlan.com. Figure 4 of the 2012 Harvard Applied Statistics Workshop provides some insight into the role of the 50 percent mark (since as a rate moves toward 50 percent, the proportion of the population associated with a given movement across the X-axis increases until the 50 percent point is reached, and thereafter that proportion decreases). As the prevalence of an outcome changes, the absolute difference will tend to change in the same direction as the smaller relative difference and the opposite direction of the larger relative difference.
The study addressed on this page provides an interesting contrast with the study that is the subject of the Education Trust GC Study subpage. The Education Trust Glass Ceiling study discussed on that subpage relied on absolute differences between rates as a measure of differences between rates at which advantaged and disadvantaged groups fell below the basic level or reached the advanced level. As explained on the subpage, that approach would tend, in circumstances of general improvements in education, to find decreasing disparities in rates of falling below basic but increasing disparities in rates of achieving the advanced level. The study addressed on this page, which analyzed disparities in terms of relative difference in falling below the basic level and relative differences in achieving the advanced level, would tend, in circumstances of improvements in education, to find increasing relative differences in falling below basic and decreasing relative differences in reaching the advanced level. Thus, the approach would tend to systematically reach opposite conclusions from the approach employed in the Education Trust Glass Ceiling study. The contrast with the Education Trust Glass Ceiling study highlights a tendency for observers who rely on absolute differences to reach different conclusions from observers who rely on relative differences whenever the situation examined involves very high rates for one outcome with corresponding very low rates for the opposite outcome. For in such cases, the observer relying on the relative difference will tend to focus on the relative difference for the outcome with the low rate simply because it will be the larger relative difference. And, as noted, that relative difference will tend to change in the opposite direction of the absolute difference.
***
In April 2009, McKinsey & Company issued a report titled “The Economic Impact of the Achievement Gap in America’s Schools.” Along with 24-page report itself, McKinsey issued a 199-page document titled “Detailed findings on the economic impact of the achievement gap in American’s schools.” Much of the discussion in the report is about demographic differences in achievement scores – that is, differences in continuous variables. While not necessarily agreeing with the report’s analyses of such issues, my concern here is with binary variables –i.e., achieving or failing to achieve a certain level of proficiency.
In the instances where the report discusses differences regarding binary variables, it relies of relative differences (or measures that are functions of relative differences) with respect to the larger relative difference – that is, (a) relative difference in the adverse outcome with respect to falling below/or achieving the basic level and (b) relative differences in the favorable outcome with respect to falling below/or achieving the advanced level. Since, as the prevalence of an outcome changes, the larger relative difference tends to change in the opposite direction of the absolute difference, the McKinsey approach will tend to reach conclusions that are the opposite of those based on absolute differences between rates.
There are at least two matters where the McKinsey approach discusses disparities in binary measures. The first involves differences in rates of failing to reach the basic competence level in Massachusetts, where the study regarded the relative difference to be especially larger. The second involves difference in reaching the advanced level where the study regards the fact that the relative decline in rates of reaching between the fourth and twelfth grades is greater for minorities than whites to be an indication of increasing disparity. In neither case does the report reflect an awareness of the way the measure employed tends to be affected by the prevalence of an outcome or of the potential for the relative difference as to the opposite outcome of that examined or the absolute difference to yield a conclusion that is different from the report’s conclusion. These two cases are discussed in Sections A and B. Section C uses data from the report to illustrate certain patterns, though without reference to the characterizations of differences in the report.
A. Relative Differences in Falling Below/Above Basic Level
There are occasional statements in the report suggesting that one would expect that as scores improve overall, there will be a reduction in the disparity or that areas with high achievement scores should have lower disparities (though such statements are made in the context of pointing out that such is not the case). I do not know that there is a basis for such expectation even as to continuous measures. There is, however, a basis to believe that relative differences in adverse outcomes will be associated with the level of achievement. But the association is one whereby the higher the general level of achievement, the larger will tend to be the relative difference in the adverse outcome.
In any event, some statements in the report about the absence of smaller achievement gaps in areas with generally high achievement levels are accompanied by discussions of relative difference in failure to achieve the basic competency level. These include the following statement in the report (at 10):
“Interestingly, the size of the racial achievement gap is not correlated with overall state performance. Massachusetts, for example, has among the highest overall scores on NAEP, but blacks and Latinos there are eight times more likely to underperform in fourth grade math than are whites.” [i]
As noted above, and explained on many pages of this site, there is reason to expect that states with generally high achievement levels will have larger relative differences in adverse outcomes, though smaller relative differences in the corresponding favorable outcomes, than states with generally low achievement levels simply because states with high achievement scores will tend to have low adverse outcome rates.
The McKinsey study’s Detailed Findings (at 26) provide the actual rates at which whites, blacks and Latinos fall below the basic level for reading and math in the fourth grade. In doing so, the study relies on these relative differences in falling below basic as indicating that “the relative achievement gap in Massachusetts is also among the highest.” The data set out there, in conjunction with the national data found at page 34, allows one to compare Massachusetts data with national data.
Table 1 below presents the pertinent rates for whites, blacks, and Latinos, along with rate ratios for falling below basic the basic level and reaching the basic level, the absolute difference between rates, and the EES (for estimated effect size). For three of the four comparisons (white-black, math and reading; white-Latino, reading), we observe the common prevalence-related pattern whereby relative differences in the adverse outcome is larger, while the relative difference in the corresponding favorable outcome is smaller, in the setting where the adverse outcome is less common.
The absolute difference column indicates that, if the study had relied on absolute differences between rates to measure the gap, as is the most common approach to the measuring outcome disparities in education (as in the Education Trust and Annie E. Casey Foundation studies that are the subjects of the pages in studies mentioned in the prefatory note), the McKinsey study would have found Massachusetts to have below average disparities in three of the four cases.
The EES tells us that, to the extent the difference in circumstances reflected by a pair of rates can be measured, it is greater in Massachusetts than nationally in all four cases. But the magnitude to the difference is less than it would appear according to the relative differences in adverse outcomes.
Table 1: White and Minority Rates of Falling Below Basic Levels in Math and Reading in Massachusetts and the United States, with Measures of Differences between White and Minority Rates [b5424 a 2]
|
Subject
|
Comparison
|
Area
|
W
|
M
|
Min/Wh Ratio Below
|
Wh/Min Ratio Above
|
AbsDiff
|
EES
|
Math
|
White-Black
|
Mass
|
3%
|
25%
|
8.33
|
1.29
|
0.22
|
1.2
|
Math
|
White-Black
|
US
|
9%
|
37%
|
4.11
|
1.44
|
0.28
|
1
|
Math
|
White-Latino
|
Mass
|
3%
|
23%
|
7.67
|
1.26
|
0.2
|
1.14
|
Math
|
White-Latino
|
US
|
9%
|
31%
|
3.44
|
1.32
|
0.22
|
0.84
|
Reading
|
White-Black
|
Mass
|
13%
|
43%
|
3.31
|
1.53
|
0.3
|
0.945
|
Reading
|
White-Black
|
US
|
23%
|
54%
|
2.35
|
1.67
|
0.31
|
0.84
|
Reading
|
White-Latino
|
Mass
|
13%
|
45%
|
3.46
|
1.58
|
0.32
|
0.985
|
Reading
|
White-Latino
|
US
|
23%
|
51%
|
2.22
|
1.57
|
0.28
|
0.765
|
The report makes the same point about Massachusetts with respect to disparities between high and low income students, and shows the Massachusetts figures for falling below basic by eligibility for free or reduced lunch at page 43. But the report does not provide the same type of data for the nation. So it is not possible to create a table on differences by income similar to Table 1.
B. Relative Difference in Reaching/Failing to Reach Advanced Level
The report relies on an aspect or relative differences in favorable outcomes when discussing disparities in rates of reaching the advanced level for average scores on reading and math. At page 35, the report shows rates of reaching the advanced level for the white, blacks and Latinos in grades 4, 8, and 12. The report does not present a measure of the differences. But, in stating that the trend in underrepresentation of blacks and Latinos among the high performers “is amplified over the life of the student,” the study points out that the white rate of reaching the advanced level declines by 60% between the fourth and twelfth grads, while the black and Latino rates decline by 87% and 65%.
Table 2 sets out the relevant figures. In this case, the pattern whereby the rarer an outcome the greater tends to be the relative difference in experiencing it and the smaller tends to be the relative difference in avoiding is manifested in the pattern whereby groups with lower baseline rates tend to experience a larger proportionate reduction in the declining outcome than groups with higher baseline rates, while the other groups experience the larger proportionate increase in the corresponding increasing outcome.
Comparing whites with both minority groups, we observe the typical patterns of relative changes – that is, the groups with lower baseline rates of achieving advanced status (blacks and Latinos) experience a larger proportionate decrease in their rates of reaching the advance level than whites, while whites experience the larger proportionate increase in the opposite outcome.
We do not, however, observe that patterns for relative changes when comparing Latinos with blacks. Latinos, while having a higher baseline rate than blacks, experienced a larger proportionate decline than blacks. The EES figure indicates that, to the extent that the change can be measured, it was greatest for Latinos, followed by whites, and then blacks.
Relying on absolute differences as a measure of disparity (as in the Education Trust and Annie E. Casey Foundation studies) the decline would have been deemed largest for whites followed by Latinos and then blacks.
Table 2: White and Minority Rates of Reaching the Advanced Level for Math and Reading Combined in Fourth and Twelfth Grades, with Measures of Changes in Rates [b5436a1]
|
Race
|
4
|
12
|
Adv Per Decrease
|
Below Adv Perc Increase
|
Abs Dec
|
EES
|
White
|
10%
|
4.0%
|
60%
|
6.67%
|
0.06
|
0.469
|
Latino
|
3%
|
0.4%
|
86.67%
|
2.68%
|
0.03
|
0.77
|
Black
|
2%
|
0.7%
|
65%
|
1.33%
|
0.01
|
0.4
|
Table 3 now shows the patterns in terms of the differences between white and minority rates of achieving or failing to achieve the advanced level.[ii] And we observe the usual pattern for the standard measures.[iii] As the favorable outcome decreased, relative differences in the favorable outcome increased while relative differences in the adverse outcome decreased. And, as commonly happens in the rate ranges at issue, absolute differences decreased. The EES figure, however, shows that, in accordance with the EES patterns in Table 2, the disparity between whites and blacks decreased while the disparity between whites and Latinos.
Table 3: White and Minority Rates of Reaching the Advanced Level for Math and Reading Combined in Fourth and Twelfth Grades, with Measures of Differences Between White and Minority Rates [b5436a1]
|
Comparison
|
Grade
|
W
|
M
|
Wh/Min Fav Ratio
|
Min/Wh Adv Ratio
|
Abs Diff
|
EE
|
White-Black
|
4
|
10%
|
2%
|
5
|
1.09
|
8%
|
0.77
|
White-Black
|
12
|
4%
|
0.70%
|
5.71
|
1.03
|
3.30%
|
0.71
|
White-Latino
|
4
|
10%
|
3%
|
3.33
|
1.08
|
7%
|
0.6
|
White-Latino
|
12
|
4%
|
0.40%
|
10
|
1.04
|
3.60%
|
0.9
|
C. Further Illustrations Regarding Falling Above/Below Basic Level
At page 34 of the Detailed Findings, in order to illustrate the way disparities in reaching various levels persisted across subjects and across years, the McKinsey study presents figures on the proportions of whites, blacks, and Latinos falling into three categories: (a) advance, (b) proficient or basic, and (c) below basic. The study did not quantify the disparities reflected in the data.
The data are nevertheless useful for illustrative purposes. From the categorical figures on page 34, one can derived percentages of each group falling above or below the cut point for basic and the cut point for advanced. Table 4 presents information based on falling above or below the basic level cut point.
(The minority figures for reaching the advance level are sufficiently low that rounded figures shown on the page cannot be reasonably analyzed.[iv] At any rate I am not doing so at this time.)
Table 4 presents, along with rates at which each racial group falls below the basic level in reading and math in the fourth and eighth grades, the same the measures of differences between white and minority rates shown in Table 1 and 3. The table illustrates the following patterns with respect to the relative differences.
Between the fourth and eighth grades, the below basic rates reading rates decreased for all groups. With respect to both the white-black and Latino-white differences, as commonly happens in those circumstances, the relative differences in falling below basic increased, while the relative difference in reaching basic or above decreased. Between the fourth and eighth grade, the below basic math rates increased for all groups. With respect to both the white-black and Latinos-white differences, as commonly happens in those circumstances, the relative differences in falling below basic decreased, while the relative difference in reaching basic or above increased.
In both cases, as commonly happens, the absolute differences changed in the same direction as the smaller relative difference. The EES indicates that, to the extent that the disparities can be measured, for reading, the black-white and Latinos-white disparities were essentially the same in the eighth grade as in the fourth grade; for math, the black-white disparity was essentially the in the eighth grade as the fourth grade, while the Latinos-white disparity increased somewhat between the fourth and eighth grades.
Table 4: White and Minority Rates of Falling Below the Basic Level for Reading and Math in Fourth and Eighth Grades, with Measures of Differences Between White and Minority Rates [b5436a1]
|
Subject
|
Comparison
|
Grade
|
W
|
M
|
RRFav
|
RRAdv
|
Abs Df
|
EES
|
Reading
|
White-Black
|
4
|
23%
|
54%
|
1.67
|
2.35
|
0.31
|
0.84
|
Reading
|
White-Black
|
8
|
17%
|
46%
|
1.54
|
2.71
|
0.29
|
0.84
|
Reading
|
White-Latino
|
4
|
23%
|
51%
|
1.57
|
2.22
|
0.28
|
0.765
|
Reading
|
White-Latinos
|
8
|
17%
|
43%
|
1.46
|
2.53
|
0.26
|
0.77
|
Math
|
White-Black
|
4
|
9%
|
36%
|
1.42
|
4
|
0.27
|
0.98
|
Math
|
White-Black
|
8
|
18%
|
53%
|
1.74
|
2.94
|
0.35
|
0.995
|
Math
|
White-Latino
|
4
|
9%
|
31%
|
1.32
|
3.44
|
0.22
|
0.84
|
Math
|
White-Latino
|
8
|
18%
|
45%
|
1.49
|
2.50
|
0.27
|
0.785
|
[i] The report commonly uses the term “times higher” when it should be using the term “times as high.” This usage is the subject of the Times Higher subpage of the Vignettes page of jpscanlan.com.
[ii] The relationship between Tables 2 and 3 can be compared to the relationship between Tables 2 and 2a of the main Educational Disparities page and between Tables 1 and 2 of the Criminal Record Effects subpage of the Scanlan’s Rule page.
[iii] Note that Table 3 compares whites with blacks and whites with Latinos. As indicated above, a comparison of blacks and Latinos would not show the usual pattern, in consequence of an increase in the Latino-black difference that is sufficient to cause the usual prevalence-related patterns not to be observed.
[iv] See discussion of Tables 2 and 3 in the Education Trust GC Study subpage regarding the potential for more precise figures to yield results different from those derived from rounded figures when the reported outcome rates are very low.