James P. Scanlan, Attorney at Law

Education Trust Glass Ceiling Study

(April 26, 2014; rev. June 28, 2014)

This page is related to the Education Trust HA Study subpage of the Educational Disparities.  Like that subpage, this subpage involves an Education Trust study that attempted to evaluate educational disparities without an understanding of the extent to which observed patterns are functions of normal distributions.  The criticism of the Education Trust’s analyses on this subpage could as well be said of the health and healthcare disparities research of the Harvard School of Public Health, the Harvard Medical School, the Centers for Disease Control and Prevention , the Agency for Healthcare Research and Quality, the National Center for Health Statistics, and the Institute of Medicine, as well as the virtually every other research institution in the world (as recently addressed at pages 24 to 32 of the Federal Committee on Statistical Methodology (FCSM) 2013 Research Conference paper “Measuring Health and Healthcare Disparities.”)    

Most subpages like this take for granted that the reader will have some familiarity with the statistical patterns described on scores of pages on this site.  In this case, however, I set out the basic patterns as they bear on the particular issues examined the Education Trust Glass Ceiling study – (a) falling below the basic level and (b) achieving the advanced level.  The rarer an outcome the greater tends to be the relative difference in experiencing it and the smaller tends to be the relative difference in avoiding it.  Thus, as is pertinent to the subject of this subpage, which involved a period where favorable outcomes were increasing the corresponding adverse outcomes were decreasing, (a) as an adverse educational outcome like falling below the basic proficiency level decreases in overall prevalence, relative differences in experiencing it tend to increase while relative differences in avoiding it tend to decrease; (b) as a favorable outcome like achieving the advanced level increases in overall prevalence relative differences in experiencing that outcome tend to decrease while relative in failing to experience it tend to increase.  It should be evident that (a) and (b) involve the same pattern viewed from different perspectives, since as one outcome increase the other necessarily decreases.  While prevalence related patterns of absolute differences are more difficult to describe, it suffices to note here that the rates of falling below basic are in ranges where decreases in the outcome will tend to reduce absolute differences between rates while the rates of achieving the advanced level are in ranges where increases in the rates will tend to increase absolute differences.  Fuller explanations of these patterns may be found in the introductory section to the Scanlan’s Rule, as well as, among many other places, in the FCSM paper mentioned above note and the October 9, 2012 Harvard University Measurement Letter.

Inasmuch as the Education Trust Glass Ceiling Study relied on absolute differences between rates, its approach may usefully be contrasted with the approach in the 2009 McKinsey & Company issued a report on the titled “The Economic Impact of the Achievement Gap in America’s Schools,” which discussed demographic differences in falling below the basic level in terms of relative difference in the adverse outcome rates (which will tend to increase as those outcomes are decreasing) and demographic differences in reaching the advanced level in terms of relative differences in achieving that level (which will tend to decrease as those outcomes increase).  Thus, the McKinsey approach would tend to reach opposite conclusions from the Education Trust approach. 

I also note here that there is a systematic tendency for the relative difference upon which an observer focuses to change in opposite directions from the absolute difference.  That occurs because the absolute difference tends to change in the same direction as the smaller relative difference.  But observers tend to focus on the larger relative difference simply because it provides the most striking disparity.

 

 

***

On May 14, 2014, the Education Trust issued a report styled “Breaking the Glass Ceiling of Achievement for Low-Income Students and Students of Color.”   The report (at 1) described itself as the first of a series attempting to shed light of racial/ethnic and income differences within the high end of the student populations.  The Education Trust HA Study discusses the April 1, 2014 study that is the second of the series.

The Glass Ceiling report attempted to appraise the size of racial/ethnic and income disparities in meeting or falling below certain proficiency levels based on absolute differences between rates.  The report, however, reflected no understanding of the patterns by which absolute differences between rates tend to change solely as a consequence of overall changes in rates.  Consequently, the report does not provide a sound analysis of whether racial/ethnic and income disparities have been increasing or decreasing.

The report’s most detailed analysis pertains to fourth graders.  One group of findings in the report related to racial/ethnic and income differences in failing to meet the basic level in math during a period from 2003 to 2011 when rates of reaching the basic level improved for all groups.  The report failed to recognize that the rates at issue were in ranges where general decreases in the rates examined would tend to reduce absolute differences regardless of whether there occurred a change in the comparative situation of the two groups. 

Table 1sets out the rates of failing to reach the basic level for the disadvantaged group (DG) and advantaged group (AG) in each comparison (indicated in the second column), along with ratio of AG’s rate of experiencing the favorable outcome (meeting basic) to DG’s rate of experiencing the favorable outcome (RRFav), the ratio of DG’s rate of experiencing the adverse outcome (falling below basic) to AG’s rate of experiencing the adverse outcome (RRAdv) and the absolute difference between rates (AD).  The table also shows the EES (for “estimated effect size”), which is a measure of the difference in circumstances of the two groups (in terms of number of standard deviations between mean scores of the hypothesized underlying distributions) that is unaffected by the prevalence of an outcome.  The table shows that each of the three standard measures behaved as it commonly does in the circumstance.   That is, the relative difference in the generally increasing (favorable) outcome decreased; the relative difference in the generally decreasing (adverse) outcome increased; and the absolute difference decreased.  The EES indicates that, to the extent that the difference in the circumstances of the two groups reflected by each pair rates can be measured, it declined.

Table 1.  Rates at which disadvantaged groups (DG) and advantaged groups (DG) in fourth grade fell belowbasic level in math in 2003 and 2011, with measures of differences.

 

Issue	Compariosn	Yr	DG	AG	RRFav	RRAdv	AD	EES
Below basic	Low v high income	2003	38.00%	12.00%	1.42	3.17	0.26	0.87
Below basic	Low v high income	2011	27.00%	8.00%	1.26	3.38	0.19	0.79
Below basic	Black v white	2003	46.00%	13.00%	1.61	3.54	0.33	1.02
Below basic	Black v white	2011	34.00%	9.00%	1.38	3.78	0.25	0.92
Below basic	Hispanic v white	2003	38.00%	13.00%	1.40	2.92	0.25	0.82
Below basic	Hispanic v white	2011	28.00%	9.00%	1.26	3.11	0.19	0.75

 

The report’s conclusion as to decreasing disparities was thus broadly correct.  But its implication that the absolute difference was a sound measure of disparity in the context examined is incorrect. 

A notable reflection of the failure to understand the patterns be which relative as well as absolute differences tend to be affected by the prevalence of an outcome may be found in the report’s reference to a relative difference as a measure of disparity.  After stating that the decline in absolute differences between rates reflected significant progress, the report states (at 6):   “Still, black, Hispanic, and low-income students were more than three times as likely as their peers to perform within the lowest achievement category in 2011.”  But, as shown in RRAdv column, rather than failing to decline more (as the characterization in the report implies), the relative difference in the adverse outcome increased.

The report also relied on absolute differences to conclude that while most groups increased their rates of reaching the advanced level, disparities increased.  Here, too, the report failed to recognize that the rates were in ranges (well below 50%) where general increases would tend to increase absolute differences.  In this case, however, there was less consistent a pattern of increases for all groups (than in the case of decreases in rates of falling below basic).  Thus, a pattern of increasing disparities could be found for all standard indicators (in which case the EES also would necessarily increase). 

But one exception regarding the Hispanic-white disparity warrants note.  The first two rows of Table 2 present figures discussed in the report for comparison of Hispanics with whites.  In this case, all standard measures changed as they typically do in the circumstances.  And the EES shows an increase in disparity.  But it was a very small increase (from .69 to .71 standard deviations), rather than the fairly large increase suggested by the increase in the absolute difference from 4 to 7 percentage points.

Table 2.  Rates at which Hispanics and whites in fourth grade reached advanced level in math in 2003 and 2011, with measures of differences. [ref b5225a4]

 

Issue	Numbers	Yr	His	White	RRFav	RRAdv	AD	EES
Advanced	Rounded	2003	1.00%	5.00%	5.00	1.04	0.04	0.69
Advanced	Rounded	2011	2.00%	9.00%	4.50	1.08	0.07	0.71
Advanced	Refined	2003	0.80%	5.20%	6.50	1.05	0.044	0.78
Advanced	Refined	2011	2.00%	8.90%	4.45	1.08	0.067	0.70

 

In this case, however, the rates were low enough that the rounding to the nearest percent employed in the report could make a difference.[i]  Thus, the second and third rows of Table 2 show the Hispanic and white rates of reaching the advanced level carried out to a tenth of a percent (as estimated based on the lines in Figure 1b at page 5).  Using these rates, we observe that while again all standard measures behaved as they commonly do in the circumstance, the EES indicates that the disparity decreased (from .78 to .70 standard deviations). 

The report also relies on absolute differences to make a point that the widening of racial and ethnic gaps in reaching advanced level math was more pronounced among higher income groups than lower income groups.  But as shown by the EES figures in Table 3, with respect to the Hispanic-white comparisons (the only comparison for which there are data in Figure 2b at 7), while the gap widened for the lower income group it, decreased for the higher income group.  Thus, not only was there not a more pronounced increase in the gap within the higher income group than the lower income group, to the extent that the gap can be effectively measured, it declined in the higher income group. 

Table 3.  Rates at which Hispanics and white in fourth grade reached advanced level in math in 2003 and 2011, by income level, with measures of differences.

 

Issue	Income Level	Yr	Hisp	White	RRFav	RRAdv	AD	EES
Advanced	Low	2003	0.80%	1.50%	1.88	1.01	0.01	0.23
Advanced	Low	2011	1.50%	3.20%	2.13	1.02	0.02	0.31
Advanced	High	2003	2.00%	6.00%	3.00	1.04	0.04	0.50
Advanced	High	2011	5.00%	12.00%	2.40	1.08	0.07	0.47

 

The report also discusses some gaps among 12^th graders for the years 2005 and 2009, again relying on absolute differences.  That discussion also suffers from the failure to recognize the way absolute differences tend to change simply because the overall prevalence of an outcome changes.