James P. Scanlan, Attorney at Law

California Reading Instruction Competence Assessment

(July 12, 2019)

            The pattern by which relaxing a standard, while tending to reduce relative differences in rates of meeting the standard, tends to increase relative differences in rates of failure to meet the standard is discussed in many places on this site.  The pattern is most easily illustrated with test score data, as in Table 1 (at 329) of “Race and Mortality Revisited,” Society (July/Aug. 2014) and Table 1 (at 20) and Figure 1 (at 22) of my Comments for Commission on Evidence-Based Policymaking (Nov. 14, 2016).  Such fact makes especially remarkable the failure to understand that large relative racial/ethnic differences in failure to meet academic standards (or the large proportions racial/ethnic minorities make up of persons failing to meet the standards) is a function is the leniency rather than the stringency of the standards, as I discussed almost three decades ago in “An Issue of Numbers,” National Law Journal (Mar. 5, 1990), and “The Perils of Provocative Statistics,” Public Interest (Winter 1991).  That issue appears to be no better understood today than when those items were published.  In essence, observers typically that lowering a test cutoff tends to reduce relative differences in pass rates.

            A situation where the issue may prove pertinent is discussed in a May 25, 2019 EdSource article regarding attention given recently to difficulties teachers have in passing a test called the Reading Instruction Competence Assessment (RICA).  While there appear to a number of objections to the test, including its general difficulty, some part of the opposition to it is apparently based on the fact that Latinos and African Americans have greater difficulty passing the test than other persons.  The article casts the matter in terms of both failure rates and pass rates.  With regard to the period between 2012 to 2017, it notes both that failure rates for first time takers were 33 percent overall and 45 percent for Latinos and African Americans and that 91 percent of all test takers eventually passed compared with 86 percent of Latinos and 85 percent of African Americans.  The article does not quantify any of the differences.

            The figures presented in the EdSource article themselves allow for an illustration of the ways measures tend to be affected by the prevalence of an outcome that is comparable to that in Table 1 of "Race and Mortality Revisited" and Table 1 of the CEP Comments.  The two rows in the referenced tables show the pass (and corresponding failure) rates for a higher- and lower-scoring group at a higher cutoff, where pass rates are 80% and 63%, and a lower cutoff, where pass rates are 95% and 87%.  The pattern shown in the tables whereby lowering the cutoff reduces relative difference in pass rates while increasing relative differences in failure rates is similar to what one would commonly observe if persons who initially failed the test were allowed to retake the test.

            Table 1 below, which is based on the above-mentioned pass or failure rates discussed in the May 2019 EdSource article, show separately for comparisons of outcome rates of (a) test takers who are neither Latino nor African American (NLAA)[i] with Latino test takers and (b) test takers who are neither Latino nor African American with African American test takers.  The first and second column show the comparison being made with respect to either initially passing (or failing) the test and ultimately passing (or never passing) the test.  The next four columns show the pass and fail rates of the advantaged group (AG) and disadvantaged group (DG) being compared.  The next two columns show that, as would be expected in the circumstances, allowing persons to retake the test reduced relative differences in securing the credential but increased relative differences in rates of failing to secure the credential.  The last column shows the measure described in "Race and Mortality Revisited" and the CEP Comments – which I commonly term EES for estimated effect size and which is also termed probit d' – that is theoretically unaffected by the prevalence of an outcome. 

Table 1.  Advantaged and disadvantaged racial/ethnic groups’ rates of initially passing (or failing) and ultimately passing (or never passing) the RICA, with measures of difference

            The differences between the EES values for initially passing (or failing) and ultimately passing (or never passing) are small.  Given the groups’ rates for initially passing (or failing) the test, and NLAA figures for ultimately passing (or failing), one would predict Latino and African American rates of ultimately passing (or failing) that are very close the figures actually shown.[ii]  That will commonly be the case in situations where persons failing a test are allowed to retake it.  But it does not have to be the case.  The figures EES figures in two rows could differ substantially even when the pattern whereby the two relative differences change in opposite patterns is shown.  The important thing to keep in mind, however, is that it is the variation in EES values that may provide a reason for studying underlying processes.  But neither the increasing relative difference in the decreasing outcome nor the decreasing relative difference in the increasing outcome, over time or from setting to setting, would not warrant an effort examine the underlying processes, at least not without attention to whether the pattern is greater or less than what would be expected in the circumstances.  See discussion of Table 2 in "Race and Mortality Revisited" (at (329-330, 343).

            The RICA test is now being examined by a literacy expert group commissioned by the California Commission on Teacher Credentialing.  It is not clear how much attention will be given to the racial/ethnic impact of the RICA in discussions of whether the test is too difficult or otherwise warrants revision.  Nor is it clear whether the attention to racial impact will focus or relative differences in passing the test that making the test easier will tend to reduce or relative differences in rates of failure to pass the test will making the test easier will tend to increase.  As with the NCAA eligibility requirements and the Georgia teacher test discussed in the “An Issue of Numbers” and “The Perils of Provocative Statistics,” the easier the test, and thus the more commonly the relative racial/ethnic differences in failure to pass the test (and the proportion disadvantaged groups make up persons failing to pass the test) will be very large, the greater the likelihood that the focus will be on failure to pass the test. 

            The EdSource article also discussed the expense and inconvenience of retaking the test.  Like making the test easier or lowering the cutoff, making the less expensive and inconvenient will tend to reduce relative racial/ethnic differences in ultimately passing the test but increase relative differences in failure ever to pass the test.