Many studies have found that the proficiency levels needed to "meet standards" on the Illinois Standard Achievement Test (ISAT) (given to third- through eighth-graders) are among the lowest in the nation, that they are much lower than the proficiency levels needed to "meet standards" on the Prairie State Achievement Exam (PSAE) (given to eleventh-graders), and that they are even further behind when compared to the proficiency levels needed to be on track to college and career readiness. See sidebar.
In a recent filing with the U.S. Department of Education, the Illinois State Board of Education (ISBE) said it will raise the proficiency level needed to meet standards on the ISAT to align them with the PSAE and with college and career readiness. An internal ISBE study, though, shows that the ISAT proficiency level that ISBE settled on in its filing is significantly below the level needed to be on track to college and career readiness, as measured by the benchmarks identified by the ACT.
ISBE told the RoundTable that nothing is final and that it will convene a "standard setting process" this summer to determine the new cut scores (e.g., the proficiency level) for the ISATs. However, the statements ISBE made in its recent filing with the U.S. Department of Education, like a blade of grass, may show which way the wind is blowing.
ISBE’s Stated Plan in Its Waiver Request
In September 2011, the U.S. Department of Education invited states to apply for waivers from the accountability measures of the No Child Left Behind Act. President Barack Obama said, "To help states, districts and schools that are ready to move forward with education reform, our administration will provide flexibility from the law in exchange for a real commitment to undertake change." The flexibility is intended to support reform efforts in critical areas, including "transitioning to college and career-ready standards and assessments."
On Feb. 23, ISBE filed a request for a waiver with the U.S. Department of Education saying it was aggressively moving forward with a plan to transition to college ready standards. As part of this effort, ISBE said it will raise the cut scores (e.g., the proficiency levels) to "meet standards" on the ISATs to align them with college and career readiness benchmarks.
To do this, ISBE says it decided to use the cut scores to "meet standards" on the PSAE as the benchmarks for college readiness. This decision was based on its study that compared the cut scores needed to meet standards on the PSAEs with the ACT’s college readiness benchmarks using a method called "standard equipercentile equating," said ISBE.
"In general, the PSAE scores differed little from ACT’s college benchmarks," says ISBE in its waiver request. "Thus, we are confident that high school students are college ready in reading, mathematics, or science when their PSAE scores fall in the ‘meets standards’ or ‘exceeds standards’ categories."
After deciding to use the PSAE "meet standards" cut scores as the benchmark for college and career readiness, ISBE back-mapped those cut scores to ostensibly identify cut scores on the ISATs that are aligned with the college and career readiness benchmarks of the ACT. ISBE says in its waiver request that it "anticipates that the new [ISAT] cut scores will result in approximately 50 percent of Grades 3-8 students meeting or exceeding standards on ISATs," and thus being on track to college and career readiness.
In other words, ISBE anticipates that the benchmark for college and career readiness will be scores at approximately the 50th percentile on the 2011 ISATs.
Using PSAE "meet standards" cut scores as the benchmark for college and career readiness low-balls the standard. ISBE’s own study found that the PSAE "meet standards" cut scores are significantly lower than those needed to meet the college and career readiness benchmarks of the ACT.
ISBE’s Study and the Choice of a Benchmark Score
In submitting its waiver request, ISBE told the RoundTable, it relied on a report entitled, "College & Career Readiness for Illinois State Assessments," prepared by Pooja K. Agarwal, Ph.D. and Rense Lange, Ph.D. They were both employed by ISBE at the time they prepared the report.
The report itself found that all of the PSAE scores needed to "meet standards" are lower than the PSAE scores that correspond with the college readiness benchmarks of the ACT. For example, the PSAE "meet standard" cut scores are positioned at about the 50th percentile, while the PSAE scores that correspond with the college readiness benchmarks of the ACT range from the 65th percentile up to the 80th percentile.
Despite these differences, the ISBE study decided to use the PSAE "meet standards" cut scores as the benchmark for college readiness. The report explains, "It can be observed that except for science, the projected PSAE college readiness scores differ little from existing PSAE ‘Meet Standards’ scores. Given that the PSAE college readiness scores and the PSAE current ‘Meet Standards’ scores are closely aligned, it is reasonable to operationalize college readiness in terms of the existing PSAE ‘Meets’ scores."
Thus, according to the study, high school students are deemed to be college ready in reading, mathematics, or science whenever their PSAE scores exceed the PSAE "meet standards" cut scores for those subjects, even though those scores are substantially lower than the PSAE scores that correspond with ACT college readiness benchmarks.
Table 1 below compares the PSAE "meet standards" cut scores with the PSAE scores that Drs. Agarwal and Lange found corresponded with ACT college readiness benchmarks:
What appear to be relatively small numerical differences in scale scores actually translate into very substantial differences in statistical value. These statistical differences are illustrated in Table 2 below using percentile ranks that are taken directly from the Agarwal/Lange report:
The RoundTable asked Paul Zavitkovsky of the Urban Education Leadership Program at the University of Illinois-Chicago about these differences. Mr. Zavitkovsky said, "It’s not really possible to judge how closely aligned scale scores are until you convert those scores into percentile ranks. Once you do that here, the differences in percentile values for reading and math are substantial and the difference in science values is enormous."
He added, "It’s heartening to see that ISBE is using the NCLB waiver process to align the way elementary and high school test results will be reported in the future. But it’s hard to see how the numbers justify using the meet standards cut score on the PSAE as a surrogate for college readiness."
One way to see the practical impact of using the meet standard PSAE scores instead of the college ready PSAE scores is to compare the difference between a) the percentages of Illinois eleventh-graders who met standards on the PSAE in 2011 and b) the percentage who met the college readiness benchmarks on the ACT in 2011. As reflected in the accompanying chart (Figure 1), 51% of all 11th graders tested met standards in reading on the PSAE, but only 40% of those same students met the ACT college readiness benchmark. For math, 51% met standards, while only 39% met the ACT benchmark. For science, 49% met standards, yet only 26% met the ACT benchmark.
In numerical terms, about 72,000 eleventh-graders met standards in reading and math on the 2011 PSAEs, yet only about 58,000 of those same students met the college readiness benchmarks of the ACT. In science, about 71,500 met standards on the PSAEs, yet only about 37,700 met the ACT benchmark.
Using the PSAE "meet standards" score, rather than the PSAE score that corresponds to college readiness on the ACT, makes a significant difference. Using the lower PSAE "meet standards" scores significantly lowers the bar for what ISBE is calling "college readiness."
In separate interviews, the RoundTable explored why Drs. Agarwal and Lange said in their report, "It is reasonable to operationalize college readiness in terms of existing PSAE ‘Meets’ scores." Dr. Agarwal told the RoundTable she personally thought the PSAE would be better to use than the ACT because the PSAE includes both the ACT and the WorkKeys assessment, which is an indicator of career readiness. When asked why she used the PSAE "meet standards" scores, rather than the PSAE scores that corresponded to ACT college readiness, she said she disagreed with the way ACT set its benchmarks. She said she thought educators should look at the ISATs to set a cut score that indicated being on track to college readiness, adding that her report was preliminary and that ISBE planned to set cut scores in a "full standard setting" that she supported.
Dr. Lange said a group of experts selected the cut scores to meet standards on the PSAEs in the early 2000s, but he acknowledged they were not trying to align the cut scores with college readiness at that time. When asked, "If you wanted to determine an ISAT score that best corresponded to being on track to meeting the ACT’s college readiness benchmarks, should you back map from the PSAE college readiness score, as opposed to the PSAE meet standards score." He replied, "Yes, if you wanted to identify the score that best corresponded."
The Back-Mapped ISAT Scores Low-Ball the Benchmark for College Readiness
The Agarwal/Lange study back-mapped from the PSAE "meet standards" cut scores to ostensibly determine ISAT cut scores for third- through eighth-grades that indicate being on track to college readiness. The ISAT cut scores they identified for eighth-graders are 251 for reading and 271 for math. The ISAT cut score for science, which is tested in seventh but not eighth grade, is 239. These scores all equated to approximately the 50th percentile in 2011.
Because the PSAE "meet standards" cut scores are lower than the PSAE scores that correspond with the ACT college readiness benchmarks (See Table No. 1 on page 20), one would expect the back-mapped ISAT scores to be lower than the scores needed to be on track to ACT college readiness. In fact, a number of studies already indicate that the proposed ISAT scores for college readiness are substantially lower than they need to be to predict ACT college readiness with at least a 50% probability.
For starters, Dr. Agarwal’s and Lange’s own report contains a chart that shows the "relationship between the probability of students being college ready in math on the  PSAE and their prior  ISAT grade 8 math scores." The chart (Figure 2) shows that an eighth-grader with an ISAT score of 271 in math (the ISAT score identified in the Agarwal/Lange report as being on track to college readiness) has only about a 20% percent chance of obtaining PSAE college readiness score in eleventh grade.
A 20% chance of being on track to college readiness is very low. Dr. Lange told the RoundTable that deeming a student to be on track to college readiness in eighth grade if they have only a 20% chance of being college ready in eleventh grade "doesn’t make any sense." He said he would want the probability to be at least 50%.
The chart (Figure 2) also shows that, on average, an eighth-grader in 2008 needed to obtain an ISAT score of about 284 in math to have a 50% probability of obtaining a PSAE college readiness score as an eleventh-grader in 2011.
The chart from the Agarwal/ Lange study is reproduced here as Figure 2, with lines added that move up from the ISAT score of 271 and down from the point where students have a 50% probability of meeting or exceeding the PSAE college readiness score in 11th grade.
Drs. Agarwal and Lange’s report says, "Analogous results were obtained for Reading and Math," but those results were not displayed in the report. The RoundTable asked ISBE for the results for reading and math, but was told they were not available.
Second, as reported in earlier articles published by the RoundTable, Mr. Zavitkovsky compared multiple years of ISAT and ACT results to estimate the ISAT scale scores that indicate students are likely to be on-track for college readiness at grades 3-8 (i.e. have a 50% chance or better of meeting or exceeding college-readiness benchmarks in eleventh grade). The ISAT scale scores he identified for eighth-graders are 256 for reading and 284 for math.
Significantly, the eighth-grade ISAT score of 284 in math identified by Mr. Zavitkovsky is the same score identified in the Agarwal/Lange chart (Figure 2) as having a 50% probability of achieving a PSAE college readiness score in eleventh grade.
Mr. Zavitkovsky’s estimates are based on five years of statewide and district-level comparisons between eighth-grade ISAT and eleventh-grade ACT scores, three years of comparisons between fifth and eleventh grade scores and five years of comparisons from grades three through eight. In all cases, he found that students statewide typically needed to be at or above the 60th Illinois percentile in reading and at or above the 66th Illinois percentile in math, to be on track to ACT college readiness in 11th grade.
Instead of lining up with the 60th or 66th percentiles, the Agarwal/Lange report and ISBE settle on the 50th Illinois percentile.
A third way to look at the ISAT scores identified in the Agarwal/Lange report as being indicators of college and career readiness is to compare: a) the percentage of eighth-graders meeting or exceeding those ISAT scores with, b) the percentage of eighth-graders meeting the college readiness benchmarks on the EXPLORE test, a test in the ACT family of tests.
In its waiver request filed with the U.S. Department of Education, ISBE says that of the Illinois eighth-graders who took EXPLORE in 2011, 44.7% met the college readiness benchmarks in reading, 34.1% met the benchmarks in math, and 16.3% in science. By contrast, approximately 50% of Illinois eighth-graders met or exceeded the ISAT cut scores identified in the Agarwal/Lange report as being indicators of college and career readiness. Figure 3 illustrates the difference. (It should be noted that only 80% of the students who took the ISAT also took the EXPLORE test.)
The ISAT cut scores proposed in the Agarwal/Lange report are set at a significantly lower proficiency level than the benchmarks for college and career readiness identified by ACT for the EXPLORE test.
Other studies have identified ISAT cut scores indicating college readiness that are significantly higher than those identified in the Agarwal/Lange report. In "From High School to the Future: Pathway to 20" (2008), researchers at the Consortium on Chicago School Research at the University of Chicago found that the cut scores indicating college readiness for eighth-graders in the Chicago School System were in the low 260s for reading and the low 290s for math. In "Illinois Comparability Study Linking EXPLORE to ISAT" (2009), ACT researchers found the scores were 259 in reading and 286 in math.
What’s Next at ISBE
When asked to comment on these issues, Mary Fergus, spokesperson for ISBE, told the RoundTable that the Agarwal/Lange report is preliminary and "it is just one report and it may or may not be used in the final decision." She added there is often give and take in negotiating a waiver request with the U.S. Department of Education and that nothing is final.
In its waiver request ISBE said it "will finalize the new cut scores with our Technical Advisory Committee in spring 2012." Ms. Fergus told the RoundTable that ISBE will begin a "standard setting process" this summer, and that the Technical Advisory Committee will be involved in that process. "A standard setting generally involves convening a large number of educators, including teachers from each grade level who are familiar with the range of capabilities of students in that grade, as well as other experts, to determine cut score by grade," she said.
The level at which ISBE will set the final ISAT cut scores for college readiness remains to be seen.