In February 2012, the Illinois State Board of Education (ISBE) filed with the U.S. Department of Education an application for a waiver from certain requirements of the No Child Left Behind Act.
In its application, ISBE acknowledged that the cut scores (e.g. the proficiency levels) to “meet standards” on the Illinois Standard Achievement Test (ISAT) (given to third- through eighth-graders) were not aligned with the cut scores to “meet standards” on the Prairie State Achievement Exam (PSAE)(given to eleventh-graders). Nor were they aligned with the proficiency level needed to be on track to meet ACT’s college readiness benchmarks by eleventh grade.
“To ensure that our continuous system of assessments aligns to college- and career-ready benchmarks, Illinois will raise the cut score for the ISAT assessment,” said ISBE in its waiver application. ISBE theorized that the cut scores to “meet standards” on the PSAEs “differed little” from ACT’s college readiness benchmarks, so if cut scores to “meet standards” on the ISATs were aligned to the cut scores to “meet standards” on the PSAEs, the ISAT cut scores would be aligned to college readiness.
On Jan. 24, 2013, ISBE decided to raise the cut scores to “meet standards” on the ISATs. In a prepared statement announcing the change, State Superintendent of Education Christopher A. Koch said, “The lower expectations of the previous performance levels did our students a disservice by not adequately assessing their ability to succeed after high school. The new, higher expectations will provide more accurate information about a child’s development and allow us to provide the appropriate supports and interventions earlier in a student’s academic career to ensure he or she is on track to enter college or career-training programs.”
While raising the cut scores to “meet standards” on the ISATs is a positive step, an internal study conducted by ISBE and other analyses show that the new cut scores are still below the proficiency levels needed to be aligned with the cut scores to “meet standards” on the PSAEs. Moreover, they are not aligned with ACT’s benchmarks for college readiness.
The chart below graphically shows the misalignment.
ISBE’s Waiver Application
For many years, the proficiency levels to “meet standards” on the ISATs have been among the lowest in the nation and substantially below the proficiency level needed to “meet standards” on the PSAEs. This has caused consternation throughout the State because approximately 40% of the eighth-graders who “met standards” on the ISATs have failed to “meet standards” on the PSAEs as eleventh-graders.
In its 2012 waiver application ISBE acknowledged, “ISAT results are not aligned to PSAE results.” ISBE pointed out that 85 percent to 86 percent of third- through eighth-graders “met standards” in reading and math on the 2011 ISATS, yet only 51 percent of eleventh-graders met standards on the PSAE.
ISBE also acknowledged the misalignment was due to using low cut scores – low proficiency levels – to determine whether students “met standards” on the ISAT.
ISBE said, “We do not attribute this drop in proficiency to a drop in student learning in Grades 9 and 10. To the contrary, we are confident that this drop in proficiency reflects the misalignment between the ISAT ‘meets standards’ scores and the PSAE ‘meets standards’ scores.”
ISBE also told the U.S. Department of Education that it had conducted an equipercentile analysis and if the cut scores to “meet standards” on the ISATs were aligned with the cut scores to “meet standards” on the PSAEs, it “will result in approximately 50 percent of Grades 3-8 meeting or exceeding standards on ISAT.”
In other words, to align the ISATs with the PSAEs, the cut scores to “meet standards” on the ISATs would need to be set at approximately the 50th percentile.
ISBE told the RoundTable that the equipercentile analysis referred to in its filing with the U.S. Department of Education was contained in a report, “College & Career Readiness for Illinois State Assessments,” prepared by Pooja K. Agarwal, Ph.D. and Rense Lange, Ph.D., both ISBE employees at the time of the study.
The RoundTable obtained a copy of the report through a freedom of information request in March 2012. That report identified cut scores to meet standards on the ISATs that were aligned with the cut scores to meet standards on the PSAEs. The ISAT cut scores identified in the Agarwal/ Lange report correspond to the 51st percentile for reading and the 53rd percentile for math.
New ISAT Cut Scores Misaligned With PSAE
In adopting new cut scores to “meet standards” on the ISATs on Jan. 24, 2013, ISBE did not adopt the ISAT scores that Drs. Agarwal and Lange found were aligned with the PSAE cut scores to “meet standards.” Instead, ISBE adopted lower scores.
In a prepared statement announcing the new cut scores, ISBE indicates that if the new adopted cut scores were applied, about 60% of students would “meet standards” on the ISATs, indicating that the new cut scores correspond to the 40th percentile, rather than the 51st or 53th percentiles.
Paul Zavitkovsky, of the Urban Education Leadership Program at the University of Illinois-Chicago, determined the percentile rank of new cut scores for the ISATs. The percentile rank for the cut scores varies by grade level, but it corresponds to the 42nd percentile for both reading and math at the eighth-grade level.
“Controlling for the number of students who drop out of school between 8th and 11th grade,” he said, “the new cut scores for meeting standards on the ISAT are still about 10 percentile points lower than the current cut scores for meeting standards on the PSAE.”
Statistically, this means about 16% of eighth-graders who “meet standards” on the ISATs will likely not “meet standards” on the PSAE.
The RoundTable invited ISBE in a Feb. 21 email to comment on why the new ISAT cut scores were set at the 42nd percentile rather than the 50th percentile and to provide any support for setting them at that level. ISBE did not provide a comment.
In a Jan. 28 Freedom of Information Act Request, the RoundTable asked ISBE to produce all additional studies that were conducted in setting the new ISAT cut scores. The RoundTable also asked for other documents that might explain the rationale for setting the new ISAT cut scores to “meet standards” below the level needed to align them with the PSAE cut scores to “meet standards.” ISBE refused to produce the documents, citing three exemptions under the FOIA.
New ISAT Cut Scores Are Far Below College Readiness Benchmarks
The new ISAT cut scores – 248 for eighth-grade reading and 267 for eighth-grade math – also lag far behind the proficiency level needed to be on track to college and career readiness as measured by the ACT.
The Agarwal/Lange report said the cut scores to “meet standards” on the PSAEs were “closely aligned” with the scores on the PSAE that correspond to ACT’s college readiness benchmarks. Their study found, though, that the PSAE scores that correspond with ACT’s college readiness benchmarks were at the 65th percentile in reading and the 66th percentile in math. In stark contrast, the new cut scores to meet standards on the ISATs are set at the 42nd percentile in both eighth-grade reading and math.
If a student needs to be at the 65th or 66th percentile in eleventh grade to be on track to college readiness, an ISAT cut-score set at the 42nd percentile in eighth grade falls short of the mark.
A chart in the Agarwal/Lange report gives a glimpse of how far off the mark the new ISAT cut scores are. The chart from the Agarwal/ Lange study is reproduced here as Figure 2, with the red and green lines added by the RoundTable.
The chart shows the “relationship between the probability of students being college ready in math on the  PSAE and their prior  ISAT grade 8 math scores.” The chart (Figure 2, see green line) shows that an eighth-grader with an ISAT score of 267 in math (the new cut score for eighth-graders to meet standards in math) has about a 10% percent chance of obtaining a PSAE college readiness score in math in eleventh grade.
A 10% chance of being on track to college readiness is very low. Dr. Lange told the RoundTable in April 2012 that deeming a student to be on track to college readiness in eighth grade if they have only a 20% chance of being college ready in eleventh grade “doesn’t make any sense.” He said he would want the probability to be at least 50%.
The chart (Figure 2, see red line) also shows that, on average, an eighth-grader in 2008 needed to obtain an ISAT score of about 284 in math to have a 50% probability of meeting or exceeding ACT’s college readiness score as an eleventh-grader in 2011. Mr. Zavitkovsky told the RoundTable that a score of 284 was at the 64th percentile on the 2012 ISATs.
Drs. Agarwal and Lange’s report says, “Analogous results were obtained for Reading and Math,” but those results were not displayed in the report. The RoundTable asked ISBE for the results for reading and math, but was told they were not available.
Numerous other studies indicate the new ISAT cut scores are substantially below where they need to be to indicate being on track to college and career readiness. As reported in earlier articles published by the RoundTable, Mr. Zavitkovsky compared multiple years of ISAT and ACT results to estimate the ISAT scale scores that indicate whether students are likely to be on-track for college readiness at grades 3-8 (i.e. have a 50% chance or better of meeting or exceeding college-readiness benchmarks in eleventh grade).
Mr. Zavitkovsky told the RoundTable he found that, on average, students statewide needed to be at or above the 60th Illinois percentile in reading and at or above the 66th Illinois percentile in math, to be on track to ACT college readiness in 11th grade.
By contrast, the new cut scores to “meet standards” on the ISATs for both eighth-grade reading and math correspond to the 42nd percentile.
Mr. Zavitkovsky’s conclusions are supported by at least two other studies that identified ISAT scores that correspond to be being on track to ACT college readiness. Both studies identified ISAT scores that were above the 60th percentile in reading and the 66th percentile in math.See “Illinois Comparability Study Linking EXPLORE to ISAT” (2009), conducted by ACT; and From High School to the Future: Pathway to 20” (2008), conducted by the Consortium on Chicago School Research at the University of Chicago.
The Agarwal/Lange report identified cut scores to “meet standards” on the ISATs that were aligned with the cut scores to “meet standards” on the PSAEs. The ISAT cut scores identified in the Agarwal/Lange report correspond to the 51st percentile in reading and the 52nd percentile in math.
In adopting new cut scores on Jan. 24, 2013, ISBE adopted scores that were lower than the ones identified in the Agarwal/Lange report, and that corresponded to about the 42nd percentile.
In a Feb. 21 email, the RoundTable invited ISBE to comment on why the new ISAT cut scores were set at the 42nd percentile rather than the 50th percentile and to provide any support for setting them at that level. The RoundTable asked ISBE to respond by Feb. 25 so it could incorporate ISBE’s comments in an article, but ISBE did not respond before the above article was posted on Feb. 27.
After the article was posted, Mary Fergus, public information spokeswoman for ISBE, emailed this comment to the RoundTable: “The suggestion of 50th percentile or 42nd percentile made by Agarwal/Lange was never used. We adopted the bridge study method to link grade 11 PSAE and grade 8 ISAT results through cohort students who took both tests. We then compared the results with statewide PSAE average to calculate percentage for each performance level. The equal-percentile method was applied to grade 3 through grade 7 based on the new grade 8 percentage on each performance level.”
ISBE did not summarize the results of the “bridge study method” referred to in the email, and did not provide any support for setting new cut scores to “meet standards” on the ISATs at a level that corresponded to the 42nd percentile.
In a memorandum submitted to ISBE for its Jan. 23-24, 2013 Board meeting (at which the Board decided to raise ISAT cut scores), Christopher A. Koch, Superintendent, and Susan C. Morrison, Deputy Superintendent/Chief Education Officer, said staff used “an ‘equipercentile equating process’ to determine the new performance levels. With this process, a cohort-group analysis was used to backmap performance levels from the Prairie State Achievement Examination (PSAE) to 8th grade ISAT performance. …”
In a Jan. 28, 2013 Freedom of Information Act request, the RoundTable asked ISBE to produce all documents that were prepared as part of or that refer to the “equipercentile equating process,” the “cohort-student group” analysis, or the “backmapping” referred to in Dr. Koch’s memo. The RoundTable also asked ISBE to produce any other studies or reports that attempted to equate ISAT scores with PSAE “meet standards” scores or to equate ISAT scores with college readiness benchmarks of the EXPLORE or ACT tests.
In its response to the FOIA request, ISBE said the Agarwal/Lange report “is responsive to this request,” but it refused to produce any other studies, analyses or documents.
On March 19, 2013, the RoundTable appealed ISBE’s refusal to produce any other documents called for in its Jan. 28 FOIA Request. In that appeal, ISBE acknowledged that it has a report entitled, “Promoting College and Career Readiness for Illinois Students: A Review of Illinois Assessment Performance Categories;” a report entitled, “NCEA ACT 2012;” minutes of meetings of its Technical Advisory Committee at which the new cut scores were discussed; and six excel spreadsheets which it says are responsive. ISBE asserts these documents are exempt under what is sometimes called a “deliberate process” exemption. The RoundTable disputes that exemption applies. Briefing of the issue was complete on May 6. The Public Access Officer of the Attorney General’s Office has not yet decided the appeal.
On May 30, 2014, the Office of the Attorney General’s Office ordered ISBE to produce a report that recommended the new ISAT cut scores that ISBE adopted in January 2013. After numerous requests, ISBE finally produced that report on Sept. 9, 2014. The RoundTable published an article referencing that report, “The ISAT Cut Score Fiasco: Where’s the Accountability?” in its Oct. 9, 2014 print edition. A link to that article is below.
ISAT Cut Scores
A student who takes the ISAT is given a score based on the number of his or her correct responses on the test. In order to “meet standards” on the ISAT, a student’s score must be at or above the “cut score” set by the ISBE to meet standards on the test. The cut score, which can be set at any point along the achievement spectrum, determines the proficiency level needed to meet standards.
ISATs Historically Set at Very Low Levels
Many studies have found that the proficiency levels (e.g., the cut scores) to meet standards on the ISAT were set at a very low level and among the lowest in the nation. See, e.g., “Mapping State Proficiency Standards Onto NAEP Scales: 2005-2009” (2011), conducted by NCES at the U.S. Department of Education (at eighth grade: 41st in reading and 45th in math.)
In “From High School to the Future: ‘The Pathway to 20’” (2008), researchers with the Consortium on Chicago School Research at the University of Chicago found that Chicago eighth-graders who just “meet standards” in math, have less than a 5% chance of meeting ACT benchmarks for college readiness in 11th grade.
Another study, “Something’s Wrong with Illinois Test Results” (2009), conducted by Paul Zavitkovsky of the Urban School Leadership Program at the University of Illinois-Chicago, found that Illinois eighth-graders “met standards” in reading on the 2006 ISATs if they were at the 22nd Illinois percentile. Yet, they need to be at the 60th Illinois percentile to be on track for ACT college readiness in reading.