The Illinois State Board of Education’s history in setting cut scores to “meet standards” on the Illinois Standard Achievement Test (ISAT) is a disturbing one. For many years the cut scores (i.e., the proficiency levels) to “meet standards” on the ISATs were among the lowest in the nation. The cut scores were set so low that students who were at serious risk of academic failure could still “meet standards.”
On Jan. 24, 2013, ISBE decided to raise the cut scores to “meet standards” on the ISATs as part of an effort to convince the U.S. Department of Education to grant Illinois a waiver from certain provisions of the No Child Left Behind Act. ISBE represented to the U.S. Department of Education in its application for the waiver that it would raise the cut scores so they would align with college and career ready standards.
In a prepared statement announcing that ISBE raised the cut scores, State Superintendent of Education Christopher A. Koch said, “The lower expectations of the previous performance levels did our students a disservice by not adequately assessing their ability to succeed after high school.” He also said the new higher cut scores will provide a “more accurate” indicator of whether a student was on track to college and career readiness.
Subsequently, ISBE boldly represented to the public in report cards that the new ISAT cut scores “align with college and career ready expectations.”
It appears, though, that the new cut scores still fall far short of the mark. The impact may not be long-lived because the State is shifting to a new assessment system, PARCC, this year. But it does raise questions about ISBE’s responsibility in setting and reporting test data, and whether there is any accountability.
A report prepared by ISBE titled “Promoting College and Career Readiness for Illinois Students: A Review of Illinois Assessment Performance Categories” (the Report), recommended the new ISAT cut scores that were adopted by ISBE in January 2013.
ISBE produced that report to the RoundTable on Sept. 9, more than one-and-one half years after the RoundTable requested it under the Freedom of Information Act, and more than three months after the Public Access Counselor of the Attorney General’s Office (AG) ordered ISBE to produce it.
The new cut scores fall short of aligning with ACT’s college readiness benchmarks for two reasons.
First, the Report makes clear that no attempt was made to align the new ISAT cut scores with the college readiness benchmarks identified by the ACT – which ISBE has often said is the “gold standard” for college readiness. Instead, the Report says the new ISAT cut scores are aligned with the proficiency level needed to “meet standards” on the Prairie State Achievement Exam (PSAE). That proficiency level is below the level needed to demonstrate college readiness.
Second, while the Report says the new cut scores to “meet standards” on the ISATs align with the cut scores to “meet standards” on the PSAEs, ISBE acknowledges that it has no study or analysis in its possession or control that aligns the new ISAT cut scores to the PSAE “meet standard” scores. Available data indicates, though, that the new ISAT cut scores are below even those needed to align to the PSAE cut scores to “meet standards.”
The combined impact of these shortcomings is significant. Data shows that an 8th-grader who scores right at the new “meet standards” boundary on the ISATs has a 10% chance or less of obtaining an ACT college ready score in 11th grade.
B. Disconnect No. 1: Aligning With “Meet Standards” on the PSAE, Not ACT College Readiness
ISBE could have aligned the new 8th-grade ISAT scores to ACT’s college readiness benchmarks but it chose not to do so. Instead, it decided to link them to a lower level of proficiency.
The Report says that an earlier report, “College & Career Readiness for Illinois State Assessments” prepared by Pooja K. Agarwal, Ph.D. and Rense Lange, Ph.D. (the “Agarwal/Lange Report), was prepared to examine the relationship between the ACT, the PSAE and the ISAT. The Agarwal/Lange Report identified PSAE scores that equated with the ACT’s college readiness benchmarks by using an equipercentile equating approach. Those scores are each higher than the PSAE “meet standards” scores.
The Agarwal/Lange Report concluded, though, that the PSAE scores to “meet standards” on the PSAEs were close to the PSAE scores that equated with ACT’s college readiness benchmarks and that it was “reasonable to operationalize college readiness in terms of existing PSAE ‘Meets’ scores.”
Based on that assertion alone, the Report purports to link eighth-grade ISAT scores to the “meet standard” scores on the PSAE, rather than to the ACT’s college readiness benchmarks.
The table below reflects 1) the cut scores to “meet standards” in reading and math on the PSAE, and 2) the PSAE scores that the Agarwal/Lange Report determined equate to the ACT’s college readiness benchmarks. The percentile rank of each score is in parentheses.
What appears to be relatively small numerical differences in scale scores actually represent significant differences in statistical value. The cut scores to “meet standards” on the PSAEs are positioned at the 49th percentile, while the PSAE scores that equate with the college readiness benchmarks of the ACT are at about the 65th percentile.
The RoundTable asked Paul Zavitkovsky of the Center for Urban Education Leadership at the University of Illinois-Chicago to comment about these differences. Mr. Zavitkovsky said, “It’s not really possible to judge how closely aligned scale scores are until you convert those scores into percentile ranks. Once you do that here, the differences in percentile values for reading and math are substantial.”
He added, “Based on the data provided in the report it’s hard to see how the numbers justify using the meet standards cut score on the PSAE as a surrogate for college readiness.”
One way to see the practical impact of using the cut scores to “meet standards” on the PSAEs rather than the scores that equate with the ACT’s college ready benchmarks is to compare the difference between 1) the percentages of Illinois eleventh-graders who met standards on the PSAE with 2) the percentage who met the college readiness benchmarks on the ACT. This information is available because the ACT is embedded in the PSAE, and ISBE reported these percentages on its website.
ISBE reported that for 2011 (the base year in the Report):
• 51% of all 11th graders tested met standards in reading on the PSAE, but only 40% met the ACT’s college readiness benchmark in reading.
• In math, 51% met standards on the PSAE, while only 39% met the ACT’s college resdiness benchmark.
Another way to look at it is to identify the students who obtained a “meet standards” score on the PSAEs, and determine the percentage of those students who obtained an ACT score that met or exceeded the college ready benchmarks of the ACT.
At the RoundTable’s request, Mr. Zavitkovsky conducted this analysis using student-level data files that are released annually by ISBE. He determined for 2011 (the base year in the Report):
• Only 11% of the students who had a reading score of 155 on the PSAE (the “meet standards” score) had a reading score of 21 or higher on the ACT (the ACT college readiness benchmark in 2011).
• Only 3% of the students who had a math score of 156 on the PSAE (the “meet standards” score) had a readiness score of 22 or higher on the ACT (the ACT college readiness benchmark).
ISBE thus decided to align the new ISAT cut scores to a PSAE score where students have only a 3% chance or an 11% chance of being college ready in eleventh grade.
In separate interviews in April 2012, the RoundTable explored why Drs. Agarwal and Lange thought it was proper to use the “meet standards” benchmark on the PSAEs as a surrogate for college readiness, rather than the college readiness benchmarks identified by the ACT. Dr. Agarwal told the RoundTable she “personally disagreed” with the way ACT set its benchmarks, but the report itself does not point out any deficiencies in the way the ACT set its benchmarks for reading or math.
Moreover, an August 2014 ISBE memo under the names of ISBE’s Board President and ISBE’s Superintendent says the ACT “is the gold standard for college and career readiness.” Also, ISBE touts ACT’s college readiness benchmarks in its online report cards.
Dr. Lange said a group of experts selected the cut scores to meet standards on the PSAEs in the early 2000s, but he acknowledged they were not trying to align the cut scores with college readiness at that time. When asked, “If you wanted to determine an ISAT score that best corresponded to being on track to meeting the ACT’s college readiness benchmarks, should you backmap from the PSAE college readiness score, as opposed to the PSAE meet standards score?” He replied, “Yes, if you wanted to identify the score that best corresponded.”
C. Disconnect No. 2: The New ISAT Cut Scores Are Too Low to Align Even With PSAE “Meet Standards”
In adopting the new cut scores to “meet standards” on the ISATs in January 2013, ISBE adopted the scores recommended in the Report. These scores were each lower than the scores the Agarwal/Lange Report identified as aligning with the PSAE “meet standards” scores. The table below compares the ISAT scores identified in Agarwal/Lange Report and the new scores adopted by ISBE for eighth-graders.
While these scores are numerically close, the Agarwal/Lange scores are positioned at about the 51st percentile in reading and the 53rd in math. The scores that were adopted by ISBE are positioned at about the 41st percentile.
1. The Lack of Any Underlying Analyses
ISBE has provided only a vague, general statement on how it aligned the new ISAT cut scores to “meet standards for 8th-graders to the PSAE cut scores “meet standards” scores for 11th graders. In a memo to the ISBE Board in January 2013, Dr. Koch said:
“Student Assessment Division staff utilized an ‘equipercentile equating process’ to determine the new performance levels. With this process, a cohort-student group analysis was used to backmap performance levels from the Prairie State Achievement Examination (PSAE) to the 8th grade ISAT performance.”
The RoundTable asked ISBE to produce the studies, analyses and other records that were prepared as part of the equipercentile equating process, the cohort student group analysis, and the backmapping process, in a request made under the Freedom of Information Act.
ISBE refused to produce these records, claiming they were exempt. The RoundTable appealed to the AG. Fourteen months later, the AG ordered ISBE to produce the Report and six other documents. After numerous demands, ISBE finally produced the records, but none of the records contains the underlying analyses that equated the 8th-grade ISAT scores and 11th-grade PSAE scores.
The Report is the only document that even touches on the methodology, and it says in vague, general terms:
“A final analysis of the ISAT scores from 2008 through 2011 was conducted using the equipercentile equating process. A backmapping process was used with assessment results from the 2008 to 2011 PSAE to ISAT grade 8 data by using a cohort-student group and an all-student group. The percent of students at each performance level in PSAE was calculated.”
This sheds no more light on how it was done than Dr. Koch’s memo to the ISBE Board. The RoundTable followed up and asked ISBE to produce the underlying analyses, specifically the records that showed the data used, the cohorts and cohort sizes used, the formula used, and the calculations made.
Matt Vanover, ISBE’s director of public information, told the RoundTable in an email, “We have provided you with all of the responsive documents and spreadsheets that make up the equipercentile equating process, the cohort-student group analysis and the backmapping process. .. [W]e have not withheld any spreadsheets or analyses regarding your request.”
But, ISBE did not produce a single record, paper or digital, that reflects the underlying analyses that equated 8th-grade ISAT scores and 11th-grade PSAE scores. It appears there are none.
The RoundTable also asked Mr. Vanover to make the persons who prepared the Report and who performed the underlying analyses available for an interview. ISBE refused.
The RoundTable then asked ISBE in written questions to lay out in detail how the analyses were performed, including the data used, the cohorts and cohort sizes used, the formula used, the assumptions made, and the calculations made.
Again, ISBE provided only a vague, general response – too vague to allow an independent expert to assess what ISBE did.
2. An Independent Analysis
The RoundTable asked Mr. Zavitkovsky to independently determine 8th-grade ISAT scores in reading and math that equated to PSAE “meet standards” scores. He performed the analysis for four cohorts of students: 2009 ISATS/2012 PSAEs, 2008 ISATs/2011PSAEs, 2007 ISATs/2010 PSAEs, and 2006 ISATs/2009 PSAEs.
Mr. Zavitkovsky concluded, after adjusting for students who dropped out between 8th and 11th grades, that an 8th-grader would need to score at the 53rd percentile in both reading and math on the 2008 ISATs to meet standards on the 2011 PSAEs.
His findings are in line with those of the Agarwal/Lange Report, which concluded that 8th-grade ISAT scores at the 51st percentile in reading and the 53rd percentile in math equated to “meet standards” scores on the PSAE.
Yet, the new cut scores adopted by ISBE correspond to the 41st percentile.
“Controlling for the number of students who drop out of school between 8th and 11th grade,” Mr. Zavitkovsky said, “the new cut scores for meeting standards on the ISAT are still about 10 percentile points lower than the current cut scores for meeting standards on the PSAE.”
D. The Combined Impact of the Two Shortcomings is Significant
The new ISAT cut scores are positioned at about the 40th percentile. A wealth of data, though, shows that the scores need to be positioned at the 60th percentile or higher to align with college and career readiness.
As reported in earlier articles published by the RoundTable, Mr. Zavitkovsky compared multiple years of ISAT and ACT results to estimate the ISAT scale scores that indicate whether students are likely to be on track for college readiness at grades 3-8 (i.e. have a 50% chance or better of meeting or exceeding college-readiness benchmarks in eleventh grade).
Mr. Zavitkovsky found that, on average, students statewide needed to be at or above the 60th Illinois percentile in reading and at or above the 66th Illinois percentile in math, to be on track to ACT college readiness in 11th grade.
At least two other studies have identified ISAT scores that correspond to ACT’s college readiness benchmarks. Both identified ISAT scores that were above the 60th percentile in reading and the 66th percentile in math. See “Illinois Comparability Study Linking EXPLORE to ISAT” (2009), conducted by ACT; and From High School to the Future: Pathway to 20″ (2008), conducted by the Consortium on Chicago School Research at the University of Chicago.
The Northwest Evaluation Association, the owner of the MAP test, has identified RIT scores in reading and math that align with ACT’s college readiness benchmarks. Those scores are slightly above the 70th percentile. See “A Study of the Alignment of the RIT Scales of NWEA’s MAP Assessments with the College Readiness Benchmarks of EXPLORE, PLAN, and ACT” (2011).
The Agarwal/Lange Report found that the eleventh grade PSAE scores that equate to ACT’s college readiness benchmarks are positioned at the 65th percentile in reading and the 66th in math.
On the ACT portion of the 2012 PSAEs, 40% of the 11th graders taking the test met ACT’s benchmarks in reading and 39% in math, meaning that they correspond to the 60th and 61st percentiles.
In practical terms, when the new ISAT cut scores are used, about 60% of Illinois 8th-graders are viewed as being on track to college and career readiness. If the scores were positioned at or above the 60th percentile, 40% of the students or less would be viewed as being on track. The lower cut scores give a puffed-up message to students and the public and make things seem better than they are.Another way of looking at it is this: ISBE purported to set the new 8th-grade ISAT cut scores at a proficiency level that equated to an 11th- grade benchmark at which students have only a 3% chance of achieving an ACT college ready score in math at 11th grade and an 11% chance in reading. Because it appears that ISBE fell short in linking the 8th-grade ISAT scores to even that low benchmark, an 8th-grader who scores right at the new “meet standards” boundary of the ISATs has even lower odds of being college ready in 11th grade.
A chart in one of ISBE’s own reports, the Agarwal/Lange Report, demonstrates that the new ISAT cut scores are way off the mark. The chart is reproduced above, with the red and green lines added by the RoundTable.
The report shows the “relationship between the probability of students being college ready in math on the  PSAE and their prior  ISAT grade 8 math scores.” The chart (see green line) shows that an eighth-grader with an ISAT score of 267 in math (the new cut score for 8th-graders to meet standards in math) has about a 10% percent chance of obtaining a PSAE college readiness score in math in 11th grade.
A 10% chance of being on track to college readiness is very low. Dr. Lange told the RoundTable in April 2012 that he would want the probability to be at least 50%.
Significantly, the chart (see red line) also shows that an 8th-grader in 2008 needed to obtain an ISAT score of about 284 in math to have a 50% probability of meeting or exceeding ACT’s college readiness score as an eleventh-grader in 2011. A score of 284 was at the 64th percentile on the 2012 ISATs.
ACT’s College Readiness Benchmarks
The ACT benchmarks represent the level of achievement required for students to have a 50% likelihood of doing B-level work or better in college freshman courses and a 75% probability of doing C-level work or better. The benchmarks are based on a sample of 214 institutions and more than 230,000 students from across the United States. The ACT has established benchmark scores on the ACT test for reading, math, science and English. A composite benchmark score is the average of the four subject matter benchmarks.
ISBE endorses the ACT’s college readiness benchmarks in the online report cards posted on its website. It says, “”While the composite ACT score is a simple way to gauge college readiness, ACT has determined subject-level benchmarks that more precisely measure college readiness in each individual subject.”” The college readiness benchmark score for both reading and math is 22.New ISAT Scores Reflect ‘Basic’ Skills Per NAEP
ISBE’s Report contains a table of ISAT scores for four achievement levels that line up with the achievement levels that a student needs to demonstrate “”below basic,”” “”basic,”” “”proficient”” or “”advanced”” skills on the National Assessment of Education Progress (NAEP), which is sponsored by the U.S. Department of Education.
The new ISAT cut scores adopted by ISBE in January 2013fall in the middle of the range of scores that comprise the achievement level designated by NAEP as “”basic.”” The new scores are below the level needed to be “”proficient”” or “”advanced.””
The scores to be “”proficient”” in reading and math on NAEP correspond to the 64th percentile.