In 2006 the Illinois State Board of Education (ISBE) attempted to peg the cut scores to “meet standards” on the Illinois Standards Achievement Test (ISAT) at the 38th national percentile. In other words, the intent was that students would have to perform at a level equivalent to or better than 38% of the students in the nation to meet standards on the ISATs. ISBE used the Stanford Achievement Test, tenth edition (SAT-10) in the recalibration process.
Since 2006, however, several national studies have found that the cut scores to meet standards on the ISATs equate to much lower percentiles on national tests other than the SAT-10. These studies indicate that the cut scores to meet standards may be set at much lower levels of achievement than originally contemplated.
NAEP, MAP and State of Illinois Percentiles
The RoundTable recently obtained data showing that the cut scores to meet standards for eighth-grade reading and math on the ISAT coincide with the 20th national percentile of the 2007 National Assessment of Academic Progress (NAEP). The NAEP is given every two years to a representative sample of students in fourth and eighth grades from every state. The test is sponsored by the U.S. Department of Education and is often referred to as the Nation’s Report Card.
In a study conducted by the National Center for Education Statistics (NCES) and the Institute of Education Sciences at the U.S. Department of Education, “Mapping State Proficiency Standards Onto NAEP Scales: 2005-2007” (2009), researchers determined that Illinois’ cut scores to meet standards in reading and math for fourth- and eighth-graders on the 2007 ISATs were equivalent to the following scale scores on the 2007 NAEP test:
• fourth-grade reading – 200 (26th national percentile on NAEP);
• fourth-grade math – 208 (13th national percentile on NAEP);
• eighth-grade reading – 236 (20th national percentile on NAEP); and
• eighth-grade math – 251 (20th national percentile on NAEP).
The national percentile ranks for each of the NAEP scale scores were not published with the original report. But on Nov. 8, following RoundTable requests to the National Center for Education Statistics, the Education Testing Service provided that information.
These national percentile ranks on NAEP are very close to those found in a separate study, “The Proficiency Illusion” (2007), conducted by the Northwest Evaluation Association and the Thomas B. Fordham Institute, using national norms from the Measures of Academic Progress (MAP) test as a yardstick. The MAP is used by District 65 to assess student progress and teacher effectiveness throughout the school year.
National percentile ranks from both the NAEP and MAP studies also match up closely with the “State of Illinois percentile” ranks reported by Paul Zavitkovsky of the Urban School Leadership Program at the University of Illinois-Chicago. Illinois percentiles reflect how students taking the ISATs score in relation to all other Illinois students taking the same test. For example, an eighth-grader who scored at the 20th Illinois percentile in reading on the 2010 ISATs scored at or better than 20% of all other students who took the same test.
The accompanying table, prepared by Mr. Zavitkovsky, summarizes the national percentile ranks from the NAEP and MAP studies as well as the Illinois percentile ranks that coincide with the cut scores to “meet standards” on the ISATs. The table shows that these percentile ranks are all within a close range for the grade and subject indicated. The percentiles are within one or two points for 2006 and 2007, where there are common data points.
Mr. Zavitkovsky told the RoundTable that the close match-ups between NAEP, MAP and State of Illinois percentiles are consistent with his earlier finding that the Illinois test population is highly representative of the national population. The evidence for this finding is that, within standard errors of measure, Illinois percentiles have been more or less interchangeable with national percentiles on the NAEP for most of the past decade, he said.
The accompanying table also shows the median SAT-10 percentiles of students at the cut scores to meet standards on the 2009-10 ISATs. The SAT-10 median percentiles were determined by Mr. Zavitkovsky using data provided for the first time for the 2009 ISATs by ISBE. Median SAT-10 percentiles are provided because the students who score at a given ISAT cut score have a range of SAT-10 percentiles.
In stark contrast to the close correspondence of the NAEP, MAP and Illinois percentiles, the SAT-10 generates much higher percentiles – 16 to 28 points higher – at the cut score boundaries.
According to ISBE, SAT-10 percentiles are based on an abbreviated version of the SAT-10 that is incorporated into the ISATs. A student’s score on the abbreviated version is extrapolated to a full-length score, and a percentile rank is then obtained from a 2002 table prepared from a national norm sample.
When asked by the RoundTable, the Illinois State Board of Education (ISBE) could not explain why the SAT-10 portion of the ISATs was generating much higher results than NAEP and MAP. ISBE was also unable to respond to other basic questions about that portion of the ISAT. ISBE said it has no studies analyzing the reliability of the national percentile scores generated by the SAT-10 portion of the ISAT. See Sept. 15 issue of the RoundTable.
Pearson Education, Inc. is the owner of the SAT-10 exam. The RoundTable has asked Pearson to explain why the abbreviated version of the SAT-10 produces national percentile ranks that are much higher than NAEP and MAP. To date, Pearson has not responded to Roundtable requests.
District 65 Administrators Cling to ‘Meet Standards’ Benchmarks on ISATs as Measure of ‘Proficiency’School District 65 posted “”Student Achievement Highlights”” on its website in December. In its highlights the District presents charts showing “”8th Grade Cohort Proficiency”” and “”8th Grade Proficiency”” over time. Two of the charts say the data provided for the period 2001-10 demonstrate the improvement in eighth-grade reading and math “”proficiency”” and illustrate the reduction in the reading and math “”proficiency gap”” between ethnic groups.
The District’s highlights say “”proficiency”” is measured by the percentage of students meeting or exceeding standards on the ISATs.
District 65 administrators thus cling to and endorse a very low benchmark for measuring “”proficiency,”” one that sets very low expectations for students, and one that substantially masks the extent of the gap in achievement between ethnic groups.
The accompanying article presents data showing that several national studies peg the cut scores for eighth-graders to meet standards on the ISAT at the 20th national percentile, using NAEP and MAP as the measures. These national studies also found that the cut scores to meet standards on the ISATs are among the lowest in the nation.
Another study, “”Pathway to 20″” (2008), conducted by the Consortium on Chicago School Research at the University of Chicago, found that eighth-graders in the Chicago Public School System who just “”meet standards”” on the ISATs have less than a 5% chance of meeting ACT college readiness benchmarks in eleventh grade.
More recently, Paul Zavitkovsky, of the Urban School Leadership Program at the University of Illinois-Chicago, found that eighth-graders could meet standards on the 2010 ISATs in reading and math by scoring at the 16th Illinois percentile. He has reported that the four-year statewide average to be on track to meet ACT college readiness standards, though, is the 60th Illinois percentile for reading and the 66th for math.The ISAT Meet Standards is at the 22nd National Percentile Using ITBSUPDATE NOV. 16, 2011. In a report, “”Trends in Chicago’s Schools Across Three Areas of Reform,”” (September 2011) published by the Consortium on Chicago School Research, the authors found that, “”The ISAT ‘meets standards’ point lines up with approximately the 22nd national percentile for the ITBS [Iowa Test of Basic Skills].”” See Figure 15 and accompanying text in the report.This is consistent with results using NAEP and MAP as the yardsticks.