At the School Board’s Oct. 3 meeting, Paul Brinson, chief information officer of School District 65, said, “I didn’t see anything in the RoundTable studies that leads me to believe that we haven’t presented that same kind of information to you over time. There’s no startling revelations in that. You can talk about the degree of difference…and whether or not the degree of difference is there. But the difference is real.”
Paul Zavitkovsky’s study gives a much different and much more nuanced picture of student achievement at District 65 than the picture presented by District 65 using the “meet standards” reporting method of the Illinois Standard Achievement Test (ISAT). Mr. Zavitkovsky, of the Urban Education Leadership Program at the University of Illinois-Chicago, reported the results of District 65 students on the ISATs for the period 2001-09 using State of Illinois norms, such as the percent of students performing at the 50th Illinois percentile, Illinois percentile ranks, and the percent of students scoring in each Illinois quartile and in each Illinois stanine. He reported similar data for ETHS students on the Prairie State Achievement Exam (PSAE) for the period 2004-09.
While his analysis shows progress has been made in student achievement at District 65, particularly for African American students, his study also illustrates that the ISAT “meet standards” reporting method of the ISATs gives a misleading picture of student achievement and growth over time.
He found that the achievement gap at District 65 is much larger than portrayed by the “meet standards” measure of the ISATs, that about 70% of African American eighth-graders are not on track for college readiness when they graduate from District 65, and that the difference in the percent of students meeting standards on the ISATs versus the PSAEs is due in large part to a misalignment of the two tests.
The ISATs Use a Low Benchmark
To Measure Whether Students Meet
Mr. Zavitkovsky found that the benchmark to meet standards on the ISATs is set very low. Eighth-graders who scored at the 17th Illinois percentile in reading (at or better than just 17 percent of students who took the same test) “met standards” on the 2009 ISATs. In order to be on track for college readiness in reading, though, a student needed to be at the 60th Illinois percentile (at or better than 60% of the students taking the same test).
The benchmarks to “meet standards” on the ISATs are grossly misaligned with the proficiencies needed to be on track for college and career readiness.
There is a Wide Achievement Gap
When Expectations Increase
In October 2009, District 65 administrators reported that the District had made “significant” progress toward closing the achievement gap. They reported that on the 2009 ISATs, 82% of African American eighth-graders in the District “met standards” in reading, compared to 99% white students, a gap of 17 points. On math, 85% African American eighth-graders “met standards,” compared to 99% white students, a gap of 14 points.
Working with exactly the same test data, however, Mr. Zavitkovsky found that the gap is much larger when the standard is the percent of students exceeding the 50th Illinois percentile – a percentile rank often used as an indicator of performing at “grade level,” and one which has higher expectations than the “meet standards” proficiency level of the ISATs.
On the 2009 ISATs:
• 46% of African American eighth-graders in the District performed at or above the 50th Illinois percentile in reading, compared to 91% white students, a gap of 45 points.
• 50% of African American eighth-graders in the District performed at or above the 50th Illinois percentile in math, compared to 94% white students, a gap of 44 points.
While African American eighth-graders, as a group, have made progress and are approaching or are at the statewide average, there is still a substantial achievement gap when compared to the high level of achievement of white students. The gap is larger when viewed from the perspective of how many eighth-graders are on track for college readiness.
Almost 70% of D65 African
American Eighth-Graders Are
Not on Track for College Readiness
Based on four years of state-wide test data comparisons, Mr. Zavitkovsky’s reports provided estimated benchmarks to assess whether students were on-track to college readiness at the third, fifth and eighth grades. These benchmarks are the bottom edge of the sixth Illinois stanine for reading (the 60th Illinois percentile) and the mid-point of the sixth Illinois stanine in math (the 68th Illinois percentile).
He also presented numerous charts and data tables for the period 2001-09 showing the percent of white and African students who scored in each of the nine stanines at third, fifth and eighth grades for the period 2001-09.
Using these benchmarks, the data shows that on the 2009 ISATs:
• 32% of African American eighth-graders were on track for college readiness in reading, and 29% in math.
• 86% of white eighth-graders were on track for college readiness in reading and 84% in math.
Using Mr. Zavitkovsky’s estimates, about 70% of the African American students who graduate from District 65 are not on track for college readiness. The District has not reported this type of data before.
While District 65 began to report its eighth-graders’ scores on the EXPLORE test last year, it did not report the percent of white or African American students who were on-track to college readiness based on benchmarks identified for EXPLORE. Moreover, District 65 administrators argued that the EXPLORE test is not aligned with the Illinois learning standards and that it was not a valid tool to use to assess District 65 students.
Yet, the State of Illinois recently said it would use eighth-grade test results on EXPLORE as a primary indicator of school effectiveness toward meeting the goals of the Common Core State Standards.
The ISATs and the PSAEs
Starting in 2007, School District 65 administrators publicly said that District 65 was sending high performing eighth-graders to ETHS and that the gains made at District 65 were being eroded at ETHS. For example, 72% of African American eighth-graders met standards in reading on the 2006 ISATs; three years later only 36% met standards in reading as eleventh-graders on the 2009 PSAEs.
Mr. Zavitkovsky’s study illustrated that this drop is due in large part to a “gross misalignment” of the ISATs and the PSAEs. Simply put, there is a lower benchmark to meet standards on the ISATs than there is on the PSAEs. Mr. Zavitkovsky presented numerous charts and data tables to elaborate on this point for six cohorts of students.
The SAT-10 and National Percentiles
Mr. Brinson has said that the cut scores to “meet standards” on the ISATs were set at the 38th national percentile. In 2006, the State used the Stanford 10 (SAT-10) in an attempt to set the cut scores to “meet standards” at that percentile. Several national studies, though, including one conducted by Center for Education Statistics and the Institute of Education Sciences at the U.S. Department of Education, found that the cut scores were set below the 25th national percentile.
Historically, District 65 has reported “national percentiles” using the results generated by an abbreviated version of the SAT-10 that is embedded in the ISAT. A student’s score on the abbreviated version is extrapolated to a score on the full length version of the test and a national percentile rank is then obtained from a 2002 norms table, which was developed through a sample.
An important finding from Mr. Zavitkovsky’s analysis is that the percentiles reported using the SAT-10 are typically 15 to 25 points higher than Illinois percentiles and 15 to 25 points higher than comparable national percentiles produced by the National Assessment of Education Progress test (NAEP, the “Nation’s Report Card), the EXPLORE test, and the Measures of Academic Progress test (MAP). Mr. Zavitkovsky says, “When there’s a conflict, I think there’s good reason to privilege the NAEP.”
When asked by the RoundTable, the Illinois State Board of Education (ISBE) could not explain why the SAT-10 portion of the ISATs was generating much higher results than NAEP and could not respond to other basic questions about that portion of the test. ISBE said it has no studies analyzing the reliability of the national percentile scores generated by the SAT-10 portion of the ISAT. See Sept. 15 issue of the RoundTable.