At the District 65 School Board’s Oct. 20 meeting, administrators presented a 70-page report on student achievement. While administrators presented results on the 2009 Illinois Standard Achievement Test (ISAT) just two months ago (reported in the Aug. 19 issue of the RoundTable), School Board members asked the administrators to provide them with students’ results on the Measures of Academic Progress (MAP) test and the EXPLORE test.
The District has administered the MAP test to District 65 middle school students since 2007; District 202 has administered the EXPLORE test to District 65 eighth-graders for more than ten years. While District 65 eighth-graders showed good progress on the 2009 MAP and EXPLORE tests, the results for MAP in 2008 were down from 2007 and the results on EXPLORE have been essentially flat over the prior seven years. This is in contrast to the dramatic gains made over time by the District’s students on the ISATs.
Superintendent Hardy Murphy said, “We have made progress and it’s ongoing. …Each year we are graduating more students from District 65 prepared for success at the high school. …That doesn’t mean that the work is done. …All we are trying is to get people to acknowledge is that progress is occurring.”
After presenting the data, Paul Brinson, chief information officer, said, “I don’t see how anyone can come to a conclusion other than our students are doing well, they’re progressing. …We can see that all the way into the eighth grade and into the high school.”
Board members were laudatory of the report and the progress being made by students at the District. Board president Keith Terry said that as a public school district, District 65 educates students from all walks of life. “We have seen an increase in performance across every group that comes into District 65. …I celebrate what we’ve achieved in this District.” He added, “Yet we’re not perfect,” and said the Board would continue to challenge the administration to improve student achievement.
Tracy Quattrocki said, “I think there’s a lot to celebrate,” but she asked if the state’s standards were too low. She also asked if the District should be concerned about African-American students stagnant progress on the EXPLORE tests and about Illinois students poor showing on the 2009 National Assessment of Educational Progress (NAEP) test that was released two weeks ago.
In addition to presenting the achievement report, the administrators released the District’s report card, prepared by the Illinois State Board of Education, which contains results for both District 65 students and Illinois students for comparative purposes. At the RoundTable’s request, the District provided additional information reflecting average scale scores of students on the ISAT and MAP tests.
Progress on the ISAT
On the 2009 ISATs, 86.8% of District 65 students met or exceeded standards on the 2009 ISATs, up slightly from 86.5% in the previous year. On a statewide basis, 79.1% of the Illinois students met standards on the ISATs, compared to 79.8% in the previous year.
The percentage of District 65 students who met or exceeded standards on the 2009 SATs increased slightly in reading at the third- and fifth-grades, but declined slightly in all other subjects at the third- though seventh-grade levels. The biggest gain was at the eighth-grade level, where 90% of the students met standards in reading, compared to 85% in the previous year, and 93% met standards in math compared to 85% last year.
On a long-term basis, the District has shown substantial gains on the ISATs in both reading and math. On a composite basis, 67% of the students met standards in reading and math on the 2003 ISATs, that percentage grew to 88% on the 2009 ISATs, a 21-point increase. On a statewide basis, 61% of Illinois students met or exceeded standards on the 2003 ISATs, that percentage grew to 76% on the 2009 ISATs, a 15-point increase.
Black students have shown even greater gains over time. On a composite basis, 47 % of the District’s black students met or exceeded standards on the 2003 ISATs; that percentage increased to 78% on the 2009 ISATs, a 31-point gain. On a statewide basis, 38% of Illinois black students met or exceeded standards on the 2003 ISATs; that percentage increased to 66 % in 2009, a 28-point gain.
The District has also increased the number of students who exceed state standards. Mr. Brinson presented data showing that the percentage of students exceeding standards in reading has increased every year since 2006, from 32% in 2006 to 40% in 2009. In math the percent of students exceeding standards increased from 44% in 2006 to 48% in 2009.
The District’s achievement report contained a scattergram chart printed from ISBE’s website showing that District 65 was at the top of about 20 schools having a similar demographic profile as the District.
Debate About the ISATs
Whether the dramatic increase in improvement on the ISATs reflects real academic progress or is attributable to lowering the bar – or some of both – is a topic of debate locally, statewide and nationally.
One of many charts presented by District administrators reflects the year-by-year percentage growth in the ISAT scores between 2003 and 2009. Almost one-half of the gain during that period occurred on the 2006 ISATs, the year in which ISBE changed the tests.
In 2006, the ISBE established a continuous vertical scale for the ISATs, and as part of that process changed the cut scores (essentially the passing grades) for a student to make it into the “meets” and “exceeds” performance categories. ISBE intentionally lowered the bar for eighth-graders to meet standards in math. A “Bridge Study” conducted by ISBE concluded that that change would increase the number of eighth-graders meeting standards in math by 26 percent; the study concluded that the new cut scores for all other grade levels and subjects were essentially equivalent to the old cut scores when it came to measuring student performance.
One research report, however, “The Proficiency Illusion,” conducted by researchers with the Northwest Evaluation Assessment (which oversees the MAP test) concluded the changes to 2006 ISATs had a more sweeping impact. The study concluded that the changes to the ISATs would yield increased proficiency rates of 17%, 3%, and 14% in third, fifth and eighth-grade reading, and 2%, 5%, and 27% in third, fifth and eighth-grade math.
Since then, other reports, “From High School to the Future: The Pathway to 20,” and “We Can Do Better, Advancing Education in Illinois,” concluded that the ISATs set a very low bar. Arne Duncan, Secretary of the U.S. Department of Education, said in a recent interview with reporters from Bloomberg News that the ISATs and other state tests have been “dumbed down” to make compliance with the annual yearly progress (AYP) goals of No Child Left Behind easier to achieve.
Two weeks ago, NAEP issued the Nation’s Report Card for mathematics, which reported that only 33% of Illinois eighth-graders were “proficient” in mathematics on the 2009 NAEP. By contrast, 82% of Illinois eighth-graders met standards in math on the 2009 ISATs. The “proficient” level has been set as the goal for student performance by the National Assessment Governing Board.
Ms. Quattrocki asked whether it was a concern that Illinois students did so poorly on the 2009 NAEP test. She said, “Is it unsettling at all to read those reports that Illinois is doing so poorly on those tests? Is that at all a concern to us? Should it be a concern in this District?”
She also said, “People think that the state standards are too low. …Are we concerned that our standard for meeting standards in the District, that that bar, is too low, as evidenced by performance on the national test?”
Mr. Brinson said the NAEP test assessed students using different learning standards than the ISAT tests.
Dr Murphy said, “The real answer to the question is we don’t doubt that there’s significant improvement that could occur in the District. … Every year we’re trying to look across the waterfront and see where can we do better. … We can do better for every student in the District.”
He added, “All we want anybody to say or do is recognized we made some progress.”
Board member Kim Weaver said, “Like Dr. Murphy said, we bought into the ISAT. We’re using other tests as mechanisms to make sure we have continuous improvement. …“I don’t understand how it’s going to help us as a Board and as a school district,” to look at a lot of other tests.
Keith Terry said, “The crux of it is that there’s a cohort of kids that are underperforming badly at the high school. And everyone’s talking about, ‘It’s not me, it’s not me, it’s District 65.’ … They’re sending us bad kids.’ … When we look at it, this problem does not go away for us until the problem goes away for ETHS. … I say it because to some degree perhaps they are right. … We need to do a better job of teaching and sending them better-educated children, but I think over time we’ve shown that we’ve tried to move it along.
“But, to Ms. Quattrocki’s point of looking at it from different angles, public education is complex, and, unfortunately, when [students] leave us, we can’t teach them any more,” Mr. Terry continued. “Perhaps that could be a reason for looking at this from different perspectives.”
While there is an ongoing debate about whether the bar for meeting standards on the ISATs has been lowered and is set too low, the table on page 24 shows that District 65 students have made good gains in years in addition to 2006 (the year in which ISBE changed the test) and have improved at a rate greater than Illinois students generally.
Students in the Top 50th Percentile: ISAT and MAP
School District 65 officials say the gains on the ISATs are corroborated by the number of students performing in the top 50th percentile nationally. On the ISATs, a subset of 30 questions taken from the SAT10 are included in the test, and the answers to these 30 questions are extended out and compared to a national sample, said Mr. Brinson.
The District’s Achievement Report reflects that high percentages of District 65 students are performing in the top 50th percentile nationally as measured by the sampling of questions on the ISAT SAT 10. Performing at the 50th percentile is often associated with performing “at grade level.”
On Sept. 9, the School Board decided to consider results on the MAP test as an additional measure to assess student achievement. The MAP test is a computerized adaptive test that the District has given to middle school students since 2007. The test may be aligned to an individual state’s curriculum standards.
A comparison of the results on the ISAT SAT 10 and the MAP test, show that results on the ISAT SAT 10 are substantially higher than the MAP test results. A third data point, the EXPLORE test, falls between the two.
The percent of black and Hispanic eighth-graders who performed above the 50th percentile nationally in reading and math on the 2009 ISAT SAT10, the spring 2009 MAP, and Dec. 2008 EXPLORE tests are as follows:
• black eighth-graders, reading: 69% (ISAT), 52% (MAP), 63% (EXPLORE);
• black eighth-graders, math: 86% (ISAT); 53% (MAP); 60% (EXPLORE);
• Hispanic eighth-graders, reading: 65% (ISAT); 45% (MAP); 54% (EXPLORE);
• Hispanic eighth-graders, math: 86% (ISAT); 56% (MAP); 65% (EXPLORE). ]
Another measure which may be used to compare how District 65 students are doing with other students is to compare their average scale score on the ISATs with the average scale score of all Illinois students on the ISATs.
On the reading portion of the 2009 ISATs, the average scale scores for the District’s black and Hispanic students were 246 and 248 respectively, each slightly below the state average of 249. In math, the District’s black and Hispanic eighth-graders average scale scores were 271 and 276 respectively; the state average was 272.
The EXPLORE Test
While District 65 students have shown great strides in meeting standards on the ISATs since 2003, they have shown little improvement on the EXPLORE test.
School District 202 gives the EXPLORE test to District 65 eighth-graders each year as one of several tests and methods used to assess incoming freshmen. EXPLORE is part of the ACT family of tests, and it is designed to be given to eighth- or ninth-graders. District 65 eighth-graders’ composite scores by subgroup on the EXPLORE test for the last eight years have generally been flat.
The numbers below are the composite scores of the subgroup of students indicated for the school years 2001-02 though 2008-09:
• Black: 14.1, 13.3, 13.6, 14.0, 14.2, 14.1, 14.2, 14.5
• Latino: 14.5,14.7, 14.9, 14.2, 14.9, 14.8, 14.4, 14.8
• White: 19.5, 19.5, 19.4, 19.3, 19.4, 19.5, 19.4, 19.7
According to EXPLORE and ACT technical manuals, the benchmark composite score for college readiness on the EXPLORE test is 16.25. Students who score a 16.25 on EXPLORE have a 50% chance of achieving a composite benchmark score of 21.25 on the ACT, which is the minimum score required for students to have a high probability of success in college, say the manuals.
The average composite scores of District 65’s black and Hispanic students have consistently fallen short of the 16.25 benchmark for college readiness.
Board member Tracy Quattrocki asked, “When we see that the African-American EXPLORE scores did not essentially change that much in the last four or five years, is that something we should be concerned about?”
Mr. Terry said, “Absolutely. To be frankly honest with you, we should be screaming in the streets. And I mean the parents, the African-American parents … This is to me unacceptable.”
Dr. Murphy said, “We are not going to hold our teachers accountable for the EXPLORE test. …We’re going to continue teaching and holding them accountable for how [students] perform on the federal accountability system [the ISATs].” He said under the No Child Left Behind Act’s accountability system, 77.5% of each subgroup of students must meet standards on the 2010 ISATs. He said adding EXPLORE to the accountability system would change the focus and be like “robbing Peter to pay Paul.”
The idea “is to make our instructional program robust enough so the EXPLORE scores go up,” Dr. Murphy said.
Bonnie Lockhart said a majority of the Board recently decided not to use EXPLORE as an accountability standard. She also defended the progress made by African-American students, saying they are doing much better now than when Dr. Murphy took over as superintendent.
Mr. Quattrocki asked, “Why do you think we haven’t seen any movement [on the EXPLORE test], as much movement as we’ve seen on the ISATs, why has it been close to stagnant for African-American students?”
Suzanne Farrand, the District’s math and enrichment coordinator, said she was not speaking from a research-based point of view, but said math instruction has changed dramatically in the elementary schools over the last 10 years. She said, “EXPLORE is exactly the same test as it has probably been for the last 40 years, for all I know. …I think the kinds of things that we teach benefit children in terms of understanding and knowledge in mathematics, and that is not captured on EXPLORE.”
Yet, Ed Colby, spokesperson for ACT, told the RoundTable that “EXPLORE is designed to reflect the knowledge and skills that are taught in schools across the United States,” and it is revised periodically, based on a survey of thousands of teachers across the country, to keep up with current trends in math instruction and learning. He also said “EXPLORE does emphasize quantitative reasoning and understanding, rather than memorization of formulas and computational skills.”
The EXPLORE test is closely aligned with the ACT, which is given to eleventh-graders as part of the Prairie State Achievement Exam (PSAE), the state’s mandated assessment at the high school level. ISBE has recently encouraged school districts to give the EXPLORE test to eighth- or ninth-graders.
Readiness for High School
Mr. Brinson said, “When our students are leaving this District [District 65], they’re leaving this District apparently very well-prepared to succeed, but three years later they’re not doing as well.”
The table below gives the percent of eighth-graders meeting standards on the 2006 ISATs and the percent of eleventh-graders meeting standards on the 2009 PSAEs:
Reading ’06 ISAT ’09 PSAE
White: 97 93
Black: 71 39
Hispanic: 77 50
White: 98 93
Black: 76 37
Hispanic: 84 60
The drop in the percentage of eighth-graders meeting standards on the ISATs and the percentage of eleventh-graders meeting standards on the PSAEs is a statewide phenonomen. For example, on a statewide basis, the percentage of black eighth-grade students who met standards in reading on the 2006 ISATs was 53%, the percentage dropped to 28% on the 2009 PSAEs. For Hispanic eighth-graders the percentage dropped from 52% on the 2006 ISATs to 37% on the 2009 PSAEs.
Andrea Preston, communications specialist for ISBE, told the RoundTable in a 2007 interiew that ISAT scores do not predict PSAE scores. Matt Vanover, spokesperson for the ISBE, told the RoundTable in an interview in 2008, “We have heard from the field for a number of years about the misalignment between students at the elementary schools and the high schools.”
Judith Levinson, director of research and evaluation, told the RoundTable that a 2008 study, “From High School to the Future: The Pathway to 20” conducted by the Consortium on Chicago School Research at the University of Chicago (the U of C Study), confirmed the misalignment between the tests.
The U of C study analyzed how cohorts of students in the Chicago School System did on the ISAT and later on the ACT. The report found that students who just barely meet the state standards on the ISAT have virtually no chance of reaching a 20 on the ACT, which are given as part of the PSAE. The report says, “This suggest major misalignment between our expectation for what students should know and be able to do at the end of elementary school and whether or not they are on track for college readiness.”
On Track for College Readiness
An ACT study and the U of C study give some insight into what EXPLORE scores and ISAT scores mean in terms of predicting how a student will do on the ACT and whether a students is on track for college and the workforce.
On the last three EXPLORE tests, District 65 black and Hispanic students had average composite scores ranging between 14.1 and 14.8. A 2008 study by the ACT, “The Forgotten Middle,” found that students who have a composite score of 15 on EXPLORE, on average, have a composite score of 18 on the ACT. The U of C study found that only 11% of the students who scored at or below 15 on EXPLORE, went on to score a 20 on the ACT.
District 65’s ISAT scores predict a better outcome. In the last three years, District 65’s black eighth-graders had average scale scores of 268, 262 and 271 in math; Hispanic students had average scale scores of 275, 266, and 276. The average scores have thus ranged from 262 to 276. The U of C study says an ISAT score of 265 in math equates to about a 20 percent chance of reaching a 20 on the ACT three years later, and an ISAT score of 275 corresponds to about a 40 percent chance of reaching 20 on the ACT three years later.
Stuart Luppescu, one of the authors of the U of C study, told the RoundTable he would expect the same type of relationships between ISAT and EXPLORE scores and ACT scores found in the U of C study to exist at other school districts, but said there could be variations either up or down. The report also says that students who attend a strong high school and earn good grades show higher gains than other students.
The U of C study picked out a score of “20” on the ACT because the Chicago School System picked that as a goal for its students, and it was slightly below the state and national averages. The report says a composite score of 20 would give a high school graduate “some moderate chance” of being accepted into Southern Illinois University, Northern Illinois University and University of Illinois at Chicago, where an ACT score of 20 ranked at about the 25th percentile rank for the college class of 2005. The ACT’s composite benchmark score for college readiness is 21.25.
The District presented data from an ACT report that the typical ACT scores of students accepted to a college with a “traditional” admissions policy are in the 18-24 range, and for a college with a “liberal” admissions policy, the range is 17-22. The range of ACT scores is higher at colleges with “highly selective” or “selective” admissions policies, says the report.
As far as improvement on the ACTs, ETHS reported that its senior class had a record-high composite score of 23.5 on the ACT college-entrance exam in 2009. In the last five years, the composite ACT scores for white students were 26.6, 26.5, 27.0, 27.0 and 27.2; for black students they were 17.4, 18.2, 17.2, 17.6 and 19.2; and for Latino students they were 20.7, 18.8, 19.0, 18.1 and 19.0.
Table No. 1 – D65 8th Grade Scores on EXPLORE
These tables give District 65 eighth-graders’ composite scale scores on EXPLORE and also the percentage of those students who ranked above the 50th percentile nationally in reading on EXPLORE. The data is obtained from reports published by School District 202.
Composite Scale Scores on EXPLORE
2002 2003 2004 2005 2006 2007 2008 2009
- Black 14.1 13.3 13.6 14.0 14.2 14.1 14.2 14.5
- Hispanic 14.5 14.7 14.9 14.2 14.9 14.8 14.4 14.8
- White 19.5 19.5 19.4 19.3 19.4 19.5 19.4 19.7
Percentage Above the 50th Percentile in Reading
Black 45 40 50 49 47 51 49 62
Hispanic 47 50 63 55 55 60 58 54
White 94 93 95 95 94 95 94 94
Table. No. 2- MAP Test – Percentage of D65 Eighth-
Graders Above 50th Percentile
Reading 2007 2008 2009
Black 45 46 52
Hispanic 45 38 45
White 89 89 92
Black 46 43 53
Hispanic 51 42 56
White 90 89 93
Table No. 3 – Percentage of D65 Students
On 2003-09 ISATs
Reading 2003 2004 2005 2006 2007 2008 2009
Black 43 51 51 64 69 74 75
Hispanic 46 47 49 70 78 68 69
Asian 79 83 88 91 93 91 92
White 89 92 94 95 97 97 97
IEP 25 30 37 44 49 51 52
LEP 31 35 33 64 71 49 42
Math 2003 2004 2005 2006 2007 2008 2009
Black 51 57 61 77 81 82 82
Hispanic 60 61 58 80 83 84 85
Asian 93 90 92 96 95 96 95
White 92 93 94 98 98 98 99
IEP 33 37 45 58 63 64 64
LEP 41 45 41 68 72 77 72
Table No. 4 – D65 ISAT Scale Scores
Another way to look at ISAT performance is to look at the average scale score of students. Students taking the ISAT are given a “scale score” which is based on the number of right and wrong answers that a student has on the test. ISAT reading and math scores are reported on a “continuous standard scale across grades.” The Chicago Sun Times uses the average scale scores of students to compile its annual list of best performing schools.
The table below gives the cut-off scores (similar to a passing grade) to make it into the “meets” and “exceeds” performance categories on the ISATs, the average reading scale scores for District 65 students on the 2009 ISATs, and the state average scale score of students in the State on the 2009 ISATs.
Cut-off Score D65 Students Score
Grade Meets/Exceeds White Black Hispanic State
3rd 191/227 237 201 195 206
4th 203/237 249 214 208 218
5th 215/247 260 227 230 230
6th 220/257 264 234 231 239
7th 226/267 272 237 237 243
8th 231/278 272 246 248 249
The table reflects that the average score of District 65 white students substantially exceeded the average scores of all students in the state, and that white fourth-graders had an average score of 249, which equaled the state average score for eighth-graders.
Mary Fergus, spokesperson for ISBE, told the RoundTable that this meant white fourth-graders in District 65 were reading at the same level as the average eighth-grader in Illinois. “The same scores at different grades have the same meaning, they indicate the same skill levels as far as difficulty is concerned. The only caveat is that the contents of the tests tend to differ from grade to grade,” she said.
The average scores for black and Hispanic students were generally below the state average. At eighth-grade, their average scores were only a point or so below the state average; they were, however, about 25 points below the average score of white eighth-graders.
Addressing the Needs of Students Not Meeting StandardsMr. Brinson presented data that 612 students were not meeting standards in reading and 394 were not meeting standards in math. He said about 60% of these students are students with a disability and have an Individual Education Plan (IEP) and/or are Limited English Proficient (LEP). Some of the students are also from low-income families.
Mr. Brinson said 16% of the District’s students were performing below expectations. “”We have to look at our interventions in terms of the specific kinds of learning needs that children have.””
Assistant Superintendent Susan Schultz said, “”We’re looking at changes to instruction to address the needs of the 16% who are below expectation. Our goal is to serve those students in the general education classroom. That’s where they can be most successful.””
Ms. Schultz said the District would use Response to Intervention, RTI, to help target strategies for struggling students, continue the implementation of the inclusion model to make sure all students have access to the general curriculum, continue with differentiated instruction to create classrooms that are responsive to the needs of students; and continue with professional development.
“”We’re focusing on changing classroom instruction,”” she said. “”We’re not focusing on adding programs around that might or might not address their needs.””