Peter Godard, the District’s new chief of research, accountability and data, presented a Performance Data Snapshot to the District 65 School Board at its Dec. 1 meeting. It was Mr. Godard’s first day on the job, although he prepared the report beforehand. He previously served as Chief Performance Officer for the Illinois State Board of Education (ISBE).
Paul Goren, superintendent of District 65, opened the discussion telling Board members there was a lot of information in the District’s prior achievement reports. “Our ultimate goal and our immediate goal is to make the information usable for you, who are decision makers, and all of us who are deeply interested about the schools.
“We’re on a journey to make the achievement and accountability report a useable one, something that is understandable, something that can really have legs inside the District,” Dr. Goren said.
Dr. Goren said the presentation by Mr. Godard would be a preliminary report, a “snapshot of our perspectives on accountability and achievement.” He said administrators wanted feedback from the Board to help guide them to “frame and reframe what has been a long report with lots of information that is not as understandable as I wish it would be.”
The next generation report will be provided to the Board in January in time to discuss at their Jan. 20 Board meeting, Dr. Goren said.
Measurement of ‘Proficiency’
Mr. Godard’s snapshot report presented data on “Academic Success” and the “Gaps in Academic Success” using the percentage of students who “met standards” on the Illinois Standard Achievement Test (ISAT), rather than the percentage of students who were on track to college and career readiness.
ISBE raised the cut scores to “meet standards” on the ISAT in January 2013. Mr. Godard said, though, the new cut scores “fall short of meeting the ACT college readiness benchmarks. They fall short of the proficiency levels that have been looked at in Evanston in the past.”
He explained he was using the “meet standards” benchmark in his presentation because it was publicly reported by ISBE and thus readily available. By moving ahead with data that was publicly available “we didn’t have to wait another month-and-a-half to begin to have a conversation about what does our performance profile look like,” Mr. Godard said.
Board President Tracy Quattrocki said she was disappointed the new ISAT cut scores were lower than the proficiency level needed to indicate being on track to college readiness. She noted that the new cut scores were set at about the 40th percentile; and the benchmarks used by District 65 to indicate college and career readiness were the 60th percentile in reading and the 68th in math. These benchmarks were identified by Paul Zavitkovsky of the Urban Education Leadership Program at the University of Illinois-Chicago and have been used by District 65 since 2011.
“We as a Board have defined academic success as being on track to college and career readiness,” said Ms. Quattrocki. “When you do our internal analysis, we use a higher bar than meets expectations on the ISATs.”
Using the higher bar makes a significant difference. The RoundTable asked Mr. Zavitkovsky to calculate the percentages of white and black students, disaggregated by income, who met three different benchmarks on the 2014 ISATs: a) “meet standards” on the ISATs (using ISBE’s new cut scores), b) at/above the 50th percentile, and c) on track to ACT college readiness, using the 60th percentile in reading and the 68th in math as the benchmarks. He did so using publicly available student-level data sets. The charts below were prepared using data provided by Mr. Zavitkovsky.
The above charts show the percentage of District 65 African American and white students, by income level, who met three different benchmarks on the 2014 ISATs: a) “meet standards,” using ISBE’s new cut scores; b) at/above the 50th percentile, and c) on track to ACT college readiness, using the 60th percentile in reading and the 68th in math as the benchmarks. For non-low-income African American students, N=225; for low-income African American students, N=786. For non-low-income white students, N=1,839; for low-income white students, N=119. In accordance with FIRPA requirements, some African American and white students may have been excluded from the public data used to produce these charts. Data provided by Paul Zavitkovsky, Urban Education Leadership Program, University of Illinois.
The charts illustrate that significantly lower percentages of students were on track to meet ACT’s college readiness benchmarks, than met standards on the ISATs. The charts also show that student achievement is statistically related to household income.
In a separate interview, Mr. Godard told the RoundTable that the January report would report the percentage of students who were on track to college and career readiness using two different tests: 1) the ISAT, using the 60th percentile in reading and the 68th percentile in math as the benchmaks; and 2) the Measures of Academic Progress (MAP) test, using RIT scores that correspond to the ACT’s college readiness benchmarks.
Because 2014 is the last year in which the ISATs will be given, Mr. Goddard said he anticipates District 65 will report progress toward college and career readiness in the future using MAP and also using benchmarks developed for the PARCC assessment, which is replacing the ISAT. He said it may take three years before benchmarks corresponding to college readiness are available for the PARCC assessment.
Student Growth Using ISBE’s Model
Mr. Godard presented data on student academic growth using a “Value-Table” model that was adopted by ISBE in January 2012. Each school district’s average growth, as computed under the model, is reported in ISBE’s online report cards.
Under ISBE’s model, student academic growth is calculated based on students’ changing performance levels on the ISAT from one year to the next. When a student takes the ISAT, his or her score falls within the range of one of four performance levels: academic warning, below standards, meets standards, and exceeds standards. For the growth model, each of these performance levels is also divided into two sub-levels, making a total of eight performance levels ranging from 1A for the lowest scores to 4B for the highest scores.
ISBE provides each school district with a table of cells showing the number of students who advanced one or more performance categories, the number who stayed in the same performance category, and the number who dropped one or more performance categories from 2013 to 2014.
ISBE’s “Value Table” assigns a numerical value for movement between performance categories and for staying at the same performance level. For example, if a student moves up from performance category 3B in 2013 to the next highest category, 4A, in 2014, that is worth 135 points; if a student stays at 3B from 2013 to 2014, that is worth 110 points; if a student drops from 3B to 3A, that is worth 80 points.
Under the model, the points for all students in a school or district are added up and then divided by the number of students to obtain an average growth score for the school or district. A score above 100 is deemed to show positive growth. A score below 100 is deemed to be negative growth.
On the 2014 ISATs, District 65 had an overall score of 103 in reading, which put it in the top quartile of all school districts across the State in academic growth in reading, said Mr. Godard.
District 65 had a growth score of 111 in math, which placed it in the top 10% of Districts in the State, he said.
A number of issues have been raised about the growth model. One key issue is the subjective scoring system. Mr. Godard said a group of educators worked with a statistician to determine the points that would be awarded for staying in the same performance category and for moving up or down from one performance category to another from one year to the next. “Part of what this entailed,” he said, “is what a representative group of educators valued. … That’s why it’s a value table.”
In a “Fact Sheet: New Growth Model Using Value Tables (August 2013),” ISBE says the educators decided to award more points to those students who maintain or increase achievement at the higher end of the achievement scale (i.e., categories 3A to 4B) than to students who start out in categories 1A to 2B. The model thus gives more weight to constancy and growth at the higher end of the distribution scale than at the lower end.
The difference in point values is readily apparent on the face of the Value Table. Students who stay in the same performance category from one year to the next are awarded 50 points if they started in the lowest category 1A; then 85 points if 1B; 90 points if 2A; 95 if 2B; 100 if 3A; 110 if 3B; 115 if 4A; and 130 if 4B.
In an interview with the RoundTable, Mr. Zavitkovsky said, “The net effect of this weighting system is that higher-achieving schools end up looking like they are gaining ground when they may actually be staying flat or declining somewhat; and lower achieving schools end up looking like they’re losing ground when they’re actually staying flat or even making modest gains.”
At the RoundTable’s request, Mr. Zavitkovsky ran a simulation of what would happen under the model if one assumed that all of the students attending Dewey and Oakton Schools stayed in their same performance categories from 2013 to 2014. Dewey has a relatively higher achievement profile while Oakton’s profile is lower by comparison. Under this simulation, completely flat growth at Oakton produced a flat growth score of 100 in reading and a negative score of 99 in math. At Dewey, completely flat growth produced a positive growth score of 110 for both reading and math.
When asked to comment on this, Mr. Godard told the RoundTable that the ISBE growth model is not the most statistically sophisticated model. He acknowledged that there are better growth models, and that District 65 would not want to use a model that incentivizes growth at the high end of the distribution scale more than at the low end.
Nonetheless, he said the January report would likely report growth using the ISAT growth model “because it’s publicly reported as part of the State’s accountability reports, and with weaknesses in the model, it’s still a model that gives us some information.”
Debate about the ISAT growth model may be moot next year because the ISAT is being replaced by PARCC. Mr. Godard said, “I, we, will be active in the discussion as PARCC comes out to try to make sure that the growth model and the measurements that come out are going to meet our needs at District 65.”
He told the RoundTable that the January report would also report growth using MAP’s system, which can be used to compare a student’s actual year-end growth with their predicted growth.
Number of Students Changing Performance Categories
Mr. Godard presented charts showing the number of District 65 students who stayed in the same ISAT performance category in reading and math between 2013 and 2014 and who moved up or down one of more performance categories. He said he thought this provided the District some good information about where changes in achievement were occurring in the achievement distribution.
With regard to the reading chart (see below), Mr. Godard said growth is occurring at all levels of the distribution scale, with the exception of category 4B, the highest category, where there is no room to move up.
In 2013, 459 students were in category 4B. On the 2014 ISATs, 197 of the 459 students stayed in category 4B, and 262 dropped to a lower performance level. Many students, however, moved up to the 4B category from lower categories.
Board member Richard Rykhus noted that of the students in category 4A, the second highest category, twice as many dropped to a lower category as increased to a higher category. In 2013, there were 698 students in category 4A. Of those, 137 moved up to category 4B, and 284 dropped to lower performance categories.
On an overall basis, a total of 895 students moved up one or more performance categories in reading, and 1,181 dropped down one or more performance categories.
The chart for math was more positive. A total of 1,202 students moved up one or more performance categories; and 755 dropped down one of more performance categories.
Ms. Quattrocki asked Mr. Godard to include in the January report the range of percentile ranks included in each performance category so Board members would have a better feel where the changes in achievement were actually occurring.
Katie Bailey said, “I would like to peel back what some of it really means.”
Mr. Godard presented data showing the achievement gap that exists for students from low-income families, black and Hispanic students, students with a disability, and English language learners. The analysis was based on the percentage of students who “met standards” on the 2014 ISATs. He said all subgroups scored higher than their peers statewide, with one exception.
He started with the gap for students from low-income households. He said, “The reason we start here is really to emphasize what the research literature suggests, which is that poverty is the driver of the opportunity gap for our students. If we only look at one gap, this is probably the one to look at.”
Mr. Godard presented data showing that 44% of students from low-income households met standards in reading on the ISATs, while 86% of those from non-low income households met standards. In math, the percentages were 53% and 89%.
The gaps for black and Hispanic students, students with a disability and English language learners are equally pronounced. Many of these students are from low-income households.
Ms. Bailey asked if the data for black and Hispanic students could be disaggregated by household income – for example to report data for black students who are from low-income families and for black families from non-low income families.
Board member Candance Chow suggested separating out the achievement data for students who were entitled to “free-lunch” from those who were entitled to “reduced-fee lunch” to see if depth of poverty was having an increased impact. She said this may help to target interventions where they are needed the most.
According to District 65’s Opening School Report, 2,610 students, or about 36% of the District’s enrollment, are entitled to receive free lunch. These students are from households below 130% of the poverty line.
Board member Omar Brown asked if the District could break out the growth rate for low-income students.
Mr. Godard said breaking out growth for different subgroups of students would be at the top of the list of things to do.
Board Members Ask For More Detailed Data
Ms. Quattrocki asked Mr. Godard to report the percentage of students, by subgroup, who are on track to college and career readiness using the 60th percentile in reading and the 68th in math as the benchmarks.
She added that the Board would like to see the grade levels where the District is making gains or losing ground, and asked for grade-level data. Ms. Bailey also expressed interest in grade-level data to assess whether the District was gaining or losing ground in the middle schools.
Ms. Chow noted that the Board’s three-year goals included goals to increase the percentage of students who scored at/above the 50th percentile, to increase the percentage who were on track to college and career readiness, and to decrease the percentage who were in the bottom quartile. She asked if the achievement report would contain data to track progress in meeting those goals.
“I would think so,” said Ms. Quattrocki.
Mr. Rykhus said the Board had spent a lot of time in the past identifying “slices of data” that it was interested in tracking to monitor student achievement. “The different slices are something we should try to retain. You have the prior report. I would like you to pull out the slices and apply them to the data we’re looking at.”
Ms. Quattrocki referred to the goal in the District’s last five-year strategic plan that students who have been in the District since kindergarten be able to read at grade level by third grade. “I still feel like we haven’t dug deep enough. What can we learn from somebody whose growth is higher?” she said. “There must be schools and grade levels where there’s stand-out data and we’d love to see some of those comparisons so we could figure out what’s going on.”
Ms. Chow said, “Within our own district the average is here, but there are probably some schools that are way up here, we can take a look at our own best practices. Look at those high spots that we can model.”
Suni Kartha said she would like to see data tracking the growth of students who participated in Foundation 65’s reading program, the YMCA’s summer reading program, and the District’s extended day program. “What other sorts of support are students getting that are showing growth so we can decide if that’s something we need to expand?” she asked.
Mr. Rykhus brought up breaking out a snapshot of performance by school. Ms. Chow asked the administration to assess the data by school to determine if there were common threads that could be replicated in other schools.
Dr. Goren said, “I love these questions.” He said the purpose of presenting the Performance Profile Snapshot was to obtain Board members’ input to help shape the January report.
Mr. Godard told the RoundTable he would like to provide a narrative report in January, with charts and graphs and an appendix with more data runs for persons interested in delving into specific areas. He said he planned to include a separate report on programs, such as early childhood education, special education, the Two-Way Immersion program, and the African Centered Curriculum. Going forward, he said, he would like to do a more in depth and more frequent analyses of these programs.
Dr. Goren said, “We’d really like to build a suite of reports over time that is embedded in continuous improvement and understand very publicly where we are, where we’re going and what progress we’re making.”
He said the January report may be more reflective of what the reports looked like in the past, rather than “what we’d like to see in future reports. Our real commitment over time is to make it usable, to make it in regular doses and generate information that we all can use. That is our two to three year plan.”
ISBE Low-balled the New ISAT Cut Scores
In previous articles, the RoundTable presented data showing that the new ISAT cut scores are set at approximately the 40th percentile, and that scores that predict being on track to college and career readiness correspond to the 60th percentile in reading and the 68th in math. Data shows that students who score right at the new ISAT cut score to “”meet standards”” have less than a 10% chance of meeting ACT’s college readiness benchmarks in 11th grade. See, “”The ISAT Cut Score Fiasco: Where’s the Accountability,”” available at www.evanstonroundtable.com.
The RoundTable’s articles also reported data showing that the new cut scores are set about 10 percentile points below the proficiency needed to align them with the cut scores to “”meet standards”” on the PSAEs. In a separate interview, Mr. Godard said, “”What you’ve reported is probably accurate.””