On June 17, School District 65 administrators presented their analysis of an algebra pilot program established at the start of the 2012-13 school year at King Lab and Bessie Rhodes magnet schools. The pilot eliminates tracking and collapses Algebra 1(high school algebra 1) with Algebra 8 (eighth-grade algebra). Thus as one administrator said, the classes in the pilot have “students from the zero percentile to the 99th percentile.”
Algebra 1 has historically been offered to students at or above the 80th percentile rank. Algebra 8 has been offered to all other students. Algebra is the only course that is tracked in District 65. Seventy-five percent of the school districts in the nation track courses in algebra. Evanston Township High School tracks math-related classes.
While Algebra 1 (high school algebra 1) is offered to higher-performing students, District administrators say the students in Algebra 8 are offered a more rigorous curriculum than the one offered to students in Algebra 1.
The administration’s goal for the pilot is to increase the number of kids who take geometry after an algebra course. “The real question we’re testing here is what happens to kids who aren’t necessarily slated to go to geometry [after taking algebra],” said one administrator. “Do those kids benefit from [having] higher-achieving kids in their classroom?”
To us, other important questions are these: Does the pilot’s approach increase the achievement of all students? And, if the pilot is expanded to the middle schools, which have significantly larger algebra classes, will the teachers be able to effectively differentiate instruction to meet the needs of all students? This is no small feat. The average student in the top quartile is performing about four grade levels above the average student in the bottom quartile, using ISAT scale scores as the measure.
The pilot should answer these important questions, and do so in a transparent, thorough manner. We think the administration’s analysis falls far short of doing so.
The pilot was essentially put in place last year before the School Board knew about it, and Board members had no input on its implementation. On June 17, the School Board decided to continue the pilot at the magnet schools next year, but not to expand it to the middle schools. The Board said it will decide on goals for the pilot in the fall, and also decide how to assess the pilot.
Too Many Students In the Bottom Quartile
The analysis of the pilot presented by administrators shows that more than 120 eighth-graders taking algebra in the 2012-13 school year scored in the bottom quartile in math on the fall 2012 MAP test.
On June 13, administrators presented additional data showing that 29% of black students and 21% of Hispanic students in the District scored in the bottom quartile in math on the 2012 ISATs. The vast majority are from low-income households.
Board members rightly expressed concerns about these results. One of the Board goals adopted in August 2011 is that the District decrease the number of students in the bottom quartile. One Board member has said on at least three occasions when the pilot has come up for discussion that the lack of preparation does not suddenly materialize in eighth grade. It exists long before then. On June 17, he said he wanted the District to do a “root-cause analysis,” and figure out a way to better prepare students early on so they will be better prepared by eighth grade.
We agree. The administration should also analyze why such high percentages of black and Hispanic students are performing below grade level and below college readiness benchmarks when they enter eighth grade.
The Analysis of the Pilot Falls Short
A recent meta-analysis cites research finding that low-performing students do better in tenth grade if they take algebra in ninth grade, rather than in eighth, and finds that when struggling students made achievement gains in algebra, they “occurred in settings where policies were accompanied by strong supports for struggling students, particularly more time for algebra instruction.” Algebra: A Challenge at the Crossroads of Policy and Practice (2011).
In addition, the meta-analysis and a more recent report found evidence that high-performing students may lose ground in de-tracked algebra classes.
The analysis presented by District 65 administrators does not adequately assess the impact of the pilot on low-performing and high-performing students.
The analysis itself shows that low-performing students taking algebra in the pilot at the magnet schools did worse than low-performing students taking Algebra 8 in the middle schools. On a net basis, no students moved up from the bottom quartile in the pilot. By contrast, 28 percent of the students taking Algebra 8 moved up. A more nuanced approach would shed more light on this problem.
Moreover, the analysis does not assess how the pilot is impacting high-performing students. While it analyzes how many students were in the top quartile at the beginning and the end of the school year, it does not analyze whether there are any shifts in achievement within that quartile.
For example, hypothetically, the percentile rank of the average ISAT scale score of students in the top quartile could decline from the 90th percentile to the 85th percentile, and this would not show up in the administration’s quartile analysis.
We think the administration should use a much more nuanced approach to analyze the impact on low-performing and high-performing students. There are many way to do this, including using changes in percentile ranks and changes in stanine levels. There are roughly three stanines in both the bottom and the top quartiles.
We also think the administration should measure changes in student achievement by comparing a student’s year-end scores to his or her prior year-end scores, rather than by comparing the student’s beginning-of-the-year scores on the fall MAP test to his or her year-end scores on the spring MAP test. Comparing year-end scores to prior year-end scores gives better assurance that the tests are given under comparable conditions, and that there is in fact growth from one year to the next.
In the interest of transparency, the administration should also provide scores on the placement tests, and the number of students actually placed in geometry honors, geometry and geometry with supports. Several Board members said they want to see how students in the pilot do in ninth and tenth grades, compared to other students. We think this would provide important information.
Setting Up a Valid Pilot
Administrators presented data showing that a higher percentage of students in the pilot were advanced to geometry than students in the middle schools. Several Board members said they did not think the pilot’s results can be used to predict how students in the middle schools will do if the pilot is expanded to those schools. We agree.
There are many ways in which the pilot is not comparable to the algebra classes in the middle schools. First, the average class size for the algebra classes in the pilot is 16.4 students, compared to 20.2 for students in Algebra 8 and 27.6 for students in Algebra 1. Second, 74 percent of the algebra students in the pilot scored above the 50th percentile in math on the fall MAP test, compared to 65 percent in the middle schools. Based on these two factors alone, one would expect that the pilot would have better results.
Other factors may include the level of supports provided to teachers in the pilot; the quality of teachers and teachers’ expectations; and the criteria, if any, used to select students for algebra classes.
If the administrators are serious about conducting a pilot, they should make the class sizes in the pilot comparable to those at the middle schools (about 26 students), make sure the supports provided to teachers in the pilots can and will be made available to teachers in the middle schools if the pilot is expanded, and take other steps to ensure that the data generated in the pilot is predictive of what will happen if the pilot is expanded to the middle schools.
Administrators should continue to present data for the pilot, Algebra 1 and Algebra 8. They should also present aggregated data for all algebra students in the middle schools.
We strongly support the goal of increasing the number of algebra students who are prepared to take geometry honors and geometry. One way to do this is to raise the achievement level of students well before they reach eighth grade. The District should take a fresh look at how to do this. The District should also construct a valid pilot and conduct a much more rigorous analysis of the pilot.
The decision on whether or not to expand the pilot, or some variation, can then be made on the basis of better information.