Throughout the nation, there is an increasing focus on improving the effectiveness of teachers as a way to increase not only student achievement, but student engagement in the classroom. The issue was crystallized in a recent study, “The Irreplaceables,” which found that urban school districts – which were the focus of the study – were losing many top-performing teachers, while retaining a large number of low-performing ones.
In setting goals this year for Superintendent Hardy Murphy, the District 65 School Board has made improving the quality of its teachers a priority. One of the Superintendent’s goals is to conduct a study to review and evaluate the quality of teaching in the District, to identify the characteristics of successful teachers, to develop a plan to attract teachers who possess those characteristics, to examine whether the District is retaining its best teachers, and to outline professional development for teachers.
At the School Board’s Jan. 22 meeting, Dr. Murphy laid out how he plans to meet that goal. Dr. Murphy said he planned to conduct a study to review and evaluate teacher quality using both quantitative and qualitative data. “The results of this study will help identify areas for continuing professional development and help identify characteristics of effective teachers,” he said. “The information can be used to help in teacher recruitment, hiring and retention.”
While he proposed to retain Clayton J. Graham of Advantage Analytics, llc to assist in the quantitative aspect of the study, Dr Murphy told the RoundTable on Jan. 29 that the District would not be retaining him.
The Proposed Quantitative Study
At the Board meeting, Dr. Murphy proposed that the quantitative data be provided by Clayton J. Graham whom he said is a management consultant with Advantage Analytics, llc. Dr. Murphy envisioned that Mr. Graham would analyze student test data and identify teachers who have increased student growth either more or less than expected. Administrators would then talk to successful teachers to help identify what makes them successful.
Advantage Analytics, llc, is a limited liability company formed under Wyoming law in November 2009. Margaret Graham is the manager, according to records of Wyoming’s Secretary of State. Other public records indicate that she is Mr. Graham’s wife.
Mr. Graham told School Board members he would convert student scale scores on the Illinois Standard Achievement Test (ISAT) and the Measures of Academic Progress (MAP) test to an “index” so that the scores would be comparable across subjects, grades and years. He said he would then quantify how much each teacher’s students grew either above or below “what you expect,” taking into account student demographics such as income status. He said his study would cover a four-year period.
Dr. Murphy said the District will use also qualitative data as part of this analysis. The qualitative data will be obtained through classroom observations of teachers and interviews of focus groups, he said. The focus groups, composed of stakeholders, will be asked to talk about what they believe makes an effective teacher.
Dr. Murphy said, “The idea here is to come up with somewhat of an objective measure of what has happened over time with students and teachers and then to pair that with the subjective and narrative side of what people think and see are the qualities and characteristics of teachers that they appreciate. The two sides should work together in order for us to come up with some ideas of what teacher effectiveness is.”
Several Board members expressed concerns about proceeding with a study to identify effective teachers based on Mr. Graham’s methodology, at the same time the District was trying to determine the best methodology to use to rate teachers as “excellent, proficient, needs improvement, or unsatisfactory” in a teacher appraisal system.
Tracy Quattrocki said, “I was surprised by the presentation today because there is a national debate going on about how to effectively measure teachers through the mining of data, and we internally are having a very transparent and educated conversation about this.
“What I’m hung up on is the fact that we are trying to come up with the most reliable way to look at data for our teachers in our teacher appraisal system. And it seems to me if we are going to come up with the best way to look at data for our teachers for the evaluation system, then that should be the same way we’re looking at teacher effectiveness for other reasons.
She asked Mr. Graham to explain the methodology he planned to use to determine whether teachers were effective. Mr. Graham did not explain his methodology, but instead said, “The paper that Max McGee and I gave out in Las Vegas in November covers a lot of that stuff.”
The paper referred to by Mr. Graham, titled “Education Effectiveness: Fact, Fantasy and Fraud,” contains about 90 slides, mostly charts, but no narrative explaining the methodology he uses to quantify teacher effectiveness. The presentation was made to the 2012 “Palisades Risk Conference.”
In response to the RoundTable’s follow-up questions, Mr. Graham explained how he converts ISAT scale scores to an index. Essentially an index score represents the percentage a student’s scale score is above or below the state average scale score. But he did not describe the methodology he would use to quantify teacher effectiveness, including how he would determine a student’s expected growth and how he would take into account student demographics such as income status.
Eileen Budde said it appeared the District was using two different processes to basically identify great, good, and less good teachers. She asked, “Is it just me who thinks they should be wrapped together?”
Mr. Rykhus agreed. If there are two different systems being used to evaluate teachers, he said, “It seems like we’re setting up the possibility of giving two different messages.”
Two other Board members took an opposing view. Kim Weaver said, “I love this whole concept.” Andy Pigozzi said, “I see no downside in moving forward and collecting this data and analyzing it. We’ve always said before that we like to use a mix of measures.”
Ms. Budde suggested the District take advantage of what was learned in the three-year study conducted as part of the Measures of Effective Teaching (MET) project sponsored by the Melinda and Bill Gates Foundation. Ms. Budde said the final report, issued in January, contained a lot of good information that should be part of the conversation about what makes an effective teacher. See sidebar on page 22.
DEC President Jean Luft said, “DEC has some grave concerns about moving forward with Mr. Graham’s study.” She said the study will only assess teacher effectiveness at producing a certain score on a certain test and that other components of teaching would not be measured. She added that there was a concern that the findings and the teacher’s identity may not be exempt from a request made under the Freedom of Information Act.
Considering Other Options
On Jan 29, Dr. Murphy told the RoundTable that District 65 would not retain Mr. Graham. He said other school districts felt his approach had a lot of merit, but “I’m not sure we need him at this point.” He added there was nothing wrong with Mr. Graham’s methodology.
Dr. Murphy said he still planned to use classroom observations and focus groups to help identify the traits of successful teachers. He added that he planned to validate the qualitative part of the study with a longitudinal (multi-year) analysis of student achievement that would identify teachers who were consistently producing high rates of student growth in achievement. He said the District could then determine what traits those teachers possessed and pair up the findings from both parts of the study.
Graham: District 65 Moves from 333rd to 75th in State in Last 10 Years
Clayton J. Graham told School Board members on Jan. 22 that District 65 has “”done some remarkable things”” in the last ten years. Ten years ago, he said, District 65 ranked 333rd out of 800 elementary school districts in the State. Today, District 65 ranks 75th, he said.
Mr. Graham told the RoundTable that he ranked schools in the State by norming the ISAT data – he converted each student’s scale score to an index so the results are comparable across years, grades and subjects.
To quantify the progress in terms of percentile ranks, the RoundTable examined data provided last fall by Paul Zavitkovsky at the Urban Education Leadership Program, University of Illinois-Chicago. On the 2001 ISATs, the average scale score of all District 65 students was at the 58.8th percentile, meaning that the average scale score of all District 65 students was above 58.7% of the students in the State. On the 2012 ISATs, the percentile rank grew to 68.0%, an increase of 9.3 percentile points over eleven years. Most of the growth, 6.3 percentile points, took place between the 2001 and the 2006 ISATs.