A study released on Oct. 4, 2007 titled “The Proficiency Illusion” claims that a substantial portion of the dramatic increases on the 2006 Illinois Standard Achievement Tests (ISATs) is due to changes made to the test, rather than to improvement in student achievement. The 238-page study, conducted by the Northwest Education Association (NWEA) and the Thomas B. Fordham Institute, analyzed the tests used by 26 states to measure progress under the No Child Left Behind Act.

As part of the “Proficiency Illusion” study, researchers analyzed whether the changes made by the State Board in 2006 made it easier to meet standards on the 2006 ISATs compared to prior years. They concluded that much of the reported gain on the 2006 ISAT was due to changes in the tests.

School District 65 Superintendent Hardy Murphy told the RoundTable, “We don’t agree with that. We’re seeing not only an increase in ISAT scores, but also an increase in the percentage of students scoring above the 50th percentile rank nationally in reading and math.”

Matt Vanover, spokesperson for the Illinois State Board of Education (State Board), told the RoundTable, “The Fordham Institute is formalizing a plan for a national curriculum and national standard. The Fordham folks are pushing a national agenda. The study has some shortcomings.”

In the five-year period 2001-2006, District 65 improved its overall performance on the ISATs by 22 percent. Almost two-thirds of that improvement took place on the 2006 ISATs.

“The Proficiency Illusion”

In 2006 the State Board made a number of changes to the ISATs. Among these changes were altering the “cut scores,” those used to determine whether a student met standards.

With the exception of eighth-grade math, the State Board maintains that the new cut scores were changed to convert to a new vertical scale and that they are equivalent to the old cut scores when it comes to measuring student performance. The State Board acknowledges it lowered the bar to meet standards for eighth grade.

The State Board also extended the time to take each section of the reading and math tests, which on a cumulative basis added up to 60 more minutes to take the tests; a new firm was retained to prepare the test questions; and other changes were made.

As part of the “Proficiency Illusion” study, researchers with NWEA analyzed whether the changes made by the State Board in 2006 made it easier to meet standards on the 2006 ISATs compared to prior years. The NWEA did this by analyzing reading and math results from a group of elementary and middle schools in which almost all students took both the ISATs and the Measure of Academic Progress (MAP) test in 2003 and 2006. The MAP test was developed by NWEA and is used in more than 2,500 school systems in 49 states.

NWEA researchers concluded that the changes made it significantly easier for third- and eighth- graders to meet standards in reading on the 2006 ISATs and for fifth- and eighth-graders to meet standards in math on the 2006 ISATs. Gains shown on the ISATs at those and other grade levels were not present on the MAP tests.

NWEA also estimated the impact the changes made by the State Board would have on the 2006 ISAT results, using the MAP test results as the yardstick.

The State Board maintains that the new cut scores were changed to convert to a new vertical scale and that they are equivalent to the old cut scores when it comes to measuring student performance. The State Board acknowledges it lowered the bar to meet standards for eighth grade.

As an example, NWEA researchers determined that third-grade students in 2003 had to score at the 52nd percentile nationally on the MAP test in order to meet standards in reading on the ISATs; in 2006, third-graders had to score only at the 35th percentile nationally on the MAP test in order to meet standards on the ISATs.

Since the MAP test was constant, they concluded that the changes to the ISATs increased the number of students meeting standards on the ISATs by 17 percent – the difference between the percentile ranks on the 2003 and 2006 MAP tests.

Using this approach the NWEA researchers estimated that the changes made by the State Board increased the number of students meeting standards on the 2006 ISATs by the following percentages: third- grade reading -17%; fifth-grade reading – 3%; eighth-grade reading – 14%; third- grade math – 2%; fifth-grade math – 8%; and eighth-grade math – 27%.

The table below compares NWEA’s estimated increases due to the changes made by the State Board and the actual increases experienced by School District 65 students on the 2006 ISATs for the grades and subjects indicated:

The “Proficiency Illusion” study also concluded that meeting standards in math on the 2006 ISATs was easier than meeting the math standards of 80% of the other states reviewed. For reading, the study concluded that it was more difficult to meet standards on the ISATs at the third-grade level than 70% of the other states reviewed, but at the eighth-grade level meeting reading standards on the ISATs was easier than 80% of the states reviewed.

“In our looking at it, regardless of what the cut scores are, we’re seeing more students scoring at or above the fiftieth percentile rank nationally. … We’re seeing demonstrated improvement.” — District 65 Superintendent Hardy Murphy

According to the study, Illinois’ eighth-graders who performed at the 21st and 22nd percentiles nationally on the MAP tests in reading and math (meaning that 79 and 78 percent of the students taking the test did better than they), met standards on the 2006 ISATs.

D65 Points to increases in percentile ranks

Dr. Murphy said the study was apparently done as part of a national dialogue about using national standards versus state standards to measure progress under NCLB. “In our looking at it, regardless of what the cut scores are, we’re seeing more students scoring at or above the fiftieth percentile rank nationally,” he said. “We’re seeing demonstrated improvement.”

Starting with the 2006 ISATs, about 30 questions from the nationwide Stanford Achievement Test were included in the ISATs to enable school districts to compare how their students were doing in relation to students nationwide.

Paul Brinson, director of information services for District 65, told the RoundTable that an estimated 50% of the District’s third-graders scored above the 50th percentile rank nationally in reading on the 2003 ISATs; the number grew to 79% on the 2007 ISATs. In math, the percentages grew from 61% on the 2003 ISATs to 74% on the 2007 ISATs.

Dr. Murphy said the District also had substantial gains at the fifth-grade level. In reading, an estimated 59% percent of the District’s fifth-graders scored above the 50th percentile rank nationally on the 2003 ISATs; the number grew to 80% on the 2007 ISATs. In math, the percentages grew from 73% on the 2003 ISATs to 85% on the 2007 ISATs.

Mr. Brinson said the percentile ranks for 2003 are all estimates.

“Our percentile shift is in the right direction,” Dr. Murphy said.

The State’s Board’s response

Mr. Vanover told the RoundTable there are a number of problems with the “Proficiency Illusion” study. He said, “We don’t know the size of the sample they used. We don’t know how many students or what schools take the MAP test.”

The State Board questions whether the study is part of a debate on national standards.

He added, “What they fail to take into account is that there is a framework for Illinois learning: the Illinois learning standards. What they discount is that students are actually learning. What they’re saying is that if you have a large number of students taking a test and the scores are going up [there is something wrong].”

The Proficiency Study disputes Mr. Vanover’s comments. The study states, “NWEA curriculum experts evaluate the particular state’s content standards” and “the MAP test is purposely aligned to each state’s standards.”

The study also states that enough students were in the sample for each state to make the results reliable; an appendix to the study states the sample sizes in Illinois ranged from 1179 to 1654 students per grade level – roughly the same sample size used by the State Board in some of its own test procedures.

Before changing the cut scores in 2006, the State Board conducted a “Bridge Study” in an attempt to make the new cut scores it adopted equivalent to the old cut scores.

With the exception of eighth-grade math, the State Board maintains that “the only thing that has changed is the numerical value of the cut score. Thus, meeting the Illinois Learning Standards according to the new vertical scale requires as much knowledge as (and no more than) the old ISAT scale did.”

The State Board acknowledged that it lowered the bar to meet standards for eighth-grade math. The Bridge Study states the State Board did so because the score had been placed too high and was out of line with the other grades.

The Bridge Study estimated that the change to the cut score for eighth-grade math would increase the number of students meeting standards by at least 26 percentage points.

October 17, 2007

Impact of Changes Made to the ISATs

 

The following gives the increase in Illinois students meeting standards on the 2006 ISATs due to test changes in 2006 (as estimated by the Proficiency Illusion report), followed by the increases in School District 65 on the 2006 ISATs for the grade and subject indicated:

     -3rd/reading 17%, 18%

     -5th/reading 3%, 12%

     -8th/reading 14%, 11%

     -3rd/math 2%, 9%

     -5th/math 8%, 10%

     -8th/math 27%, 27%