School District 65 administrators said at a School Board meeting on Nov. 5, 2007 that District 65 has made substantial gains on the Illinois Standard Achievement Test (ISAT) and that those gains erode at Evanston Township High School by the time eleventh-graders take the Prairie State Achievement Examination (PSAE).

Paul Brinson, director of information services for District 65 said, “One of the things the high school says is if we sent them better students, they could do better. Well, we’re sending them pretty good students.”

Mr. Brinson’s remarks come on the heels of District 202’s being placed in restructuring status for failing to meet Adequate Yearly Progress under the No Child Left Behind Act for black, Hispanic and other student subgroups for the fifth year in a row.

“We know that there is a lot of good work with students going on at District 65,” Eric Witherspoon, Superintendent of District 202 told the RoundTable. “Students leave eighth grade and make the transition to high school at lots of different levels and proficiencies. Some are highly accelerated and other students come struggling greatly.”

Judith Levinson, director of research, evaluation and assessment at ETHS, says the ISATs and PSAEs are not aligned and that an eighth-grader’s ISAT results cannot be used to predict how he or she will do as an eleventh grader on the PSAEs.

During the same time-frame in which District 65 has made substantial gains on the ISATs, the District’s eighth-grade students’ performance on another test, the EXPLORE test, has been flat. The EXPLORE test is “designed to measure students’ curriculum-related knowledge and the complex cognitive skills important for future education and careers,” according to the technical manual for the test.

The various tests raise questions concerning whether achievement levels are eroding at the high school and/or whether a substantial number of students are being graduated from District 65 without the knowledge and cognitive skills to succeed in high school and later in college.

The ISAT vs. PSAE debate

Mr. Brinson presented slides at the Nov. 5 District 65 School Board meeting that showed District 65’s gains on the ISATs during the last four years. Black and Hispanic eighth-graders have made substantial gains, particularly on the 2005 and 2006 ISATs. The percentage of black eighth-graders who met or exceeded standards in reading increased from 54% to 71% between 2004 and 2006. In math, the percentage of black eighth-graders meeting or exceeding standards increased from 37% to 75% between 2004 and 2006. Hispanic eighth-graders showed similar gains.

To make his point that gains at District 65 are eroding at ETHS, Mr. Brinson presented a chart that showed the percentage of District 65 eighth-graders who met standards on the 2004 ISATs, versus the percentage of ETHS eleventh-graders who met standards on the 2007 PSAEs. The cohort analysis showed substantial drops standards on the 2004 ISATs, versus the percentage of ETHS eleventh-graders who met standards on the 2007 PSAEs. The chorot analysis showed substantial drops at the eleventh-grade level. An abbreviated version of Mr. Brinson’s chart is shown below:

Relying on a study conducted by MetriTech, Inc. for the Illinois State Board of Education that analyzed the relationship between ISAT and PSAE scores, Mr. Brinson said that eighth-grade ISAT results are predictive of eleventh-grade PSAE results. He said if 54% of black eighth-graders met standards on the 2004 ISATs, he would expect that about 54% of that cohort group would meet standards as eleventh graders on the 2007 PSAE test. Instead, scores dropped from 54% on the 2004 ISATs to 33% on the 2007 PSAEs.

In commenting on the drop, Mr. Brinson said, “There really is something not going on for those students, because we would expect there’d be higher percentages of students meeting expectations.”

Mr. Brinson added based on the 2007 ISAT scores of the eighth-grade class that just graduated from District 65, “we would expect that 82% would meet standards in reading and 88% percent would meet standards in math on the 2010 PSAEs.”

Superintendent Hardy Murphy said the growth in eighth-graders’ ISAT scores was obtained through an instructional environment that included differentiation of instruction, enrichment and an array of support systems.

“If students find similar kinds of support awaiting them as they matriculate from our system out into the next system, then you will see those correlation coefficients being validated,” he said. “If they move into an environment that’s substantially different, then you’re going to see a difference in terms of their performance from what would be predicted based upon the State analysis.”

The MetriTech study Mr. Brinson says he relied on states that an eighth-grader’s ISAT “performance level” (i.e. “warning,” “below standards,” “meets standards” or “exceeds standards”) matched his or her “performance level” as an eleventh grader on the PSAEs 62% of the time in reading and 70 % in math. Mr. Brinson acknowledged in an interview with the RoundTable that this meant that eighth-grade ISAT performance levels would not match eleventh-grade PSAE performance levels in reading 38% of the time and in math 30% of the time. Mr. Brinson said, though, that for the students whose performance levels did not match, he would expect that one-half of the time their performance level would go up and one-half of the time down, so that overall eighth-grade ISAT scores would predict eleventh-grade PSAE scores.

Judith Levinson, director of research, evaluation and assessment for District 202, told the RoundTable there is no data supporting the assumption that one-half the scores would go up and one-half down, but it would depend on how the scores were spread. She also said that Sam Krug of MetriTech told her that performance on the ISATs is not predictive of performance on the PSAEs. Andrea Preston, communications specialist for ISBE, told the RoundTable the same thing.

Statewide results on the 2004 ISATs and 2007 PSAEs appear to support claims that performance levels on the eighth-grade ISATs are not predictive of performance levels on eleventh-grade PSAEs. The Statewide trends are similar to the trends at ETHS.

For example, on a Statewide basis, the percentage of eighth-grade white students who met or exceeded standards on the 2004 ISATs in reading was 76%; the percentage dropped for those students as eleventh graders to 64% on the 2007 PSAEs. For black students the percentage dropped from 48% on the 2004 ISATs to 28% on the 2007 PSAEs. For Hispanic students the percentage dropped from 51% on the 2004 ISATs to 33% on the 2007 PSAEs.

“This is a big problem for Illinois,” Dr. Levinson told the RoundTable. “It’s not good for kids, and it’s confusing for parents. The tests need to be aligned so that there is one set of standards along the way, K through 12.”

The EXPLORE Test

During the same period that District 65 has made substantial gains in increasing the number of students who meet or exceed standards on the ISATs, the District has also increased the number of students with a national percentile rank of 50 or above on the national norm portion of the ISATs, or in other words, it has increased the number of students performing among the top one-half of students in the nation.

In reading, the percentage of white students in the top 50th percentile in the nation increased from 77% on the 2003 ISATs to 90% on the 2007 ISATs. For black students the percentage increased from 37% in 2003 to 60% in 2007. For Hispanic students, the percentage increased from 44% in 2003 to 70% in 2007.

Despite these increases, the performance of District 65 eighth-grade students on the EXPLORE test has been generally flat. The percentage of white, black and Hispanic students who scored above the 50th percentile rank in the nation on the EXPLORE test in December 2003, 2004, 2005, and 2006 is as follows:

• White students: 95%, 95%, 94%, and 95%;

• Black students: 50%, 49%, 47%, and 51%.

• Hispanic students: 63%, 55%, 55%, and 60%.

Dr. Levinson said the passing standard for EXPLORE is the 50th percentile. In a memorandum dated Oct. 4, she said, “For EXPLORE, you need to be at least at the 50th percentile to meet standards and be on track to pass the PSAE/ACT and be successful in college work.”

Using EXPLORE as the measure, about 50% of District 65’s black eighth-grade students and about 40-45% of the District’s Hispanic eighth-grade students have not met standards for the last five years. District 202 provides remedial programs for many of these students.

Various theories have been offered to explain why District 65 students have shown substantial gains on the ISATs, but shown flat performance on the EXPLORE test.

Mr. Brinson said, “The EXPLORE test is not aligned with State standards. It is not the standard we work toward. It is not the appropriate standard.” He said the test is used as a “counseling tool” for eighth-graders entering the high school.

Dr. Murphy said, “The EXPLORE test is an attempt to grab national standards and come up with some kind of alignment with the State standards.” He added, “With the PSAE, there is more of an alignment with our State standards so the instructional precision is greater.”

Dr. Levinson has a different view. She told the RoundTable that the EXPLORE test is part of the same family of tests as the ACT, that it is directly aligned with the ACT and that the ACT is given as a part of the PSAE tests.

She thinks that students may do better on the ISATs than EXPLORE because the ISAT cut score to “meet standards” corresponds to the national percentile rank of the 37.5 percentile rank, while the passing standard for students on the EXPLORE test is the 50th percentile rank.

Another theory is that changes to the 2006 ISATs have made it easier to pass the ISATs, while the EXPLORE test has not changed. See sidebar.

The Technical Manual for the EXPLORE tests, which cover English, mathematics, reading and science, states “These tests are designed to measure students’ curriculum-related knowledge and complex cognitive skills important for future education and careers…The fundamental idea underlying the development and use of these tests is that the best way to determine how well-prepared students are for further education and for work is to measure as directly as possible the knowledge and skills needed in those settings.”

The EXPLORE test thus provides benchmarks for college readiness, which are minimum scores required for students to have a high probability of success in college, by the time they graduate from high school. For the EXPLORE tests the composite benchmark is 16.25 out of 25 scale points. For the ACT tests, the composite benchmark is 21.25 out of 36 scale points.

The table below gives the composite scores for white, black and Hispanic eighth-grade students at District 65 on the 2004 EXPLORE tests and as eleventh-graders at ETHS on the 2007 ACTs. The benchmark for college readiness is in parenthesis:

         2004 EXPLORE       2007 ACT

White 19.4 (16.25)             27.6 (21.25)

Black 13.6 (16.25)             18.6 (21.25)

Hisp. 14.9 (16.25)               21.2 (21.25)

Many black and Hispanic students fall short of EXPLORE’s college readiness benchmark when they transition to ETHS. While white and Hispanic students make gains in relation to the benchmarks for college readiness during their high school years, the composite score for black students still falls short of that mark.

Nov. 14, 2007

NAEP Tests: 2/3 of Illinois Students Not ‘Proficient’

The Nations Report Card shows that only about one-third of Illinois students performed at a “”proficient”” level on the 2007 National Assessment of Educational Progress (NAEP) tests. The NAEP is given to a sampling of fourth- and eighth- graders every two years in all 50 states as part of the No Child Left Behind Act. About 700,000 students were given the NAEP in 2007. The results are reported in the Nation’s Report Card under the auspices of the U.S. Department of Education.

For reporting purposes, the NAEP uses four achievement levels: below basic, basic, proficient and advanced. The “”proficient”” level is defined as having “”demonstrated competency over challenging subject matter, including subject matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter.””

On the 2007 NAEP tests, 32% of Illinois’ fourth-graders met or exceeded the proficient level in reading and 36% met or exceeded the proficient level in math. More than twice as many students met or exceeded standards on the 2007 ISATs: 74% of fourth graders met standards in reading and 86% met standards in math.

The dichotomy is similar at the eighth-grade level. On the 2007 NAEP tests, 30% of Illinois eighth-graders met or exceeded the proficient level in reading and 36% met or exceeded the proficient level in math. On the 2007 ISATs, 82% of eighth-graders met or exceeded standards in reading and 81% in math.

 

Changes to the ISATs May Make Test Easier

A number of changes were made to the 2006 ISATs, which some critics say are responsible for or contribute to the higher scores: 1) The State Board of Education sharply lowered the bar for eighth-graders to meet standards in math on the 2006 ISATs; 2) the State Board changed other cut scores for other grades and subjects, which it maintains had no effect on pass rates; 3) students were given an extra 10 minutes to take each section of the reading and math tests, which on a cumulative basis add up to an extra 60 minutes to take the tests; 4) a new firm was retained to prepare the test questions; and 5) other changes were made.

A study called “”The Proficiency Illusion”” concluded that a substantial portion of the higher pass rates on the 2006 ISATs were due to these changes; this is disputed by the Illinois State Board of Education. See Oct. 17, 2007 issue of the RoundTable.