A new report, “Upstate/Downstate,” by Paul Zavitkovsky and Steven Tozer at the University of Illinois at Chicago, analyzes how changes in low-income enrollment and school effectiveness have impacted student achievement in 709 Illinois school districts in the period between 2001 and 2016.
The study considers more than 24 million test scores in reading and math for all Illinois third- through eighth-graders who took the Illinois Standard Achievement Test (ISAT) from 2001 through 2014 and the PARCC test in 2015 and 2016. During that period, the percentage of low-income students in the State increased from 37% to 50%.
“Fifteen years after No Child Left Behind was signed into law, race and family income still predict standardized test scores with remarkable accuracy in most school districts,” says the Report. “But unlike fifteen years ago, the connections between low-income enrollment and lower achievement are now being enacted with the same depressing regularity in predominantly white communities as they have been for years in low-income communities of color. When it comes to academic achievement, economic distress is an equal opportunity disruptor.”
One key finding of the study is that most school districts in central and southern Illinois, which are predominantly white, saw significant drops in achievement as their low-income enrollment rose. “Without fundamental improvements in school effectiveness, higher concentrations of low-income enrollment have been accompanied by steep declines in achievement,” says Mr. Zavitkovsky.
But a theme running through the Report is that gains in school effectiveness can blunt, or in some cases reverse, the impact that low-income enrollment typically has on achievement. The report provides examples where the Chicago Public School system has shown significant increases in reading and math achievement at each grade level, 3-8. The Report cites additional data that suggests growth in school effectiveness could reasonably increase student achievement by between 0.75 and 2.0 grade equivalents statewide.
“It’s still not widely accepted that schools and districts have any real ability to disrupt the strong association between student achievement, household income, and other external factors,” said Mr. Zavitkovsky. “The central message of the report is that shifts in school effectiveness in Illinois are having at least as big an impact on student achievement as external factors are.”
Improving Achievement at Third Grade
“There is a long line of research on the close association between reading proficiency at the end of third grade and academic success in later years,” says the Report, citing the 2012 study, “Double Jeopardy.” As the first standardized measure of achievement, third grade scores represent everything that has occurred from birth through third grade.
The Report finds that Chicago has not only made much greater progress than the rest of the State in improving third grade achievement between 2001 and 2016, but it now is consistently outscoring the rest of the State in apples-to-apples comparisons that control for race and household income. To preserve the comparability of scores over the years, the report excludes scores for students who are temporarily identified as English Language Learners (ELL).
Figures 1 and 2, below, reflect the percentage of low-income white, Hispanic and black third-graders (who are not ELL) who scored at or above the 50th percentile in reading in the years indicated:
Chicago/Low-Income: Figure 1 shows that low-income black third-graders in Chicago moved up 13 percentage points between 2001 and 2011, and after that their growth was generally flat. Low-income Hispanic and white students in Chicago showed significant growth through the entire period, moving up 26 and 23 points respectively.
Outside Chicago/Low-Income: Figure 2 shows that low-income Hispanic (non-ELL) students outside Chicago showed significant growth between 2001 through 2011; otherwise, growth was generally flat for Hispanic, black, and white third-graders.
Figures 3 and 4 provide the same data, but for non-low-income students.
Chicago/Non-Low-Income: Figure 3 shows that achievement of non-low-income black, Hispanic, and white third-graders in Chicago has grown between 25 and 37 points between 2001 and 2016.
Outside Chicago/Non-Low-Income: Figure 4 shows that black and Latino third-graders showed growth between 2001 and 2011, but growth has declined or been flat after that.
Mr. Zavitkovsky attributes the growth in Chicago at third grade to increases in school effectiveness. In light of recent research by Sean Reardon and associates at Stanford’s Center for Education Policy Analysis that achievement/opportunity gaps are due primarily to differences that occur before third grade, it is significant that school effectiveness can move the needle at the third-grade level.
School District 65 has long recognized the importance of increasing reading proficiency by third grade, and the School Board has adopted third-grade reading goals on many occasions, including in its 2009 five-year strategic plan.
Figure 5 shows the percentage of black third-graders in District 65 who scored above the 50th percentile in reading for the years indicated. Data for the period 2001-2009 is taken from Mr. Zavitkovsky’s study of District 65’s achievement, done at the RoundTable’s request in 2010. Data for the years 2010 -2011 is taken from District 65 reports. The RoundTable lacks data for subsequent periods.
For the period 2001 -2011, the growth of black students in Evanston is comparable to that in Chicago.
Variances in Achievement Between School Districts with Same Concentration of Low-Income Students
The Report illustrates that school districts that have the same percentage of low-income students have significantly different achievement results.
Figure 6 plots 709 school districts in the State, based on 1) the percentage of low-income students in the district (the horizontal axis), and 2) the percentage of students in the District who scored above the 50th percentile in reading on the 2016 PARCC test (the vertical axis). Each diamond in Figure 6 represents one school district.
The trend line shows there is still a strong relationship between income and achievement, said Mr. Zavitkovsky. On average, as concentrations of poverty increase, achievement declines.
But the chart also shows that some school districts that have the same concentration of low-income students have significantly different achievement results.
For example, among the school districts that have 40% low-income students, some have only about 20% of their students scoring above the 50th percentile; others have about 75% scoring above the 50th percentile.
This data and the data compiled by Dr. Reardon in a nationwide study of more than 3,000 school districts show that the difference between the highest and lowest achieving school districts having identical percentages of low-income students typically amounts to two or three grade level equivalents, says Mr. Zavikovsky. He attributes this difference, in part, to school effectiveness.
“The good news from the variability of achievement among school districts with the same level of low-income students is that school effectiveness could reasonably increase achievement in most districts by between 0.75 and 2.0 grade-equivalents,” he says.
Regional Shifts in Low-Income/Achievement
The Report also analyzes how composite achievement changed for third- through eighth-graders between 2001 and 2016 in different regions of the State. Again, the measure of achievement used by the Report is the percentage of students scoring at or above the 50th percentile of students in the State. The Report found:
• Chicago: Low-income enrollment in Chicago stayed constant at about 85%, but reading achievement rose by 14 percentage points, and math by 18 points.
While some of this increase is due to shifts in black and Latino enrollment in Chicago, there have been increases in each racial and ethnic group. For example during the 16 year period, non-ELL low-income eighth-graders showed these increases in reading: 6 percentage points for black students, 21 points for Hispanic, and 14 points for white. For non-ELL non-low-income eighth-graders, the increases in reading were 13 points for black students, 17 points for Hispanic, and 4 points for white.
• Suburban Chicagoland: In 229 school districts in suburban Chicagoland, low-income enrollment rose by an average of 22 percentage points, but average achievement declined by an average of only 1 point in reading and 3 points in math. This was much less than would be expected in light of the significant increase in low-income students.
Viewed another way, in 2001 typical school districts in suburban Chicagoland that had 50% low-income enrollment had about 35% of their students scoring above the 50th percentile; in 2016, school districts in this region that had 50% low-income enrollment had about 50% of their students scoring above the 50th percentile.
“While individual districts all had their own unique profile, evidence for the region as a whole makes it pretty clear that increased school effectiveness is strongly influencing achievement in suburban Chicagoland,” said Mr. Zavitkovsky.
• Central and Southern Illinois (south of Interstate 80): In 421 school districts in central and southern Illinois, low-income enrollment rose by an average of 19 percentage points; average achievement fell by 9 percentage points in reading and 7 points in math. “Without fundamental improvements in school effectiveness, higher concentrations of low-income enrollments have been accompanied by steep declines in achievement,” said Mr. Zavitkovsky.
Figure 7 illustrates the average changes in reading achievement and in low-income enrollment for school districts in Chicago, suburban Chicagoland, and central and southern Illinois during the period 2001-2016.
Mr. Zavitkovsky says that school districts across the State have responded in different ways to jumps in low-income enrollment. In Chicago, where enrollment has remained constant at 85%, students have shown significant gains. In suburban Chicago, school districts have maintained achievement despite significant increases in low-income students. In contrast, most school districts in central and southern Illinois, which are predominantly white, have shown significant declines in achievement as the percentage of low income students increased.
“The data … make it clear that rising concentrations of low-income households have depressed achievement more consistently in predominantly white communities in central and southern Illinois than in other parts of the State.
“For the first time, concentrations of white poverty are beginning to appear around the state,” said Mr. Zavitkovsky. “While these concentrations are typically not as severe as those that have long been present in communities of color, we’re seeing declines in achievement that pretty much match what low-income communities of color have been experiencing for years.
“On the other hand, large scale gains in Chicago and suburban Chicagoland, and more isolated gains in individual districts across the State, offer good evidence that school effectiveness is making a difference and that demographics are not the only important influence on student learning outcomes.”
Why the Regional Differences?
When asked why school effectiveness was not improving in central and southern Illinois school districts, Mr. Zavitkovsky said, “It’s not entirely clear why school districts in central and southern Illinois didn’t show the same gains in school effectiveness during the period 2001 – 2016 that Chicago and most suburban Chicagoland districts did. In the Report, we speculated that some of the responsibility lies with how poorly ISAT test reports kept districts apprised of what was actually going on.” The low benchmarks to “meet standards” on the ISATs led school districts to believe they were doing well, when they were actually falling behind.
Mr. Zavitkovsky said another contributing factor may have been the widening disparities in per-pupil spending for instruction between upstate and downstate districts with average to high concentrations of low-income students.
“Compared with suburban Chicagoland, per pupil spending for instruction in most northern, central and southern Illinois districts was less equitable in 2016 than it was in 2001,” he said.
For example, the Report found that among the school districts in each region that had the highest concentrations of low-income students, per pupil spending was about the same as suburban Chicagoland in 2001, but was 8 to 17 percentage points below parity in 2016. Among the school districts in each region with mid-level concentrations of low-income students in 2001, school districts in northern, central and southern Illinois were all about 9 percentage points below parity in 2001. By 2016, however, they were 18, 24, and 30 points below parity respectively.
The Report found that regional differences in student achievement corresponded pretty closely with regional differences in per pupil spending for instruction. The Report added, though, “While that correspondence does not prove that the one caused the other, it raises powerful questions about the relationship between district resources and a district’s capacity to improve school effectiveness.”
“A key lesson from America’s experiment with No Child Left Behind is that state and federal policy have not been effective in strengthening school effectiveness at scale,” concludes the Report. “In the end, school effectiveness is a local challenge that cannot be successfully confronted without local determination, local ingenuity and local leadership.
“But statewide policies create the conditions in which schools and districts operate. Those conditions influence the likelihood that improvements in school effectiveness will or won’t gain traction at the local level.” The Report recommends three shifts in state policy that it says would boost the odds for growing school effectiveness at scale in every region of the State: test less and report results in ways that are more useful for parents, teachers, and other end-users; follow through on early advances in State support for principal leadership preparation and development; and fund schools equitably.
Paul Zavitkovsky is currently a member of the Illinois State Board of Education’s Assessment Review Committee and the Illinois P-20 Council’s Data, Assessment and Accountability Committee. He is a leadership coach and assessment specialist at the Urban Education Leadership Program of the University of Illinois at Chicago.
Steven Tozer, Ph.D. is a Professor of Education and Director of the Center for Urban Education Leadership at the University of Illinois at Chicago. Prof. Tozer is Co-Chair of the Illinois School Leader Advisory Council, and is co-author of Schools and Society (2012).