Taking stock: How the pandemic has impacted student achievement at School District 65

The pandemic has taken a social and emotional toll on students, parents, teachers and administrators, and it has also had an impact on student learning. This article takes an initial look at how District 65 students are doing academically, two years into the pandemic, using two different data sets.  

First, administrators presented a report (the Joint Report) analyzing the progress being made on a Joint Literacy Goal to the School Boards at their Feb. 23 joint Board meeting. The goal, adopted by the boards in January 2014, provides:  “District 65 and District 202 will ensure that all students are proficient readers and college and career ready by the time they reach 12th grade.” The report shows that high percentages of Black and Latinx students are not proficient in reading when they graduate from District 65 and enter Evanston Township High School.

The authors of the report, Stacy Beardsley, Assistant Superintendent of Curriculum and Instruction at District 65, and Pete Bavis, Assistant Superintendent of District 202, recommended that the boards revise the Joint Literacy Goal. All members of the boards who spoke at the meeting supported the recommendation and added wide-ranging suggestions. Some urged that the districts partner with the community to address barriers caused by economic hardship. Others urged that the districts take a closer look at racism in the schools. Still others urged administrators to zero in on what programs were working and to replicate those programs in the schools.

Second, last week, in response to an Freedom of Information Act request, the RoundTable obtained the results of District 65 students on the Measures of Academic Progress (MAP) test given on five different occasions in the last several years. The trends are generally similar to what researchers have found at the national level, that achievement has slipped more in math than in reading during the pandemic. The biggest declines are for Black and Latinx students in math.

I. Progress in meeting the joint literacy goal

Setting proficiency at a point below grade level

How to define “proficiency” and to measure progress in meeting the Joint Literacy Goal consistently across the K-12 span has been a challenge because the districts have used different tests. ETHS currently uses the STAR test while District 65 uses the MAP test.

For many years now, ETHS has given the STAR test in September to incoming ninth graders, and it selected a grade equivalent (GE) score of 8.3 on that test to reflect “proficiency” in reading for an incoming ninth grader.

This is not a high bar. A GE score of 8.3 means the incoming ninth grader has the literacy skills of a “typical” student in the third month of eighth grade, according to information provided by STAR.

Ninth graders scoring below a GE of 8.3 are “not reading at grade level,” Scott Bramley, Associate Principal for Instruction and Literacy at ETHS, said in a Jan. 10, 2019, memo.

Nonetheless, for purposes of measuring progress in meeting the Joint Literacy Goal, administrators of Districts 65 and 202 decided to use that score – the GE score of 8.3 – as their benchmark for “proficiency” for students entering ETHS. They further decided to identify a score on the spring eighth-grade MAP test in reading that aligned with that level of proficiency.

In a joint study completed several years ago, administrators determined that a RIT score of 227 on the reading MAP test given in the spring of eighth grade aligned with a GE score of 8.3 on a STAR test given in September of ninth grade.

The districts thus agreed that District 65 would use a RIT score of 227 on the eighth-grade spring MAP test in reading as its benchmark for proficiency in reading. Students with scores at or above 227 would be viewed as proficient. Students with scores below that level would be viewed as not proficient.

“There is a high degree of consistency” between the two measures, said Bavis. Administrators have conducted follow up studies that confirmed their findings.

After identifying the MAP score for 8th grade, District 65 identified scores that indicate proficiency for grades 3 – 7. *

Percent of students meeting the proficiency threshold

The Joint Report on the Joint Literacy Goal shows the percentages of third- through eighth-graders, by subgroup, who met the proficiency thresholds in reading on the spring 2021 MAP test. That data is shown in the chart below.

The chart shows that higher percentages of eighth-graders are proficient in reading than third-graders, and that there is a jump in the percentages at seventh and eighth grades. But even so, at eighth grade only 35% of Black students are proficient in reading, compared to 47% of Latinx students, and 85% of white students.

A high percentage of Black and Latinx students in District 65 are from low-income households. The beige bar shows the percentage of low-income students who are proficient in reading.

Thus, 65% of Black students, 55% of Latinx students, and 15% of white students are not proficient in reading when they graduate from District 65. The districts reported the median RIT score of students who failed to meet proficiency. The percentile ranks of the median RIT scores are: Black students – 32nd percentile; Latinx students – 35th percentile; white students – 46th percentile, according to NWEA’s 2020 norm study.  

“These patterns largely remained consistent, and we continue to see an unacceptable opportunity gap by race,” Beardsley said.

The Joint Report does not present data showing the percentage of ninth-graders at ETHS who met the proficiency threshold on the STAR test in the fall of 2021. But it does present data showing the percentage of students who met both 1) the proficiency threshold in reading on MAP at the end of eighth grade, and 2) the proficiency threshold in reading on STAR as incoming ninth-graders at ETHS. And it reported the percentage of students who failed to meet the thresholds on both tests. The chart below illustrates that data for the ETHS class of 2015 (i.e., the incoming ninth graders in the fall of 2021).

Significantly, 51% of Black, 41% of Latinx, and 10% of white students failed to meet proficiency on either of the two tests.   

The Joint Report did not provide data, by subgroup, for previous years, but the data is available from prior years’ reports. The charts below show the percentage of third-graders and eighth-graders, by subgroup, who met proficiency thresholds in reading on the spring MAP tests in 2017, 2018, 2019 and 2021. District 65 did not administer MAP tests in the Spring of 2020.

For third-graders, there was a drop between the spring of 2019 (pre-pandemic) and 2021, which might be attributable to the pandemic, but there were fluctuations in prior years as well.

For eighth-graders, there was a drop for Black students in 2021, after good increases in 2018 and 2019. Latinx students and white students showed a relatively small increase between 2019 and 2021.

II. Results on five MAP tests

In response to an Freedom of Information Act request, District 65 provided data showing the percentage of students who scored above the 50th percentile on five MAP tests administered in the winter of school year 2019-20 (SY’20), the fall, winter and spring of SY’21, and the fall of SY’22.

The charts below show the percentage of students in grades three through eight, by subgroup, who scored at or above the 50th percentile in reading and math on the MAP tests indicated. In each case, the scores correlating with the 50th percentile are lower than the scores indicating proficiency in reading for purposes of evaluating progress in meeting the Joint Literacy Goal and lower than the scores that indicate being on track to college readiness in reading and math.

The pandemic hit in March 2020, and District 65’s schools were closed for in-person learning through about mid-February 2021. The only pre-pandemic test is the one administered in winter SY’20.

For reading, the data shows that white students scored about the same on all tests. Black students showed a decline, and Latinx students showed an increase. Comparisons of the fall scores to spring or winter scores may not be valid because results in the fall are often a few points lower due to summer learning loss.

For math, there is a more pronounced downward trend.

“Early research indicates that the initial impact [of the pandemic] is greater on younger students and in the area of math,” says Beardsley. “Our District data reflected this trend coming into the 2021-2022 school year.”

The charts below show the percentage of students who met growth targets identified by NWEA in reading and in math on the MAP tests indicated. NWEA’s growth targets represent the average growth of students at the same grade who started out at the same level of achievement.  

 In reading, the percentage of white students meeting targeted growth has declined. On the SY’22 fall test, 51% of Black students, 55% of Latinx students, and 52% of white students met targeted growth.

In math, the percentage of students meeting targeted growth declined for each subgroup. On the SY’22 fall test, only 31% of Black and Latinx students and 41% of white students met targeted growth.

Statistically, 50% of the students are expected to meet targeted growth. It is important to recognize, however, that even if a student meets targeted growth year after year, it does not mean that the student will be on track to college readiness at the end of eighth grade.

Meeting targeted growth aligns more with maintaining the status quo – achieving average growth, not accelerated growth. NWEA says in its 2020 norm study that if a school district is interested in accelerating students’ growth or closing achievement gaps, it can set customized accelerated growth targets. District 65 has not done so.

At District 65, it appears that the pandemic has had a more significant on student achievement in math than in reading. This is consistent with findings made by NWEA researchers in a report issued in December 2022. The NWEA report found that after most schools were closed for in-person learning starting in March 2020, students began the 2020-21 school year with reading achievement “roughly comparable to a typical year, but that math achievement was between 5 to 10 percentile points lower, with students in earlier grades experiencing larger declines.”

At the start of the 2021-22 school year, the NWEA report found that achievement in reading was down 3 to 7 percentile points and in math it was down 9 to 11 percentile points. “In other words, we find continued evidence of significant unfinished learning.”  
The NWEA report also found that the percentage of students meeting targeted growth in reading between fall 2019 and fall 2021 neared pre-pandemic growth rates; but that math gains were well below average. “This finding suggests that school-related disruptions continue to have a more significant impact on students’ acquisition of math skills/content compared to reading and highlights the need for continued focus on supporting the development of math skills.”

The NWEA report also said the largest declines in achievement during the pandemic were observed for Black and Latinx students.

Revising the joint literacy goal

“While developing and measuring the joint literacy goal is important, it’s also important to remember that the students in front of us today have not had a stable school experience,” said Bavis. “That is why it is critically important to revisit and revise the Joint Literacy Goal to reflect our continued commitment to reading proficiency, college and career readiness, and to acknowledge that the work going forward needs to be responsive to the students that we are serving today.”

The Joint Report recommended revising the Joint Literacy Goal, taking into account the following:

● How might the goal be revised to acknowledge our current context?

● How might the goal be revised to align with each districts’ commitment to race and equity?

● How has college and career readiness evolved in 8 years?

● What are the research-based indicators of success?

● Where do we want to be in one year? In five years?

● What does each district commit to investing in?

Board members who spoke at the Feb. 22 meeting concurred with the recommendation, and added some things they wanted administrators to consider.

Patricia Maunsell, a member of the District 202 School Board since 2017, said, “We’re still seeing those huge gaps. And it’s nothing any of us are happy with. …  I feel like this is an opportunity to really think about things differently in terms of the literacy.”

Pat Savage-Williams, a member of the District 202 Board since 2013 and President for most of those years and is now the President, said she liked the questions, “Where do we want to be in one year? Where do we want to be in five years?  I really like looking at this goal in a way that we will continue to keep ourselves accountable and look at what we’ve done and what progress we’ve made.” She stressed there was urgency to revise the goal.

Monique Parsons, a member of District 202 School Board since 2015, said, “I appreciate you guys giving us this report and not dwelling on the data because since I’ve been on the board, it hasn’t changed much. I mean, let’s be honest, COVID has impacted a lot … But we discussed the literacy goal or the lack of literacy of our children not reading at grade level before COVID hit.”

Parsons likewise focused on the urgency. “And with that urgency, my expectation is that the considerations are bold, not just creative. But how are we going to do school differently? Because we can’t keep doing what we’ve been doing for the past eight years, because it hasn’t worked.” She said she hoped administrators would bring “different voices to the table ,” “that we throw heavy supports into our community where students are struggling the most,” and there be “accountability, of making sure that our students are reading, so that they can succeed in life.”

Sergio Hernandez, a member of the District 65 Board since 2017, said, “We need to first ensure that children and their families have the comforts of a warm home, that they’re well fed, that their social emotional needs are met and they are ready to come to us so that they can be educated and succeed.”

He said a large percentage of people in the community face “barriers to economic and educational opportunity.” He said the districts needed to work with community partners, to “put wrap-around services around not just the child, but their family as well. … So, we need to take this comprehensive community approach to education, … and that means the city and Northwestern, that means social service agencies, all of us need to be at the table.”

Anya Tanyavutti, who has served as Vice-President or President of the School Board since April 2017 and is now the President, said, “Children are mandated by law to come to school every day, in a place that is hostile to their presence, that is neglectful to their presence, are not having a happy experience. They’re not having an experience of feeling loved, safe and cared for. And that kind of pain is not acceptable for the most vulnerable people in our community.

 “And we need to be accountable as institutions, for not just thinking about outcomes, but the experience of children.  …. So, I’m not only concerned about how our students are reading in high school, I’m concerned with are they showing up and feeling a sense of belonging? Are they showing up with the social emotional skills to really benefit from all the richness that school has to offer? And if not, where did we miss? How can we course correct, what can we do more of or differently, or more collaboratively to make sure that’s happening? And I hope that’s the kind of vision that’s captured in the portrait of a graduate.”

Soo La Kim, a member of the District 65 School Board since 2019, asked how District 65 could make the culture and climate shifts in the district’s building “so that every child feels that they can learn, that everyone around them believes that they can learn. …  So, I think we can’t divorce the academics from that context.

Administrators plan to a revised Joint Literacy Goal to the School Boards at their joint meeting in the fall.

Footnote

*As it happens, a score of 227 in reading on the spring eighth-grade test is the score that was identified as the minimum score indicating being on track to college readiness by the Northwest Evaluation Association, the owner of the MAP test, in a 2015 study. NWEA also identified college ready benchmark scores for grades 5, 6 and 7. District 65 back mapped to identify college ready scores for grades 3 and 4. These scores are being used to identify the level of proficiency needed to meet the Joint Literacy Goal.

It is important to recognize, though, that District 65’s joint study with ETHS determined that an eighth-grade score of 227 in reading on the Spring MAP test is aligned with a GE score of 8.3 on the STAR test given to incoming freshmen.

From the high school’s perspective, a GE score of 8.3 is the borderline between reading at grade level and reading below grade level, and needing supports. From STAR’s perspective it means the incoming freshman is reading at the level of a student in the third month of eighth grade.

So, an eighth grader at District 65 who scores a 227 in reading on the Spring MAP test is led to believe they are scoring above the national average and that they are on track to college readiness.

Once they set foot into the high school, though, there is a different story. That student is on the borderline of performing below grade level in reading from ETHS’s perspective, or perhaps six months below grade level from STAR’s perspective.

The RoundTable has published articles showing that MAP normative results portray that students are faring much better than other tests.

MAP Normative Data Shows Significantly Higher Achievement Levels for CPS Students than NAEP or PARCC

NWEA’s Expected Growth Targets Are Not Aligned With the Growth Students Need to Be on Track to College Readiness by 8th Grade