At the joint meeting of the District 65 and 202 School Boards on Feb. 25, David Figlio, Dean of the School of Education and Social Policy (SESP) at Northwestern University, and Lila Goldstein, Research Data Analyst Lead at SESP, gave the Boards an update on the development of what is now called the Northwestern-Evanston Education Research Alliance (NEERA) Persistence Compass Project. The project has been in the works for the last three years.
NEERA is a partnership between Northwestern University and School Districts 65 and 202.
The NEERA Persistence Compass is designed as a “diagnostic tool” specifically for School Districts 65 and 202, and it will give educators information about “the degree to which individual students are progressing to be prepared to be successful in college,” said Dr. Figlio in a memo presented to the Boards. It will “allow the Districts to focus attention on flagging students that most need support.”
Dr. Figlio said the diagnostic properties of the Persistence Compass will help the School Districts “to be as strategic as possible in their choice of programs and interventions” that they select for students. The Compass can also be used to track progress in preparing each student to be “successful in college.”
Developing the model followed a three step process: 1) define the desired post-secondary outcome for students; 2) identify ETHS graduates who achieved that post-secondary outcome; and 3) determine the aspects, or factors, of those students’ experiences at ETHS that can be used to predict whether current students will meet the desired post-secondary outcome.
How Success in College is Defined
The starting point of the model is to define the desired outcome for ETHS students. The NEERA team decided that graduation from a post-secondary institution is the desired outcome, but it decided to use an earlier post-secondary outcome that is predictive of college graduation as a proxy for graduation from college.
The proxy selected is “continuous enrollment for five or more semesters in any tier post-secondary program,” and it includes “shorter degree or certificate programs.”
As an example, if a student obtains an associate’s degree or a certification from Oakton Community College in two years, that would count as college persistence.
Because success is defined as persisting for five semesters in any tier college, including Tier III and two-year colleges, the model is not designed to measure what skills and experience a student needs to persist in more competitive Tier I and Tier II colleges such as DePaul University or the University of Illinois at Chicago (defined by ETHS as Tier I colleges), or Northern Illinois or Northeastern Illinois (defined by ETHS as Tier II colleges).
Also, because students may generally persist in college if they have a GPA of 2.0, the model is not designed to measure what is needed to earn a GPA of 2.5, or a 3.0. Due to grade inflation, the average college GPA in 2013 was 3.15, according to a recent study.
The Predictive Models
NEERA has identified ETHS students who have persisted to five semesters in any tier college, including two-year colleges. As the next step, NEERA is identifying aspects, or factors, of those students’ experiences at Districts 65 and 202 that will be used to predict whether current students will persist to five semesters in a post-secondary institution.
The NEERA team has developed a predictive model for eighth and ninth grades that it shared with District 65 and 202 staff. But the model and the factors identified in that model have not yet been presented to the School Boards or the public.
In a presentation last fall, Ms. Goldstein said examples of some possible factors were grades, attendance, standardized test scores, discipline and course-taking patterns.
When asked by District 65 Board member Candance Chow what factors might be included in addition to grades and course-taking patterns, Ms. Goldstein said they are able to take a closer look at a student’s experiences, such as how a student’s credits in 9th grade were distributed, what electives were taken, what levels of courses were taken, and what the grades were in each subject.
“We’re looking at in-school factors and things the District is measuring already that are sort of low-hanging fruit,” she said.
Dr. Figlio’s memo mentioned possible factors such as the number of electives taken in freshman year, and the interaction between the number of Fine Arts courses taken freshman year and the level of World Language courses taken.
“Of course we are looking at other variables both in high school and in middle school,” said Dr. Figlio. “The predictive tool will not have every variable that we examine, but a combination of variables. We are aiming for a model with high predictive power that does not oversimplify students’ experiences, but is still usable and understandable.”
He added, “We collectively ran a million regressions,” and the Northwestern team is using machine learning to try to figure out the very best combinations of variables that will predict persistence to five semesters.
NEERA is also developing thresholds that can be used to identify students who need supports and to identify areas where the Districts might consider modifying instruction or programs.
As an example, Dr. Figlio said, if students taking three electives persist at about the same rate as those taking four electives, setting a threshold at four rather than at three might not be helpful. Intervening to encourage a student to take a fourth elective would not yield a significantly different end result.
Dr. Figlio’s memo said the model will be “stable across racial and other demographic subsets of students.” In light of that Ms. Chow asked if NEERA would determine if certain factors or thresholds might be more effective for a subgroup that has been marginalized, and if so, whether those factors or thresholds would be a priority, even if they did not have a stable outcome for all demographic groups?
Dr. Figlio said the model would not identify different factors for different subgroups. He explained, “If you break up the performance model into subgroups, you get into really tiny sample sizes in a hurry, so you get into the possibility of massive measurement error that might lead us down into rabbit holes.”
Dr. Figlio added that the model would provide information that the Districts could use with other information they have to improve outcomes for particular subgroups.
Jonathan Baum, District 202 Board member, noted that the Persistence Compass was a diagnostic tool and not a prescriptive one. He asked, “If it’s not prescriptive, how can it be used to inform instruction and practice?”
Dr. Figlio said, “We need to have diagnostics to help school districts to make more informed decisions.” He said they viewed the model as “a flashlight tool” that can help narrow down a potential number of curricular changes or other changes to “a few different places that the Districts might really want to prioritize.”
Dr. Figlio added, “There’s a cycle of continuous improvement that educators are engaged in.” He said school administrators use their own best judgement, observations, ideas, and research to try out new curricular interventions, or other types of interventions. And then they evaluate the interventions, and either refine them or scrap them. The School Districts’ collaboration with Northwestern will help with each part of that process and accelerate each aspect of that process, he said.
“There’s lots and lots of Northwestern researchers who would love to partner with both School Districts and try out different types of interventions – for example, jointly develop interventions with practitioners,” said Dr. Figlio.
District 202 Board member Jude Laude asked, “Are there any non-academic factors included in the Compass that impact learning, like executive functioning?”
Dr. Figlio responded, “Right now, the collaborative work on the Compass itself is focused on the variables that are measured in School District administrative data.”
He added, “We all know that social and emotional learning, executive functioning skills and others are absolutely, extraordinarily important.” He said he hoped that the information provided through the Compass may inspire the School Districts to take a more in-depth look at some of those issues.
“It’s important for us all to keep in mind what the Compass can and cannot do,” said Dr. Figlio. “We all perceive the Compass as a relatively small part of a great suite of collective work that we’re doing trying to jointly facilitate.
“Nobody in this room or who is watching this Board meeting should put too much hope or too many eggs in the basket of the Persistence Compass. It’s an important tool that will help the School Districts to do more, faster and better, because it will help to supplement their judgement, but it’s only a tiny fraction of the joint work that we’re doing.”
Ms. Goldstein said NEERA is on track to identify factors and set thresholds for eighth and ninth grades by this coming summer. The next steps include creating and adjusting interfaces to enable the Districts to use the Persistence Compass, expanding the model to kindergarten through twelfth grades and updating the model based on more recent data and policies.