On May 21, the District 65 and 202 School Boards were provided an update on a project in which the Northwestern-Evanston Education Research Alliance (NEERA) is working to identify multiple characteristics of District 65 and ETHS students that can be used to predict college persistence. NEERA will first identify the characteristics – or on track indicators – for eighth and ninth grades, and then expand to other grade levels.
NEERA is a partnership between School Districts 65 and 202 and Northwestern University (NU). David Figlio, Dean of the School of Education and Social Policy (SESP) at Northwestern University, and Lila Goldstein, Research Data Analyst Lead at SESP, joined by four other members of the NU team working on the project, made the presentation to a joint School Board meeting held at ETHS. Dr. Figlio said many other researchers at NU are involved in the project.
The Lewis-Sebring Family Foundation and the Spencer Foundation have made significant grants to help fund the project.
“This is such important exciting work because it changes the paradigm of what research about education should be about,” said District 65 Superintendent Paul Goren. “Typically research is done on school systems. This opportunity, this partnership is shifting that so the issues and concerns that are being addressed are issues of problems and practices generated by School Districts 65 and 202.
“Then in partnership with our esteemed researchers we really get to dig deeper into what some of the potential data are, what the solutions are, and it really builds a whole new mode of doing research with and for school systems, as opposed to on school systems.”
An Overview/Some Background
Eric Witherspoon, Superintendent of District 202, said, “We definitely know that it’s important to have our students be college and career ready.” He added, though, that it is not only important to help students gain admittance to college, but to get them through college.
Dr. Witherspoon said what NEERA is doing is it is taking a look at ETHS graduates who have persisted to five consecutive semesters in college – which is used as a proxy for college graduation, and then identifying the traits of those students in ninth and eighth grades that can then be used to predict whether ninth and eighth graders are on track to persisting in college. The traits would include such things as grades, courses taken, and attendance rates.
Then, he said, if the Districts determined that students were not on-track, they could provide them additional supports and additional rigor so that they will be prepared to persist and graduate from college.
In constructing the model, NEERA has decided to use an earlier post-secondary outcome that is predictive of college graduation, rather than graduation. In October 2016, Dr. Figlio said there were two important reasons for this. First, he said, the further that a student is from high school graduation, the harder it is to argue that the student’s success in college is due to his or her high school education as opposed to intervening factors. Second, he said, the multiple measures identified to predict college graduation would be two years further removed if college graduation was used, rather than persistence to five semesters.
In deciding to use persistence to five semesters as the outcome, District 202 analyzed college graduation rates of ETHS students, in relation to the number of semesters that they persisted to in college and in relation to whether they enrolled in a Tier 1, Tier 2 or Tier 3 college (see sidebar for description of tiers). An ETHS March 9, 2017 memo reported:
• At day one: 88% of students who enrolled in a Tier 1 college went on to graduate, compared to 58% who enrolled in a Tier 2 college, and only 37% who enrolled in a Tier 3 college.
• At five semesters: 94% of the students who were continuously enrolled for five semesters in a Tier 1 college went on to graduate, compared to 82% in a Tier 2 college, and 80% in a Tier 3 college.
Based on the analysis, ETHS decided to use persistence to five semesters as the outcome.
Dr. Figlio said he wants “bring together our [NU’s] research mission, our teaching mission, and our service mission in the service of our hometown.
“The goal here is we’re interested in looking to see what happens over the course of a young person’s life … Can we predict at various times in people’s lives, maybe first grade, second grade, third grade all the way up to twelfth grade, what is predicting college success. And again, success in this case is we’re looking at attending and persisting in college.
“Our goal is to carry out research that is informed, that is collaborative with the experts in Evanston schools,” said Dr. Figlio. “Our long term aspiration is to develop this further with other organizations, such as McGaw YMCA, and other leading organizations in Evanston as well.”
Lila Goldstein, Research Data Analyst Lead at SESP, provided information concerning the ninth-grade classes at ETHS: Overall, 11% moved out of the District. Of those that remained, 96% graduated from ETHS, and 4% did not. Of the students who graduated from ETHS, about 85% enrolled in college. Of those, 64% persisted to a fifth semester in a four year college or completed a two-year program. The above chart prepared by NEERA illustrates this data by race and ethnicity.
Of the students who persisted to a fifth semester in college, 89.7% went on to graduate, according to a Mach 9, 2017 memo prepared by ETHS.
Closer Look at the Model
Ms. Goldstein laid out the approach in more detail, talking about what college persistence means, possible factors to use to measure whether students are on track to college persistence, and the need to identify thresholds that will trigger interventions.
The Desired Outcome
NEERA has agreed to use persistence to a fifth semester in a four-year college, including Tier 1, Tier 2 and Tier 3 colleges, as “college persistence.”
In addition, Ms. Goldstein said, “We are counting completion of shorter programs in those five semesters as college persistence as well.”
As an example, if a student obtains an associate’s degree or a certification from Oakton Community College within the five semester period, that would be counted as college persistence.
By including all three tiers of four-year colleges as well as two-year colleges in the mix, NEERA will not be identifying the characteristics needed to gain admittance to and persist in Tier 1 or Tier 2 colleges.
Ms. Goldstein did say, though, that NEERA has discussed “continuing to use our college persistence measure, but tying that in with college type or college tier to see whether we can predict students’ success within different kinds of college environments.”
Identifying the Factors That Are Predictive of Persistence
NEERA has identified three cohorts of students who graduated in 2012, 2013 or 2014, and who have persisted to five semesters in college. The next step is to determine multiple characteristics of those students in ninth grade that will predict persistence. NEERA is doing the same thing for one eighth-grade cohort at District 65 who graduated from ETHS in 2014.
The goal, Ms. Goldstein said, is, “We are looking for actionable, trackable factors” – factors that correlate with college persistence, and things that the Districts have the capacity to collect and maintain, such as grades and attendance records.
The actionable factors “basically boil down to about six things in a student’s academic career that we end up looking at,” said Ms. Goldstein. They are:
• Grades, such as an “A” or a “B”
• Progression, such as whether a student was held back, or whether a student was young for his or her class
• Course Taking Patterns, such as whether a student took Algebra or Geometry in freshman year at ETHS
• Standardized Test Scores. At District 65, they would look at MAP scores, and at ETHS, ACT or SAT scores.
• Attendance Rates, such as how many days did a student miss
• Disciplinary Incidents.
“The goal is to take those factors and figure out how to combine them to give us probabilities of persistence,” said Ms. Goldstein. Using multiple factors in combination, she said, will give a “more nuanced picture” or “more subtle information” about where students end up persisting.
Ms. Goldstein gave some examples based on preliminary work. She said that grades in specific courses have different levels of predictive power. For example a chart showed that ninth graders who had an “F” in English or Math were a little less likely to persist in college than students who had an “F” in social studies or English. The chart also shows, though, that students who do not have an F in any one of four courses have about a 60% probability of persisting in college.
Another chart combines two factors and shows that ninth graders who have higher than a 2.0 GPA in English and fewer than 10 absences have almost a 70% chance of persisting in college, By contrast, ninth graders who have lower than a 2.0 GPA in English and 10 or more absences had a little less than a 50% chance of persisting in college.
Another chart combines three factors: a grade in English, a grade in math, and whether or not a student met the college readiness benchmarks on the MAP test in eighth grade. The chart is reproduced below. The chart suggests that English grades are more predictive than math scores. In addition, Ms. Goldstein said, “The grades are tracking more with college persistence than meeting or not meeting college readiness benchmarks in eighth grade.”
The RoundTable asked the NEERA team about the data showing that students in the sample who did not obtain an F in any one of four subjects had a 60% chance of persisting in college, and suggested the high predictive power of not getting an F could be due to the fact that the students in the sample had also met ETHS’s graduation criteria and about 70% of the students in the sample met the admissions criteria of a Tier 1 or Tier 2 college. The RoundTable asked how the NEERA team would ensure that the factors selected to predict college success are in fact relevant and have predictive power standing alone.
Dr. Figlio said “The point of this slide is not to focus on whether students have Fs or not, but to show how measuring different aspects of students’ performance can give us different information.”
Dr. Figlio told the RoundTable they are testing the correlation of each of the six factors (e.g., grades, courses taken, attendance rates, etc.) standing alone and in combination with one another, with college persistence.
“As we build the model from these data, we are taking, in essence, a two step approach,” he said. “The first step is to estimate many thousands of logistic regressions relating college persistence with different combinations and permutations of the variables, and then to let the data speak – which of these various iterations together best predicts college persistence?
“Specifically, we are evaluating each of these iterations with Bayesian Information Criteria to narrow down these thousands of options to a “top ten” list of highest-prediction-power combinations of available data. The second step then involves choosing amongst these “top ten” list members, returning to our original selection criteria: What are the easiest data to collect and use? Which combination will make sense to people?”
He added that each of the factors in the final formula is going to have “considerable independent predictive power standing alone.” But he added, “[W]hat we care most about is the amount of collective predictive power that the variables have standing together and in interaction with one another.”
Ms. Goldstein said, “We’re making sure that anything we’re picking up is predictive of college persistence no matter a student’s race, no matter a student’s sex, no matter a student’s family income, birth country or home language.”
Identifying Thresholds to Intervene
Another step is to identify thresholds “for flagging students for support,” said Ms. Goldstein. She said NEERA could set a threshold for a GPA, and “figure out if we’d want to intervene with a student and give supports to students who have a 3.0 or a 2.0.”
Dr. Figlio said the plan is also to determine the type of interventions that might help a student who is not meeting the on-track indicators. As an example, he said, “Suppose we find out that in sixth grade there are certain indicators that suggest that a student is falling off of track to be prepared to be successful in college. … What are things that we might be able to do at that particular place that might be able to make sure that by seventh grade the person is achieving the indicators that are back on track.”
“That’s where you want to get the subject area experts at District 65 with the subject area experts at Northwestern together with evaluation experts, using data to develop … praticeable potential solutions that might be implemented in the school system to see whether it actually makes a difference.
“That’s the type of collaboration we have in mind,” he said.
District 202 Board member Jonathan Baum said he was “grateful” for everyone’s work and the foundations’ support, but frustrated it was taking so long.
Dr. Figlio said a lot has been done, including cleaning up and harmonizing the data between the two Districts, hiring staff, and finalizing a data-sharing agreement.
“We are getting to the place where all of that prior investment that was necessary before you could accomplish anything has been made.”
He added they were able to do in one year with funding from the Spencer Foundation and the Lewis Sebring Foundation what would have taken five years to do without the funding. He added that “We fully expect by this time next year to have “something that could be very actionable and practiceable.”
District 202 Board member Jude Law asked if the team was looking at factors that occur on the college campus that are important for college success. He mentioned access to financial aid, building a network of supports on the campus, learning about academic supports available at the college, culture shock when people do not look like you on the campus.
Dr. Figlio said, “It’s important to think about what can and can’t be done within this particular project. What you’re describing, of course, are absolutely essential questions that higher education researchers are grappling with, including many at Northwestern. This particular collaboration with District 65 and ETHS is not going to address those particular questions because the primary point of this collaboration between District 65, ETHS and Northwestern is college persistence which is a stand-in for are students in twelfth grade prepared to be successful in college.”
Dr. Figlio added, “One thing that could be extremely important down the road is if we could identify what are the attributes of the schools, colleges and universities that lead to persistence that might help us with the college advisory model at ETHS.”
Some high schools and after-school programs in Chicago, and some Evanston programs track how Black and Hispanic students do at various colleges, and they use that information in advising students in the college selection process.
District 65 Board member Joey Hailpern asked if there would be a way to identify indicators for kindergarten and early childhood.
Dr. Figlio said, “One of the things that’s very beneficial to this approach is the notion of being able to come up with cascading on-track indicators. The ultimate goal is to be able to come up with indicators for the early grades” that are actually synchronized with the indicators in the later elementary grades and the middle school grades and into high school and into twelfth grade. The aspiration is to be able to take it all the way back to the early grades.
District 65 Board member Sergio Hernandez said, “I see these indicators being able to be backtracked to early childhood education. All of these measures give us a broader view of childhood development. For me, it’s really exciting.”
Mr. Hailpern suggested that teachers should be collecting data about what interventions students were receiving and what experiences children were receiving, so these would be documented and could be used later on.
District 65 School Board President Suni Kartha said, “We don’t want to underestimate the importance of what we can learn from these measures, but we have a lot of other information about what would be indicators of success.” She added that the District was providing interventions and students were receiving enrichment activities, and she asked if these would be tracked.
One member of NU’s team said, “One of the things we launched this year with District 65 was an initiative to document all the out-of-school opportunities and participation patterns of students.” She said they plan to expand the initiative to ETHS.
Dr. Figlio said that many variables could be identified that predict college persistence. He said, though, “For an on-track indicator, you have to be parsimonious. But that doesn’t mean that’s the only thing that matters. … Just because they’re things we care about doesn’t mean they should fit into the on-track indicator.”
He said over time, some factors might be replaced by other factors.