School Districts 202 and 65 are making progress toward defining what they deem is a successful college outcome for their students. Carrie Levy, Director of Research, Evaluation, and Assessment for District 202; Peter Godard, Chief Officer, Research, Accountability, and Data for District 65; and David Figlio, a professor at Northwestern University and Director of the Institute for Policy Research (IPR) at Northwestern; gave the progress report at a joint meeting of the School Boards on Oct. 24.
In the past, School District 202 used the college readiness benchmarks on the ACT test, as well as other metrics, to measure whether students were prepared to succeed in college. Students scoring at ACT’s benchmarks have a 50% chance of earning a B in freshman year of college. To assess whether students are on track to college readiness, School District 65 uses scores on the Measures of Academic Progress (MAP) test that are linked to the ACT’s college readiness benchmarks.
Under a proposed new approach, Districts 202 and 65 will agree on a college outcome that is short of – but predictive of – graduation. At the Oct. 24 meeting, the discussion focused on using persisting to either a third semester or a fifth semester of college as the college outcome. Once the college outcome is agreed upon, then the Districts, in collaboration with IPR, will identify “multiple measures” that predict whether their students will reach the agreed-upon college outcome.
At the low end, students may persist from one semester to the next in college if they have a Grade Point Average (GPA) of 2.0 or qualify for probation. The measures will thus be linked to that level of achievement, which is the borderline between passing and failing.
Determining a Post-Secondary ‘Outcome’
Administrators have decided not to use graduation from college as the “outcome,” but instead to use an earlier post-secondary outcome, such as persistence to a third semester in college or persistence to a fifth semester in college, that is predictive of ultimate graduation.
Dr. Figlio said there are two important arguments for using an outcome short of graduation. First, he said, the farther that a student is from high school graduation, the harder it is to argue that the student’s success in college is due to his or her high school education, as opposed to other intervening factors. Second, he said, students often graduate five or six years after high school, and if measures are identified for those students, the measures will be six years removed for high school seniors, 10 years removed for high school freshmen, and 11 or more years for District 65 students. “Think about how different the measures are today from 6 to 10 years ago, or from 11 years of more in District 65,” Dr. Figlio said.
The Districts are thus looking for an outcome short of graduation that is predictive of college graduation. To assist in making this determination, Dr. Levy said the team analyzed the college graduation rates of ETHS students who graduated from high school in 2008 and 2009. They determined the graduation rates of the students who started out in college, and of the students who persisted to a third semester of college, to a fourth semester of college, and to higher numbers of semesters, using data available from the National Student Clearinghouse.
For purposes of the analysis, the Districts defined college graduation as receiving a certificate or an associate degree from a community college, or a bachelor’s degree from a four-year college. The definition is thus broader than graduation from a four-year college.
The study found that of all ETHS students who started out in any type of college (including two-year and four-year), 68% went on to graduate. Of the students who persisted to a third semester of college, 80% went on to graduate. Of the students who persisted to a fifth semester, 89% went on to graduate.
The team also took a look at whether there was a difference in graduation rates depending on the type of college a student enrolled in. Dr. Levy said they categorized colleges based on the “college tiers” as defined in accordance with Barron’s Profile of American Colleges:
• Tier 1 includes the “most competitive” colleges (e.g., Northwestern University, Washington University, Carleton College, Macalester College), “highly competitive” colleges (e.g., University of Illinois/Urbana-Champaign, University of Wisconsin/Madison, University of Michigan, Lawrence University), and “very competitive” colleges (e.g., DePaul University, University of Iowa, University of Illinois/Chicago, Loyola University Chicago); 48% of ETHS graduates enrolled in Tier 1 college.
• Tier 2 includes “competitive” colleges (e.g., Northern Illinois, Southern Illinois University, Northeastern Illinois University, and University of Kansas); 17% of ETHS graduates enrolled in a Tier 2 college.
• Tier 3 includes “less competitive” or “non-competitive” colleges (e.g., Oakton Community College, Columbia College Chicago, College of Lake County, Robert Morris University); 23% of ETHS graduates enrolled in a Tier 3 college.
The study found that the graduation rate differed significantly depending on the tier of college that ETHS students enrolled in. Significantly, 88% of ETHS students who started out in Tier 1 colleges graduated; 59% of students who started out in Tier 2 colleges graduated; and only 32% of students who started out in Tier 3 colleges graduated.
Of the ETHS students who persisted to a third semester at a Tier 1 college, 92% graduated; compared to 68% at a Tier 2 college, and 44% at a Tier 3 college.
Of the ETHS students who persisted to a fifth semester at a Tier 1 college, 93 graduated, compared to 80% at a Tier 2 college, and 61% at a Tier 3 college.
Predicting graduation based on persistence to a certain number of semesters thus varies significantly depending on whether a student was enrolled in a Tier 1, Tier 2, or Tier 3 college. The differences are illustrated in the accompanying chart, which is based on data provided by the Districts.
Choosing the number of semesters to use as the outcome is a bit of “art” and a bit of “science,” said Dr. Figlio. “If you look at the three contiguous semesters of enrollment, that might be sufficient to have a very good predictor of ultimate graduation, but you could also have arguments for five semesters, especially if you’re interested in moderately competitive or less competitive groups.”
He added that the “art” part is “extremely important because it’s what the values are of the school district, what do you consider to be successful, all of those things are philosophical, educational, political decisions.”
District 202 Board member Jonathan Baum pointed out that only 44% of students who persisted to a third semester in a Tier 3 college went on to graduate, and that only 68% of students who persisted to a third semester in a Tier 2 college went on to graduate. “That’s not good enough,” he said.
Dr. Levy said the next step is that the Boards will choose the number of semesters that they think adequately predicts graduation from college. It is anticipated that the team will present a recommendation to the Boards at their joint meeting in February 2017.
Determining the Measures
Dr. Figlio said once the School Districts decide on the college outcome, the team will identify multiple measures for District 202 and multiple measures for District 65 that predict whether each District’s students are on track to achieving that outcome.
“By having District 65 and 202 both agreeing to the same long-term measure, then you don’t have to worry about finding something in 65 that corresponds to something in 202. You’re using data in 202 to predict the downstream measures. You’re using data in 65 to predict the downstream measures. And the two of them automatically align,” Dr. Figlio said.
As an example, Dr. Figlio said District 202 might identify measures such as an ACT or SAT score, taking particular courses, success in the courses taken, grades, taking AP courses and success in those courses. He added that different weights might be given to different measures.
District 65 might use scores on the MAP test, attendance rates and truancy, behavioral factors, grades in middle school, and other factors. He said these could also be weighted.
Dr. Figlio said it does not matter if the measures used at District 202 differ from those used at District 65 because they would each be aligned with the same college outcome.
Paul Goren, Superintendent of District 65, cautioned, though, that District 65 would be backmapping from a college outcome to identify measures as far back as third grade. The lapse of time could be more than 10 years. The District will need the best analysts to assist in the backmapping, he said.
Dr. Figlio said, “We need more analytic support in the Districts and at Northwestern to help facilitate the information flow and to be sure we can do the analyses.” He said this requires financial support to hire staff capable of providing the analytic support. Evanston residents Chuck Lewis and Penny Sebring have made a generous donation to jump-start the process, Dr. Figlio said, and the School Districts and IPR were finalists for a large grant from a major national philanthropist.
District 202 Board member Monique Parsons said, “What I’m gathering is the creation of an educational pathway of students in Evanston to and possibly through college. We’re going all the way back and that will help students be successful not only reaching college, but getting through. That’s powerful.”
“Persistence” in Terms of Academic Success
The data available through the National Student Clearinghouse does not provide information on the courses a student has taken or on grades earned in college.
A student, however, may persist from one semester in college to the next if he or she has an overall college grade point average of 2.0, or by having less than a 2.0 grade point average if he or she qualifies for academic probation. This is the case at every college checked by the RoundTable, including at DePaul University, Loyola University Chicago, Oakton Community College, Northeastern Illinois, Northern Illinois, Northwestern University, Roosevelt University, Southern Illinois, University of Chicago, University of Illinois at Urbana/Champaign (with some variations), and the University of Illinois at Chicago.
ETHS is thus proposing to link its “multiple measures” to a GPA of 2.0 or less, which is the borderline between passing and failing in college. One recent study found that 77% of the grades given in four-year colleges are As and Bs. An updated study found that by 2013, the average college student had a GPA of 3.15. “Where A is Ordinary: The Evolution of American College and University Grading 1940-2009” (2012) by Stuart Rojstaczer and Christopher Healy; and “GradeInflation.com” (Updated, March 2016).
Another factor is that the proposed outcome is not persistence in a four-year college, but persistence in either a four-year college or a community college. As such, the measures may be significantly influenced by the academic characteristics of students who persist to the chosen number of semesters in a community college.
The Illinois State Board of Education reports that 55% of ETHS graduates who enroll in a community college are required to take a remedial course.