A research team that studied the test results of students in the state’s largest voucher program say a far more rigorous evaluation of the program and its outcomes for students is needed.
Education researchers at North Carolina State University and the Friday Institute this week released the results of an unpublished evaluation of the Opportunity Scholarship, a state-funded voucher that helps low-income students attend private schools. The research came out with positive results for the voucher recipients who participated in the study – but the authors say those results come with many caveats.
The roughly 250 voucher recipients who participated in the study scored significantly higher across math, reading and language arts on a nationally-normed standardized test compared to similar public school students who took the same test. However, the study states those results cannot be generalized.
“Can you look at our results – which are large, positive, across all three subjects examined – and then say, ‘Okay, book closed, we know that kids getting a voucher in North Carolina are doing great'?” posed researcher Anna Egalite, of North Carolina State University, “We caution against jumping to that conclusion."
Egalite and her colleague Trip Stallings of the Friday Institute can’t say with confidence that those test results are representative of the typical voucher student’s experience. Details in the Opportunity Scholarship statute prevented them from getting a larger, representative sample of participants in order to conduct a more informative study of the program’s effects.
“We would love if we could do that in North Carolina, because it would allow us to say, ‘This program works or it doesn’t,” Egalite said. “Right now, we can say ‘This sample [of students] we have, in this way that we’ve studied them, is associated with higher performance, and there’s just so many asterisks to that.'”
Do students do better in math and reading as a result? Maybe. - NC State Researcher Anna Egalite
Obstacles to a More Rigorous Study
The law that created the Opportunity Scholarship called for a formal evaluation of the program’s causal effects on student performance – a high bar for experimental research. Egalite and Stallings say this is not that study. Instead, they hope future research can take on that task, but first researchers would need access to a random, representative sample of voucher recipients.
One difficulty is that voucher recipients and public school students are not required to take the same standardized tests. The voucher program also does not offer recipients any incentive for participating in such a study. Egalite and Stallings recruited volunteers with the help of public school officials and a school voucher advocate, Parents for Educational Freedom in North Carolina. That recruitment, the paper notes, could introduce bias into the selection of participants.
More than half of the private schools that participated in the study were Catholic schools, while they represent a small minority of voucher-receiving schools. The study’s participating private schools were on average majority-white, while the participating public schools on average had majority-minority populations.
The researchers took measures to match up the private school students with public school students who had similar incomes, past test scores and lived in the same area. But in order to make a true “apples-to-apples” comparison between voucher recipients and their public school counterparts, Stallings says you’d also have to account for the desire to have a voucher. That would mean comparing a set of students who got vouchers and moved to private schools against students who applied for and were eligible for a voucher, but did not get one. Currently, the Opportunity Scholarship goes to nearly every eligible student coming from a public school who applies and accepts one.
“At the end of the day, the North Carolina Opportunity Scholarship – it’s still a black box. There’s still much that we don’t know," Egalite said. "And not just of the nuances, but of the big questions: Do students do better in math and reading as a result? Maybe.”
Egalite says she took a professional risk to pursue research that may not be accepted in top peer-reviewed journals in her field. The research team viewed the study as a public service, a means for showing lawmakers what it would take to comprehensively review the program.
“There are many in our state who think there isn’t sufficient accountability in the program already, so we hope this is a demonstration of [how] this is our best effort to evaluate the program given how it’s currently designed,” Egalite said, “but I think we can do better.”
