When Caroline Hoxby published her report on How New York City’s Charter
Schools Affect Achievement in September, its release was choreographed for maximum political effect. Within a matter of days, the Bloomberg campaign issued a call for the unfettered and unregulated expansion of charter schools across New York, citing the Hoxby report. The National Alliance for Public Charter Schools and the New York Charter School Association were proclaiming that Hoxby’s analysis proved wrong the growing body of solid research showing that the academic performance of charter schools is mixed, including this authoritative Stanford study — with their own calls for the complete deregulation of charter schools in New York.
A crucial component of this choreography was the fact that the Hoxby report was issued without any peer review, a break with the commonly accepted standards for the publication of serious academic research. The absence of peer review was crucial because it meant that reporters, who do not have training in rigorous academic research based on complex statistical modeling, were in no position to question the reports’ methods and thus its conclusions.
Now we have an analysis of the Hoxby report by Think Tank Review, a joint project of the Education and Public Interest Center at the School Education of the University of Colorado-Boulder and the Education Policy Research Unit at Arizona State University. Reviewer Sean Reardon, Associate Professor of Education and Sociology at Stanford, finds a number of major flaws in the methods employed by Hoxby. Specifically:
• The report relies on an inappropriate set of statistical models to analyze the data.
• The report includes claims regarding the cumulative effects of attending a New York City charter school from kindergarten through eighth grade that are based on an inappropriate extrapolation.
• The report does not include adequately detailed information in some areas to allow a reader to fully assess its methods, results, or generalizability.
• The report uses a criterion for statistical significance that is weaker than that conventionally used in social science research.
• The report describes the variation in charter school effects across schools in a way that may distort the true distribution of effects by omitting many ineffective charter schools from the distribution.
As a result of the flaws in the report’s statistical analysis, [the Hoxby report] likely overstates the effects of New York City charter schools on students’ cumulative achievement, though it is not possible—given the information missing from the report—to precisely quantify the extent of overestimation.
Over at Gotham Schools, Aaron Pallas explains the import of these flaws in layperson’s terms — or as “lay” as you can get when discussing the world of statistics.