Measuring School Performance

One of the realities that teachers and others who work in schools know from practical experience is that there is considerable variance in the academic aptitude of classes, year after year. For the statistician, there are technical terms to explain why, even with exactly the same inputs from teacher and school, student outputs will vary. In layperson’s term, the crux of the matter is this: even if students were randomly assigned to classes and schools [and Jesse Rothstein's study has demonstrated that they are not], the classes can be quite skewed academically, especially in small schools since they have smallest samples.

Two wonderful posts — one by guest blogger Daniel Koretz at Eduwonkette and one by Philissa Cramer at Gotham Schools — do a masterful job of explaining this problem [and others] with the schools progress report grades being issued by the New York City Department of Education, pointing out how the resulting sampling error and measurement error casts considerable doubt on the reliability of the grades.

What is especially scandalous about this flaw in the School Progress Reports is that there is a relatively simple method for mitigating the worst excesses of the sampling and measurement error that comes from year to year variations in class composition: do a three year measure. This was one feature of the UFT’s proposed accountability system, and we have advocated for its adoption by the DoE. But Chancellor Klein has refused to adopt a three year measure, on the grounds that it would take in longer for a school to improve its grade and longer for a tailspin in school performance to be manifest. So he has his “in time” grades. One small problem: they are not reliable.

Tagged: data, progress reports

 Print