“Supposed” Problems with Progress Reports are Real

Joel Klein held a press conference at PS 46 in Queens last week in yet another attempt to deflect criticism from his problematic progress reports.

The progress reports, which label our schools with single-letter grades, rely upon a virtually incomprehensible formula that significantly distorts the record of many city schools. To add to the confusion, Klein’s grades are at odds with state evaluations of our schools. The city-state discrepancies came to light just before the December holidays, and so there was Klein, just after that holiday on the steps of PS 46 trying to lend legitimacy to a failing formula by rewarding cash to grade-A schools.

PS 46 is indeed one of the city’s great schools and it deserves recognition. But 134 schools got rewards, and Klein’s choice of PS 46 for the press conference was not accidental. PS 46 is a consistently high-performing school, and the progress reports have come under attack in part because they tend to punish consistently high-performing schools (see here and here, for example). In fact, the grades are so distorted that some of the city’s top schools got D’s and F’s.

PS 46 escaped that fate. In fact, given Klein’s formula, one might even say that in spite of consistently strong performance, PS 46 received an A.

No wonder then that Klein chose PS 46 for the press conference. Highlighting its success might undermine the criticism that great schools were hurt by his reports.

The press conference didn’t turn out as planned (Klein was taken to task by Assemblyman Mark Weprin for turning our schools into testing mills), but in the Principal’s Weekly Klein released later in the week he got to take his shot. In it, Klein said,

What this school has accomplished is remarkable. It serves a high-performing population, and yet its progress has not been stymied by a supposed “ceiling effect.” [bold added]

“Supposed,” says Klein, and what that means is clear: Don’t believe what you’re hearing about “ceiling effect.” If those supposedly good schools truly were good, then they’d be getting an A (not an F or D), just like PS 46. That’s the spin, and now that it’s in Principal’s Weekly, we’re likely to hear it again and again.

Well, Mr. Klein. Nice try. And I am glad you have taken it upon yourself to praise one of our many excellent schools – it’s a nice change from the constant threats of punishment. But this recognition of success does not in any way change the essential flaws with the progress reports. It gets tiresome to do this, but records online become records of fact unless someone comes along to correct them. For the record, then, let’s take a closer look at PS 46 and another high performing school that I know well, PS 35.

PS 35 received an F because its math scores were too high to go up, and its ELA scores dropped a bit between 2006 and 2007. But let’s look at the whole picture, and let’s look also at what would have happened if the progress reports included data that was more long term. As the chart that follows shows:

• PS 35 has been a consistent high performer, clearly competitive with PS 46;
• PS 35 slightly outperforms PS 46, evan after having a drop in ELA in 2007;
• PS 46 had significant drops in scores just one year previous to the reports – an even steeper drop than PS 35 had this year. In other words, had the progress reports been formulated a year ago, Klein probably would have been praising PS 35 – and wishing PS 46 would go away.

	PS 46	PS 35
Percent passing ELA 2006	82	93
Percent passing ELA 2007	84	87
Percent passing Math 2006	94	98
Percent passing Math 2007	90	98
Not included in progress report
ELA progress in passing from 05-06 (grade 4 only*)	- 17.1 (to 78.6%)	+5 (to 95%)
Math progress in passing from 05-06 (grade 4 only*)	- 7.6 (to 89%)	+2 (to 100%)
Percent passing ELA 2004-07 (grade 4 only*)	85	89
Percent passing math 2004-07 (grade 4 only*)	95	98
FINAL GRADE	A	F

(state tests for grades 3 and 5 did not begin until 2006)

Let me make something clear. PS 46 is a great school. The sudden single-year drops I highlighted are virtually meaningless. PS 46 and PS 35 are terrific institutions, and both deserve to be recognized for their success.

As do many schools:

	Overall passing rate last two years	Grade
PS R005	89	C
PS R036	84	F
PS R055	86	D
PS Q046 (Klein’s rewarded school)	87	A

One DoEer has commented to me that schools like these are merely “outliers” – extreme anomalies by which the system can’t be judged. But that’s not true. I talk about these kinds of examples because they are easier to talk about – easier for the public to see and understand. But what they really indicate is that there is something fundamentally flawed with all of the progress reports, and that tampering with the edges won’t make them better.

The progress reports are driving the system. We have 1400 schools, 80,000 teachers and over one million students dancing to their tune, but that tune is almost wholly predicated on unreliable tests wrapped into one grand, unintelligible and seriously flawed formula. Pointing out the problems with them isn’t about nit-picking and isn’t about being obsessive. It’s about trying to get the maestro at the DoE to stop with the conducting and the spinning.

Save the few high notes that work.

Rewrite the rest.