Log in  |  Search

Burying the Bias in the Teacher Data Reports

At minimum, the DOE’s Teacher Data Reports are biased against teachers who work with high-performing students.

And the DOE seems to be doing everything it can to make sure that teachers and the public never find that out.

The bias is (to my eyes, anyway) huge. In elementary schools, for example, teachers who work with high-performing math students are 40 times more likely to fall in the bottom 5% of all teachers than in the top. Those findings come from information found on last year’s Teacher Data Reports, [1] but with an apparent eye to a possible public release and with clear contempt for the teachers they are supposed to support, this year the DOE decided to simply leave it out. [2]

The specific information missing from the new reports is the proficiency scores, namely the average test scores of students prior to taking the class, and the average when they left. By leaving it out, DOE dispenses with all pretenses that the Reports are designed to be useful for teachers. But the omission also allows the DOE to keep the focus right on the number it wants the potential public to see: the teacher’s percentile, free of any kind of context whatsoever. Were the students in the teacher’s class high or low performing? We won’t know. How much did their scores change? We won’t know that either. And how much better were the so-called “better” teachers, and how much worse the worst? Without proficiency scores, we simply will not know.

From a DOE that has professed the “public’s right to know,” it’s pretty shabby treatment.

Here are the details. There were 690 math teachers whose students were expected to score at the top levels of the test (with a proficiency score of at least 4). If we look at all teacher results for the most recent year on the reports, we see that 63 of these teachers had a result that placed them in the bottom 5th percentile of all teachers, but only 2 had scores that placed them in the top 5%. That’s a 30 to 1 ratio. At the elementary level, the ratio of bottom to top was 41 to 1.

Math after One Year of Teacher Data

Number of Teachers
Grade Level Predicted Score Total Bottom 5% Top 5%
All ≥4 690 63 2
4th & 5th ≥4 531 41 1

Not all teachers fell in the top or bottom, of course — but of course the bias extended to teachers throughout the spectrum. Note, for example, the relative number of teachers who fall in the DOE categories of Below Average and Above Average.

Math after One Year of Teacher Data

Number of Teachers
Grade Level Predicted Score Total Bottom 5% Top 5 % Below Average
5th to 25th
percentile
Above Average
75th to 95th percentile
All ≥4 690 63 2 194 82
4th & 5th ≥4 531 41 1 154 64

And, though we are continually told that value-added becomes more reliable with more years of data, that’s clearly not the case here. In fact, after four years, the bias is simply confirmed:

Math after Four Years of Teacher Data

Number of Teachers
Grade Level Predicted Score Total Bottom 5% Top 5% Below Average
5th to 25th
percentile
Above Average
75th to 95th percentile
All ≥4 105 6 0 34 9
4th & 5th ≥4 88 4 0 30 9

Fewer teachers have four years of information, but for those who do, the results have much worse implications. After all, the urban (school) legend is that more years of information ensure more legitimate results. But that doesn’t happen when the fancy formulas work against our teachers in a systematic way.

Keep in mind that the bias would not be limited to those teachers who work exclusively with the city’s top students. In the DOE files, we can only see teachers whose overall class average is high, but regardless of the class average, any class might have high-scoring students in it, and that would affect results. What is more, biases that show up in such an exaggerated fashion on high-performing students might be at work in the results tabulated for every single child. Considering that small differences in student progress can translate into very different outcomes for teachers (more on that in a future post), any statistical bias might mean a lot.

Besides biases in the formulas, there are other sources of bias. We know, for example, that the reports do not account for things that we know make a difference like regular academic intervention (tutoring), 504 test modifications, attendance the year of the test, and the huge variation among English Language Learners. [3] These things are not even included, much less included correctly.

Similar patterns emerge in ELA. Students tend not to score as highly in ELA (test prep can take a student’s reading ability only so far), but the pattern is the same whether we look at results after one year …

ELA after One Year of Teacher Data

Number of Teachers
Grade Level Predicted Score Total Bottom 5% Top 5% Below Average
5th to 25th
percentile
Above Average
75th to 95th percentile
All ≥3.9 29 8 0 5 3
4th & 5th ≥3.9 28 7 0 5 3

…or four:

ELA after Four Years of Teacher Data

Number of Teachers
Grade Level Predicted Score Total Low
bottom 5%
High
top 5%
Below Average
5th to 25th
percentile
Above Average
75th to 95th percentile
All ≥3.9 10 3 0 3 1
4th & 5th ≥3.9 10 3 0 3 1

I asked a teacher to call the Teacher Data Department at the DOE for an explanation of where the scores are. With the kind of gentleness usually reserved for kindergartners, the DOE staffer explained that DOE had decided the inclusion of proficiency scores was “confusing to its teachers.” What condescension — and what a thin deceit. These are the same teachers who have their noses rubbed in proficiency scores morning, noon, and night by a DOE intent on substituting test-prep lessons for the reading, writing and arithmetic we’d really like to teach. Believe me, teachers understand proficiency rankings. Besides, if DOE were so worried that proficiency scores might confuse our pretty little heads, why did they provide a separate “data” document that lists proficiencies translated into statistical formulations that no one understands?

Well, official DOE is sure to have its official answer, and maybe the answer will even make some sense. One cannot escape the feeling, however, that the purpose of the DOE’s actions is either to keep the machinery of teacher persecution at full throttle, or to make sure the press doesn’t see just how disastrous the Teacher Data Reports really are.

I suspect the answer may be both.


1 Last year, the DOE released a spreadsheet that contained some of the data listed on every Teacher Data Report, (with the names of teachers and their schools redacted. The file was of limited use, and crucial columns of information were missing (such as the number of students), but it did give us a glimpse of some of the major trends citywide.

2 The UFT is back in court this week working to block the DOE from releasing the TDR’s. Though the case centers on last year’s reports (which were based on spring 2009 tests), a decision is likely to apply to future Reports as well, including those just released (and based on the spring 2010 tests).

3 The Reports do take ELL status into consideration, but in a way that is wholly insufficient. A Math teacher of English Language Learners from Mexico, for example, is likely to contend with different levels challenge in his classes than a teacher of ELL kids from Shanghai.

Print

8 Comments: