I am the UFT representative who sits on the DOE’s Technical Expert Panel for the Value-Added Teacher Initiative — the project that is attempting to develop a value-added method to evaluate teachers based on their students’ test scores. (So now I have outed myself. CitySue is actually the UFT’s Director of Policy Research.) But that seat implies a lot more participation than reality conferred.
I am probably on this panel — indeed the panel itself was probably created — for one reason only — so the DOE can say that the UFT “participated” in the project, as they have done for the last couple of days. Let me tell you how it really was.
The panel consisted of a dozen or so academic researchers, all but two of whom I believe are or were employed by the DOE. (Yes, I am one of the exceptions.) The UFT had been invited to join the panel only after President Weingarten had angrily refused to endorse the project last summer and had won a concession that results would not be used to evaluate any UFT member.
At that time, the union’s opposition to the relentless focus on high-stakes tests was the main reason for its objections to the experiment. But by December, after we had learned more about DOE’s plans, we had much more cause for concern. Still, to skip to the bottom line before I fill in the details, remember we have a signed contract until October 2009. By then Klein and Company will be packing their bags.
Initially, I was pleased to join the panel. I’d been reading so much about the controversy swirling around the usefulness of value-added (VA) analysis in developing fair and objective ways to evaluate teacher performance. For myself, I can’t remember ever having received an observation report that was truly helpful or thoughtful, and I’d received some that made it clear that the AP was reading his mail throughout my lesson. Maybe, I thought, VA would be an improvement. Hope springs eternal.
The Panel met only twice, in September and October. I say “met” advisedly, because we convened only on the Internet. The discussions centered almost entirely on whether or how to create statistical substitutes for missing, “dirty” or unavailable data. In fact, good data was so scarce that the researchers set a very low bar for the amount of data considered adequate for evaluating a teacher: all that was needed to reach a conclusion on the effectiveness of any teacher was the scores in the current and prior year of three students! (Yes, you read right. Three. About 10 percent of an elementary teacher’s students and 2 percent of a middle school teacher’s students.)
That begins to tell you how flimsy the analysis was. And, as you can imagine, with many teachers’ ratings depending on so few students, just a couple of kids who were having a bad test day could have a huge effect on that teacher’s value added. Even the most zealous statisticians admit they can’t control for everything. Did mom and dad have a fight that morning? Did the dog die? Was a cold coming on? The possibilities are endless.
The biggest data problem was the timing of NYC standardized tests. Ideally, a value-added assessment system should use a pre and post test in September and June (before summer learning loss or a boost from summer school affect the outcome). End-of-year tests are second best for measuring learning growth over a year’s time.
But here tests are given mid-year, ELA in January and math in March. Growth from year to year is the result of at least two teachers’ work. (That doesn’t even count the contributions of after-school and summer school teachers, private tutors, involved parents, cluster teachers, teachers of other subjects, etc. who are never accounted for.) How to divide the gain (or loss) in scores between the current teacher and last spring’s teacher? Without a test in between, there is no way to know.
After much debate, however, the answer was simple. Apportion the change in scores in proportion to the amount of time spent with each teacher. For example, 60 percent of the math score change can be attributed to the current teacher for her work from September to February (6 mos.) and 40 percent can be attributed to last spring’s teacher who had the student from March to June (4 mos.). The problem? This method runs totally counter to the principle that underlies the entire concept of value-added: that teachers have varying skill levels and contribute to their students’ learning at different rates. If every teacher’s contribution can be measured by the time they spend with the student, why go through this elaborate exercise?
The upshot of this one decision is that no teacher’s contribution is accurately measured. The measures, as flawed as they may be to start with, are only applicable to each pair of teachers!
I could go on but I think you’ve heard enough. No wonder even avid critics of teacher unions, and those who most loudly demand more teacher accountability, have conceded that value-added assessment has a ways to go before it can be reliably used to rate individual teachers.
By the way, you may have noted that these statistical problems are mostly obliterated when groups of teachers, instead of individuals, are considered. When the numbers are larger, one or two anomalies have less effect. Most important, there is no need to determine whether the fourth grade teacher or the fifth grade teacher — or even the second or third grade teachers — contributed more to a fifth grader’s score. Everybody gets credit. In schools, for example, where literacy is incorporated into every subject class, teachers of science and social studies are recognized for the reading instruction they offer. That’s why schoolwide progress is a more reliable accountability method.
There are many other reasons to question the wisdom of this kind of evaluation. We haven’t even talked about the reliability of the tests that underlie the method, or the comparability of tests designed for different grades, or, most important, the effect this system would have on a balanced curriculum. But suffice it to say that, as we learned more about the project, we became more alarmed. Fortunately, we had a promise that the data were not to be used in any actual evaluations.
In late November, as the individual teacher reports were being prepared, we demanded to know what schools had participated in the experiment so we could alert the unwitting teacher participants. But we were rebuffed becasue the principals, who had volunteered to take part in the pilot, had been promised confidentiality. (What were they afraid of?) We did win agreement, however, that principals had to keep individual teacher results confidential, not to be shared with parents or colleagues. We were still talking with DOE officials when Deputy Chancellor Chris Cerf went public, thus cutting off any chance that we could uncover the schools and alert our members.
Of course, that is not the end of the story. We’ll hear that only when an arbitrator or judge rules on their right to use this method of assessing teachers. It’s clear to us that teacher evaluation is a contractual issue and if they want to change it they have to negotiate that. Fortunately — though not by mere coincidence — we signed a new contract early and it is effective until October 2009, right near the end of this administration’s tenure.
So while this project is instructive becasue it illustrates the dangers of DOE’s obsession with measuring over learning, and their blind faith in data, UFT members have little to fear. Like the threats to tenure that we thwarted in Albany, and the rumblings about merit pay, which we turned into schoolwide bonuses and 55/25 to boot, this will turn out to be lots of sound and fury, signifying nothing.
14 Comments:
1 jd2718
· Jan 23, 2008 at 8:50 am
Sue,
Leo wrote
and I absolutely distrust the DoE’s data people,
which leaves me wondering three things:
1) what was the panel’s stated purpose?
2) who decided we should participate, and why?
3) Was the membership informed and I missed it?
My biggest question is the “why” at the end of #2. (if you don’t know, maybe you could tell us what person or body made the decision?)
Jonathan
2 LabourStart -
· Jan 23, 2008 at 10:34 am
[...] Wal-Mart employees sign up for health care plan amid labor union battles [Pravda] 23-Jan-2008 USA The real true facts behind DOE teacher value added project [Edwize] 23-Jan-2008 USA Shuttle drivers to vote on union [Emory Wheel] 23-Jan-2008 USA [...]
3 CitySue
· Jan 23, 2008 at 11:21 am
Jonathan:
Keep in mind that the project, which was privately funded, was going ahead with or without us. To me, the decision to know what was going on and eventually to be able to critique and fight it rather than purposely remain ignorant was a no-brainer. Do you disagree?
4 Sherman Dorn: Value-added, with botulism
· Jan 23, 2008 at 11:36 am
[...] Kevin Carey proclaims that value-added [method] comes of age, he might want to read the real true facts behind the New York City teacher value-added project, wherein we learn that the city’s great statistical experts thought three children were enough of a [...]
5 NYC Educator
· Jan 23, 2008 at 12:26 pm
Like you, City Sue, I will shed few tears when Klein and Bloomberg pack their bags. I do remember, though, similar sentiments about our erstwhile leader, Rudy Giuliani.
I think it’s important for the UFT to become more proactive, more assertive, and less dependent on who may or may not be the next mayor, governor, president, or whatever. Governor Spitzer, for example (and I voted for him as enthusiastically as anyone), has just drastically reduced funds that could’ve been used to reduce class sizes in NYC.
We can’t count on a friend in City Hall. I’ve been teaching for 23 years, and we’ve almost never had one. While I remember a brief flash of sympathy from Mayor Dinkins, he quickly turned his back on us, rather than defending education as important.
We need to stand up for ourselves and aggressively push a pro-teacher, pro-education agenda no matter who is in charge. Certainly if we don’t, no one else will do it for us.
6 jd2718
· Jan 23, 2008 at 1:34 pm
City Sue,
I am not certain whether it made sense or not to participate (although after the fact, it may not matter much). I would hope that we consider options carefully, and I would hesitate to characterize anything controversial as a “no-brainer.”
My questions are real questions. Can you or someone else answer them?
Jonathan
7 NYCDoE rates teachers - more response « JD2718
· Jan 23, 2008 at 2:32 pm
[...] brings me to the UFT. City Sue, our director of policy research, sat on a panel from September that considered, in theory,…. We should have known. We should have informed our [...]
8 CitySue
· Jan 23, 2008 at 5:27 pm
Here, Jonathan, are the DOE’s stated goals for the VA project:
· Develop models to measure the value that individual teachers add to student achievement
· Provide teacher value-added data to principals in an accessible form
· Evaluate the potential benefits and uses of these data
Notice there is no mention of evaluating teachers.
I know that Randi has frequently written and spoken at member meetings about the chancellor’s efforts to use student test scores in tenure decisions, in evaluating teachers and in individual merit pay. Her member letter of Jan 2007 devotes a couple of paragrafs to it. As I recollect, she has mentioned this project specifically at either a DA or executive board meeting, maybe both and maybe more than once. From my vantage point, I cannot trace the precise decision making chain as I do not take part in those meetings. Obviously, if we had decided not to sit on the panel, we would know much less than we do now, and I would not have been able to offer the above criticism.
9 Otter
· Jan 23, 2008 at 5:55 pm
CitySue,
You say in an earlier comment that this program was privately funded. By whom?
10 jd2718
· Jan 23, 2008 at 6:05 pm
Sue,
fair enough on those answers, and thank you. We have been clear on opposing test data for ‘high stakes’ decisions, and I recall the letter.
I don’t know about participating. I am uneasy about lending legitimacy to their projects. And I worry about how this affects our ability to rally our members, as it were. You picked up information, but was it worth it? These are questions.
We should keep our membership better informed. That’s not a question.
But more importantly, going forward, is our response. The resolution that the Exec Board passed last night is a good one. (I am looking for the text – it probably won’t be on the website until the Delegate Assembly ratifies it).
Jonathan
11 MichaelB
· Jan 23, 2008 at 7:24 pm
I have to hand it to Klein – this is a smart political move. He sets the agenda, while we have the difficult task of explaining to the public that his plan is bogus while not appearing to be standing in the way of sensible reform.
There seems to be a pattern here. Klein announces “improvements” in test scores, gets the positive headlines, and the UFT is left to explain how they aren’t really improving. Klein announces improvements in the graduation rates of the new small schools and we have to point out that his numbers are bogus, but the papers keep repeating his version of the story. Meanwhile, no one notices that his reforms aren’t all they’re cracked up to be.
I feel like we’re going to be stuck playing defense until we make more of an issue of teacher retention. Our numbers show an important story – that teachers are leaving the system under Klein as fast as ever. Everyone agrees this is a problem, but no one talks about it. Why can’t we put consistent public pressure on Klein to address this issue? Why couldn’t we have said the success of his tenure would depend on by whether or not he could cut in half the attrition rate and then hold him to it? Why can’t we denounce his ill-advised plans as smokescreens to cover up his terrible retention numbers?
As long Klein gets to use his numbers to tell his story, he’ll be able to set the agenda. What about us and our story?
12 jd2718
· Jan 26, 2008 at 11:25 am
Well-said Michael. There is a battle in the media, and we don’t need to lose it.
Retention should be front and foremost, but there are other pro-kid, pro-school, pro-teacher issues to push as well. Testing is huge, and we should make more of it. The moment passed on NCLB, where we finessed what should have been a loud “no.” The physical conditions of the schools. Overcrowding. The bus routes.
But to do this we need to be willing to say clearly that Bloomberg’s record on education is one of failure.
Jonathan
13 Peeing into the Wind: Why is Klein Picking Losing Fights With the Teachers Union? « Ed In The Apple
· Jan 27, 2008 at 9:35 pm
[...] folks over at Edwize, Leo Casey and City Sue, skewered the concept, and, the blogosphere in general was [...]
14 Creating Lifelong Learners » Blog Archive » Carnival of Education #156
· Jan 29, 2008 at 8:34 pm
[...] The Real True Facts Behind the DOE Teacher Value-Added Project by Steve Perez in Edwize. [...]
Leave a Comment