(This is the second of two posts written in response to Joel Klein’s manifesto, The Failure of American Schools, which was published in the June issue of Atlantic Monthly. In the first post, I addressed Klein’s attribution of an apocryphal anti-union quote to the late UFT and AFT President Al Shanker.)
The defining characteristic of a bad faith argument is not that it is wrong, although it certainly is that, but that the person wielding it knows that it is wrong, or with a minimal exercise of due diligence, would discover that it is wrong. Klein’s Atlantic essay is replete with bad faith arguments, so many that it would require a small pamphlet to respond to every such argument. Consequently, I will restrict myself to analyzing a number of the more central propositions he puts forward.
Using Value-Added ‘Teacher Data Reports’ To Evaluate Teachers
Klein declares that he is “still shocked” that the UFT opposed his efforts to use the value-added ‘teacher data reports’ he had developed for high stakes decisions on tenure and discontinuance. “As a result, even when making a lifetime tenure commitment,” Klein writes, “under New York law you could not consider a teacher’s impact on student learning. The Kafkaesque outcome demonstrates precisely the way the system is run: for the adults.”
What Klein avoids addressing, but what he certainly knows well, is that both his value-added “teacher data reports” and the New York State English Language Arts [ELA] and Math exams used to develop these reports had serious, fundamental flaws that prevented them from being a meaningful measure of educational progress.
When the scores on state ELA and Math exams improved at a extraordinarily high rate that was simply not credible, especially when compared to National Assessment of Educational Progress (NAEP) ELA and Math scores which were basically flat, the State Education Department commissioned noted testing expert Daniel Koretz, Harvard Professor of Education, to review the exams.* Koretz found that there was rampant grade inflation in the tests [see here, here and here] on a scale that invalidated the results. “We have to stop lying to our kids,” Regents Chancellor Meryl Tisch said. “We have to be able to know what they do and do not know.” When the New York State Department of Education recalibrated cut scores on the tests, the performance of New York City students fell dramatically. But it was these invalid tests that Klein wanted to use, and which he presents as a true and complete measure of “student learning.”
What is worse is that Klein knows that the statistical modeling used in his ‘teacher data reports’ has produced calculations which are literally off the charts in terms of inaccuracy and unreliability: the average margin of error on the last set of ‘teacher data reports’ was ± 28 points. It would be difficult to understate just how much this particular measure casts into disrepute the use of these reports for making the high stakes teacher evaluation and teacher tenure decisions advocated by Klein. For statisticians, the margin of error reflects how far the actual ‘true’ figure could vary from the reported calculation: the lesser the margin of error, the greater the confidence one can place in the accuracy of the figure. A key determining factor here is how widely and randomly a particular universe is sampled: for example, all other things being equal, one can be much more confident that a poll with a sample of 1000 New York City residents will produce a truer reading of the city’s sentiments than a poll with a sample of 100 New York City residents. This difference would be expressed in terms of the two poll’s margins of error. A margin of error of ± 28 points on the teacher data reports means that the ‘true’ score of a teacher who scores at the 50th percentile could be as high as above the 75th percentile or as low as below the 25th percentile. Similarly, the ‘true’ score of a teacher who scores at the 25th percentile could be as low as the 1st percentile and as high as over the 50th percentile. In short, it means that there is so much ‘noise’ in this set of statistics as to make them virtually unusable for any meaningful insight into individual teacher performance, let alone as a basis for making high stakes evaluation and tenure decisions.
All value-added modeling of test scores for individual teachers has had disturbingly large error rates: a 2010 US DoE study found that in value-added modeling “error rates for comparing a teacher’s performance to the average are likely to be about 25 percent with three years of data and 35 percent with one year of data.” But even against this backdrop, New York City’s margins of error are extraordinarily wide, a statistical outlier. Joel Klein is fully aware of this fact, because he decided for political reasons — against the advice of his own team and the outside statisticians working on the model — to use a single year’s test scores rather than three year’s scores. Since a very significant portion of the large margin of error in value-added modeling of test scores is traceable to the very small and less than random samplings on which they are based, using test scores from multiple years could reduce and ameliorate that problem. But Klein decided he wanted a measure that could be attached to every annual evaluation, even if it were considerably more inaccurate and unreliable as a consequence.
This is why, when the UFT and NYSUT negotiated the framework for a new teacher evaluation system with the State Education Commissioner and the Chancellor of the Regents last year, we limited the role of value-added modeling from standardized exams to a 20% portion of the total evaluation and required the use of a valid and reliable value-added metric, unlike the one employed in NYC’s ‘teacher data reports.’
Klein neglects to note why so few New York City teachers are prepared to accept his word on anything related to his ‘teacher data reports': the way in which he broke his explicit, written promise to them that the reports were being produced to improve classroom instruction and would consequently be shared only with the teacher and her immediate supervisors. But as soon as Klein saw a potential political advantage to breaking his promise, the word about town was that Tweed had begun soliciting Freedom of Information requests from the NYC media outlets asking for the publication of the reports. When the requests were filed, Klein immediately announced his intention to provide the reports. (Some reporters were so disgusted about the way in which Klein and the DoE were using the media that they threatened to quit over the issue.) The UFT went to court to prevent the publication of the reports, and final disposition of the case is still pending.
There is indeed something profoundly Kafkaesque about this issue has played out, although not in the way that Klein dishonestly suggests. In what could be a passage straight out of Kafka’s Trial, New York City public school teachers now face the threatened professional stigma that their name could be published in newspapers and on the Internet, attached to wildly inaccurate and unreliable teacher data reports which are, in turn, based on New York State exams that the State Education Department itself has pronounced as broken. And if that were not bad enough, Klein demands that we use these flawed instruments to make decisions that could terminate a professional career, such as the choice of awarding tenure or discontinuing a teacher. And he is “shocked” that anyone would not understand that such a state of affairs is all in the best interests of the students.
The Academic Performance of New York City Charters Schools
Klein writes that New York City charter schools were “getting significantly better reading and math results with their students than were comparable traditional public schools,” and singles out Eva Moskowitz’s Harlem Success Academy 1 as a charter school whose students “are demographically almost identical to those attending nearby community and charter schools, yet gets entirely different results.”
What Klein neglects to say, but what he most certainly knows and knows well, is that when the New York State Education Department ended the rampant grade inflation in its Math and ELA standardized exams last year, the test scores of New York City charter schools plummeted, falling at a much faster rate than the test scores of the traditional public schools.† Klein’s own school progress reports for elementary and middle schools, which he based largely on the student grades from those exams and used to make high stakes decisions about closing schools, puts this development in relief: 61% of the traditional elementary and middle public schools received an ‘A’ or a ‘B’ on their school progress report, compared to 46% of the elementary and middle charter schools.
Moreover, as Christina Collins pointed out here on Edwize, the demographics of Harlem Success Academy 1 are starkly different from nearby traditional public schools: according to the data Harlem Success released to the state, only 3% of its students are English Language Learners, compared to 12% in the average Harlem district school. No students at Harlem Success require the most intense level of Special Education services, compared to 31.4% of special needs students in local district schools, and almost 80% of Harlem district school students qualify for free lunch, the widely accepted proxy for poverty, compared with just over 60% at Harlem Success.
The Financing of New York City Charter Schools
Klein writes that “to support effective choice, moreover, we need to provide real funding equity: the money must be for the child, not the school. So if Juan goes to PS 11, which gets $20,000 as a result, then that same $20,000 must go to a KIPP charter school if Juan decides to go there.” The clear inference of this statement is that district schools such as PS 11 receive greater public funding than charter schools like KIPP.
The Independent Budget Office, an arm of New York City government, has studied the issue of comparative funding for charter schools and district schools, and found that two-thirds of New York City charter schools receive greater public financial support than district public schools. In a 2010 report and a 2011 follow-up, the IBO found that charter schools housed in New York City public school buildings (all of the KIPP NYC schools fall into this category) received $649 more in per student public support than district public schools. Moreover, the IBO noted that when this year’s 8% increase in funding for charter schools was added to the formula, this advantage from public funding will become even greater, an advantage we at Edwize calculated as over $2000 per student, for a 12.5% differential in favor of most charters. Klein established the policy — the provision of free facilities, free utilities and free school safety — that created this advantage for two-thirds of New York City charter schools, and he is most certainly aware of its results.
It is also worth pointing that charter schools have a significant advantage over district schools in raising private and philanthropic funds. It takes some extraordinary chutzpah on Klein’s part to use a KIPP school in this sort of comparison, since KIPP’s fundraising from private and philanthropic sources far outstrips both district schools and other charter schools. In their recent comprehensive study of KIPP schools, Gary Miron, Et. Al. found that KIPP schools were getting an average of $5760 per student in private funding, and that when public and private funding were combined, KIPP schools had a total of $18491 in funding per student — a massive funding advantage of more than 50% over the traditional public schools in the districts where KIPP was located (p. ii).
Due Process for Teachers: The Saga of the Rubber Room
The extent of this “no one gets fired” mentality is difficult to overstate—or even adequately describe. Steven Brill wrote an eye-opening piece in The New Yorker about the “rubber rooms” in New York City, where teachers were kept, while doing no work, pending resolution of the charges against them—mostly for malfeasance, like physical abuse or embezzlement, but also for incompetence. The teachers got paid regardless. (To add insult to injury, these cases ultimately were heard by an arbitrator whom the union had to first approve.) Before we stopped this charade—unfortunately by returning many of these teachers to the classroom, as the arbitrators likely would have required—it used to cost the City about $35 million a year… No one—and the union means no one—gets fired.
For clarity of exposition, let’s dissect this passage into different points.
- Take special note of the initial use of passive voice: the “rubber rooms” were places “where teachers were kept, while doing no work.” In fact, it was Klein himself who created the “rubber rooms.” Before Klein’s tenure as Chancellor, teachers with charges pending against them were placed in DoE offices and in other schools where they would perform non-teaching work for the school system while they awaited disposition of the charges against them, much as they now do after the agreement to close the “rubber rooms.”
- Having created the “rubber rooms,” Klein instituted a number of perverse incentives that led to a significant increase in the number of teachers placed in them. While a teacher who was excessed from his school remained on that school’s payroll until he found a position at another school, a teacher who was sent to the “rubber room” was removed from the school’s payroll after a brief period. Add to that policy the elimination of meaningful oversight of principal decisions to send teachers to the “rubber room,” and patterns of abuse began to emerge. Some schools and principals had large numbers of staff sent to the “rubber room”; others had very few. When the ‘rubber rooms’ were finally closed, the numbers confirmed that they had been misused: of the approximately 750 cases in the “rubber room,” close to 25% were terminated, resigned or retired as a result of a hearing. (So much for “No one—and the union means no one—gets fired.”) In over 60% of the cases, teachers were returned to active service, either because they were completely exonerated of the charges or because the findings against them were so minimal that a fine was the only penalty levied. The remainder of the cases are still pending.
- Now, take note of Klein’s shift to an active voice: “before we stopped this charade…” In truth, Klein opposed the agreement closing of the “rubber rooms,” as he believed that they provided him with a public relations bludgeon to use against teachers and their union. It was only under the direct orders of Mayor Bloomberg that an agreement was reached with the UFT to end them.
- Klein’s conducted an ongoing campaign around the “rubber rooms.” Steven Brill, a close personal friend of then Deputy Chancellor Chris Cerf, was brought in to write the New Yorker article to be used in this campaign, and was given complete access to the “rubber rooms” for that purpose. At the same time, a filmmaker making a film critical of the NYC DoE role in establishing and running the “rubber rooms” was arrested for entering and filming one of the sites.
- In the above passage and elsewhere in the Atlantic article, Klein mentions that charges against teachers are heard and decided by “union approved” arbitrators. What he neglects to note is that the very same arbitrators must also be approved by his Department of Education. The law requires that both parties approve the arbitrators, in order to ensure that they are impartial and neutral decision makers. When a lawyer such as Klein describes arbitrators in this misleading, one-sided fashion, he is engaged in a calculated misrepresentation of the process. What Klein really wants here is the elimination of the ability of teachers to have a fair, impartial hearing: he wants one side — the DoE — to control completely the person making the decisions, much like he controlled the DoE officers who heard and decided appeals of unsatisfactory ratings. The result is a ‘rubber stamp’ of whatever the principal wants: in the over 1300 ‘U’ rating appeals that have been heard by the Department of Education over the last two school years, exactly 3 cases have had that rating overturned.
Collaboration: The Elixir of the Status Quo Crowd?
During my tenure, I fought to break this institutional stranglehold of defenders of the status quo. I did so because I believed that our kids are not getting the education they deserve, that we have clear examples showing dramatically better results, and that we won’t achieve those results if we just keep tinkering. Since 2007, my colleague Michelle Rhee, in Washington, D.C., has been making the same noises. The response, often from friends as well as opponents, was that we were unrealistic: complex systems don’t change easily, impatience is immature, and directly challenging the educational establishment is not a winning strategy. “You need to be more collaborative and less controversial,” we were repeatedly admonished.
That’s bad advice. Collaboration is the elixir of the status-quo crowd.
Klein goes on to cite two schools — Brooklyn Generations and the New American Academy — as exemplars of innovative school design. What he knows and knows well but neglects to say, is that both of those schools were founded by educators who believe strongly in collaboration and who involved teachers and the UFT in their founding and operations. Both schools exist because the UFT negotiated contractual side agreements that allowed their experimental designs to be enacted. What is more, the highlights of the new NYC schools begun over the last decade are found in those, such as the International Schools and the New Century Schools of New Visions, which were explicitly collaborative projects, done together with teachers and the UFT. By contrast, those new schools with leaders who have embraced the antagonistic and confrontational posture of Klein have produced some of the greatest failures.
Klein ignores the real world falsification of his anti-collaboration posture, and elides the fact that the very schools he embraces as innovative and producing strong results are strongly collaborative, because he is so thoroughly enmeshed in ideological thinking. Ideological thinking of the sort that Klein embraces rests on two dispositions: intellectual hubris and moral self-righteousness. Klein’s intellectual hubris, like that of co-thinker Michelle Rhee and their fellow so-called education reformers, is based on the dogma that a very simple first principle — schools should function as businesses and education should be operated as a market — holds the answer to every question that confronts us. This reliance on a first principle to answer every question is the hallmark of the ideological ‘true believer.’ It is invariably combined with a moral self-righteousness: those who disagree with your dogma do so because they are morally inferior. In the hands of Klein, this disposition takes the form of the following thesis: one must agree with his first principles of market fundamentalism to be truly interested in the needs of children. If you disagree, if you believe that education is more than a personal good, if you believe that education must serves public purposes, most preeminently, that of the democratic self-rule of an educated citizenry, if you believe that markets undermine those public purposes and breed inequality, then Klein concludes that you have to be motivated by the needs of adults over children. Klein’s entire Atlantic magazine treatise is riven with this moral self-righteousness.
Combine these two dispositions of ideological thinking — intellectual hubris and moral self-righteousness — and the idea that there is virtue in confrontation and antagonism follows logically. Those with whom you disagree are the enemy, to be fought at every turn. With this mindset, ‘bad faith’ arguments are simply the weapons of war, to be used if they suit your purpose.
* The National Assessments of Educational Progress are a series of national exams of elementary and secondary student achievement sponsored by the US Department of Education. They are considered the ‘gold standard’ of exams assessing student academic achievement.
† New York States now provides standardized Math and English Language Arts exams, grades 3 through 8. For elementary and middle schools, 85% of their school progress report grades are based on how the students in these schools perform on the tests. New York City charter schools are disproportionately elementary and middle grades schools.