Click on the graphs for larger versions.

It’s standard fare to question the effects of high-stakes testing on education. Mention tests and people will tell you about the narrowing of the curriculum, the lowering of the standards, and the changed understanding of what it means to be educated, which used to have something to do with pleasure and imagination and now has mostly to do with isolated, testable skills. But in New York City, testing is not the only thing that is high stakes, and as it turns out, there are more ways to dumb down an education than to add another high-stakes test to it. Consider, for example, New York City’s high stakes credit accumulation scheme.

In this city, the number of credits awarded to students in high schools truly is high stakes. It counts as nearly one third of each high school’s Progress Report grade, and the Progress Report counts for just about everything, including the removal of principals and the closing of schools. Since the Progress Reports were introduced in 2006-2007, the percent of students earning 10 or more credits each year has leapt a (truly) incredible 16 percentage points citywide. For schools with the highest concentration of high need students (the schools most likely to be threatened with closure) the jump is 18 points. Most schools accrued those gains between the first and second years of the Reports.

Dramatic rises in student performance are rarely worth the spreadsheet they are printed on, and these soaring statistics are no exception. In June, the state informed the city that, in spite of the rapid rise in classroom credits, only 20% of our students are college ready. In other words, all those credits might have been good for accountability’s bottom line but they probably were not good for students who may have done less to earn them (through credit recovery policies, for example¹), and who would now need to take remediation classes once they got to college. That’s a serious consequence, every bit as serious as the negative results of high-stakes testing. But when the fate of schools, principals, and mayors hang on the accumulation of credits, students get their credits whether they learned very much or not.

And the DOE’s solution to the college-ready crisis? To add college ready standards to the high school Progress Report, of course. I’d like to blame it on pure cynicism (they like the gaming and want it to continue), but honestly, I think rewards and punishments are the only thing they know.

Meanwhile, Bloomberg continues to crow over his Texas New York miracle. Deprived of bragging rights in the K-8 sector after the state announced that steeply rising test scores were mostly full of hot air, the mayor has shifted the spin to graduation rates. But graduation rates are driven by credit accumulation, which is driven in turn by Progress Reports dangling like a sword over everybody’s head.

As if things were not bad enough, new horrors await us next year when the Progress Reports will also give extra credit to schools that rewrite the education plans of special need students so they can be placed in larger classes among students with very different academic and social needs. Schools will get credit for moving self-contained students into CTT or regular classes. As with credit accumulation policies, this is a political decision and a money decision, and as with credit accumulation it is being masked as a decision in the best interest of students. Those who try to stand against it will be accused of being heartless, old-fashioned and worse.

I wrote about the inherent conflict of interests in the Progress Report four years ago, and I wasn’t the only one. No one listens to us, of course — we are teachers. But sooner or later, when the shouting is done, the schools have been closed, and another generation has been sent off to college to pay for the no-credit remediation they were never given in Bloomberg’s schools — sooner or later everyone will scratch their heads, wonder what happened, and impose new and “rigorous” standards.

And then blame us.

Notes: Credit accumulation data can be found on the Progress Report data sets. In the second chart, I determined student need levels by the peer index on the Progress Reports. The schools represented in that chart are those that fell into the bottom 20% for each year.

The college readiness figure cited above comes from the state and DOE data files, which define students as College Ready if they have attained a 75 or higher on their English Regents and an 80 or higher on their highest Math Regents.

1 Credit recovery allows students to pick up credits for classes that they did not pass, and was unregulated through most of the years covered in this post. During the 09-10 school year the state issued guidelines. Their impact remains unclear.

Mr. Brooks has misrepresented my views. While I have criticized charter schools, I am always careful to point out that they vary widely. The overwhelming majority of high-quality research studies on charters shows that some are excellent, some are abysmal and most are no better than regular public schools.

Some charters succeed because they have additional resources, supplied by their philanthropic sponsors; some get better results by adding extra instructional time. We can learn from these lessons to help regular public schools.

The Times adds this note below the letter:

We invite readers to respond to this letter, as part of our new Sunday Dialogue feature. We plan to publish a sampling of responses in the Sunday Review, and Diane Ravitch will be given an opportunity to reply. E-mail: letters@nytimes.com

What did those of us staying in the same downtown hotel as Hollywood stars like Samuel L. Jackson think we could do to help fix the Detroit Public Schools?

I asked myself that question at the time, and I ask it especially now, when I’m amazed to read about the “draconian” measures being taken by DPS Emergency Financial Manager Robert Bobb. While he concedes his plan makes neither financial nor academic sense, Bobb is nonetheless attempting to solve DPS’s $327 million budget shortfall by closing nearly half of Detroit’s schools and increasing class sizes in the remaining ones to as high as sixty. It seems an insane idea to me, especially since I feel responsible for a big chunk of that deficit.

The company I worked for, you see, is owned by Houghton Mifflin Harcourt, and Houghton Mifflin Harcourt just completed a 15-month contract with the Detroit Public Schools worth $39,859,925.00. That’s right, almost forty million dollars, or more than 12 percent of DPS’s entire budget shortfall, for HMH’s “managed instruction” in reading and math. While I don’t know exactly what forty million dollars of “managed instruction” looks like (who does?), I know some of those millions were used to pay for the tests I helped slap together (mostly recycling passages and questions from our item bank that had been used many times before) and to sponsor my travels to Detroit.

They are coaching their kids themselves and those who can afford it have engaged professional tutors to plant test-acing strategies in their kids’ minds. They are racing to figure out how to crack the secrets of the test so that their kid beats the next guy’s kid to the fast track.

The training centers are commonly referred to as “boot camps.”

This story, reported recently in the Chicago Tribune, provides a perfect example of the pathological role that testing is playing these days. Not only does it substitute for curriculum, it also spoils and supplants the beautiful innocence of childhood.

The natural talents of children will not necessarily win out over the effects of dull but efficient gaming of the system. Anyway, most of the great men and women in all fields and generations around the globe probably, judging from their biographies, would not have been accepted into these programs. The nature of creativity and genius is too elusive, unpredictable, and unidentifiable at this pre-tender age. That’s even true of the nature of ordinary worldly competence.

It’s bad enough that kids no longer are taught to think critically but rather have hammered into them insipid “right” responses necessary to get them selves an admission ticket and, in some situations, bonuses for CEOs.

How absurd that such young children need to be exploited to make a political statement in veneration of the benefits of testing.

But a lot more happened than that.

As State Education Commissioner David Steiner explained at the state’s press conference, the state tests have not simply become too easy. They have become bad tests.

They have been assessing only a very narrow band of state standards and virtually ignoring the rest of the state curriculum. They have repeated questions from year to year, making it easy to game the tests. And they do not reflect what students need to succeed in college and careers.

That is going to change. Over the next three years, the  tests will become longer. They will test more material, have more open-ended questions and require more writing. They will aim to assess not whether students learned “test-taking tricks,” in Steiner’s words, but whether they can apply knowledge and explain their answers. By 2014-15 the goal is that our state tests will be able to tell students honestly if they are on track to succeed in college and beyond.

Revelations by Subgroup

Surprisingly, the city’s education leadership conveyed little of these developments. Mayor Bloomberg told reporters at the city’s press conference later in the day, “Nothing has changed. You’re writing a story about a change in definition.”

As far as he was concerned, they’d just moved the goalpost.

But the mayor was overlooking crucial signals from the tests. For one, the black-white performance gap doubled in math and grew by 50 percent in ELA. It wasn’t just that everyone went down the same. Many higher-needs students, who have been the focus of the mayor and chancellor’s education mission since 2003, were shown to be hovering just barely over the Level 3 line, the result of just enough test preparation but not enough solid education. When the bar was raised, their lack of mastery over grade-level skills and knowledge was cruelly revealed.

Here are the figures for the percentages of city students meeting standards, by race:

What they show is Black and Hispanic students lost much more ground, as a group, than did whites or Asians. If you compare the racial gaps from 2009 to the 2010 gaps you see:

MATH: The black-white gap widened to 34.1 points in 2010 from 17.2 points in 2009. The Hispanic-white gap widened to 28.3 points from 13.7 points in 2009.

ELA: The black-white gap widened to 31.6 points in 2010 from 21.9 in 2009. The Hispanic-white gap increased to 30.5 points from 22.8 points.

(Statewise, racial performance gaps widened as well, but not as much. In math, the state black-white gap widened to 30 points from 17 points. The Hispanic-white gap widened to 24 points from 13. In ELA the state black-white gap widened to 30 points from 22, and the Hispanic-white gap widened to 28 points from 21.)

The city’s other high-needs students lost more ground, proportionally, as well. English Language Learners, who have logged steady performance gains over the last several years, fell back hard, to just 13.4 percent meeting ELA standards from 34.8 percent in 2009, while English–proficient students lost only a third of their gains. Just 23.2 percent of special education students met math standards this year, a drop of two-thirds, compared to a 31% drop for general education students.

Test prep vs education

At the city press conference, NPR reporter Beth Fertig asked the mayor if teachers’ longstanding complaints of excessive test prep resonated with him now.

The mayor snapped that “the things we are focusing on are the basics and until kids can do the basics,” talk of a more well-rounded education is “nice,” but irrelevant. “If you want to teach children to hold hands and sing ‘Kumbaya,’” he said derisively, trailing off to the obvious point that these squishy skills are pointless until children’s test scores show they can read and add. Real life is full of tests, he said, and students have to learn to take them.

What he didn’t seem to acknowledge is that a test-prep curriculum has failed to even produce mastery of basics, as the new tests indicate. And by calling for a continuing focus on “the basics,” the mayor dodged the issue of whether prepping for a bad test is a good idea.

It’s not. What Steiner and other educators were saying rather bravely Wednesday about our state assessments is that they have failed. They don’t assess 21st-century reasoning or analytic skills and knowledge. Drilling for them — as teachers have said until they are blue in the face — isn’t leading to student success. One could argue that in the case of black and Hispanic children and special-needs students, it has amounted to fraud.

The good news is this is on track to change, spurred by state and federal efforts to fix the assessments and rewrite curriculum. It will take several years, and of course the outcomes are not guaranteed. But at least it is crystal clear now that the test-prep emperor has no clothes, and that is a major step forward.

Though she makes the absurd claim in the article that her mission to change public education started as early as first grade (while most of us were concerned about which cartoon lunchbox we would get), Moskowitz’s real mission is to increase her own political power. During her early 2000’s tenure on City Council, Moskowitz conducted a series of education oversight hearings. (Which, according to the article, satisfied her childhood “Watergate” fetish.) In many respects, she intended these hearings to be a launching pad to higher office — but the plan backfired, as her coarse personality turned off voters and resulted in a nine point loss in the race for Manhattan borough president. Her current resurgence of interest in educational issues is intended as a pathway back into the public light, and perhaps higher office.

Moskowitz’s contradictory views on standardized testing are one hint that her interest in public schooling is more about playing to political rhetoric than thinking about what urban students really need to succeed. Her story about quitting the PSATs “midstream” in high school because other students were cheating and her denials that her charter schools try to “game” the tests leaves out the story of how high-pressure, high stakes testing feeds the drive to raise scores at any cost — the same drive she now encourages at her own schools.

In fact, in the article the testing culture of Harlem Success resembles a bad version of Orwell’s 1984. Paul Fucaloro, director of instruction (and Moskowitz’s proclaimed “right hand”) is quoted as saying that “we have a gap to close, so I want the kids on edge, constantly. By the time test day came, they were like little testing machines.” At Harlem Success the boot camp for “testing machines” begins in kindergarten, “where they get drilled for two weeks on how to behave in the zero noise corridors (straight lines, mouths shut, arms at one’s sides) and the art of active listening (legs crossed, hands folded, eyes tracking the speaker).” The only teacher who went on record for this piece confirmed the brutality of the testing sweatshop. Her comments are shocking: “Life at Harlem Success is very, very structured, even the twenty-minute recess. Lunches are rushed and hushed, leaving little downtime to build social skills. Many children appear fried by two o’clock, particularly in weeks with heavy testing. We test constantly, all grades. During the TerraNova, a mini-SAT bubble test over four consecutive mornings, three students threw up. I just don’t feel that kids have a chance to be kids.” Harlem Success doesn’t allow for vacation, either — that’s prime test prep time. Students have only two days off; Christmas and New Year’s. If you are having trouble mastering an assessment and deemed one of “the real slow ones” by Fucaloro, you have to stay an extra 30 minutes past the dismissal time of 4:30 p.m. If Moskowitz believes that having students “on edge” and producing “little testing machines” through these harsh methods puts her schools on the forefront of education reform, she’s dangerously mistaken.

The emphasis on standardized testing in Moskowitz’s schools displays another glaring problem with her views on education reform — a blatant disregard for students with special needs. “I’m not a big believer in special ed,” Fucaloro says — and indeed, Harlem Success rosters include very few students with special needs as compared to the surrounding district schools. Fucaloro condescendingly claims that Harlem success fixes special needs kids by “undoing what parents allow kids to do in the house — usually mama.” The article, however, goes on to explain the real means by which Moskowitz’s schools solve the special needs “problem”: “When remediation falls short, according to sources in and around the network, families are counseled out,” and students are then “dumped” into nearby district schools to get the services they desperately need. If Moskowitz is really the savior of education, then why are these children ignored? Two words: test scores.

While Eva Moskowitz is no savior of public education, her misdeeds are helpful in one way — they point to what needs to be fixed in public education; over-reliance on standardized tests and ignoring students with the highest needs. School uniforms, pristine hallways and bathrooms, and gamed test scores does not equal education reform. In a world where perception is reality, Moskowitz might have some fooled. But this was never about getting it “right” for our kids. Raising test scores builds the illusion to the public that children are learning, but teachers and parents know better. In a culture where teachers are demonized for the nation’s ills, the truth falls flat. Opportunists, whether it is for financial or political gain, have rooted themselves in education. Who will save our children from such “saviors?”

The ideology to which I am referring of course is the penchant everywhere to replace real learning with a spreadsheet education: education by the numbers sliced and diced and then sliced and diced again. The enthusiasm is understandable; data has its appeals. It isn’t hard (I say this from experience) to get hooked into spending too much time clicking through Excel spreadsheets, looking for patterns, seeking the grail. Data appeals because it allows us to believe in — and even impose — a kind of order on the chaotic world in which we wake up everyday.

In the case of schools, the chaos we try to codify is education. “Well, look at that!” says the principal. “In your class, 25% of students got question #7 wrong, which tests their ability to know the meaning of words in context. Teach them that skill, and then…? Test them again.”

It is comforting, but it is wrong.

You can’t teach by the numbers. The tests are not reliable, and even if they were, you can no more translate an isolated skill into real knowledge than you can transfer attendance at a lecture about unicycles into riding down a Manhattan street on a unicycle, in the rain.

Teachers know that, but teachers have been forced.

And still, after eight years of this kind of failure, when TUDA shows — again — flat scores for NYC, and that the achievement gap has not narrowed one iota, the papers don’t even bother to ask a teacher what is wrong.

Hoxby, a Hoover Institution fellow and Stanford professor who has published extensively on charter schools (favorably) and teacher unions (unfavorably), looked at students who won admittance by lottery to certain New York City charters and compared their performance to students who applied but were not admitted.

As Jonathan Gyurko writes in an earlier post, she found an incremental scale-score improvement of 2.4 to 3.6 points (on a 325-point scale) more per year  in reading and math for charter pupils over those who lost the lottery and did not attend a charter. But she then projects that a Harlem student who attended charters from K-8th grade would make the same gains every year and could  narrow his or her achievement gap with Scarsdale students by 86% in math and 66% in ELA.

What Hoxby did was take this point difference, this “charter effect,” and present it as a persistent, undiminishing causal effect that can work educational miracles over eight years on the same student. And it’s unlikely she has test scores for very many students who’ve been continuously enrolled in a charter for eight years.

But she did have a nice construct, one that would make anyone sit up and take notice. Scarsdale is one of the top performing school districts in New York State, even in the United States. Its campus-like schools boast rich electives, high-tech labs and music rooms, green rolling playing fields, helicopter parents and relaxed, highly-paid teachers. Below, the Scarsdale High School library. Under that, a Harlem Charter School.

The gaps between Harlem and Scarsdale students are about far more than test scores, and closing the academic ones will take a lot more than test prep

SOME HARLEM-SCARSDALE GAPS

To perpetuate the fiction that if they could just attend charter schools, Harlem’s struggling students would morph into Scarsdale over-achievers (if indeed they even wanted to) is a disservice.

Hoxby’s study was hailed by the Wall Street Journal on page 2 and greeted at the final word on charter superiority in a Daily News editorial the next day.

But researchers know better. The black-white test score gap has been shown to persist even between middle-class blacks and whites, for deep and complex reasons. Ronald Ferguson, a Harvard professor who studied the racial achievement gap in Shaker Heights, Ohio  (Hoxby’s home town), says it would take at least 25 years to close the racial achievement gap, even between students of from families with similar incomes.

Ferguson found that blacks scored on average nearly 100 points below whites on SATs. Even in Shaker Heights the average grade for a black senior was C+ versus B+ for whites. The gap is not a result of effort — he found blacks studied harder than whites — but of the persistence of poverty’s ills even after incomes had equalized.

So by all means let’s equalize the resource gaps with Scarsdale. But it’s not right to generalize that a few points average gain on a standardized test means high-needs students will continue to make the same gains year after year. Nor that those decimal points are all it takes to close America’s racial achievement gaps.

Quite a lot, apparently, and it shows in New York City’s data. Every year, the DOE surveys teachers, secondary students, and parents to find out what the school looks like to the people who actually spend time there. Basically, the survey asks whether the school community is a welcoming one that holds high standards, inspires kids to learn, and cultivates collaborative culture. The city tabulates the results, combines them with attendance data and then converts that to a letter grade. In addition, the DOE gives schools a separate letter grade that is based on student progress on state exams from one year to the next.*

Compare those two letter grades — for environment and progress — and what you will find is that environment matters. The better the school community is at working together, the more likely it is that kids will make academic progress. Or to put it more harshly, when the environment deteriorates, so does the education students receive. Seventy-five percent of the middle schools that received an A for environment also received an A for progress on the state exams. That percent gets cut nearly in half, however, when the school environment rates a D. Only forty percent of D-rated environments earned an progress grade of A.

Now, I would hazard a guess that very few people reading this blog are much impressed by progress on the state exam, where scores shot up obscenely this past spring. They probably think, as I do, that though our students are doing very well, those scores do not reflect their work. Rather, they are the result of either dumbed-down tests, too much test prep, some kind of statistical phenomenon, or a combination of all three. But in this case, I am looking at comparisons of progress, and not the progress in and of itself. Comparisons show us that in middle schools where there is too little respect, collaboration, inspiration and communication, students didn’t do as well as they did in middle schools where the whole community feels more engaged in its own work.

This was a large sample of schools, and over 220,000 teachers, students, and parents completed the survey. Still, of course, the usual caveats apply. I’m an English teacher, not a statistician. Besides that, compiling this takes a lot of cut and paste, and though I am careful, one never really knows. So, if someone wants to double-check, be my guest.

Update: I neglected to include the five middle schools that received an F for environment.  Of those schools, 0% received an A for progress.

___________________________________________________

* Along with a third grade for overall achievement, these two letter grades, get rolled into a single final grade, and that is what the public generally reads about in the papers. Those final grades have little meaning for teachers, however because the environment accounts for only 15% of the total. The rest is based on the state exams, and those results have little relevance to our work.

In 2006, third-grade students had to get 43.6 percent of the points on the math test to earn a Level 2 — but by 2009, they needed to get only 28.2 percent of the points. On the English language-arts test, the cutoff to earn a Level 2 in sixth grade dropped from 41 percent of the points in 2006 to just 17.9 percent in 2009.

… In grades three through eight, the number of New York City students who scored at Level 1 in math fell by an astonishing 80 percent in only three years.

… Some of the city’s lowest-performing schools have few or no Level 1 students because the state lowered the bar.

Lower standards on state exams are not the mayor’s fault; nor are they Joel Klein’s. Rather they are what happens when two tests go high stakes to the virtual exclusion of all other indicators, as has been the case under NCLB.

But even if initial blame lies with NCLB, the city compounds it by basing its own high stakes decisions (in this case student promotion) almost exclusively on tests. Ultimately, the city undermines the very goal it seeks to achieve: higher standards. In fact, anecdotally, teachers have often told me that under this system more students wind up inappropriately promoted than ever before. After all, though it may be technically possible to retain a student who has achieved a Level 2, in practical terms it is very, very difficult. How can a teacher make an effective argument for retaining a child whose educational level has the official sanction of the city, as attested to by an official state exam?

In the Daily News article, state spokesman Tom Dunn said, “Our exams were not designed to determine whether students were ready for promotion.” Exactly so. And I would add that decisions about promotion must be based upon a broad spectrum of knowledge about a student’s academic abilities that cannot be wholly captured by a test.