Our “Outside Auditor” Is About To Report

Math and reading scores on the 2007 National Assessment of Educational Progress (NAEP) are set for release Tuesday (9/25), and should function as a revealing “audit” of the New York State ELA and math tests following two years of increasing controversy over the state assessments.

When the state reading scores rose dramatically in 2005, the N.Y. Times featured City Council hearings in which experts testified about their concerns over the possible gaming or misuse of state tests in an increasingly high-stakes, driven education environment. More recently, the Daily News called into question the comparability of math tests from year to year. And the N.Y. Sun, citing an internal UFT research report, noted just recently that there were questions about the comparability of 4th grade ELA tests from 2004 to 2005 after the city’s 10-point gain.

In schools, there has been widespread skepticism about the validity and reliability of state tests, especially as they become more and more the sole measure of student, teacher and school performance. It seemed to many teachers that the 2005 ELA test was easier. The state does not release the details of how it scores the tests so there isn’t any surefire proof. But the fact is that a 10-point gain in one year is suspect. Tests are not perfect instruments, and they cannot always be perfectly adjusted in the scoring.

What will we learn when the 2007 results come out Tuesday? NAEP scores are reported at the state level only–there will not be results for New York City, for example–but they measure the same 4th and 8th grade achievement that the state tests do. And in 2005, when NAEP reading and math results were last reported for 4th and 8th grades, many states, including New York, were embarrassed to see they had far lower student achievement levels on the national tests than their state results had shown.

For example: While there was a 6-point gain in the percentage of New York state’s 4th graders meeting reading standards on the state tests from 2003 to 2005, NY State’s 4th grade NAEP scores for the same period were completely flat. While 8th graders gained 3 points on their N.Y. state reading tests from 2003 to 2005, they dropped two points on the NAEP reading exam. In math, New York state tests showed 4th graders improving seven points; on NAEP it was less–just three points. For NY state 8th graders, a 5-point gain on the state math tests was contradicted by a 1-point drop on NAEP math.

NAEP is not perfectly aligned with our state tests. It has three achievement levels–”below basic,” “basic” and “proficient”–rather than the four levels N.Y. state uses. “Proficient” under NAEP represents a much higher scale score on that test than a Level 3 represents on the state test. But what is completely comparable is the direction of a gain or loss. When the state says that 8th graders went up in reading and math and NAEP says they went down, there is a problem.

In preparation for Tuesday, here is the percentage of N.Y. state 4th and 8th graders meeting state standards for the two years from 2005 to 2007:

Grade 4 ELA 70.4 to 68.0, down 2.4 points

Grade 8 ELA 48.1 to 57.0, up 8.9 points (wow)

Grade 4 math 84.9 to 79.9, down 5 points

Grade 8 math 55.5 to 58.8, up 3.3 points

Did the state’s 8th graders really make that progress? No one can tell, but flag that 8th grade reading result. That is a big move for a short, two-year time frame.

No matter how they come out, the NAEP results will also throw fuel on the No Child Left Behind reauthorization fire, which is already white-hot. U.S. Education Secretary Margaret Spellings used the 2005 scores to “prove” that NCLB was working. Others tried to prove her wrong. In this round, the tests will be used to “prove” that NCLB is, or is not, improving student achievement across the nation and should, or should not, be reauthorized.

Hang on to your hat. The ride is about to get fast and bumpy.