Last week, I spent time learning about the Department of Education’s new predictive and interim ELA assessments. According to Klein, the tests are “more accurate than what our schools have today” and will “make it easier for our educators to tailor instruction.” But Klein’s not a teacher, and frankly, I think that’s poppycock. The new testing system is one snazzy product, but in spite of all the money spent, I doubt it will make better readers and writers of our kids. In fact, it’s more likely to undermine that goal.
But first the good news because there really is good news. The DoE says the tests cannot be used against teachers, and to assure us of that, they say they will place the tests online in advance of their administration. Accessibility makes these tests low stakes and no stakes, strictly to be used as classroom tools. In addition, teachers will not be grading the tests. Multiple choice sheets will be picked up by courier and graded by the DoE’s vendor. The results – question-by-question and student-by-student breakdowns – will be available to teachers five days later, online. That’s the plan. It’s an expensive plan, and plans do not always work out as expected, but for now let’s assume swift couriers and smooth scanning. Let’s assume it works.
But if it works, that’s where the problem lies. Leaving aside for a moment the likelihood that the DoE has no plans to give teachers the time they would need to make meaningful use of the data, let’s consider an even more fundamental question: how useful is the costly data to begin with? What will it teach us about our students that we don’t already know?
The answer to my mind is that it will teach us very little. True, the DoE has attempted to tie every multiple choice question to a particular state standard. Theoretically then, right and wrong answers would reveal a student’s specific strengths and weaknesses. But while an overall multiple choice score might give us a broad sense of where students are at, it is useless to draw specific inferences from specific questions and then teach to the junk data that results. Individual multiple choice questions are something of a crap shoot, their reliablity dependent on everything from the phrasing of the question to a student’s urge to guess. Thus, will a multiple choice question tell us which students can evaluate an idea, or form one of their own? Can it even tell us whether students can identify a theme? Doubtful. Yet by just such doubtful data teachers must determine their instruction. Multiple choice testing items cannot bear that kind of weight.
In other words, the problem is not so much that the interim tests are comprised of multiple choice questions. Rather it is that teachers will be asked to use the fetishized results to plan instruction. And that’s ridiculous. It’s as if the DoE has built a huge and many-armed contraption just to wash a cup, or shell an egg.
Teachers don’t need a Dr. Seuss slicing-dicing miracle machine to decide what students need. Rather, teachers want to look at their students’ writing, and also to engage them in substantive discussions about the books they read. From these activities they learn the
strengths and weaknesses of their students, and plan accordingly. The DoE’s new tests do not forbid this kind of “data analysis” of course, but they do not enrich it either. And since time in education is a zero sum game, then for every day that teachers are forced to spend teaching kids on the basis of junk data, there will be one less day for real assessment and for delivering a real education.
On some level, the DoE must know this – though maybe not, given its lack of experience with real classrooms. Howsoever that may be, in its enthusiasm for turning all of education into a spreadsheet, the DoE has (once again) undermined the high standards it claims it wants. Gold-plated square pegs, surely, these assessments. But headed toward round holes, nonetheless.
One final note. The predictive tests (and possibly the other interims) do include extended response questions that schools can administer if they want to. But vendors can’t grade essays (too costly, too time consuming), and if teachers are to grade them, they must be given time. Teachers have other essays of their own to read, essays that will give them greater insight into what their students need. Remember, time is zero sum.
But whether or not teachers choose to use the canned essays, the new assessments are shifting the focus in our classes. Multiple choice is where the money is going. Multiple choice is what generates those computer quantities the DoE so dearly loves. From the multiple choice results, we’ll be forced to teach.
It’s a shame, isn’t it? The DoE will have spent many, many dollars parsing multiple choice questions where they will not find the data that they seek. Rather that data will be found where it has always been: in the interactions that happen (relatively inexpensively) between the teachers and their kids.
7 Comments:
1 Peter Goodman
· Oct 30, 2007 at 2:18 pm
Jackie
Maybe I sat in on a session with a better instructor … I was impressed … with a school system filled with newer teachers the ready availability of predictive assessment data, IF PROPERLY USED, can be useful.
In collabortive schools with Teacher Centers at the core of school PD data can drive and inform instruction …
What is sad is that some of Klein’s ideas, example, Inquiry Teams, are excellent ideas … mandating and driving down school throats guarantee failure.
2 rainyvines
· Oct 30, 2007 at 5:58 pm
I agree with Jackie Bennett (and it’s not the first time). I see many problems with this costly use of multiple choice data to “inform and drive instruction.” In addition to those she already mentioned, these tests are utterly inappropriate for beginner and intermediate-level English language learners, who may not understand the texts or questions in the first place.
Also, regardless of population, the tests assume a mindset that may be unnatural to young children and even teens. When children have difficulty locating the “main idea,” they may be blessed, not deficient. Children (even into high school) often find supreme importance in things that adults might consider irrelevant at best (ever read The Little Prince?) A skilled teacher can help them come to recognize overarching themes and concepts, without destroying their love for the particulars.
Multiple choice tests assume that children have the same conceptual hierarchy as adults (or should). Like so many other aspects of test-obsessed culture, these tests and the accompanying test prep disregard and destroy children’s individual ways of thinking.
Devil’s Advocate: Children should learn about “main idea” and such before they get slammed with it. They will need it in high school and beyond, so the earlier they get used to it, the better.
Refutation: Yes, they will need to use it. It does not follow that they need to begin in fourth grade, or even sixth. And it should be introduced in the context of compelling ideas and literature, not canned, pasteurized, adulterated, abridged stuff.
3 Jackie Bennett
· Oct 30, 2007 at 11:07 pm
Hi Peter – actually my instructors were wonderful and so was the workshop. It was professional. There was a presenter from the vendor who was smart and patient with our many questions and not for one second did I feel like I was being condescended to. And that is just as true for the DoE liaison who accompanied her in order to answer questions. She was great, very engaging. Both of these women exhibited enthusiasm and grace under pressure, even when my colleagues and I asked a lot of questions. These women were terrific.
But I disagree about the usefulness of the parsed data (won’t repeat my post) and I also disagree that this is what our younger teachers need. I am often extraordinarily impressed by our new teachers, and given the limited resources in any organization, I’d prefer the money had gone to supporting them in what they most seem to want support in, which is in how to handle 30 children at one time. Would love to know if the board is being as exacting with providing them mentors and assistance with class issues as it is with getting the numbers into ARIS in time.
(And here, here to rainyvines. The literature is in the language. Heaven save us from the pasteurized, adulterated and abridged garbage they pass off as reading material in English classes these days. I still believe in books, as you surely seem to. As to skills without context, and education without knowledge – a depressing subject, to take up in some later post.
4 NYC Public School Parents: Periodic assessments: a waste of precious time and dollars
· Oct 31, 2007 at 6:40 am
[...] piece by Jackie Bennett on the UFT blog Edwize about the so-called periodic assessments – which start next week. She says they will produce [...]
5 jd2718
· Oct 31, 2007 at 7:45 am
Certainly in math, there is no comparison between looking at a list of right and wrong answers, on the one hand, and looking at student work on the other. Only in the latter case do we know what’s going wrong.
If a multiple choice test analysis comes back saying that Steven is having trouble with fractions, what do we do? We test him with short answer questions so we can see his work so that we can find out what’s wrong.
Jonathan
6 Angie
· Nov 6, 2007 at 12:29 am
Jackie, I loved your article. You’re right on the money. I remember many, many years ago I had a student who struggled so hard, it broke my heart. He was in 5th grade, but was performing on a 3rd grade level. Well, when he got a high score on his citywide test, the principal and his parents were thrilled! I stared at them with my mouth wide open. These seemingly intelligent people were willing to believe one test score over a year of low performance. I was a brand new teacher, but I knew how ridiculous the whole thing was. Obviously he had guessed well on the multiple choice questions. It was luck that got him the high score, not intelligence. Maybe the people pushing these assessments got to where they are today because of multiple choice tests!!
Angie
7 Steve Perez
· Nov 9, 2007 at 1:29 pm
Angie, thanks for sharing your story – it’s a real reality check on the current testing mania.
Leave a Comment