Asking Hard Questions About “What Works”

[Editor's note: Data analysis conducted by Rhonda Rosenberg.]

This week, the Daily News published yet another editorial taking an unjustly negative view of district schools in comparison to the charter sector — in this case, arguing that the relatively high proficiency levels in upper grades at schools in the Harlem Success and Harlem Village charter chains are primarily due to those schools’ extended days and school years. However, the latest available official data indicates that the schools in these two chains are also characterized by lower proportions of high-needs students than local district schools, and by extremely high rates of student attrition over time — in one case, a 68% drop in cohort size between 5th and 8th grades.

Even charter advocacy groups such as the New York City Charter School Center have warned against making simplistic assumptions about the causes of charters’ performance on state tests, noting in their own thoughtful analysis of this year’s results that

it would be inaccurate to draw absolute conclusions about school or sector superiority solely from these data points. There are important differences in demographics and enrollment practices that need to be taken into account in making policy conclusions. In order to control for these variables deeper analysis is required.

Using unofficial data for 2010-11 (which has not yet been made available to the general public), they concluded that

In citywide comparisons, charter schools have lower enrollment rates for students with disabilities, much lower rates for English Language Learners, and higher rates for students whose family income makes them eligible for Free or Reduced-Price Lunch (FRPL), though very slightly lower rates if considering only those families eligible for Free Lunch. How a particular charter school compares to nearby district schools will always depend on the school, neighborhood, and grade level in question.

With that in mind, it’s important to consider the comparison between the two chains praised by the Daily News and the district schools in their Harlem neighborhoods. The latest official data available from the state (from 2009-10) indicates that all of the schools in these chains have demonstrably lower proportions of the city’s neediest students — those eligible for free lunch, English Language Learners, and students who receive special education services in self-contained classrooms or those with collaborative team teaching — than the average district school in Harlem.

Harlem Success and Harlem Village vs. Neighborhood Demographics, 2009-10

Area	School Name	% FRPL	% FL	% RL	% LEP	% SC/CTT
	Harlem Success Academy 1	76%	63%	13%	2%	0%
	Harlem Success Academy 2	78%	66%	12%	3%	0%
	Harlem Success Academy 3	74%	59%	15%	4%	0%
	Harlem Success Academy 4	74%	58%	16%	4%	0%
	Harlem Village Academy	68%	55%	13%	3%	0.30%
	Harlem Village Academy Leadership	76%	61%	15%	2%	0.30%
HARLEM DISTRICT SCHOOLS AVERAGE:		83%	78%	5%	12%	13%

Even more striking, however, are the rates of student attrition at these schools. Both chains made their reputations based on the rising proficiency scores among their earliest cohorts of students — but an examination of state testing and enrollment data reveals that the proportion of students who have disappeared from those classes over time is disturbingly high.

Harlem Success Academy 1

		Cohort Size
		2006-07	2007-08	2008-09	2009-10	2010-11	% Decrease
Entrance Year	K 2006-07	83	80	79	63	58	-30%
	Gr 1 2006	73	73	62	59	42	-42%

Perhaps the most shocking statistic is the disappearance of students from the cohort of students who began 5th grade at Harlem Village Academy in 2006-07. Of the 60 students who took the state math tests that year, only 19 remained to take them in 8th grade — a 68% drop.

Harlem Village Academy

		Cohorts (Gr 5-8)
		2003-04	2005	2006	2007	2008	2009	2010	2011	% Decrease
Entry Year	2003-04	73	48	46	39					-46%
	2005		52	49	38	30				-42%
	2006			58	43	44	39			-33%
	2007				60	43	29	19		-68%
	2008					71	40	48	44	-38%

Harlem Village Academies Leadership

		Cohorts (Gr 5-8)
		2006	2007	2008	2009	2010	2011	% Decrease
Entry Year	2006	59	49	46	42			-29%
	2007		51	45	56	39		-24%
	2008			62	56	47	41	-34%

While it is possible that the declining cohort sizes are caused by reasonably higher levels of student retention, such a policy (if no students were actually leaving the school) would result in a visible increase in size for the following year’s cohort — the opposite of the pattern which shows up in this data. Another possibility is that extremely high retention rates are repeated on an annual basis — that is, the spots of students held back each year are taken by students held back from the previous grade. In the case of these two schools, this would mean that a quarter to over two-thirds of students at these chains have been held back over the course of their time at the school. Even if some of the attrition is caused by retention and some by students leaving the school, it is likely that the students most likely to leave are those who are told they will be held back if they stay — by definition, those students with the lowest academic performance, further raising the schools’ proficiency levels over time.

While it is vital to look broadly when seeking the most effective means of educating all of our city’s students, it is also vital to acknowledge that some current charter sector characteristics can’t (and shouldn’t) be replicated in district schools — including enrolling systematically lower proportions of high-needs students, a failure to replace students who leave their schools or are held back, and (possibly) extraordinarily high rates of student retention or the departure of students who have been unable to meet the school’s performance requirements. With last year’s passage of a charter law which requires authorizers to hold the state’s charter schools accountable for these patterns, hopefully we will now be able to move forward in having a more honest and productive conversation about what truly works in educating our students.

Tagged: attrition, Daily News, ELL, Harlem Success Academy, Harlem Village, special education