Does the New York Times Know What a Good School Looks Like?

We’re usually fans of The Upshot section of the New York Times, which focuses on making complex data accessible through clear, interactive graphics. But a recently published piece on home prices and student achievement in various urban areas left us and others disappointed.

The basic question: Where can families find affordable housing near excellent schools? To answer it, The Upshot uses home price information from Redfin and student achievement results from the Stanford Educational Data Archive to try to identify “suburban sweet spots” where there are “schools that deliver on quality with homes that are relatively cheap.”

So what's the problem?

The problem is how The Upshot approaches school quality. Their results rely on average student performance across entire districts.

Well, first of all, just because the typical student in a school is two grades ahead of schedule doesn’t mean that if you move your child to that school, he will immediately zoom ahead, too. That’s like assuming if you move onto a street full of exercise fanatics, you’ll automatically get into better shape. Second, within districts, school performance can vary considerably. And third, how well students do on tests is simply a narrow judgment of school quality.

In reality, student performance arises from many factors. In many cases, the students who show excellent outcomes are the same ones who enter school well-equipped to succeed; they’ll do well no matter what happens at school.* In other cases, schools lift students to greater outcomes because they are truly, well, great schools.

Stanford researcher Sean Reardon (whose data is used in The Upshot’s article) nicely summarizes the core issue in his comment on the New York Times’ site:

“The article repeatedly conflates test scores with "school quality." This is a common, but significant, error. Average test scores in a school district are better thought of as a reflection of the sum total of educational opportunities and resources children in those communities have had to learn the tested material -- opportunities and resources in their homes, pre-school programs, neighborhoods, afterschool programs, and - yes - their K-12 schools. To attribute test scores solely to "school quality" ignores the powerful role that family background plays in shaping opportunity.”

If you are a parent, at best these graphics can point you to communities where the other kids are scoring well on tests. That’s not nothing. Students with successful peers do tend to perform slightly better, and to plenty of parents, that’s enough. But you deserve better data. Sitting next to a high-achieving friend is not a substitute for having fantastic teachers, interesting stuff to learn, and a culture of high expectations.

Let's ask a better question

What families really want to know is, to what extent does each local school district help students learn more than they would learn somewhere else? That’s a different – and trickier – question. To answer it, you need to know how students were performing before they entered a district and what assets they had at home. For instance, children of very well educated parents tend to do better no matter where they attend school, because of they come from an educated family.

The Upshot did not include any Louisiana cities in its analysis. However, if you live in Louisiana, we can lend a hand. We’ve published letter grades for each K-8 public school, and those grades take the rate of student learning into account.

Otherwise, we’ll have to wait until The Upshot gets hold of student growth information for each school district nationally. When that happens, we’re confident they’ll create a much-improved version of this analysis—one that focuses on schools that seem to be boosting students to success, and not the other way around.

* Also worth keeping in mind is the fact that even the most privileged American kids lag far behind students in other countries in academic performance, which raises the question of how good our “good” suburban schools really are. But that’s another post for another day.