A recent blog by
Stephen Sawchuk cautions that even the analysis of a data-rich test like NAEP
(the federally administered National Assessment of Educational Progress) can collapse
under misinterpretation.
NAEP provides an informational
goldmine. Since it’s administered to a sample of students across a broad swath
of districts and states every year, it enables comparisons of achievement
trends across states. And because NAEP has no accountability stakes it’s less
susceptible to the gaming and cheating that may compromise other exams.
But Sawchuk cautions that some
analysts’ interpretations are venturing beyond what NAEP can support. Policy
advocates of all stripes have laid claim to NAEP as a proof point for various
positions. However, NAEP only provides evidence of a correlation between scores
and policies – and as anyone who’s taken Stat 101 knows, correlation does not prove causation.
Another frequent misstep is overplaying the
longitudinal nature of the test – while it provides data from multiple years,
it doesn’t test the same group of students.
So to all those NAEP data
prospectors out there, remember that while NAEP is rich in information, if you
try to oversell what NAEP can offer, you may find yourself left with fools’ gold.
