A recent blog by Stephen Sawchuk cautions that even the analysis of a data-rich test like NAEP (the federally administered National Assessment of Educational Progress) can collapse under misinterpretation.
NAEP provides an informational goldmine. Since it's administered to a sample of students across a broad swath of districts and states every year, it enables comparisons of achievement trends across states. And because NAEP has no accountability stakes it's less susceptible to the gaming and cheating that may compromise other exams.
But Sawchuk cautions that some
analysts' interpretations are venturing beyond what NAEP can support. Policy
advocates of all stripes have laid claim to NAEP as a proof point for various
positions. However, NAEP only provides evidence of a correlation between scores
and policies - and as anyone who's taken Stat 101 knows, correlation does not prove causation.
So to all those NAEP data prospectors out there, remember that while NAEP is rich in information, if you try to oversell what NAEP can offer, you may find yourself left with fools' gold.