Evaluation relies on two sources of data:
Syllabi for all required courses within the program that address literacy instruction
All required textbooks for each required literacy course
Methodology in Brief
A team of analysts use course catalogs to determine the required coursework for each elementary program we are evaluating. Analysts then read course titles and descriptions to pinpoint courses that address reading instruction. Textbook information is gathered through syllabi and university bookstores.
A separate team of expert reading analysts — all professors and practitioners with advanced degrees and deep knowledge of how children learn to read — evaluate reading syllabi and textbooks using a detailed scoring protocol.
Fifteen percent of syllabi are randomly selected for a second evaluation to assess scoring variances. Each course is analyzed for its coverage of each of the five components of early reading instruction, as identified by the National Reading Panel (2000): phonemic awareness, phonics, fluency, vocabulary, and reading comprehension. Course analysis focuses on three main elements:
Use of course time to address each component, as specified by the lecture schedule.
Whether students are required to demonstrate knowledge of individual components through assessments, assignments, or instructional practice
If the assigned text or texts accurately present the components of reading instruction. Ratings of reviewed reading textbooks are provided here.
Reviewers analyze every required textbook for its coverage of the science of reading. The process of reviewing a book follows these steps:
The reviewer ascertains if the text can be used either as a 'comprehensive' text (covering all five of the components as well as analyzing how the text approaches assessment and strategies for struggling readers), or if the text is designed only to teach one or a combination of the components, but not all ('specialized').
The reviewer determines if the content defines and presents each component in light of the science, shedding old unproven practice and advancing a depth of knowledge not only about how students learn to read, but specifically how to teach students to read -- not just guide, encourage, or support.
References are perused for primary sources, researchers, and trusted peer-reviewed journals that present the consensus around the science of reading.
Each of the five components is assessed separately within each course. Points awarded for use of course time, demonstration of knowledge, and text coverage are combined to create five separate component scores (phonemic awareness, phonics, fluency, vocabulary, and reading comprehension) for each course. If a program includes multiple reading courses, the program score for each component is determined by the highest course score for that component. The five program-level component scores are used to determine the overall grade.
Scoring with less information
Due to the critical importance of reading instruction, NCTQ developed a means of evaluating elementary programs on this standard in cases where course details are missing from submitted material or where we could not obtain all reading syllabi.
This process relies on two key sources of data:
The syllabus for at least one course focused on foundational literacy. The syllabi for peripheral courses that may touch on literacy instruction, but are not core foundational literacy courses, are never substituted.
The assigned textbooks for all required literacy coursework. Where this information cannot be sourced from a syllabus, we identify the required textbooks using the institution's bookstore.
If we cannot obtain both pieces of information, the program is not scored.
Non-traditional programs
Our analysis of non-traditional programs only considers the coursework that is required before candidates become teachers of record. Reading instruction is simply too important for teachers to be learning while on the job. To account for the limited time-frame to complete such coursework, we additionally consider the requirement of a passing score on a reading-specific licensing test prior to entering the classroom.