Are Ratings from Tiered Quality Rating and Improvement Systems Valid Measures of Program Quality? A Synthesis of Validation Studies from Race to the Top—Early Learning Challenge States

Publisher: Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education
Apr 29, 2019
Authors
Lindsay Fox, Moira McCullough, Pia Caronongan, and Mariesa Herrmann

Key Findings:

  • All nine states used external measures of quality to examine the validity of their ratings; eight used an independently collected measure of program quality and eight used at least one measure of children’s outcomes.
  • The ratings distinguished between programs with differing quality; higher-rated programs had higher scores on independent measures of quality. However, the overall level of quality for higher-rated programs could not be described as high based on these independent measures.
  • The ratings were not related to differences in children’s outcomes; children who attended higher-rated programs did not have better developmental outcomes than those attending lower-rated ones.
  • Researchers from all nine states reported that their non-experimental designs limited the interpretation of findings. In addition, most researchers perceived recruiting child care providers for the studies and attaining sufficient representation across the rating levels as the most challenging aspects of the studies.

The usefulness of quality ratings for early learning and development programs depends on how accurately they measure programs’ quality, that is, their validity. To inform states’ continued development of tiered quality rating and improvement systems (TQRIS) and future validation studies, this report synthesizes findings from validation studies conducted by nine states that received Race to the Top—Early Learning Challenge (RTT-ELC) grants. It also describes the challenges that researchers faced when conducting these studies.