After interviewing the key players who have developed and conducted this ambitious study, and reviewing all reports and instruments, we have identified four key features that make this study distinctive.
TOP staff noted that communication is key to this relationship. TOP built relationships with school districts, schools, teachers, and administrators, educating them about the goals of the study and why it was important to their overall mission.
Parents were provided with a longitudinal study brochure explaining the premise of the study. TOP program staff noted that generally the only families who do not give consent were guardians of foster children. In order to keep current TOP graduate and parent contact information, TOP staff sends out reminder postcards annually, and maintain a Facebook page (Top South, Top North, Top Northwest) to reach out to parents. Family consent was only half the battle. In order to collect data for the study, school districts’ and teachers’ willingness to cooperate was crucial.
In addition to education and incentivizing participation, WSU researchers have worked to make the teacher survey less cumbersome, so it takes less time for them to complete. Teacher participation has also been excellent. One TOP staff stated that teachers want more TOP children in their classrooms because of their readiness for school.
Targeted Use of Professional Researchers
A second key feature of the TOP study is the targeted use of professional researchers. Rather than simply handing over the reins of the project to WSU researchers, TOP has collaborated with them on the design, data collection, and communication of the results. There are multiple benefits to this approach of using professional researchers in targeted ways. It seems to create a nice collaborative relationship in which the researcher is closely working with staff to ensure they are producing something useful to them. Families and school districts may be more inclined to participate in a study when approached by an excellent local preschool, rather than a previously unknown university researcher.
Strategic Approach to Data
A theme throughout our interviews with TOP staff and WSU is focusing on the key information needed to understand TOP’s impact.
- Data or analyses are not strictly necessary to convey the point of the project and may be a waste of effort.
- Being selective about what data are presented is key to communicating to a broad audience.
It simplifies data collection greatly, and likely makes it easier to get district and teacher cooperation. Furthermore, it may be true that including individual-level variables is just not that important to persuading others.
There are multiple benefits to this strategic approach to data. It offers necessary focus: on the goals of the project; on respecting time and effort of partners and staff; and finally, on the intended audience. This is not a study designed to sit on a shelf; it’s a study designed to persuade laypeople about the value of high-quality preschool. Second, a strategic approach to data has offered staff and evaluators the flexibility to change what’s not working, or could be working better.
Commitment to Demonstrating Outcomes
Demonstrating outcomes is a central value of this organization. Being able to track and demonstrate effectiveness was always a secondary goal of TOP. The value of doing so seems self-evident to every person we spoke to: all perceived that continued support of TOP is dependent on being able to show long-term results on issues the public cares about.
The study aims to produce convincing data that quality early childhood programs improve children’s well-being – and doing so is considered to be essential to TOP’s ability to continue, grow, and thrive as a program.
This level of commitment to demonstrating results can be rare in the world of early childhood services. Many early childhood program staff worry that evaluation takes time and money away from the real work of the organization – the work that will directly benefit children. In a context in which everything is tight – time, money, expertise – and the need is vast, growing, and desperate, it can be difficult to justify directing resources towards evaluation.
In a world in which demand is great and resources are limited, it is crucial that programs be as effective as they can be in helping their target populations. This requires tracking outcomes continuously, systematically weighing the evidence, and continuously looking for ways to improve. Evaluation is part of how you make sure you are helping kids to the best of your ability.