Methodological challenges associated with assessment of the impact of a parental education program on child development and psychosocial stimulation: Findings from a secondary analysis of data from a quasi-experimental program evaluation in Tijuana, México

Abstract

Background: The director-general of WHO recently declared that attainment of a child’s full developmental capacity is a human right, and is critical to “equitable prosperity and sustainable progress of societies.” Inadequate psychosocial stimulation has emerged as an important modifiable risk factor for poor child development, but best practices for scale-up of practical and cost-effective interventions are still undefined. We aimed to determine whether a parental education program developed and administered by World Vision (WV) under field conditions resulted in improved child development. Methods: We conducted a secondary analysis of de-identified data derived from a quasi-experimental study conducted in two communities (intervention and comparison) in Tijuana, México. The intervention consisted of a parental education program. Data on child milestone attainment and household characteristics were obtained from maternal self-report in baseline and post-intervention household surveys administered to randomly selected households with young children. The primary analysis employed a difference-of-differences approach to compare change in the proportion of children meeting milestones between baseline and endline assessments in intervention vs. comparison communities, in multivariate logistic regression models controlling for potential confounders. Findings: Data were available for 396 households with children 6 to 23.99 months of age (comparison: 120 baseline, 36 endline; intervention: 109 baseline, 131 endline); 375 were retained in the final multivariate analysis. In the final model, the odds ratio (OR) for attaining all developmental milestones evaluated at endline vs. baseline in intervention vs. comparison communities was 3.70 (p = 0.02, 95% confidence interval [CI] 1.27, 10.78) in a model including child age, maternal depression, and provision of iron-rich foods (all significantly associated with milestone attainment). The final model was derived from unplanned analyses, because observed odds ratios for associations between covariates within the psychosocial stimulation domain and milestone attainment varied significantly and inexplicably by arm and time in the prespecified multivariate model. There was no statistically significant association between household-level exposure to the intervention and milestone attainment. Examination of the comparison community at endline revealed that it appeared to differ in significant and fundamental ways from all other study subgroups for reasons we cannot fully explain within the constraints of this data set. Interpretation: Assessment of the impact of the WV program was significantly limited by methodological constraints related to study design and implementation. While findings of multivariate models suggest that the WV intervention was associated with improvements in child development within the intervention community relative to comparison community, we are unable to definitely attribute these to the WV intervention. Ultimately, the overall relationships regarding psychosocial stimulation, exposure to the intervention, and child development within this data set did not form a consistent pattern. While the methodological weaknesses of this study prohibit us from drawing robust inferential conclusions, this analysis provides important insights into some of the common challenges faced by research seeking to evaluate early child development programs. We end with a section of recommendations for future studies that draw attention to some of the key lessons learned in this study in an effort to help future studies avoid similar pitfalls.