External/Impact Evaluation | Addis Ababa Parental Coaching for Early Childhood Development Programme At Bernard van Leer Foundation

TERMS OF REFERENCE

External/Impact Evaluation | Addis Ababa Parental Coaching for Early Childhood Development Programme

Van Leer Foundation / 2024

Application Due: March 5th, 2024; EOIs will be reviewed on a rolling basis.

Timeline

Phase I – April-May 2024

• Review the programme documentation + logic model
• Inception Report
• Detailed evaluation plan – including baseline/endline method

Phase II – June-October 2024

• Implementation of the baseline study
• Baseline study report / Presentation of findings
• Implementation of the endline study
• Endline study report/ Presentation of findings

Phase III – December 2024

• Draft Report / Presentation
• Final report/ Presentation

Budget range: 75,000 to 90,000 euros (any travel to be covered within budget)

We would like to invite potential contract holders to submit a high-level estimate of envisaged costs or a few possible budget scenarios (any travel to be covered within the estimated budget) as part of their proposal.

Overview

The Addis Ababa City Administration has rolled out a comprehensive, integrated, mutli-sectoral early childhood development (ECD) strategy named ‘Children: The Future Hopes of Addis Ababa (ECD) Initiative’ to provide universal access to comprehensive ECD services for all children in Addis Ababa by 2026. Specifically, the City Administration aims to reach vulnerable children and their caregivers in 330,000 low-income households with intensified ECD services. The Parental Coaching for Early Childhood Development (PC-ECD) Programme, coordinated by the Bureau of Health (BoH) and overseen by the Strategic Programme Management Office (SPMO) based at the mayor’s office, is one of the critical interventions.

The PC-ECD Programme is a home-based intervention to break the inter-generational cycle of poverty. It supports low-income families with children ages 0 to 6 years to improve their caregiving behaviours, health, and skills to promote early child development. To achieve this goal, trained PC-ECD workers provide regular home visits to eligible families guided by these objectives: (i) Improve pregnancy outcomes by supporting pregnant women to improve prenatal health; (ii) Improve child development by supporting caregiver-child interactions through play and modelling positive caregiving behaviours and practices; and (iii) Improve families’ social context by supporting partnerships and cross-sectoral collaboration.

This evaluation aims to assess the effectiveness, outcomes, and impact of the PC-ECD Programme on participating parents and their children. The contract’s expected outcomes encompass interrelated phases of measurement guided by a process-oriented approach. Deliverables include interim reports and presentations of each measurement phase to inform ongoing work to key stakeholders. A draft and a final evaluation report summarizing the key findings on what works and what does not in the PC-ECD Programme are included. The evaluation should provide valuable insights into the Programme’s strengths and weaknesses and inform future improvements and decision-making.

About the Van Leer Foundation

Founded in 1949, the Van Leer Foundation works worldwide to ensure that babies, toddlers and the people who care for them have a good start in life. Our mission is to improve opportunities for all young children, especially the millions of young children growing up in circumstances of social, economic and environmental disadvantage around the world.

We believe the major challenge is the transition to scale of support for babies, toddlers and those caring for them. By working together with governments, civil society, philanthropy, academia, international organisations and business, we find the best programmes and policies to improve children’s lives. We partner with them by providing funding, expertise, networks and support to scale these up for maximum impact.

Geographically, our investments are made in focus countries, selected to reflect global diversity in economic, geographic and cultural terms, and a global programme through which we engage strategic replication partnerships that span multiple countries, in some cases extending beyond our core set of countries.

Responsibilities

Overview

The Van Leer Foundation seeks a qualified team to design and implement the PC-ECD Programme external/impact evaluation, inform ongoing work and partial results to key stakeholders, and report the final findings. We are particularly interested in approaches that include process and outcome indicators that help understand the Programme’s effect on parent-child interaction (covering the four key priority behaviours[1]), child development, and staff adherence to the Programme’s protocols.

Overview of the scale of evaluation

The PC-ECD Programme has three management levels. A technical team based at the BoH oversees the general coordination (central level). Each of the 11 sub-city health offices should have one Programme coordinator (intermediate level). Implementation teams – one trained supervisor and PC-ECD workers – respectively supervise and deliver the intervention at the community level. PC-ECD workers are assigned to the beneficiary families per block. Currently, 1988 PC-ECD workers, 62 supervisors, four sub-city health office coordinators, and three BoH specialists have been hired for the Programme. The potential Programme coverage is around 111,328 households currently.

Goals

The primary goals of the external/impact evaluation are as follows:

• To identify, in collaboration with the BoH, SPMO, VLF, and other stakeholders, a set of quality and impact indicators that can be monitored and used for adapting the PC-ECD Programme in the future. The Programme’s logic model should include the essential indicators.
• To assess the effectiveness and the costing data of the PC-ECD Programme in achieving its stated goals and objectives, according to the agreed Programme’s logic model.
• To measure the short-term and intermediate outcomes for participating parents and their children.
• To identify the factors that contribute to the success or challenges of the Programme. Explore the underlying mechanisms and processes through which the parenting programme brings about changes. This involves analyzing the fidelity of programme delivery and components such as curriculum, activities, dosage (i.e., frequency, duration, and intensity of the visits provided to families), supervision, and workforce competency (knowledge, skills, and attitudes). Identify strengths and weaknesses of the PC-ECD Programme and provide actionable recommendations for its improvement. This may include suggestions for refining programme content, delivery methods, and outreach strategies.
• To provide evidence-based recommendations for programme improvement and decision-making. Offer evidence-based recommendations to programme administrators and policymakers for making informed decisions about programme continuation, expansion, or modification.

Tasks and deliverables

Phase I

• Review the programme documentation, materials, and curriculum.
• Plan and conduct a validation workshop on the PC-ECD Programme’s logic model. Agreeing on a feasible logic model before starting the evaluation will help define the goals and outcomes and measure the impact of the PC-ECD Programme correctly. The programme’s objectives serve as benchmarks for assessment (if the logic model is not yet finalized, this workshop will be a consultative meeting to finalize it)
• Review the ECD-related indicators of the Integrated Monitoring Framework of the ‘Children: The Future Hopes of Addis Ababa (ECD) Initiative’ under development by Lalibela Networks Plc and the monitoring of the PC-ECD Programme carried out by local managers to seek integration and consistency with the external evaluation. Provide a synthesis report with recommendations.
• Prepare and approve the inception report. It is an initial planning document outlining the scope, objectives, methods, and critical aspects of the external/impact evaluation, including the baseline study. This planning document must be presented and approved by the key stakeholders (BoH, SPMO, VLF, among others). Its purpose is to provide a roadmap for the evaluation process and ensure that all stakeholders are on the same page regarding the evaluation.
• Develop, finalize, present, and approve a detailed evaluation plan. Comprised of:
• Introduction
• Scope and Objectives (specify the main objectives or research questions that the evaluation will seek to answer)
• Methodology for baseline and endline studies and wider evaluation (describe the research methods and data collection techniques that will be used in the evaluation. This section should include details on the sampling strategy, data sources, and any tools or instruments that will be employed. Identify/adapt a reliable and valid instrument to evaluate the interaction between parents and children and a valid instrument to measure child development)
• Data Collection Plan (provide a detailed plan for data collection, including timelines, responsibilities, and any ethical considerations. Comparative analysis, if applicable, with a control group or baseline data)
• Data Analysis Plan (outline how the collected data will be analyzed and interpreted to answer the evaluation questions. This can include statistical methods, qualitative analysis techniques, and any software to be used)
• Stakeholder Engagement (describe how stakeholders will be engaged throughout the evaluation process. This may involve consultations, workshops, or feedback mechanisms to ensure that their perspectives are considered)
• Risk Assessment (identify potential risks or challenges that may arise during the evaluation and propose strategies to mitigate them. This could include issues related to data access, ethical concerns, or logistical challenges)
• Timeline (present a timeline that outlines the key milestones and deadlines for the evaluation, including data collection, analysis, and reporting)
• Budget (provide a budget estimate including costs related to personnel, data collection, analysis, and any other expenses. This helps stakeholders understand the financial aspects of the evaluation)
• Reporting and Communication (explain how the evaluation findings will be reported and communicated to stakeholders, including the format of the final report and any interim updates)
• Roles and Responsibilities (define the roles and responsibilities of the evaluation team members, including their specific tasks and areas of expertise).

Phase II

• Conduct the baseline and endline study. Collect data from the programme’s participants, including surveys, interviews, observations, and/or focus groups; analyse data and reporting.
• Present the baseline and endline study reports and discuss the findings with the key stakeholders (BoH, SPMO, VLF, among others).

Phase III

• Reflection on results Hold strategic consultation/meetings with the PC-ECD Programme coordination team and relevant stakeholders running the programme in the municipality to reflect on the outcomes of observations, focus groups, and measurements in real-time to inform and course-correct on a more frequent basis.
• Develop and present a preliminary report.
• Develop a final evaluation report incorporating the suggestions provided by key stakeholders.

Additional Asks

• Participate in periodic meetings (every two weeks) with key stakeholders (BoH, SPMO, VLF, among others).

Deliverables will include:

• A synthesis report of the ECD-related indicators collected in Addis Ababa with recommendations on consistency between the ECD indicators included in the Integrated Monitoring Framework by Lalibela, the monitoring data used by the Programme’s managers, and the evaluation’s potential process and outcome indicators.
• Jointly developed plan (agenda, list of stakeholders) of the PC-ECD Programme logic model consultative meeting.
• The PC-ECD Programme logic model.
• Inception report/evaluation plan (methodology/matrix of the evaluation study, including the baseline and endline study and its instruments for data collection) and a PowerPoint presentation summarizing the main content.
• Baseline and endline study reports and PowerPoint presentations summarizing the main content.
• Preliminary report/draft of final report findings and a PowerPoint presentation summarizing the main content; meeting with stakeholders to discuss.
• A well-written final report of the evaluation findings.
• A 1–2-page executive summary of key impact.
• Any supporting data visualization.
• Any additional material, including interview transcripts, collected data, workshop reports, etc.

Note: although the data collection will be in Amharic, all the reports and presentations should be done in English.

Primary audience: Policymakers and managers of the Addis Ababa Bureau of Health, SPMO, and the local government in general, the Addis Center of Excellence, the VLF team, and our partners from Big Win and ELMA Philanthropies.

Qualifications

Education and work experience

Interested individuals and/or organisation should demonstrate their qualifications and proven experience in research, evaluation, and writing.

There is a possibility of proposing semi-experimental methods, if appropriate. The PC-ECD Programme will continue to be gradually scaled up/expanded, and this could present opportunities for a more rigorous and interesting evaluation approach (taking waitlisted or later-phase families as controls) than just a pre-/post.

Core skills

• Expertise in Programme Evaluation: The firm should have a proven track record of evaluating complex social programs, including those focused on parenting and child development.
• Subject Matter Expertise: Understanding of ECD and parenting programmes, child development, and relevant theories in the field of family studies to assess programme components effectively.
• Evaluation Planning: Effective project management skills to plan and execute the evaluation within established timelines and budgets.
• Ethical Considerations: Knowledge of research ethics and the ability to navigate ethical issues related to informed consent, confidentiality, and data protection when working with vulnerable populations.
• Data Collection and Analysis: Proficiency in data collection techniques such as surveys, interviews, focus groups, observations, administration and coding of standardized measurements. Expertise in both quantitative and qualitative data analysis, including statistical analysis.
• Stakeholder Engagement: The ability to engage programme staff, participants, and other stakeholders collaboratively and constructively throughout the evaluation process.
• Data Visualization and Reporting: Proficiency in creating clear and informative reports and visualizations to communicate evaluation findings to both technical and non-technical audiences.
• Cultural Competency: Sensitivity to cultural diversity and understanding how cultural factors may influence programme outcomes and participation.
• Collaborative Work: The evaluation firm should be able to work collaboratively with VLF, SPMO, and BoH to ensure that the evaluation is helpful and informative. The firm should be willing to listen to the concerns and suggestions and to adjust the evaluation plan as needed.