Program Evaluation- READY Initiative At Save the Children

Timeframe: December 2021-March 2022
Level of Effort: Up to 35 working days
Location: Remote – available to work across various time zones in the Americas, Middle East, Africa

I. Background
In 2018, the United States Agency for International Development’s Bureau for Humanitarian Assistance funded Save the Children to lead a three-year initiative to augment global capacity to respond to major disease outbreaks. Through a consortium of partners, the READY initiative supplements existing efforts to strengthen coordination between global humanitarian outbreak structures and operational organizations responding to outbreaks. READY is building and retaining capacity among operational consortium members, non- governmental organizations (NGOs), and other stakeholders to respond to major outbreaks more quickly and effectively, including COVID-19, with an integrated approach that addresses holistic needs of affected communities—with community engagement and communications at the center. For more information on READY’s work, please visit https://www.ready-initiative.org/.

Over the life of the initiative, READY has carried out routine project monitoring and activity specific evaluations for key activities, such as the Outbreak Preparedness Planning/COVID-19 Readiness Workshops, the COVID-19 Micro-Trainings, and the Epidemiology and Modeling of Infectious Diseases in the Humanitarian Setting training series. Additionally, lessons learned from internal, activity-specific after-action reviews and external participant feedback have been used to adapt and strengthen READY approaches and activities/events. As READY approaches the end of the initiative in March 2022, we are seeking a Consultant to lead on an evaluation of READY’s work as a whole.

II. Evaluation Purpose
The purpose of this evaluation is to identify to what extent engagement in READY activities and events successfully augmented NGO readiness and capacity for outbreak response. READY wants to identify what we did well, what we can improve on, and where we can add the most value for future projects.

III. Evaluation Type
READY plans to carry out an external, mixed methods performance evaluation to review how well the initiative achieved its objectives, identify challenges faced, and highlight lessons learned and best practices for future programs of a similar nature. Though the methodology will be finalized with the support of the selected Consultant, it is anticipated that the evaluation will largely rely on qualitative methodology with a focus on interviews and discussion with key stakeholders.

IV. Evaluation Questions
READY is interested in understanding the relative effectiveness of its various activities in strengthening NGOs’ ability to respond to outbreaks, within each of the project’s three objectives: 1) To improve I/NGOs’ operational and technical coordination with critical disease coordination platforms, 2) To strengthen the operational capacity of organizations to more efficiently launch an integrated humanitarian response to infectious disease outbreaks, and 3) To develop effective technical preparedness and readiness across relevant sectors for infectious disease response in humanitarian settings.

To evaluate the current activities under READY, the evaluation will explore the effectiveness of the overall project in achieving its stated goal. The key question this evaluation seeks to answer is “Did organizations or individuals who engaged in READY activities and events (trainings, webinars, e-learning courses, etc.) improve their readiness to respond to major disease outbreaks?”

Key themes to explore:

• Were the chosen operational and technical activities the right ones to build organizational and individual capacity to respond to major disease outbreaks, and to what extent were they successful at doing so?
• Which operational and technical activities were perceived as the most useful and relevant for improving organizational level NGO readiness to respond to disease outbreaks?
• Did READY engage in the right global outbreak and humanitarian coordination structures at the right time? How did engagement in these mechanisms help/or not READY to achieve impact READY also seeks to document lessons learned from support activities that can be used by READY or similar initiatives for future readiness support. Focus areas include:
  • Localization commitments
  • Structure and engagement of the READY Advisory Groups (RAG) and Strategic Advisory Group (SAG)
  • Engagement with and functionality of the READY Knowledge Management Hub

V. Evaluation Methods
The proposed methodology includes:

• Document review including review of donor reporting, after action reviews from READY activities, participant feedback, pre-and post- test scores, activity learning objectives, etc.
• Online survey targeting READY email subscribers (a global network of over 2,000 individuals who have subscribed to READY updates and/or engaged in READY trainings, events, and activities), stakeholders, and partners. Note: Given the nature of the survey and READY’s target populations, there is limited capacity to develop a scientific sample. This will be a non-probability sample using self-selection sampling.
• Qualitative interviews (key informant interviews and/or focus group discussions) with READY staff, consortium partners, advisory group members (RAG and SAG), external stakeholders who have engaged with READY’s work (webinars, capacity building trainings, technical guidance development and operationalization, etc.), and coordination bodies (Health Cluster, RCCE Collective Service, etc.)

The Consultant will work with the READY M&E Advisor and the READY Core Team to develop an evaluation plan and methodology at the start of the consultancy. READY can provide support and access to online survey platforms such as Survey Monkey, Google Forms or Kobo Toolbox.

VI. Evaluation Findings Dissemination Plan
READY will share findings from the evaluation with key stakeholders, and relevant partners. It will also upload the final report to the Development Experience Clearinghouse and READY’s website. A full dissemination plan will be developed with the READY team and BHA and will include a final report and presentation of findings to key stakeholders.

VII. Expected Deliverables & Timeline

Recruiting and onboarding the consultant will take place in final quarter of 2021. Data collection for the evaluation will take place in the final quarter of the program (Jan-March 2022). READY will aim to have the report finalized and disseminated by the end of March 2022.

The Consultant’s deliverables and tentative timeline (subject to the commencement date of the evaluation) are outlined below. Key deliverables are in bold. The Consultant and the READY M&E Advisor will agree on final milestones and deadlines at the inception phase.

Deliverables and Tentative Timeline

• December 2021– The Consultant is contracted and commences work
• December/January 2022– The Consultant will facilitate a workshop with the relevant stakeholders at the commencement of the project to develop the inception report**
• January 2022– The Consultant will submit an inception report including:
  • evaluation objectives, scope, and key evaluation questions
  • description of the methodology, including design, data collection methods, sampling strategy, data sources, and evaluation matrix against the key evaluation questions
  • data analysis and reporting plan
  • caveats and limitations of evaluation
  • risks and mitigation plan
  • ethical considerations including details on consent
  • stakeholder communication and engagement plan
  • key deliverables, responsibilities, and timelines
  • resource requirements
  • data collection tools (for feedback from stakeholders including the SAG and RAG)

Once the report is finalized and accepted, the Consultant must submit a request for any change in strategy or approach to the M&E Advisor and Chief of Party.

• January 2022– The Consultant will finalize the data collection tools (including survey instrument and interview guide) and finalize the analysis and data management plan.
• February 2022– The Consultant will lead on data collection and analysis
• February/March 2022– The Consultant will share the Draft Report including the following elements:
  • Executive summary
  • Background description of the program and context relevant to the evaluation
  • Scope and focus of the evaluation
  • Overview of the evaluation methodology and data collection methods
  • Findings aligned to each of the key evaluation questions
  • Specific caveats or methodological limitations of the evaluation
  • Conclusions outlining implications of the findings or learnings
  • Recommendations
  • Annexes (Project logframe, evaluation ToR, Inception Report, Evaluation schedule, List of people involved)

Note: A consolidated set of feedback from key stakeholders will be provided by Save the Children within 2 weeks of the submission of the draft report.

• March 2022– The Consultant will develop and share the Final Report incorporating feedback from consultation on the Draft Evaluation Report
• March 2022 – The Consultant will produce key knowledge translation materials: • PowerPoint presentation of findings

VIII. Consultant Profile

• Master’s Degree, preferably in public health or a related field;
• Proven experience in program evaluation;
• Strong experience in qualitative methods, including conducting interviews or group discussions
• Excellent English language oral and written communication skills (required);
• Knowledge and experience in humanitarian response and outbreaks (preferred);
• Familiarity with humanitarian and global health coordination mechanisms (preferred);
• Experience with USAID/BHA Monitoring and Evaluation practices (preferred);
• Comfort working in teams as well as acting independently in the implementation of specific tasks, multitasking and prioritizing, working under pressure, and meeting deadlines;
• Willingness to work across multiple time zones as needed.