Consultancy: Endterm evaluation of project ‘Lifesaving shelter assistance for the most vulnerable, conflict-affected persons in newly accessible areas At ZOA

1. Background

Overview of the Project

The objective of this action funded by ECHO is to provide lifesaving shelter support to 2.835 (including co-funding) of the most vulnerable persons in the targeted areas of Kherson and Mykolaiv oblasts. The action is needs-driven and maintains agility to shift in case new areas open up if the context changes, particularly to Newly Accessible Areas (NAA). The total budget of the action is 2.2 million EUR.

The action is based on primary data collection, including rapid needs assessments conducted in Mykolaiv and Kherson oblasts. The main need identified is shelter assistance, indicated as the most urgent need by 90% of the households in Kherson and 80% of the households in Mykolaiv. The action aims to increase access to safe and dignified shelters by repairing houses with light and medium damage. An integrated approach using a combination of conditional cash for shelter and technical support is used to maximize shelter outcomes. To ensure efficiency and effectiveness, the intervention makes use of the RedRose digital solution for its cash distributions. This solution is tailor-made to the Ukraine context and successfully used in other cash interventions in Ukraine, including Kherson. Provision of technical support to targeted households is integrated throughout the project cycle.

Most vulnerable individuals are targeted by using a combination of community-based targeting approach and vulnerability criteria. Lastly, this action is strongly linked to early recovery/nexus by repairing residential houses damaged as a result of the conflict.

Summary of the activities implemented are listed below by sector:

• Shelter: provide lifesaving cash for shelter support to enable 2.835 of the most vulnerable persons to repair their light or medium damaged house in Kherson and Mykolaiv oblasts

2. Purpose of the evaluation

ZOA is responsible for the coordination and implementation of the endline evaluation. The purpose of the endline evaluation is to provide a holistic understanding of the project progress, performance and achievements, including the extent to which the project has been adapted according to the changing context and the way in which the project has impacted the lives of affected populations in Ukraine. Additionally, the objective is to provide learnings and recommendations for future projects, including a potential cost extension with ECHO. The evaluation should be structured around the OECD-DAC evaluation criteria, in particular efficiency, effectiveness and coherence.). The specific objectives of the evaluation include:

• To assess the efficiency of the Cash Modality used in the project (Efficiency).
• To assess the performance of the project, paying particular attention to the outcomes and outputs of the project interventions against its key result indicators (Effectiveness).
• To assess how the project has implemented the main ZOA prioritized cross-cutting themes, such as accountability, technical quality assurance and support, localization (coordination with local authorities and affected communities), gender sensitivity and inclusiveness, and conflict sensitivity/ do no harm programming (Coherence).
• To assess complementarity of project interventions and the collaboration with other humanitarian actors and ECHO (Coherence).
• To document lessons learned (what worked well, what did not work well) and provide evidence-based recommendations for future interventions.

3. Scope of the evaluation

The scope of the endterm evaluation is the full implementation period up until the date of the evaluation (August 2024) with a focus on the different project components and their complementarity and the added value of the response within the larger humanitarian response. The evaluation should cover and provide updates on achievements in targeted sectors (Shelter) across the intervention locations in targeted Oblasts (Kherson and Mykolaiv). The target groups to be included in the end evaluation are staff of ZOA in country, project beneficiaries, local authorities and other relevant stakeholders. The target audience for sharing the evaluation findings include ECHO, ZOA staff, the humanitarian community in Ukraine, local authorities and affected populations.

4. Evaluation questions

The evaluation should be structured around the OECD-DAC evaluation criteria and the additional criteria as mentioned in the table below. The evaluation questions are indicative. The consultant is expected to review and improve the evaluation questions in the initial stage of the evaluation exercise.

Evaluation Criteria & Key Questions

Efficiency

Efficiency concerns an assessment of how well resources are used to achieve intended objectives.

• To what extent are activities timely and cost-efficient?

• What could be done differently to complete the project more timely and efficiently?

• To what extent can funds be re-allocated in time during implementation to respond to new developments?

• How is ZOA’s coordination contributing to the efficiency?

Effectiveness

Effectiveness is about the extent to which the project has achieved its objectives.

• To what extent are the project targets and objectives achieved, and how effective was the project in responding to the needs of affected people?

• What are the successful approaches applied by the program, and what evidence is there to support their effectiveness?

• What are the major factors influencing the achievement or non-achievement of the objectives?

• Is the intervention causing a significant change in the lives of the intended beneficiaries?

• How likely is it that any positive changes may be sustained in the short, medium and long term?

Coherence

Coherence refers to the compatibility of the intervention with other interventions (and policies) in the country, sector and / or institution. The distinction is made between internal coherence (synergies with interventions carried out by the same institution/government, and with international norms and standards) and external coherence (consistency of the intervention with other actors’ interventions: complementarity, harmonization, coordination and added value).

• To what extent was internal and external coherence of the project ensured?

• How did coordination and collaboration with other humanitarian organizations and other key stakeholders happen and contribute to coherence?

• How is the intervention integrating the nexus approach, and what are the benefits and challenges of doing so?

Other criteria

Gender sensitivity and inclusiveness

• To what extent is a gender-sensitive programming approach applied in the project?

• How is the gender and age marker used?

• To what degree is the program addressing the needs of marginalised groups?

• Do all the intended target groups, including marginalised people, benefit equally from the intervention?

Protection mainstreaming

• How does ZOA mainstream protection throughout their interventions and to what extent have efforts been effective?

• What can be learned for the future to improve protection mainstreaming?

Accountability

• To what extent was the affected population meaningfully informed and involved in the design and planning process, and throughout implementation?

• What feedback mechanisms are in place and to what extent are these mechanisms accessible for all targeted groups? How is beneficiary feedback being addressed?

• How did ZOA uphold the CHS commitments? (Focus on commitment 4 and 5).

Conflict sensitive and do-no-harm programming

• How are conflict-sensitivity and do-no-harm programming integrated into the project design and applied during project implementation?

Localization and equitable partnerships

• What learnings can be provided on improving collaboration with local authorities and affected communities?

5. Methodology

The consultant is expected to determine the methodological approach for the end-term evaluation of the ECHO project. The methodology should be further detailed in the inception report and will be approved by ZOA, including the research tools. ZOA provides the following guidance for the development of the methodology:

• The consultant is expected to use and review existing project documents during the end evaluation. These documents include the project proposal and logframe, monitoring and progress reports, financial reports, policies and strategies, and any other relevant project document. ZOA will provide the external consultant with all available project documentation at the beginning of the consultancy.
• A mixed research methodology is preferred, which includes the collection of qualitative and quantitative data through surveys, key informant interviews, focus group discussions and document review. Participatory research methods are encouraged.
• Triangulation of data is important. Therefore, the consultant is expected to use multiple data sources to allow for the verification of results.
• The sample size must be statistically representative of the target population. The evaluation team should ensure that people from different groups targeted by the project, including people with disabilities, different genders and age categories, are included in the sample size. The consultant’s proposed methodology and sampling should take into consideration comparability with the baseline and generate evidence to show the most significant impacts. The analysis among others should be disaggregated by governorate, gender, age and disability.
• Field work is mandatory. The consultant / consultancy team will undertake field visits to the project implementation areas to collect primary data amongst a variety of stakeholders, including local authorities and beneficiaries. Participation of stakeholders in the evaluation should be maintained at all times, reflecting opinions, expectations, and visions about the contribution of the project towards the achievement of its objectives.
• The consultant must consider participants’ safety throughout the evaluation (including recruitment and training of research staff, data collection / analysis and report writing) as well as research ethics (confidentiality of those participating in the evaluation, data protection, age and ability-appropriate assent processes) and quality assurance (tools piloting, enumerators training, data cleaning). The consultant should ensure prior informed consent for participation in the study and for any picture being taken.
• The consultant is expected to be aware of and understand political sensitivities, consult ZOA regularly and adapt the evaluation approach if needed. ZOA will inform the consultant about the accessibility and security situation of the project target areas before starting of the field data collection exercise and will agree with the consultant on feasible ways of data collection. The consultant should consider mitigation measures if project locations are deemed inaccessible.

  6. Deliverables

The information below provides an overview of the expected deliverables.

INCEPTION REPORT: The inception report must include findings from the desk review, an evaluation matrix (table describing per key question the method, data source, tools), detailed workplan, procedures for data collection, analysis, sampling and the data collection tools. The inception report and data collection tools both need to be approved by the lead before the start of data collection. Tentative timeframe: 21 June 2024.

DRAFT REPORT: The draft report should structurally mimic the final report (see below for structure) and include data analysis, initial findings, conclusions and recommendations. Tentative timeframe: 1 August 2024

FINAL REPORT: In the final report the feedback on the draft report needs to be addressed. The final report will be a maximum of 30 pages (excl. annexes) and consist of the following sections at a minimum:

1. Table of contents

2. Executive summary

3. Intervention description

4. Scope of the evaluation

5. Purpose and objectives of the evaluation

6. Methodology

7. Findings

8. Lessons learnt

9. Conclusion

10. Recommendations

11. Annexes – photos, human stories, final research tools, bibliography of secondary data used, a list of persons interviewed.

Alongside the evaluation report the consultant will provide the raw data.
Tentative Timeframe: 16 August 2024

PRESENTATION / LEARNING SESSION: The consultant is required to develop a presentation / learning session with selected stakeholders to present findings and recommendations. Tentative timeframe: 23 August 2024.

7. Roles and responsibilities

ZOA is responsible for the recruitment and contracting of the endterm evaluation consultant. The ZOA Manager of Programme Quality and the Programme Manager for South East Ukraine will be the main focal persons for this assignment and ensure to provide the consultancy team timely with relevant documentation during the inception phase, respond to additional requests for information and facilitate the initial connections with relevant local stakeholders. ZOA will oversee the endterm evaluation process, keep track of deliverables and coordinate feedback from partner organizations. In case needed, ZOA will facilitate visa and in country travel approvals for the consultant, and ZOA will inform the consultant about the accessibility and security situation of the project target areas before starting of the field data collection.

ZOA will provide project documentation relevant to the endterm evaluation, including beneficiary lists (if feasible and if confidentiality can be maintained), facilitate connections with project teams in the field for data collection and provide practical support in case needed. Staff from ZOA, will be available to participate in an interview if requested by the consultant. Additionally, ZOA will appoint a focal person (e.g. the Programme Manager) for the endterm evaluation and support the consultant in the selection of participants in the data collection and introduce the consultant to relevant stakeholders.

The endterm evaluation consultant / consultancy team is expected to:

• Organize an inception meeting with the ZOA project team;
• Design the evaluation methodology, including evaluation matrix, data collection tools for the program and relevant program indicators (including translation into Ukrainian if needed) and sampling technique and size;
• Conduct desk reviews of secondary information and program documents;
• Develop the inception report and submit it to ZOA for feedback, including the data collection tools;
• Address feedback on the inception report and data collection tools;
• Recruit and train enumerators (ensure male and female enumerators) who will pre-test the data collection tools. If necessary, make final adjustments to data collection tools;
• Collect data from a representative sample of individuals from the target groups and key program relevant stakeholders using methodologies as outlined in the approved inception report. The consultancy team is responsible for the transportation of enumerators in the field;
• Process and analyze the data (data entry, cleaning, verification and analysis);
• Report writing and submission of first draft of the report;
• Organize a session for relevant stakeholders to present findings of the end evaluation and share learnings for discussion;
• Finalize the report incorporating feedback and submission of final report.

  8. Tentative timeline

Task / Date (deadline):

Posting of request for proposals: 22 March – 14 April

Deadline submission proposals: 14 April

Proposal considered, short listing and follow up enquiries completed: 15-30 April

Interviews with selected consultants: 1-15 May

Signing of contract with selected consultant and kick-off meeting : 7 June

Submission inception report: 21 June

Data collection: 1-5 July

Submission draft report: 1 August

Presentation of results: Between 5 and 9 August

Submission final report and final deliverables: 16 August 2024

9. Budget

Interested consultants are required to provide a budget including travel costs, number of working days per specific activity, daily rate and any other costs. The maximum budget for this evaluation is 13,294 EUR. In case an agreement is reached that the consultant will also collect baseline data, additional budget will be made available.

10. Required expertise

The specific requirements for this assignment are hands on experience in evaluating emergency interventions implemented by international partners, and experience in evaluating multi-sectoral humanitarian response programs. Additional required qualifications are detailed below.

• At least a master’s degree in economics, developmental studies, business administration and social science or related field for the lead consultant / a minimum of bachelor’s degree on the relevant academic areas with 7 years of progressive experience in research.
• Extensive experience in research work and in assessments/evaluations. Knowledge of mixed research methodologies and application of various tools including practical experience in assessments, planning, implementation, monitoring and evaluation of humanitarian interventions. Proven experience of using participatory methods is required.
• Strong experience in humanitarian response, Child Safeguarding and knowledge of humanitarian standards (CHS, Sphere, Code of Conduct), and expertise on project sector Shelter. Expertise on other sectors (FSL, NFI, MPCA, WASH) will be an added advantage.
• Understanding of and experience in evaluating cross-cutting themes, including gender equality, accountability and conflict sensitivity / do-no-harm.
• Good understanding of the Ukrainian context and previous work experience in Ukraine are required.
• Excellent analytical and report writing skills with skills in using statistical packages such as SPSS, STATA etc.
• Fluent in English. Understanding of Ukrainian of the lead consultant will be an added advantage. Staff responsible for data collection need to be fluent in Russian or Ukrainian.
• Cultural awareness and ability to operate in politically complex and sensitive environments are required.
• An experienced protection/safeguarding person should be included as part of the team.
• Willing and able to travel to and work in the project locations in the de-occupied territories of Khersonska, Mykolaivska Oblast, and Dnipropetrovsk Oblast.