Evaluation of “Strengthening Disaster Risk Reduction and Livelihoods Capacities in Refugee-Affected Community in Bangladesh At International Organization for Migration

This is a two- and half-year multisectoral project including Disaster Risk Reduction, Livelihoods, WASH and Shelter support for the identified vulnerable host communities in Ramu, Moheshkhali, Ukhiya and Teknaf Upazilas of Cox’s Bazar district under the financial support from Bureau for Humanitarian Assistance (BHA) of United States Agency for International Development (USAID).

Evaluation context

The primary aim of the project is to contribute to increase the disaster risk reduction capacity of the vulnerable host communities involving resilient livelihoods, improved WASH and shelter facilities. To achieve this, the project has seven outcomes, the first of which is to enhance DRR capacity of Local authorities and institutions in Cox’s Bazar District through providing capacity building support to the Upazila and Union Disaster Management Committees, Cyclone Shelter Management Committees, Fire Service and Civil Defense, Cyclone Preparedness Program and local communities. IOM also renovates cyclone shelters under this outcome. The outcome also addresses landslide early warnings in collaboration with Food and Agriculture Organization (FAO). The second outcome is to increase the resilience of host communities to external shocks by providing improved livelihoods, income generation and short-term employment opportunities. A wide range of livelihoods training is provided to the vulnerable host community beneficiaries including inputs for starting their business. The third outcome is to enhance disaster risk reduction initiatives through cash for work, enabling the local communities to earn an income and contribute to the rehabilitation of small-scale disaster mitigation activities e.g., repair of roads, culverts, guide walls, drainage systems, etc. The fourth outcome is to reduce the vulnerabilities and exhibit an enhanced capacity of the communities through the upgradation of evacuation shelters. The same beneficiaries also receive improved WASH facility under outcome six. Under the seventh outcome, IOM provides unconditional cash grants and communication and emergency response equipment support to the fishermen and host community beneficiaries affected by the Covid-19 pandemic.

While the program life cycle is completed, a final evaluation is anticipated to understand the impact of the project which will also guide the future initiatives to achieve sustainable results. Throughout the TOR, the term “Evaluator” refers to both an individual and/or a team.

Evaluation purpose

The main objective is to evaluate the project’s performance against the desired results as articulated in the project’s results framework. Further, the objective is to understand the impact and outcomes of the resources and interventions implemented. It will also help to assess the relevance and accountability of the project to the intended beneficiaries. The final evaluation will provide results to support management decisions on intended improvements, learnings and lessons learned to ensure sustainability of future activities.

Recommendations will be used to improve the program processes as well as the anticipated results in the most sustainable, effective, relevant, and efficient manner. Recommendations will also be used for future programme design.

The evaluation specific objectives aim to:

• Assess to what extent the project realized the outputs, outcomes, and objectives as expressed in the project documents
• Assess the project’s overall performance from planning, implementation, and knowledge management by identifying the key strengths and areas of gaps and providing necessary recommendations for future improvement.
• Identify impacts of the interventions (livelihoods/skills activities on the beneficiaries – e.g. whether the beneficiaries managed to develop businesses including the impact of several COVID 19 lockdowns, on what were the unconditional cash grants spend mostly and reasoning why; in terms of DRR interventions what were the impacts of interventions on local government structures/ communities) and document lessons-learned, innovations, and best practices resulting for future strategies and interventions
• Understand better what the project has achieved in relation to gender and disability mainstreaming and overall results
• Support the use of relevant and timely contributions to organizational learning, informed decision-making processes resulting from the analysis, conclusions, or recommendations as well as and accountability for results

Evaluation scope

The evaluation will look at the project objective; “Leverage DRR actions in Cox’s Bazar District to protect host communities’ lives and livelihoods” and focus on the seven outcomes of the project which was implemented from August 2019 to February 2022.

1. Enhanced DRR capacity in Cox’s Bazar District. The beneficiaries are local disaster management institutions, cyclone shelter management committees, CPP, Fire Service and Civil Defense, local communities. Activity locations are in; Ramu, Moheshkhali, Ukhiya and Teknaf
2. Host communities are resilient to external shocks due to improved livelihoods, income generation and short-term employment opportunities. The beneficiaries are 700 host communities in Ukhiya and 500 young entrepreneurs in Ukhiya and Teknaf.
3. DRR and market function is enhanced through CfW initiatives. The beneficiaries (687) are cash for work laborers for the selected small scale disaster mitigation activities in Ukhiya and Teknaf
4. 500 Host Community members, particularly persons with heightened vulnerabilities, exhibit an enhanced capacity to upgrade and maintain disaster resilient shelters. Activity locations as Nhila, Sabrnag, host community inside Camp 9 and Camp 8E
5. Naf River-based communities and relevant institutions have enhanced DRR and preparedness capacities. The beneficiaries are four Ward Disaster Management Committees, six primary schools and local communities of nine wards in Ukhiya and Teknaf
6. Targeted 2223 households (11,570 individuals) in host communities have access to safe and disaster-resilient water, sanitation and hygiene services.
7. COVID-19 reprogramming activities to enhance institutional and individual resilience to COVID-19. The beneficiaries are 6218 fishermen from host communities of Ukhiya, Teknaf, Ramu and Moheshkhali, and 700 day-laborers from Ukhiya

While looking at the project outcomes, the evaluation will answer the following relevant indicators:

1. % of local actors with increased knowledge on early warning systems (outcome 1)
2. % of targeted households that exhibit a marked increase knowledge and skills concerning DRR shelter upgrades and maintenance (outcome 4)
3. % of beneficiaries that report improvements in household financial literacy due to the trainings offered (outcome 2)
4. % of beneficiary households who report improvement in their household income (outcome 2)

Evaluation criteria

Program performance will be evaluated according to the Organisation for Economic Co-operation and Development/ Development Assistance Committees (OECD/DAC) evaluation quality standards based on the criteria of relevance, effectiveness, efficiency, impact, and sustainability. Due to the nature of final evaluation, impact and sustainability would be prioritized including other criteria. These five evaluation criteria form the basis and guidance for the evaluation questions, in addition to cross-cutting issues (gender, environment and climate change) as presented in Table 01.

Evaluation questions

The evaluation shall be on the quality of project performance as per the evaluation criteria described above. The guiding evaluation questions are presented in Table 01: Evaluation guiding questions. The evaluation questions are related to the programme processes and delivery strategies based on the seven outcome areas outlined under the evaluation scope in section three. The evaluator is encouraged to identify additional questions to gather adequate data needed and to suggest the best approach.

Evaluation methodologies

Evaluation team: External

The evaluation will be conducted through mixed approaches:

• Review of project documents (proposal, narrative reports, work plans, monitoring data, budget, financial reports, annexes etc.); relevant agency strategies and guidance; relevant projects and documents under the project (home-based, desk study).
• The evaluator should consider covid-19 situation while designing methodologies. Any community level data collection must be performed taking approval from respective authorities observing social distances.
• Project implementation process observation at selected sites.
• Quantitative and qualitative data collection and analysis.
• KIIs/ FGDs and/ or surveys with beneficiaries taking necessary approvals from authorities (due to Covid-19, field visits in selected locations should be considered following Covid-19 guidelines.
• Key in-depth interviews with program staff, Government Authorities (local Government of Cox’s Bazar, Agricultural Department), implementing partners, local NGOs/ CSOs, co-partners.

The evaluator is expected to provide a reviewed description of the methodology in the proposal, as well as a detailed methodology in the inception report.

The evaluation must follow the IOM Data Protection Principles, UNEG norms and standards for evaluations, and relevant ethical guidelines.

Considering the current COVID-19 pandemic, the evaluation should consider constraints both regarding programmatic changes the program had to undertake as well as mobility restrictions. This Terms of Reference (TOR) considers IOM’s Evaluation/OIG guidance on ‘Continuity of Monitoring and Evaluation Interventions during COVID-19’ (version 8 April 2020). While data collection should be conducted virtually whenever it is possible, it would need to be adjusted to its specific context and appropriate alternatives to field visits and data collection are recommended to ensure the continuation of monitoring and evaluation activities during this period and should be further explored by the evaluator(s) in coordination with IOM.

Evaluation deliverables

The evaluator is expected to produce: (1) Work plan and inception report/plan per the IOM template (will be shared later) including the tools/materials/templates. (2) Briefing and debriefing meetings in addition to the routine meetings and discussions with the M&E officer, programme manager/ project team, project focal points and management. (3) A presentation of the initial/ preliminary findings and a final draft report outlining the lessons-learned and recommendations. (4) A final report with a summary of the evaluation brief (2-pager evaluation brief per IOM template- will be shared later) – clearly describing concrete steps which will include an approach on how the four-thematic outcomes of the project (as outlined under the evaluation scope) were implemented and how it will be implemented in future. (5) A Final PowerPoint Presentation with key result for future use/ reference. Final Report (report body 30 pages maximum) including an Executive Summary, approach, and methodology, with gender equality addressed throughout. Final report should annex the complete database of findings, survey records, questions, and answers. The template for IOM evaluation matrix as part of the report could be collected from Annex 1.

Evaluation workplan

The total assignment is for 8 weeks. It is expected to start the evaluation as soon as signing the contract with IOM for this assignment. The draft report should be submitted by the end of 6^th week after signing the contract. IOM will review the report in the 7^th week and send back the report to the evaluator. The evaluator will prepare the final report based on the comments and submit the final report with presentation for endorsement in the 8^th week. A detailed workplan is sought with the proposal highlighting potential challenges related to Covid-19. The evaluator will directly work with the National Monitoring, Evaluation & Reporting Officer under the Social Cohesion Division, IOM with the programmatic support from DRR, Livelihoods, WASH and Shelter units.

Intellectual property rights

The information collected, and the resulting findings shall be the property of IOM and shall not be disclosed or revealed to any third party without expresses written permission of IOM. The evaluator shall not, either during the term or after termination of the assignment, disclose any proprietary or confidential information related to the evaluation without prior written consent. Proprietary interests on all materials and documents prepared by the evaluator under the assignment shall become and remain the property of IOM.

Data collected for the study is the property of IOM. Any intentional fabrication of data would be considered fraudulent and IOM holds the right to act against that.

Requirements

Any national or international candidate with good knowledge and experience in project/ programme evaluation, especially on energy & environment, forestry, agriculture, and livelihoods will be preferred. Demonstrated knowledge and experience in developing and implementing gender sensitive survey methodologies in humanitarian emergency settings. Knowledge of Rohingya refugee crisis in Cox’s Bazar, or previous experience on similar assignment will be an added advantage. Considering worldwide travel restrictions due to Covid-19, candidates (national/ international) currently in Bangladesh will be prioritized.

The below requirements would be considered while selecting the evaluator.

• Extensive expertise in evaluations of complex programs and a minimum of five or more years of experience in conducting complex evaluations.
• Familiarity or fluency in Bangla is an advantage.
• Proven experience in conducting value for money and cost and benefit analysis, including other efficiency-related tools and techniques for humanitarian and programmatic interventions
• Proven experience in using quantitative and participatory qualitative methods for data collection.
• Proven experience in conducting evaluations with international humanitarian organizations and knowledge of livelihoods and other relevant programs.