Mid-term evaluation – Norad cooperation agreement 2021-25 At Norwegian Red Cross

Background

Within the framework of the 2021-2025 cooperation agreement with Norad, the Norwegian Red Cross seeks to conduct a mid-term evaluation. The current cooperation agreement, named “GLO-0604 QZA-20/0295 Norwegian Red Cross Agreement 2021-2025”, is valid for the period January 2021 to December 2025. The “…project’s expected effect on society is that people affected by conflict, crisis, and climate change have improved health and protection (impact)”. At an outcome level, the expected effects for the target groups are: 1) Improved and safer access to health care, 2) Improved collective humanitarian impact of the RCRC Movement, and 3) Improved financial management of National Societies. The intended target group is vulnerable people in the countries included in the agreement: Burkina Faso, Nigeria, Somalia, South Sudan, Myanmar, Iraq, and Syria. In addition, three thematic areas are funded: 1) Finance development: accountable and effective National Societies, 2) Multi-country scale-up of RCRC community-based surveillance, and 3) RCRC Movement and universal health coverage.

The main modality of implementation at country level is through country-level projects in partnerships with the respective Red Cross or Red Crescent National Societies. A key principle in Norwegian Red Cross approach to partnerships is alignment with the auxiliary role of the National Society and partner’s strategic priorities.

Evaluation purpose, criteria, and scope

In accordance with the requirements in the 2021-2025 cooperation agreement with Norad, the Norwegian Red Cross seeks to conduct a mid-term evaluation focussing on effectiveness, impact, sustainability, and efficiency based on case studies of Myanmar, Somalia, and Burkina Faso. The overall purpose is to evaluate project performance to provide insight and enable effective and cost-efficient project delivery and management. More specifically, the evaluation shall focus on:

• Effectiveness and impact: by documenting progress to date and evaluating the extent to which the country-level projects are expected to achieve their objectives and their targets at the end of 2025 (effectiveness), assess if the projects are on track to reach impact at the end of 2025 based on the global theory of change. The analysis should also assess the contribution of the country-level project the overall objective of the agreement. An analysis of the inclusion of vulnerable groups should be included in this part of the evaluation.
• Sustainability: based on current institutional policies and project plans, evaluate the extent to which the net benefits of the interventions are likely to continue after 2025. This section should also evaluate organisational sustainability and system strengthening of partners based on Norwegian RC support to capacity building and organisational development.
• Efficiency: the evaluation should conduct one in-depth cost efficiency case study that consider Norad’s guideline on the use of cost analysis in impact evaluations (Norad 2023 “Norad’s expectations for knowledge utilisation by grant recipients…”).

The scope of the evaluation is as follows:

• Geographically, for the evaluation criteria of effectiveness, impact, and sustainability, and the inclusion of vulnerable groups, conduct 3 in-depth case studies in respectively Myanmar, Somalia, and Burkina Faso. Norwegian RC has had partnerships with the National Societies since respectively 2019 (Myanmar), 1982 (Somalia) and 2021 (Burkina Faso). This includes an assessment of Norwegian RC contribution to organisational sustainability and system strengthening of partners based on support to capacity building and finance development.
• Thematically, the evaluation is limited to primary health care service delivery. For the evaluation of organisational sustainability and system strengthening of partners, a broader focus is necessary. The evaluation should also take into consideration results prior to the current cooperation agreement period to document longer term impact (based on existing reports and evaluations)
• For cost efficiency, the evaluation should focus on primary health care service delivery through fixed and mobile health clinics in Myanmar and Somalia.

The main audience of the evaluation is the Norwegian Red Cross and the relevant National Society partners as well as Norad. The evaluation report will be published on Norad’s evaluation database.

Evaluation questions

A tentative list of evaluation questions and sub‐questions should be suggested in the technical proposal and a final list agreed with Norwegian Red Cross during the inception phase. It is expected that revised international evaluation criteria and their principles for use adopted in December 2019 inform the design of the proposal, including the evaluation questions. The below focus areas and questions are indicative of the types of questions to be addressed in this evaluation:

• Effectiveness and impact
  • Evaluate progress towards objectives and targets for the case studies.
  • Evaluate if and how any identified change can be attributed to Norwegian Red Cross supported projects or if any contribution can be identified.
  • Evaluate the quality of baseline studies to ensure they can be used for comparison with endline studies in 2025 for documentation of impact.
  • Evaluate inclusion of vulnerable groups in the project cycle (community engagement and accountability, assessments, project design, baseline studies, etc).
• Sustainability: Important components to consider when evaluating sustainability (non-exhaustive list):
  • How can the sustainability of the intervention be assessed?
  • To what extent are the projects built on and contribute to develop existing local capacities?
  • Can the achieved results of the projects be considered sustainable in a medium- to long-term?
  • What are the major factors which influence the achievement or nonachievement of sustainability of the project?
  • For organisational sustainability, the evaluation should include a longer time frame than the current agreement period that assesses Norwegian Red Cross contribution to sustainability of partners based on support to capacity building and organisational development
• Efficiency: A methodological approach to cost efficiency should be developed as part of the evaluation of the cost efficiency of primary health care service delivery through fixed and mobile health clinics.

Evaluation methodology and data collection methods

The technical proposal (see below) should include the outline of a suitable methodology and appropriate data collection methods adapted to the purpose of the evaluation. It is free to suggest any additional methodological approaches and data collection methods and questions that are not mentioned in the ToR.

The evaluator must adopt a consultative and participative methodological approach when relevant. The approach should strive to include the following elements:

• Given limited access to the project locations in all three contexts, the methodological design and composition of the evaluation team should take this into consideration. It is preferrable to do field data collection in at least one of the three contexts.
• Data collection from Norwegian Red Cross, IFRC, ICRC and NS staff at national and local level as well as with volunteers and beneficiaries as relevant. Norwegian RC staff, together with National Society partners, will support organising data collection as needed.
• Data collection from relevant stakeholders such as authorities and other national and local actors
• Comparative analysis across the case studies
• Documentation of results prior to current agreement period should build on existing reports and evaluations.

Relevant written documents will be provided by Norwegian Red Cross as well as the relevant partner National Societies. Key sources of written information include:

• Project documents (assessments, project plans and logical frameworks, budgets, indicator tracking tables, baseline and endline studies, annual reports, evaluations)
• Norwegian Red Cross result framework
• Annual internal Norwegian Red Cross result reports
• Annual report from Norwegian Red Cross to Norad
• Annual project audits and audits of Norwegian Red Cross
• Relevant background and strategy papers (from Norwegian Red Cross and partners)
• Grant agreement between Norad and Norwegian Red Cross and correspondence related to grant management

Deliverables and timeline

The proposal should include, but not be limited to, a plan for the following deliverables:

1. An inception report with a detailed description of methodology to be used, a data collection plan, overview of information sources, a timeline for deliverables and solutions to identified challenges.
2. Feedback workshop with each Red Cross Red Crescent National Society involved in the evaluation based on preliminary findings and recommendations.
3. Draft consolidated report in English including background, findings and conclusions, lessons learned and recommendations.
4. Feedback workshop in Oslo or online to present draft consolidated report before revision and final approval of the report.
5. The final report shall include one chapter for each of the case studies that can be shared with the respective NS as well as a section summarising findings from all case studies including a comparative analysis. The main body of the text of the report should be maximum 25 pages (excl. executive summary and annexes). The reports should as a minimum include the following:
   1. Executive summary
   2. Background
   3. Evaluation methodology and limitations
   4. Findings, conclusions and lessons learned
   5. Recommendations
   6. Appropriate appendixes

The deadline for submitting a proposal is 16 August 2024. Tentative dates for key deliverables are:

• Start of evaluation: 13 September
• Inception report: 27 September
• Data collection: 13 September – 13 November
• First draft report: 27 November
• Final report: 4 December

Evaluation quality and ethical standards

The evaluators should take all reasonable steps to ensure that the evaluation is designed and conducted to respect and protect the rights and welfare of people and the communities of which they are members, and to ensure that the evaluation is technically accurate, reliable, and legitimate, conducted in a transparent and impartial manner, and contributes to organisational learning and accountability. Therefore, the evaluation team should adhere to the evaluation standards and specific, applicable process outlined in the IFRC Framework for Evaluation. The IFRC Evaluation Standards are:

• Utility: Evaluations must be useful and used.
• Feasibility: Evaluations must be realistic, diplomatic, and managed in a sensible, cost-effective manner.
• Ethics & Legality: Evaluations must be conducted in an ethical and legal manner, with regard for the welfare of those involved in and affected by the evaluation.
• Impartiality & Independence; Evaluations should be impartial, providing a comprehensive and unbiased assessment that considers the views of all stakeholders.
• Transparency: Evaluation activities should reflect an attitude of openness and transparency.
• Accuracy: Evaluations should be technical accurate, providing sufficient information about the data collection, analysis, and interpretation methods so that its worth or merit can be determined.
• Participation: Stakeholders should be consulted and meaningfully involved in the evaluation process when feasible and appropriate.
• Collaboration: Collaboration between key operating partners in the evaluation process improves the legitimacy and utility of the evaluation.

It is also expected that the evaluation will respect the seven Fundamental Principles of the Red Cross and Red Crescent: 1) humanity, 2) impartiality, 3) neutrality, 4) independence, 5) voluntary service, 6) unity, and 7) universality. Further information can be obtained about these principles at: www.ifrc.org/what/values/principles/index.asp

Evaluator(s) qualifications

The competency requirements for the evaluator(s) are:

• University degree(s) at post-graduate (master) level in relevant fields of study and relevant work experience in health/humanitarian/development programme.
• Familiarity with trends and developments in humanitarian work.
• Minimum of 5 years of experience in evaluation and social science, or similar, methodology required.
• Demonstrated experience in planning and implementing final project evaluations required, inclusive of competency and track record in managing quantitative and qualitative data collection methods and analysis.
• Experience from the relevant geographical areas required.
• Knowledge and experience of working with the Red Cross Red Crescent Movement preferred.
• Excellent written and spoken English skills required, , French is strongly preferred.
• Excellent writing and communication skills required.