Mid-Term Evaluation (MTE) of the URCS ONE PLAN 2023-2025 At Danish Red Cross

Terms of Reference (TOR) for the Mid-Term Evaluation (MTE) of the

URCS ONE PLAN 2023-2025

1. Summary

Purpose:

The purpose of the evaluation is to assess the mid-term progress made by the URCS and its partners in addressing the urgent humanitarian needs arising from Russia’s military attacks against Ukraine. Specifically, the evaluation will examine the effectiveness, efficiency, and relevance of the humanitarian response efforts, including the delivery of life-saving assistance and services to internally displaced people, returnees, and the host population. It will also evaluate the coordination and collaboration among the URCS, RC Movement Partners, and other international actors and agencies to ensure a comprehensive and scaled-up response to the crisis. Ultimately, the evaluation aims to provide actionable insights and recommendations to enhance the ongoing humanitarian response and better meet the evolving needs of affected populations throughout the remaining period of ONE PLAN implementation.

Commissioner(s):

Illya Kletskovskyy, Deputy Director General (Programmes), National Committee, Ukrainian Red Cross Society

Audience:

The primary audience for this MTE would be the Ukrainian Red Cross Society, IFRC Ukraine Country Delegation, Europe Regional Office, Secretariat in Geneva, and Movement partners involved in the development and implementation of the ONE PLAN. The secondary audience would be donors and the government as well as other local and international humanitarian actors.

Duration of consultancy:

Approximately 40 working days, with approximately 15 days in the field.

Estimated dates of consultancy:

August 2024 –October 2024 (Tentative)

Geographical scope of consultancy:

In Ukraine (comprising 24 Oblasts and Kyiv city), the consultant has the flexibility to suggest an appropriate sampling approach for location selection based on the response and programmatic volume and intensity outline in the URCS ONE PLAN (2023-2025)

Reporting to:

Evaluation Management Team (EMT)

2. Background

The conflict between Ukraine and the Russian Federation (RF) commenced in 2014, and tensions heightened significantly on 24 February 2022 with the onset of Russian military attacks across Ukraine. These attacks deliberately targeted both military and civilian infrastructure, leading to the annexation of territories including Luhansk, Donetsk, Zaporizhzhia, and Kherson by the Russian armed forces. The captured regions were subsequently recognized as federal subjects of the RF.

As of 1 February 2022, Ukraine’s population, excluding Crimea and Sevastopol, stood at 41,130,432 according to data from the State Statistics Service of Ukraine. The conflict’s impact has been profound, affecting an estimated 24 million people within Ukraine. The number of individuals requiring humanitarian assistance surged from 2.9 million before 24 February 2022 to 17.6 million by January 2023. Humanitarian organizations in Ukraine responded swiftly, significantly expanding their operations and reaching over 15.8 million individuals by the end of 2023. Notably, approximately one million people residing in areas temporarily under the military control of the RF faced challenges in accessing humanitarian aid. Amidst the escalating crisis, the number of humanitarian organizations in Ukraine has increased five-fold, totaling over 700 by the latest available data.

Leveraging its auxiliary status, the URCS endeavors to complement and bolster government structures to alleviate the suffering of vulnerable populations, particularly those in hard-to-reach communities that remain inaccessible to other aid providers. The URCS faced heightened demand, prompting the development of the URCS ONE PLAN 2023-25. This strategic initiative aims to extend assistance to individuals, bridge existing gaps, empower communities and local organizations, reinforce governmental efforts, and pave a clear pathway towards Ukraine’s resilience, reconstruction, and recovery. For this MTE the original version of The ONE PLAN will be used.

Even though the URCS could facilitate much beyond its current facilitation, the URCS ONE PLAN underscores merely a fraction of the programming opportunities that a robust and knowledgeable local actor such as the URCS can facilitate. Guided by innovative strategies and an optimistic outlook, the plan capitalizes on the opportunities presented by Ukraine’s developed context and robust governance structures.

URCS’s core mandate and operations (existing pre-escalation) were Home-Based Care, Health Promotion and Disease Prevention, Relief Distribution, and Search & Rescue. URCS recognized the additional needs that arose with the escalation of the conflict, and the OnePlan encompasses the provision of additional essential services that the URCS and its branches across the nation are committed to delivering.

3. Purpose and Scope

The purpose of this MTE is to assess the relevance, efficiency, effectiveness and coordination of the ongoing execution of the OP led by URCS, to inform the continued response in Ukraine, as well as to provide lessons and recommendations for the latter half of execution. Additionally, the MTE will be conducted to assess the feasibility of the ONE PLAN, whether it is strategically sustainable and the extent to which the programmatic part of the plan has been achieved i.e., Pillar 1: Humanitarian Action, Resilience and Recovery and partially the Pillar 2: coordination and communication.

The MTE will focus on the Federation-wide response including the collaboration within-country participating National Societies, the IFRC Secretariat, ICRC and other humanitarian partners as appropriate, from the beginning of OP implementation until the time the evaluators collect data.

• Relevance would include the extent the OP has been targeted to meet the needs of the most vulnerable, and the ways in which the OP can continue to support the recovery of people affected by the IAC.
• Efficiency and effectiveness would reflect on successes and challenges faced by URCS, the IFRC Secretariat and participating National Societies in ensuring effective and efficient delivery of services and aid to those in need.
• Coordination would review the support provided by actors within and outside the Movement towards the implementation of OP and the extent the interventions were complementary and avoiding duplication of efforts.

In addition to Red Cross Red Crescent stakeholders and the target population of relief assistance and services, the MTE will also consult with key external stakeholders including partner organisations and government agencies relevant to the review objectives. Geographically, the review will encompass all 24 oblasts and Kyiv city in Ukraine served by URCS as part of the IAC-raised complex crisis response. The focus will be on areas where multiple programs overlap, emphasizing regions where the volume and intensity of the programs intersect the most compared with the regions or areas with a relatively lower presence of the URCS interventions. This specification will be clarified at the outset of the review.

4. Evaluation Criteria and Key Questions

The review criteria and key questions to be addressed in this MTE are listed below as guidance and are expected to be further finetuned by the MTE team in consultation with the Evaluation Management Team and relevant stakeholders.

A. Relevance

• Is the assistance provided by the URCS with the support of movement partners i.e. ICRC, IFRC and participating National Societies relevant to the needs of the affected population including both women and men, and is of a quality and scale that meets Red Cross Red Crescent commitments and expectations?
• To what extent were the interventions and targeting of beneficiaries based on assessed and expressed needs, risks, vulnerabilities and capacities of the communities, including those who are the most vulnerable or marginalized?
• How relevant were the plans and strategies developed for the response, and what adjustments or improvements are needed to address gaps in the response at this time?
• What sectors are most relevant in the short term vs. those in which URCS’s relevance long-term is essential?
• Whether the M&E reports and progress towards targets were used as guidance for where to prioritize time and resources, exploring the relevance of the indicators with low progress to date?
• Whether The ONE PLAN is instrumental in setting up organizational systems/ structures to deliver against the plan and beyond its life?

B. Efficiency and Effectiveness

• To what extent is an effective management structure in place within URCS, IFRC, and participating National Societies providing direction, clarity, and well-communicated decision-making for the operation and the prioritization of interventions?
• What organizational systems strengths can we build on, and what weaknesses need to be addressed immediately (HR, staff care, IT, supply chain, finance, admin, security, etc.)?
• How well are the monitoring and learning processes along with information management systems used in the response and can they be improved?
• Has resourcing for the response been timely and adequate? What could be done differently to improve visibility and resource mobilization for this operation?
• How timely and effective was the mobilization of global tools including rapid response and ERUs? How well did the IFRC surge capacity build on local capacities and support the response operation?

C. Coordination

• What have been the good practices, challenges, and lessons learned in coordination, communication, and collaboration for the response operation from initial assessment and planning to service delivery:

(i) Within the URCS (at local, regional, interregional and National Committee levels)

(ii) Within the Movement, among the host URCS, participating National Societies, ICRC and IFRC

(iii) Between the Movement and external stakeholders including the government, UN agencies and other humanitarian actors.

5. Review Methodology and Process

The MTE will be managed according to the IFRC Guide and Procedures. The methodology will adhere to the IFRC Framework for Evaluations, with particular attention to the processes upholding the standards of how evaluations should be planned, managed, conducted, and utilized. For this evaluation, a ‘Review Board’ will be established to review the findings and endorse it. The Review Board will have members from URCS and key Movement Partners (ICRC, IFRC and representatives from the Partner National Societies especially those who have had a prolonged and notable presence in the nation with the experience of Ukraine Crisis Response. The Review Board will be entrusted to guide the evaluation process, including by provide strategic inputs across the whole process, from the design phase to the delivery and comment on the final report.

Interviewees will include URCS, PNSs, IFRC and ICRC personnel (e.g. sector leads, program managers, cross-cutting sector leads, field officers, direct implementers, volunteers and community mobilizers including the regional and interregional offices’ staff of the URCS), beneficiaries and potentially, other non-movement donors and partners (WFP, UNICEF, GIZ, Save the Children, etc.), and people who did not receive assistance.

The specific evaluation methodology will be detailed in close consultation between the EMT, Commissioner and Review Board, but will draw upon the following primary methods:

• Desk review of operation background documents, relevant organizational background and history, including URCS and National policies and SOPs, prior RCRC reports, and any relevant sources of secondary data, such as Post Distribution Monitoring (PDM), learning review report, and other surveys and report from Movement partners and participants in the operation.
• Field visits/observations Selected locations of the response particularly those areas of multi-sectoral and diversity of the response (complex vs simple).
• Key informant interviews (institutional and beneficiaries as appropriate).
• Focus group discussions (institutional and beneficiaries) as time and capacity allow.

The successful evaluation team will develop an inception report after the initial desktop review for submission and approval to the EMT. This inception report should outline a detailed proposed methodology that must consider:

Sampling method is to be recommended by the evaluator, as long the final sample to be evaluated on includes both URCS involved in the operation interventions, and the ‘most vulnerable’ beneficiaries.

Data collection methods and pace are to be decided by the evaluator, in consultation with the Evaluation Management Team but should consider the reality of difficult-to-reach areas. The evaluation consultant should visit a representative number of communities in the operational areas. This will be agreed with Evaluation Management Team (EMT) based on resources available.

The evaluation consultant (company/team) will need to work together with evaluation management team. The evaluation consultant (company/team) will be responsible for obtaining ethical approvals for the purpose of this MTE.

6. Review Deliverables and Illustrative Timeline

The evaluation consultant is expected to deliver the following outputs:

• Inception Report: The inception report will be a scoping exercise for the review/evaluation and will include the proposed methodologies, data collection and reporting plans with draft data collection tools such as interview guides, the allocation of roles and responsibilities within the team, a timeframe with firm dates for deliverables, and the travel and logistical arrangements for the team.
• Debriefings / feedback to management at all levels: The team will report its preliminary findings to the EMT prior to leaving the country to validate the findings in a participative manner. This will include a presentation to key stakeholders as determined by the EMT, a detailed PowerPoint presentation and sufficient time for discussion and suggested edits. This workshop will be attended by representatives from branches, sectoral leads, regional and interregional staff.
• Draft report: A draft report identifying key findings based on facts and will separate from the report opinions or rumours, conclusions, recommendations, and lessons for the current and future operation, will be submitted by the evaluation consultant within a week after presenting the initial findings.
• Final report: The final report will contain a short executive summary (no more than 1,000 words) and a main body of the report (no more than 10,000 words) covering the background of the intervention evaluated, a description of the evaluation methods and limitations, findings, conclusions, lessons learned and clear recommendations. Recommendations should be specific and feasible. The report should also contain appropriate appendices, including a copy of the ToR, cited resources or bibliography, a list of those interviewed and any other relevant materials. The final report will be submitted two days after receipt of the consolidated feedback from EMT. Details of the final report are outlined in the table below.

Suggested final report outline:

1. Executive Summary

Summarizes the overall findings of the evaluation with key conclusions and not more than 10 key recommendations. Executive Summary must be specific to the evaluation and clearly outline the specific context of the interventions.

2. Background

Outlines the overall objectives, aims, intervention strategy, policy frameworks, targets, main stakeholders, financial frameworks, institutional arrangements.

3. Methodology

Outlines the overall approach used and the rationale why the approach used, the tools applied and the key assumptions. It will focus on consideration for relevance and sustainability, coverage, efficiency / effectiveness / accountability, impact and connectedness and sustainability in function of the internal and external issues. Any limitations the evaluation faced should be listed here.

4. Findings

Outlines the findings of the evaluation based on the SADDD criteria.

5. Conclusions

Outlines the main conclusions that have emerged from the findings

6. Lessons learned and recommendations

Provides general overall recommendations, including on cross-cutting issues that affect not only the URCS programmes and operations but also on URCS ONE PLAN and strategies, advocacy, and volunteer management.

All products arising from this evaluation will be owned by the URCS. The evaluator(s) will not be allowed, without prior authorization in writing, to present any of the analytical results as his/her own work or to make use of the evaluation results for private publication purposes.

The draft and final reports will be submitted through the EMT, who will ensure the quality of the report, providing input if necessary. The EMT will submit the report to the key stakeholders interviewed for any clarifications. The Commissioner will oversee a management response and will ensure subsequent follow-up.

The following is an illustrative timeline that will be revisited and refined with more details during the inception stage of the MTE:

Time Schedule

Week 1

Activities:

• Desktop review of background information
• Initial briefings and interviews to inform the development of an inception report

Deliverables:

• Knowledge gained on the current URCS OP 2023-25.

Week 2

Activities:

• Virtual key informant interviews
• Development of detailed inception report, or data collection/analysis plan and schedule, draft methodology, and data collection tools

Deliverables:

• Inception report with detailed data collection and analysis plan, methodology, and data collection tools

Week 3-5 (in-country)

Activities:

• Field data collection in target communities according to the data collection schedule
• Participatory validation workshop

Deliverables:

• End-of-week progress report.
• Validation workshop

Week 5-6

Activities:

• Prepare the draft MTE report

Deliverables:

• Draft MTE report

Week 7-8 (intermittent)

Activities:

• Review by IFRC, ICRC, URCS, and participating National Societies

Week 9

Activities:

• Revise and submit the final MTE report.
• Final presentation on the MTE report

Deliverables:

• Final MTE report
• Final virtual presentation

1. Review Board

A review board will be formed to review, guide, and endorse the report and ensure that it upholds the key components of the IAC Ukraine crisis. The Review board will consist of maximum of three members of Movement partners and URCS DDG (Programmes) who have extensive experience of protracted crisis response and National Society Development (NSD) support and are engaged with the evaluation.

2. Evaluation Management Team (EMT)

An evaluation management team will be appointed to manage and oversee the evaluation to ensure it upholds the IFRC Framework for Evaluation. The management team will consist of six members: the Head of Quality Control and Accountability (QCA) Department of URCS, the Head of the Research Centre (if the person is already on board), and three representatives from the Movement Partners. A maximum of six people will be part of the evaluation management team. The Head of Quality Control and Accountability (QCA) Department of URCS will be the team leader.

3. EVALUATION QUALITY AND ETHICAL STANDARDS

The evaluator(s) should take all reasonable steps to ensure that the review/evaluation is designed and conducted to respect and protect the rights and welfare of people and the communities of which they are members and to ensure that the evaluation is technically accurate, reliable, and legitimate, conducted transparently and impartially, and contributes to organizational learning and accountability. Therefore, the evaluator(s) should adhere to the evaluation standards and specific, applicable processes outlined in the IFRC Framework for Evaluation. The IFRC Evaluation standards are:

The IFRC Evaluation Standards are:

1. Utility: Evaluations must be useful and used.
2. Feasibility: Evaluations must be realistic, diplomatic, and managed in a sensible, cost-effective manner.
3. Ethics & Legality: Evaluations must be conducted in an ethical and legal manner, with regard for the welfare of those involved in and affected by the evaluation.
4. Impartiality & Independence: Evaluations should be impartial, providing a comprehensive and unbiased assessment that considers the views of all stakeholders.
5. Transparency: Evaluation activities should reflect an attitude of openness and transparency.
6. Accuracy: Evaluations should be technically accurate, providing sufficient information about the data collection, analysis, and interpretation methods so that its worth or merit can be determined.
7. Participation: Stakeholders should be consulted and meaningfully involved in the evaluation process when feasible and appropriate.
8. Collaboration: Collaboration between key operating partners in the evaluation process improves the legitimacy and utility of the evaluation.

It is also expected that the evaluation will respect the seven Fundamental Principles of the Red Cross and Red Crescent: 1) humanity, 2) impartiality, 3) neutrality, 4) independence, 5) voluntary service, 6) unity, and 7) universality.

4. EVALUATOR AND QUALIFICATIONS

Evaluator(s), with knowledge of local context who will provide an independent, objective perspective as well as technical experience on evaluations, will be hired to conduct the review/evaluation. The evaluator will be the primary author of the evaluation report. S/he will not have been involved or have a vested interest in the URCS operation or context being evaluated, and will be hired through a transparent recruitment process, based on professional experience, competence and ethics and integrity for this evaluation. The evaluator will report on progress or challenges to the evaluation management team. The evaluator should have the following characteristics:

• Demonstrable experience in leading evaluations of humanitarian programmes responding to refugee and displacement crises and preferably previous experience of conducting evaluation in Ukraine or impacted countries.
• Knowledge of activities generally conducted by humanitarian organizations in the shelter and non-food items including winterization of response.
• Field experience in the evaluation of humanitarian or development programmes, with prior experience of evaluating RCRC programmes desirable.
• Strong analytical skills and ability to clearly synthesize and present findings, draw practical conclusions, make recommendations and to prepare well-written reports in a timely manner (examples of previous work).
• Previous experience in coordination, design, implementation, and monitoring and evaluation of humanitarian programmes.
• Experience in qualitative data collection and data analysis techniques, especially in emergency and recovery operations.
• Knowledge and experience working with the Red Cross Red Crescent Movement preferred.
• Demonstrated capacity to work both independently and as part of a multi-discipline, multi-national team.
• Ability to work within tight deadlines and manage with available resources.
• Fluent in spoken and written English. Advantageous to be fluent in Ukrainian.
• Strong interpersonal skills.
• Relevant degrees or equivalent experience.
• Availability for the time period indicated.