Consultant to conduct an external final evaluation of its project in Ukraine At Caritas Austria

Caritas Austria seeks a consultant to conduct an external final evaluation of its project in Ukraine. The consultant will guide and facilitate an external final project evaluation and lessons learned exercise, including preparation and facilitation of an online workshop with relevant project stakeholders and the preparation of a comprehensive report.

The project “ ECHO Protection “ is funded by the European Union and Caritas Austria and implemented by Caritas Ukraine.

The intervention aims to enhance the ability of conflict-affected girls, boys, women and men in Ukraine to protect and sustain their lives. It provides assistance to vulnerable girls, boys, women and men, including IDPs and host communities across a wide range of different sectors. To respond to the multitude of identified needs of affected populations, the implementation is based on a location-specific plan to tailor assistance to the unmet needs of beneficiaries.

Protection services include case management and legal support, psychosocial support, and Сhild-Friendly Spaces (CFS) for girls and boys, as well as support to cover basic needs as regards access to food and hygiene through in-kind modality to prevent negative coping mechanisms. In the regions where the project currently is being implemented, there are unmet needs of people with disabilities (PwD) – particularly people who have acquired disabilities as a result of the war. Assistance is provided in the form of i) individual hygiene kits for PwD – supplementing family hygiene kits – and ii) strengthening the crisis centre team with an additional case manager who will provide assistance to PwD.

Objectives:

To contribute to emergency aid and increased resilience and protection of vulnerable girls, boys, men and women affected by the armed conflict in Ukraine.

Project outcomes:

Vulnerable IDPs, refugee and host community households have an improved ability to protect and sustain their lives. The intervention aims to enhance the ability of conflict affected girls, boys, women and men in Ukraine to protect and sustain their lives. The intervention will therefore provide assistance to vulnerable girls, boys, women and men, including IDPs, refugees and host communities across a wide range of different sectors.

Expected results of the project:

• Vulnerable women, men, boys and girls received immediate basic food support.
• Vulnerable women, men, girls, and boys received immediate basic hygiene support.
• Vulnerable women, men, girls and boy have access to quality protection services.

Locations: Sumska oblast, Chernihivska oblast, Zaporizhzhia oblast, Kharkivska oblast

Project duration: 01.04.2023 – 31.07.2024 (16 months).

The purpose of this project evaluation is to both capture the results and outcomes achieved during the implementation of the project and to identify key lessons learnt by the implementing partners. The project evaluation will include reviewing outputs and analyzing short term outcomes reached through interviewing the different project stakeholders (implementing organizations and beneficiaries) and getting feedback in the project evaluation and lessons learnt workshop at the end of the process. This workshop which will be facilitated by the consultant will give room for further discussion and allow for deeper participation of all implementing partners in the evaluation process. The final report will include a section on recommendations for future action based also on the interview and workshop results and embedded in a wider humanitarian context of learning, innovation and future anticipation. The final report should provide an understanding of the cause and effect links between: inputs and activities, and outputs, outcomes and impacts. The final report should serve accountability, decision-making, learning and management purposes.

The scope of the evaluation is the whole ECHO Protection project with the aim to focus on the thematic sector of the protection activities, emphasising the relevance and effectiveness of the interventions provided.

The protection activities in the project are provided by the Caritas Crisis Centres (CC) and Child Friendly Spaces (CFS). They aim to provide complex accompaniment to persons with protection needs, who are finding themselves in challenging life situations, mainly caused by the war in the country.

The CCs offer a range of complimentary services:

• crisis consultation
• psychosocial support
• legal advisory
• referral system (by hotline and social worker)
• emergency case management
• cash for complex social accompaniment
• community facilitation and engagement.

The CFS provide support and respite to children and parents through education, psycho-social support and entertainment activities. These activities are conducted by the following team:

• Coordinator
• Child Psychologist
• Social Pedagogue
• Animator

Despite the fact that the evaluation is performed at the end of the project, the main objective is to learn how the activities of the CCs & CFSs can be improved. Caritas Ukraine has been working since 2017 with their CCs approach, by constantly adapting methodologies to the evolving needs.

Within the project at hand, the target group of CCs has been extended to people with disabilities (PwD), which are mainly individuals with physical disabilities and war-caused disabilities. In order to evaluate the piloting of working with the additional target group, a special attention is expected to PwD.

The focus should be on the following two criteria:

• The relevance of the intervention
• The effectiveness of the intervention

Proposed Evaluation Questions (to be developed further by the consultant):

General:

• If the project were to continue, how could we improve it to better deliver on our objectives for beneficiaries in the area of protection.
• Which practices did not produce the expected results and outcomes or were not appropriate to the needs of the beneficiaries?
• Which practices worked well to produce the expected results and outcomes for the beneficiaries involved in the project?
• In relation to the effectiveness of the project implementation, what worked well for local Caritas project staff and what elements should be improved?

Protection:

• To what extent were the protection services effective and relevant to the needs of beneficiaries?
• To what extent do protection services, including referrals, in the CC complement each other?
• Where referrals effective for beneficiaries, in line with international standards?
• What improvements to the provision of protection services should be recommended?

Relevance:

• To what extent was the intervention provided according to (fast changing) local context and needs?
• To what extent was the intervention implemented according to donor requirements?
• What have been the supporting criteria to foster project’s relevance? What have been hindering aspects?
• Were the selection criteria for beneficiaries relevant and appropriate in targeting the most in need of protection services?
• Were there any exclusion patterns identified as a result of the beneficiary selection process?
• Has there been a risk of duplication with similar interventions by any other actor in the area with similar activities (Poshmishka for example)?

Effectiveness and efficiency:

• To what extent has the project achieved its intended results or outomes, with a focus on Protection?
• To what extent have PwD benefitted from the results and outcomes of this project?
• What factors were decisive for achievement/non-achievement of the intended results?
• To what extent have interventions contributed to strengthening local capacities?

Coherence

• How coherent was the project implementation with DG ECHO policies? How coherent was the project with Caritas internal policies?
• How coherent was implementation with other relevant international standards and guidance?

METHODOLOGY

The evaluation will be undertaken in line with OECD/DAC standards for evaluations (Applying Evaluation Criteria Thoughtfully, 2021). The final methodology and questionnaire are to be designed and elaborated by the evaluation team.

The following methods should be used:

• Desk review, including the narrative proposal, the logframe of the project, reports as well as relevant secondary data
• Key informant interviews (KII) with the key staff members for the project
• Focus group discussions (FGDs) with beneficiaries and other stakeholders. Depending on the security situation online or offline formats need to be considered. Due to vulnerability of target groups safeguards for confidentiality, privacy, and respect of dignity need to be respected. In addition, certain groups may not be interviewed directly but questions may be transmitted by the staff or case worker due to their safeguarding needs
• Please consider including a relevant external expert in the FGD/KII. Eg. a relevant member of the Protection Cluster.
• Project evaluation and lessons learnt workshop with Caritas Austria and Caritas Ukraine and local partners in August 2024. Preferably in an online format. The consultant is asked to suggest an impactful/innovative/participatory methodology for the workshop
• The report and the workshop are supposed to be in English, interviews can be held in Ukrainian or Russian

While the above mentioned are desired methods by Caritas Austria, we remain open to additional suggestions regarding methodology based on the expertise of the consultant, which should be outlined in the technical proposal.

RESPONSIBILITIES AND CONDITIONS

Confidentiality of information: All documents provided to the evaluator(s) and all data collected during field research must be treated as confidential and used solely for the purpose of this evaluation. Respondents shall not be identified in reports without their explicit written permission. Photo, video, and other research data that can be traced back to individual research participants shall be anonymized unless agreed otherwise, in writing, with the person concerned.

Ownership of data, findings, products: Caritas Austria retains full ownership of all data, findings, and products produced as part of this assignment.

Support by CAT and partners: All relevant documents including programme proposal, reports, etc. will be provided for the consultant(s). Field research can be supported and facilitated by Caritas Jordan based upon the methodology and approach defined. Relevant contacts of other stakeholders will be shared upon the consultant(s)’ request.

Safeguarding and protection: For the time of the assignment, the evaluator commits to safeguarding and protection policies and procedures that are shared and reiterated during the preparatory phase.

Distribution Policy: Intended users of deliverables are Caritas Austria, Caritas Ukraine, back donors of Caritas Austria, other partners of Caritas Austria (e.g. Caritas International Member Organisations).

Absence of conflict of interest

The tenderer must not be affected by any conflict of interest and must have no equivalent relation in that respect with other tenderers or parties involved in the project. Any attempt by a tenderer to obtain confidential information, enter into unlawful agreements with competitors or influence the evaluation committee will lead to the rejection of its tender.

Respect for human rights as well as environmental legislation and core labour standards

The tenderer and its personnel must comply with human rights and applicable data protection rules. In particular, and in accordance with the applicable basic act, tenderers and applicants who have been awarded contracts must comply with the environmental legislation including multilateral environmental agreements, and with the core labour standards as applicable and as defined in the relevant International Labour Organisation conventions (such as the conventions on freedom of association and collective bargaining; elimination of forced and compulsory labour; abolition of child labour).

Zero tolerance for sexual exploitation, abuse and harassment:

Caritas Austria applies a policy of ‘zero tolerance’ in relation to all wrongful conduct which has an impact on the professional credibility of the tenderer.

Physical abuse or punishment, or threats of physical abuse, sexual abuse or exploitation, harassment and verbal abuse, as well as other forms of intimidation shall be prohibited.
Anti-corruption and anti-bribery

The tenderer shall comply with all applicable laws and regulations and codes relating to anti-bribery and anti-corruption.

All deliverables generated through this evaluation will be subject to approval by CAT before public dissemination.

Safety and Security

During the time of field travel the consultant is obliged to follow the Caritas Country Forum security protocol. CAT will provide security briefing prior to the Ukraine mission.

The consultant has to take care of Ukraine insurance her-/himself, covering health treatment and war risk. CAT can provide recommendations on insurance companies. Insurance costs can be covered within the cooperation contract.

CAT will not take legal liability for the consultant, travelling in Ukraine.