CONDUCTING A PROJECT FINAL EVALUATION

TERMS OF REFERENCE

for

CONDUCTING A PROJECT FINAL EVALUATION

Project Name: Building resilience among refugees and their Jordanian hosts (Phase II)**

Country: Jordan

Project Number: 2826-02/2019**

Project Duration: November 2019 – 28 February 2022

Name of Organization: CARE Jordan

Project Budget: € 800,000.00

Austrian Development Agency: € 745,000

CARE: € 55,000**

1. Background

CARE is a humanitarian non-governmental organization working around the globe to save lives, fight poverty and achieve social justice. CARE is committed to working with poor women, men, boys, girls, communities, and institutions to have a significant impact on the underlying causes of poverty. CARE seeks to contribute to economic and social transformation, unleashing the power of the most vulnerable women and girls.

CARE International established a presence in Jordan in 1948, created in the wake of the Palestinian refugee crisis. Over 70 years later, CARE has served the needs of the Palestinian, Iraqi, Somali, Sudanese, Syrian refugee communities, and vulnerable Jordanians. Jordan’s central position and relative political stability have contributed to its role as a safe haven for refugees.

2. Project Brief

Funded by the Austrian Development Agency (ADA), with a total amount of EUR 800,000 the “Building Resilience Among Refugees and their Jordanian Hosts” (phase II) project aims to improve access to dignified, sustainable livelihoods for the most vulnerable men and women refugees and Jordanians in highly-impacted host communities, while promoting the prevention of GBV and increasing female economic and social participation. The project is targeting the areas of Amman and Zarqa and the Azraq refugee camp and runs from 1 November 2020 – 31 October 2021. The Project has two partners specifically for the Village Saving and Loans Association (VSLA) activities; namely Sanabel Alkheir Association for Social Development in Amman, and Qudurat Association for Youth Development and Empowerment in Zarqa, both Jordanian Community Based Organizations (CBOs).

The project will reach 2,889 direct beneficiaries, specifically:

• Total direct Jordanian beneficiaries targeted will be 1037 individuals comprised of 700 in the work preparedness program, 23 in Village Savings and Loans Associations (VSLAs), 10 in VSLA phase-I, 64 grants for entrepreneurs, and 10 role models and 230 beneficiaries of cash assistance.
• Total direct Syrians targeted are 2,013 individuals (100 for work permits, 200 for life skills, 22 in Village Savings and Loans Associations, 1,500 in camp sewing and life skills training, and 191 Incentive-Based Volunteer participants in the camp).

Through the intervention, 15,731 beneficiaries, including children and the elderly will indirectly benefit from improved livelihood. 3,500 beneficiaries will benefit through media and outreach activities.

The main expected results of the project are:

· Increased awareness of rights and ability to meet legal job market requirements among vulnerable urban refugees in target communities: This result will disseminate correct and up-to-date information to refugees about the process for applying for a work permit and entering the formal labour market, especially targeting women and female-headed households. By facilitating pathways to legal work, and assisting refugees in obtaining work permits, this result will provide access to improved livelihoods to at-risk refugees and protect them from human rights violations (GBV, child labour, dangerous work, etc.) endemic in the informal job market and as they seek to meet urgent needs.

· Improved access to and participation in the labour market for vulnerable refugees and host Jordanians, especially women: Barriers that stand in the way of labour market participation by vulnerable Syrian refugees and Jordanians will be removed. Beneficiaries will be prepared to enter/re-enter the workforce through capacity building programs including but not limited to workforce readiness, life skills, and business skills, along with linking them with employers through a job fair and enhancing their saving and entrepreneurial capacities via saving groups (Village Savings and Loans Associations). Targeted groups include women and female-headed households, in order to support female participation and protect against GBV, as well as Jordanian youth, who also face particular challenges entering the workforce. Entrepreneurs will be supported through training and financing/grants.**

· Increased economic participation and ability to secure dignified, sustainable livelihoods for vulnerable refugee camp women: Through this component, vulnerable Syrian refugee camp women will be trained in advanced vocational skills such as sewing that will help them to start home-based work for additional income and self-sufficiency. Beneficiaries will also be trained in life and business skills, including marketing, GBV and women’s rights, bookkeeping and other essential aspects of entrepreneurship, and will be supported in their involvement in the Incentive-Based Volunteering program.

· Minimized the socio-economic impact of COVID-19 on vulnerable and at risks refugee and Jordan households: This component will cover the urgent needs during the COVID-19 crisis of vulnerable Jordanian households through Emergency Cash Assistance. The targeted community members will be informed about hygiene and social behaviours that are important to stop transmission and prevent the spread of COVID-19.

3. Scope, purpose, and objectives

CARE International in Jordan will commission an external qualitative final evaluation of “*Building resilience among refugees and their Jordanian hosts” (phase II)* project between 3 October 2021 and 18 January 2022 to supply credible and reliable findings for evidence-based decision-making and give account of corresponding results achieved to ADA. The final evaluation for the project will be inclusive and participatory in nature, will give full consideration to gender and other categories of marginalization at all stages of the process, be cognisant of ethical research principles, and will ensure that lessons learned are disseminated to ADA and relevant stakeholders within the project including CARE team members and implementing partners.

The final evaluation should analyse the progress the project towards its aims to enhance resilience and protection for the most vulnerable Syrian refugee and Jordanian households in the urban areas of Amman, Zarqa, and Azraq (Town and Camp). The primary users of the final evaluation will be CARE, local partners and ADA.

The key objectives of the evaluation are the following:

1. Assess performance against higher-level results.
2. Document lessons learned and best practices and facilitate future decision-making through assessment of successes and failures, analysis of what caused these, and recommendations for improvement.

4. Evaluation key questions

The final evaluation will attempt to answer the following based on four evaluation criteria, specifically, relevance, effectiveness, impact and sustainability, as well as Environmental, Gender and Social Impact Management (EGSIM).

Relevance:

1. Assess the interventions’ relevance to targeted community members. To what extent was the project aligned with the needs and priorities of men, women and other key groups? Do the intervention results respond to the needs of all stakeholders, men, women and other key groups, as identified at the design stage?

Effectiveness:

1. Examine if the project model approach is effective in enhancing resilience and protection, especially from gender-based violence for the most vulnerable Syrian refugees and host Jordanians: a) increase awareness of rights and ability to meet legal job market requirements, b) improve access to and participation in the labour market for vulnerable refugees and host Jordanians, and c) increase economic participation and ability to secure dignified, sustainable livelihoods for vulnerable refugee camp population.
2. Evaluate the satisfaction of targeted groups toward the core program interventions.
3. Evaluate the effectiveness of new modalities of work during the COVID-19 pandemic.

Impact:

1. Document key achieved results and outcomes of the project’s interventions.

Sustainability:

1. Examine the extent to which the interventions can deliver its intended benefits over an extended period of time after the end of the project.
2. Identify the factors hindering the sustainability of the intervention.

Environmental, Gender and Social Impact Management (EGSIM);

1. To what extent have EGSIM issues[1] been integrated into the design, planning and implementation of the project and the results achieved?
2. Which gender equality principles were used (e.g. equality, participation, social transformation, inclusiveness, empowerment, etc.) in the implementation of the intervention and the results achieved?

5. Evaluation approach and methods

The methodology must include qualitative and quantitative methods to allow for sufficient triangulation and robust findings. These will likely include literature reviews, survey of beneficiaries (with a representative sample size), key informant interviews, focus group discussions with beneficiaries. Particular emphasis is placed on the full integration of gender and ethical research considerations in the selected data collection and evaluation tools and methods. The key tools foreseen for the evaluation are:

• Desk review: The evaluation team need to conduct desk review of the project documents, baseline report, progress reports, and any other relevant documents.

• Survey: The evaluation team will conduct a beneficiary’s survey to assess changes on key outcome indicators and prospects on impact. The scope of the survey is expected to be comparable to the project baseline survey. The confidence level is 95% and margin of error is 5%.

• Focus Group Discussions (FGD): The evaluation team will conduct gender sensitive focus group discussions with key stakeholders to assess implementation experiences and effectiveness, document successes, challenges and lessons learned, and develop recommendations for improvement.

• Key Informant Interviews (KIIs): The evaluation team will visit various stakeholders and undertake key informant interviews using pre-designed checklists and get wide-ranging and in-depth feedback and learning. The visit will include selected project key stakeholders including relevant local government representatives.

The evaluation consists of three phases:

1. Inception Phase

Contract, Kick-Off and Access to Data: Contract is signed and a discussion of the assessment takes place with the evaluation team. First documents, including project documents, log frame, progress reports, baseline, project Monitoring and Evaluation Matrix, assessment reports, etc. The inception phase includes:

• Desk Review/Study: The evaluation team studies all necessary project documents; re-constructs and analyses the intervention logic and its assumptions. Existing data needs to be analyzed and interpreted.

• Inception Report: In the inception report, the evaluators will describe the design of the evaluation and will elaborate on how data will be obtained and analyzed. The use of a data collection planning worksheet or a similar tool is required. On the basis of the information reviewed and analyzed, the evaluation team shall prepare and submit an inception report including details of methodology, sampling procedure and size, quantitative and qualitative data collection tools, detailed work plan with indicative targets to be interviewed, surveys to be undertaken, dates of field work/itinerary, and name of team members in charge, if any. The survey data collection tool shall be submitted in both English and Arabic version for review and comments. The evaluation team is required to undertake a sample pre-test of its data collection tools at field level and adapt the tools for data collection based on the pretest findings before actual data collection. Data triangulation and quality control are very important and need to be discussed in the inception report. The field trips will only take place upon official approval of the inception report by CARE. The inception report shall be submitted both in hard and electronic copies, as per the reporting requirements (see ADA Guidelines for Project and Programme Evaluations – Annex 5 of the Guidelines).

1. Data Collection and Interview Phase

Data needs to be gathered, analyzed and interpreted. Interviews with selected beneficiaries and stakeholders are conducted on the spot – adjusting the data gathering methods to fulfil all protection measures related to the COVID-19 pandemic – or via Zoom in case of traveling and gathering restrictions linked to the COVID-19 pandemic. During the field data collection, the evaluation team shall hold an independent briefing meeting with the appropriate project field staffs; ensure adequate contact and consultation with, and involvement of, the different stakeholders. At the field, the evaluation team will collect the most reliable and appropriate sources of information and will harmonize data from different sources to allow ready interpretation. It is mandatory for the evaluation report to include quantitative and qualitative data disaggregated by age, sex and vulnerability group.

Debriefing of Partners: It is expected that the evaluation team will summarize the findings of the field work, discuss the reliability and coverage of data collection. Presentation of preliminary findings after first collection of data and field trips in a meeting with CARE Jordan, and via skype or video conference with CARE Austria (if needed).

1. Data Analysis and Final Report

· Data Analysis/Synthesis and Submission of Final Report Draft: This phase is mainly devoted to the analysis of data and interpretation of the results, preparation of the draft report and submission of the draft of the final report, for CARE to provide feedback. The evaluation team will make sure that the evaluation survey is objective and balanced, affirmations accurate and verifiable, and recommendations realistic. It is expected that the evaluation team will present concrete recommendations which are addressed to the specific stakeholders. The data analysis and presentations shall include data disaggregated by sex, age and vulnerability status. Following the data analysis, the final draft report shall be submitted both in hard and electronic copies, as per the reporting requirements (see ADA Guidelines for Project and Programme Evaluations – Annex 6 of the Guidelines). The reports have to be submitted in English.

If CARE Jordan considers the final draft report of sufficient quality, they will circulate it for comments to CARE Austria, donor/ADA and others, and convene a meeting in the presence of the evaluation team. On the basis of comments expressed by CARE’s team, and collected by the focal person, the evaluation team will amend and revise the draft final report.

· Validation Workshop/Seminar: Design and deliver a one-day workshop (in Arabic or English) with the project staff and project partners, for debriefing on preliminary findings of the Final evaluation and discussing implications on the implementation and integration of the evaluation recommendations into future project interventions. On the basis of comments made by workshop participants, and collected by the focal person, the evaluation team has to write the final version of the report, in which the rules applying to the integration of comments are those stated in the previous section. The final evaluation report will be published on the ADA and CARE websites.

· Submission of Final Report: After amending and revising the final draft report as per the comments of CARE, ADA and others, the evaluation team will submit the final report in both hard and electronic copies, as per the reporting requirements.

The Guidelines for Project and Programme Evaluations developed by the Austrian Development Agency^[2] need to be considered throughout the entire evaluation process. OECD/DAC Evaluation Quality Standards are to be applied and compliance with these standards is deemed necessary for the credibility of the evaluation. The evaluation team must follow the norms, principles, and standards as well as ethical guidelines for evaluations that are specified by the Development Assistance Committee of the Organisation for Economic Cooperation and Development (OECD/DAC), the German Evaluation Society (DeGEval) and the United Nations Evaluation Group (UNEG) as well as the Evaluation Cooperation Group of the international financial institutions (ECG). Also, the evaluation team should identify potential harms for participants of the evaluation and for evaluation team and should develop mitigation measures (Evaluation policy of the Austrian development cooperation).**

The evaluation team is expected to submit all the reports in English: one copy in hard copy/paper and one copy in electronic version. The executive summary should summarize key findings and recommendations (three to five pages) and needs to be submitted as part of the final draft report. The findings and recommendations of the draft final report and final report have to be structured according to the evaluation questions.

CARE’s reference group will be actively engaged in the evaluation process to contribute to the quality and credibility of the evaluation. Quality assurance, including the verification of factual accuracy will be conducted both in the design phase and implementation of an evaluation. It is performed at four key points in the evaluation process: (i) the terms of reference, (ii) the draft inception report, (iii) the draft evaluation report and (iv) the final evaluation report.

6. Key deliverables

All deliverables must be submitted in English. They are:

1. Draft and final inception report (maximum 10 pages, excluding annexes)
2. Draft and final evaluation report (maximum 30 pages, excluding annexes), including the Results Assessment Form (RAF)[3]
3. A PowerPoint presentation based on the draft report for the validation workshop

Structure and content of the deliverables (inception report, evaluation report and RAF) must follow the related templates and checklists in the ADA Guidelines for Programme and Project Evaluations.[4]

The evaluation team must follow the standards and principles of the Austrian Development Cooperation for evaluation, which include independence, impartiality, credibility, transparency, utility, feasibility, fairness, accuracy, participation, and partnership. The quality of the reports will be judged and accepted according to the criteria set in ADA Guidelines for Project and Programme Evaluations.

7. Work Plan

The dates will be confirmed based on the date of signing the contract for this assignment**.**

Desk Review and Inception phase (LOE (evaluation team): 16 working days)

A. Time Frame: 3- 9 October-2021

Activity:

• Finalizing the evaluation plan
• Conducting a Desk Review
• Developing the data collection and analysis tools
• Sharing the inception report with the reference group for their feedback and approval.

Responsibility: The evaluation team **

B. Time Frame: 10 -14 October 2021 **

Activity:

• Reviewing the inception report including tools by the reference group

Responsibility: CARE and ADA

C. Time Frame: 15 -19 October 2021[5]**

Activity:

• Sharing the updated version of the inception report

Responsibility: The evaluation team **

D. Time Frame: 20-21 October 2021**

Activity:

• Reviewing the second version of the inception report**

Responsibility: CARE and ADA**

E. Time Frame: 22 -26 October 2021**

Activity:

• Sharing the final version of the inception report

Responsibility: The evaluation team

Data Collection phase (LOE (evaluation team): 23 working days)

A. Time Frame: 27 October– 18 November 2021 **

• Data collection

Responsibility: The evaluation team

B. Time Frame: 19 – 29 November 2021 **

• Data analysis and writing transcriptions and coding

Responsibility: The evaluation team

Synthesis and Final Report phase (LOE (evaluation team): 14 working days)

A. Time Frame: 30 November – 14 December 2021

• Report Writing

Responsibility: The evaluation team

B. Time Frame: 15 – 22 December 2021

• Reviewing the first draft report by the members of reference group.

Responsibility: CARE and ADA

Validation workshop (LOE (evaluation team): 1 working days)

A. Time Frame: 23 December 2021

• Conducting Validation Workshop/Seminar**

Responsibility: The evaluation team and CARE

Draft final report

(LOE (evaluation team): 3 working days)

A. Time Frame: 27 December 2021 – 5 January 2022

• Amend and revise the draft report and share the second version of the evaluation report

Responsibility: The evaluation team

B. Time Frame: 6-12 January 2022

• Reviewing the second draft report by the members of reference group.

Responsibility: CARE and ADA

(LOE (evaluation team): 3 working days)**

A. Time Frame: 13 – 18 January 2022

• Finalizing the final evaluation reports based on the feedback of the members of reference group.**

Responsibility: The evaluation team **

Any changes to the dates should be reported and discussed with CARE Jordan before making any amendments.

8. Reporting Requirements and relationship

All reports should be submitted to CARE Jordan, and will be only accepted in quality and professional English language and according to the criteria set in ADA Guidelines for Project and Programme Evaluations.

The evaluation team leader reports to the Programs Quality focal point with the support of the Project Manager.

It would be obliged for the evaluation team leader to inform the Quality and Accountability Coordinator about any protection case, taking into account the ethical considerations. The evaluation will be oversight of a Reference Group consisting of representatives from CARE Jordan and CARE Austria.

The main responsibilities of CARE International in Jordan are:

§ To ensure that the consultant team has access to and has consulted all relevant information sources and documents related to the project;

§ To validate the evaluation questions, primary data collection tools; and planned evaluation methodology, together with ADA, as part of the Inception Report;

§ To discuss and comment on notes and reports delivered by the consultant team. Comments by individual group members are compiled into a single document by the consultant team leader and subsequently transmitted to the consultant team;

§ To assist in feedback of the findings, conclusions, lessons and recommendations from the evaluation.

§ Invite the participants to attend the FGDs.

§ Arrange for the KIIs.

§ Conduct phone-based surveys.

§ Sharing the quantitative survey data.

The main responsibilities of the consultant are:

§ Conduct desk review for the project’s documents.

§ Develop the inception report, which includes the detailed methodology, sampling, tools for FGDs, survey, and KIIs, and detailed work plan.

§ Conduct the FGDs virtually or in-person.

§ Conduct the KIIs.

§ Analyse the qualitative and quantitative data.

§ Writing the final evaluation report.

§ Sharing the Results Assessment Form (RAF).

§ Sharing a PowerPoint presentation based on the draft report for the validation workshop.

§ Conduct a validation workshop.

9. Financial Arrangements

At the submission and CARE’s approval of each of the following milestone, CARE will pay the agreed upon amount within 30 days from submitting the invoices, and as follow:

§ 20% after the submission of inception report.

§ 30% after the submission of first draft report.

§ 50% after the submission and approval of final report and deliverables.

10. Qualifications

§ The evaluation team leader should have a minimum of five years’ experience in leading evaluations for INGO and in the field of sustainable development and economic empowerment. Evidence of experience is required.

§ In case of multiple team members, all team members should have a minimum of three years’ experience in conducting evaluations, including data collection and analysis

§ Demonstrated experience in using human rights based approach and gender responsive approaches.

§ Demonstrated experience with quantitative and qualitative research methods.

§ Strong analytical skills and writing abilities.

§ Knowledge of the Jordanian context will be considered an asset.

§ Fluency in English and Arabic is required.

CARE will be evaluating the received applications based technical criteria (50%) and the received net financial offer (50%).

Notes:

1) *Concluding the assignment in high quality and timely manner is of great essence.*

2) The total amount of the contract value includes all requirements needed to carry out the consultancy, including transportation, stationery, and any other cost of any related logistics.

3) The outcomes and deliverables (reports, photos, and any other outcome) of this consultancy is an intellectual property for CARE International in Jordan, and it’s not allowed to be used by the consultant in any format, nor disseminated or announced through any media channel without prior written approval by CARE Jordan.

4) Any media outlet, should be approved by CARE Jordan, and should include and attribute the credit to CARE Jordan. **

Annex I – Reference Documents

Brief project information from ADA webpage: https://www.entwicklung.at/en/projects/detail-en/staerkung-der-resilienz-von-fluechtlingen-und-ihrer-jordanischen-aufnahmegesellschaft-phase-ii

ADA Guidelines for Programme and Project Evaluations available at: https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluierungs_Leitfaeden/Guidelines_for_Programme_and_Project_Evaluations_ADA_2020.pdf

ADA Manual Environmental, Gender and Social Impact Management, available at: https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluierungs_Leitfaeden/Guidelines_for_Programme_and_Project_Evaluations_ADA_2020.pdf

[1]All issues relevant to ADA’s EGSIM Manual need to be assessed in relation to each evaluation criteria and evaluation question. The EGSIM Manual can be found in the following link:

https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Publikationen/Handbuecher/Environmental_and_Social_Impact_Management/EGSIM_Manual_Juni2018.pdf

[2] ADA Evaluation Guidelines: https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluierungs_Leitfaeden/Guidelines_for_Programme_and_Project_Evaluations_ADA_2020.pdf

[3]https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluierung_Templates/Annex9_Results_AssessmentForm_Template.xlsx

[4]https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluierungs_Leitfaeden/Guidelines_for_Programme_and_Project_Evaluations_ADA_2020.pdf, See in particular Annex 5 for Inception Report, Annex 6 for Evaluation Report and Annex 9 for the Results Assessment Form.

[5] Eid Al-Adha’s holiday: from 19 to 22 July 2021