TERMS OF REFERENCES FOR THE FINAL EVALUATION OF WAXBARASHADU WAA IFTIIN III (EDUCATION IS LIGHT III) PROJECT (in Puntland State of Somalia) At CARE

1. INTRODUCTION

CARE is an International NGO working in Somalia. CARE and its partners work with vulnerable communities to address the underlying causes of poverty, promoting peace and development, by supporting sustainable livelihood and economic development initiative programmes, strengthening civil societies, responding to emergencies, and advocating for policy change.

The Puntland State of Somalia is amid momentous political, social, and economic transformation since its leaders declared the territory an autonomous region in 1998. The region has managed to maintain relative peace and security in recent years, which has enabled it to establish political and administrative institutions, basic social services, an active civil society, and a growing private enterprise community. In addition, much progress has been made in the education sector in the last few years. Despite these improvements, however, educational provision, participation and completion in Puntland is still among the lowest in the world. The GER[1] for primary and secondary education as per the Ministry of Education and Higher Education (MoE&HE) 2019/2020 EMIS stands at 71.3% (Girls: 66%; Boys: 76%) and 20.5% (Girls: 16.2%; Boys: 24.5%) respectively.

The major challenges experienced in the education sector are multifaceted factors and include weak technical and institutional capacity of the MOEHE, low access to quality, relevant and equitable education system at all levels (Early childhood. Primary (including special needs education), non-formal (NFE), Alternative Basic Education (ABE), secondary Education, Higher Education and TVET) as well as limited pedagogical skills and the availability of teachers, especially in rural and nomadic areas. All these constraints lead to poor service delivery in Puntland state.

It is against these backdrops that a consortium comprising of CARE and Save the Children are implementing a three-year EU-funded Waxbarashadu Waa Iftiin II (Education is Light II) project to address the challenges in close coordination, support, and guidance of the MoE&HE in Puntland. The project aims to improve the educational attainment and the employability of relevant target groups, contributing to poverty reduction within an increasingly peaceful, secure and democratic Somalia and specifically consolidates and expands, equitable and inclusive quality education, and vocational training opportunities for Somali children, youth and adults.

Through this ToR, CARE (lead agency) is planning for a final evaluation of the 3 year consortium project to ascertain the extent of implementation and achievements against the set indicators as well as establish lessons learnt and recommendations to inform the subsequent phase of the project on the needs and priorities of the MOE&HE for continuity and consolidation of the existing gains of the EU funded consortium project. The project began on 24^th November 2018 and is scheduled to end on 24 November 2021. The project interventions are implemented in 9 regions (Bari, Cayn, Gardafuu, Hayland, Karkaar, Mudug, Nugal, Sanaag and Sool regions) of the Puntland State of Somalia and are expected to achieve the following objectives and results;

*Overall objective:**** Improve the educational attainment and the employability of relevant target groups, contributing to poverty reduction within an increasingly peaceful, secure, and democratic Somalia.*

*Specific Objective: Consolidated and expanded, equitable and inclusive quality education, and vocational training opportunities for Somali children, youth, and adults.**

Results:

Result 1: Increased access to quality education and training for all children, youth and adults including vulnerable groups

Result 2: Strengthened public systems for effective education service delivery

1. OBJECTIVES OF THE EVALUATION

The results of this evaluation are expected to provide a detailed measure of the degree to which stated objectives and results were achieved (or not) in the project. It should generate relevant findings, lessons learned and recommendations which will be shared with key stakeholders of the project and used by the implementing agencies to guide and inform on the best practices and strategies in for improving other future phases and education programs. The end term evaluation of Waxbarashadu Waa Iftiin III will provide some useful augmenting base line data for Waxbarashadu Waa Iftiin IV. The final evaluation will have full participation of the project staff and stakeholders.

The specific objective of the evaluation is to: “provide information on the performance of the project against all indicators and the project’s overall relevance, effectiveness, efficiency, validity of design, sustainability, factors affecting performance, alternative strategies and its strengths and weaknesses.

1. THE SCOPE OF THE EVALUATION

The Consultant will perform an outcome and impact level assessments. He/she should take into consideration the evaluation criteria of relevance, effectiveness, efficiency, sustainability and project performance rating. The criteria should include assessment of all the different components of the project including Impact and Sustainability, activities implemented or not done about the indicators outlined in the project log frame. Other cross cutting approach that should looked in include human rights, gender sensitivity, inclusivity, safety and protection, promoting health, and community engagement.

The consultant will use the rating scale below for grading the level of achievement of the main activities implemented by the project and give a summative mean grade for all the activities evaluated and graded. The reason for choice of a score should be adequately explained in detail and examples provided to justify the scores. The mean grade will provide the overall performance of the project.

Table 1: The Rating Scale

IMPLEMENTATION: The activity implemented:

SCORE

IMPACT/Likely: Achievement

A completely appropriate, efficient, and timely manner

1

Completely achieved

A largely appropriate, efficient, and timely manner

2

Largely achieved

A moderately appropriate, efficient, and timely manner

3

Partially achieved

An appropriate, efficient, and timely manner to a very limited extent

4

Achieved to a very limited extent

Neither an appropriate, nor an effective or timely manner

5

Not achieved to any discernible extent

Unverifiable

X

Unverifiable

Differences among areas of intervention, related to different constraints and problems, should be analysed, and reported for the different level of analysis. The evaluator should assess the following:

Relevance

• Investigate and make conclusions on the appropriateness of project concept to the problems it was supposed to address considering the prevailing political and social economic situation in Puntland.
• Review the complementarity of the project with other CARE interventions in Puntland and particularly how this project has contributed to the reduction of vulnerability through the provision of quality primary education and non- formal education.
• Review the project design including the risks and assumptions and their impact on the project.

Efficiency

• Assess how well the project activities transferred the available resources into the intended results in terms of quantity, quality, and time.
• Assess how the project strategy used induced sustainable change, particularly in regard to establishment of Consortium relationships with key actors and participatory processes.
• Assess how inputs and means have been converted into activities and the quality of the results achieved.
• Analyse the efficiency of the project implementation and monitoring at result level using the indicators of the log frame and the proposed timetable.
• Assess the level of collaboration with other agencies and the various project stakeholders.
• Verify the appropriateness of the consortium approach and the innovativeness and response by project management to bring changes to the social environment in which the project operates.

Effectiveness

• Measure the extent to which the project achieved its outcomes and purpose using the OVIs column of the Logical Framework
• Assess whether planned benefits have been delivered and received as perceived by collaborating partners and target groups and will continue towards the intended purpose.
• Analyse the effectiveness of project strategies and capacity building efforts to determine if they are sustainable.
• Determine the progress made in achieving results of the project at all levels.
• Assess the quality of operational work planning, budgeting and how the project managed the originally identified risks and others that may not have been foreseen.
• Assess the quality of reporting.
• Check if the assumptions were correct and if not, how this has affected the project achievements.
• Assess the quality of the developed training curricula.

Impact

a) Analyse whether the project has made milestones that will contribute to the WWI programme goal of sustainably improving access and literacy levels of school aged children and adult learners and other disadvantaged groups.

b) Determine the impact the project has made on access to primary education with a particular focus on girls and minority groups.

c) Assess ownership of the WWI by the Government/regional authorities in Puntland, CECs and the general population and commitment to continue with the support after the expiry of the current initiative.

Sustainability

d) Evaluate the stakeholder participation in the management/implementation of the project.

e) Measure the level of local ownership and confirm whether key stakeholders are committed to continue and build on the achievements of the project.

f) Confirm whether the community has the intent and capacity to continue the support activities (of the current initiative) including financial and technical capacities.

g) Assess if the regional environment, socio-economic, political, and cultural factors are conducive to the continuity of the project support activities.

h) Assess to which level sustainability has been considered in planning and execution of activities particularly in respect to institutions, formal and non-formal, as well as technical and cross cutting issues.

i) Validate the exit strategy in place to hand over the Project to support WWI at the end of the programme.

j) To enable a possible replication, the evaluator is to check whether the process and lessons learned are being documented appropriately so they can be translated into a methodology which will enable others to replicate the activities more easily and effective (multiplier effects).

Cross-cutting Issues:

k) Particular attention will be given to special needs, vulnerable and marginalized groups, human rights, gender, inclusion, cultural identity, peace building, safety and protection, environmental, HIV/Aids education, and health/hygiene promotion in the project.

Based on the above-mentioned assessment areas,

• Measure the extent to which the project is successful in achieving its purpose in the targeted regions. Analyse strengths and weaknesses of the project and identify lessons learned.
• Develop detailed recommendations for how to strengthen: project design, project management (including monitoring); implementation (including approaches) and relations with stakeholders in the project area. The recommendations should be placed into the context of how the project should be implemented to maximize impact going forward.
• EVALUATION METHODOLOGY

CARE Somalia proposes a participatory, multi-stakeholder approach in undertaking this assignment. The project evaluation will employ a mixed method approach in order to draw conclusions on the contributions in improving quality education in Puntland. Both qualitative and quantitative methodologies of data collection should be employed.

a) Literature review

Literature review will involve a study of various project and MoEHE documents including the two documents earlier mentioned (Annex 1). The consultants will also be provided with the following key documents for review and background information: the WWI III Log-frame (Annex 2), Project Proposal (Annex 1), Project Funding Contract between CARE and EC (Annex 3), Project Baseline Survey (Annex 4), the project MTR report and various projects implementation reports to the EC plus any other document considered necessary for this exercise.

b) Data collection

Quantitative methods using structured questionnaires and ‘School fact establishment tools’ (adopted from the baseline) will be used to establish quantitative benchmarks reflected in the log frames. Focus Group Discussions, Individual questioning with a scientific representation of all the stakeholders will be used to collect detailed qualitative information. It is expected that the consultant will adapt the baseline tools and come up with innovative ways of combining the various research tools and methodologies to get the right and accurate information.

During the study, the consultant will interview all the stakeholders including beneficiary communities, students, Ministry of Education/regional education authorities, Community Education Committees, School children teachers, head teachers and staff of implementing consortium partners (CARE, Save the Children and MoEHE), men/women groups, religious leaders, opinion leaders, women leaders and other actors including Local and International NGOs in the region.

c) Sampling design and procedure

The 9 regions (Bari, Cayn, Gardafuu, Hayland, Karkaar, Mudug, Nugal, Sanaag and Sool regions) of the Puntland State of Somalia will form the sample frame. Both simple random and purposive sampling will be used to determine the sample population based on the consensus reached between the Lead Agency, the Consortium members and the MOEHE. Simple random sampling will be applied at school level to select teachers, students, and parents, while purposive sampling will be applied to select the head teachers, centre managers, and CECs.

Data Collection Instruments. The consultant will use the five (5) data collection instruments that were used to capture qualitative data at baseline and adopt the same for the EoP evaluation. The instruments include semi-structured interview questionnaires; one- on-one personalized in-depth interview questionnaires for key informants; focus group discussions (FGDs) with discussion guidelines; institutional fact establishment sheets. The questionnaires will be administered to the respondents, while Institutional fact establishment sheets will be used to collect specific quantitative and qualitative facts at the sampled institutions, including physical and non-physical facilities, enrolment, teacher establishment, etc. **

d) Reporting and feedback

After the evaluation, the consultant(s) is expected to do a debriefing with the with the MOEHE and the consortium partners and project staff on the ground and incorporate their relevant inputs. The consultant(s) should then prepare a draft report and make a presentation to the MOEHE and consortium partners and EC delegation in Nairobi for their inputs after which the final report is prepared. Both draft and final report should have a power point summary not exceeding 20 slides. The main text of the final evaluation report should be presented using Arial font 11 and not exceed 50 pages inclusive of Annexes. The Executive Summary in the main report should not be more than five pages with fully cross-referenced findings and recommendations. In addition, a short, separate summary of one page should be provided during the presentation of the final report.

The consultants/Consulting firm needs to produce five well bound hard copies of the final report and soft copies of all the reports sent electronically to a list of recipients in the consortium to be provided. The electronic soft copies should be in Adobe Acrobat and MS Word.

1. EXPERTISE REQUIRED

The consultants/consulting firm should be an experienced and have expertise in the following: –

· Extensive experience in research work and in education assessments/evaluations, including previous working experience in Somalia.

· A degree in education planning and management and/or curriculum development or other relevant fields. Postgraduate qualification in the same will be an added advantage.

· Demonstrated experience in community development including community targeting, rapid rural appraisal, and project cycle management.

· Knowledge in research methodologies and application of various tools including practical experience in assessments in EGRA/EGMA, planning, implementation, monitoring and evaluation of community-based interventions.

· Have excellent analytical and writing skills

· Previous evaluations, especially under EC funded projects will be an added advantage

1. WORK PLAN AND TIME SCHEDULE

Under the direct supervision of the Consortium Coordinator and Program Coordinator the consultant will undertake the following tasks;

#

Tasks

Duration (Man days)

Literature review/Desk Review

3

Preparation of data collection tools, translation into Somali and testing/revision of tools

2

Travel to and from Puntland

2

Enumerator training

3

Field data and information collection

7

Data and information analysis

6

Report writing and presentations

7

Total

30

This assignment will be carried out in a total of 30 days as per the breakdown contained in the table above.

1. COST ESTIMATE AND TIMEFRAME

The Consultants are requested to quote for this assignment and to mention when they will be available to carry out this assignment. Due to the urgency of the assignment, time will be of essence in the award of this contract.

1. RESPONSIBILITIES OF THE CLIENT – CARE SOMALIA (Lead Agency)

In support to the consultant to undertake the assignment, CARE Somalia will;

• Pay the entire consultancy fee (as per the contract) after the field work, submission, and acceptance of the report, outputs, and attachments. The consultant will be paid subject to the completion of the report in a satisfactory manner. This payment will also be taxed in accordance with the Somali Government laws, exact amounts will be specified in the contract
• Cater for transport and upkeep as well as other related logistics to and from the field.
• Facilitate access to relevant offices in the field through prior arrangements and consultations.
• Access to existing and useful resource materials while undertaking this assignment.
• CONSULTANT’S RESPONSIBILITIES

During the period of carrying out this assignment, the consultant will need to establish close coordination arrangements with the sub office in Puntland. In particular, the evaluator will be expected to work closely with the Consortium Coordinator, Program Coordinator, Project Managers of the consortium partners (SAVE the Children and CARE (lead agency) and the MoEHE. The consultant under this assignment will: –

• Be available for the assignment.
• Prepare all their travel documents.
• Commits to complete a fully satisfactory end product within the agreed time frame;
• Provides the necessary material for use during and after the study.

INTELLECTUAL PROPERTY RIGHTS

All documentation related to the assignment shall remain the sole and exclusive property of CARE and the consortium partners.**

[1] Puntland MOEHE EMIS Statistics Yearbook 2019-2020