End Term Project Evaluation Consultancy: Emergency Food Aid for Malnourished School Children in Cueibet County, Lakes state, South Sudan At Vétérinaires Sans Frontières Germany

Introduction
This survey is commissioned by VSF Germany, South Sudan for the evaluation of the DKH funded school feeding project (K-SSD-2021-4027) and will cover the period from April 2021 to March 2023.
Vétérinaires Sans Frontières (VSF) is an international Non-Governmental Organization, providing humanitarian aid and development assistance to pastoralists and vulnerable communities in areas where livestock is of importance. VSF support is in animal health; livestock related agriculture, marketing, food safety, drought responses and mitigation, capacity development of communities and governmental institutions, peace and conflict resolution with the ultimate aim of food security and strengthened livelihoods of pastoralist communities.
This end term evaluation shall inform stakeholders on relevance, effectiveness, efficiency and sustainability towards meeting its overall objective. The information gathered will be important in understanding to what extent the project achieved or didn’t achieve expected results. It will serve as accountability assessment and report to DKH, community of Cueibet county, Lakes State, Humanitarian Stakeholders and Lakes State Government.

A. Background
1.1. Background of The Project
The project targeted the population of Cueibet county. Lakes is one of the 10 states of South Sudan and it is located in the Bahr el Ghazal region. The population in Cueibet county is estimated as 218,811.
The following outputs were achieved: (1) 184 MT of sorghum and 51.8 MT of groundnuts were procured from farmers in Cueibet, 15,000 of fortified cooking oil and 7 MT of iodized salt was procured from local suppliers in South Sudan and delivered to the project location for warehousing and distribution and (2) Provided one nutritious meal per day for 15,000 school going children in Cueibet during the implementation period.
1.2 The key result areas of the project and indicators for achievement:
a. During the period of acute food deficiency, 15,000 school children of 47 schools have received 1 meal every day
• 47 schools of Lakes State contribute to the immediate and long-term nutritional security and resilient livelihoods of the population.
1.2. Project Activities
• Procure and distribute HGSF rations to school feeding committees
• 15000 School children feed on a hot meal at school the on a daily basis
• Conduct ToT trainings on nutrition, innovative and resilient agro-pastoral production
• Develop and disseminate educational materials for schools on (e.g. sanitation, hygiene, school gardening, nutrition, agro-pastoral innovation)
• Provide extension service and inputs to schools and groups through the ToTs
• Establish model-gardens/units at schools
• Evaluation of HGSF– an end of project evaluation will be conducted one month after the last distribution to school stores/satellite warehouses.

B. Purpose
An overview of the purpose of the evaluation
The evaluation shall assess the projects’ relevance, effectiveness, efficiency, sustainability and impact. Furthermore, the evaluation will also focus on accountability to both beneficiaries and donor against intended results and learning.
Specific objective
• To evaluate the project against the project objective and indicators using the DAC criteria of relevance, effectiveness, efficiency, coherence impact, and sustainability
• To document lessons learned throughout the lifespan of the project to inform future program design in a similar context.
• To compare, the project baseline against end-line evaluation findings and to provide the recommendation for future programming
• To provide information on current Humanitarian situations in the project intervention area

C. Methodology
The evaluation team shall evaluate both the process of implementation’s outputs, and the outcomes of the intervention. In addition, an impact survey shall be assessed upon data availability for the EFAM project.
A process evaluation will assess the implementation of the school feeding activity by both PROMISE and EFAM projects in Cueibet through the analysis of indicators, review of programme documents, monitoring reports, case studies as well as interviews and FGDs with key informants among the different stakeholders. The FGDs will be organized among farmers, school staff, parents and government officials to clarify details of the implementation as well as to get a better understanding of the challenges faced by the programme and how corrective mechanisms were adopted (or not) and why. They shall also inquire about the perception of participants’ regarding their roles in the school feeding program, which may provide comparative data between both projects.
An outcome analysis will assess both projects performance as well as the perceived impacts or effects of the programmes among key stakeholders.
In order to measure the effectiveness, efficiency, and outcomes of both projects, the evaluation shall collect qualitative and quantitative data in both supported schools and non-supported ones. Collecting data from non-supported schools is crucial in order to construct a counterfactual, against which the outcomes of the programme can be compared. This approach will help to disentangle changes, which can be attributed to the projects, from changes that have occurred due to external factors.
The process evaluation will draw on both the analysis of quantitative indicators and on qualitative methods. While quantitative results provide progress as per logical framework indicators, the qualitative methods will complement and provide explanation to the quantitative results as well as assessing the evaluation questions.
The outcome evaluation shall be based on the analysis of quantitative indicators, as well as on qualitative methods.
The general rule related to acceptable margins of error (5%) in social research will apply to the study. Sampling criteria should be agreed upon with VSF SSD National office, a 30 x 30 cluster sampling is recommended, however, the Consultant may have the discretion to employ any other sampling methodology upon giving appropriate justification which will then be reviewed by VSF SSD programme team before being adopted.
Triangulation of data is important and comparisons with the baseline have to be presented.
Important: Data collected needs to be disaggregated into Gender, age, location, and any other relevant category to the issue at hand (e.g. duration of residency, HH status, etc.).
For HH surveys, the use of electronic data collection is highly recommended with preference of Kobo toolbox/ODK.
The process summary
• Desk review of secondary data (proposals, reports and HGSF previous studies)
• Interviews with regional teams, advisers and project officers and partner staff
• In-depth explorative Interviews with groups of beneficiaries (incl disaggregation by gender, and age)
• Focus group discussions with beneficiaries using structured questions (incl disaggregation by gender, and age)
• Visits to selected project sites, PRA related methods to enhance participation preferred (Gender sensitive)
• Interviews with other stakeholders (e.g. local authorities, donors, other NGOs, UN, non-beneficiaries)
• Sharing of initial findings and learning with regional team and partners in country

D. Evaluation Criteria
The DAC- OECD – criteria on humanitarian aid 2016 shall be used to carry out this evaluation:
The evaluation shall be based on the conception that evaluation is an assessment “to determine the relevance and fulfillment of objectives, developmental efficiency, effectiveness, impact and sustainability” of efforts supported by aid agencies.
Below are areas for examination by the evaluation:
Appropriateness/Relevance
The evaluation shall gauge extent to which the school feeding (K-SSD 2021-4027 -EFAM III) project is suited to the priorities and policies of the target group, recipient and donor. In evaluating the relevance , it is useful to consider the following questions
– To which extent were the needs of different stakeholders (women/ men, girls/ boys) met by the intervention?
– To what extend have recommendations and lesson learnt from previous evaluations been incorporated in the design?
– Was a thorough risk assessment carried out to identify risks to the project, including possible mitigation measures? How was this used to shape the project, what can be learnt from this?
– To what extent are the objectives of the program still valid?
– To what extend are the indicators of progress and impact in the design of good quality?
– Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives?
– Are the activities and outputs of the program consistent with the intended impacts and effects?
– Did the project address their problems of target group to some extent? Or has it exacerbated their condition?
– Is the intervention better in their opinion?
– Or could there be another best way of intervening that DKH and VSF did not consider?
– What could that best way be?
– Is the project aligned with government’s education and school feeding policies and strategies
– Does the project complement other donor-funded interventions such as PROMISE?

Effectiveness
This is to measure the extent to which EFAM III (K-SSD-2021-4027) project attained its objectives. In evaluating the effectiveness of the project, it is useful for the evaluator to consider the following questions:
– To what degree has (and has not) the project resulted in the expected results and outcomes?
– Have student literacy, attendance, retention, attentiveness, and student health improved? If yes, to what extent?
– Did assistance reach the right beneficiaries in the right quantity and quality at the right time?
– How many of the beneficiaries (schools and/or farmers) have received training as per project implementation plan?
– To what degree were targets in terms of schools and/or farmers met?
– Is the initiative in a position to regularly provide school meals to children in the beneficiary schools? Are such meals adequate, following any dietary guidelines and assuring food safety (handling and preparation, water availability, minimum infrastructure for school kitchen, hygiene practices and related trainings)?
– Does the community actively participate in the school-feeding programme activities?
– How has the HGSF approach influenced the food basket diversity of school meals in beneficiary schools in Cueibet?
– Is the intervention in a position to reach male and female smallholder farmers and provide them with stable markets for their products?
– Were there any significant barriers for local traders and smallholder farmers to participate in the programme? In particular:
o Was the payment mechanism effective? Was there any time delay in payments to beneficiary farmers? Why?
o Were the contractual mechanisms effective? Was there any legal barrier for the participation of targeted beneficiaries?

Coordination and stakeholders’ perspective
Coordination covers the extent to which the interventions of different actors are harmonized with each other, promote synergy, avoid gaps, duplication, and resource conflicts. Questions to be formulated are to be within below context:
– How well coordinated has school feeding been, across the region, with what consequences?
– How can we reduce gaps in school feeding support by the different agencies?

Efficiency
Under efficiency, the evaluation shall measure the outputs—qualitative and quantitative— in relation to the inputs. This will inform whether EFAM III used the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. When evaluating the efficiency, it is useful to consider the following questions while formulating questionnaire.
– Were activities cost-efficient? What were the costs of providing one meal per day under EFAM III project (per beneficiary)?
– Were objectives achieved on time?
– Was the EFAM III project implemented in the most efficient way compared to alternatives?
– How efficient has the project been in its approaches?
– To what extent has the project utilized the funding as per the agreed work plan to achieve its targets?
– To what extent was the M&E systems utilized by VSF to ensure effective and efficient project management?
– What was the total cost of the project per beneficiary?
– What were the external costs borne by the beneficiary?
– How have school meals contributed to the enrolment and regular participation of students?
– How do the prices paid under the food procurement differ from market prices?
– How does the intervention’s efficiency compare to other school feeding programs or social protection programmes?

Coverage
The evaluation will gauge to what extent did HGSF in Cueibet reach the most vulnerable children in schools. Questions to gauge this shall be designed in the following context:
– To what extent has HGSF benefitted the most vulnerable households?
– How can VSF ensure that marginalized individuals also have access to HGSF?

Coherence
Coherence shall question to what extent to which security, developmental, trade, and military policies as well as humanitarian policies, are consistent and take into account humanitarian and human rights considerations. Questions to be formulated should be within below category:
– To what extend is the intervention logically coherent and accurate?
– How coherent are agency policies on protection, and what are the implications?
– How could we advocate that other donors take human rights into account in funding decisions?

Impact
To gauge the impact, the evaluation shall interrogate the positive and negative changes produced by School Feeding, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The evaluation should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions. When evaluating the impact of the School Feeding project, it is useful to consider the following questions:
• To what degree has the project made progress toward the results in the project-level framework?
• Have there been any unintended outcomes, either positive or negative?
• To what extent have the outcomes been achieved? What were the major factors influencing their achievement or non-achievement?
• What are the outcomes on school participation, enrolment and educational performance? Are outcomes different for boys, girls, orphans etc.?
• What are the effects on dietary diversification of PROMISE /EFAM approach and how does school meals satisfaction and child nutrition compare in both projects?
• How has the programme improved the situation of the families of the participating school children? Have eating habits been changed at home?
• Have the EFAM III/PROMISE schools improved their fund management capacity?
• Is the intervention in a position to strengthen/empower local institutions and facilitate the capacity development of local leaders?
• To which extent has the project changed attitudes towards gender? Is the intervention in a position to empower girls/women?
• Is the perceived social inclusion different among individuals participating in each project?

Sustainability
This shall be measuring whether the benefits of EFAM III are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability , it is useful to consider the following questions:
– What progress has the state and national government made toward developing a nationally owned school feeding and what remains to be addressed?
– What were the major factors which influenced the achievement or non-achievement of sustainability of the program or project?
– Have farmers and local traders built capacity in a sustainable way to participate in institutional and non-institutional markets even with a reduced external support in terms of training and inputs?
– How are local communities involved in and contributing toward school feeding?
– Is the program sustainable in the following areas: strategy for sustainability; sound policy alignment; stable funding and budgeting; quality program design; institutional arrangements; local production and sourcing; partnership and coordination; community participation and ownership?
– Will schools continue buying from local smallholder farmers after the end of the initiative?
– Will the agricultural improvements related to PROMISE/EFAM be sustained, even after the end of project activities?
– How sustainable are the achievements and what recommendations could be made to improve sustainability Or to encourage productive investment?

E. Evaluation Team – Roles and Responsibilities
The expected roles and responsibilities of the evaluation team should be laid out here. This includes the number of expected team members and their specific contribution to the evaluation. Generally, this includes a Team Leader as well as any other evaluation team members who will conduct primary or secondary research.
Each member of the evaluation team shall have designated roles and responsibilities for the delivery of the evaluation.

Scope of Work and responsible parties
a) Evaluation tools are properly developed and in line with VSF SSD evaluation standards
• Review of the available relevant project documents i.e. project proposal, log frames.
• Coordinate HH Interviews for selected respondents during the evaluation study.
• Training of enumerators.
• Conduct interviews with selected partners, staff and other stakeholders
• Visit selected project sites and utilise engaging appraisal methods to ensure effective participation of women and men, girls and boys, elderly and other particularly vulnerable groups in the review process.
Responsible: Consultant

b) Coordination of the household interviews and well us providing guidance to the focus groups discussions
• Household interview and FGD participants are properly selected.
• Guide the FGD and interviewers on proper data entry and documentation/filing.
• Analyse key themes and topics, prepare the summary reports of each FGD as per agreed upon format, and analyse data, compare and triangulate across all FGDs and other tools utilised.
• Ensure completeness of the survey instruments questionnaires.
Responsible: Consultant

c) Apply effectively the quantitative and qualitative tools
• Obtain consent from the household or FGD members to conduct the evaluation.
• Record household data on questionnaire as per training instructions.
• Record all key notes related to issues arising while interviewing.
• Review and submit the survey data to survey team leader in accordance with the schedule

Responsible: Consultant, Project Manager

d) Provide overall coordination, logistics, and technical oversight in the Evaluation process.
• Brief stakeholders about the purpose of the evaluation.
• Brief stakeholders about the purpose of the evaluation.
• Provide technical oversight and monitoring of the Survey and ensure timely completion and compliance with international evaluation standards.
• Avail all project related secondary data.
• Avail all the required logistics including vehicles for the Evaluation.
• Assist in organizing meetings with stakeholders.
• Avail all the required logistics including vehicles for the Evaluation.
• Recruit and pay the evaluation enumerators.
• Supervise adequate utilization of the resources allocated for the study.
• Budget preparation and management during the evaluation period/ensuring that all expenses are properly documented.
• Budget preparation and management during the evaluation period/ensuring that all expenses are properly documented.
• Oversee accommodation and meals for data collection teams.
Responsible: Project Manager and Consultant

a) Evaluation team composition and required competencies
The evaluation team shall compose of male and female members with a comprehensive mix of competencies in agro-pastoral production and social research methodologies. These will be complemented with at least five years’ experience in related programming and programme research. Extensive experience in the specific fields in the Horn of Africa and South Sudan in particular will be required. Excellent communication skills and well as demonstrated writing and presentation skills are requisite. The team shall comprise of at least one male and one female member conducting relevant data collection methods throughout the entire field mission.

b) Management arrangements
The consultant should be informed of some issues, situation and conditions as they are or may arise during the exercise.
a. Travel: All international flights land in Juba, it is not possible to fly to Lakes on the same day. The consultant should take in to consideration this challenge that should not lead to cancellation of the exercise. VSF SSD will cover the cost of all internal flights and transport.
b. Accommodation: Consultants will be housed in hotels in Juba and whilst in Cueibet they will be housed at the organizations compound accommodation rooms. However, electricity for powering laptops is not guaranteed at all times. Internet access will be available at VSF SSD offices.
c. Data entry: VSF SSD will not supply data entry clerks or computers for data entry. Consultants are responsible for all data entry and management. All hard copies of tools will need to be transported by consultants to the place where data entry will be done. All data sets must be provided to VSF SSD in soft copy at the time of submission. They are the property of VSF SSD and the communities from which the data will be collected and may be used for future analysis.
Data will in most instances be collected from non-English speakers. However, tools will not be translated into the local language. A way around this matter will be developed in discussion with project staff, VSF SSD program staff and the consultant.
d. Operation arrangement
• Accommodation and transport will be provided by VSF SSD;
• Translators/enumerators, drivers, facilitators, office space, printing of questionnaires etc. will be provided by VSF SSD;
• The contact person in South Sudan will be the Country Programme Manager;
• The focal person in the field will be the Project Manager;
• Security advisory issues will be provided by VSF SSD;
• VSF SSD will take care of internal travels but in case of international flights, the consultant will organize and VSF SSD will pay reasonable prices incurred only.
e. Reference materials
Relevant documents will be availed for the consultant to support during the secondary information desk reviews. The consultant will be encouraged to identify any other sources for appropriate additional information that may be required to supplement what is provided by the project.
The Project Team will share the following documents with the Consultants for reference.

1. Full project proposal/concept note
2. Project agreement contract
3. Project log frame
4. Project activity reports
5. Sudan Population and Housing Census report 2008
6. FAO/WFP food security assessment/survey reports
7. Humanitarian Response Plan 2019, 2020, 2021,2022
8. South Sudan Infrastructure Action Plan, Chapter 6
9. Investing in Agriculture for Food Security & Economic Transformation – UNDP

c) Budget and payment
The evaluation budget will take care of the expenses such as the consultant’s charges, international travel, in country travel, accommodation full board, and fuel for vehicles to be used during the survey period, enumerators’ fees and lunches, stationery and communication costs. VSF will take care of the following
a) Flights (internal and international for the consultant)
b) Accommodation in South Sudan
c) Field transport (fuel and car rentals)
d) Enumerators fees
e) Stationeries
The following costs will not be covered by the organization and should be factored into the consultancy and related fee which the consultant will submit with the application.
• Costs for data handling, entry and processing
• Communication cost
• Report writing and printing
• Any medical expenses by the consultant during the assignment

F. Timeframe
Month – May
Tasks
Secondary Data/Information Review – 1 day
Data collection tools development – 1 day
Share Data Collection tools with VSF programs – 1 day
Enumerators selection and Training -1 day
Pretesting, refining and printing/deploying of evaluation tools – 1 day
Data collection and interviews – 5 days
Data Analysis and validation – 3 days
Draft Report of Evaluation – 5 days
Final Report with Feed back – 3 days
TOTAL DAYS – 21 days

G. Outputs and audiences

Stakeholders and Interest in the evaluation and likely uses of evaluation report to this stakeholder

VSF South Sudan Country Office: Responsible for the country level planning and operations implementation related to food procurement, food delivery and school feeding. It has a direct stake in the evaluation and interest in learning from experience to inform decision-making and advocacy with the government for adequate investment in HGSF. It is also called upon to account internally as well as to its beneficiaries and partners for performance and results of its operation.
VSF Regional Office Nairobi (RON): Responsible for both oversight of country office and technical guidance and support to VSF South Sudan activities in general, VSF RON management has interest in an independent/impartial account of the operational performance as well as in learning from the evaluation findings to apply this learning to other country offices.
Diakonie Katastrophenhilfe (DKH): DKH has an interest in the lessons that emerge from surveys and evaluation, particularly as they relate to DKH strategies, policies, thematic areas, or delivery modality with wider relevance to DKH programming. The South Sudan Head of mission at Juba is a particularly important stakeholder.
Beneficiaries: As the ultimate recipients of food assistance and productive and agricultural outputs marketing support, beneficiaries have a stake in determining whether assistance provided is appropriate and effective. As such, the level of participation in the evaluation of women, men, boys and girls from different groups will be determined and their respective perspectives will be sought. The beneficiary groups targeted shall include learners, community members, Parent Teacher Association (PTAs), school committees, small holder farmers, etc.
Government (National and state): The Government has a direct interest in knowing whether the project activities in the country/state are aligned with its priorities, harmonized with the action of other partners and meet the expected results. Issues related to capacity development, handover and sustainability will be of particular interest. For the school feeding program, key government ministries include Ministry of Education, Ministry of Agriculture and RRC

Outputs from Evaluator

1. The evaluation report should be no more than 20 pages (excluding annexes) and include:
   • Inception Report with clear evaluation logic in reference to questions
   • Executive summary
   • Key recommendations
   • Introduction – scope, purpose, methodology
   • Presentation of main findings
   • Detailed recommendations for VSF SSD.
2. The case study can also be provided