Provision of Consultancy Service for Project Evaluation – RFP-SSD-JUB-2023-004 At Danish Refugee Council

Terms of Reference (TOR) for Consultancy Service for Project Evaluation – RFP-SSD-JUB-2023-004

Who is the Danish Refugee Council

Founded in 1956, the Danish Refugee Council (DRC) is a leading international NGO and one of the few with a specific expertise in forced displacement. Active in 40 countries with 9,000 employees and supported by 7,500 volunteers, DRC protects, advocates, and builds sustainable futures for refugees and other displacement affected people and communities. DRC works during displacement at all stages: In the acute crisis, in displacement, when settling and integrating in a new place, or upon return. DRC provides protection and life-saving humanitarian assistance; supports displaced persons in becoming self-reliant and included into hosting societies; and works with civil society and responsible authorities to promote protection of rights and peaceful coexistence.

Danish Refugee Council is a leading, international humanitarian displacement organization, supporting refugees and internally displaced persons during displacement, in exile, when settling and integrating in a new place or upon return. DRC provides protection and life-saving humanitarian assistance. DRC has been operational in South Sudan since 2005, implementing multi-sectoral responses including Protection, Camp Coordination and Camp Management (CCCM), Economic Recovery, Shelter and Non-Food Items (SNFI), Humanitarian Disarmament and Peacebuilding (HDP). DRC is implementing projects in Upper Nile, Unity, Jonglei, Western Bahr El Ghazal, Eastern Equatorial States and country wide through static and mobile response teams.

Purpose of the consultancy

The main purpose of the final program evaluation for DRC’s BHA-funded Multi-Sectoral Emergency Response South Sudan is to measure project effectiveness, efficiency, coverage, and relevance of activities. The evaluation will cover the accomplishment of all programme objectives as outlined in the programme document, theory of change, and the detailed implementation plan.

Background

In 2021 there was a deterioration in the humanitarian situation in South Sudan, which has seen the number of individuals in need of humanitarian assistance. South Sudan remains a fragile humanitarian context with continued reports of protection violations and sub-national violence occurring across the country. An estimated 8.9 million people require humanitarian assistance in South Sudan in 2022. Over two million people were internally displaced and an estimated 8.3 million people, including refugees, experienced acute food insecurity in the lean season (May-July 2022), with 13 counties experienced extreme levels of food insecurity in 2022, compared to six in 2021. Sub-national violence and flooding continued the main triggers for displacement. Communities across South Sudan continue to be affected by climatic shocks, with flooding displacing an estimated 220,382 people in Unity State between 2021-22 and 126,00 individuals affected by flooding in Upper Nile State. To support displacement-affected communities in South Sudan, DRC signed a modification of its ongoing BHA award, which would enable continuation and also expansion of its multi-sectoral emergency response program through static and mobile Protection, Humanitarian Coordination, Information Management and Assessments, and Shelter and Settlements interventions.

DRC proposed to continue to implement an integrated static and mobile humanitarian response that builds upon its year 1 BHA award while addressing new crisis through the sectors of Protection, Shelter and Settlements and Humanitarian Coordination, Information Management and Assessments. The activities built upon the activities implemented in Year 1 and lessons learned as well as the contextual changes have been considered in the design of the proposed activities for Year 2. The goal of the program is increase access to lifesaving basic services and information for displacement affected communities. The proposed Theory of Change (ToC) states that: if the safety and wellbeing of vulnerable individuals are protected and they are supported to recover from rights violence, if individuals have access to essential services, non-food items and dignified shelter, if community based structures and protection mechanisms are supported and strengthened through coordination and information sharing, and if the political, security and protective environment remains conducive, then the resilience and coping mechanisms of displacement affected communities to handle shocks and crises is strengthened, increasing their self-reliance.

Purpose of Program is the following:

1. To improve the protective environment in target locations in South Sudan and increase access to essential protection prevention and response services.
2. To ensure the inclusive delivery of assistance to vulnerable populations in camps and out-of camps settings through effective coordination and accountability mechanisms.
3. To provide basic and essential shelter and non-food items (NFI) to vulnerable households to reduce vulnerabilities.

To provide emergency livelihood inputs to households at risk of acute food insecurity

Objective of the consultancy

The specific objectives of this Final Project Evaluation will include the following:

1. To assess the achievements of the project in relation to the goal, objectives, results, and targets.
2. To examine the validity of the project’s theory of change.
3. To determine how well the resources were used in achieving the desired project results
4. To identify best practices, lessons learned and strength of the project implementation including project design or proposal development phase.
5. To identify key challenges and recommendations for future programming including evaluation management response plan.

Scope of work, evaluation questions and Methodology

The Consultant will be required to prepare a detailed methodology and work plan indicating how

the objectives of the project will be achieved, and the support required from DRC.

Scope of final evaluation

The evaluation will focus on assessing the achievements of BHA project from the start of the programme to date (September 2021 to April 2023) in targeted counties of Rubkona and Guit in Unity State and Malakal Fashoda, Baliet counties of Upper Nile State and accessible hard to reach areas (Upper Nile/Jonglei state) by the Mobile Response Team (MRT). The key stakeholders to be involved in the evaluation process will be target populations, community leaders, heads of chief, women representatives, youth representatives, local authorities, UN Agencies, cluster leads including Protection, CCCM, and Shelter/NFI clusters and other NGOs operating in the target locations.

Evaluation Questions

This section presents key final evaluation questions for each evaluation criteria. The key evaluation questions and criteria mentioned as follows.

1. Did the project intervention address the needs of the target population? Did the program increase access to i) lifesaving basic services and ii) information for displacement affected communities? Why or why not?”
2. Did the programme resources (human, financial and materials) lead to achievement of programme objectives, target, /results in a timely manner?
3. Has the intervention caused a significant change in the lives of the intended diverse beneficiaries including women, men, children, people with disabilities and/or communities depending on the intended programme objectives and nature of the different activities?) If yes, what changes have occurred in the target population that may be attributable to project activities? How could the impact/results be strengthened further? Did the intervention lead to any unintended positive or negative results?
4. How likely is it that the project’s results will be sustained after its completion? And have communities shown ownership of structures?
5. What are the main lessons learned for each of the project objective areas?

Evaluation Methodology

The final evaluation data collection will use both quantitative and qualitative methods of data collection. The data collection includes a household survey to measure values of outcome indicators. The evaluation will use multiple data collection methods that include desk review, key informant interviews (KIIs), focus group discussions (FGDs), case studies and observation of project activities. The desk review will include a detailed reading and assessment of key programme documents, including work plans, MEAL plan, budgets/expenditures, baseline report, annual and semi-annual performance reports. DRC will share these documents with the selected consultant to support preparation and submission of Inception report.

For the household survey, the same locations and respondents who attended on the baseline survey will be interviewed. However, in areas where the population has been displaced due to conflict or flooding, the quantitative end line survey design is flexible to expand its target areas beyond the baseline survey areas. The sampling frame will consider the entire direct beneficiary household that were assisted during the project implementation period.

KIIs will include key DRC staff members (Technical Coordinators, Project Managers, Head of Programme, Grant Team, Area Managers, Project Officers, M&E team), local authorities, youth representative, community leaders, women representative and other relevant stakeholders.

Once selected, the consultant/evaluator will also be required to provide additional details on their qualitative research methods. This will include

• Identifying specific categories of respondents for each question and a rationale must be provided as to why those respondents were selected
• Clear justification for selected data collection methods including criteria for selection of participants for FGDs and KIIs
• Outlining methods that will be used for qualitative data analysis (process tracing, outcome harvesting, etc.).

This consultant will detail the above information or approaches in the inception report, and submits to DRC for feedback and review

Desk Review

The DRC will provide to consulting firm relevant program documents including activity proposal; MEAL plan; baseline report, workplans, semi-annual and annual performance reports; Indicator Progress Tracking Table (ITT), post distribution monitoring reports, intention and satisfaction reports and; success stories and other data that may be relevant for the desk review.

Quantitative household survey

The household survey will collect data on the same outcome indicators that were assessed at the baseline. The quantitative data will be collected through beneficiary household interviews with representative sample size and within the same locations selected for baseline survey. The survey will be conducted using Kobo Toolbox, an open-source suite of tools that allows data collection using Android-tablet devises and data submission to an online server. The survey will assess the performance of the key outcome indicators using the activity participant households detailed in Table 2 above. The information will be collected from the actual activity participants who have been participating in one or more of the program activities during the implementation period. The survey will be conducted by an independent external consulting firm.

Sample households will be selected using a systematic random sampling technique based on the beneficiary database. A structured questionnaire will be used to collect quantitative data from the sample households.

Sample design

The evaluation will use multi-stage cluster sampling methods. The study population will comprise same counties as that of baseline survey (Rubkona and Guit counties in Unity State, and Kodok, Baliet and Fashoda counties in Upper Nile States). All the counties, sample Payams and Bomas where the activity has been operational will be considered for proper sampling. To ensure representation of samples allocated for each target County and clusters/enumeration areas, the sample size will be proportionally distributed using Probability Proportional to Size (PPS) technique. This is to ensure that participant households in the different counties have the same probability of selection despite their different beneficiary sizes. The last stage of the sampling will include the selection of households from the selected counties using simple random sampling technique based on the total number of activity participant households in that county.

The sampling frame will be developed based on the target participant of the selected Counties and the Payams. The sampling unit is the participant households, preferably the household head will be answering the questions. If the household head is not available, another member of the household who is 18 and above years old will be interviewed. The detail sample size estimation will be done by consultant based on number of beneficiaries at the targeted counties.

In addition, the selected consulting firm will be expected to include their sample size calculations and other key statistics such as effect size, margin of error, design effect, and non-response rate.

Deliverables

The evaluation team shall deliver the following deliverables in accordance with the technical approach and specific evaluation design within the agreed timeline:

1. Inception Report – outlining key scope of the work and evaluation criteria, intended evaluation questions, detailed methodology and data collection tools, data sources and analysis approach, sampling and sample size, roles and time frame, and suggested work-plan. The inception report will be submitted to DRC 5 days after signing the contract. This is to ensure that both the evaluator and DRC team have a common understanding of the evaluation.
2. Implementation plan – will include the detailed evaluation process, timeline with detailed calendar of key activities and milestones, supervisor and enumerator training manual, field guidance, sampling approach and methods, data collection tools, data quality assurance and data analysis plan showing how each question will be analysed from the data collected.
3. Draft Evaluation report: the evaluator should submit the draft report to DRC according to the recommended structure within the agreed timeline.

• Executive summary
• Brief background (as directly relevant to the report’s analysis and conclusions)
• Methodology (including challenges and data collection approach)
• Key findings and conclusions by sector/result (supported with relevant data, and data sources)
• Lessons and recommendations for implementation of future projects and for wider learning, including by sector/result. Recommendations should include details as to how they might be implemented)
• Annexes including list of all data collection tools, interview guides, work schedule, and any other technical details

The report will be reviewed to ensure that the evaluation meets the required quality criteria. Comments from relevant DRC programme staff will be provided within 5 working days after the reception of the Draft Report.

1. The final report: (30 to 40 pages): The consultant will be required to address all the comments provided by the DRC. The report should be finalized in 7 working days after receiving the revised first draft from DRC. The consulting firm should maintain the requirements for the content, format, depth and length, overall quality and approved timelines. The evaluation team will edit and format the final report as appropriate to ensure a high-quality deliverable. The consulting firm should deliver all files, including, quantitative data sets (raw and refined products, if applicable, transcripts of qualitative data and others in an easy-to-read format and organize them professionally.

Outline of the Evaluation Report

The consulting firm should follow the format given in the reporting requirement section (see below).

• Format and Size – The report must not exceed 50 pages (excluding all annexes) in MS Word and use the standard page set-up, margin, fonts, and line spacing.
• Cover Page, Table of Contents, List of Acronyms.
• Executive Summary – must be a clear, specific, and concise stand-alone document that states the most prominent findings, conclusions, and recommendations of the evaluation and gives the readers the essential contents of the evaluation in two to three pages.
• Introduction – must include programme background, evaluation objectives, purpose, synopsis of task and evaluation questions.
• Methodology and study design – must describe the methodology and sampling design,
• Constraints and limitations in carrying out the study.
• Tabular summary and graphical presentation of quantitative and qualitative evaluation results.
• Findings and Discussion – must present results from the evaluations and associated evidence. Results must be analysed and discussed, using findings from the quantitative and qualitative investigations in a complementary manner.
• Conclusions and Recommendations – must provide high-level conclusions about the impact of the program, and how it has positively contributed to increase access to essential protection prevention and response services of the beneficiary households and the target community. Targeted recommendations need to be provided by sector (protection, CCCM, and shelter and settlement sector), as well as broader programmatic and operational recommendations regarding emerging trends from data analysis. All conclusions must be based solidly on the presented findings.
• Programmatic Limitations and Challenges – must provide a list of key technical and/or administrative limitations.
• Final Evaluation Dataset (in excel or CSV)

Duration, timeline, and payment

The final evaluation work will start on 3rd April and end on 20 May 2023. The final report will be submitted to BHA on 30th June 2023.

Budget

The Consultant will provide a financial proposal for the provision of these services. The cost of travel and other expenses should be included in the overall quotation for the service. Agreed rates will be based on prevailing market competitive rates and value for money. The consultant is responsible for the following:

• The consultant will be responsible of payment of any tax or other fees related to this assignment.
• The consultant is responsible for any insurance including medical and travel related to this assignment.
• The consultant is responsible for his/her working tools such as computer, mobile phones, tablets for data collection etc.
• The payments will be in three instalments, 30% after submission of Inception report, and 30% after the draft report and 40% submission of final acceptable report to DRC.
• Please note that a partial payment hold-back will be in effect until a final report has been approved by DRC

Proposed Composition of Team

• Team Leader
• Programmer
• Data Analyst
• Quality Assurance

Eligibility, qualification, and experience required

• A post graduate qualification in Monitoring and Evaluation, Statistics or any other related field is required.
• A qualification in Social Sciences, Development studies or other related disciplines is also strongly required.
• At least 7 years of experience in carrying out baseline and end line surveys, and programme evaluations in South Sudan. Proven experience in conducting primary data collection including in fragile/developing context
• Strong understanding of the South Sudan context, the humanitarian system, protection risks, and conflict dynamics in South Sudan
• Willingness and ability to travel to remote field locations
• Strong skills in qualitative (drill down approach) and quantitative data analysis.
• Strong experience in questionnaire development, interview techniques and managing focus group discussions.
• Experience in using mobile data collection systems such as Kobo Collect will be an added advantage.
• Good communication skills, including good report writing in English
• Proof of financial capacity to carry out the assignment

Technical supervision

The selected consultant will work under the supervision of MEAL Manager.

DRC MEAL Manager will lead the overall coordination and facilitation of the final evaluation including enumerator selection and hiring, providing lists of program sites and numbers of direct beneficiaries for all activities, sharing relevant program documents, advising about local protocols, making logistical arrangements including accommodation and transportation for field data collection, connect the evaluation team with beneficiary and key informants, arrange meetings between the evaluation team and beneficiaries, community members, authorities, and community leaders.

DRC MEAL Manager will be the focal person and responsible for:

• Overall leadership, coordination, and facilitation of the final evaluation.
• Connect the evaluation team with key programme staff.
• Provide project documents including proposal narrative and relevant attachments.
• Annual and semi-annual performance reports, quarter reports and sources of performance indicators.
• Project work plans, MEAL plan and financial information
• Baseline report, PDM reports, other assessments.
• Monthly Indicator Performance Tracking Table report and tools.
• Provide domestic logistical arrangement for the core team including accommodation, transportation, and printing and office.

Location and support

The locations for final evaluation will include Rubkona and Guit in Unity state and Malakal Fashoda, Baliet counties of Upper Nile state

The Consultant will provide her/his own computer and mobile telephone

Travel

• DRC will facilitate all travels to the fields related to this Final Evaluation

Submission process

• International consultants shall submit via a secure tender email provided in the RFP documents
• Local/National consultants shall submit their proposals in a tender box located at the supply chain office

Evaluation of bids

• The RFP shall go under three stages of evaluation namely Administrative, Technical and Financial evaluation respectively.
• Only vendors that qualified at technical evaluation will be contacted for an interview with the panel to ensure their understanding of the consultancy services.

Please find complete bid documents in the following link: RFP-SSD-JUB-2023-004 – Consultancy Service for Project Evaluation