Evaluation of Africa Educational Trust’s Education on Air II Programme At Africa Educational Trust

Africa Educational Trust (AET) is seeking bids from experienced consultants to deliver a high-quality evaluation of our Education on Air II programme in South Sudan implemented in partnership with UNICEF and the Ministry of General Education and Instruction.

Background:

Education on Air II is a 15 month project (August 2021 – October 2022) that seeks to build on some of the successes made with ‘Education on Air’ during the nationwide closure of schools with a specific emphasis on working with the MoGEI to improve the quality of the lessons, making them more learner-centered and interactive; strengthening the management of the programme; and introducing a robust monitoring and evaluation system. Although schools have now re-opened the Education on Air programme has been an important supplement to ongoing schooling as it follows the same syllabus; it will enable learning to continue if schools are forced to close again and it gives children that are out of school the opportunity to learn. Lessons have been recorded resulting in a library of radio lessons that can be utilized for years to come.

Purpose of the Final Evaluation:

The purpose of the Final Evaluation is to assess the scope, reach, accessibility, and effectiveness of Education on Air II for promoting education continuity for primary learners in South Sudan.

Full details can be found in the attached Terms of Reference.

Required skills and competencies of the evaluator:

• Strong background in primary education and distance learning
• Strong background in utilising qualitative and quantitative research methods
• Experience in working with vulnerable groups
• Experience in conducting evaluations or research in Africa

TERMS OF REFERENCE FOR THE EDUCATION ON AIR II PROJECT

Between 26th March 2020 and 3rd May 2021, the Government of the Republic of South Sudan implemented a country-wide closure of all education and learning institutions in an effort to control the spread of Covid-19. In an effort to keep children engaged and learning during the school closures the Ministry of General Education and Instruction (MoGEI), in partnership with UNICEF, delivered radio lessons, ‘Education on Air’ on South Sudan Broadcasting Association (SSBC) and Radio Miraya. The lessons were developed by teachers and delivered and recorded live, enabling students to call in if they had questions. The Education on Air programme successfully broadcasted 5 lessons a week, covering primary level English, Maths and Science. Despite this significant achievement, reporting from the MoGEI and UNICEF identified some fundamental challenges, most notably:

• Quality issues in radio scripts, presenting skills, and lesson delivery.
• High turnover and absenteeism of teachers’ delivery of the programme due to lack of payment and delays in payment
• Lack of monitoring system or tools to measure quality and reach of the programme.

This project seeks to build on some of the successes made with ‘Education on Air’ during the nationwide closure of schools with a specific emphasis on working with the MoGEI to improve the quality of the lessons, making them more learner-centered and interactive; strengthening the management of the programme; and introducing a robust monitoring and evaluation system. Although schools have now re-opened the Education on Air programme has been an important supplement to ongoing schooling as it follows the same syllabus; it will enable learning to continue if schools are forced to close again and it has given children that are out of school the opportunity to learn. Lessons have been recorded resulting in a library of radio lessons that can be utilized for years to come.

Implementation method:

Radio lesson development: This includes reviewing the Education on Air programme; adapting the scope and sequence of the radio lessons for each subject; revising lesson templates; recruiting and training teachers as scriptwriters; recruiting and training teachers as radio presenters; developing scripts; reviewing scripts; broadcasting and recording lessons; uploading lessons onto SD cards and developing and broadcasting an awareness-raising campaign. The process of developing scripts and recording lessons will be a continuous and iterative process, learning and adapting to feedback on the lessons.

Roll-out of Radio lessons: This includes contracting radio broadcasters (national and state level); sharing content with radio broadcasters; procuring and distributing MP3 radios; establishing agreements with phone network providers for toll-free calls; broadcasting awareness campaign; broadcasting lessons on a weekly basis; homework assignments;

Monitoring, Evaluation, and Learning (MEL): This includes a review of data from the Education on Air programme; developing MEL framework and tools; baseline data collection; end line data collection; listener surveys; feedback mechanisms for teachers and students with learning feeding into adaptive programming.

Purpose of the Final Evaluation

The purpose of the Final Evaluation is to assess the scope, reach, accessibility, and effectiveness of Education on Air II for promoting education continuity for primary learners in South Sudan.

Scope of work

At the project level

The objectives are to:

• Assess whether the project activities generated the planned outputs and whether they delivered on time?
• Assess the achievement of project outputs and outcomes and the overall impact on the lives of beneficiaries and local communities
• Assess the accessibility of radio-based distance learning instruction for primary learners
• Explore equity in access to radio instruction for boys, girls, children with disabilities, and other marginalised children
• Evaluate the suitability of radio as medium of instruction to deliver Maths, English, and Science lesson content to primary learners
• Identify the processes by which learners engage in radio lessons
• Assess the capacity of the Ministry of General Education and Instruction to implement and monitor the Education on Air programme
• Review significant achievements the project has brought about. What worked particularly well and why and which less well and why? Any unexpected or unintended outcomes – both positive and negative?
• Gather feedback about the project from people affected by the activities and provide an opportunity for them to participate in analyzing project achievements
• Assess the extent to which the benefits received by the target group are having a wider overall effect on larger numbers of people in the communities or regions as a whole.
• Assess whether the positive outcomes of the project are likely to continue after the end of the project.
• Make recommendations for future interventions and projects. What additional support is needed for the project to be handed over to the MoGEI?

At organization level

• Assess the extent to which the project’s structure has enabled it to meet its objectives and the needs of the target groups
• Assess the effectiveness of the monitoring, evaluation, and learning system and tools used to measure success

On partnerships

• Assess the level of effectiveness of partnerships at all levels of the project e.g. UNICEF head office, UNICEF State offices, MoGEI at national and state levels, and local communities.

It is important that the Final Evaluation is a transparent examination of activities against the planned results so that lessons for future projects can be learned. It is recognised that there will be some unexpected outcomes, both positive and negative.

The Final Evaluation Process

The Final Evaluation will take place during the final months of the project between September 2022 and October 2022 as agreed between AET and the evaluator, with the field visit preferably taking place in early October 2022 while lessons are still taking place. This is likely to include 3 days of prep, 12 days of field visit (including travel days), and 5 days of report writing with an estimated delivery time for the final report being the end of November 2022. This will be agreed upon later between the Project Manager and consultant.

The evaluation will be conducted by a regional consultant in conjunction with AET staff. The consultant will undertake a visit to South Sudan to conduct fieldwork in Juba and at least two other locations.

The evaluation process should include the following activities:

• A desk review of project information provided by AET
• Interviews with staff from AET to collect information on achievements and impact and difficulties faced during implementation, including management, monitoring, and reporting
• Focus Group Discussions and Interviews with key project stakeholders and beneficiaries in the field.
• Interviews with officials from the Ministry of Education.
• Training of enumerators for state-level data collections
• Collation and analysis of data collected during evaluation and during the project
• The consultant will present a preliminary overview of their findings to AET before preparing their report by end of 31st October 2022.
• The consultant should submit the draft report to AET for written comments before finalising the report, to minimise the chance of inaccuracies and to maximise ownership of the findings

The following documents should beexamined as part of the Final Evaluation:

• The approved project proposal documents
• The baseline report
• The quarterly reports
• Data collected
• Any other relevant documentation, as requested by the consultants

The following stakeholders and beneficiaries should be consulted:

• AET staff, including AET Education Advisor
• MoE staff at national and state levels, including the MoGEI focal points
• UNICEF focal point and state-level education officers
• Parents, guardians, and caregivers of children
• Children (both in-school and out of school)
• Radio station staff

The scope of this evaluation is limited. It is not feasible for it to be an exhaustive review covering all of the work undertaken. The project target beneficiaries are high and the evaluator will need to consider sampling techniques.

Outputs of the Final Evaluation

The Final Evaluation should result in a report of no more than 25 pages for the main body, including an Executive Summary. The report should address the areas listed in this TOR. The report should include a good level of critical analysis and the report’s authors should support their analysis of a project’s achievements with relevant quantitative and qualitative data and state how this has been sourced. Recommendations should also include details as to how they might be implemented

Roles and Responsibilities

The roles and responsibilities for the Final Evaluation are set out below:

1. Evaluator:
   • Undertaking a desk review of key project documents
   • Designing the interviews and focus group questions/ approach
   • Reviewing and adapting (if necessary) the data collection survey
   • Training enumerators on data collection and sampling for state-level data collection.
   • Collation and analysis of data collected during endline and throughout the project
   • Presenting the preliminary findings to AET
   • Producing a draft report and a final version in response to comments from AET
2. Africa Educational Trust
   • Co-ordinating the Final Evaluation and agreeing on the budget
   • Commissioning and supervising the evaluator
   • Liaising with the in-country team on the logistics and schedule for the field visit
   • Taking a final decision on the interpretation of these Terms of Reference, should any dispute or confusion arise
   • Agreeing on the format for the evaluation report
   • Providing feedback on the preliminary findings and draft report, and agreeing on the final report
   • Submitting the final evaluation report to the donor and responding to any feedback or queries from the donor
   • Providing logistical support including vehicle, driver, fuel, airtime, and accommodation in the field[1]
   • Working with the consultant to identify a suitable sample of stakeholders and beneficiaries and set up the focus groups and interviews
   • Scheduling meetings
   • Providing translation support for focus groups and interviews with beneficiaries (as needed)

[1] Accommodation costs will be covered by the consultants’ DSA but AET should book the rooms.