Terms of Reference for Endline Evaluation of the Changes 2.0 Project At Save the Children

Millions of women and girls around the globe live and encounter daily the negative impact of harmful social norms and discriminatory practices. Social norms and discriminatory practices dictate cultural beliefs, attitudes, and practices, which collectively determine how a society treats and rewards men and women or boys and girls. In Africa, where the most severe norms are practiced, defying existing norms and practices attracts dire consequences, while observing them is applauded and rewarded.

Generally, African societies value boys and men more than girls and women, respectively. Close attention is paid to the behaviour and attitude of boys and girls from as young as 3 years, where observed signs of deviant behaviours and attitudes are punished. Social exposure prepares boys as men and girls as women and introduces a wide range of deeply rooted inequalities between boys and girls, and in later stages, between men and women. For instance, girls are expected to undergo harmful practices in preparation for their status in society; higher attention is given to the views and opinion of boys and men than girls and women; and better access to socio-economic opportunities are accorded to men and boys than women and girls.

Somalia is one of the African countries with some of the most severe norms and practices, and as such the country is listed in the top five on the gender inequality index[1] (0.777 out of 1). The index looks at three dimensions: reproductive health, empowerment, and labour market participation, with Somalia scoring poorly on all three dimensions. In Somalia, gender inequality exists in economic, social, and political domains with a wide access gap between men and women in economic and social fields. For instance, male leaders are preferred and seen as better decision-makers than female leaders. This widespread preference for male political leaders over female political leaders has hindered the realization of the 30% reserve quota. According to the CHANGES 2.0 project annual reports[2], despite the increased number of women aspirants in the 2022 parliamentary elections, women managed to secure just 20.6% of the 315 seats in both houses.

Discriminatory practices are still common in Somalia and disproportionately affect girls and women more than boys and men. Existing norms expect women and girls to do most of the household chores such as cooking, washing utensils and clothes, or making beds, and give boys and men more free time to interact and socialize with their peers or concentrate on their personal goals and aspirations. According to CHOICES 2.0 project pre-post survey, girls spend about 4.8 hours on household activities compared to about 3 hours for boys.

Gender disparity in education and gender stereotypes are also common, negatively impacting girls and women. Most Somali girls, especially those from rural communities, hardly finish their primary or secondary education due to limited educational opportunities, household poverty and gender stereotypes that discourage girls from pursuing their dreams or limit their chances of achieving their goals. Adolescent boys also believe that they deserve more educational opportunities than girls or that they can outperform girls.

Female genital mutilation (FGM), classified into Pharonic (most severe) and Sunnah types, is the most common and widely supported discriminatory practice in Somalia. FGM has been challenging interventions due to its superior adaptive capacity. Whereas support for and prevalence of Pharonic FGM has declined over the years, support for and prevalence of Sunnah FGM has increased. Social and behaviour change experts have linked this unfortunate outcome to flaws in advocacy and campaign design which treated FGM and Pharonic FGM as the same, and focused on the negative consequences of Pharonic FGM to discourage its practice. This has indirectly validated Sunnah FGM and has motivated the medicalization of FGM.

Child Early and Forced Marriage (CEFM) is another common discriminatory practice in Somalia. However, unlike FGM, the support for CEFM has been declining in past decades. An evaluation by the previous CHANGES 1.0 project in 2021 observed a significant reduction in CEFM prevalence. In addition, community perceptions on the ideal age of marriage for girls has also improved from 18 to 20 years in 2017 to 20-25 years in 2021[3].

CHANGES 2.0 Overview

CHANGES 2.0 is a four-year project (2021-2024) that is co-funded by Global Affairs Canada (GAC) and the Royal Norwegian Embassy (RNE), and implemented by the CHANGES consortium, comprised of Save the Children International, CARE International, and IRC. CHANGES 2.0 is a strategic intervention designed to challenge harmful social norms – FGM, CEFM, intimate partner violence – through interconnected community and national level interventions. At the community level, the project targets adolescent boys and girls and their caregivers, community influencers composed of religious and traditional leaders, and duty bearers. At the national level, the project targets Women Rights Organizations (WRO), media associations, and government officials composed of law enforcement officers, social security departments, and members of parliament and other government institutions engaged in law and policies.

The project is carried out in 16 targeted districts and 76 communities (34 SCI, 24 CARE, 16 IRC) from Somaliland, Puntland, Galmudug, HirShabelle, Banadir and Jubaland.

Save the Children (SCI) implements the project in Somaliland (Hargeysa, Burao) in collaboration with NAGAAD Network; in Puntland (Galkacyo North, Galdogob) in collaboration with Asha Gelle Foundation; in Galmudug (Adado, Abudwak), HirShabelle (Beledweyn) and Southwest State (Diinsor) with Somali Peace Line (SPL)
CARE International implements the project in Somaliland (Badhan, Erigabo, Buhodle) in collaboration with NAGAAD Network and Tadamun Social Society (TASS); in Jubaland (Kismayo) in collaboration with Somali Women Study Center (SWSC)
IRC implements the project in Galmudug (Galkacyo South, Hobyo) in collaboration with Daryel Bulsho Guud (DBG); in Banadir (Karan, Howl-wadag) with Save Somali Women and Children (SSWC)

The CHANGES 2.0 project aims at:

Challenging harmful social norms including FGM and CEFM, and
Increasing women’s social and economic empowerment.

The CHANGES 2.0 project is composed of four outcome chains designed to bring about the required behaviour, attitude, and perception changes, and improve skills and uplift the economic status of women. The project aims at decreasing discriminatory social norms and practices that perpetuate and validate sexual and gender-based violence against women and girls (Outcome 1100). To achieve this, the project trained adolescent boys and girls (10-19 years) and their caregivers to improve their attitude towards positive social norms. Specifically, adolescents were trained on CHOICES (+) curricula, while caregivers were socialized on gender equality and positive social norms using the SC VOICES videos. The project sensitized religious and traditional leaders on key messages and awareness raising activities to improve their commitment towards positive social norms. Community influencers developed and implemented action plans to prevent violence against women and girls in their communities.

Secondly, the project trained adolescent boys and girls on adolescent sexual and reproductive health and rights (ASRHR) and life skills to improve their voice and agency to make and act on life decisions for themselves, including with regards to ASRH and SGBV (Outcome 1200). Under this outcome, the project used the Girl Shine curriculum with adolescent girls (10-19 years) and the Boys Wise curriculum (10-19 years). The project also engaged Women Rights Organizations (WRO) to support adolescent girls and boys to lead advocacy activities on gender equality and rights in their respective communities.

Thirdly, the project aimed to enhance the enabling environment in support of gender equality and women’s and girls’ rights, specifically working with WROs, media associations, and duty bearers at the Federal and regional government levels (Outcome 1300). Under this outcome, the consortium strengthened the capacity of WROs to influence and hold government officials accountable to their mandate and promises on gender equality, in addition to sensitizing law enforcement officers and gender desk staff officers on existing and upcoming gender equality laws and policies. Furthermore, the project trained media associations on existing and upcoming gender equality laws to increase their capacity and willingness to inform the public and hold government officials accountable on their mandate to protect the rights of women and girls through the adoption and implementation of gender equality laws at the federal and regional levels.

Finally, the project aimed to enhance the social and economic empowerment of women to improve their participation and involvement in household and community decision-making (Outcome 1400). The project trained women in setting up and managing Village and Savings Loan Association (VSLA) groups, and on business skills, savings practices, and accessing credit services. The project distributed tools and equipment to women to help them start and run their own businesses. The project also facilitated gender discussion sessions for VSLA women and their male household members to improve male attitude towards women’s decision-making power in the household.

Following the successful completion of the project, the CHANGES consortium is commissioning an endline evaluation to establish the outcome and impact of the project. Specifically, the endline evaluation is meant to evaluate progress made against the ultimate, intermediate and immediate outcome indicators, and consequently generate evidence to inform strong and strategic interventions towards the elimination of harmful norms and practices in Somalia. In addition, the evaluation is expected to shed light on contextual limitations or challenges hindering the full realization of a gender inclusive Somali society.

The overarching objective of the endline evaluation is to evaluate the performance of the ultimate, intermediate and immediate outcome indicators against the milestone targets and the baseline as specified in the PMF.

Performance Measurement Review Indicators:

The Endline Evaluation will cover the following project indicators:

Ultimate Outcome Indicators:

% of girls who have ever been married before their 18th birthday
% of girls who have undergone any form of female genital mutilation or cutting (FGM)
% of VYA and OA girls (and boys) who report they are empowered according to the Power Index

Intermediate Outcome Indicators:

% of VYA and OA girls (and boys) who report existence of discriminatory social norms and practices
% of VYA and OA girls (and boys) who consider a husband justified in hitting or beating his wife
% of caregivers who have taken at least one action against child marriage in the past 6 months
% of VYA and OA girls (and boys) who report decision-making power with regard to marriage
% of VYA and OA girls (and boys) who report decision-making power with regard to ASRH rights
% of men and women who are not in favour of Pharonic / Sunnah FGM/C
No. of actions – e.g. national/state action plans, policy statements, public declarations, enforcement practices, budget allocations – taken by the government at the federal and state levels to implement relevant GE laws related to FGM/C and CEFM
Level of commitment by federal and regional governments to adopt and/or implement relevant GE laws
% of women and men, girls and boys who feel that the government has made progress in adopting and/or implementing relevant GE laws that support women and girls’ rights
% of women who have started or sustained an income generating activity after receiving project-related training and inputs
% of women who decide how their income earned from IGA will be used
% of women who report decision-making power with regard to household and community decisions

Immediate Outcome Indicators:

% of VYA and OA girls and boys who hold positive attitudes towards gender equality and equal rights for girls and boys (gender construct, access to education, FGM and bodily autonomy, age of marriage)
% of VYA and OA girls and boys who hold positive attitudes towards equal roles and responsibilities of women and men (household decision-making power, division of household chores, income generating work, women roles, leadership positions)
% of caregivers and community members who hold positive attitudes towards gender equality and equal rights for girls and boys (gender construct, access to education, FGM and bodily autonomy, age of marriage)
% of caregivers who hold positive attitudes towards equal roles and responsibilities of women and men (household decision-making power, division of household chores, income generating work, women roles, leadership positions)
% of VYA and OA girls and boys who demonstrate knowledge of ASRH rights, SGBV and life skills
% of men and women who hold positive attitudes towards women’s decision-making power in the household

Evaluation Criteria

The following key questions will guide the endline evaluation’s assessment of the project against the DAC Criteria for Evaluating Development Assistance. The consultant will be asked to review and update the existing data collection tools (from the baseline) to incorporate the questions needed to answer the following DAC criteria questions:

Effectiveness: The extent to which the project attained its outcomes.
To what extent were the outcomes achieved? To what extent were the indicator targets achieved?
What were the major factors influencing the achievement or non-achievement of the outcomes?
- How effective were the consortium structures and approaches in the systematic coordination and engagement with external stakeholders and actors.
Efficiency: The extent to which the project used the least costly resources possible in order to achieve desired results, considering inputs in relation to outputs.
- Assess and establish the efficiency and specific and combined contribution of intervention models (CHOICES (+), Girl Shine/Boys Wise, SC VOICES videos, and EASE) in changing the behaviour, attitude, and knowledge of the respective target groups.
- What, if any, challenges in project implementation were encountered, including managerial, organizational and any other unforeseen factors?
- To what extent did the project collaborate with national and sub-national partners and stakeholders (technical, advocacy, funding, etc.) to achieve results in an efficient manner?
Impact: The positive and negative changes produced by the project, directly or indirectly, intended or unintended.
- What has happened as a result of the project – either intended or unintended, positive or negative?
Relevance: The extent to which the project was suited to the priorities of the target beneficiary group(s), stakeholders, and to the donor.
- Was the project relevant to the needs of the beneficiaries, as identified at the project inception/design stage?
Sustainability: The extent to which the benefits of the project (outputs, outcomes) are likely to continue after project completion.
- What is the likelihood of the continuation and sustainability of the project outcomes and benefits after completion of the project?
- What commitments (financial, human resources, etc.) have been made by stakeholders to maintain or improve results?
- How will improvements in stakeholder knowledge, attitudes, capacities, etc. contribute to maintaining results?
- To what extent is support available from the external environment to maintain or improve results?

Study Design and Sampling Framework

The endline evaluation is using a longitudinal study design to single out significant changes in behaviour, attitude and knowledge amongst adolescent boys and girls, their caregivers, and community leaders, and the ensuing ultimate impact of the project on the prevalence of FGM and CEFM among adolescent girls, the empowerment of adolescent girls and boys according to the Power Index[1], and the support for FGM among female and male caregivers and community leaders. Endline respondents will match those respondents surveyed/interviewed at the baseline, wherever possible. The endline is also utilizing a comparison/control group as a further measure of project progress. The consultant will be asked to analyze progress against project targets, as well as between the the implementation and control groups.

The endline evaluation will adopt a mixed methods approach. The quantitative method will use a sample of 5,300 individuals divided equally between 50 intervention communities and their 50 comparison/control communities. The sample covers four strata: 1600 Very Young Adolescents (800 girls and 800 boys), 1600 Older Adolescents (800 girls, 800 boys) and 1600 caregivers (800 women, 800 men) and 500 community leaders (250 women, 250 men) stratified across the intervention and comparison groups. A caregiver and community leader survey will used to collect data from adult men and women, while an adolescent survey will be used to gather relevant data from the adolescent boys and girls.

A tracking list containing relevant contact information of the caregivers and community leaders will be used to track respondents at the endline. Adolescent boys and girls will be traced through their caregivers and unavailable respondents will be replaced. For the comparison communities, replacement matching the characteristics of the respondents (age, sex, education level and economic status) will be randomly selected from the community members. In the intervention communities, replacements will be randomly selected from the project beneficiaries e.g. a replacement for an OA boy will be selected only from the OA boy beneficiaries. Table 1 of Quantitative Sample Size located here.

For the qualitative method, the endline evaluation will utilize key informant interviews (KII) with government officials, members of parliament, and women rights activists; and focus group discussions (FGD) with beneficiary groups to collect additional information to complement the quantitative methods and assess presence and extent of community level conversations between direct and indirect beneficiaries.

Scope of Work

The consortium will hire two external consultants herein referred to as qualitative consultant and evaluation consultant to undertake the endline evaluation.

The qualitative consultant will be required to collect and analyze the qualitative data (KIIs and FGDs) and produce a qualitative report which will be shared with the evaluation consultant**. A separate TOR will be posted for this consultant.**
The evaluation consultant (the current TOR) will be responsible for analyzing the quantitative data, and utilizing the findings from the qualitative report to produce the Final Endline EvaluationReport for the CHANGES 2.0 project.

The implementing partners (SCI, CARE, and IRC) will be responsible for the quantitative data collection (adolescent, caregiver, community leader surveys) in their respective locations with the Consortium MEAL manager overseeing the overall data collection exercise. The MEAL focal points from the implementing partners will be responsible for training the enumerators, supervising data collection and conducting data quality checks. Quantitative data will be collected using Kobo and the data collection account will be under the custody of the Consortium MEAL manager, with access limited to the MEAL team and the evaluation consultant.

At the consortium level, the evaluation will kick off with a three-day Training of Trainers (ToT) workshop for the 10 MEAL staff, consisting of the Consortium MEAL manager, the MEAL focal persons of the implementing partners and the local partners. In the workshop, the team will,

Go through the evaluation study design and methodology in addition to reviewing quantitative and qualitative collection tools (.docx and. xlsx versions) and data quality controls,
Develop field plans encompassing enumerator training, field supervision and safety and security risk mapping of the field team,
Review, appropriate and finalize the quantitative data collection budget
Review and update the caregiver and community leader, and adolescent survey respondent tracing lists, and
Update the intervention and comparison communities as per the intervention.

After the workshop, each implementing partner will hire and train their enumerators in their locations and will take lead in managing logistics and ensuring data quality of their respective working districts under the project. The implementing partners will cover the field costs in their locations. The implementing partners will hire an equal representation of male AND female enumerators, to ensure that women and adolescent girls are interviewed by female enumerators, and men and boys are interviewed by male enumerators. SCI will be responsible for the overall supervision of the field data collection.

Distribution of Roles and Responsibilities

The CHANGES consortium will:

provide the evaluation consultant with relevant and requested documents including project proposal, PIP, Annual Reports, Work Plans, PMF, LM, Baseline Study, Midterm Review
review and approve inception report and survey tools,
translate survey tools into Somali
provide any other support needed by evaluation consultant
provide the evaluation consultant with
- clean data sets in Excel,
- an indicator analysis plan documenting how each indicator is to be measured; and
- the qualitative report in English, containing an analysis of the FGDs and KIIs
review draft report and provide constructive feedback to the evaluation consultant
approve and sign-off on the Final Endline Evaluation

The evaluation consultant will be required to:

review project documents, including but not limited to: Project Implementation Plan, Logic Model, Performance Measurement Framework, Baseline Study, Annual and bi-annual Reports, Work Plans, annual budgets and expenditures, data collection tools, etc.
review and update the existing data collection tools to incorporate the questions needed to answer the DAC evaluation questions.
produce an inception report demonstrating his/her technical capacity to handle the evaluation by explicitly explaining his/her evaluation methodology and design in line with the details on these already provided above. The inception report should exhaustively cover the proposed design and methodology, analysis plan, suggested final report format among others.
prepare and submit final draft report following the CHANGES 2.0 Baseline report and Midterm report formats. The consultant should work closely with the Consortium MEAL Manager and respond to the feedback in good time.
produce four thematic factsheets (max 2 pages) aligned with the project impact indicators (FGM, CEFM and Women Economic and Social Empowerment and Adolescent Sexual Reproductive Health and Rights) summarizing the major approaches and findings of the project.

Qualification and Experience

Masters in Gender Studies, Development Studies, Public Health, or Social Science with demonstrable experience in GBV and CP Humanitarian projects or gender equality. Alternatively, master’s degree in development studies, social sciences with statistics and economics background will also be accepted.
Demonstrable experience in working with children preferably in conflict prone countries such as Somalia.
Advanced data analysis skills in regression and econometric analysis using STATA, R or SPSS,
Considerable record of accomplishment and proven experience in quantitative and qualitative methods of data analysis and data management.
Profound experience and technical knowledge in gender equality, adolescent sexual reproductive health and rights, and social and economic and social empowerment of women
Previous experience in Somalia is an added advantage.

Duration of Consultancy

The evaluation consultant is expected to initiate data analysis upon receipt of data no later than October 2024. The CHANGES 2.0 Evaluation Report is expected to be completed by 28 February 2025.

Tentative work dates:

July: Posting of TORs for the Evaluation Consultant
August: Contracting of Evaluation Consultant; contract signed by end of August
August: Consortium MEAL Manager and SCC MEAL Advisor finalize data collection tools, with input from Evaluation Consultant (quantitative and qualitative)
July-August: SCI, CARE, IRC prepare logistics for data collection – hiring enumerators, developing logistics plans, identifying beneficiaries
August: Posting of TORS for the Qualitative Consultant
September: Contracting of Qualitative Consultant
September: Quantitative data collection takes place in September for 3 weeks
September: Data cleaning 1- 2 weeks
September-October: Qualitative data collection takes place
October: Quantitative data ready for the endline consultant to start work
October: Qualitative analysis takes place
November: Qualitative report shared with Evaluation consultant
October-December: Preparation of Endline Evaluation report takes approximately 3 months (Oct-Dec), which includes SC review cycles
January: Final submission of Endline Evaluation January 10, 2025
February: Final Narrative Report submission to GAC February 28. 2025

Consultant Payment

The consultant will be paid in accordance with the below payment plan.

20% upon signing of contract and submission of Inception Report
30% upon submission of first draft of Endline Evaluation for review
50% upon submission of final and approved Endline Evaluation and and fact sheets

Safeguarding Policy

In addition to ensuring quality data is collected, the consortium will ensure field enumerators are trained on and adhere to Safeguarding policies of the consortium members. Enumerators should be sufficiently enlightened on the ethical issues they are required to observe while collecting data. This includes the Do No Harm principle. In consultation with Save the Children’s National Child Safeguarding Manager, the consultant shall develop and facilitate Safeguarding session for the facilitators during the training.

Although a lot will be covered in the CSG session, some of the general steps that will be taken include:

No child will be interviewed without their consent and voluntary participation. Enumerators will be required to do comprehensive introduction including making children understand how the data will be used, confidentiality and safety of the data.
Adolescent boys and girls will be interviewed by enumerators of the same sex and interviews should not take place in the same place at the same time.
No child will be interviewed in an unconducive environment or un-safe place. Enumerators will not be allowed to interview children inside old, cracked, or dark buildings. Enumerators will be required to look for harmful objects in the interview area before starting the interview.
Children and their parents will be oriented on how they can report complaints they may have on the interviews conducted.
Children will be allowed to terminate the interview at any time as they wish.
Enumerators will not be allowed to interview children inside a house or in a private set-up.

Application Process

Applications for the consultancy must include the following:

Detailed technical proposal clearly demonstrating a thorough understanding of this ToR and including the following:
1. Demonstrate previous experience in coordinating and administering studies of a similar nature,
2. A proposed timeframe detailing schedule of work for writing the endline evaluation report.
A financial proposal with a detailed breakdown of all costs including consultancy fees/costs.
Curriculum Vitae(s) of the consultant team outlining relevant experience.
Names and contact information of three references who can be contacted regarding relevant experience.
A copy of a previous report of a similar nature undertaken on: a) baseline study; OR b) endline evaluation.

The proposal will be scored on both technical (methodology) and financial (budget) aspects.

Completed applications should be submitted electronically in English to the following address: gtimpl@savethechildren.ca with the header: Changes 2.0 Endline Evaluation

Closing date for submission of the application package is end of business day on: Monday, August 12, 2024

How to apply

Completed applications should be submitted electronically in English to the following address: gtimpl@savethechildren.ca with the header: Changes 2.0 Endline Evaluation

Closing date for submission of the application package is end of business day on: Monday, August 12, 2024