Final Evaluator – STEEP At Internews

ABOUT THE OPPORTUNITY

Internews is seeking a qualified consultant or firm to conduct a final evaluation for its project, STEEP (publicly known as Infuse), from October 2024 to February 2025 (timeline subject to change slightly). The evaluation will be conducted fully remotely/online.

ABOUT THE PROJECT

The STEEP project is behind the development and piloting of Infuse. Infuse is a community-developed framework providing pathways for digital protectors (digital security trainers, technologists, auditors, etc.) to advance their knowledge and skill sets in areas of specialized technical expertise (STE). Such areas include detecting and analyzing malware, investigating malicious infrastructure, and web security. Infuse aims to make much-needed STE more widely available in contexts where at-risk communities need it the most, by upskilling protectors who already have established trust with these communities. This involves localizing expertise, as well as expanding accessibility to and increasing the diversity of providers of specialized digital security support.

LOGISTICS

Internews has offices in Arcata, CA, Washington, DC and London, UK. Ultimately this is a remote-based project and candidates based anywhere in the world will be considered. However, there are some locations in which Internews is not able to support remote work.

The duration of the evaluation is expected to take approximately 4 months.
The estimated budget for this evaluation is 9,000-11,000 USD.

OUR COMMITMENT TO FOSTERING A CULTURE OF BELONGING

We are an organization of dynamic, mission-driven individuals who are passionate about our core values and about supporting positive change in the world. We pride ourselves on our commitment to innovation and flexibility. We believe that diverse teams are strong teams and work to support an ethic of belonging, dignity, and justice for all people. Our current team includes a mix of genders, parents and non-parents, and people of multiple races, nationalities, ages, sexual orientations and socioeconomic backgrounds. We are an EEO employer and encourage candidates of all races, genders, ages, orientations, ethnicities, and national origins to apply, and welcome those with alternative backgrounds and experiences.

Evaluation Purpose and Objectives

The purpose of the final evaluation is to assess the project’s performance and success against expected outcomes, using the OECD DAC criteria as framework. This includes evaluating the project relevance, coherence, effectiveness, efficiency, impact and sustainability. The evaluation should identify ways of improving where relevant, looking the indications of potential impact and assess early signs of sustainability of project results.

Specific objectives of the evaluation include:

• Assess the relevance of the STEEP/Infuse project in addressing the needs and priorities of the targeted beneficiaries and stakeholders, considering the overall context;
• Assess to what extent the project achieved the set outcomes and identify any unintended results;
• Assess the project results on key target beneficiary groups which primarily include project fellows (Infuse learners) but also may include organizations and individuals which the fellows provided support to, organizations which conducted programming alongside learners, and others involved in the Infuse framework (e.g., those who provided mentorship to learners);
• Examine the project’s efficiency in using resources to deliver results;
• Identify early signs of lasting impact and assess potential sustainability of the Infuse framework. Provide analysis regarding the likelihood that the framework will be used in the long term and what conditions or resources would be necessary to achieve long term sustainability;
• Analyze underlying factors beyond Internews’s control that affect the achievement of the project results;
• Provide actionable recommendations for potential future interventions, related to STEEP/Infuse or similar projects.

Additional components may include the following (optional):

• Producing ‘spotlight’ case studies which showcase the successes and stories of select project fellows for funder and external audiences.

The key audiences of the final evaluation are the Internews STEEP/Infuse team, the broader organization, the funder and project relevant stakeholders, and partners.

Key Evaluation Questions

In line with the evaluation purpose and objective, the following key evaluation questions are proposed for the final evaluation. However, these can be expanded upon, narrowed down, and modified by the evaluator in consultation with Internews.

• How relevant is the STEEP/Infuse project, particularly, the Infuse framework and resources, in addressing the needs and priorities of digital protectors and at-risk communities worldwide?
• To what extent does Infuse align with the challenges identified in the digital protection landscape?
• How adaptable and localizable is the Infuse framework for diverse contexts and communities?
• How well do the STEEP/Infuse project and Infuse Framework complement other initiatives in the digital protection field?
• To what extent has the project achieved its set outcomes, particularly in developing and piloting the Infuse framework?
• How effective are the Infuse learning paths in increasing digital protectors’ knowledge and skills specific to relevant areas of STE and improving their ability to serve at-risk communities?
• To what extent has Infuse bolstered networks and communities of digital protectors?
• To what extent has Infuse made skill-building materials for digital protectors more accessible?
• To what extent has it made skillsets for digital protectors more standardized?
• What unintended results, if any, has the project produced?
• How efficiently has the project utilized its resources to develop and implement the Infuse framework?
• How effective was coordination with core partners on the project?
• How has the project impacted participating digital protectors (primarily learners (fellows) but also those who provided mentorship to learners)?
• What external factors, beyond the control of Internews, have influenced the achievement of project results, and how have these external factors impacted the overall success and challenges of the project?
• What are early signs indicating the potential for long-term sustainability of the Infuse framework?
• To what extent are there indications that the program’s results will last after its completion?
• What recommendations can be made for future interventions in similar contexts or with similar objectives?

Evaluation Approach and Methodology

The evaluator has the freedom to utilize any relevant evaluation approach with relevant and robust research methods, either quantitative or qualitative, that would fulfil the purpose and objectives of the evaluation and respond to the evaluation questions. A blend of both primary and secondary data is recommended.

The evaluator should, at a minimum, in their methodology, include the following data collection methods:

• Document review: Thorough review of pertinent program-related materials, including but not limited to, quarterly narrative reports, M&E reports and tracking tools, survey results, and content produced by the program.
• Data research: Thorough review pertaining to relevant external sources to ensure appropriate level of knowledge of the context of the project.
• Qualitative Interviews (Focus Group Discussions and Key Informant Interviews): The evaluator is expected to hold interviews and meetings with relevant staff of the program, partners, and beneficiaries.
• Evaluators, proposing the integration of specific evaluation approaches such as Most Significant Change, Ripple Effect Mapping, Outcome Harvesting, Outcome Mapping, Contribution Analysis, etc., would be advantageous.

Estimated Timeframe and Budget

The final evaluation report should be submitted February 14, 2025. The evaluation will comprise of an estimated 22-34 days of work and involve the following elements (please note that deadlines may be subject to change):

REQUIRED TASKS

ESTIMATED Level of Effort (DAYS)

Intro meeting to set goals and priorities

1

Conduct desk review

4-5

Draft and submit of inception report including evaluation matrix with Internews template provided

1-3

Finalize evaluation design and methodology, and along with preparation of data collection tools

1-3

Conduct data collection

4-6

Conduct data analysis

4-6

Debrief Internews team on preliminary findings

1

Update the initial evaluation outputs and prepare the final outputs

4-6

Finalize and submit the final outputs based on Internews’s comment

1-2

Final debriefing on key findings with the Internews team and the donor (if requested)

1

Expected Tasks and Deliverables

• Kick-off meeting to set goals and priorities

Inception Phase:

• Conduct a comprehensive desk review that involves reviewing programmatic documentation produced by the project team to grasp the structure, evolution, and needs of the project. Documentation may include quarterly reports, research reports based on interviews and focus group discussions, notes from meetings with partners and fellows, project M&E mechanisms and reporting, funder requirements, etc.;
• Submission of an inception report outlining detailed the final set of evaluation questions, overarching evaluation approach and methodology with associated data collection strategy and data collection instruments. The Inception report should include the Evaluation Matrix. An Internews-provided template for the inception report will be available.

Data Collection and Analysis Phase:

• Conduct data collection, including interviews, focus group discussions, and/or other data collection activities with project staff and partners, fellows (learners), mentors to learners, and other stakeholders aligned with the evaluation approach and methodology;
• Prepare data for analysis (transcription, cleaning, etc.) with submitting raw data;
• Conduct a comprehensive data analysis by bringing together the relevant data collected in the previous phases to respond to the evaluation framework and develop strategic and operational recommendations;
• Discussions with partners and our team will help confirm, challenge, or contribute to additional insights.

Report Writing Phase

• Submission of a final evaluation report (25-50-pages (not including appendices)) in English with findings and recommendations. An Internews-provided template for the report will be available for use and adaptation. The body of the report will provide a narrative description of results achieved, operational lessons, and recommendations. Executive summary or similar document should be included. Where appropriate, it will also include illustrative graphs and charts or infographics that provide useful summaries of project data. Appendices will provide detail such as cross tabulations from reviewing quarterly reports and interviews, as well as a list of people interviewed, and survey results (if any), so readers can clearly see the data that supports the findings and conclusions;
• [Optional] Submission of ‘spotlight’ case studies which showcase the successes and stories of select project fellows for funder and external audiences. These should be separate from but accompanying the final evaluation report.

Dissemination and Learning Phase

• PowerPoint presentation and debrief with Internews staff and partners on the evaluation findings and recommendations;
• [Optional] Conduct learning session

QUALIFICATIONS WE’RE LOOKING FOR

Required

• Demonstrated experience conducting evaluations of advocacy and human rights programs, particularly for US donors;
• Experience with qualitative and quantitative M&E data collection and analysis methods;
• Strong facilitation, presentation, writing and communication skills;
• Fluency in both written and spoken English;
• In all services, uphold Internews’ Core Values and demonstrate commitment to fostering a culture of belonging.

Preferred

Note: Candidates who do not have these preferred qualifications, but who are interested and willing to learn, are encouraged to apply.

• Experience in the fields of internet freedom, digital security, digital rights, etc.;
• Formal education (e.g., bachelor’s degree, graduate degree, professional certification, etc.) in a related field;
• Fluency or advanced proficiency in Spanish. French or Arabic fluency may also be helpful.