Evaluation of the ICRC Hospital Surgical Capacity Building (HSCB) Project At International Committee of the Red Cross

Please see the full RFP document plus appendices here

https://alnap.org/about/get-involved/vacancies/evaluation-of-the-icrc-hscb-project/

Introduction to topic/intervention

Following the ICRC-commissioned independent analysis of the status of its hospital and surgical projects, the ICRC Assembly approved the Hospital Surgical Capacity Building Project (HSCB) in December 2019. Concurrently with the Assembly’s decision, the Norwegian Red Cross recognized similar needs and, consequently, fully financed the ICRC HSCB project as a collaborative effort to address the requirements for the care of war-wounded patients.

The mandate of the project was to enable ICRC’s hospital and surgical projects to provide and support cutting-edge humanitarian surgical and hospital care for victims of armed conflict and other situations of violence (OSV) in low-resource settings. ICRC would achieve this by developing and maintaining a high-qualityworkforce together with being a trusted partner of choice to authorities and clinical institutions in the field, academic institutions, and private partners[1]. The project’s mandate was developed in consultation with ICRC’s hospital team and approved by the project steering committee in February 2021. The HSCB project was originally envisioned to develop the operational capacity of the Movement hospital and surgical staff. However, COVID-19 highlighted the importance of strengthening local capacity to ensure sustainable health outcomes. This resulted in the conceptualization of the Surgical Learning Hub (SLH) and Global Network.

The project considered a phased approach to building capacity progressively from ICRC, to partner national societies, to local institutions. Since its start in 2020, the HSCB project has developed new training pathways and new digital learning tools, strengthened the ICRC HR system, and established local and global partnerships. It has also constructed the Hospital Services Capacity Building Framework, a reference document describing how to build capacity at different levels – namely, individual, organizational, and system – ensuring sustainable outcomes.

Following an extensive evaluation, Bukavu in South Kivu (DRC) was selected as a suitable site for the SLH. The SLH also entails a Network joining together the Movement, academia, development, and humanitarian actors involved in the provision of hospital and surgical care to ensure knowledge-sharing and collaboration. In 2023, an external evaluation assessed the process of establishing the SLH in Bukavu, the findings showed great buy-in from the local partners. A feasibility study was conducted in 2023 to determine the viability of establishing a Surgical Learning Hub (SLH) in South Sudan. The overall conclusion of the feasibility study was that it is not advisable to establish a Surgical Learning Hub (SLH) in Juba or any other location in South Sudan and instead the study proposed other alternative solutions.

The project has entered its transition year and will close in 2024. However, the SLH project – a deliverable of HSCB – will move forward as an ICRC MYMP (multi-year multi-partnership) project. The objectives were aligned with the previous Institutional strategy and aligned with the new Institutional Strategy. Through the MYMP process, the SLH project has been recognized as coherent with ICRC long-term objectives, within and outside of the Health Unit. This evaluation provides an opportunity to document outcomes, challenges, and lessons learned to be used for the planned ICRC MYMP project.

In 2024 the final activities will be concluded. This includes disseminating the Hospital Services Capacity Building Framework across the Hospital Services Programme. Additionally, the project will contribute to setting up the Emergency Hospital training in Nairobi and establishing linkages with the Surgical Learning Hub (SLH) and Global Network. The completion of the War Surgery Training also constitutes a key step in fulfilling the project mandate and ensuring the financial sustainability of the SLH. As the project draws to a close, its governance structure will also be undone, including its Steering Committee. The complete timeline of the project is available in the full RFP document.

[1] HSCB Project mandate

Description of the theory of change or logic model

The initial project benefits were identified by using the Theory of Change (ToC) with four overarching values guiding the delivery of the described activities and based on four hypotheses. These in turn were conceptualized into four interconnected workstreams, aligning within the ICRC Project Management Framework and envisaged to achieve the overarching vision of the project.

WORKSTREAM 1 – INCLUSIVE PARTNERSHIPS Capacity building paired with partnerships in the field of surgical care for victims of armed conflict that is recognized by competent institutions will contribute to sustainable humanitarian outcomes by strengthening local health care systems.

WORKSTREAM 2 – FIT FOR PURPOSE ONBOARDING AND CAPACITY BUILDING MECHANISM A fit-for-purpose onboarding and capacity building mechanism will ensure sustainability of capacity building efforts through flexibility in today’s increasingly complex contexts (i.e., adapting to potential risks by allowing changes and/or rotation of locations as needed).

WORKSTREAM 3 – FUTURE FIT AND FLEXIBLE WORKFORCE ICRC’s hospital/surgical quality capacity building activities supported by adequate talent management tools will foster staff development and retention.

WORKSTREAM 4 – FUTURISTIC AND DIGITAL CAPACITY BUILDING A futuristic capacity building approach that uses digital tools and technologies for learning and quality assurance will increase quality of care in ICRC surgical programs -and wider hospital care.

However, due to unforeseen events, the project has undergone slight modifications to some initial aspects of the Theory of Change (ToC), which were only partially retained. Therefore, this adjustment must be considered when evaluating the effectiveness criteria documenting results and deliverables.

The context of implementing the intervention

Since 2015, the ICRC has faced a significant increase in demand for hospital services, including surgical care, in complex conflict settings and emergencies. Deploying hospital staff, especially surgical personnel, poses unique challenges, as their specific requirements often don’t align with standard procedures or current HR systems. Field missions are typically short, and each team member requires specific skills and qualifications. Over several years, weakened selection, onboarding, and oversight mechanisms have reduced the number of available and experienced surgical staff, coinciding with major growth in ICRC’s field surgical and hospital activities.

The tensions between supply and demand of qualified surgical staff put the ICRC at risk of failing to respond to critical needs of people affected by armed conflict and violence. In response, ICRC governance approved a proposal to enhance its surgical and hospital capacity in late 2019. Concurrently, shifts in the humanitarian sector prompted the ICRC to adopt a more proactive approach to sustainable humanitarian outcomes. This supported the ICRC health unit’s phased plan to build capacity at Partner National Societies and local levels. It is assumed that the supply-demand tension for surgical care will persist, necessitating to build ICRC, Partner National Society and local staff capacity. Ignoring these trends could harm the ICRC’s reputation and hinder sustainable humanitarian outcomes.

The HSCB project was tasked with addressing this challenge developing better mechanisms for capacity building in ICRC and the Movement at large.

About this evaluation

The HSCB project has made substantial progress in meeting its initial objectives. However, certain adaptations were made to the project implementation due to external factors such as the impact of Covid-19 and the evolving financial situation at the ICRC. Lessons learned emphasize the importance of internal process changes, collaborative efforts across departments, and the need for aligned priorities between headquarters and the field. Scope adjustments were observed, at first expanding to local capacity building and then scaling down the initial aspirations of establishing three Surgical Learning Hubs due to the institutional emphasis on financial sustainability. Coordination challenges, internal readiness issues, and communication problems underscore the complexity of driving change through projects in ICRC. In summary, the project has demonstrated resilience and adaptability in navigating unforeseen challenges. The evaluation will help in documenting the achievements of the HSBC project and lessons learned from the implementation process in view of the challenges outlined above and provide valuable insights for future projects within the ICRC.

1. Evaluation purpose

The purpose of this final evaluation of HSCB project is to document results achieved, identify success factors and challenges during implementation of HSCB project, and document lessons learned for the MYMP SLH project and other projects within the ICRC. The evaluation will have both learning and accountability angles.

Specific Objectives

• Generating evidence of HSCB project (2020-2024) results and deliverables considering the rescoping and adaptation from the initial design.
• Understanding the enabling and disabling factors for success and challenges of HSCB project implementation, including:
• Capturing the factors behind changes in the project approach
• Lessons from delivery on continuous adaptation and mitigation with regards to change in scope, the relation with stakeholders and financial management
• Alignment between institutional ambitious and priorities of delegations
• Availability of project management support and guidance throughout the implementation phase
• Generating lessons learned from implementation of an ICRC institutional project within an operations-oriented program (Hospital Services Program) and fully funded by a Partner National Society
• Developing recommendations for future implementation of the MYMP SLH project based on the findings.
• Documenting lessons learned for the ICRC as an institution for future projects based on the lessons from HSCB implementation and in view of the current institutional priorities.

Key evaluation questions

The following evaluation questions are proposed, and the external evaluation consultants are invited to suggest sub-questions during the proposal and/or inception phase to elaborate their approach. External evaluation consultants are also invited to propose any revisions to the questions and provide justification.

Relevance

1. How relevant was the HSCB project approach in addressing the institutional and Movement’s need for improved capacity-building practices?
2. What adjustments – if any – need to be made to the SLH MYMP project based on the evaluation findings to ensure its continued relevance to the ICRC in the near future?

Effectiveness

1. To what extent have the desired outcomes of the HSCB project been achieved?
2. What were the internal and external challenges to the effective delivery of the HSCB project?

Efficiency

1. To what extent the ICRC systems, processes, and operational model were conducive to deploying and managing the HSCB project?
2. How did the partnership with the Norwegian Red Cross influence the implementation of the project?
3. To what extent did the outcomes of the project respond to the partner’s expectations?

Lessons learned

1. What lessons can be learned from managing the HSCB project and how can these be applied to future implementation of SLH and other projects to ensure efficient project management and implementation?

Scope

Technical scope: The evaluation will have a collective nature to document the results of the HSCB project, some lines of inquiry regarding the implementation process, lessons learned and recommendations for future implementation of MYMP SLH project. In addition, lessons learned will be generated for the ICRC as an institution based on the evidence from the HSCB implementation. The project has gone through changes and adaptations since its inception due to internal and external factors requiring the managing team to continually mitigate and respond to these circumstances. Therefore, it is important to capture these nuances of the process to understand better the challenges and limitations for implementing projects within the ICRC.

Temporal scope: The evaluation will cover the period from 2020-May 2024. During the transition year of 2024 some of the project activities will be still on going until the end of 2024, therefore this needs to be considered when discussing the results.

Geographical scope: The evaluation will have a strategic focus on the overall HSCB project results and implementation without any specific focus on the contexts where the partnerships and hubs have been established. There have been other evaluations and reviews that looked more in depth into these elements of the HSCB project.

Intended users

• Primary intended users: The primary audience for this evaluation is the Health Unit, the Norwegian Red Cross, the PES Department, the Assembly, EODG. Department of Mobilization, Movement and Partnership
• Secondary intended users: Department of Operations, RC/RC Movement partners

Key stakeholders

The list of key internal and external stakeholders and their modes of engagement in the evaluation will be developed and finalized by the evaluation manager in discussion with the external evaluation consultants. Key stakeholders and participants will be engaged in the evaluation through different methods including as part of the validation meetings with the Evaluation Advisory Group and discussions of the findings when applicable, during inception phase consultations and key informant interviews as part of the data collection phase.

This list will be composed of the following types of stakeholders:

Internal ICRC and Movement stakeholders – Health Unit staff, stakeholders within PES, ICRC Assembly, RC/RC Movement partners,

External stakeholders – potentially project partners in DRC to be determined during the inception phase.

Evaluation framework

The evaluation will draw on the OECD DAC criteria of relevance, effectiveness, and efficiency to document both results produced by the HSCB project, and the challenges and lessons learned from the implementation process.

Evaluation approach

Specific methods

The evaluation will deploy both primary and secondary, qualitative, and quantitative methods for data collection. At a minimum, qualitative data collection methods should include document review, key informant interviews, group interviews, and workshops where appropriate. Analysis methods will require triangulation between multiple data sources and methods with an analysis matrix submitted as part of the evaluation report.

Primary data collection expectations, disaggregation

Primary data collection can include key informant interviews, group discussions with ICRC and Movement stakeholders and other relevant external stakeholders, and surveys if applicable. Quantitative methods may be primary (e.g., survey) and/or secondary (HSCB project documents and data). External evaluation teams are invited to suggest the most applicable data collection strategies based on the scope, objectives, and timeframe of this evaluation and justifying their choices in the proposal.

Depending on the methods and the sources, data should be collected in a way that it can be disaggregated by individual attributes (e.g., sex, age, disability, etc.) as per the Sex, Age and Disability Data Disaggregation Framework[1] and/or contextual attributes if applicable depending on the evaluation approach.

Secondary data availability

Secondary data will include HSCB project reports, past evaluations, and feasibility studies with relevance to the project. Relevant strategies to be reviewed as part of this evaluation will include the ICRC Institutional Strategy 2019-2022, the health strategy 2020-2023, ICRC learning and development strategic objectives 2019-2022 and the ICRC hospital vision paper and strategy, ICRC Institutional Strategy 2024 – 2027, Assembly orientation note 2018 and HSCB project mandate. This list of documentations is not exhaustive, and the final list will be discussed and agreed upon with the evaluation consultants during the inception phase.

Translation requirements (if applicable)

All written communication and reporting on the evaluation will be in English, except where needed to communicate with colleagues speaking French. Members of the consultant team will need to be able to converse in French.

[1] 4698_002_Sex, Age and Disability. How a data disaggregation Framework Can Make Our Operations More relevant; 06.2023 (icrc.org)

Risks, limitations

The ICRC’s 2023-2024 restructuring led to higher workloads and additional responsibilities for staff members and managers throughout the organization, thereby reducing their availability and receptiveness for processes such as this evaluation. The current operational environments for ICRC require especially high workload from health staff responding to multiple emergencies. The evaluation will need to be accommodating regarding the availability of staff members, and in the case of non-availability contingency plans for data collection will be needed to ensure the validity of datasets.

Rescoping the project from the initial Theory of Change and project mandate will need to be thoroughly captured when assessing achieved results and deliverables. In addition, there are activities still planned for the transition year of 2024 and will still be underway concurrently with this evaluation. Evaluation teams are required to expand on risks and include methodological risks, limitations, and mitigations in their proposal and during the inception phase.

Deliverables

1. A 1-page briefing outlining the purpose, timing, and key messages to explain the evaluation process to stakeholders. This acts as a communication tool internal within the evaluation particularly during the data collection phase. It includes the contact details of the evaluators, the evaluation commissioner, and the ICRC’s integrity weblink.
2. An inception report with PPT detailing a proposed methodology, evaluation matrix, list of stakeholders to be consulted, workplan and timeline, and the tools for data collection to be presented to the Evaluation Advisory Group (EAG) and the commissioning Unit/Delegation for validation.
3. A preliminary findings and recommendations presentation of the key findings, recommendations, lessons learned, and best practice addressed to the Advisory Group and the commissioning Delegation.
4. A draft evaluation report. This should be clear and simply written, free of unnecessary jargon. The main body of the report should not exceed 40 pages. The report outline will be agreed with the supplier as part of the inception report. The report will include the following sections:

• Executive Summary – a short overview of the report
• Context and background
• Methodology
• Main findings of the evaluation, and conclusions
• Concise recommendations, with consideration of the degree of prioritization and any necessary sequencing for actions and the responsible persons

Annexes:

• Terms of Reference
• Evaluation Matrix
• Analysis framework
• List of stakeholders interviewed, anonymized.
• List of documents consulted
• Data collection tools and consent forms
• Any other relevant documents to support the report.

1. A final report, with feedback integrated from the EAG and other internal stakeholders as relevant.
2. A visual communication product such as a poster, infographic or max 4 pages brief that presents the key messages and recommendations of the evaluation in an engaging manner. When relevant only and agreed upon at the inception phase with the evaluation manager.

Ethical considerations and safeguarding

Evaluators are required to adhere to international best practices and standards in evaluation. It explicitly requires evaluators to abide by the Professional Standards for Protection Work; the ICRC’s Code of Conduct; the ICRC’s Code of Ethics for Procurement; and the ICRC Rules on Personal Data Protection. The evaluation design and implementation must apply the ICRC’s guiding principles and approach on Accountability to Affected People. Evaluators will be expected to obtain informed consent from those interviewed and ensure that if an interviewee is quoted, the interviewee can’t be identified unless with consent. This evaluation will not require formal Ethical Review Board approval of the inception report.

Evaluation team members or consultants must not have any conflict of interest (COI). An evaluator cannot evaluate an intervention if they have previously participated in its design or implementation. Perception of COI is a broader consideration and bidders are required to explicitly identify any potential and any likely perceptions of COI and propose mitigation strategies.

The evaluation should not collect any personally identifying information and is therefore not considered a data processer on behalf of the ICRC for the purposes defined in the GDPR. However, data generated through the evaluation may be sensitive, and the confidentiality of stakeholders must be upheld. The raw data generated by the external evaluators is owned by the ICRC but as an independent evaluation the ICRC does not request access to this material unless there is a serious issue, dispute, contractual failing on the part of the supplier, or for other reasons not yet determined. In the case of the ICRC requiring the supplier to handover primary data, it must be cleaned and anonymized to protect the privacy and confidentiality of participants.

Management of the evaluation

The Health Unit is the commissioning team of this evaluation (“Evaluation Commissioner”), with the HSCB Project Manager having overall responsibility for managing the external evaluation team and convening the Evaluation Advisory Group. The Evaluation Manager is responsible for implementing the evaluation from conception to the management response and using the findings and recommendations.

At the scoping and preparatory phases, the Evaluation Manager develops the TOR, identifies, and convenes the EAG, organizes the procurement (with the Eval Office depending on the procurement route), and organizes the collation of relevant documentation to share with the external team.

At the inception phase, the Evaluation Manager sets the parameters for engagement with the external evaluation team (expectations, communications, document sharing) and identifies and introduces the evaluation team to key stakeholders to consult as part of their onboarding and planning for the evaluation. The EM manages the feedback process on the draft inception report among the EAG (and the Evaluation Office facilitates the formal quality assurance process).

At the data collection phase, the EM facilitates introductions to internal and external stakeholders as needed, supports the internal distribution of surveys if relevant, convenes the EAG for the presentation on findings, and facilitates the feedback process on the draft report.

At the dissemination phase, the EM proposes and facilitates sharing of the findings (e.g., the report, its executive summary, webinars) and manages the completion of the management response to the recommendations.

The team members of the HSCB project and Health Unit will contribute to the processes of onboarding the external evaluation team, accessing documentation, and providing supporting information.

All evaluation deliverables will be submitted simultaneously directly to the Evaluation Manager and the ICRC Evaluation Office in due course for a first review. The Evaluation Manager will mobilize and share the inception report and final report with other key ICRC stakeholders for feedback, such as the Evaluation Advisory Group. The Evaluation Manager will provide consolidated feedback to the external consultants, including external quality assurance reviews overseen by the Evaluation Office. The key dates for deliverables and milestones will be determined during the inception phase in consultation with the Evaluation Manager.

The Evaluation Office at HQ will accompany the process providing technical advice and feedback to the Evaluation Manager at key stages, including scoping of the exercise, identifying and recruitment of the external evaluation team, providing feedback during the inception phase on the proposed methodology, methods, and data collection instruments, providing feedback on the inception and the final reports directly and through the established Evaluation Quality Assurance process and ensuring impartiality.

Evaluation Advisory Group

An Evaluation Advisory Group (EAG) will accompany this evaluation, providing their expert advice and feedback on key stages. The members of the EAG may be included in the initial onboarding process for the external evaluation team. The EAG will be consulted for the inception phase presentation/discussion on the plan for data collection and analysis, and for the presentation of the final report findings. The EAG for this evaluation consist of the following members:

• Yves Giebens, Hospital Service Program Coordinator
• Catherine Savoy, Dep. Head of Health
• Etienne Penlap Temdie, Health Coordinator, DRC
• Raphael Tenaud, OPS Coordinator Africa
• Araceli Lloret, Head of DM Norwegian Red Cross
• Julie Fuvel, EPMO Adviser

Timeline

The evaluation is anticipated to start in August 2024 with all deliverables expected by the end of January 2025. The ICRC does not mandate that working days must be consecutive. The ICRC envisages that the consultancy will be mostly done remotely without any field visits. During the inception phase, it is recommended for the team leader to visit ICRC HQ office in Geneva for initial in-person consultations.

A timeline is provided in the full RFP document.

Evaluation Quality Assurance

The process will conform to the ICRC’s quality standards for evaluations, which set out what the minimum standards are for an independent, professional, and reliable evaluation. These sets of quality standards are in the form of checklists and the key products of the evaluation (Terms of Reference, Inception Report, and Evaluation Report), are all impartially assessed by an independent expert. This process is managed by the Evaluation Office. The quality standards are provided to the evaluators at the start of the contract. In addition, regarding sequencing, the Inception Report must be signed off by the internal evaluation manager and the Evaluation Office before any data collection activities begin.

Publication of final report

The ICRC’s Evaluation Office’s Policy is to publish the Executive Summary and or the full report on the ICRC’s website for transparency as per international evaluation best practices. The reports are fully anonymous and will be published unless there are strict limitations to its publication or if it includes sensitive information based on the categorization identified in the ICRC`s Access to Information Policy.[1] This decision is discussed at the ToR preparation phase to ensure it is clear to all stakeholders.

Dissemination of findings

The ICRC’s Evaluation Office will publish the Executive Summary and or the full report based on the final agreement with the HSCB team on the ICRC’s website. The full evaluation report with findings will be disseminated through different modalities internally, including through publication on Evaluation Office community site to the following stakeholders:

• ICRC Assembly
• PES
• OPS
• EODG
• MMP
• Norwegian Red Cross and other donors if applicable

The evaluation consultants are expected to present the final report to selected ICRC internal audience in addition to the EAG members during one dedicated event organized by the Evaluation Office or the Commissioners of the evaluation.

Follow-up of recommendations

In order to strengthen the use of the evaluations at the ICRC, fostering ownership over the process of change and ensuring accountability for results, the intended users of this evaluation will initiate the management response process as a follow-up action facilitated by the Evaluation Office. The Evaluation Office will support the Evaluation Commissioner and responsible teams and stakeholders in developing and tracking the management response actions through dedicated systems and processes. An Evaluation Management Response template is provided at the end of the evaluation to the evaluation commissioners to keep all accepted recommendations in a centralized manner. The Evaluation Office organizes 6-month and 12-month check-ins to discuss the status of the recommendations’ implementation and collect feedback on the relevance of recommendations.

Budget range or anticipated working days

The budget ceiling for this evaluation is 60,000 CHF, however overall cost will be taken into consideration for the selection decision including cost effectiveness of the proposal. The consultants are expected to elaborate in the proposal on the number of anticipated working days for this assignment based on the budget range, proposed timeline and level of efforts required at each stage of the evaluation. ICRC is recommending for the team leader to visit the ICRC HQ in Geneva for initial consultations during the inception phase. The travel cost associated with the visit to ICRC HQ should be included in the budget.

[1] https://www.icrc.org/sites/default/files/document_new/file_list/access-information-policy.pdf

Expertise required

Team Leader:

• Substantial experience (minimum 15 years) in leading teams and designing and delivering rigorous evaluations in the international and humanitarian sectors, meeting international evaluation quality standards.
• Proven experience in evaluation of health-related interventions in humanitarian settings
• Minimum academic qualification: post-graduate degree in evaluation, humanitarian response, or other field relevant to this evaluation.
• Understanding of the ICRC’s mandate and work, previous experience with ICRC as a staff member or consultant would be a strong asset given the candidate does not have a conflict of interest and have not worked on the design or implementation of anything falling within the scope of this evaluation.
• Proven expertise in facilitating participatory workshops involving different groups and participants

Across the team:

• Proven experience of conducting mixed- methods humanitarian evaluations
• Expertise in developing and adapting qualitative and quantitative evaluation tools for different stakeholders, geographic contexts, and thematic areas.
• Knowledge, understanding, and/or experience of the ICRC’s institutional and operating models.
• Expertise and technical knowledge on project management in humanitarian settings, health related interventions in humanitarian settings, assistance, and protection in the humanitarian sector.
• Understanding of the ICRC’s mandate and work, previous experience with ICRC as a staff member or consultant would be a strong asset given the candidate does not have a conflict of interest and have not worked on the design or implementation of anything falling within the scope of this evaluation.
• Knowledge of English is a must, ability to work in French is an advantage