Evaluation Vietnam – Project for Promotion of Inclusion of Persons with Disabilities and Mitigation of Climate Change At Caritas Germany

Terms of Reference

Midterm Evaluation, Caritas Da Nang, Project for Promotion of Inclusion of Persons with Disabilities and Mitigation of Climate Change, Diocese of Da Nang

1. Project Information

Project Title:

Promotion of Inclusion and Mitigation of Climate Change: Installation and Operation of Solar Energy Systems with Vocational Training and Job Creation for People with Disabilities, Da Nang, Vietnam

Project Number:

DCV project no: P.325-2019-006

BMZ project no: PN: 5119

Project Period:

01.05.2021 – 28.02.2025

Project Evaluation Period:

01.05.2021 – 31.12.2023

Overall Budget:

660,000 EUR

Project Donors:

German Federal Ministry for Economic Cooperation and Development (BMZ) (75%) and Caritas Germany (25%)

Project Partner:

Caritas Da Nang, Vietnam

2. Background information and Context

Background

Situational background: With a current total population of around 95.5 million people, it is estimated that about 7.5 million people with disabilities live in Vietnam, approx. 7.8 percent of the population, with an increasing trend. They are often among the poorest of the poor (vicious cycle of poverty and disability) and are disproportionately vulnerable, often stigmatized and largely excluded from key areas of life such as health care, education and vocational training, gainful employment and social life in general. The majority of persons with disabilities have no vocational qualifications, even if they may have managed to go to school (inclusive schooling is still the exception) and may even have a high school diploma, because inclusive vocational training courses are hardly available or even unaffordable. Vocational training and gainful employment are key factors for the social participation of people with disabilities.

As a reaction to the increasing energy and climate problems, the renewable solar energy market in Vietnam has been growing in recent years and will surely have growth potential in the future. With the solar energy sector, which requires an increase of skilled workers, a new economic sector and labor market has been opening-up, which can offer opportunities for vocational training and employment of young people with disabilities and thus for their inclusion in the working world. The installation and maintenance of the photovoltaic systems in the present project on the roofs of two church-supported institutions provide opportunities to train young people with disabilities for the job of solar worker cum electrician while facilitating their entry into the solar energy and general electrician labor market adapted to their individual abilities.

Key stakeholders and donors:

The main stakeholders involved in the project activities are: Caritas Da Nang, Thanh Tam Center Da Nang, the private Technical Vocational Training Institute Phuong Dong College of Da Nang, the Solar Energy Company Quang Minh Dat Electrical Construction and Consulting Co. Ltd (abbrev. QMD Company) responsible for the installation and maintenance of the solar energy systems created with the support from this project, the management of the An Binh Clinic and of the Social-Pastoral Center of Da Nang Catholic Diocese, the parents and relatives of the young trainees with disabilities, the trainees themselves, and the (potential) employing companies for the trainees who have graduated from the training program. – The donors of the project are the German Federal Ministry for Economic Cooperation and Development (BMZ) which is providing 75 percent of the project funds; DCV/Caritas Germany International Department is contributing 25 percent of the project funds; local resources come in by way of personal family support in-kind for the young trainees.

Former projects: The idea for this project arose from the many years of expertise and community-based project experience of the two state-recognized church institutions Diocesan Caritas Da Nang and the Thanh Tam Center of the Catholic Sisters Congregation Soeurs de St. Paul de Chartres in Da Nang, the latter specifically in the sector of social and professional inclusion of people with disabilities.

Initial consultations with the Vietnamese project partner Caritas Da Nang on this project topic already took place in 2018/2019. For the concrete preparation of the project, a technical feasibility study was prepared in 2019, which was financed by DCV/Caritas Germany International Department from its own funds, supplemented by workshops in which the concept for the vocational training of young people with disabilities was developed together with the local partner. The project was proposed for funding to the BMZ funding title “Private Träger” and received an A-categorization in the third attempt by Engagement Global / BENGO in 2020. The full application submitted to BENGO in May 2020 was amended several times in response to BENGO queries and finally approved at the end of February 2021. The BENGO forwarding contract was received in March 2021, and the project then began officially in May 2021.

Context

Intended Objectives:

Overall objective / impact: The use of modern solar energy technology contributes to effective climate protection, leads to sustainable management of social institutions and thus to the improvement of their services to their users / target groups and opens up the new occupational profile “electrician and solar worker” for the professional and social inclusion of people with disabilities.

Project objectives / outcomes (social as well as technical or economic and ecological):

1) Young people with disabilities (40 in the course of the present project) are trained and independently earn income in the occupational field of electrician and solar worker with different employment opportunities.

2) Patients of the rural An Binh Clinic and users of the diocesan Social-Pastoral Center of Da Nang receive improved health care as well as services in the form of educational and cultural programs.

3) The use of solar energy actively contributes to national and global climate protection.

Note: For detailed sub-objectives / outputs & indicators, see BMZ PT full proposal

Target Groups: With the operation of the installed solar systems, the target groups and beneficiaries directly reached are the clients of two facilities: patients of the rural An Binh Clinic (about 7,200/year) and participants in the programs and events of the Social-Pastoral Center of the Diocese of Da Nang in Da Nang City (about 2,000/year).

With the project’s social inclusion component of linkage to vocational training and employment promotion especially developed for young people with disabilities, about 40 selected trainees with disabilities are directly reached as a pilot group of later electrical and solar workers.

Indirectly, all people are reached who work and live together with the trainees with disabilities and who experience the importance of inclusion concretely: the families of the trainees, the staff of the training/employing companies and their customers, neighbors and the general environment of the residential groups in which the young people with disabilities live during vocational training.

The implementation activities throughout the project as well as the current state of the project’s implementation to date (end of December 2023) are described in the project partner’s annual narrative reports for the years 2021, 2022 and 2023 (the latter was sent to Caritas Germany on 22/01/2024) as well as in Caritas Germany’s annual narrative reports to BMZ/bengo for the years 2021 and 2022 (2023 is to be submitted by Caritas Germany to BMZ/bengo by 30/04/2024).

Some notable successes and some problems in short:

• The solar power systems that have been installed are working well and have increased the service capacities of the rural An Binh Clinic and of the diocesan Social-Pastoral Center. However, the initial plan to sell surplus solar power to the national power grid EVN, and thereby generate income, is on hold due to an unexpected change of the Vietnamese Government’s energy policy, suspending payment for the intake of privately produced electric power into the public grid.
• The young trainees with disabilities are motivated and enjoy their training. They have gained self-confidence in daily life. To date, the majority of them have found employment, but not all of them yet. It is not easy for the implementing project partner to find work placements for them and to provide on-the-job accompaniment and problem solving for both sides, the employees and the employers, among other reasons due to the project partner’s limited project staff capacity.
• In the beginning, the progress of project implementation was slowed down due to the effects of the Corona pandemic, but the project partner was able to compensate and move slowly forward step by step, partly with the help of remote virtual work meetings.
• A recurrent problem has been project staff fluctuation: resigning of the first project coordinator and of the lead social worker; both have been replaced, but such transitions always cause interruptions of project continuity.

The project has been operational since May 2021 and will continue until February 2025. The local cost of the project (partner budget) amounts up to EUR 559,745.00 whereby 75% (EUR 419,808.75) are funded by the German Federal Ministry for Economic Cooperation and Development (BMZ), the rest is financed by Caritas Germany’s own funds (25% = EUR 139,936.25).

3. Objective of the Evaluation

The overall objective of this mid-term evaluation is to analyze and understand better the impacts of this project’s interventions in its different components and to improve further intervention design and management in the remaining project period and for working towards post-project sustainability.

Specific objectives of the evaluation:

On the social inclusion side: This mid-term evaluation is being undertaken at this time, because – even though somewhat beyond mid-term – it appears meaningful now that 25 (62.5%) of the originally envisaged number of 40 trainees with disabilities have graduated from the project’s vocational training courses from the beginning of the project until the end of 2023. So, at this point there is a good opportunity to extract lessons learned from the implementation of the training courses delivered so far and from the job placement efforts, and based on this, to make any necessary adjustments in this regard in 2024 (in which the project partner wishes to provide two more training courses).

On the technical & economic / ecological side: Now is a good moment to analyze (quantitively and qualitatively) how the installed solar power systems actually presently benefit the service provision of the An Binh Clinic and of the Social-Pastoral Center.

• Type of evaluation: participatory mid-term evaluation
• Reasons for scheduling this type of evaluation: due to slow onset of project implementation, resulting from the effects of the Corona pandemic, mid-term is removed to slightly later than exactly mid-term.
• What does the evaluation seek to accomplish: to draw lessons learned for the remaining project period and for already working towards post-project sustainability.
• How will the evaluation benefit the stakeholders: the stakeholders will benefit from the improvements undertaken on the basis of the lessons learned.
• Who will use the evaluation results: the evaluation results will be used by the local project partner Caritas Da Nang and by Caritas Germany.
• How will the findings of the evaluation be used in terms of accountability, learning and improvement: Caritas Da Nang and Caritas Germany will discuss how to use the conclusions and recommendations of the evaluation for improvement; this includes transparency and accountability vis-à-vis all the stakeholders of the project.

4. Scope of the Evaluation

• Thematic coverage: Which activities or aspects of the program are being evaluated?

Mainly the social inclusion aspects and related activities; if possible the technical and economic / ecological aspects.

• Time period: What time period of the project implementation should be analyzed during the evaluation process? (E.g. first, second phase; years; the whole program)

The whole program from the beginning to date: 2.5 years.

• Geographical focus: Which primary geographical area should be evaluated? Should all cities/region/countries covered by project implementation be analyzed during the evaluation process? Or just some specific area/region?

The area within the boundaries of the diocese of Da Nang, which is the operating area of the project partner and the area from which the trainees with disabilities are recruited.

• Target group: which target groups should be evaluated*?*

All stakeholders: The project staff of Caritas Da Nang, the project advisors of Thanh Tam Center Da Nang, the 25 trainees with disabilities who were trained by the end of 2023 and who either found employment or did not yet find employment, the families of the trainees, the private technical vocational training institute Phuong Dong College of Da Nang, the solar energy company Quang Minh Dat Electrical Construction and Consulting Co. Ltd., the (potential) companies for providing employment for the graduated trainees, the management and staff of the An Binh Clinic, the management of the Social-Pastoral Center of Da Nang Catholic Diocese.

• Out-of-scope: Specify issues that are explicitly outside of the scope of the evaluation]

Outside the scope of the evaluation are the patients of the An Binh Clinic and the users of the Social-Pastoral Center.

5. Evaluation Criteria and Key Evaluation Questions

This evaluation will focus on 5 OECD-DAC evaluation criteria and related key questions, which are:

1. Relevance (from the perspective of the project beneficiaries) – key evaluation questions: To what extent are the project’s objectives and activities consistent with the needs and priorities of the beneficiaries?
2. Effectiveness – key evaluation questions: To what extent have the project’s objectives been reached? If this has been challenging, has some progress been made towards their achievement? – Are the project activities adequate to realize the objectives? – What were the problems and constraints encountered during the implementation? – What steps need to be taken to increase the effectiveness of the project (on organizational level; on country, province, community level; on donor level)?
3. Efficiency (economical perspective) – key evaluation questions: Were the implemented project activities cost-efficient? Are the project results being achieved at an acceptable cost?
4. Impact (significant and/or lasting effects produced by the project: directly, indirectly, intended and unintended) – key evaluation questions: What real difference has the project made for the beneficiaries; how have their living conditions changed or not changed? – What is the assessment of the project’s contribution to human and institutional capacity building in terms of knowledge and competencies? – What are the social, economic and environmental effects on individuals, communities and institutions?
5. Sustainability (the probability of continued long-term benefits) – key evaluation questions: Are the achieved positive changes likely to be maintained after donor funding will have ceased? – What are the factors (internal / external) that will help to sustain the positive effects of the project? – To what extent has the project established processes and systems that are likely to support the continued implementation of the project? – What steps need to be taken to ensure project sustainability? – Is the project contributing to social change and institutional capacity?

6. Approach and Methodology

The Evaluator(s) is (are) expected to perform the evaluation in a strictly methodical manner in order to produce verifiable information and make recommendations that are sufficiently valid and reliably based on dependable data and analysis, which are accessible to the client.

The evaluation will adopt a participatory, transparent and solution-oriented approach using internal and external stakeholders. The evaluation process, as well as the final results should be gender-sensitive and human-rights-based. The evaluator(s) will use mix-methods that are able to appropriately address the primary evaluation questions, as well as properly support its derivative conclusions and recommendations.

The evaluation methods to be used may include, but are not limited to:

• Review of project documentation (Desk Research): An array of major documents that should be studied (e.g. project documents, monitoring reports, final reports, as well as statistical data, documents pertaining to similar projects led by other donor organizations, etc.) Existing project documents and reports are to be shared with the evaluator in order to facilitate the realization of the tasks. The evaluator is invited to request additional documents that may be needed for the completion of the evaluation.
• Interviews with all the key informants and key players: (semi-)structured Interviews with beneficiaries and other important stakeholders as already indicated above are a must and should be supported by interview protocols and the list of respondents (e.g. direct project beneficiaries, local staff, etc.). Among others, it will be important to interview (potential) employers of the graduated trainees with disabilities, to know their attitudes towards inclusion of persons with disabilities among their workforce, and to assess the possible need for more intensified lobbying and support vis-à-vis such employers.

1. The selection criteria should be based on gender, age, beneficiaries/non-beneficiaries and geographical spread in the project region.
2. The exact sample size should be proposed by the evaluator(s) and discussed together with Caritas Da Nang and Caritas Germany before the start of the evaluation. Direct observation during field visits: CaritasDa Nang will organize field visits at different project sites.

• Focus group discussion with beneficiaries, local leaders or local partner staff (if feasible within the timeframe)
• Standardized Survey and questionnaires: to quantify the result and deliver a statistical analysis.

7. Deliverables and Timeframe

Optional deliverables for the evaluation are:

Inception Report:

An inception report will be submitted by the evaluating consultant in order to demonstrate his understanding and planning of the evaluation, which will be reviewed and discussed in cooperation with Caritas Da Nang and Caritas Germany.

The inception report should include an evaluation matrix[1] *(*including the final evaluation questions and indicators); the overall evaluation design and methodology with a detailed description of the data collection methods and data analysis techniques, as well as a proposed timeframe for the activities and deliverables. The evaluator is free to suggest additional methodological framework for the evaluation.

The inception report of the evaluation should not exceed 5 pages and follow a predefined structure as described in Annex 1.

Workshops/ Meeting:

The evaluator is free to select the type of workshops (e.g. kick-off/mid-term/validation workshop) during the conduction of the evaluation.

The Kick-off workshop provides an opportunity to discuss organizational processes, methodology, data collection instruments, timetable, milestones, reporting dates, task and responsibilities. Moreover, the mid-term workshop is useful to discuss problems that may arise in the course of an evaluation and allow a joint development of strategies for solutions.

During a validation workshop, the evaluator will present the findings collected in the field to Caritas Da Nang and Caritas Germany. The workshop will be organized in order to discuss and validate findings, lessons-learned and recommendations proposed by the evaluator. Stakeholders are invited to make recommendations for amending the review.

Draft Report:

The consultant is expected to submit the primary findings resulting from his evaluation, as well as his initial recommendations to the desk officer of Caritas Germany. The Draft Report should be presented after the field work has been concluded and should incorporate comments supplied by Caritas Germany and its partner organizations.

Final Report:

The final report serves to illustrate the relevant evidence corresponding with the evaluation issues, questions and criteria listed in the Terms of Reference. The final report should provide an executive summary in English. The Final report should take all aspects reviewed during evaluation workshops or meetings into consideration and is subject to approval by Caritas Da Nang and Caritas Germany.

The proposed report structure is outlined in Annex 2; its length should not exceed 28 pages (excluding appendices). All documents and tools are to be written in English language.

8. Timeframe for Deliverables

The actual evaluation work is to be carried out from 8 April 2024 to 19 April 2024, resulting in a total of 10 days.

The timeline for the activities consists of the following phases:

Preparation Phase: 2 days

• Analysis of relevant project documents, as well as further research
• Preparation of inception report and exchange with Caritas Da Nang and Caritas Germany
• Kick-off Workshop involving Caritas Da Nang and Caritas Germany (the latter virtually via Zoom)

Field Phase: 5 days

• Briefing with Caritas Da Nang, Caritas Germany and other relevant actors in Da Nang Diocese (Da Nang City and Quang Nam Province)
• Field visits
• Validation workshop involving Caritas Da Nang and Caritas Germany (the latter virtually via Zoom)

Synthesis Phase: 3 days

• Reporting including management response workshop
• Draft evaluation report
• Produce final report including revision as per feedback

The key products expected for the evaluation are the following:

Deliverable / Dates:

Inception Report by 28 March 2024

Draft Report by 3 May 2024

Final Report by 17 May 2024

The evaluator should present a detailed work plan and timeframe of all activities including the relevant resource allocation (see Annex 3).

9. Roles and Responsibilities

Caritas Da Nang is responsible for organizing and facilitating the logistics to and in Da Nang diocese. Caritas Da Nang and Caritas Germany will provide access to all relevant project documents.

The consultant will be working under and reporting to the Caritas Germany desk officer and the Caritas Da Nang partner organization during the evaluation phase.

10. Guiding Principles & Donor concepts

The consultant is to conduct the evaluation in accordance with the principles outlined in the “Caritas Internationalis management standards” document[2], the “Caritas Code of conduct”,[3] as well as the “Guidelines on Combating Fraud and Corruption in the Project Work of Caritas Germany”[4]. (See Annex 5)

The evaluator must take all required steps to ensure that the evaluation is designed and conducted to respect and protect the rights and welfare of the people and the communities of which they are members, as well as to ensure that the evaluation is technically accurate, reliable, and legitimate, and conducted in a transparent and impartial manner. Moreover, the evaluation should ideally contribute to organizational learning and accountability.

11. Professional Qualification and Experience

The evaluation consultant should meet with the following, required skills and competence profile:

• Experiences of evaluation regarding empirical social research, evaluation of programs in international cooperation, application of OECD-DAC criteria for evaluation, managing participatory evaluation process.
• In-depth knowledge and experience in social inclusion aspects and related activities; if possible, also in technical and economic / ecological aspects.
• Ability to provide strategic and practical recommendation to key stakeholders.
• Use of methods in empirical social research (qualitative and quantitative methods, data collection and data analysis).
• Strong analytical skills and ability to clearly synthesize and present findings, draw practical conclusions, make recommendations and prepare well-written reports in a timely manner.
• Expertise in using participatory tools.
• Familiarity with transparency and downward accountability processes and their values.
• Fluency in written and spoken English, ideally in conjunction with good skills in spoken Vietnamese.
• Country and regional experience in Asia, Vietnam / Da Nang.
• Good understanding and appropriate sensitivity with regard to different cultures and traditions.

12. Terms of Payment

The evaluator is to receive payment from Caritas Germany in 3 instalments:

• 1st Payment: 30 % upon signing of the contract*;*
• 2nd Payment: 30 % upon the submission of the first draft of the evaluation report;
• Final Payment: 40 % upon the submission of the final evaluation report, subject to its acceptance by Caritas

Caritas Germany will not settle any payment unless the consultancy institution accomplished all the tasks in a timely fashion. The basis for payment scheduling is to be determined during contract negotiations.

Annexes:

• Annex 1: Table of content for the Inception Report
• Annex 2: Table of content for Evaluation Report
• Annex 3: Template for Work plan – Timeframe
• Annex 4: Evaluation Matrix Template
• Annex 5: Financial Proposal
• Annex 6: Caritas Internationalis and Caritas Germany Ethical Guidelines and Code of Conduct.

Homepage: https://www.caritas-international.de/

Full ToRs and all Annexes (bottom of page):

https://www.caritas-international.de/ueberuns/jobs/jobs.aspx