Consultancy: Project Mid-Term Evaluation

Terms of Reference for Mid-term Project Evaluation

Project Title: KAMEH (Knowledge Agriculture Mutuality Exchange for Hope)

Country: Syria; Rural villages in Hama, Tartous and Lattakia governorates

Duration of project and scope of evaluation:

Total project duration: 01.06.2023 – 31.05.2026
Period to be covered by the Mid-term Evaluation: 01.06.2023 – 31.05.2025

Partner Organisations:

• Caritas Austria (CAUT) – contract holder and consortium lead
• Caritas Syria (CS) – implementing partner
• Greek Orthodox Patriarchate of Antioch and all the East – Department for Ecumenical Relations and Development (GOPA-DERD) – implementing partner

1. Introduction/ Background

Under the EuropeAid/ DG MENA-funded KAMEH project, CAUT, CS and GOPA-DERD contribute to increasing the resilience of vulnerable rural communities affected by the protracted crisis, food insecurity and climate change (overall project objective).

Thereby, activities will target farmers in six sub-districts in three governorates: Qalaat al-Madiq in Hama, Bahlouhiyah, Kfariyah and Aramo in Latakia, and Safita/ Mashta al-Helu and al-Qadmous in Tartous. By providing agricultural inputs, such as seeds, fertiliser, tools, etc. (excluding fuel) for smallholder farmers, KAMEH will address access to and availability problems of food items. Additionally, practical technical trainings as part of so-called Farmer Field Schools (FFS) will present practices to farmers that reduce the cost of farming and improve the quality and quantity of crop yields, which will be reflected in their market values. FFS will also include technical business trainings to improve farm management through bookkeeping and better access to markets. By setting up food processing businesses, vulnerable community members without access to land will receive access to income generating activities. Furthermore, cooperation within farmer collectives will facilitate the sharing of resources (and knowledge) and cooperation among beneficiaries.

The present tender aims to select an experienced external consultancy team for supporting the project manager and implementing partners with an external mid-term evaluation.

The consultant(s) will work in close co-operation with the KAMEH consortium (Caritas Austria and implementing partners), and will be contracted by Caritas Austria, Storchengasse 1, A-1160 Vienna. They will be expected to use participatory means to engage with project participants and other stakeholders as they carry out the evaluation. The final report submitted by the consultant(s) will be presented to the project implementing partners, Caritas Austria, as well as to the donor, EuropeAid/ DG NEAR.

Caritas Austria wishes to conduct the mid-term evaluation in order to determine and document to what extent the project’s theory of change is holding true during implementation and which mid-course corrections, if any, may be required in order to better serve its participants and have a transformational impact on their lives. In doing so, the evaluation should apply and assess the project using OECD/DAC criteria[1].

Note: The donor is conducting an external ROM (Results Oriented Monitoring) Evaluation of the project in May 2025. These Terms of Reference may be adapted to build on the ROM evaluation results. The mid-term evaluation will start after receipt of the ROM evaluation report, in June 2025.

KAMEH Project Background and Overview:

The KAMEH project is being implemented in villages in the mountainous regions of Lattakia and Tartous, where farmers grow olive and fruit trees, etc. and herd cows. In addition, the project targets villages in the fertile Sahel el-Gharb plain in northern Hama, where farmers practice irrigation-fed agriculture of vegetables and wheats. The KAMEH project started in June 2023 and will run until the end of May 2026. With the help of a holistic approach, that encompasses practical trainings for smallholder farmers focusing on conservation agriculture, distributing agricultural inputs to individual farmers and to farmer collectives, rehabilitating small irrigation works, creating food processing businesses, which provide work opportunities for vulnerable community members, and by implementing business trainings, CAUT and partners improve the living situation of rural communities. Whereas trainings and the distribution of inputs have already been started – and were completed for some project participants – the implementation of other activities (rehabilitations, creating food process businesses) is still in the preparation phase.

Specifically, the overall objective and project outputs are as follows:

Overall Objective: Contribute to increasing the resilience of vulnerable rural communities affected by the protracted crisis, food insecurity and climate change.

Output 1 – Social capital developed:

This output subsumes activities that focus on the cooperation within communities. Activities include creating farmer collectives, which receive/ are responsible for managing agricultural machines. Each collective consists of around 25 farmers and is being led by a so-called resource farmer, who received a training to conduct farmer field schools. In total, CAUT and partners created 56 farmer collectives.

Output 2 – Targeted smallholder farmers have increased access to productive inputs, tools, equipment and infrastructure:

All activities that include the procurement and distribution of agricultural inputs are subsumed under this output. Inputs include machines, tools, irrigation equipment, seeds, seedlings, water tanks, fertilizer and pest control measure, etc. for farmers and home/ kitchen garden owners. Farmer collectives receive larger machines to use jointly as a group. In addition, an activity focusing on the rehabilitation of irrigation systems is part of this output. In total, around 1400 individual smallholder farmers and around 145 home/ kitchen garden owners receive inputs.

Output 3 – Capacities and knowledge of smallholder farmers, home/ kitchen garden beneficiaries, food processing entrepreneurs and staff of implementing partners on climate-smart and agro-ecological farming are developed:

All activities, which focus on building capacities of farmers and home / kitchen garden owners, are part of this output. Resource farmers who go through a ToT deliver practical training sessions on their farms (Farmer Field Schools). In addition, project participants receive business trainings and trainings focusing on data collection. The same farmers, who receive agricultural inputs, also receive trainings.

Output 4 – Food processing businesses are upgraded or newly established in each target location:

This output subsumes activities that focus on creating new food processing businesses, led by local communities. The businesses process further primary goods, which are produced in the targeted rural villages, such as cheeses, yoghurts, pickles and olive oil. Activities under this output have not started yet.

Output 5 – Up-to date information and data available:

Activities under this output focus on the creation of useful tools for farmers to collect data on their farms that will help them improve farm management. In addition, MEAL activities and knowledge sharing activities are part of this output.

2. Objective of the mid-term evaluation

1. The main objective of the evaluations is to assess the theory of change promoted and followed by the project. Thereby, the evaluation should engage with questions related to the relevance, efficiency and effectiveness of the project as well as its coherence. The results of the mid-term evaluation should serve Caritas Austria’s and implementing partners as a management tool. The evaluation should help to deepen insights into assumptions and approaches underpinning the project, focusing on training methods, types of distributed agricultural inputs, and promoted agricultural approaches (conservation agriculture).
2. The evaluation should also provide insights in how far the project addresses crosscutting issues, including gender sensitivity, climate resilience, conflict sensitivity and social inclusion.

3. Specific mid-term evaluation questions:

Relevance:

• Assess the relevance of the project, including against the backdrop of the recently changed political situation in Syria.
• To what extent are project design and approach appropriate to the local situation and needs of target communities? Are they appropriate in regards to the market situation (and current changes to the market)?
• Are training activities designed in the best way so as to meet the learning needs of farmers?
• Do delivered agricultural inputs meet the needs of targeted farmers (tools, machines, seeds, seedlings, fertiliser, pest control measures, and veterinary services) and are appropriate for local use and maintenance?
• Are the planned outputs of the project logically designed and able to contribute to the achievement of the outcome and the overall goal (impact) – as part of the analysis of the log frame / project theory of change?
• Are the assumptions holding true and the risks mitigated?
• Did beneficiaries consider the selection criteria (farm size, etc.) fair and transparent and understand the selection process? Are/ were any groups missed out in their opinion?
• To what extent are the interventions appropriate: culturally, social-economically, gender sensitively and environmentally?

Coherence:

• To what extent are project design and approach appropriate in regards to the priorities of the new authorities on the local and national level (looking at the Humanitarian Affairs Coordination, Ministry of Agriculture and Ministry of Local Affairs, Directorates of Agriculture, etc.)?
• To what extent are project design and approach appropriate in regards to the strategies of Caritas Austria, implementing partners and the donor community, particularly the EU and EU member states?
• To what extent are project design and approach adhering to Sphere standards, Core Humanitarian Standards, Protection Mainstreaming Standards and the Do No Harm Principle?
• How did Caritas Austria and partners harmonise and coordinate their interventions with public stakeholders (relevant line ministries and directorates before and after December 8, 2024) and other international and national NGOs? How was duplication of assistance avoided?

Efficiency:

• Assess the efficiency of the project, looking at financial efficiency, management of the project (by both Caritas Austria and partners) and the methodology of activities.
• To what extent did the delivery of trainings and other meetings, but also the distribution of inputs, ensure access, safety and dignity?
• To what extent did the project provide services and agricultural inputs in such a way to offer the highest value for money?
• Is the training methodology (farmer field schools) suitable for the targeted farmers?
• Is the service delivery modality (in-kind distribution of agricultural inputs) suitable in the context, in which targeted communities live in?
• Is the project being implemented in the most efficient way, financially and in terms of resource allocation, but also in terms of time allocation, planning, coordination, and communication between the three consortiums partner?
• Are constraints and risks regularly identified and analysed, and plans adapted accordingly?
• Are the budget and available financial resources realistically planned for the achievement of the intended objectives and outputs (including staffing, trainings for capacity building)?
• Are there clear processes in place to support monitoring and use of the monitoring-results for management and decision-making?

Effectiveness:

• Assess the extent to which the project is effective/ was effective in delivering the planned outputs and outcomes.
• To what extent has the project already produced its expected outputs or will be likely to achieve them?
• Are, and if yes, what are the targeted farmers learning in training activities? Can farmers be observed applying learning outcomes on their farms? Do they face any barriers to applying them?
• Are farmers using the provided agricultural inputs (tools, machines, seeds, seedlings, fertiliser, pest control measures, veterinary services)? Do they face any barriers to using them?
• Can the effects of provided trainings and agricultural inputs, e.g. through increased farm outputs or increased income of farmers, already be seen?
• To what extent did project participants assume ownership of the project’s activities?
• What have been the key challenges and/or risks hampering implementation of the project? How have these been addressed by the programme management?
• Are there any observable indications that actions from farmers are as environmentally friendly as possible, e.g. related to using water efficiently?

Environmental Sustainability:

• How are local farmers managing water availability for irrigation, and what training or support has been provided to ensure sustainable water management?
• How are farmers being educated on the proper use of fertilizers to minimize negative environmental impacts such as soil acidification or nutrient runoff into water systems?
• To what extent has the project promoted agro-ecological and climate smart practices?
• How has the project addressed the use of pesticides and other agrochemicals, and to what extent have more sustainable alternatives been promoted (e.g., integrated pest management, bio pesticides)?
• What is the level of awareness and understanding among farmers regarding the environmental and health risks of excessive pesticide use, and how are they being educated on safer alternatives?
• How does the project monitor and evaluate its environmental impact, including water usage, fertilizer application, and biodiversity health?
• How effective are these monitoring systems in identifying any unintended negative environmental consequences of the project?

The consultant(s) are not required to assess impact and sustainability of the project. However, if there are early indications of such, they may be documented in the report.

4. Methodology:

It is expected that the project review will employ both quantitative as well as qualitative methods, making use of different methods, such as analysis of documents, farmer records, structured interviews, focus group discussions, semi-structured interviews (face-to face or remotely), and observations during field visits, etc. Data analysis should disaggregate collected data collected by sex of beneficiaries, be sensitive to gender disparities, seek to include persons with disabilities, and be prepared to probe into unintended project outcomes.

The methodology applied in the proposal should include but not be limited to:

• Review of key project documents including proposal, annual plans, log frames, risk matrices, project reports, PDMs, baseline study, consortiums SOPs and other related documents.
• Review of KAMEH process and outcome monitoring data as well as other data collected in the course of the KAMEH intervention (e.g. primary data for outcome and process monitoring, etc.)
• Interviews with key staff associated with the project and other stakeholders (face-to-face or remotely). All interviews should be documented.
• Interviews (KII, FGDs) with project participants (farmers and home/kitchen garden owners). All interviews should be documented.
• Physical verification visits to project sites to affirm level and results exhibited.

Consultant(s) are asked to specify the proposed methodology for the mid-term and final evaluation in their technical proposal (part of the application form). This should include an explanation of how qualitative and quantitative data will be triangulated; how potential biases will be handled and a clarification of the sampling methodology.

Consultants will be asked to coordinate data collection (KII, FGDs) with implementing partners, who will facilitate contacts to relevant stakeholders, project participants, etc. and organise visits to project target locations.

5. Deliverables:

The evaluation should entail the following deliverables:

Inception Report: In the inception report, the consultant(s) shall describe the design of the mid-term evaluation and elaborate on how data will be obtained and analysed. The report should at a minimum:

• Describe the conceptual framework reflecting these ToRs
  • Highlight any reservations regarding the feasibility and/ or limitations
  • Outline the methodology and provide a timeline for the data collection and review
  • Include tools/ templates for data collection
  • Include selection of locations for data collection and sampling methodology
  • Provide an analysis of consulted documentation
  • Present outline of the final report

The project management has the possibility to give feedback on the inception report and proposed design. Data triangulation and quality control should be discussed in the inception report. The inception report needs approval by CAUT before continuing with data collection and analysis. CAUT is aware that the limited timeframe for the evaluation can pose a challenge to building the trust/rapport required to get to in-depth conversations; the evaluator(s) is/are invited to reflect on this in the inception report.

Draft Mid-term Evaluation Report: The report should include a draft executive summary and clear findings and recommendations to be read and commented by all project partners. The findings and recommendations of the draft final report should be structured according to the review questions. To be approved by CAUT.

Final Mid-term Evaluation Report[2]: The report should be clear, well-structured (along the DAC criteria and according to the evaluation questions) and should not exceed 30-35 pages. An outline of the report’s structure needs to be agreed upon during the inception phase, but should include at least:

• Cover page, table of contents, list of abbreviations
  • Reflection on research ethics and limitations
  • Executive summary including recommendations (three to five pages)
  • Description of the review objective(s), methodology and activities
  • Discussion of findings (presented per specific tasks – Relevance, Efficiency, Effectiveness, Impact and Sustainability)
  • Conclusions and Recommendations
  • Annexes (e.g. list of reference documents, list of interviewed persons, minutes of meetings)

Findings and recommendations should be based on a critical analysis. To be approved by CAUT.

Validation workshop: To be organised following data gathering phase with CAUT and implementing partners.

Final Workshop: To be delivered after the final report for CAUT and implementing partners staff.

6. Process and Timeline for the Mid-term Evaluation:

Contracting: 12.05.2025

The exact deadlines for each step will be agreed on upon defining the starting date of the evaluation.

Desk Review: The evaluation team studies all necessary project documents including project proposal, logical framework, risk matrices, reports, existing monitoring (MEAL) data, training materials and produced studies, etc.

Inception Report: The evaluation team uses a data collection planning worksheet or a similar tool based on the agreed methodology. The first interviews take place. The phase will be concluded with the delivery of the inception report: see deliverables. Field trips shall only take place upon official approval of the inception report by the contractor.

Review/ Field phase: Data needs to be gathered, analysed and interpreted. Consultant(s) are expected to make use of existing project data and generate new data as agreed in methodology (e.g. through interviews, FGD, etc.) either through face to face modality or remote modalities adapted accordingly.

Validation Workshop: Presentation of key findings (feedback workshop) at the end of the office visits / field trips.

Draft Final Review Report: Submission and presentation of draft final review report, inclusion of comments from partners and evaluator.

Final Review Report: Submission of final review report.

Final Report Workshop: Presentation of final report findings, key learning points and recommendations.

7. Evaluation Team:

Key qualifications to the review team should be:

• Relevant academic degrees.
• A minimum experience of three years and expertise in conducting evaluations, reviews, business analyses, market assessments, etc. inside Syria.
• Team leader has led or conducted at least three evaluations/reviews in the last five years in the field of agriculture, livelihoods and food security.
• Each team member has conducted at least two evaluations/reviews.
• Proven experience with quantitative and qualitative data collection, analysis and evaluation methods.
• Understanding of local economic and political development context in Syria.
• Knowledge of OECD/DAC criteria.
• Experience in evaluation/review development and/or humanitarian projects, especially in the sphere of agriculture, livelihoods/ food security.
• Ability to conduct meetings with senior government, UN and NGO level personnel.
• Experience and expertise in evaluating cross-cutting issues.
• Excellent oral and written English & Arabic skills.
• Ability to travel in the indicated project locations in Syria.
• Excellent MS Office skills.

The consultants must not have been involved in the design, implementation or monitoring of this project.

8. Responsibilities and Conditions:

Confidentiality of information: All documents provided to the evaluator(s) and all data collected during field research must be treated as confidential and used solely for the purpose of this evaluation. Respondents shall not be identified in reports without their explicit written permission. Photo, video, and other research data that can be traced back to individual research participants shall be anonymized unless agreed otherwise, in writing, with the person concerned.

Ownership of data, findings and products: CAUT retains full ownership of all data, findings, and products produced as part of this assignment.

Support by CAUT and partners: All relevant documents including programme proposal, reports, etc. will be provided for the consultant(s). Field research can be supported and facilitated by CAUT and partners based upon the chosen methodology and approach defined. Relevant contacts of stakeholders will be shared upon the consultant(s)’ request.

Safeguarding and protection: For the time of the assignment, the evaluator commits to safeguarding and protection policies and procedures (including for data protection) that are shared and reiterated during the preparatory phase.

Distribution Policy: Intended users of deliverables are CAUT, project implementing partners and donors.

[1] https://www.oecd.org/en/topics/sub-issues/development-co-operation-evaluation-and-effectiveness/evaluation-criteria.html

[2]The quality of the review reports will be assessed according to the following criteria: Is the results-matrix format part of the report? Were the Terms of Reference fulfilled and is this reflected in the report? Are all evaluation questions answered? Are the methods and processes of the review sufficiently documented in the project review? Are cross-cutting issues analysed in the report? Are the conclusions and recommendations based on findings and are they clearly stated in the report? Does the report clearly differentiate between conclusions, lessons learnt and recommendations? Are the recommendations realistic and is it clearly expressed to whom the recommendations are addressed to? Were the most significant involved stakeholders consulted? Does the report present the information contained in a presentable and clearly arranged form? Can the report be distributed in the delivered form? Does the report take to specific frame conditions of the intervention areas into consideration?

How to apply

Qualified applicants are requested to submit all required documents listed below to:
bewerbung-COE@caritas-austria.at with the subject line “KAMEH mid-term evaluation”

Deadline for the submission of applications: April 27, 2025

The application should include as a minimum:

• Technical proposal within the completed application form (Annex 1), inicluding workplan/ timeplan and proposed methodology, description of experts’ qualifications and previous experience.
• Information about professional background:
  • CVs of all research team members, professional certificates
  • At least two samples of previously completed evaluations in the fields of agriculture/ food security in Syria.
• Signed and stamped declaration of the application form (Annex 1)

Financial offer, including daily fees per expert, envisaged travel costs, material costs if applicable, other costs. Please prepare the offer in EUR.

Within Caritas Austria there is no space for exploitation, abuse, discrimination, or corruption. The successful candidate(s) will be required to sign and adhere to Caritas Austria’s Code of Conduct and Safeguarding Policy. The applicant or any team member must not meet any exclusion criteria (Annex 2).

Annex 1 and 2 can be downloaded under the following link: https://wolke.caritas.at/s/SGWM79PJZ2gLcn8

Password: KAMEHMIDTERMEVALUATION

Selection of offers:

Submissions will be evaluated considering both quality of the offer and ability to deliver the tasks (service specifications) as outlined in the technical offer (70%) as well as considering the financial offer (30%). Applications will be assessed by an independent evaluation committee; the evaluation committee reserves the right to interview shortlisted applicants in a second round of the evaluation process. Incomplete applications will not be considered.

The offers will be assessed based on the following criteria:

• Quality of proposal
• Professional expertise and experience
• Price (financial offer) in EUR.

For questions and inquiries, please contact: majjed.khatteeb@caritas-austria.at