Health Programmes Meta Evaluation 2018 - 2020

The programmes were scored according to the DAC and expanded DAC criteria for evaluations of development and humanitarian response. The scoring is on a scale from 0 to 4 as follows:

• 4 - Outstanding: Highly Satisfactory (exceeding expectations in a number of areas)
• 3 - Satisfactory: performance in line with what would be expected of a well-functioning organisation (doing exactly what it should do)
• 2 - Acceptable: generally acceptable performance but with some clear, and documented, shortcomings
• 1 - Minimal: Barely acceptable performance with some major shortcomings and reservations
• 0 - Unacceptable: totally unacceptable performance or insufficient data to make an assessment

Relevance: Of the 28 evaluations included in this review, 26 addressed this issue, and based on the evidence presented in the reports all interventions were felt to fall in to one of the two upper categories (a score of either 4 or 3), meaning that on average Concern’s Health interventions perform slightly above expectation in terms of their relevance. Relevance continues to be the criteria that our programmes score best on, with evaluators highlighting how the interventions meet the need of the community.

Efficiency: Of the 24 evaluations that provided enough data to make an assessment, 2 were considered highly satisfactory, 15 as satisfactory and 7 as satisfactory but with some reservations. Challenges identified not budgeting appropriately and not spending at an appropriate rate. Inefficiencies are connected to lacking staff and high levels of staff turnover. One evaluation highlighted how efficiency gains could have been achieved in terms of “having a framework agreement, or pre-qualified suppliers, for the purchase of emergency response items”. Historically, efficiency is one of the criteria Concern has struggled with the most.

Effectiveness: The reviewers, basing their conclusions on effectiveness on data available at the outcome level found only one programme to be highly satisfactory, 17 to be satisfactory and 7 to be satisfactory, but with some (major) reservations. Issues flagged included targets not being clearly identified in the results framework making it impossible to know whether the progress was in line with expectations. Some evaluations looked at the question of Value for Money (VfM) under effectiveness, in particular the two Community-based Management of Acute Malnutrition (CMAM) Surge evaluations. A Cost-Effectiveness Assessment found that the Surge approach as implemented in Niger was "highly cost effective”. The ability to assess an intervention’s effectiveness has been greatly enhanced by the availability of good quality baseline and endline data.

Impact: Overall, 21 of the 28 evaluations either provided a rating, or it was possible to estimate one based on the data provided in the report, with 17 assessing impact as satisfactory and 4 as satisfactory but with some reservation, suggesting our work overall is satisfactory in this regard. A number of evaluations highlight areas that need more work to achieve a greater overall integration. In Lebanon the evaluators highlight how “It was difficult to evaluate protection and WASH together and provide them with one rating. Attribution of impact to Concern’s work on its own can be challenging. For example, the Kenya evaluation highlighted how “improvement in nutrition situation in Marsabit County as a result of the project interventions alongside other intervening factors such as the rains that were witnessed in the month of April to May 2018”.

Sustainability: Amongst those classified as ‘Development’, 3 were considered to have addressed sustainability in a satisfactory manner, with 6 being assessed as satisfactory, but with shortcomings, with 2 not providing enough information to make the assessment. One area that is receiving greater attention under the health interventions is strengthening capacities at district level, often with government run facilities, and even in the more short-term interventions evaluated. Emergency evaluations in general do not look at issues of sustainability, looking instead at connectedness though 7 of the 17 reviewed did touch on it.

One of the biggest challenges to sustainability identified is a tendency to do it ourselves. This was brought to the fore in the emergency nutrition response in Ethiopia where the evaluators highlighted, “The Concern team is doing direct implementation of CMAM and in July 2018 agreed to manage two stabilisation centres”. A point is raised in some evaluations – that it is very difficult to assess sustainability immediately at the end of a short-term intervention. In some evaluations, such as in the DRC WASH Consortium. The issue of time frames is also raised in terms of the length of time necessary to take on board behaviour change messages in a way that will last. Also, this focus on the short term nature is at times seen to translate into a prioritisation of hardware over software, reported in the Liberia WASH Consortium evaluation.

Coherence and How Concern Understands Extreme Poverty (HCUEP): Coherence has only been assessed in 11 of the 17 emergency evaluations, but HCUEP has been applied particularly strongly in the HPP evaluations. A number of evaluations highlight the lack of a coherent or strong approach to addressing issues of inequality in our health interventions. Concerns raised in the North East Syria evaluation were the lack of attention to issues of gender and disability in programme design. Risk and Vulnerability appears to be better addressed within the Health programming than in the Livelihoods and Education sectors. For example, the Chad / Sudan BRACED evaluation focus on how the activities undertaken have addressed the systemic risks. HIV and AIDS continues to be poorly addressed in health programmes however.

Coverage and Connectedness: Coverage was assessed in 11 of the 17 Emergency evaluations, with one being seen to be highly satisfactory, six satisfactory and four satisfactory with reservations. In South Sudan the evaluators praised the work of the mobile clinics “Overall, the programme has been successful at reaching the most vulnerable". The evaluators considered 7 of the emergency interventions satisfactory in this regard and a further 3 satisfactory but with reservations, with no assessment made for seven. Challenges were identified in Lebanon, with more positive reports from Syria in this regard.

Innovation and Promising Practices: These include CMAM Surge, Adolescent Sexual and Reproductive Health (ASRH) interventions, the Economic Approach introduced by the DRC WASH Consortium, Urban Early Warning Early Action approach as adopted in Kenya and Health Vouchers are being piloted in Bangladesh urban areas.