At a basic level all IFAD-supported projects monitor and report on the progress of project activities and outputs – such as how many people have been trained or how many hectares of land are covered by new irrigation systems. Most projects also report on impact, with at least data on the RIMS anchor indicators of child malnutrition, food security and assets being collected from a sample of project participants at baseline, mid-term and completion. These impact indicators aim to provide evidence of achievement of the goal of poverty reduction. Some projects also collect data on the intermediate outcome indicators – the immediate results of project outputs such as adoption of new technologies, improved irrigation water supply or increased crop production.
In the development community – or at least those involved in monitoring and evaluation - there has been a demand for a more rigorous approach to the evaluation of the effectiveness of international development assistance. The International Initiative on Impact Evaluation (http://www.3ieimpact.org) and others have argued that more convincing evidence is needed to attribute development outcomes to project interventions.
Randomised Control Trials (RCT) have emerged as the “gold standard” for measuring the results of a development intervention. This approach, which is also used in trials to test new drugs, involves dividing members of a target population at random into two groups, one of which is given the development intervention and one of which is not. Indicators for the two groups are measured and the difference between the two groups at the end of the trial is the measured impact of the intervention.
Although RCT can produce useful insights into what works and what does not work, it does not seem to be a tool that can be used to measure the impact of a development project. Project participants are not chosen at random from a target population – they may volunteer to join or projects may recruit people they think best fit the target group or are locations that will benefit most from project interventions. Impact surveys for such projects can still use people who have not participated in the project as a comparison or “control group”, but this group, as it was not chosen at random from the same population, will not be identical to those participating in the project. In other words, people who do not participate in the project are not participating because they do not wish to, or may not fit the profile of the target group as well, or they may live in a location which has different characteristics. However it is still possible to use such a control group in what is known as a “semi-experimental design” for an impact survey.
The idea behind such a semi-experimental design is to compare changes that take place in the project group with changes in as good a comparison group as is practically possible. There are various statistical techniques that can help in selecting a control group that is a near-match to the project group, such as Propensity Score Matching, but a practical and straight forward approach is to select as group that fits the profile of the target group as closely as possible (such as being small farmers from nearby villages) and then using the “difference of difference” technique to measure project impact – this being the difference between the two groups in change in indicators from pre-project to post-project dates.
Such an approach requires pre-project (baseline) surveys for both project and control groups, which are often not done (or otherwise difficult to do). But we can still produce evidence of project impact without data on the “difference of difference”. One way is measure changes in the project group though baseline and impact (post-project) data, and then collect some more qualitative information from a control group for comparison. For example, baseline and impact surveys may show an increase in yield of an average of 25% for 90% of project group farmers, while only 15% of farmers in a control group reported having any significant increase in yield (note that, without a control group baseline, we are not trying to measure the size of the yield increase for the control group).
It is also possible to produce better evidence of project impact by using a “results chain”. A project that provided training in agriculture to 90% of its participants, and then reported that the proportion of underweight children fell from 40% to 25%, has not produced much of a case to show that training resulted in better child nutrition. We do not know if the training was effective, and some other factor may have improved child malnutrition. However if the project can produce evidence of a results chain, this can make the case that the training did result in reduced child malnutrition. For example, evidence can be collected to show that the training actually provided farmers with information that they did not obtain from other sources, and this resulted in adoption of new technologies, that in turn increased crop production, and this extra production was either consumed at home, or used to buy food, that meant that periods of food shortage were reduced and diet was improved, then we can claim that the training did have an impact on child malnutrition.
If projects can generate convincing evidence of their results and impact, not only will people, including those outside of IFAD and its implementation partner, be persuaded that the project was successful, but useful lessons will be learned for the planning of future interventions. To produce this evidence of results, projects need the capacity and resources to collect, analyse and interpret data that links project implementation to outcomes and impact. I wonder if projects feel that they have the resources they need, and should the design of new projects provide additional resources?