Discussions Discussions

«Back

Any good baseline survey report to share?

Display Date: 7/19/11

Baseline survey is one of the most important components of M&E system in IFAD projects. In some countries, such as China and Philippine, the baseline survey has two parts: 1) the project baseline which sets the thredhold of a project, and covers the indicators to be measured by project outcome and output; 2) RIMs survey.

You may agree that data collection is very labour intensive. So as the data entry and cleaning. Therefore it is very important that the data can be fully used and analyzed, to make maximum output.

I am wondering whether HQ or any project has a good baselinse survey report to share? So we will learn from you and improve the reporting capacity of our projects.

Thanks to you all!

By the way, I donot have baseline report to share with you, but we just finished a 1328 HH survey report on food security . It was well edited. If you are interested in, pls let me know.

Next
Comments
Hi everyone. I am Edward Mallorie, a consultant who is providing back-up support to monitoring and evaluation in Asia Pacific Region.

Weijing is absolutely right when she says that we need to make the greatest possible use of survey data. What concerns me is that, too often, baseline surveys provide little in the way of useful information. I recall a baseline survey of a project in Bangladesh (closed some years ago) where a baseline survey was carried out shortly after the project started. This was a huge task, and the report ran to 62 volumes (with data from 20,000 households in 450 villages). It was quite useless in terms of providing baseline information on indicators against which the results of project interventions can be measured. This was partly because it was carried out before members of project groups were recruited, and it therefore could tell us nothing about the pre-project situation for those households who actually participated in the project. In addition the vast amount of data collected defied attempts at analysis.

In order to make the greatest use of data collected, surveys need to provide, in addition to indicators of project impact, information on more immediate project outcomes. Such information can be useful for project management decision-making and lesson-learning. For example, has a new variety resulted in an increase in crop yield? However to show this clearly, it could be better to have separate data from a sample of crop producers, rather than a general survey of all project participants if this includes people who were not crop producers, or who did not get project support for crop production. This suggests that, to get useful information on project performance, it may well better not to do general surveys of a whole project, but rather look at groups of participants who are getting a specific package of support, or are at a specific stage of development – alternatively one can plan an overall survey in a way that will generate useful information for specific sub-groups.

To make such measurements of project outcomes and performance, we need information on the selected indicators in the pre-project situation - which means some sort of baseline data is needed. This can come from initial surveys carried out at the time that project groups are formed, or from some other form of data gathering such as profiles of each household completed as they join the group. The all-purpose baseline survey, which tries to gather data on all outcome and impact indicators from a sample of all project households, may not be so useful.

I would be interested to hear of the experiences that people have had in carrying out surveys. What has worked well? Where are the problems? What useful things have been learned?

Posted on 7/19/11 5:32 PM.

Top
Hi Edward, Thanks for these very interesting remarks in reply to Weijing's query. In anticipation of comments from others who have had experience in carrying out baseline surveys, could you tell us about "annual outcome surveys" that many projects are now implementing and how they square with the baselines and the issues that you have raised above.

Posted on 7/22/11 9:33 AM in reply to Edward Mallorie.

Top
To respond to the questions raised by Cleona, the idea behind annual outcome surveys is to regularly collect and analyse outcome and impact information from households participating in a project. The IFAD standard methodology for impact measurement, the RIMS (Results and Impact Monitoring System) anchor indicator surveys provide information on indicators of poverty (and hence show impact of projects), they do not cover more indicators of more immediate results or outcomes – as I described in my earlier posting. In addition these surveys are only done two or three times during the life of a project (at start-up, possibly at mid-term and at completion).

Annual outcome surveys would enable projects to: (a) regularly measure changes and outcomes taking place at the household level; (b) provide early evidence of project success or failure; and (c) check if project participants are being correctly targeted. This information should enable project management to: (a) take corrective actions if required; (b) report on initial results in periodic project progress reports and other forums; and (c) assist in completion of level 2 indicators in annual RIMS table reports to IFAD.

IFAD has proposed that annual outcome surveys cover a sample project participant households selected at random from all project participants, and a sample of non-participating households. A sample size of 200 has been suggested – although this should be 200 for each of the project and non-participating households.

My suggestion in my previous posting was that these surveys would be even more useful if samples were selected from sub-groups of project participants who might be from different target groups, or who were getting different packages of support from the project. In this way the performance of specific project activities and outputs can be assessed. However this means covering sufficient number of households in each sub-group to get a reasonably accurate and reliable result – which may be around 200 – so if there were four sub-groups, the total sample would be 800. This will involve more work than a simple sample of 200 drawn from all project households.

The best way forward may be to start with a basic survey of all project households. If it is felt that it would be useful to obtain information on specific interventions or sub-groups, then in following years, surveys could cover specific sub-groups. Such surveys carried out at the start of a project will collect data prior to project interventions taking place, and hence will act as a baseline.

Posted on 7/24/11 3:39 PM in reply to Cleona Wallace.

Top
Thanks very much to Edward for your advice. Very insightful. The sub-group sample is useful to provide project-specif information, but be suitable in annual outcome survey/middle term/completion survey. In the baseline, it could be difficult as sub-groups are not identified yet.

In China, we do the baseline with the whole population sampling ending up with 900 HHs as it is carried out together RIMS survey, while in middle term and completion, we will visit same households, and add one question to distinguish the beneficiary and non-beneficairy. In the baseline, essentially the social-economic data will be collected and not so much relevant to project.

In MT and completion, more project specific questions will be asked.

The survey method in China is trying to fill in the gap of RIMs and provide as much project info. as possible. However, it has at least two pitfalls:

1) some sub-group in MT and completion could be too small to be statistically robust
2) some non-beneficiary could be doing better than the beneficiary because of other interventions from gov. or other agencies, which we have difficulty in telling

For the first proble, I think we can still compare a subgroup (say, rural finance) with other groups in terms of key indicators (e.g. income/wealth), but not able to compare within subgroup ( e.g. how much % of participant is satisfied with the project intervention).

For the second one, we have to leave it there.

thanks to you all!

Posted on 7/25/11 5:09 AM in reply to Edward Mallorie.

Top
Dear Edward, it is really useful remark for the Baseline Survey, I think the survey would be fine for two times one in the starting of the project like Baseline Survey and second would be at the end of the project like Post Survey then the project staff would be able to do impact evaluation. In the mid term is really difficult to measure the impact at the middle of the project.

Does Annual Outcome Monitoring and Annual Outcome Survey has difference or same? If the Annual Outcome Survey are conducting for the above three purpose, may we will do just monitoring and get the result as we want in the AOS because in monitoring we can also achieve; (a) regularly measure changes and outcomes taking place at the household level; (b) provide early evidence of project success or failure; and (c) check if project participants are being correctly targeted.

Could you please more explain the sample size for example we have 100 members beneficiaries, how many number we have to select as a sample? what do you think about the stratified sampling or random sampling which one would be good? If they arranging focus group interview for the survey, how many number have to participate in the focus group? So it mean that when they are doing subgroup sampling it should be cluster sampling because its divided on sub categories.

For the baseline survey they have select control group because it help them in the impact evaluation otherwise it would be difficult at the end of the project to measure the impact. In the baseline survey she has to keep in mind that she has to survey those groups which she has interviewed during the baseline survey. The control group would be 80-85% similar with treatment group.

Dear Wand i already posted one baseline survey report in the website, if you are unable to find please send me one test email then i will share you the report.

Posted on 7/25/11 5:18 AM in reply to Edward Mallorie.

Top
Weijing – thanks for your reply and for raising these interesting points.

Potential problems with carrying out a baseline survey of the whole population and then later identifying people who join the project are: (i) only a small proportion of the population covered in the baseline survey may join the project, and so we end up with too small a sample for project participants (or conversely almost everyone joins so we have too small a control group); and (ii) due to targeting or the support offered by the project, there are underlying differences between project and non-project households (such as land ownership), which means the control group is not a good basis for measuring changes in the project group.

Any household, either project or non-project, may be affected by other programmes. Unless other programmes specifically avoid project households, we have to take this as part of the underlying environment in which the project is operating. The sample should be of sufficient size so such variations between individual households do not distort the survey results.

In order to be able to compare data from different sub-groups, we need a large enough sample from each sub-group. Depending on sample design, the required precision and probability level, and the variability of the population, a sample of around 150 to 200 households for each sub-group should suffice for most socio-economic investigations. The standard sample size for a RIMS anchor indicator survey is 900, and it could be possible to compare results for different sub-groups within this survey. I do not know the basis for calculating the sample size for RIMS surveys, but it may be in order to have extra precision to identify small changes in anthropometric measurements of child malnutrition.

Posted on 8/2/11 5:53 AM in reply to Wang Weijing.

Top
Gulla

Thanks for your response to my ideas on baseline surveys.

Annual Outcome Monitoring and Annual Outcome Surveys are the same thing. I think if projects can carry out such surveys they may be more useful than baseline, mid-term and impact surveys. Although one can argue that baseline, mid-term and impact surveys can collect more comprehensive data, more frequent collection of data on project results will be more useful for project management. Agricultural programmes are often affected by year-to-year variations in weather and market prices, so there is a real risk that one of the three surveys at baseline, mid-term and impact will take place in an “abnormal” year, and so not provide a good basis for comparison of the the results of other surveys.

If you only have a small number of beneficiaries in your project (you mention 100), the sample size can be adjusted downwards by a “finite population correction” (FPC) factor. If you calculate the required sample based on simple random sampling (SRS), and this sample turns out to be over 10% of the population, then it is worth estimating a reduction in sample size through the FPC. The FPC is calculated as:
1 _
1+((n-1)/N)

where: n = the initial estimate of sample size
N = the population size

For instance if the proposed SRS sample size is 67, but the total population is only 100, then the FPC will be:

1 = 0.602
1+((67-1)/100)

and the adjusted sample size will be: 67 * 0.602 = 40

This is before any further adjustment is made for clustering of the sample. In most surveys for rural development projects we use a clustered sample to avoid travelling to individual households all over the project area. This means selecting the sample in two stages – the first being a random selection of clusters such as a village or project groups, and then a second stage random sample of individual households in the selected cluster. Although this makes it easier to carry out the interviews as we only need to go to a limited number of clusters, we do need to interview a larger total sample to overcome what is known as the “cluster” or “design” effect.

I think I have written enough for now – but will post another note on focus group interviews and sample stratification.

Posted on 8/2/11 5:56 AM in reply to Gulla Jan Ahmadzai.

Top
Gulla

I said I would give you some information on the use of stratification in sample surveys.

To improve the accuracy of the results of the sample survey, a sample may chosen in such as way as to reflect variations (“strata”) in the population. For example, if 25% of villages are remote from a road, and 75 are close to a road, then, out of a sample of 22 villages (clusters), 25% (5) could be chosen from remote villages and the rest from accessible villages. This ensures that the correct proportions of the sample are chosen from remote and accessible villages so the final result is more accurate. However with only 5 villages in the remote group, the sample is likely to be too small to compare with those in accessible villages. To make such a comparison, a sample of about 22 villages should be selected from both remote and accessible villages (these becoming “domains of investigation”), and the combined results from both samples weighted in the ratio 25:75 to give an accurate picture of the overall area.

This is an example of stratification in a 2 stage cluster survey, so a number of households – say 4 to 6 – would be chosen at random in each of the selected villages. However stratification can also be applied to simple random samples where clusters are not used.

I hope this is useful.

Posted on 8/7/11 3:17 PM in reply to Gulla Jan Ahmadzai.

Top
dear Edward,

as we know that baseline survey is done before doing anything or any intervention for the identification of socio-economic situation of a region, where intervention intend to be implement , so it is important to select just primary stakeholder or direct beneficiaries or we can also select indirect beneficiaries for baseline information achievement .

thanks

Posted on 8/12/11 6:59 AM in reply to Edward Mallorie.

Top
Eidullah,

Baseline surveys can have two functions. One is, as you say, to identify the baseline socio-economic situation in a project area, assess the causes and consequences of poverty, and suggest appropriate project interventions. The other function is to measure key indicators in the target population to provide a baseline against which the impact of the project can be measured.
In some projects, baseline surveys are carried out prior to the start of project implementation and before any project groups have been formed. It can be difficult for such a survey to provide a basis for assessing benefits for households participating in the project unless: (i) the project is going to benefit the entire population of the project area; or (ii) a sufficiently large sample is covered by the survey so that, once the project starts, sample households who end up participating in the project group can be identified and used as the baseline for future impact assessment.

Some projects do aim to benefit the entire population of a project area, and in this case it may well be appropriate to carry out a baseline survey before the start of project activities – bearing in mind the risk that if activities are delayed, then there could be a change in the baseline situation before the start of the project. Infrastructure development (roads, markets etc.) is an example of such a project.

However most projects have the objective of directly benefiting a specific group of households (members of project groups or recipients of project services), and baseline data needs to be collected from this group. Thus the baseline baseline survey may need to be delayed for some time after project start until a reasonable number of people have been recruited (but before project interventions have made a significant impact). Some (non-IFAD) projects carry out a baseline survey each year as they recruit a new batch of entrants each year. Alternatively a household profile form including appropriate indicators can be completed for each household as they enter the project.

Some projects also aim to generate significant indirect benefits for a wider population – for example a project may provide agricultural support to a specific target group while also developing infrastructure for the wider population. In such a situation a combined approach should be used – with baseline data collected from a sample drawn from the entire population of the area, along with an additional sample from directly participating households.

As I said in my second paragraph, it may be possible to later identify project participant households from amongst the sample households included in a survey carried out prior to the start of a project. However such an approach runs the risk that insufficient households will be identified to form a valid sample for assessment of project benefits.

To conclude, the baseline survey should collect data from households who are directly participating in a project – as these are the primary beneficiaries. For some projects it may also be useful to carry out a separate survey of the wider population if the project aims to this population with significant indirect benefits. It is also worthwhile to consider alternative approaches to the measurement of project benefits. In particular the use of a control group – this is a group who are similar to the project beneficiary group, but who are not benefited by the project, and so can be compared with the beneficiary group in order to measure the impact of the project. Both baseline-impact (“before and after”) and control-project (“with and without”) approaches have their advantages and disadvantages, but if used together can provide strong evidence for the impact of a project.

Posted on 8/14/11 10:59 AM in reply to Eidullah Mustafa Zazai.

Top
Weijings concern is genuine. I could share annual outcome survey reports with her and also baseline survey reports from the India Country Programme. I have seen good monitoring report and survey reports from Vietnam and Phillippines as well.

We need to confine the log frame outcome indicators corresponding to project components to the RIMS plus surveys. This is what we have done in India. As Edward mentioned that we might need to do separte studies to examine a results chain of an intervention. For this what we are trying to do next in the new projects is to do a baseline of an annual outcome survey to not only track outcomes but to capture intermediate outcomes and relevent to interventions that would be cover specific to a project. design The salient distinction would be that in the past we used the annual outcome survey with indicators almost cross cutting all project designs with outcome indicators selected from IFAD strategic framework, but now we would make the annual outcome survey fit the project design with intermediate indicators and the main interventions with respect to the project in question and do a baseline. Why we are doing this is because from our experience a common annual outcome survey questionnaire may not be suitable to all projects for example, in the Post Tsunami Sustainable Livelihoods Programme (PTSLP) we had to redesign the Annual Outcome questionnaire with respect to the project component, as the project had unique set of components.

The other aspect that I would like to empahsis is that the baseline survey is predominantly used for impact evaluation and as Edward mentioned to also see how the results chain emerges from the use of inputs to the creation of outputs then these leading to intermediate outcomes and finally contributing to the project impact. However, baseline surveys have been also seen by a large number of projects as a project planning tool..I am not sure if this is the right perspective.

Posted on 9/4/11 7:01 AM in reply to Cleona Wallace.

Top
Weijing, we might need to do separate sub-groups studies and the baseline of the annual outcome surveys could be a basis for comparision or else we use the year in which such sub-groups will start a particular activity. For example, from the planning process we could identify which cluster of farmers would start a specific activity, in India it we would have milk producing cluster, poultry cluster, etc and then do a baseline of that before the intervention begins. This is one possibility. The other is we use information that is available for a set of indicators from the project baseline, which is also a reason why we need sound project level baseline to make them more useful. Perhaps specific short impact assessment surveys would be more useful in combiantions with KAP surveys. These shouldn't be too time consuming, because in a typical IFAD projects we don't have nowadays two to three components and a few sub-components to implement, mostly credit, insurance, marketing, production of food and cash crop, livestock and infrastructure and perhaps one or two non-farm sector activities. In my view confining to one or to critical intermediate outcome indicators to say for example a value chain would be helpful. I must admit,we don't have much experience in this yet, but this is still a possibility. I will share our experience once the newly designed annual outcome surveys are up and running.

Posted on 9/4/11 7:12 AM in reply to Wang Weijing.

Top

Image Gallery