Evaluation has been defined as" the systematic and scientific process determining the extent to which an action or set of actions were successful in the achievement of pre-determined objectives. It involves measurement of adequacy, effectiveness and efficiency of health services. "WHO (1969).
Some important terms used in evaluation are given below:
• Effectiveness - whether or not a programme achieves its stated objectives; that is, did it work?
• Efficiency - the amount of effort in terms of time, staffing, resources and money required to achieve the objectives; was it worth the effort?
• Formative evaluation - monitoring progress during the programme, involving measurement of intermediate objectives; that is, what has been achieved so far?
• Summative evaluation - measurement of impact or change at the end of the programme; have the objectives been achieved?
So evaluation involves showing that:
• Change has taken place.
• The change took place as a result of the programme.
• The amount of effort required to produce the change was worthwhile.
One reason that evaluation is difficult is that it is often only at the end of a programme that thought is given to how the programme should be evaluated. It is important to plan at the beginning how to evaluate the work.
Communication stages and evaluation
By asking the questions listed below, it is possible to find out at which stage a failure in communication took place. The fault can then be corrected and the programme improved.
Questions to ask when evaluating communication activities
Were the communication activities carried out?
How many radio programmes were broadcast; talks or training sessions given; community meetings held; leaflets distributed; posters put up?
Did the intended audience see or hear the message?
How many people saw the posters; were able to receive the radio broadcasts; came to the talks; passed by the exhibition?
Did the intended audience pay attention to the communication?
What was the coverage of the programme - how many people saw the poster; listened to the radio programme; stopped to look at the exhibition; were paying attention at the meeting?
Did the intended audience understand the messages?
How many people could correctly repeat back the messages of the posters, radio programmes, talks, meetings?
Did it convince the audience?
How many people accepted and believed the message?
Did it result in a change in behaviour?
How many people changed their behaviour as a result of the communication?
Did it lead to improvements in health?
How many people's health improved as a result of the programme? What changes in the level of disease incidence or prevalence took place?
It is best to be realistic when selecting changes to be measured in an evaluation. Changes in knowledge, understanding, awareness and belief might take place soon after the communication. Changes in behaviour and health, however, usually take longer. It is a good idea to carry out a short-term evaluation fairly soon after the activity and a follow-up evaluation later to look for long-term changes.
Choice of indicators for evaluation of diarrhoea programmes
• mortality from diarrhoea;
• morbidity from diarrhoea.
• disposal of faeces;
• water handling;
• oral rehydration therapy.
Preconditions for behaviour change
• ability to prepare oral rehydration solution;
• recognition of symptoms of cholera;
• belief that dehydration is dangerous;
• acceptance of importance of oral rehydration therapy;
• exposure to education on cholera;
• knowledge of cholera control measures.
Showing that change took place because of your programme
If the objectives have been clearly defined at the outset it is not usually difficult to show that change has taken place in a community. It is, however, much more difficult to show that it took place because of the programme's intervention and efforts, and not for some other reason. There are two ways of showing that change was caused by the programme's intervention (this is called proving causality).
Another group can be set up, for instance another classroom, group of mothers, or village, as a control group who do not receive the education. The two groups should be as close as possible in age, education and income. If the group that received the educational programme achieves a better performance than the control group, this will provide strong evidence for the success of the communication activities.
Indirect method without controls
If it is not possible to set up a control group, an indirect method will have to be used for excluding other reasons for any changes. One way is to look carefully at what has taken place - could there be any other possible explanation for the changes that took place? Another way is to interview samples of the community and ask them why they changed their behaviour - was it because of the activities or were there other reasons that were unrelated to the activities?
Two approaches to evaluation
Evaluation may be carried out in two ways, as indicated below. Using controls
• Measure baseline at the beginning ("pre-test").
• Test group receives education.
• Control group does not receive education.
• Measure levels at the end of the programme ("post-test") of test group and control group.
Indirect evaluation without a control
• Measure baseline at the beginning.
• Give education.
• Measure levels at the end. Ask questions to find out why people changed and find out if it was because of educational efforts.
Participatory evaluation. Involvement of the community in evaluation helps to create a bond of trust with the community. It gives an opportunity to find out their feelings about the benefits and weaknesses of the activities, and to draw on their experiences and insights on what has happened. Evaluation becomes a learning process, and the community are able to reflect on their experiences and plan future activities.
Summary guidelines for evaluation
• Decide at the beginning of a programme how it is going to be evaluated.
• Prepare a set of realistic, achievable and measurable indicators for success. Consult employers, funding bodies and the community when deciding on objectives.
• Wherever possible, set up control groups who do not receive the education. Make sure that these controls are similar to the test community. If controls are not possible, collect data that will help to show that it was the education that led to improvements.
• Look for changes in the short term as well as the long term. Find out if any benefits are long-lasting.
• Do not limit the evaluation to just finding out if the objectives have been achieved - look out for any unplanned benefits or unexpected problems.
• Look out for ways of involving the target groups in all stages of the evaluation process, including setting objectives, collecting data, judging outcomes and deciding on future activities.
• Learn from failures as well as successes. Find out why programmes succeeded or failed and what lessons can be drawn for the future.
• Share the successes or failures with others. Tell others about what has been done; circulate any reports and look out for newsletters and journals to send articles to for publishing.