 |
|
|
|
|
|
Many educational and community programs are good candidates for evaluation with qualitative techniques. In a child development evaluation, ratings by observers were used to measure the quality of and intensity of interactions of mothers with their young children (Grotberg and Badri, 1986, p. 14). The variables included ratings of the mothers' skills in stimulating their children, their patience with their children, and ways mothers interacted with their children. These are hard variables to measure. A survey approach, using a questionnaire, would not have produced the depth and variety of data obtained from first-hand observation of the mothers with their children. Also, data were accumulated over a number of visits. This allowed the observers to check the accuracy of previous observations. Well-documented and presented qualitative data have a compelling, built-in validity, and can be very persuasive. A survey finding that most workers in a factory are dissatisfied with working conditions, for example, could be dismissed by managers as complaints by lazy workers. But, suppose that in addition to the survey results, the evaluator obtained extensive, unstructured responses from the workers showing their dedication to their jobs and dissatisfaction with specific working conditions; then the managers might be more willing to respond to the evaluation results. Frequently, qualitative evaluations of community activities are based on extensive interactions between the evaluator and stakeholders. The evaluator acts more like a consultant to stakeholder groups, serving as a facilitator who helps the stakeholders work through the evaluation process. Consultations are carried out in group meetings and in one-on-one meetings with individual stakeholders. The questions to be answered by the evaluation are established in consultation with stakeholders. Stakeholders help select the indicators and ways of measuring them. The evaluation design is the result of extensive two-way communication between the evaluator and stakeholders. Regardless of the extent of communication and planning, qualitative designs are frequently loose and flexible, sometimes simply based only on agreement for extensive interviews and periods of observation of community events. Data collection is done informally, similar to the processes described for participant observation. The evaluator may observe and record how well he or she thinks some community program is functioning, based on criteria developed cooperatively with stakeholders. Additional data may be extracted from records of group meetings and focus groups. Notes on meetings with individuals or small groups of stakeholders may provide additional data. As in participant observation, the evaluator is an outsider to the group and its activities, but is accorded a special status by the group. This status allows the evaluator to participant in the activities of the community, observe and record information, ask questions, and to build up a body of information on the activity being evaluated. Insights and assessments of the observer are shared on a regular basis with stakeholders, who then provide feedback to the evaluator. Through this process, the evaluator comes to understand the stakeholders' views and is able to test the validity of his or her emerging conclusions. Qualitative evaluation also makes great demands on an evaluator. The evaluator is constantly aware that the conclusions being developed depend on his or her skill and judgment. There is the ever present danger of faulty observation and analysis. Intensive consultations with stakeholders protect against errors in observation and faulty analysis. Consultations, however, require a large amount of time, and can create additional stress. Differences may arise between the evaluator and stakeholders over means of data collection, the meaning of observations, and over the interpretation of findings. Stakeholders may fail to realize the importance of proper methods of collecting data and push for means the evaluator knows are open to bias. Should the evaluator insist on the highest standards for data collection or should something closer to what stakeholders think is sufficient be used? How far should an evaluator go in accepting alternative interpretations by stakeholders? Differences may also arise among sets of stakeholders, putting additional pressure on the evaluator. There is no single best way to resolve differences with stakeholders. Evaluators have to balance their commitment to adhering to the rules of data collection and analysis against the possibility that observations or interpretations may be biased. The best course of action is to seek confirmation of observations and other data by drawing on additional sources of information and to be able to back up every conclusion with solid data. All the ways to triangulate should be explored and used. Qualitative community or organizational evaluations also provide an opportunity for building skills and resources in the group participating in an evaluation. The process of consultation with stakeholders can be used to build group cohesion, teach problem solving skills, introduce elements of research methodology, and to demonstrate the analysis of observations and other data. Communities and organizations can be left far richer and more resilient as a result of participating in a consultative evaluation. This point is developed more fully as part of the discussion of Rapid Rural Assessmen t in Chapter 15. We found one Web site dealing specifically with qualitative evaluation. This site, The Use of Qualitative Interviews in Evaluation, describes various ways qualitative interviewing is used, the advantages and disadvantages of qualitative interviewing, how to conduct qualitative interviews, and how to handle ethical issues that may arise. |