Review: Alternative Approaches Review: Alternative Approaches IIII

What three approaches did we last cover?What three approaches did we last cover?

Describe one benefit of each approachDescribe one benefit of each approach

Which approach focuses on the Which approach focuses on the marginalized?marginalized?

What were the five cautions the authors What were the five cautions the authors shared about the alternative approaches to shared about the alternative approaches to evaluation?evaluation?

Guidelines for Planning Guidelines for Planning Evaluations:Evaluations:

Clarifying the Evaluation Clarifying the Evaluation Request and ResponsibilitiesRequest and Responsibilities

Dr. Suzan Ayers Dr. Suzan Ayers

Western Michigan UniversityWestern Michigan University(courtesy of Dr. Mary Schutten)(courtesy of Dr. Mary Schutten)

Individuals who Affect or are Individuals who Affect or are Affected by an Evaluation StudyAffected by an Evaluation Study

Sponsor:Sponsor: authorizes the evaluation, authorizes the evaluation, provides resources for its conductprovides resources for its conduct

Client: Client: requests the evaluationrequests the evaluation

Stakeholders: Stakeholders: those who have a stake in those who have a stake in the program or in the evaluation’s resultsthe program or in the evaluation’s results

Audiences:Audiences: individuals, groups, agencies individuals, groups, agencies who have an interest in the evaluation who have an interest in the evaluation and receive its resultsand receive its results

Understanding Reasons forUnderstanding Reasons forInitiating EvaluationInitiating Evaluation

Understanding the purpose of the Understanding the purpose of the evaluation is an important first stepevaluation is an important first step– Did a problem prompt the evaluation?Did a problem prompt the evaluation?– Did some stakeholder demand it? Did some stakeholder demand it? – Who has the need to know?Who has the need to know?– What does s/he want to know? Why?What does s/he want to know? Why?– How will s/he use the results?How will s/he use the results?

It is not uncommon for the clients to be It is not uncommon for the clients to be uninformed about evaluation procedures uninformed about evaluation procedures and to have not given deep thought and to have not given deep thought about the ramificationsabout the ramifications

Frequently, the purpose is not clear until Frequently, the purpose is not clear until the evaluator has carefully read the the evaluator has carefully read the relevant materials, observed the relevant materials, observed the evaluation object, and interviewed evaluation object, and interviewed stakeholdersstakeholders

Practical Application to YOUR Plan: Practical Application to YOUR Plan: Questions to BeginQuestions to Begin

1) Why is this evaluation being requested? 1) Why is this evaluation being requested? What questions will it answer?What questions will it answer?

2) To what use will the evaluation findings be 2) To what use will the evaluation findings be put? By whom? What others should receive put? By whom? What others should receive the information?the information?

3) What is to be evaluated? What does it 3) What is to be evaluated? What does it include? Exclude? During what time period? include? Exclude? During what time period? In what settings? Who will participate?In what settings? Who will participate?

4) What are the essential program 4) What are the essential program activities? How do they link with the goals activities? How do they link with the goals and objectives? What is the program and objectives? What is the program theory?theory?

5) How much time and money are available 5) How much time and money are available for the evaluation? Who can help with it? for the evaluation? Who can help with it? Is any information needed immediately?Is any information needed immediately?

6) What is the political climate and context 6) What is the political climate and context surrounding the evaluation? Will any surrounding the evaluation? Will any political factors and forces interfere in political factors and forces interfere in gaining meaningful and fair information?gaining meaningful and fair information?

Informational Uses of Informational Uses of EvaluationEvaluation Needs AssessmentNeeds Assessment

– Determine whether sufficient need exists to initiate a Determine whether sufficient need exists to initiate a program and describe the target audience program and describe the target audience

– Assist in program planning by identifying potential Assist in program planning by identifying potential program models program models

Monitoring/Process StudyMonitoring/Process Study– Describe program implementation and whether Describe program implementation and whether

changes from the initial model have occurredchanges from the initial model have occurred Outcomes StudyOutcomes Study

– Examine whether certain goals are being achieved at Examine whether certain goals are being achieved at desired levelsdesired levels

– from the initial model have occurredfrom the initial model have occurred Cost Effectiveness StudyCost Effectiveness Study

– Judge overall program value & its relative cost:value Judge overall program value & its relative cost:value ratio compared to competing programsratio compared to competing programs

Noninformational UsesNoninformational Uses Postponement of a decisionPostponement of a decision Ducking responsibility [know decision Ducking responsibility [know decision

already but need to make it look good]already but need to make it look good] Public Relations [justify the program]Public Relations [justify the program] Fulfilling grant requirementsFulfilling grant requirements

Covert, nefarious, political uses of Covert, nefarious, political uses of information:information:– Typically more common in federal/national Typically more common in federal/national

evaluationsevaluations

Conditions under which evaluation Conditions under which evaluation studies are inappropriatestudies are inappropriate

Evaluation would produce trivial informationEvaluation would produce trivial information– Low impact program, one-time effortLow impact program, one-time effort

Evaluation results will not be usedEvaluation results will not be used– Regardless of outcome, political appeal/public support…Regardless of outcome, political appeal/public support…

Cannot yield useful, valid information Cannot yield useful, valid information (bad worse than (bad worse than none)none)

– Well-intentioned efforts, “mission impossible” evalsWell-intentioned efforts, “mission impossible” evals Evaluation is premature for the stage of the programEvaluation is premature for the stage of the program

– Fitness program evaluation in first 6 weeks will not yield Fitness program evaluation in first 6 weeks will not yield meaningful informationmeaningful information

– Premature summative evals most insidious misuse of Premature summative evals most insidious misuse of evaluationevaluation

Motives of the evaluation are improperMotives of the evaluation are improper– Ethical considerations, “hatchet jobs” Ethical considerations, “hatchet jobs” (propriety: eval respects (propriety: eval respects

rights & dignity of data sources; help organizations address all clients’ needs)rights & dignity of data sources; help organizations address all clients’ needs)

Determining AppropriatenessDetermining Appropriateness Use a tool called Use a tool called evaluability assessmentevaluability assessment

– Clarify the intended program model or Clarify the intended program model or theorytheory

– Examine the program implementation to Examine the program implementation to determine whether it matches the determine whether it matches the program model and could achieve the program model and could achieve the program goalsprogram goals

– Explore different evaluation approaches Explore different evaluation approaches to match needs of stakeholdersto match needs of stakeholders

– Agree on evaluation priorities and Agree on evaluation priorities and intended uses of the studyintended uses of the study

Methods Methods Create working group to clarify program Create working group to clarify program

model or theory, define information model or theory, define information needs, evaluation expectationsneeds, evaluation expectations– Personal interviews with stakeholdersPersonal interviews with stakeholders– Reviews of existing program documentationReviews of existing program documentation– Site visitsSite visits

Figure 10.1Figure 10.1 (p. 186): checklist to (p. 186): checklist to determine when to conduct an evaluationdetermine when to conduct an evaluation

Who will Evaluate?Who will Evaluate? ExternalExternal

– impartial, credible, expertise, fresh lookimpartial, credible, expertise, fresh look– participants may be more willing to reveal participants may be more willing to reveal

sensitive information to outsiderssensitive information to outsiders– more comfort presenting unpopular more comfort presenting unpopular

information/advocating changes, etc.information/advocating changes, etc. InternalInternal

– Knowledge of program, history, context, etc.Knowledge of program, history, context, etc.– familiarity with stakeholdersfamiliarity with stakeholders– Serve as advocates to use findingsServe as advocates to use findings– quick start upquick start up– Known quantityKnown quantity

Combination Combination – Internal collect contextual informationInternal collect contextual information– Internal collect data Internal collect data – External directs data collection, External directs data collection,

organizes reportorganizes report– Internal is there to advocate and Internal is there to advocate and

support after external is gonesupport after external is gone

Evaluator Qualifications/SkillsEvaluator Qualifications/Skills

Does evaluator have the ability to use Does evaluator have the ability to use methodologies and techniques needed in methodologies and techniques needed in the study?the study?

…….have the ability to help articulate the .have the ability to help articulate the appropriate focus for the study?appropriate focus for the study?

…….have the management skills to carry .have the management skills to carry out the study?out the study?

……maintain proper ethical standards?maintain proper ethical standards? ……communicate results to audiences so communicate results to audiences so

that they will be used?that they will be used?