Using the CDC Evaluation Fwork to Avoid Minefields on the Road to Good Evaluation Presented to: 2002...

Using the CDC Evaluation F’work to Avoid “Minefields” on

the Road to Good Evaluation

Presented to:Presented to:

2002 National Asthma Conference2002 National Asthma Conference

October 24, 2002October 24, 2002By:

Thomas J. Chapel, MA, MBAOffice of the DirectorOffice of Program Planning and EvaluationCenters for Disease Control and Prevention

Why We Evaluate

“... The gods condemned Sisyphus to endlessly roll a rock up a hill, whence it would return each time to its starting place. They thought, with some reason, that there was no punishment more severe than eternally futile labor....”

The Myth of Sisyphus

–– MMWRMMWRFramework for Program Evaluation in Public HealthFramework for Program Evaluation in Public Health 2

3

Defining Evaluation

Evaluation is...Evaluation is...

the systematic investigation of the the systematic investigation of the merit, worth, or significance of an merit, worth, or significance of an “object”“object”

–– Michael ScrivenMichael Scriven

Program is...Program is...

any organized public health any organized public health action/activityaction/activity

4

Research vs. Program Evaluation

A continuum, not a dichotomy, but at far A continuum, not a dichotomy, but at far ends may differ in:ends may differ in: Framework and stepsFramework and steps

Decision makingDecision making

StandardsStandards

Key questionsKey questions

DesignDesign

Data collection sources and measuresData collection sources and measures

Analysis timing and scopeAnalysis timing and scope

Role of values in making judgmentsRole of values in making judgments

Centrality of attribution as conclusionCentrality of attribution as conclusion

Audiences for dissemination of resultsAudiences for dissemination of results

5

The Continuum

Efficacy…does my effort work in Efficacy…does my effort work in ideal circumstancesideal circumstances

Effectiveness…does my effort work Effectiveness…does my effort work in real world settings, and work the in real world settings, and work the same way across settingssame way across settings

Implementation fidelity…is my Implementation fidelity…is my (efficacious and effective) effort (efficacious and effective) effort being implemented as intended.being implemented as intended.

Today’s Focus

Top Minefields Top Minefields on the Road on the Road Conducting Conducting Good Good Evaluation!Evaluation!


Minefield # 8

Not linking planning and Not linking planning and evaluation…evaluation…


Minefield # 7

Evaluating only what you Evaluating only what you can measure…can measure…


You Get What You Measure…

“…“…In Poland in the 1970s, In Poland in the 1970s, furniture factories were furniture factories were rewarded based on pounds rewarded based on pounds of product shipped. As a of product shipped. As a result, today Poles have the result, today Poles have the world’s heaviest world’s heaviest furniture…”furniture…”

(New York Times, 3/4/99)(New York Times, 3/4/99)


Minefield # 6

Thinking evaluatively Thinking evaluatively only at the end…only at the end…


When to Evaluate….

Good program evaluation Good program evaluation shifts our focus from shifts our focus from

““Did it (my effort) work?”Did it (my effort) work?”

toto

““Is it (my effort) working?”Is it (my effort) working?”


Minefield # 5

NotNot asking “ who (else) asking “ who (else) cares…cares…


Minefield # 4

Neglecting intermediate Neglecting intermediate outcomes…outcomes…


Forgetting Intermediate Outcomes

14

Minefield # 3

Neglecting process Neglecting process evaluation…evaluation…


Minefield # 2

Confusing attribution and Confusing attribution and contribution…contribution…


17

“Networked” Interventions

Agency A

Program A-n

Program A-1

Agency B Program B-1

Agency C

Program C-n

Program C-1

Agency D

Program D-n

Program D-1

Organization

PerformancePartnership

CommunityOutcomes

Institute

OUTPUTS

SHORT-TERMOUTCOMES

LONG-TERMOUTCOMES

SYSTEMOUTCOME

Minefield # 1

Using more “sticks” than Using more “sticks” than “carrots”…“carrots”…


Framework forProgram Evaluation

19

Standards forEffective Evaluation

Not HOW TO do an evaluation, but help Not HOW TO do an evaluation, but help direct choices among options at each step:direct choices among options at each step:

At each step, standards ask which At each step, standards ask which choice(s)choice(s) Utility (7): Best serve information needs of Utility (7): Best serve information needs of

intended usersintended usersFeasibility (3): Are most realistic, prudent, Feasibility (3): Are most realistic, prudent, diplomatic, and frugal given resourcesdiplomatic, and frugal given resources

Propriety (8): Best meet law, ethics, and due Propriety (8): Best meet law, ethics, and due regard for the welfare of those involved and regard for the welfare of those involved and affectedaffected

Accuracy (12): Best reveal and convey Accuracy (12): Best reveal and convey technically accurate informationtechnically accurate information

20

Broadening Our Thinking About Evaluation

WhatWhat to evaluate to evaluate

WhenWhen to evaluate to evaluate

WhoWho should be involved in should be involved in evaluationevaluation

HowHow to evaluate to evaluate

21

Who Should Evaluate?

22

Why Involve Stakeholders

Smoke out disagreements in…Smoke out disagreements in…

Definition of the problemDefinition of the problem

Activities and priorities of programActivities and priorities of program

Outcomes that equate to successOutcomes that equate to success

What constitutes “proof” of successWhat constitutes “proof” of success

Get their help with.. Get their help with..

Credibility of findingsCredibility of findings

Access to key playersAccess to key players

Follow-upFollow-up

Dissemination of resultsDissemination of results23

Using Logic Modelsfor Evaluation

Clarity on Clarity on What are activities What are activities

What are intended effectsWhat are intended effects

What is the sequence/order of intended What is the sequence/order of intended effectseffects

Which activities are to produce which Which activities are to produce which effectseffects

Consensus with stakeholders on all of the Consensus with stakeholders on all of the aboveabove

Focus the evaluation designFocus the evaluation design

24

25

Some Factors That Influence Choice of Evaluation Focus

Users and usesUsers and uses – Who wants the – Who wants the information and what are they information and what are they interested in? interested in?

Accountability to (other) Accountability to (other) stakeholdersstakeholders – For what effects are – For what effects are key stakeholders expecting to see key stakeholders expecting to see results? results?

ResourcesResources – Time, money, expertise – Time, money, expertise

Stage of development Stage of development –How long –How long has the program been in existence?has the program been in existence?

““Ripple effect”Ripple effect”- How far out would - How far out would an intervention of this an intervention of this intensityintensity reasonably be expected to have an reasonably be expected to have an effect? effect?

Setting Evaluation Focus: Some “Process” Issues

What are the likely key challenges to What are the likely key challenges to “implementation fidelity?“implementation fidelity?

““Dropped baton” issues are keyDropped baton” issues are key Partner failed to do their partPartner failed to do their part

Client/family/patient failed to fulfill their Client/family/patient failed to fulfill their referralreferral

Other common challengesOther common challenges Inadequate dosageInadequate dosage

Bad accessBad access

Failure to retain participantsFailure to retain participants

Wrong match of staff and participantWrong match of staff and participant

26

27

Evidence Gathering: Choosing Design

What intervention was actually What intervention was actually delivered?delivered?

Were impacts and outcomes Were impacts and outcomes achieved?achieved?

Was the intervention responsible for Was the intervention responsible for the impacts and outcomes?the impacts and outcomes?

Justifying Claims About Intervention Effectiveness

Performance vs. a Performance vs. a comparison/control groupcomparison/control group

Time sequenceTime sequence

Plausible mechanisms Plausible mechanisms (or pathways toward change)(or pathways toward change)

Accounting for alternative Accounting for alternative explanationsexplanations

Similar effects observed in Similar effects observed in similar contextssimilar contexts

28

Choosing DataCollection Methods

Function of:Function of:

TimeTime

CostCost

Sensitivity of the issueSensitivity of the issue

““Hawthorne effect”Hawthorne effect”

EthicsEthics

ValidityValidity

ReliabilityReliability

29

Maximizing Use of Results: Key Questions

Who is the audience?Who is the audience?

What will be of greatest importance to What will be of greatest importance to them?them?

How will they use the information How will they use the information provided?provided?

How much time will they be willing to How much time will they be willing to spend reading and assimilating the spend reading and assimilating the material?material?

What type of vocabulary will express the What type of vocabulary will express the information most clearly?information most clearly?

30

Some CDC Asthma Examples

Comprehensive School-Based Comprehensive School-Based Asthma ProjectAsthma Project

Controlling Asthma in American Controlling Asthma in American Cities (CAAP) ProjectCities (CAAP) Project

31

Helpful Publications @ www.cdc.gov/eval

32

http://www.cdc.gov/eval

Community Tool Boxhttp://ctb.ku.edu/

33

Using the CDC Evaluation Fwork to Avoid Minefields on the Road to Good Evaluation Presented to: 2002...

Documents

Transcript of Using the CDC Evaluation Fwork to Avoid Minefields on the Road to Good Evaluation Presented to: 2002...