Designing the Evaluation

Sonal R. Doshi MS, MPHProgram EvaluatorCDC/OSTLTS/DPHPI

TB Education, Training, and Evaluation Lights, Camera, Action: Setting the Stage for TB

Elimination18 September 2012

National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention

Division of Tuberculosis Elimination

Session Outline

Introduction Session objectives

Background What is evaluation? Research vs. evaluation CDC Evaluation Framework

What is evaluation design? Considerations for determining evaluation design Type of evaluation designs

Exercise Wrap-up

INTRODUCTION

Session Objectives

Identify key elements for determining the evaluation design.

Describe the difference between process and outcome evaluation and other evaluation designs

BACKGROUND

What is Evaluation?

“the process of determining the merit, worth, or value of something, or the product of that process.” Michael Scriven

“Evaluation is a field that applies systematic inquiry to help improve programs, products, and personnel, as well as the human actions associated with them.” *

* AEA guiding principles - http://www.eval.org/aea09.eptf.policy.handouts.pdf

Research vs. Evaluation

Knowledge intended for use

Program- or funder-derived questions

Judgmental quality Action setting Role conflicts Often not published Multiple allegiances

Production of generalizable knowledge

Researcher-derived questions

Paradigm stance More controlled setting Clearer role Published Clearer allegiance

Research EvaluationSystematicMethods

“Research seeks to prove, evaluation seeks to improve…”

M.Q. Patton

CDC Framework for Program Evaluation

MMWR, 1999. Framework for Program Evaluation in Public Health.

Standards

UtilityFeasibilityProprietyAccuracy

Engagestakeholders

Steps

Describethe program

Gather credibleevidence

Focus theEvaluation

design

Justifyconclusions

Ensure useand share

lessons learned

EVALUATION DESIGN

Evaluation and Evaluation Design To improve program… Art of evaluation design*

An “…exercise of the dramatic imagination” – L.J. Cronbach (pg 249)

Involves creating a design that is appropriate for a specific situation and particluar action or policy-making context

There is no single best plan for an an evaluation, not even for an inquiry into a particular program, at a particular time, with a particular budget

Any given design is necessarily an interplay of resources, possibilities, creativity, and personal judgements by the people involved

* From Patton MQ. Utilization-focused evaluation: the new century text. 1997. Sage Publications, Inc.

Types of Evaluation Design Many different ways to categorize

evaluation designs Examples

Randomized or true experimental/quasi-experimental/non-experiments

Summative/formative Process/outcomes Economic – CEA, CBA, and CUA Mixed methods

Considerations for Evaluation Design Determine expected uses by intended users

Judgement oriented Facilitating improvements Generating knowledge

Have a clear understanding of what, if any, decisions will be influenced by the evaluation

Reflect on the program evaluation standards Utility Feasibility Propriety Accuracy

* Citations, references, and credits – Myriad Pro, 11pt

Considerations for Evaluation Design, cont. Considerations from “Real World

Evaluation”* Understand clients information needs. Understand the dynamics and timetable of the decision-

making process. Define the program theory on which the program is

based in close collaboration with key stakeholders. Identify Budget, time, and data constraints and prioritize

their importance,. Understand political context.

Answer these questions** What is the purpose of the evaluation? How will the information be used? What will we know after the evaluation that we don’t

know now? What actions will we be able to take based on evaluation

findings?

* Bamberger M, Rugh J, Mabry L. RealWorld evaluation: working under budget, time, data, and political constraints. 2006. Sage Publications, Inc.** From Patton MQ. Utilization-focused evaluation: the new century text. 1997. Sage Publications, Inc.

EXERCISE

Click icon to add picture

Evaluation Questions What are the barriers to successful

tuberculosis contact investigations in Utah?

Did implementation of the TB cohort review process as a program evaluation tool improve treatment and contact elicitation outcomes?

WRAP-UP

Take Aways This is evaluation; not research Be clear on the evaluation question you are

trying to answer Be clear on the intended use of the

evaluation Be realistic Others??

Thank-you!

For more information please contact Centers for Disease Control and Prevention

1600 Clifton Road NE, Atlanta, GA 30333Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348E-mail: cdcinfo@cdc.gov Web: http://www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

National Center for HIV/AIDS, Viral Hepatitis, STD , and TB Prevention

Division of Tuberculosis Elimination

Designing the Evaluation

Sonal R. Doshi MS, MPH

Program EvaluatorCDC/OSTLTS/DPHPIsdoshi@cdc.gov

TB Education, Training, and Evaluation Lights, Camera, Action: Setting the Stage for TB

Elimination18 September 2012

National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention

Division of Tuberculosis Elimination