Measuring Outcomes: Potential Approaches in Evaluating Public Health William M. Sappenfield, MD, MPH...

Measuring Outcomes: Potential Approaches in Evaluating Public Health

William M. Sappenfield, MD, MPHMCH EPI Program Team Leader

Division of Reproductive Health, CDC

Carolyn B. Slack, MS, RNDirector, MCH Division

Columbus Health Department

Framework for Program Evaluation in Public Health

Adopted: MMWR 48(RR11) Sept 99 Summarizes the essential elements Provides framework Clarifies the steps Reviews standards Addresses misconceptions

Essential ShiftsShifts in Public Health Planning and Policy

EFFORTEFFORT RESULTS RESULTS

RESULTS ACCOUNTABILITYRESULTS ACCOUNTABILITY

ACCOUNTABILITY ACCOUNTABILITY BUDGETS BUDGETS

Premises of Reinventing Government What gets measured gets done. If you don't measure results, you can't tell success

from failure. If you can't see success, you can't reward it. If you can't reward success, you're probably

rewarding failure. If you can't see success, you can't learn from it. If you can't recognize failure, you can't correct it. If you can demonstrate success, you can win

public support.

Osborne and Gaebler (1992: chapter 5, "Results-Oriented Government")

Planning

Doing

Assessment

Strategies

Plan

Evaluation

Monitoring

Implement

Being Effective in Public Health

Program Evaluation StepsEngage

Stakeholders

Describe the Program

Focus the Evaluation

DesignGather

Credible Evidence

Ensure Use and Share Lessons

Learned

Justify Conclusions

Standards: Utility

Feasibility Propriety Accuracy

Step 1: Engage Stakeholders

Fostering input, participation, and power-sharing among those persons who have an investmentThose involved in program operationsThose served or affected by the programPrimary users of the evaluation

Step 2: Describing the Program

NeedExpected effectsActivitiesResources

Stage of development

Context Logic model

Scrutinizing the features of the program being evaluated, including its purpose and place in a larger context.

Columbus, OH

Columbus Health Department Perinatal Program


Program MapSelf Referral Outreach Visit Pregnancy Test

Risk Assessment

Service Plan

F/U Visit

Delivery

F/U Visit

No Show Outreach

Waiting List

Referral


Program MapSelf Referral

Outreach VisitPregnancy Test

Risk Assessment

Service Plan

F/U Visit

Delivery

F/U Visit

Nursing

Service Plan

Coordination

Medical

Treatment

Social

Counseling

Nutrition

WIC

Substance Abuse

Intervention

Food Stamps

Counseling




Risk Assessment

Service Plan

F/U Visit

Delivery

F/U Visit

Nursing

Service Plan

Coordination

Medical

Treatment

Social

Counseling

Nutrition

WIC

Substance Abuse

Intervention

Food Stamps

Prenatal Care Entry

Reduced Risk

Reduced Stress

Quit Rates

Weight Gain

Counseling




Risk Assessment

Service Plan

F/U Visit

Delivery

F/U Visit

Nursing

Service Plan

Coordination

Medical

Treatment

Social

Counseling

Nutrition

WIC

Substance Abuse

Intervention

Food Stamps

Prenatal Care Entry

Reduced Risk

Reduced Stress

Quit Rates

Weight GainReduced

LBW

Counseling

Step 3: Focusing the Evaluation Design

Purpose:Gain insightChange practiceAssess impactInquiry affects participants

Components: Users Uses Questions Methods Agreements

Planning in advance where the evaluation is headed and what steps will be taken; iterative process.




Risk Assessment

Service Plan

F/U Visit

Delivery

F/U Visit

Nursing

Service Plan

Coordination

Medical

Treatment

Social

Counseling

Nutrition

WIC

Substance Abuse

Intervention

Food Stamps

Prenatal Care Entry

Reduced Risk

Reduced Stress

Quit Rates

Weight GainReduced

LBW

Counseling

Step 4: Gathering Credible Evidence

Compiling information that stakeholders perceive as trustworthy and relevant for answering their questions.IndicatorsSources of evidenceQuality of informationQuantity of evidenceLogistics

Step 5: Justifying Conclusions

Making claims regarding the program that are warranted on the basis of the data.Stakeholder standardsAnalysis and synthesis of findingsInterpretation of the factsJudgments Recommendations

Step 6: Ensuring Use and Sharing Lessons Learned

Ensuring that stakeholders are aware, findings are used in decisions, and participants have a beneficial experience. Designed for usePreparation for findingsFeedback from all partiesFollow-up of usersDissemination processAdditional Uses

Program Evaluation StepsEngage

Stakeholders

Describe the Program

Focus the Evaluation

DesignGather

Credible Evidence

Ensure Use and Share Lessons

Learned

Justify Conclusions

Standards: Utility

Feasibility Propriety Accuracy


Understand the Theory of Action Choose the Outcome(s) to Measure Choose the Standard or Comparison

Group

What is the Intervention?

Bloom, 1968. The evaluation of primary prevention programs in Greenfield and Miller, Comprehensive Mental Health: The Challenge of Evaluation, University of Wisconsin Press.

“We are generally asked to evaluate an outcome of an undefined program having unspecified objectives on an often vaguely delineated recipient group whose level or variety of pathology is virtually impossible to assess, either before or after their exposure to the program.”


Domino Approach:

What happens in the sequence to make the whole chain of events occur?

What is the Theory of Action?




Risk Assessment

Service Plan

F/U Visit

Delivery

F/U Visit

Nursing

Service Plan

Coordination

Medical

Treatment

Social

Counseling

Nutrition

WIC

Substance Abuse

Intervention

Food Stamps

Prenatal Care Entry

Reduced Risk

Reduced Stress

Quit Rates

Weight GainReduced

LBW

Counseling



Group

Choose the Outcome(s) to Measure

Example: Smoking Cessation Ask Advise Assess Assist Arrange Stages of Change Cessation Attempts Cessation During Pregnancy Improved Low Birth Weight

Choose the Outcome(s) to Measure

Relates to the Theory of Action Sufficient Evidence Measurable Accurate and Reliable Feasible in a Public Health Setting Affordable



Group

Choose the Standard or Comparison Group

5 As

Choose the Standard or Comparison Group

Standard Internal Comparison

– Randomized

– Systematic External Comparison

Choose a Comparison GroupInternal

Starting Event

Pretest Posttest

Inter vention

•Randomize

•Select •Individual

•Groups

Choose a Comparison GroupExternal

Starting Event

Pretest Posttest

Inter vention



Group

CityMatCH Urban MCH Leadership Confluence


Case Study: “Show Me the Results!”

Midstate Health Department Perinatal Program



Risk Assessment

Service Plan

F/U Visit

Delivery

F/U Visit

Nursing

Service Plan

Coordination

Medical

Treatment

Social

Counseling

Nutrition

WIC

Substance Abuse

Intervention

Food Stamps

Prenatal Care Entry

Reduced Risk

Reduced Stress

Quit Rates

Weight GainReduced

LBW

Counseling

Measuring Outcomes: Potential Approaches in Evaluating Public Health William M. Sappenfield, MD, MPH...

Documents

Transcript of Measuring Outcomes: Potential Approaches in Evaluating Public Health William M. Sappenfield, MD, MPH...