Logic Models and Organizational Strategy and Evaluation By: Thomas J. Chapel, MA, MBA Chief...

Logic Models and Organizational Strategy and Evaluation

By:Thomas J. Chapel, MA, MBAChief Evaluation OfficerCDC/[email protected]

Disclaimer…

The findings and conclusions in this presentation are those of the author and do not necessarily represent the views of the Centers for Disease Control and Prevention.

3

Objectives

Why bother with logic models Added-value of each component of logic

model Best use of logic models in framing

planning and evaluation questions

4

Take-Home Mantras

It’s not about the model, it’s about “describing your program”

Models make the program logic clear, not true

Match the detail/format to the need Simpler is generally better

5

Presenting Problem

Complex programs operating in complex environments

Accountability focused on long-term/distal outcomes

Joint efforts of many partners necessary to reach distal outcomes

Fragmentation of program planning, performance measurement, and evaluation

6

Integrating Processes to Achieve Continuous Quality Improvement

Continuous Quality Improvement (CQI) cycle

Planning—What actions will best reach our goals and objectives.

Performance measurement— How are we doing?

Evaluation—Why are we doing well or poorly?

What do What do we do?we do?

Why are Why are we we

doing doing well or well or poorly?poorly?

How are How are we we

doing?doing?

How do we How do we do it?do it?

7

You Don’t Ever Need a Logic Model, BUT, You Always Need a Program Description

Don’t jump into planning or evaluation without clarity on:

The big “need” your program is to address The key target group(s) who need to take action The kinds of actions they need to take (your intended

outcomes or objectives) Activities needed to meet those outcomes “Causal” relationships between activities and outcomes

8

Logic Models and Program Description

Logic Models

Graphic depictions of the relationship between your program’s activities and its intended effects

9

Step 2: Describing the Program:Complete Logic Model

Activities Inputs Outputs

Intermediate Effects/

Outcomes

Short-term

Effects/ Outcomes

Long-term Effects/

Outcomes

ContextAssumptions

10

What the program and its staff actually do



Outcomes

Short-term

Effects/ Outcomes

Long-term Effects/

Outcomes

ContextAssumptions

Results of activities: Who/what will change?

11

Finding Activities and Outcomes

GoalsObjectivesActions/Tactics

ActivitiesST or MT Outcomes

LT Outcomes or Impacts

Process MeasuresProgress Measures

Impl. Measures

Outcome MeasuresImpact Measures

Key Performance IndicatorsSuccess Factors

Pla

nE

val

PM

12

Lead Poisoning: The Problem Lead poisoning is a widespread environmental hazard

facing young children, especially in older inner-city areas. Exposure lead has been linked to cognitive disruption and behavioral disorders, especially when exposure occurs early in life. The main sources of lead poisoning in children are paint and dust in older homes with lead-based paint. Lead poisoning effects can be ameliorated through medical interventions. But, ultimately, the source of lead in the environment must be contained or eliminated through renovation or removal of the lead-based paint by professionals. Short of that, families can reduce the bad effects on their children through intensive housekeeping practices and selected nutritional interventions…

13

Lead Poisoning: The Program… County X, with a suspected high number of lead-poisoned children,

has received money from CDC to support a Childhood Lead Poisoning Prevention Program. The county’s program aims to do outreach and identify children to screen, identify those whose initial and confirmatory screening results show elevated blood lead levels (EBLL) at actionable levels, assess the environments of the EBLL children for sources of lead, and case manage both their medical treatment and the correction of their environment. The program staff will also train families in selected housekeeping and nutritional practices that have been shown to reduce lead levels in homes and lead burden in children. While as a grantee the county can work to assure medical treatment and reduction of lead in the home environment, the grant cannot directly pay for medical treatment or for renovation/remediation of homes.

14

Listing Activities and OutcomesLead Poisoning

Activities Outreach Screening Case management Referral for medical tx Identification of kids with

elevated lead (EBLL) Environmental

assessment Referral for env clean-

up Family training

Effects/Outcomes Lead source identified Families adopt in-home

techniques Providers treats EBLL

kids Housing Authority

eliminates lead source EBLL reduced Developmental “slide”

stopped Q of L improved

15

Global Logic Model: Childhood Lead Poisoning Program

Early Outcomes Later OutcomesLater ActivitiesEarly Activities

If we do…

Outreach

Screening

ID of elevated kids

And we do…

Case mgmt of EBLL kids

Refer EBLL kids for medical treatment

Train family in in-home techniques

Assess environment of EBLL child

Refer environment for clean-up

Then….

EBLL kids get medical treatment

Family performs in-home techniques

Lead source identified

Environment gets cleaned up

Lead source removed

And then…

EBLL reduced

Develop’l slide stopped

Quality of life improves

16

Lead Poisoning: “Causal” Roadmap

ScreeningScreening

Do Environment

Assessment

Do Environment

AssessmentID Source and

Refer for clean-up

ID Source and

Refer for clean-up

MedicalManagement

MedicalManagement

Lead Source

Removed

Lead Source

Removed

ReducingEBLLs

ReducingEBLLs

ImprovedDevelopment

and Intelligence

ImprovedDevelopment

and Intelligence

MoreProductive

and/or QualityLives

MoreProductive

and/or QualityLives

Family performs

in-home techniques

Family performs

in-home techniques

ID kids with

EBLL

ID kids with

EBLL

OutreachOutreach

Train

Families

Train

Families

Refer for

Medical Treatment

Refer for

Medical Treatment

Case

Management

Case

Management

Activities Outcomes

Elaborating the Simple Logic Model

18

Filling in the Blanks….Mediators

18

19

Conduct training

Do career leadership

development

Competent, sustainable,

diverse workforce

Improved health

outcomesDo Strategic

workforce planning

Evidence Base

Inputs Activities Outcomes

Logic Model—Office of Workforce and Career Development

…a miracle occurs…

20

Implicit Logic Model

Conduct training

Do career leadership

development

Workforce is

CompetentSustainable

Diverse

Improved health

outcomesDo Strategic

workforce planning

Evidence Base

Inputs Activities Outcomes

21

Tangible products of activities



Outcomes

Short-term

Effects/ Outcomes

Long-term Effects/

Outcomes

ContextAssumptions

22

Lead Poisoning: Sample Outputs

Pool (#) of eligible kids Pool (#) of screened kids Referrals (#) to medical treatment Pool (#) of assessed homes Referrals (#) for clean-up

23


ScreeningScreeningScreeningScreening

Do Environment

Assessment

Do Environment


Refer for clean-up

ID Source and

Refer for clean-up

MedicalMedicalManagementManagement

MedicalMedicalManagementManagement

Lead Source

Removed

Lead Source

Removed

ReducingEBLLs

ReducingEBLLs

ImprovedDevelopment

and Intelligence

ImprovedDevelopment

and Intelligence

MoreProductive

and/or QualityLives

MoreProductive

and/or QualityLives

Family performs

in-home techniques

Family performs

in-home techniques

ID kids withID kids with

EBLLEBLL

ID kids withID kids with

EBLLEBLL

OutreachOutreach

Train

Families

Train

Families

Refer for Refer for

Medical TreatmentMedical Treatment

Refer for Refer for

Medical TreatmentMedical Treatment

Case

Management

Case

Management

Activities Outcomes

24

Lead Poisoning: “Upgraded” Outputs: More than Simple Counts

Pool (#) of screened kids (meeting likely risk profile)

Pool (#) of eligible kids (with lead level >XXd/ul)

Referrals (#) to (qualified or willing) medical treatment providers

Pool (#) of assessed (“leaded”) homes Referrals (#) for clean-up (to qualified

or willing orgs)

25

Outreach

Screening

ID of elevated kids

Do case mgmt

Refer for medical tx

Train family in in-home techniques

Assess environ’t

Refer house for clean-up

(#) of eligible kids meeting risk profile

(#) screened kids with lead < threshold

(#) referrals to qualified medical tx

(#) of families completing training

(#) of “leaded” homes

(#) referrals to qualified clean-up

EBLL kids get medical treatment

Family performs in-home techniques

Lead source identified

Environ cleaned up

Lead source removed

EBLL reduced

Develop’l slide stopped

Quality of life improves

Global Logic Model: Childhood Lead Poisoning Program

Early Outcomes Later OutcomesLater ActivitiesEarly Activities Outputs

26

Resource “platform” for the program



Outcomes

Short-term

Effects/ Outcomes

Long-term Effects/

Outcomes

ContextAssumptions

27

Lead Poisoning: Sample Inputs

Funds Trained staff Legal authority to screen Relationships with orgs for med

treatment and environmental clean-up

28


ScreeningScreening

Do Environment

Assessment

Do Environment

Assessment ID Source and

Refer for clean-up

ID Source and

Refer for clean-up

MedicalManagement

MedicalManagement

Lead Source

Removed

Lead Source

Removed

ReducingEBLLs

ReducingEBLLs

ImprovedDevelopment

and Intelligence

ImprovedDevelopment

and Intelligence

MoreProductive

and/or QualityLives

MoreProductive

and/or QualityLives

Family performs

in-home techniques

Family performs

in-home techniques

ID kids with

EBLL

ID kids with

EBLL

OutreachOutreach

Train

Families

Train

Families

Refer for

Medical Treatment

Refer for

Medical Treatment

Case

Management

Case

Management

Outcomes Inputs

Funds

Staff

R’ships

Legal

Authority

Funds

Staff

R’ships

Legal

Authority

Activities

29



Outcomes

Short-term

Effects/ Outcomes

Long-term Effects/

Outcomes

ContextAssumptions

Moderators: Contextual factors that will facilitate or hinder (i.e. “moderate”) ability of activities to produce our outcomes

30

Moderators—Lead Poisoning

Political—“Hazard” politicsEconomic— Health insuranceTechnological— Availability of

hand-held technology

31

Lead Poisoning: “Causal” Roadmap and Moderators

ScreeningScreening

Do Environment

Assessment

Do Environment


Refer for clean-up

ID Source and

Refer for clean-up

MedicalManagement

MedicalManagement

Lead Source

Removed

Lead Source

Removed

ReducingEBLLs

ReducingEBLLs

ImprovedDevelopment

and Intelligence

ImprovedDevelopment

and Intelligence

MoreProductive

and/or QualityLives

MoreProductive

and/or QualityLives

Family performs

in-home techniques

Family performs

in-home techniques

ID kids with

EBLL

ID kids with

EBLL

OutreachOutreach

Train

Families

Train

Families

Refer for

Medical Treatment

Refer for

Medical Treatment

Case

Management

Case

Managementinsurance climate

“hazard politics”

technology

Activities Outcomes

32

Note!

Logic Models make the program theory clear,

not true!

33

Logic Models and Strategy and Planning

By Clarifying Sequence of Activities Outcomes, Helps With… Identifying/refining missionDeveloping consensus on key intended

outcomes—the “staked claim”Enhances quality and clarity of BSC

perspectives and objectives

34

Helpful Resources NEW! Intro to Program Evaluation for PH Programs—A Self-Study Guide:

http://www.cdc.gov/eval/whatsnew.htm Logic Model Sites

Innovation Network: http://www.innonet.org/ Harvard Family Research Project: http://www.gse.harvard.edu/hfrp/ University of Wisconsin-Extension: http://www.uwex.edu/ces/lmcourse/ CDC/DASH:http://www.cdc.gov/healthyyouth/evaluation/

resources.htm#4 CDC/STD: http://www.cdc.gov/std/program/progeval/TOC-

PGprogeval.htm

Texts Kellogg Foundation Logic Model Development Guide:

www.wkkf.org W.K. Kellogg Foundation Evaluation Resources:

http://www.wkkf.org/programming/overview.aspx?CID=281 Rogers et al. Program Theory in Evaluation. New Directions Series:

Jossey-Bass, Fall 2000 Chen, H. Theory-Driven Evaluations. Sage. 1990

Logic Models and Organizational Strategy and Evaluation By: Thomas J. Chapel, MA, MBA Chief...

Documents

Transcript of Logic Models and Organizational Strategy and Evaluation By: Thomas J. Chapel, MA, MBA Chief...