Evaluation DesignMatching Logic ModelSelecting Indicators

Ben SillimanYouth Development Specialist

NC 4-H

Design Concerns

• Questions to match program goals, levels• Indicators of program outcome, quality• Questions on program outcome, quality• Types of evaluation designs• Adjustments to fit audience, conditions• Timeline for evaluation activities

Bridging to Program Success

How does the evaluation design address each level of the logic model?

Matching Questions to Logic Model

Formative Questions• Input

– Are resources available?– Are constraints

addressed?• Output

– Did activities occur as planned?

– Was the target audience served?

Matching Questions to Logic Model

Summative Questions• Outcome

– Were there KASA changes?– Were there practice

changes?

• Impact– Did the cumulative activity

lead to change in social, economic, educational, or cultural change?

Matching Questions to Logic Model

Bridging to Effects• Outcome

– Were there KASA changes?• Skill in crossing the bridge• Confidence in crossing

– Were there practice changes?• Sustained or advancing skill• Assistance to others learning to

cross

Matching Questions to Logic Model

It’s not about building the bridge

• Impact– Did the cumulative activity lead to

change in social, economic, educational, or cultural change?• Confidence under challenge, crisis• Grasp of how to build skills• Use of skills to improve self and

community

Keep your eye on the design

Arkansas River through wood slats on the Royal Gorge Bridge, Photo by Eric Gorecki

Search pla

Indicate exactlywhere you mean to go

Program: Drive from Mecklenburg 4-H to State 4-H Office

Outcome Indicator: Arrive at parking lot, 512 Brickhaven

Indicate exactlywhere you mean to go

Program: Travel from Mecklenburg 4-H to State 4-H Office:

Process Indicators: Mileage & Turning Points

1: Start out going NORTHWEST on ARMORY DR toward CHARLOTTETOWNE AVE/E INDEPENDENCE BLVD. 0.0 mi…

7: Merge onto I-277 N/NC-16 N via ramp on the LEFT. 1.3 mi 8: Merge onto I-77 N/US-21 N via EXIT 5A toward

I-85/STATESVILLE. 1.8 mi 9: Merge at I-85 N via EXIT 13A toward GREENSBORO. 125.7

mi10: Take I-40 E toward RALIEGH. 30.3 mi…17: Turn LEFT onto BRICKHAVEN DR. 0.0 mi18: End at 512 Brickhaven Dr Raleigh, NC 27606-1492• Estimated Time: 2 hours 44 minutes Estimated Distance:

164.60 miles

Examples of 4-H Indicators

• Health: Increase in…– Nutrition Knowledge– Exercise– -Number at Healthy Weight

• Leadership and Citizenship: Increase in:– Civics Knowledge/Skill– Role/Responsibility for Org. or Service

Examples of 4-H Indicators

• School-to-Work: Increase in…– Science Knowledge or Skill– Career Readiness, Interviewing Skill

• Life Skills: Improvement in…– Communication Skills– Goal Setting– Research Skills

Examples of 4-H Indicators

• Program Quality: High levels of…– Safety– Support– Skill Building– Service Norms– -Etc.

Indicators-to-Targets

• Health: Targeted levels– All participants will score above 80% on

Nutrition Knowledge (specify concepts)– All participants will complete __ min. of

(specified) Exercise __ times per week– 80% of sustained participants will show

progress toward/maintenance at Healthy Weight (specify standard)

Targets-to-Design & Questions

• Health: Question formats– All participants will score above 80% on

Nutrition Knowledge (specify concepts)• Test participants before and after training• Ask participants to point to foods with

specific nutritional qualities (e.g, Vitamin C) or to describe qualities of specific foods

• Observe choices at meal or snack or ask for food recall from previous week

Types of Designs

• Anecdotal• Quasi-Experimental

– Single group

• Experimental– Comparison groups

May use recall, retrospective, or pre-post, cross-sectional or longitudinal, quantiative and/or qualitative methods

Health Examples:Types of Designs

• Anecdotal: Some tell their story– Some participants or others say that/how the

program changed eating habits• Quasi-Experimental (all participants share)

– Participants report (or others observe) changes in knowledge or behavior before-and-after or just after training

• Experimental (equivalent groups compared)– Participants and comparison groups report

changes in knowledge or behavior (or others observe) before-and-after training and often as a follow-up on retention or practice

Targets-to-Types of Designs

• Health: Types of Designs– Anecdotal– Quasi-Experimental

• Retrospective, Pre-Post

– Experimental• Comparison groups

May be cross-sectional or longitudinal, quantiative and/or qualitative

Call for the Question

If stakeholders want to know…• Who participated and how long

(input)– Keep enrollment, attendance, activity

records

• Whether the program promoted change– Ask retrospective or pre-post Qs

Call for the Question

If stakeholders want to know…• How different groups benefit

– Keep records & compare group outcomes

• How much change is due to a program– Use pre-post Qs, comparison groups– Gather additional data about qualities of

the group(s) and desired change(s)

• Types of evaluation designs• Questions on program outcome,

quality• Adjustments to fit audience,

conditions• Timeline for evaluation activities