2013 Beyond TOC Slide Handout

7/27/2019 2013 Beyond TOC Slide Handout

1/14

Beyond Theory ofChange

Tips and Strategies for

Collecting, Aggregating, andReporting PM Outcomes

SW Cluster MeetingJuly 24, 2013

Susan Hyatt, Core Thoughtwww.core-thought.com

1

Session Description

Knowing WHAT data to collect to demonstrate the changeyour program creates is important.

However, data collection, aggregation and reporting tools andprocesses are critical to insure you have valid and reliable dataon which to base your findings either makes or breaks yourefforts.This session will provide you the step-wise information to helpyou insure your programs current performance measurementapproach is in alignment with your theory of change and canyield the information you need to clearly and confidently showyour outcomes.

2

Learning Objectives

By the end of the session, you will:

Know the four types of outcomes and identify your options forcollecting data based on the outcome to be measured;

Be able to identify the most appropriate data source and costeffective data collection methodology to use to measure yourintended outcome; and

Know strategies for aggregating and summarizing performancemeasurement data to make powerful statements that make yourprogram more attractive for grants and other resources.

3


2/14

Making the Case forSupport CNCS wants to support activities with

high likelihood of creating anintended change in a community need

oEvidence-based interventions: Willthe proposed intervention (design,dosage) likely lead to the desiredoutcome?

oWhats the cause and effect?

oWhat to measure to demonstrate

4

Measuring ProgramPerformance Outcomes

What isMeasurement

Question?

Who WouldKnow?

ClarifyDesired Change

Outcome

How toCollect Info?Then Do It!

What DoesInfo Tell You?

TweakProgram & Tell

Your Story

5

Measure theAMOUNT of serviceprovided (Did you do what you said?)

Number of people served (usually best)

Number of products/events/etc. created

What are Outputs?Bean Counting 6


3/14

Measure the CHANGES or BENEFITSthat occur because of service provided(individuals, organizations, communities, or theenvironment)

Increase - Decrease

More - Less

Something is different

What are Outcomes?The So What? 7

Keep It Simple....You CantMeasure EVERYTHING

Can read at grade level

Improved

selfeste

em

3rd Grade Reading Tutoring:Whats the Best Outcome

to Measure?

BetterAttendanceLessdisruptiveinclass

Improved GPA

Graduates from HS

8

How Clear Are You AboutWhat to Measure?

Problem/Need

DesiredOutcome

ProgramActivities/

Intervention

Theory of Change:Tool to IdentifyCause and Effect

Relationship

9


4/14

What Are You Trying toChange? (K-ABC)

Learning newinformation or

how to dosomething

10

12



how to dosomething

Thoughts orfeelings about

something

11

13



how to dosomething


something

Actions,conduct,or habits

12

14


5/14



how to dosomething


something

Actions,conduct,or habits

Circumstancesor status

13

15

What to Measure?Teen Health Program

Increasedknowledge ofhealthy food

More willingto eat

vegetables

More

frequentexercise

Improved

physicalcondition

14

What Do TheseQuestions Measure?

Identifying Outputs andOutcome Types

ACTIVITY

15


6/14

Now What?

You have identified the desiredchange (outcome) through your

theory of change.

Are you clear about WHAT to measure?AND

Are you measuring it the BEST way?

16

Asking the RightQuestion is Critical

Intervention: ABS driving school gives a 10 weekcourse that meets twice a week for 60 minutes thatincludes classroom-based and on road lessons on drivingskills

Desired Outcome: Students have basic drivingproficiency.

17

Asking the RightQuestion

Question: Do you like driving?

Answer: I LOVE driving!!

Do you like driving? gets information about an attitude not skill level.

18


7/14


Question: Are you a skilled driver?

Answer: I think I am a GREAT

driver!

Asking, Are you a skilled driver gets information aboutself perception, a thought not actual skill level.

Self-ratings are subjective NOT objective.

19


Question: Do you know theColorado driving laws?

Answer: I got 100% correct on mywritten drivers test!

Knowing state driving laws reflects knowledge not actual skill level even though it is objective.

20


Question: Have you demonstratedyou are a skilled driver?

Answer: I passed my road test!

An on-road driving test DOES measure skill level orproficiency and is objective.

21


8/14

Qualitative Data

Describes or characterizes through words

Focuses on meaning, experience or attitudes

Collected through focus groups, interviews, openedended questionnaire items, and other lessstructured situations.

Not the same as anecdotal Information

Qualitative Quality

22

Quantitative Data

Focus on numbers and frequencies.

Data which can be measured

Length, height, area, volume, weight, speed, time,temperature, humidity, sound levels, cost, ages, etc.

Quantitative Quantity

23

Whats the Best Way toAsk the Question?

Qualitative Quantitative

Please describe howyou have been feelinglately.

I feel great! I love my new job.Ive lost 20 pounds and feel

healthier than I have in years.

Thinking about the pastweek, how depressed wouldyou say you have been on ascale from 0 to 5, where 0means "not at all" and 5means "the most possible?

2

24


9/14

Whats Your Outcome

MeasurementQuestion?ACTIVITY

25

Whos Your Best Sourceof Information?

Data Source = The person, group or organizationthat has information on whether the intendedoutcome occurred.

Can you access this data source? How?

What are the pros and cons of relying on this data source?

What alternatives are available? What tradeoffs come withrelying on an alternative data source?

Once youve chosen an appropriate data source, make sureyoure asking them the right questions!

26

School AttachmentWho Would Know?

Students(preferred data

source)

Teachers

Tutors

Parents

27


10/14

Who is Your Data

Source?ACTIVITY

28

What Method Will YouUse?

Data Collection Method: the process used tosystematically collect data

Match method with:

Measurement question for desired outcome

Data source

Resources (time, dollars, etc.)

29

Surveys

Types of Questions:

Open ended

Closed endedScales

RankingMultiple Choice

30


11/14

Interviews & FocusGroups 31

Observations 32

Other Commonly UsedMethods

Logs and Tally Sheets

Standardized Tests

Program Developed Pre/Post Tests

Rubrics

Secondary Data

Other?

33


12/14

Choosing or Designing anInstrument

Which ever method you select, whatinstrument will you use?

Borrow vs. Develop Does it ask the right questions to get at your

desired outcome?

Does it have all the necessary components? What information will each question yield? How will you use information, if not related to

outcome?

34

Instrument Mapping

Look at each question on your data collection tool and ask:

Does this help us measure the desired outcome? Is there one question? More than one?

What kind of data will we get? Subjective? Objective? Quantitative? Qualitative?

If it doesnt measure the outcome, do we really need to ask it?How will we use the answer? Nice to know or internal use?

How will we analyze this?

What is our target? How much change is enoughtiveData

35

Instrument MappingACTIVITY

36


13/14

Data Collection Steps

After identifying a data source, method and tool(s):

1. Identify who you need to work with to collect data

2. Set a schedule for collecting data

3. Train your data collectors

4. Implement data collection

Good practice is to pilot test your tool.

FOR BEST RESULTS make key decisions about how toimplement data collection BEFORE your program begins!

37

Program PM Targets

Target describes amount of change you anticipate achieving. Based onindicator specific, measurable amount of change that will signifysuccess

Tools may vary by site but which questions get at thedesired outcome?

Need to apply program target to data collection toolsused by each site

Need to report aggregate outcome for program - noteach site separately

38

Data Aggregation andReportingACTIVITY

39


14/14

Summary of Key Points

Use your theory of change as a guide to outcomemeasurement.

The type of outcome you want to measure (attitude,

knowledge, behavior, condition) influences your choice ofdata source, method, and instrument.

Be very clear about the measurement question to ask

Choose methods based on ability to measure yourdesired outcome and ease of use.

Map your instrument

Remember: Continuous Improvement NOT Perfection

40

Ways to Connect

Susan [email protected]

303-512-0808

@Susan_Hyatthttp://www.facebook.com/CoreThoughtinc

www.core-thought.com

41

2013 Beyond TOC Slide Handout

Documents

Transcript of 2013 Beyond TOC Slide Handout