Partnership

What to Expect From This SeriesWhat to Expect From This Series

1. Learn about steps required to implement outcome evaluation

2. Design a program logic model

3. Identify appropriate evaluation/measurement tools

4. Develop an implementation plan

UnderstandingUnderstandingOutcome Evaluation:Outcome Evaluation:Core Concepts andCore Concepts and

Building the FoundationsBuilding the Foundations

Session #1/3

Purpose of Today’s SessionPurpose of Today’s Session

• To gain a basic understanding of the core concepts and basic steps in outcome measurement in particular, and program evaluation in general.

• To share strategies for building buy-in to outcome measurement.

• To begin work on a program logic model for a program of your choice.

Today’s AgendaToday’s Agenda

• Outcome evaluation language

• Outcome evaluation steps

• Break

• Outcome evaluation steps

• Lunch

• Developing logic models

Our Approach toOur Approach toOutcome MeasurementOutcome Measurement

DiscussionDiscussion

Imagine…


• Evaluation is already going on, informally, in any healthy organization and can be harnessed.

• Evaluation is a natural and necessary part of organizational development, not an “add on.”

• Evaluation strategies that help one organization may be confusing or even damaging to others: there is no “cookie cutter” approach.


• Developing common language within and between organizations is key.

• Organizations get better at measuring outcomes when they have practice, and when they identify measurement priorities themselves.

• Evaluation is about organizing, interpreting and acting on information, not about gathering mountains of data.

ClearEvidenceof Impact

ImprovedPractice

ImprovedOutcome

Measurement

Useful Insights, New Questions

A Few Key DefinitionsA Few Key DefinitionsCommunity Based Research is a term that refers to a variety of efforts to

apply research tools and strategies to the task of building communities. Community based research can take many forms, including needs

assessment, environmental scans, etc. ..

Program evaluation is one form of community based research. It focuses on the study of community interventions...

Their processes, outcomes, and context.

Outcome evaluation is one type of program evaluation,in which: 1. outcome objectives for activities are identified,

2. success in reaching objectives is measured, and3. best practices are identified, and

4. recommendations for improvement result.

Outcome measurement is a process in which:1. outcome objectives are identified and

2. success in reaching objectivesis measured

The Dilemma of TerminologyThe Dilemma of Terminology

Evaluators and Funders Use Terms Like…

When TalkingAbout…

• Inputs, Outputs, (Performance) Indicators, Targets, Benchmarks, Data Sources

The Information Used to Measure Change

• (Short and Long Term) Outcome Objectives, Goals, Impacts, Results

The Changes Caused by an Intervention

• Outcome Objectives, Implementation Objectives, Goals, Outcomes, Targets, Intended Results

The Intent of an Intervention

• Stakeholder Consultation, Participatory Approach, Utilization Plan, Lessons Learned, Implementation, Process, Evaluability

The Process of Measuring and Learning from Outcomes

Outcome MeasurementOutcome Measurement

The Basic Steps

The Seven Practical Steps The Seven Practical Steps in Outcome Measurement in Outcome Measurement

Gathering & Analyzing

Information

Acting on Findings

Laying the Foundations

Evaluation Planning

4. Design Methods and Measurement plans5. Analysis Plan

1. Get Buy-In2. Clarify Theory

6. Communicate Results7. Act on Results

3. Develop EvaluationPurpose & EvaluationQuestions

Data Doesn’t Change the World… Data Doesn’t Change the World… People Do.People Do.

Evaluation Requires an Ongoing Focus onEvaluation Requires an Ongoing Focus onRelationship BuildingRelationship Building

Situating Yourself: Clarifying Values & Beliefs, Reviewing

Resources

Gathering, Sharing, and Organizing

Knowledge

Clarifying The Issues and Identifying Research Questions

Building & Maintaining

Relationships For Action

Deciding & Acting Together Using

Knowledge

The Role of Collaboration and Teamwork The Role of Collaboration and Teamwork in Community Based Researchin Community Based Research

NEW SKILLS OR

CAPACITIES

NEW CONNECTIONS

TO OTHER PEOPLE

OR GROUPS

NEW INSIGHTS

INTO ISSUES

Step 1:Step 1:Building Stakeholder InvolvementBuilding Stakeholder Involvement

from the Beginningfrom the Beginning

Fostering Buy-in/Stakeholder Fostering Buy-in/Stakeholder Participation: How?Participation: How?

• Have all those who have a stake in the project and its evaluation been identified? Is there support for the idea of evaluation?

• Are there other ways that stakeholders could be included in the early planning phase and throughout the evaluation?

• Has the political context of relationships among stakeholders been considered?

• Have the interests of outside audiences been identified and considered?

Creative Ways toCreative Ways toInvolve StakeholdersInvolve Stakeholders

• Helping to build logic models

• “Teaching” the evaluators

• As surveyors or interviewers

• As pilot testers for data collection tools

• As note-takers at focus groups

• As co-presenters of findings

• As participants in a “data analysis forum”

• OTHERS?


• The first meeting of your evaluation committee

• In the context of measuring outcomes, when do you run into stakeholder conflict?

BREAKBREAK

Step 2:Step 2: Clarifying Your Clarifying Your

Theory of ChangeTheory of Change

What is aWhat is aTheory of Change?Theory of Change?

• A way of clearly explaining why you do what you do

• An explanation that focuses on exactly how your work will lead to positive change.


Which one of the followingtheory statements

best describesprogram impact?

We have implemented a new We have implemented a new type of women’s shelter…type of women’s shelter…

– because a local report identified the need for more shelter beds locally.

– so that our organization can provide shelter for 20% more women

– that is at a different location more accessible to younger women

– so that young women will be more likely to report abuse and seek support

– in order to identify instances of abusive behaviour earlier in relationships and increase the chances of breaking the cycle of abuse

Components of aComponents of aBasic Program Logic ModelBasic Program Logic Model

• Activities explain the concrete things you do within your program

• Short-Term Outcome Objectives identify the fairly immediate changes you expect to see as a direct result of your work

• Long-Term Outcome Objectives identify the more distant benefits or changes you hope to contribute to by achieving your short-term outcomes

• Goals are the longest term, broad vision of your program

The Really Useful IdeaThe Really Useful Idea(articulating your theory of change)(articulating your theory of change)

Identify short-term outcome objectives that are

within your control, measurable and

achievable, and use a logic model to explain

how these short-term achievements contribute

to the achievements of long-term outcomes

over time.

Activity

Short-termOutcome

Long-termOutcome

Refer clients to Job

Leaders

Improve clients’ knowledge of

job opportunities

Improve clients’

employment networks

How Do Logic ModelsHow Do Logic ModelsHelp You to Measure Outcomes?Help You to Measure Outcomes?

• They help “unpack” complex, important long-term outcomes into short-term outcomes that can more easily be measured

• They help you sort out exactly how and why you expect outcomes to occur, so you know where to look for evidence of success

• They help you distinguish actual “measurement problems” from the much more common “theory problems”


(After Rush & Ogborn (1991) CJPE)

Activity Clusters

– The concrete things you do within your program

– The major categories or clusters which describe the activities that make up your work

– Clusters should be composed of activities that are similar at a behavioural level, not activities with similar purposes (e.g., cooking in the kitchen and cooking at the barbecue are similar activities, but cooking dinner and buying the groceries for dinner are not).

Outcome Objectives- Measurable in principle

- Describes a meaningful change

- Need not specify how much change will occur or what the target level is

- Specifies a particular population and situation

- Clearly explained and theoretically defensible

- Focused on active rather than the neutral or passive change (i.e., “improved”, or “reduced” vs. “supported”, “promoted”, or “encouraged”)

Example: Increased feelings of social support among participating parents


Short-term Outcome Objectives

- The immediate benefits or changes that the target groups are anticipated to experience or display as a result of the program activities.

- It should be possible to identify a clear, direct intended causal link between at least one program activity and each identified short-term outcome objective.


Long-Term Outcome Objectives

- The more distant benefits or changes that the target groups are anticipated to experience or display as a result of the initiative.

- Generally, long-term outcome objectives are the second-order changes that result from successful achievement of short-term outcomes over time.


Refer clients to Job

Leaders

Improve clients’ knowledge of

job opportunities

Improve clients’

employment networks

Develop and support clients’ individual rehab

plans

Clients maintain or improve their level of

function

Clients become aware of +/or are connected to community resources

Clients gain insight to healthier

lifestyles/choices

Provide clinical support and interventions

Assess and develop life skills

Client becomes a more productive, independent

member within their community

Clients are able to find +/or maintain

accommodation of their choice

Lon

g-te

rm

outc

ome

obj

ectiv

es

Sho

rt-t

erm

ou

tcom

e o

bjec

tive

s

Clients have improved informal

supports

Reduce the # of hospitalizations or

shorten lengths of stay

Clients have increased confidence

in ability to live independently

Clients have longer community tenure

Act

iviti

es

Example of a well-done logic model for a program with individualized outcomes

Weekly self help group for women

Women are able to evaluate relationship with partner and

make decisions about the relationship

Women develop friendships in the

group

Women feel less fragile and are not dealing with daily

crises

Family members reconstruct a support system and maintain

family unity

Women feel supported in the decisions they

make

Lon

g-te

rm

outc

ome

obj

ectiv

es

Sho

rt-t

erm

ou

tcom

e o

bjec

tive

s

Women support each other outside of the

group

Women articulate expectations of their

partners

Women access resources

Partner is involved as part of family decision making

Act

iviti

es

Example of a well-done logic model for a crisis oriented program

Working on Logic ModelsWorking on Logic Models

2-4 pm2-4 pm

Exercise 1: Describing your ActivitiesExercise 1: Describing your Activities

Think about what your staff and/or volunteers do on a day to day basis in as concrete terms as possible.

Focus on activities that involve contact with clients or the community. Activities that should be in your model are those that you expect will impact individuals or communities in some positive way.

Staff are most often “doing things”, like “providing”, “teaching”, “raising awareness”, “creating”, etc. so try to use these types of active verbs. List as many activities as you can and try to be as honest as you can about what you actually do.

Try and “cluster” your activities if they seem to involve doing similar things.

Exercise 1: Describing your ActivitiesExercise 1: Describing your Activities

Avoid putting “outcome” language into your activities. For example, avoid terms like “prevent”, “increase”, or “improve” because these are changes that are a consequence of the activities. Identify the things your program does that leads to prevention, increases, or improvements.

Avoid “double barreled” activities (e.g., “provide education and personal support”). This is especially important if they might lead to different outcomes. Separate them if possible.

While administrative tasks are important and vital (e.g., team meetings, hiring functions, etc.), you can usually omit them in logic models because they do not tend to directly impact outcomes objectives. You can leave these out and add them later if it helps to clarify your program.

Exercise 2: Identifying the Outcome Exercise 2: Identifying the Outcome Objectives for your ProgramObjectives for your Program

Develop a list of the outcome objectives your program is intended to achieve. Include as many outcomes as you can. You might also think about your mission, your marketing materials, grant proposals, and your own reasons for doing your job.

Focus on how the program makes a difference or change in your clients or the community. For example, if an outcome objective of the program is to give people someone to talk to, ask yourself how this may help an individual.

Try to write your outcome objectives in way that refers to change that can be measured.

Think about sequence. Logically, some outcome objectives will necessarily precede others. Try to arrange them temporally from top to bottom.

Exercise 3: Putting it all TogetherExercise 3: Putting it all Together• Using a pencil, begin to draw the connections between your

activities and your short term outcome objectives and between your STOs and your long-term outcome objectives. Keep in mind the following:

All STOs should be linked to at least one activity; all activities should be linked to at least one STO.

All LTOs should be linked to at least one STO; all STOs should be linked to at least one LTO.

Pick an arrow that you think might have questionable logic or weak assumptions and make it “dashed”.

Pick an arrow that you think has strong logic and well-supported assumptions and put a check mark beside it .

• Think about logical causes. If a group support intervention is supposed to lead to greater feelings of social support, think critically about why and how that happens. Does it make sense? In some cases, this analysis might lead you to add, delete, or modify activities and outcome objectives.

Using theUsing theOn-LineOn-Line

Logic Model Design ToolLogic Model Design Tool

What to Expect at Our Next SessionWhat to Expect at Our Next Session

• Respond to questions regarding finishing up logic models

• Identify indicators• Start thinking about measurement tools

Upcoming WorkshopsUpcoming Workshops

• Workshop 2 focuses on – Refining logic models and identifying

measurement implications

• Workshop 3 focuses on– Translating measurement plans into

action

Thank You!Thank You!

Step 3:Step 3:Developing YourDeveloping Your

Evaluation Purpose &Evaluation Purpose &Evaluation QuestionsEvaluation Questions

What do you really want to know?

What kinds of questions canevaluation really answer for you?

Breaking Down the ComplexityBreaking Down the Complexity

Did our program have an impact???

What were we trying to change? (Long-Term Outcome Objectives)

How were we planning to make that contribution? (Activities)

What sorts of things would we see if the expected change was happening? (Indicators)

What particular contribution were we going to make to that change?

(Short-Term Outcome Objectives)

How can we document those observations in a systematic way?

(Methods)

The Evaluation Purpose StatementThe Evaluation Purpose Statement

• A purpose statement should specify exactly what the goals of the research project are, and what you intend to investigate in order to achieve these goals.

A Good Purpose Statement Explains:A Good Purpose Statement Explains:

What we are evaluating (and what are we NOT evaluating)

Why we are evaluating

What we will do with the results

Defining Evaluation QuestionsDefining Evaluation Questions

• The empirical questions to which your evaluation will generate answers

• Usually about 5 to 7 in number except in very complex evaluations

• The things that will keep you focused as your evaluation project unfolds

Clarifying the Levels of QuestionsClarifying the Levels of QuestionsTo determine how

effective our program is at

helping people find meaningful jobs

How many of our clients find jobs?

How satisfied are clients with their

jobs?

What parts of our program seem to help the most?

For clients: “How much do you enjoy

your job?”

For staff: “How satisfied does your

client seem?”

Evaluation Purpose

Actual Interview or Survey Items

Evaluation Questions

Ways to Clarify Evaluation QuestionsWays to Clarify Evaluation Questions

• Focus on the action-oriented and solution focused (What can you do with the answers?)

• Review what knowledge your group already has.

• Whose knowledge is valued by decision-makers? Whose language is being used?

• What are the gaps in knowledge?

• What are the empirical questions within this issue? What extra value can someone with research expertise bring to the table?

WRAPPING UP STEPS 1-3WRAPPING UP STEPS 1-3

You’ve got stakeholder buy-in,you’ve articulated your theory of change,

and you’ve figured out yourevaluation purpose

and the evaluation questionsyou want to answer…

• This is the point at which you need to make a decision about whether you have stakeholder buy-in. Outcome measurement will not be successful without buy-in.

Are You Really Ready to DoAre You Really Ready to DoOutcome Measurement?Outcome Measurement?

Is there

buy-in?

You want to do more outcome measurement

Is outcome measurement

the next step?

1. Develop Logic models & Evaluability assessment

Awareness raising & team building; recruitment of additional people

2. Develop Purpose and Questions

Needs assessment, program design, planning

3. Identify Indicators

5. Agree on purpose and questions, set roles, talk about use of findings

YES

YES

NO

NO

4. Bring together stakeholders

6. Develop workplan

Step 4:Step 4:DevelopingDeveloping

Measurement PlansMeasurement Plans

What are Indicators?What are Indicators?

• Bits of information that provide part of the answer to one of your questions

• Things you can see or touch or hear – things that are observable in the world and don’t involve ‘interpretation’

• Are often numbers but can also be (e.g.) stories, quotations, examples, pictures

Indicators are a Useful IdeaIndicators are a Useful IdeaBecause...Because...

• They help to break down the complex task of “program evaluation” or “outcome measurement” into manageable chunks

• They help others to understand what you mean in practical terms when you talk about a particular outcome

• They help you build up a strong evaluation plan by combining different kinds of information from several sources

The Notion of The Notion of TriangulationTriangulation

Explaining the AnalogyExplaining the Analogy

• When you ask an evaluation question, you’re like a navigator on a ship asking “where am I?”

• Progress indicators are like the various compass bearings a navigator takes. The more “diverse” the readings, the better!

• If all the bearings suggest you’re in the same spot, you know you’ve done a good job of measuring.

Explaining the AnalogyExplaining the Analogy

• No single indicator, regardless of how accurate, can tell you where you are!

• Even if the indicators don’t converge or triangulate, you know more than you did and you know something about how much “error” is in your measurement technique.


Indicator Exercise

Something to Think AboutSomething to Think AboutOver Lunch…Over Lunch…

• What makes a good lunch “good?”

• How would you find out whether this lunch that we are about to have was successful?

• How would you prove to someone who wasn’t at our lunch that it was (or was not) successful?

LUNCHLUNCH

ChoosingChoosingInformation GatheringInformation Gathering

MethodsMethods

Pros and Cons of Popular Pros and Cons of Popular Measurement TechniquesMeasurement Techniques

• Retrospective reflections or stories

• Self report (interview or survey)

• Peer/family/worker report

• Direct observation

• File review

• Clinical assessmentsLess Practical, More Rigorous

More Practical, Less Rigorous

Step 5:Step 5:Analyzing Your DataAnalyzing Your Data

Data AnalysisData Analysis

• Analysis is a process of bringing order to data, organizing what’s there in patterns and categories

• Interpretation involves attaching meaning and significance, explaining descriptive patterns, looking for relationships among descriptive dimensions.

Suggested Steps in Data AnalysisSuggested Steps in Data Analysis

1. Organize Data

2. Review Original Questions

3. Summarize and Code

Steps in Data AnalysisSteps in Data Analysis

4. Generate Themes

5. Begin Writing

6. Provide and Receive Feedback

WRAPPING UP STEPS 1-5WRAPPING UP STEPS 1-5

The Evaluation Plan

Evaluation Methods Planning ChartEvaluation Methods Planning ChartEvaluation Questions

Outcome Objectives

Relevant Indicators

Where will we get the information?

What, specifically, will we ask?

What data collection tools will we use?

Data Analysis

May refer to success in meeting objectives (from your logic model) or other questions stakeholders feel the evaluation should address.

Match up with Column #2 of Worksheet #2

Things you can observe that will help you to answer this question.

Who will you speak to? How would you access existing information?

How will the questions be worded?

Given what is written in columns to the left, what method is most efficient and effective?

How are you going to make sense of the data you collect?

The Seven Practical Steps The Seven Practical Steps in Outcome Measurementin Outcome Measurement

Gathering & Analyzing

Information

Acting on Findings

Laying the Foundations

Evaluation Planning

4. Design Methods and Measurement plans5. Analysis Plan

1. Get Buy-In2. Clarify Theory

6. Communicate Results7. Act on Results

3. Develop EvaluationPurpose & EvaluationQuestions

Steps 6 & 7:Steps 6 & 7:Communicating andCommunicating and

Using ResultsUsing Results

“Problems facing the poor and the powerless must be understood in the hearts and the guts, as well as in the heads. The people with the problems must talk to each other as whole persons with feelings and commitment as well as facts.

As a tool of research, dialogue produces not just factual knowledge, but also interpersonal and critical knowledge, which defines humans as autonomous social beings”

(Park, 1993 as cited in Nelson, Ochocka, Griffin, & Lord, 1998).

Communication: Communication: Questions to ConsiderQuestions to Consider

• How much interpretation do you want to do (how much do others do)?

• Who should act after learning about the findings?

• What do you want them to do with the findings?

• How do you package information in order to facilitate this?

Where to StartWhere to Start

Is there

buy-in?

You want to do more outcome measurement

Is outcome measurement the next step?

1. Develop Logic models & Evaluability assessment

YES

NOAwareness raising & team building; recruitment of additional people

2. Develop Purpose and Questions

Needs assessment, program design, planning

3. Identify Indicators

5. Agree on purpose and questions, set roles, talk about use of findings

Is there a fit with

resources?

Revise plans, prioritize, discuss with funders

YES

YES

NO

NO

4. Bring together stakeholders

6. Develop workplan

Discussion:Discussion:Navigating First StepsNavigating First Steps

• From where you sit right now in your organization, how do you feel about these first steps?

• Which step strikes you as being the most challenging?

Action ChecklistAction Checklist

• Have you got key stakeholders involved?

• Have you clarified program logic and evaluation purpose?

• Have you identified indicators for key outcome questions?

• Have you maximized the use of the data you already have or can easily get?

Partnership

Documents

Transcript of Partnership