Understanding Program Successes and

Challenges: Program Evaluation, Data

Tools and Resources

Washington State Mentors Conference

Hilary Loeb & Kelly Bay

August 23, 2012

1

Presentation Overview

Welcome and Presentation Goals

Resources to plan data collection and analysis

Case study activity

Key data sources for informing mentoring programs

Online demonstration

2

CSF Mission

10 years of proven experience, inspiring

underserved, low-income students to

finish high school, and providing the

unique integrated system of supports and

scholarships they need to graduate

college and succeed in life.

3

The Role of the Internal Evaluator

4

Volkov, B.B. (2011). Beyond being an evaluator: The multiplicity of roles of the internal evaluator. In B.B.

Volkov & M.E. Baron (Eds.), Internal evaluation in the 21st century. New Directions for Evaluation, 132, 38.

Resources to

Plan Data

Collection and

Analysis

5

Why Should We Evaluate Our Work?

Produces programmatic self-understanding

Allows you to demonstrate program outcomes

Substantiates requests for increased funding by providing evidence of effectiveness

Identifies and leverages program strengths

Source: The Pell Institute for the Study of Opportunity in Higher Education, the Institute for Higher Education Policy, and Pathways to College Network, 2011 http://toolkit.pellinstitute.org/evaluation-101/evaluation-myths-benefits/

6

Why Should We Evaluate Our Work?

Identifies and modifies ineffective

practices

Provides documentation for

performance/funding reports

Creates a foundation for strategic

planning

Produces credibility and visibility Source: The Pell Institute for the Study of Opportunity in Higher Education, the

Institute for Higher Education Policy, and Pathways to College Network, 2011

http://toolkit.pellinstitute.org/evaluation-101/evaluation-myths-benefits/

7

What Kinds of Evaluation Questions are

Relevant to Mentoring Programs? Implementation

How did our recruitment, training, matching and oversight vary across different sites?

Did we use our resources effectively?

Improvement What changes might we recommend to better serve

our students?

Impact How did the students and mentors benefit from

participation in the program?

How did the communities of our implementation sites change?

8

What Are Tools To Evaluate Our Work? Evaluation planning tools

Logic models

Theories of change

Outcome maps

Internal data collection

External data sources

9

What Is a Logic Model?

Why Should We Use It?

“ A logic model is a systematic and visual

way to present and share your

understanding of the relationships among

the resources you have to operate your

program, the activities you plan, and the

changes or results you hope to achieve.”

(W.K. Kellogg Foundation 2004)

10

What Is a Logic Model?

Why Should We Use It?

Communicating about a program in a

clear, concise way

Creating and revisiting a program

plan

Planning a program evaluation

11

What Are the Elements of A Logic

Model?

12

Source: W.K.

Kellogg

Foundation

Evaluation

Handbook

2004

13

GirlPOWER! Logic Model

http://www.mentoring.org/downloads/mentoring_624.pdf

How Can We Prepare a Logic Model? Assemble a dynamic team

Key constituents are necessary to gain a clearer understanding of your program goals, assets, and commitments

Include individuals committed to the goals of the organization and are willing to contribute over time

Disseminate key facts prior to the meeting Determine the basic information about the program

that every team member needs to know prior to the meeting

It is important that everyone “ starts in the same place” to enable the voices of multiple constituents

14

Source: Adapted from The Pell Institute Evaluation Toolkit, The Pell

Institute for the Study of Opportunity in Higher Education, the Institute

for Higher Education Policy, and Pathways to College Network, 2011

How Can We Prepare a Logic Model? Determine a method for maintaining good notes

Have a solid record of your meetings so that the evolution of particular ideas can be traced

A note taker should be determined prior to the logic model development meetings and ideally should be consistent

Create a time frame for completing the initial and ongoing review process Consider conducting a series of short term on or

off site “ retreat” meetings with the project team

The decision to schedule meetings is likely to depend on time constraints and your organization’s culture

15

Source: Adapted from The Pell Institute Evaluation Toolkit, The Pell

Institute for the Study of Opportunity in Higher Education, the Institute

for Higher Education Policy, and Pathways to College Network, 2011

Logic Model Brainstorm You are working with a group of stakeholders

to develop a logic model for a mentoring

program for recent immigrant youth

Review the Research Corner: Mentoring

Immigrant Youth (also found at

http://www.mentoring.org/downloads/mentoring_1318.pdf)

What critical program elements would you

include in the logic model in the next slide?

16

Logic Model Brainstorm

17

Resources Activities Outputs Short and Long Term Outcomes

Impact

In order to accomplish our set of activities we will need the following:

In order to address our problem or asset we will conduct the following activities:

We expect that once accomplished these activities will produce the following evidence or service delivery:

We expect that if accomplished these activities will lead to the following changes in 1–3 then 4–6 years:

We expect that if accomplished these activities will lead to the following changes in 7–10 years:

What Tools Do Evaluators Use?

Secondary data

Surveys

18

What Tools Do Evaluators Use?

Interviews

Focus groups

Observations

Other scholars’ research

19

From:

http://www.mentoring

.org/news_and_resea

rch/research_and_st

udies/research_categ

ories/#formal

Key Data Sources for

Informing Mentoring

Programs

20

Quality of Relationship Survey Questions

21

Midcourse Corrections / Post Assessment Tool

National Mentoring Center: http://educationnorthwest.org/webfm_send/212

19 Questions:

1. Youth-

centered?

2. Youth’s

emotional

engagement?

3. Youth’s

satisfaction

with

relationship?

Youth Outcomes Survey

22

Pre and Post Assessment

Developed by Big Brothers Big Sisters of America

http://oregonmentors.org/library/evaluationtools/view/7/

10-Minute Survey

Self-Esteem/ Efficacy

School & Academics

Personal Attitudes & Beliefs

Goals & Future

Family Relationships

Drug & Alcohol Use/Attitudes

Delinquency & Antisocial

Behavior

Mentor Evaluation Form

23

Post Assessment

http://www.mentoring.org/program_resources/elements_and_toolkits/tool_kit/eval/

Data Collection Brainstorm Review our immigrant mentoring program

logic model

What types of data would you plan to

collect to track the implementation and

success of the program?

24

Logic Model: What Types of Data

Might We Collect and Analyze?

25

Resources Activities Outputs Short and

Long Term

Outcomes

Impact

In order to accomplish

our set of activities we

will need the

following:

In order to address

our problem or

asset we will conduct

the following

activities:

We expect that once

accomplished these

activities will produce

the following evidence

or service delivery:

We expect that if

accomplished

these activities

will lead to the

following

changes in 1–3 then

4–6

years:

We expect that if

accomplished

these activities

will lead to the

following

changes in 7–10

years:

State and Community-Level Data:

KIDS COUNT Data Center

26

State and Community-Level Data: OSPI Report Card

27

State and Community-Level Data: College Tracking Data Services

28

State and Community-Level Data: Ask Healthy Youth Survey Query Builder

29

Concluding Thoughts

Evaluation is an ongoing learning process best completed in collaboration with stakeholders.

Evaluation is a continuous improvement process that draws from shared review and reflection about data.

"Not everything that can be counted counts, and not everything that counts can be counted." Albert Einstein

30

Online

Demonstration

and Resources

31

Other Resources Supporting

Youth Mentoring – Washington

Washington State Mentors Resources: http://www.wamentors.org/resources

College Tracking http://www.collegetracking.com/

OSPI School Report Card and online data http://reportcard.ospi.k12.wa.us/DataDownload.aspx

Healthy Youth Survey http://www.askhys.net/layout.asp?page=nanalyzer

KIDS COUNT Data Center http://datacenter.kidscount.org/WA

32

Other Resources Supporting

Youth Mentoring – Washington

Office of Financial Management’s Education Research & Data Center & Washington State Databook http://www.erdc.wa.gov/; http://www.ofm.wa.gov/databook/

Washington State Achievement Council (formerly HEC Board): http://www.wsac.wa.gov/

State Board of Community & Technical Colleges http://www.sbctc.edu/

Your school district data director or college or university institutional research office

33

Other Resources Supporting

Youth Mentoring – National National Mentoring Association Resources:

http://www.mentoring.org/program_resources

National Mentoring Association Evaluation Toolkit:

http://www.mentoring.org/program_resources/elements_and_toolkits/tool_kit/eval/

School Data Direct (Chief of State School Officers) http://www.schooldatadirect.org/

NCES Condition of Education http://nces.ed.gov/programs/coe/

NCHEMS Information Center http://www.higheredinfo.org/analyses/newmeasures.php

High School, Community College, and National Student Engagement Surveys http://ceep.indiana.edu/hssse/index.htm; http://www.ccsse.org/; http://nsse.iub.edu/

34

Other Resources Supporting

Youth Mentoring – National National Governors’ Association Center for Best Practices

http://nga.org/cms/center

American FactFinder

http://factfinder2.census.gov/faces/nav/jsf/pages/index.xhtml

National Survey of Children’s Health

http://www.nschdata.org/Content/Default.aspx

Centers for Disease Control Behavioral Risk Factor

Surveillance System http://www.cdc.gov/brfss/

Grapevine (Data on state fiscal support for higher education)

http://grapevine.illinoisstate.edu/

Public Agenda (Portal for educators)

http://www.publicagenda.org/educators

35

Resources to Learn More about

Evaluation Planning Tools

Pell Institute Evaluation Toolkit http://toolkit.pellinstitute.org/

Organizational Research Services http://www.organizationalresearch.com/publications_and_resources_featured.aspx

“Getting Started: A Self-Administered Guide to Outcome Map Development” 2009

“Orientation to Theory of Change” 2008

“How to Manage and Analyze Data for Outcome-based Evaluation” 2000

Kellogg Foundation Evaluation Handbook, 2004 http://www.wkkf.org/knowledge-

center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx

Logic Model Development Guide, 2010 http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx

36