Www.fsg-impact.org Boston Geneva San Francisco Seattle Beginning the Evaluation Journey with FSG...
-
Upload
thomasina-wilkins -
Category
Documents
-
view
212 -
download
0
Transcript of Www.fsg-impact.org Boston Geneva San Francisco Seattle Beginning the Evaluation Journey with FSG...
www.fsg-impact.org
Boston Geneva San Francisco Seattle
Beginning the Evaluation Journey with FSG
KCIC Boot Camp
March 24, 2010
Prepared for:
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors2
Today’s Discussion
Today’s Agenda
Things You Need to Do to Evaluate Your Project’s Progress and Outcomes
Articulate your Theory of Change and develop a project Logic Model
Develop an evaluation plan
Execute your evaluation plan
Participate in Knight’s evaluation process
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors3
Evaluation Overview
The KCIC Evaluation Has Three Overarching Objectives
• Knight’s 2 Goals:
– Foundations are engaged in helping meet community information needs
– Communities are more informed and engaged
Evaluate Progress
Inform and Improve Program Strategy
Promote Learning
• Inform the Knight Foundation’s program strategy for the Community Information Challenge and identifying opportunities for refinements over time
• Create data that promotes shared learning within the KCIC grantee community and the field of place-based foundations to increase understanding of how community information needs can be addressed effectively
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors4
What is Evaluation?
The systematic collection of information based on questions of critical importance to the organization or community:What do you want to know?Why do you want to know it?Where will you find what you need to know?What will it look like?How will you use what you learn?Who are the intended users of the information?
To make judgments, inform decisions, and take action
Is an ongoing process; should not be an “event”
Is best when implemented as a collaborative learning activity that builds an organization’s capacity
Definitions of Evaluation
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors5
Process
The Evaluation Process: Planned, Purposeful, Systematic
Develop Recommenda-
tions and Action Plans
Articulate Theory of Change & Develop
Program Logic Model
Focus the Evaluation
Collect DataAnalyze Data and Interpret
Findings
Determine Design & Data
Collection Methods
(numbers, words, pictures)
Des
ign
Imp
lem
enta
tio
n
5 4
321
6
Ref
ine
Str
ateg
y B
ased
on
E
valu
atio
n F
ind
ing
s
Communicating and Reporting
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors6
Instrumental use - findings reflects changes that can be observed.
Conceptual use - findings may lead to stakeholders having different kinds of conversations, greater insights into future decisions, and/or greater commitment to the program or initiative.
Political or Symbolic use - legitimate uses of the evaluation findings may be applied to lobbying for the program or initiative, securing new or additional funding, or communicating that the evaluation has taken place.
Uses of Evaluation
How Can We Use Evaluation?
Well designed evaluations will serve multiple purposes
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors7
Internal Stakeholders• Board members/Trustees• Senior Leaders/Executives• Program staff
External Stakeholders• Legislators/Policymakers• Project partners• The general public• The media• Researchers• Other funders• Journalism programs
Stakeholders
Different Stakeholders Will Have Different Uses for Evaluation
Evaluation Stakeholders
Identifying stakeholders has implications for:
• which and how many questions are asked
• what indicators will be of interest
• how findings will be communicated and reported
• how results will be used
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors8
Example
This Theory of Change illustrates at a basic level why SVCF is doing what it’s doing
• The community needs a regional plan to address climate change and sustainability
• There is a low awareness and understanding among residents of the intricacies and implications of regional development decisions
• Residents are not engaged in the planning process today
• Creation of SVCF’s Venture Fund (with investors)
• Development of a web-based information platform for citizens to learn about potential scenarios
• Planned convenings on land use and sustainable cities
• Awareness campaign (KQED)
• Interactive kiosks
• Citizens are more informed and aware of key issues
• Citizens are more engaged in planning processes
• Regional plan is developed with community input
What we are doing to address the issue – how we plan to solve the problem or create
the change (Activities)
What the issue is and why we are taking action
(Assumptions)
The change we hope to see if we are successful
(Outcomes)
Source: FSG interview; used with grantee’s permission
Envision Bay Area – Theory of Change
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors9
As You Think About Your Own KCIC Project Outcomes, Consider the Possibilities
Outcomes are…the SO WHAT?
• The short and/or long term changes, results, and impacts from implementing a project, program, or initiative.
Changes may be:
• positive or negative, singular or multiple
• in knowledge, skills, and/or attitudes
• in organizational policies, practices, capacity
• at community level in terms of behavior, values, and attitudes
Outcomes
What will be different in your community if your project is successful?
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors10
Types of Outcomes
Short term Outcomes• Achieved during program timeframe• Within program control• “Expect to see”
For example:• New knowledge• Changed opinion/values• Increased skills• Changed motivation/intent• Changed attitudes• Changed aspirations• New relationships/networks
Short term Outcomes
Adapted from: Innovation Network, Inc. (2005)
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors11
Types of Outcomes
Intermediate Outcomes• Achieved at the end/beyond program timeframe• Follow shorter-term outcomes• “Want to see”
For example:• Modified behavior• Changed policies• Changed practices• Changed social action• Changed decisions• Modified structures
Adapted from: Innovation Network, Inc. (2005)
Intermediate Outcomes
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors12
Longer-term Outcomes
• Achieved after program timeframe
• Outside direct program control
• “Hope to see”
For example:
• Changed human condition
• Changed civic condition
• Changed economic condition
• Changed environmental condition
Longer-term Outcomes
Types of Outcomes
Source: Innovation Network, Inc. (2005)
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors13
Start Building Your Evaluation Plan by Developing Your Own Theory of Change and Identifying Your Project Outcomes
1. From our interviews with each of you, we have attempted to visually depict your project’s theory of change. Review our version of your ToC and change, add/or revise as needed.
2. If your project were hugely successful in three years, what would your target audience be doing, thinking? (long term outcomes)
3. After one year, what would your target audience be doing, thinking? (short term outcomes)
4. What are some indicators that your project is achieving what you set out to do? (outputs)
In Practice
See online assignment form: http://bit.ly/9nREQl
KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors14
Next Steps
Developing a Project Logic Model and Key Evaluation Questions (Webinar #1)
Determining Indicators of Online and Offline Behavior (Webinar #1)
Choosing an Evaluation Design and Data Collection Methods (Webinar #2)
Analyzing Quantitative and Qualitative Data (Webinar #2)
Strategies for Communicating and Reporting Evaluation Findings to Multiple Audiences (Webinar #3)
Developing an Evaluation Budget (Webinar #3)
Tips for Hiring an Evaluator (Webinar #3)
Next Steps in Designing Your Project’s Evaluation Plan
Throughout 2010 – FSG will provide up to 2 hours of additional evaluation-related technical assistance to each grantee