8/2/2019 Starr Evaluation Workshop
1/48
8/2/2019 Starr Evaluation Workshop
2/48
Questions to Answer Today
1. Why evaluate social projects or programs?2. What types of evaluation are there?3. What are outcomes?4. How do I identify outcomes?5. How do I measure outcomes?6. Where does my data come from?
8/2/2019 Starr Evaluation Workshop
3/48
Why evaluate social projects orprograms?
8/2/2019 Starr Evaluation Workshop
4/48
Purposes of Evaluation
Improve Program Implementation Pilot-testing, tracking, mid-course correction, identify
variation or problems
Better planning, better structured programs This is called Formative Evaluation
8/2/2019 Starr Evaluation Workshop
5/48
Improve Program Implementation Evaluate Program Performance
Did the program meet its intended goals? Should wecontinue to administer program? Should the program beexpanded?
This is called Summative Evaluation
Purposes of Evaluation
8/2/2019 Starr Evaluation Workshop
6/48
Purposes for Evaluation
Improve Program Implementation Evaluate Program Performance Political or Administrative Reasons Required by law Required by funder
Delay action on a program Promotional
8/2/2019 Starr Evaluation Workshop
7/48
What types of evaluation are there?
Formative Evaluation What is being implemented? Adequate?
Consistent with design and expectations?
How do clients experience the program?
Summative Evaluation
What results (or net results) can be
attributed to the program? Any behavioral
change directly related to program?
Unintended or unforeseen consequences?
8/2/2019 Starr Evaluation Workshop
8/48
What are outcomes?
Inputs
Activities Outputs
Outcomes
8/2/2019 Starr Evaluation Workshop
9/48
Definitions
Program Inputs: Resources or personneldedicated to a program or project Program Activities: Service, treatment,
assistance, or training administered toachieve program goals
Program Outputs: Measures of programactivity consumption
Program Outcomes: Status transformationexperienced by individuals or entities during
or after program participation
8/2/2019 Starr Evaluation Workshop
10/48
Input, Activity, Output,
or Outcome?
8/2/2019 Starr Evaluation Workshop
11/48
Access to job postings
8/2/2019 Starr Evaluation Workshop
12/48
Volunteer
8/2/2019 Starr Evaluation Workshop
13/48
Long-term mentoring
relationship
8/2/2019 Starr Evaluation Workshop
14/48
Bridge repair
8/2/2019 Starr Evaluation Workshop
15/48
Improved Health
8/2/2019 Starr Evaluation Workshop
16/48
Food rescue
8/2/2019 Starr Evaluation Workshop
17/48
Funding via specialty
license plates
8/2/2019 Starr Evaluation Workshop
18/48
Increased confidence
8/2/2019 Starr Evaluation Workshop
19/48
Housing referral
8/2/2019 Starr Evaluation Workshop
20/48
Starr Fellowship
8/2/2019 Starr Evaluation Workshop
21/48
Increased competence
in providing care
8/2/2019 Starr Evaluation Workshop
22/48
Program Sustainability
8/2/2019 Starr Evaluation Workshop
23/48
Hip Hop curriculum
8/2/2019 Starr Evaluation Workshop
24/48
Increased engagement
in philanthropy
8/2/2019 Starr Evaluation Workshop
25/48
Equitable access to
food
8/2/2019 Starr Evaluation Workshop
26/48
Mentoring
8/2/2019 Starr Evaluation Workshop
27/48
Why Collect Information on
Outputs and Outcomes? Track performance Facilitate comparisons
Assess equity and efficiency Refine program or implementation theory Evaluate employees
Strategic planning Improve communication Strengthen the organization
8/2/2019 Starr Evaluation Workshop
28/48
How do I identify outcomes?
Ask for input from staff, participants orsimilar programs Construct a logic model
8/2/2019 Starr Evaluation Workshop
29/48
What is a logic model?
A map that outlines your theory ofchange and how you plan to assess it
Key components of a logic model:
Inputs Outputs
Activities Outcomes
8/2/2019 Starr Evaluation Workshop
30/48
New approachesto social change
Student
competence and
confidence to
effect social
change
Students pursuing
projects post-
graduation
Sample Logic Model
Outcomes
(intended)
Inputs Activities Outputs
Funding
Staff
Food
(primarily
Oreos)
Workshops
Discussions
Mentoring
with alumni
Peer
critiques
Immersion
projects
Completedprojects
Elevator
pitches
Evaluation
reports Sustainability
plans
8/2/2019 Starr Evaluation Workshop
31/48
31110624-Generation Citizen II TOC (DRAFT)TBG
Generation Citizen II/TOC
GC providescurriculum and
teacher training
GC persuades
districts/principals to
implement/fundGC program
GC provides opportunities forstudents to continue to engage(after-school programs, election
campaigns, ambassadorprogram)
Mentorsprovide
capacity toteachers, and
inspire
participants tobecome engaged
Studentparticipants
haveknowledge,
skillsnecessary to
civicallyengage
Studentsmore
engaged incommunity
in short-term (as
measured byvolunteering,following the
news,participating
incampaigns)
Participantsmore
civicallyengaged inlong term
(as measuredby voting,
volunteering,following the
news,participating
incampaigns,becoming
politicians)
GC recruits andtrains qualified,
motivatedcollege mentors
Teachersdeliver GC
curriculum 2x/wk
Teachersintegrate actioncivics principles
into broader
classroompedagogy
DRAFT
Short-term outcome
Long-term outcome
GC activity
Non-GC activity
Mentors use GCexperience as
impetus to tackleeducation and
political inequityin future jobs
8/2/2019 Starr Evaluation Workshop
32/48
Logic Model
Annie Wu | Starr Fellowship Retreat | May 8, 2011
Bettergrades
Volunteers timeand effort
Inputs Outputs Outcomes
Money
The LIAProgram
Servicing
Short Term
Long Term
By providing learning opportunities and academic guidance to underprivileged Liberian youth, wecan improve their academic performance, social adjustment, and self confidence.
Lowerincidence ofbehavioralproblems
Increasedconfidence
Betterequipped
Volunteersknowledge and
talents
Homeworkhelp
Learning &fun
A love forlearning!
UnderprivilegedLiberian youth
Books and otherresources
Lower dropout/ pregnancyrate
8/2/2019 Starr Evaluation Workshop
33/48
Metrics and Measurement
Annie Wu | Starr Fellowship Retreat | May 8, 2011
Number ofstudents
Outputs Outcomes
The LIA ProgramProgress
Measurables
Incidence ofBehavior Problems
Academicassistance
Mentoring
GradesLearningOpportunities
8/2/2019 Starr Evaluation Workshop
34/48
Inputs
Our time and C.V. Starr funding
11 undergraduate interns from BrownUniversity, Middlebury College, Barnard
College, 1 graduate student intern from theUniversity of Michigan School of NaturalResources and Environment
$2,750 from each intern to cover internshipcosts (including a direct contribution to theeducation fund)
8/2/2019 Starr Evaluation Workshop
35/48
Activities
Internship training and home-stay
Recruit new host families
English teaching curriculum and materials
English lessons and tutoring Millennium School Feasibility Assessment
Cultural Conservation
Oral History archive
Library of articles and books Raise money for education fund
Sponsor local artists and artisans
8/2/2019 Starr Evaluation Workshop
36/48
Outputs
Interns are trained as advocates for Secoya
Generate income for Secoya through grassrootstourism and jewelry
Create a framework for future volunteer Englishinstructors to work with Secoya
Help Secoya learn English
Oral history archive and library
Money for education fund
8/2/2019 Starr Evaluation Workshop
37/48
Outcomes
Interns gain motivation and knowledge to lead
Esperanza projects, raise money
More sustainable community-based tourism
operation geared towards students/volunteers
English instruction enables Secoya to overcome
proximal barriers to education
Oral history archive and library lay the foundationfor culturally-based curriculum
Increased educational opportunities for Secoya
youth through education fund
8/2/2019 Starr Evaluation Workshop
38/48
Logic Model: Inputs
-coordinators time
-volunteers time
-shareholder money-partnerships with farmers; Farm Fresh Rhode
Island; Brown Dining Services
8/2/2019 Starr Evaluation Workshop
39/48
Logic Model: Activities
-food purchasing
-market pickup
-shareholder community outreach (emails, recipe
sharing, etc.)-farm participation for shareholders (potlucks,volunteering, etc.)
-publicity and website construction
-budget reporting, compiling of previous programrecords
8/2/2019 Starr Evaluation Workshop
40/48
Logic Model: Outputs
-500 shareholders will receive food through the
program
-sustainability plan/clear business model
-how-to guide on campus/worksite marketshares model
8/2/2019 Starr Evaluation Workshop
41/48
Logic Model: Outcomes
1. local equitable food access for Brown
community
2. financial stability for farmers/dollars divertedfrom global food system
3. partnership of Brown community with Rhode
Island agricultural network
8/2/2019 Starr Evaluation Workshop
42/48
Assumptions
All logic models are based onassumptions
What assumptions have we made in theStarr Fellowship?
What assumptions have you made indesigning your project?
8/2/2019 Starr Evaluation Workshop
43/48
What is the intended outcome(s) ofyour project?
Small Group Discussion
8/2/2019 Starr Evaluation Workshop
44/48
How do I measure outcomes?
Difficult to measure something as broad asawareness or as long-termstudents
stay in school through graduation
Each outcome has one or more indicators: The specific observable, measurable
characteristic or change that will represent
achievement of the outcome The specific statistic you will calculate to
summarize level of achievement
8/2/2019 Starr Evaluation Workshop
45/48
Where does my data come from?
Record extraction
Questionnaires
Interviews Observations
Expert ratings or evaluation
8/2/2019 Starr Evaluation Workshop
46/48
Issues in Outcome
Measurement
Cost-benefit
Validity external and internal Cause Effect
Negative Externalities
8/2/2019 Starr Evaluation Workshop
47/48
Starr Deliverable Evaluation Plan
due 5/11. Logic Model of your venture
2. Three assumptions that underlie your model
3. Three outputs to be measured this summer
Metric, target, timeline
4. Three outcomes to be measured this summer
Intermediate or long-term?
Indicators this summer
Target, timeline
8/2/2019 Starr Evaluation Workshop
48/48
Next meeting small group
next week Bring a draft of your logic model to
share with a small group