Download - Taking the Mystery out of Project Evaluation

Taking the Mystery out of Taking the Mystery out of Project EvaluationProject Evaluation

Marjorie Dennis, UMass LowellMarjorie Dennis, UMass Lowell

Lisa Glickstein PhDLisa Glickstein PhD

Andover Public SchoolsAndover Public Schools

IntroductionsIntroductions

Marjorie Dennis: NSF GK-12, RET; US ED Marjorie Dennis: NSF GK-12, RET; US ED TAH, PT3; State ITQ, STEM PipelineTAH, PT3; State ITQ, STEM Pipeline

Lisa Glickstein: US ED TAH, PEP; State Lisa Glickstein: US ED TAH, PEP; State STEM PipelineSTEM Pipeline

Participant ExperiencesParticipant Experiences

Googlisms!Googlisms! Evaluation is…Evaluation is…

…”…”used to manage a project to meet used to manage a project to meet objectives.”objectives.”

…”…”the process of measuring the quality of the process of measuring the quality of performance against standards to performance against standards to determine if the standards have been determine if the standards have been met.”met.”

…”…”a systematic inquiry designed to a systematic inquiry designed to provide information to decision makers.”provide information to decision makers.”

Session OverviewSession Overview

WhyWhy do you evaluate? do you evaluate? WhoWho chooses the evaluator? chooses the evaluator? WhereWhere do you find an evaluator? do you find an evaluator? WhenWhen do you start working with the do you start working with the

evaluator?evaluator? How How do you design and carry out an do you design and carry out an

evaluation?evaluation? WhatWhat do you do if something goes wrong? do you do if something goes wrong?

WHY?WHY?

Because you have to – federal or state lawBecause you have to – federal or state law

To get feedback during the projectTo get feedback during the project Formative evaluationFormative evaluation

Because you want to answer the big Because you want to answer the big

question: question: DID IT WORK?DID IT WORK? Summative evaluationSummative evaluation

““Research seeks to Research seeks to proveprove, ,

evaluation seeks to evaluation seeks to improveimprove…” …” M.Q. PattonM.Q. Patton

(former President of the American Evaluation Association)(former President of the American Evaluation Association)

Adapted from: Maberry Consulting Group

WHO?WHO?

Who is a good evaluator?Who is a good evaluator? Who selects the evaluator?Who selects the evaluator? Who decides contract terms?Who decides contract terms? Who will the evaluator report to?Who will the evaluator report to? Who will the evaluator work with?Who will the evaluator work with? Who owns the evaluation plan, data and Who owns the evaluation plan, data and

reports?reports?

Internal versus External EvaluationInternal versus External Evaluation

Program guidelinesProgram guidelines

CostCost

AvailabilityAvailability

Existing RelationshipsExisting Relationships

Four Steps to Select an EvaluatorFour Steps to Select an Evaluator

1.1. Decide who selects the evaluatorDecide who selects the evaluator Project Director or Advisory BoardProject Director or Advisory Board Head of organizationHead of organization Grant or Business Office (lowest bidder)Grant or Business Office (lowest bidder) Grant-makerGrant-maker

2.2. Contact potential evaluators and solicit Contact potential evaluators and solicit proposalsproposals

Four Steps (cont’d)Four Steps (cont’d)

3.3. Choose selection criteria (see Notes)Choose selection criteria (see Notes) KnowledgeKnowledge Experience – overall, and project-specificExperience – overall, and project-specific CostCost ReferencesReferences Communication skillsCommunication skills

4.4. Evaluate proposals and selectEvaluate proposals and select

Intellectual PropertyIntellectual Property

The funder paid for the evaluation and can The funder paid for the evaluation and can request data or reports at any timerequest data or reports at any time

The fiscal agent/sponsor retains the hard The fiscal agent/sponsor retains the hard and electronic copies of all evaluation and electronic copies of all evaluation materialsmaterials

Evaluation plan, rubrics, data and reports Evaluation plan, rubrics, data and reports all belong to the project (not to the all belong to the project (not to the evaluator)evaluator)

WHEREWHERE??

WhereWhere do you find an evaluator? do you find an evaluator? In houseIn house ExternalExternal

WhereWhere does one learn to be an evaluator? does one learn to be an evaluator? WhereWhere are evaluators certified? are evaluators certified? WhereWhere do external evaluators work? do external evaluators work?

Professional EvaluatorsProfessional Evaluators

American Evaluation Association (American Evaluation Association (http://www.eval.org/) )

Graduate & Certificate Programs (Graduate & Certificate Programs (http://www.eval.org/p/cm/ld/fid=43) )

There is no single recognized credentialThere is no single recognized credential Consulting or in a university setting (Consulting or in a university setting (

http://www.eval.org/p/cm/ld/fid=108) )

WHEN?WHEN?

WhenWhen do you select the evaluator? do you select the evaluator? WhenWhen do you work out terms of hire? do you work out terms of hire? WhenWhen do you have the first project do you have the first project

meeting? meeting? WhenWhen and how often will you meet? and how often will you meet? WhenWhen do you expect reports? do you expect reports?

Initial ContactInitial Contact Involving the evaluator in project planning, or Involving the evaluator in project planning, or

planning evaluation during (not after) project planning evaluation during (not after) project design is a good ideadesign is a good idea

An evaluator can prepare an evaluation plan for An evaluator can prepare an evaluation plan for a proposal (usually at no cost)a proposal (usually at no cost)

Sometimes it wins points to name an evaluator Sometimes it wins points to name an evaluator in the grant proposalin the grant proposal

Be aware of bid laws in your stateBe aware of bid laws in your state Post-award some changes to the evaluation plan Post-award some changes to the evaluation plan

are okayare okay

Contract and TermsContract and Terms

Scope of services – list of activities and Scope of services – list of activities and work products (see Notes)work products (see Notes)

Cost of entire contract, or hourly rate and Cost of entire contract, or hourly rate and estimated total (8-15% of total grant is to estimated total (8-15% of total grant is to be expected)be expected)

Automatic termination if grant ends, Automatic termination if grant ends, options to terminate for cause or no causeoptions to terminate for cause or no cause

Protections for organization from failures Protections for organization from failures or omissions on part of evaluatoror omissions on part of evaluator

HOW?HOW?

How do you design and carry out an How do you design and carry out an evaluation?evaluation? Create evaluation planCreate evaluation plan Select indicators and benchmarksSelect indicators and benchmarks

How do you collect and analyze data?How do you collect and analyze data? Analyze dataAnalyze data Write reportsWrite reports

Evaluation PlanEvaluation Plan

Project’s goals, activities and timeline include Project’s goals, activities and timeline include data collectiondata collection

Goal = ultimate change project intends to make in Goal = ultimate change project intends to make in a condition (e.g. poverty)a condition (e.g. poverty)

Indicators = changes in attitudes, knowledge and Indicators = changes in attitudes, knowledge and behaviors that show progress towards meeting behaviors that show progress towards meeting goalgoal

Benchmarks = amount of change expected in a Benchmarks = amount of change expected in a given project yeargiven project year

Mini-EvaluationMini-Evaluation

Working alone or in groups, fill out the Working alone or in groups, fill out the sample evaluation form for your programsample evaluation form for your program

Focus first on what you are certain of – Focus first on what you are certain of – include details where knowninclude details where known

Focus next on what you don’t know – how Focus next on what you don’t know – how would you measure or find out? Who would you measure or find out? Who would you ask and how?would you ask and how?

Can you make recommendations?Can you make recommendations?

Research vs EvaluationResearch vs Evaluation Production of generalizable Production of generalizable

knowledge is the goalknowledge is the goal Researcher-derived Researcher-derived

questions (hypothesis)questions (hypothesis) Paradigm stance (how or Paradigm stance (how or

why does it work?)why does it work?) Setting designed to test Setting designed to test

hypothesis (controlled)hypothesis (controlled) Clearer role (principal Clearer role (principal

investigator)investigator) PublishedPublished Clearer allegianceClearer allegiance

May produce generalizable May produce generalizable knowledgeknowledge

Program- or funder-derived Program- or funder-derived questionsquestions

Judgmental quality (does it Judgmental quality (does it work?)work?)

Action setting to provide Action setting to provide direct servicesdirect services

Role conflicts (project Role conflicts (project director AND evaluator)director AND evaluator)

Often unpublishedOften unpublished Multiple allegiancesMultiple allegiances


Formative EvaluationFormative Evaluation Project proceeding as planned?Project proceeding as planned?

Activities, recruitment, data collection taking Activities, recruitment, data collection taking place?place?

All partners working together?All partners working together?

How can or should project get back on track?How can or should project get back on track?

Is project getting closer to its anticipated goals?Is project getting closer to its anticipated goals?

Summative EvaluationSummative Evaluation

Did the project meet its goals or not?Did the project meet its goals or not? Did the participants benefit?Did the participants benefit? To what extent did they benefit?To what extent did they benefit? Can you prove that activities caused the Can you prove that activities caused the

benefits?benefits? What activities were most effective or cost-What activities were most effective or cost-

effective?effective? Is the project sustainable?Is the project sustainable? Is the project replicable?Is the project replicable?

Short vs Long-term OutcomesShort vs Long-term Outcomes

OUTCOMESWhat results for individuals, families, communities..…

SHORTLearning

Changes in:

• Awareness• Knowledge• Attitudes• Skills• Opinion• Aspirations• Motivation• Behavioral intent

MEDIUMAction

Changes in:

•Behavior •Decision-making•Policies•Social action

LONG-TERMConditions

Changes in:

•Well-being•Health•Economy•Civics•Environment

Adapted from: University of Wisconsin – Logic Models

The Four StandardsThe Four Standards

UtilityUtility: Who needs the information and what information : Who needs the information and what information do they need?do they need?

FeasibilityFeasibility: How much money, time, and effort can we put : How much money, time, and effort can we put into this? into this?

Propriety:Propriety: What steps need to be taken for the What steps need to be taken for the evaluation to be ethical? evaluation to be ethical?

Accuracy:Accuracy: What design will lead to accurate information? What design will lead to accurate information?


Six methods to collect information and data…Six methods to collect information and data…

Method Overall Purpose Advantages Challenges

1. Questionnaires,

Surveys,

and Checklists

To quickly and/or

easily get a lot of

information from

people in a non-

threatening way

• Can be completed

anonymously

• Inexpensive to

administer

• Easy to compare

and analyze

• Administer to

many people

• Can get lots of

data

• Many sample

questionnaires

already exist

• Might not get

careful feedback

• Wording can bias

responses

• Are impersonal

• In surveys, may

need sampling

expert

• Doesn't get the full

story


Six methods (continued)…Six methods (continued)…


2. Interviews To fully understand

participant's

impressions or

experiences, or to

learn more about their

answers to

questionnaires

• Collects a full

range and depth of

information

• Develops

relationship with

the participant

• Can be flexible

and responsive

• May take too long

• Can be hard to

analyze and

compare

• Can be costly

• Interviewer can

bias responses




3. Documentation

review

To get an impression of

how program operates

without interrupting the

program; review of

applications, finances,

memos, minutes,

participant work

products, and so on

• Collect

comprehensive and

historical

information

• Doesn't interrupt

program or

participant’s routine

• Information already

exists

• Few biases about

information

• Often time-

consuming

• Information may be

incomplete

• Need to be quite

clear about what you

are looking for

• Not flexible, as data

restricted to what

already exists




4. Observation To gather accurate

information about

how a program

actually operates,

particularly about

processes

• View operations of

a program as they

are actually

occurring

• Can adapt to

events as they

occur

• Difficult to

interpret observed

behaviors

• Complex to

categorize

observations

• Can influence

behaviors of

program

participants

• Can be expensiveAdapted from: Maberry Consulting Group



5. Focus groups To explore a topic in

depth through group

discussion, such as

reactions to an

experience or

suggested change,

understanding

common complaints

(useful in evaluation

and marketing)

• Quickly and

reliably get

common

impressions

• Can be an efficient

way to get a range

and depth of

information in a

short time

• Can convey key

information about

programs

• Hard to analyze

responses

• Need a good

facilitator for

safety and closure

• Difficult to

schedule 6-8

people together




6. Case studies To fully understand or

depict participant

experiences in a

program, and conduct

comprehensive

examination through

cross comparison of

cases

• Fully depicts

participant’s

experience in

program input,

process and results

• Powerful means to

portray program to

outsiders

• Usually quite time

consuming to

collect, organize

and describe

• Represents depth

of information,

rather than breadth


Basic analysis of quantitative (numeric) dataBasic analysis of quantitative (numeric) data

1.1. Make copies of your data and store the original. Use Make copies of your data and store the original. Use the copy for making edits, cutting and pasting, and so the copy for making edits, cutting and pasting, and so onon

2.2. Tabulate the information in one or more spreadsheets. Tabulate the information in one or more spreadsheets.

3.3. Examine the median, mean, and distribution (range) of Examine the median, mean, and distribution (range) of data sets; think about which is/are more meaningfuldata sets; think about which is/are more meaningful

4.4. Think about segmentation: Do you have sub-Think about segmentation: Do you have sub-populations based on demographics or how much populations based on demographics or how much service was received?service was received?

5.5. Consider confounding variables: Is the effect due to a Consider confounding variables: Is the effect due to a bias rather than your service or intervention? bias rather than your service or intervention?


Basic analysis of qualitative (subjective) dataBasic analysis of qualitative (subjective) data(interview or focus group responses, or written commentary on questionnaires)(interview or focus group responses, or written commentary on questionnaires)

1.1. Read through all the data.Read through all the data.2.2. Organize comments into similar categories: Organize comments into similar categories:

concerns, suggestions, strengths, weaknesses, concerns, suggestions, strengths, weaknesses, similar experiences, program inputs, similar experiences, program inputs, recommendations, outputs, or outcome indicators.recommendations, outputs, or outcome indicators.

3.3. Look for patterns, associations and possible causal Look for patterns, associations and possible causal relationships.relationships.

4.4. Keep all raw data as recommended by the funder Keep all raw data as recommended by the funder after completion in case needed for future after completion in case needed for future reference.reference.


Evaluation is not only IndicatorsEvaluation is not only Indicators

Evaluation: Indicators in the context of:Evaluation: Indicators in the context of: Design & implementationDesign & implementation Program relevanceProgram relevance EfficiencyEfficiency EffectivenessEffectiveness ImpactImpact SustainabilitySustainability

ExperiencesExperiences

Write down the worst thing that an Write down the worst thing that an evaluator ever said to you about your evaluator ever said to you about your project…project…

……or the worst thing that ever went wrong or the worst thing that ever went wrong on a grant-funded project at your on a grant-funded project at your organization!organization!

WHAT?WHAT?

WhatWhat do we do when something goes do we do when something goes wrong?wrong? Partners don’t get alongPartners don’t get along Leadership issuesLeadership issues Disagreements over data interpretationDisagreements over data interpretation Non-performance by partner(s)Non-performance by partner(s) Change in conditionsChange in conditions Conflicts of interestConflicts of interest

The Challenge of Being an The Challenge of Being an Evaluator (Sirotnik, 1999)Evaluator (Sirotnik, 1999)

If you…are looking for highly definitive, If you…are looking for highly definitive, generalizable, cause/effect relationships generalizable, cause/effect relationships between measurable, high-validity, independent between measurable, high-validity, independent and dependent variables, then I suggest you go and dependent variables, then I suggest you go to work for a pharmaceutical company…to work for a pharmaceutical company…However, if you have a high tolerance for However, if you have a high tolerance for ambiguity and like to mess around…with ambiguity and like to mess around…with contexts that are always colliding and contexts that are always colliding and changing…in these settings that are engaged in changing…in these settings that are engaged in long-term processes of renewal and change – long-term processes of renewal and change – well then have I got a career for you!well then have I got a career for you!

If the evaluator ain’t happy, ain’t no If the evaluator ain’t happy, ain’t no one happy…one happy…

If your project isn’t going well, it is the If your project isn’t going well, it is the evaluator’s job to tell you (and ultimately to evaluator’s job to tell you (and ultimately to tell the funder)tell the funder)

If you have an issue with your evaluator, If you have an issue with your evaluator, first look at and fix issues with your projectfirst look at and fix issues with your project

Most of the time, this will repair your Most of the time, this will repair your relationship with the evaluator, or at least relationship with the evaluator, or at least make it tolerable!make it tolerable!

Conflict of PartnersConflict of Partners

Partners have conflicting ideasPartners have conflicting ideas

One or more partners are not One or more partners are not carrying through with activities carrying through with activities or data collectionor data collection

Personality clashesPersonality clashes

Complaints about distribution of Complaints about distribution of resources or other complaintsresources or other complaints

Leadership IssuesLeadership Issues

Poor communication – no contact, bad Poor communication – no contact, bad emails, late information and noticesemails, late information and notices

Bills not paid on time, delays in processing Bills not paid on time, delays in processing registrations for activitiesregistrations for activities

Bad record-keeping or data collectionBad record-keeping or data collection Failure to respond to formative evaluationFailure to respond to formative evaluation Mismanagement of project resourcesMismanagement of project resources Illegal activityIllegal activity

Different Interpretations of DataDifferent Interpretations of Data

Evaluator sees the data Evaluator sees the data one way, project partners one way, project partners see it anothersee it another

Lack of data leads to Lack of data leads to subjective assessmentsubjective assessment

One party rejects dataOne party rejects data

Non Performance of PartnersNon Performance of Partners

Failure to complete scope of servicesFailure to complete scope of services Communication issuesCommunication issues Significant, ongoing delaysSignificant, ongoing delays Personality clashes or personnel problemPersonality clashes or personnel problem Undermining professional relationshipsUndermining professional relationships Mismanagement of fundsMismanagement of funds

Change in ConditionsChange in Conditions

New competitor (threat)New competitor (threat) Change in lawChange in law Sub-contractor or partner goes out of Sub-contractor or partner goes out of

business or quitsbusiness or quits Significant economic downturnSignificant economic downturn Change in leadership (project or Change in leadership (project or

organization)organization)

Conflict of InterestConflict of Interest

Is the evaluator independent enough to Is the evaluator independent enough to objectively evaluate the project?objectively evaluate the project?

Would giving a positive or negative Would giving a positive or negative evaluation result in financial gain?evaluation result in financial gain? Future contracts with one or more partnersFuture contracts with one or more partners

Is there an appearance of conflict of Is there an appearance of conflict of interest?interest? Employer/employee or family relationshipEmployer/employee or family relationship

Presentation Wrap UpPresentation Wrap Up Additional stories and comments on the Additional stories and comments on the

evaluation processevaluation process

Questions on evaluationQuestions on evaluation

Reference materials for further Reference materials for further investigation and information on working investigation and information on working with an evaluator and the evaluation with an evaluator and the evaluation processprocess

Additional ReadingAdditional Reading

Evaluating Professional DevelopmentEvaluating Professional Development, Thomas , Thomas R. Guskey, ISBN 0-7619-7561-6, 2000.R. Guskey, ISBN 0-7619-7561-6, 2000.

““8 Smooth Steps: Steps to Your own 8 Smooth Steps: Steps to Your own Evaluation,” Joellen Killion, Evaluation,” Joellen Killion, Journal of the Journal of the National Staff Development CouncilNational Staff Development Council, p 14, Fall , p 14, Fall 2003.2003.

Practicing Evaluation: A Collaborative ApproachPracticing Evaluation: A Collaborative Approach, , Rita O’Sullivan, ISBN 0-7619-2456-5, 2004.Rita O’Sullivan, ISBN 0-7619-2456-5, 2004.

W.K. Kellogg Foundation Evaluation Handbook W.K. Kellogg Foundation Evaluation Handbook & Development Guide, & Development Guide, www.wkkf.org, 2008., 2008.

Thank you for attending!Thank you for attending!

Contact information:Contact information: Lisa Glickstein: Lisa Glickstein: [email protected]

Marjorie Dennis: Marjorie Dennis: [email protected]