Impact Measurementwhywhathow

9-14-09

Atlanta

Today ImperativesQuestionsWhy Now?Significant ChallengesBreakthroughs in the fieldCARE’s Long-Term StrategyThis YearHow do we meet the challenges?

Imperatives

…can only flow from a clearly-defined program approach that is predicated on a clear framing of the problem.

…systems must actually add value for the organization and produce reliable data that it can be utilized to inform strategic decisions.

Measurement …

Questions 1. If you do not measure our impact and demonstrate the

change, then does it matter?

2. Can we measure the most important changes or do we emphasis what we can measure?

3. How do you actually measure social impact and do those type of changes inherently defy measurement - an increase in someone’s level of opportunity or their dignity

4. Does spending on measurement and building measurement systems mean less time and resources for the actual work that we do?

Most Significant Challenges

What does ‘impact’ even mean? How do we even begin to define metrics that can cut across a plethora of disparate issues that CARE is addressing? Do we need counter-factual, experimental designs and benchmarks within the same issue areas to do so?

How do you measure outcomes rather than inputs? Especially when the intervention is removed from the ultimate desired change - i.e., girl’s education programs with the goal of increasing women’s empowerment.

Most Significant Challenges (2)

How do we measure impact for the wide variety of constituents that CARE is accountable to?

How do we build the capacity across the organization to make sure that the measurement is consistent through a well implemented system.

Breakthroughs in the Field

Shared Measurement PlatformsAllow organizations to choose from a set of measures

within their fields

Comparative Performance SystemsRequire all participants within a field to report on the same

measures

Adaptive Learning SystemsAllow people working on different aspects of a single

complex issue to engage in an ongoing, facilitated process

Why Now ?

The program approach across CARE

Critical to demonstrate intended benefits

for organizational performance management

Support fundraising

Provide evidence for advocacy for LIFT-UP

Proactively meets future compliance

Levels of Measurement The proposed system covers all levels of measurement and all stages of the program cycle:

Outputs

Effects

Impact

Activities Inputs

Levels of Measurement The proposed system covers all levels of measurement and all stages of the program cycle:

Outputs

Effects

Impact

Activities Inputs

Levels of Measurement

LEVEL 1

Input:• Resources needed to undertake a set of activities

Activity: • The actions that convert inputs into outputs

Output:• The changes that are a direct result of activities

Levels of MeasurementLEVEL 2 Effect: Changes in human behaviors and practices Improvement in services through increased competency (e.g. more effective extension systems) Changes in systems and policies (enabling environment)

LEVEL 3 Impact: Long-term sustainable change Contribution to the MDGs

Quality Assessment

Proposal Design

Proposal Analysis

Analysis of proposals across

CARE

Des

ign

Ubora Data Collection & Quality Assessment

Eva

luat

ion

Impl

emen

tatio

n

Results(Ongoing

Activities and Outputs)

Quality Assessment

Mid-implementation

Strategic Cross-Cutting AnalysisAnalysis Process

Results(Final Activities &

Outputs, plus Effects and Impacts)

Program Causal

AnalysisMore in-depth analysis of the

impact of specific

programs (e.g.

Signature Programs)Meta-Evaluation

Analysis of results across

CARE

Sectoral/Thematic ReviewsMore in-

depth, cross-cutting

analysis of the impact of

specific development areas (e.g.

water)

Strategic Impact I nquiry

More in-depth analysis of

CARE’s impact on specific

cross-cutting themes (e.g.

women’s empowerment,

governance)

Ubora Performance ReportMid-year and annual reports analyzing CARE’s performance to inform action for improvement

Reports

Women’s Empowerment ReportAnnual report showcasing CARE’s approaches, learning, stories

and impact, drawing on data and analysis relevant to the year’s theme

Quality AssessmentOverall quality

External Internal

Components

Component 1Program Data Collection SystemSectoral Portfolio Review

Component 2Annual Impact ReportStrategic Impact Inquiry

Component 3Meta-EvaluationProposal AnalysisProgram Causal Analysis

Components of IM System

Inputs Activities Outputs

Impact

Effects

Access AfricaHow are we measuring our impact

Increased general well being of participants

household

Increased in economic, social and political

empowerment participants

Monitoring & VSLA quality

Activity/Output Level R2. Retention rate R5. Annualized return on savings R8. Portfolio at riskR16. Cost per member assisted

Outcome Level% of VS&L members who have invested in productive assets

% women holding leadership position at community level Level of implementation of policies for women right on land and other productive assets

Impact Level

% population

below $2 per

day (MDG)

Impact Measurement System

Activity & Output level: MIS System

Excel based system to Manage Field Officer performance & VSLA Quality

Measures 4 categories of performance ratios (SEEP):

o Members Satisfactiono VSLA Financial Performanceo VSLA operating efficiencyo CARE operating efficiency

Quarterly data collection by projects in the field

Access Africa aggregates global data quarterly

Data enables AA to engage in Program Quality discussions with the COs

Common system used by OXFAM, CRS, PLAN, Aga Khan, FfH, etc.

Common indicators/tool across COs

Option of methodology: Client Survey: data collection on participants household Household Survey: data collection on community

members (participants and non-participants) RCT, with IPA in Uganda and Malawi Mixing quantitative and qualitative methods

Capacity building of COs – TDY, direct support from AA

Support COs in hiring relevant staff and/or consultants for M&E

What about Outcome and Impact?

Making the information Available

The DirectoryName

Program #GoalManager Contact InformationAssociated Project

Program Informatio

n

NamePN/FC #ManagerContact Information

NamePN/FC #ManagerContact Information

NamePN/FC #ManagerContact InformationName

PN/FC #ManagerContact Information

Project InformationSearchable information in both categories:

•Start Dates and End Dates of Projects and ProgramsImpact PopulationTarget GroupSignature Program componentMDG Impact Area program or project works inNumber of direct participants in the program Gender and AgeActivities and OutputsCARE partners Advocacy Goals, Partners and Targets of Advocacy Efforts

This year

Work with Country Offices to create and disseminate appropriate guidance on impact measurement

Work with Technical Units in Atlanta to help put impact measurement systems in place in select Learning Labs.

Focus on organizing consultations on and outlining a business process for impact measurement at CARE, complete with a resource development plan

Working with the Policy Advocacy Unit and Communications on CARE’s Annual Impact Report.

Finished

Finally