LESSONS LEARNED 1: IMPORTANCE OF HAVING CLEAR CONCEPTUAL FRAMEWORK TO GUIDE EVALUATION

SASA! Intervention Uganda

Clear conceptual framework for intervention about process of change trying to achieve in community

Stages of intervention focus and associated activities clearly articulated and translated into programming

Used to inform M&E design

SASA! Phases / Process of Change

Individual & Collective Capacity•Supportive environment

•Enhanced ability to prevent and

respond to VAW

Sustained Action

•Changed policies•Organized groups•Changed practice

in relationships, community, institutions

SASA! Logic Model

Action•Intention to act

•Personal change: balancing power• Public change:

sanctions ag VAW•Acceptability of

expanded gender roles

Behaviors•Balancing power•Communication

with partner•Decreased risk

behaviors•Community

activism

Acceptance and Influence

•Attitudes toward power, gender, human

rights•Silence broken

Connection•Activists/leaders/

professionals increasingly connected

and active

Skills•Response to women experiencing violence

•Hold men accountable•Promote balanced

power•Support to activists

Context

SocietalNat’l policy

makers, media

CommunityPolice, LCs,

HCPs, NGOs, Ssengas, RLs,

RelationshipRelatives,

elders, neighbors

Levels of SASA! Activities reaching each

Circle of Influence

Initial Outcomes

Knowledge•Recognizing VAW a problem

•Types of VAW•Consequences of VAW

•VAW/HIV linkage

Awareness•Root cause as imbalance of power between women and

men•Change can happen

Participation•Activists at grassroots, in leadership, in institutions

Intermediate Outcomes

IndividualWomen, men,

youth, CAs

Longer term outcomes

Impact

Socio-demographic Factors

Sex

Age

Income

Education

Employment

Religion

Place of residence

Socio-cultural Factors

Family Characteristics

Social Support

Alcohol Use

Reduced social

acceptance of VAW

Decrease in experience / perpetration

of VAW

Improved response to

women experiencing violence

Decrease in HIV/RH risk behaviors

Critical Thinking and Dialogue

•Public debate and discussion •Personal reflection

SASA! Phases / Process of Change

Key elements in conceptual framework

What are the range of potential exposures people may have to intervention?

What is expected process of change? What are likely pathways of change?

Among individuals receiving the intervention In the broader community

What is a realistic timeframe for this change to occur?

How do we measure this?

LESSONS LEARNED 2: POWER OF MULTIPLE FORMS OF EVIDENCE (IMAGE STUDY)

Did IMAGE affect MFIs financial performance?

50% reduction in centre “vulnerability” = attendance, arrears, savings Impact persisted 2 years after intervention completed

Vulnerable Centres

0

5

10

15

20

25

30

35

40

Aug-00Nov-00Feb-01May-01Aug-01Nov-01Feb-02May-02Aug-02Nov-02Feb-03May-03Aug-03Nov-03Feb-04May-04Aug-04Nov-04Feb-05

%IMAGE

Average

Similar impact on drop out rates.

Past year experience of intimate partner violence reduced by 55% (aRR 0.45 95% CI 0.23-0.91)

Past year experience of physical/sexual partner violence - pushed you - hit you - forced sex - scared to say no to sex

Adjusted risk ratio0.1 1 10

0.45 (0.23-0.91)

Pronyk et al. The Lancet Dec. 2006

How was violence reduced?

Changes in women’s relationships

“Now that we have money we are able to say how we feel without fearing that your husband will stop supporting you.”

“You can buy him cigarettes from your profit. Because of SEF’s money we are experiencing fewer problems in our households”

Women supporting women

“We are able to overcome abuse when we are in SEF because we get support from the women in the groups. When you engage yourself with other women and listen to their problems that will help you to cope”

LESSON LEARNED 3: COSTING IMPORTANT PART OF INTERVENTION EVALUATION

What does costing do?

Document all inputs into intervention Capital & recurrent expenditure Ingredients approach to costing puts a

financial value to all inputs: Monetary Non-monetary

Can be used to: Look at break down of costs by activity Estimate unit costs Estimate cost-effectiveness

2001-2004 2005-2007 2008-2010430 households 4500 households (30,000) 15 000 households (80,000)

Scaling up IMAGE in South Africa

Pilot StudyAdditional cost = US $43/client

Scale-upAdditional cost = US $13/client

CHALLENGES 1. REPORTING BIAS

Physical and/or sexual partner violence against women at baseline – SASA! study in past 12 months

INTERVENTION CONTROL

Violent Behaviour Women Men Women Men

Slapped her or thrown something at her that could hurt her 23.4% 20.5% 17.6% 23.1%

Pushed her or shoved her or pulled her hair 11.5% 7.1% 9.5% 7.2%

Hit her with his fist or with something else that could hurt her 11.2% 3.8% 8.4% 4.5%

Kicked her, dragged her or beat her up 10.2% 1.6% 8.4% 1.5%

Choked or burnt her on purpose 3.9% 0.0% 0.7% 0.0%

Threatened or actually used a gun, knife or other weapon 2.0% 0.0% 2.6% 0.0%

Threatened or intimidated her into having sexual intercourse even when she did not want to

9.5% 0.6% 7.7% 0.9%

Physically forced her to have sexual when she did not want to 10.5% 2.6% 9.5% 1.2%

Physical and/or sexual violence 29.1% 23.1% 24.5% 25.4%

The potential for reporting bias about gender norms & violence Women tend to under-report experiencing violence May be more willing to disclose following exposure

to intervention Could find higher disclosure in intervention

arms Men may not disclose perpetrating less socially

condoned acts of violence May be even less willing to report disclosure

following contact with intervention Could find higher disclosure in intervention

arms

Implications for evaluation

Importance of choosing quantitative outcome measures carefully

Rely on women’s reports on experiencing of violence, rather than men’s reports of perpetration If find reduction, effect less likely to be due

to reporting bias

Use qualitative research to triangulate

CHALLENGE 2: CAPTURING SOCIAL MOBILISATION DIFFICULT AND UNPREDICTABLE

IMAGE: community mobilisation led to a diverse range of activities

Many focused on violence and HIV

Also focused on other issues of concern to women

Difficult to fully document all of activities

Community mobilization: 40 village workshops 16 meetings with local

leaders 5 public marches 2 partnerships with local

institutions 2 new village committees

target Crime and Rape

Timelines as part of M&E activities in Uganda

Periodically using timeline with key informants Discuss key actions and key events in the

community Aim to map out scale and geographical diversity of

activities over time

CHALLENGE 3: EVALUATION DIFFICULT

Getting reliable evidence on impact is not easy

Getting strong quantitative data on impact difficult: RCTs gold standard study design, but expensive and technically

difficult Require partnership between intervention agencies & research

organisations May be rare that can show intervention impact on HIV or VAW

directly Seeking to evaluate complex interventions on complex issue

but: Trial design generally focuses on limited number of impacts –

need to identify primary outcomes Forms of social change may have multiple benefits Limited evidence about what may be reasonable timeframes for

different forms of change May be lack of recognition of importance of change in pathway

variables

The challenge of building a strong evidence base

Data from control communities important to help attribute impact Challenge of getting sufficient number of study and control

communities Difficult to identify and maintain control populations May not be feasible if intervention national level Difficult to identify and maintain control populations

Difficult to fund evaluation research: IMAGE had 9 + donors SASA! funding from 3 donors for baseline alone GBV not primary focus of most research donors

Leads to under-powered studies Less focus on statistical significance

CONCLUSIONS

The value of good M&E evidence

Growing body of experience, but extremely limited evidence about intervention impact on VAW & IPV

Importance of M&E to provide insights about: Process & scale of project implementation Impact on recipients and broader community Resources / costs required

Multiple forms of evidence important: Quantitative - scale of activity, magnitude of impacts Qualitative – acceptability, what happened, what means,

quality, role of key players, unexpected events Economic - forms of input, economic value, potential costs if

replicate Lessons from M&E important to inform

ongoing programming replication locally, nationally & internationally

The challenge of building a strong evidence base

Fundamental challenge of how to move from documenting levels of activity to level of change

Getting strong data on impact difficult: RCTs expensive and technically difficult Require partnership between intervention agencies & research

organisations May be rare that can show intervention impact on HIV or VAW

directly Seeking to evaluate complex interventions on complex issue: Timeframe for change my be longer than evaluation funding Importance of capturing broad benefits of social empowerment

interventions Mixed methods likely to provide most reliable evidence

Draws on strengths of each approach Provides opportunities for triangulation

The way forward?

Document experience and lessons learned From broader field of evaluation research – community interventions, service

delivery, media interventions From current M&E of different models of GBV intervention Experience combining quantitative and qualitative methods About realistic timeframes and process of change

Lobby for strategic investment in multi-disciplinary evaluation research: Women’s empowerment School based programmes Community mobilisation Masculinities Alcohol Service provision Role of legal and policy reform

Recognise need for methodological work Use evaluation research to learn about pathways of change, and broader

lessons for M&E How to conceptualise, define and measure key variables Integrating participatory evaluation methods with quant evaluation frameworks