2 0 1 5 E A S T A F R I C A E V I D E N C E S U M M I T J U LY 8 - 9 , 2 0 1 5 | N A I R O B I , K E N YA

COMMUNITY PERFORMANCE-BASED FINANCING IMPACT EVALUATION DISSEMINATION MEETINGJEANINE UMUTESI CONDO

PRINCIPAL COLLABORATORS AND INSTITUTIONS2

MOH-RBC-MCCHFidel Ngabo Cathy Mugeni

University of Rwanda/CMHS/SPHIna R. KalisaJames HumuzaJeanine CondoVedaste Ndahindwa

The World BankGil ShapiraNetsanet W. Workie

BACKGROUND AND JUSTIFICATION 3

• 2006-2008 Health Center PBF impact evaluation results show PBF had a significant impact on:• Increasing institutional deliveries• Improving quality of prenatal care• Increasing child preventive care utilization

• Vaccination, growth monitoring

• VCT for couples and more for discordant couples

• But….PBF did not have a significant impact on:• Prenatal / post natal care utilization• Modern contraceptive use• Short term malnutrition

COMMUNITY PBF PROGRAM DESIGN4

• Model 1: Conditional in-kind incentives for women:• Aimed at increase early prenatal care utilization, facility

deliveries, and postnatal care in order to diagnose and treat preventable threats

• This demand-side model endows mothers with gifts for:• Timely antenatal consultation: first visit in the first four months of

pregnancy

• In-facility delivery

• Timely postnatal care: consultation within the ten days after delivery

COMMUNITY PBF PROGRAM DESIGN5

• Model 2: Supply-side financial incentives for CHW Cooperatives:

• Aim: (i) improve quality of data reported at the sector level, (ii) increase utilization of priority maternal and child health services, and (iii) improve motivation and behavior of CHWs.

• Pay for reporting: Quarterly payment based on the timely submission of quality data reports related to 29 indicators.• Implemented nationally Not evaluated by this study

• Pay for performance: Offers financial rewards directly to CHWs.

COMMUNITY PBF PROGRAM DESIGN6

 Incentivized indicators Unit Fees (USD) (2010-2014/15)

2010 2011 2012 2013 2014

1. Deliveries: Women accompanied/referred to HC for assisted deliveries

2.73 1.37 0.99 0.99  0.99

1. Antenatal Care: Women accompanied/referred to HC for prenatal care within first 4 months of pregnancy

2.24 1.12 0.81 0.81  0.81

1. Nutrition Monitoring: % of children monitored for nutritional status (6 -59 months)

3.240.56

50.43

30.433  0.433

1. Family Planning: % of regular users using long term methods (IUD, Norplant, Surgical/NSV contraception)

2.11 1.06 0.77 0.77  0.77

1. FP: new users referred by CHWs for modern family planning methods

2.9 1.45 1.05 1.05  1.05

RESEARCH OBJECTIVES AND QUESTIONS

7

• Main Research Objective• The main objective of this study is to evaluate the effects of

demand-side (in-kind incentives) and supply-side (financial incentives) on health services utilization and health outcomes—The study attempts to isolate the causal impact of the incentives packages on health outcomes.

RESEARCH QUESTIONS8

• The primary research questions of the IE are:

• Do the demand and supply-side incentives affect:

• The number of woman who receive antenatal care during the first 4 months of pregnancy?

• The number of antenatal care visits?

• The number of facility deliveries?

• The number of woman-child pairs seen during postnatal care follow up care?

RESEARCH QUESTIONS 9

• Is there “a multiplicative effect” on outcomes when demand and supply-side incentives are combined?

• Do the supply-side incentives to CHWs increase the:

• Use of modern contraceptives services?

• Increase the time between births?

• Improve nutritional status in under-fives?

• Do the supply-side incentives to CHWs affect their motivation and behaviors?

Study Design and IE Surveys

EVALUATION DESIGN

• The study is a prospective, randomized impact evaluation

• CPBF interventions are randomly assigned at the sector level into four study arms:o D: Demand-side in-kind incentives to women

o S: Supply-side financial incentives to CHW cooperative

o DS: Demand-side incentives to women + Supply-side

incentives to CHW cooperative

o C: Comparison group

11

EVALUATION DESIGN

• Treatment group : sample of sectors where the interventions were implemented starting in 2010.

• Comparison group: sample of sectors where the interventions were not implemented until 2013

• In demand side and comparison sectors, CHW cooperatives received the average incentive payments distributed where the supply side intervention was implemented

12

SAMPLING DESIGN

Study Arm # Sectors# CHW

Cooperatives

# HH#

CHWs

D: In-kind Incentives50 50 600 1200

S: CHW incentives50 50 600 1200

DS: In-kind + CHW incentives

50 50 600 1200

C: Average financing50 50 600 1200

TOTAL200 200 2400 4800

13

Baseline Planned Sample Sizes by Study Arm

SAMPLING DESIGN14

Map of Sectors by Study Arm

SAMPLING STRATEGY AT BASELINE

• Cooperative Level• All 200 in 200 sectors

• Household Level• The household with the most recent birth in the

village (within last 4 months) was selected for the household interview for each of the 2400 randomly selected villages

• CHW individual Level• 2 CHWs selected for CHW individual questionnaire

for each of the 2400 randomly selected villages• Forcibly include MNH CHW (ASM)• Randomly select second

15

SAMPLING STRATEGY AT ENDLINE

• The endline survey covered the same sectors and villages as the baseline

• Changes from baseline survey:

• Doubling households interviewed

• Reduce by half CHWs interviewed

• Add health center assessments

16

SAMPLING STRATEGY AT ENDLINE

• CHWs cooperative• Same as baseline

• Households survey• Old sample: Tracking of households interviewed in baseline• New sample: Households with the most recent birth in each

of the villages visited during the baseline

• CHWs survey• Interview only the CHW in charge of maternal and neonatal

heath (ASM) in each village• Health facility survey

• A health facility assessment • A health worker survey (ANC and child curative care)• Patient exit interviews (ANC and child curative care)

17

CPBF IE DATA-AT BASELINE

• Baseline data collection:• Fielded from February to May 2010

Final baseline sample size

18

Study Arm # Sectors

#Households

#cooperatives

# CHWs

Demand-side 50 600 49 1,162

Supply-side 50 600 50 1,184

Demand- and Supply-side

50 600 50 1,186

Control 48 576 48 1,136

Total 198 2376 197 4,668

CPBF IE DATA-AT ENDLINE

• Endline data collection:• Fielded from November 2013 to June

2014

Final Endline sample size

19

Study Arm #Sectors

# Baseline HH

# New HH

#CHWs cooperatives

# CHWs

Demand-side

50 549 594 50 569

Supply-side 50 536 591 50 559Demand- and Supply-side

50 533 593 50 556

Control 48 539 565 47 536Total 198 2157 2343 197 2,220

CPBF IE DATA-AT ENDLINE

Quality checks:

During data collection: regular spot checks and supervision by field coordinator and research team

After data collection: 10% revisit of households (98% of households were really interviewed)

20

VALIDATION OF IE DESIGN: BASELINE BALANCE CHECK

T-tests were used to assess the difference

between each one of the three treatment arms

and the control

F-tests were used to test the hypothesis that the

variable means are identical among all 4 study

arms.

21

COMMUNITY PBF RESULT22

COMPARISON OF BASELINE AND ENDLINE SAMPLES OF WOMEN WITH RECENT BIRTHS

23

Baseline Endline T-test

Mean Mean Mean(Std.) (Std.) p-value

Number of households members 1.00 1.00 0.00 All 5.02 5.16 0.14**

(2.00) (2.00) 0.01 Under 5 1.62 1.06 -0.56***

(0.66) (0.84) 0.00Characteristics of head of household 1.00 1.00 0.00 Gender 0.94 0.93 -0.01

(0.26) (0.26) 0.37 Age 34.23 35.60 1.37***

(11.13) (11.73) 0.00 No school 0.20 0.18 -0.03**

(0.42) (0.40) 0.03 Primary school 0.64 0.69 0.05***

(0.50) (0.49) 0.00 More than primary school 0.16 0.14 -0.02*

(0.39) (0.37) 0.07Ownership of land 0.94 0.92 -0.02**

(0.27) (0.31) 0.01Characteristics of core woman 1.00 1.00 0.00 Age 28.15 28.91 0.76***

(6.25) (6.83) 0.00 Married 0.91 0.89 -0.02*

(0.31) (0.33) 0.08 No school 0.17 0.13 -0.04***

(0.39) (0.36) 0.00 Primary school 0.69 0.72 0.03**

(0.49) (0.47) 0.02 More than primary school 0.14 0.14 0.01

(0.38) (0.38) 0.59 Total number of births 3.01 3.07 0.06

(2.20) (2.19) 0.33

CHARACTERISTICS OF CHWS AT ENDLINE24

Mean Std. Obs.

Female 1.00 0.07 2204

Age 40.31 9.05 2205

From the village 0.37 0.50 2204

Number of years lived in the village 20.82 12.74 2185

Married 0.85 0.38 2205

Educational level: 1 0 2205

Primary school 0.75 0.46 2185

Post-primary school 0.10 0.32 2185

Secondary school or more 0.15 0.38 2185

Literacy 0.99 0.09 2192

Number of years worked as a CHW 4.26 2.35 2200

IMPACTS ON HEALTH SERVICE UTILIZATION OUTCOMES: TIMELY ANC, TIMELY PNC AND IN-FACILITY DELIVERY

• Indicators targeted by the Demand-side intervention• Timely PNC was not targeted in the supply-side

intervention

• Sample: 2334 women who were pregnant or gave birth shortly before the endline survey

Control Demand Supply D + S

Timely ANC 0.70958766 0.80558442 0.7412024 0.80364976

52.50%

57.50%

62.50%

67.50%

72.50%

77.50%

82.50%

87.50%

Impact on first ANC visit during the first 4 months of the pregnancy:

• A positive and significant impact of the demand-side in-kind incentives of about 10 percentage points

• The CHW incentives are not found to have a significant effect• No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms

Treatment Arm

Impact on having at least one ANC visit:

• Not targeted by the program!• Not impacted by the program but this outcome is already almost universal.

Control Demand Supply D + S

1 ANC 0.99085053 0.9881807 0.98708024 0.98712576

52.50%

57.50%

62.50%

67.50%

72.50%

77.50%

82.50%

87.50%

92.50%

97.50%

Impact on completing at least 4 ANC visits:

• Not targeted by the program!• Higher in the intervention sectors, but not statistically significant at the 10% level

Control Demand Supply D + S

4 visits 0.38929389 0.44691936 0.4273102 0.4472739

22.50%

27.50%

32.50%

37.50%

42.50%

47.50%

Impact on skilled-attended in-facility deliveries:

• No statistically significant difference between the treatment arms• Rate has increased substantially in the duration of the study for other reasons

Control Demand Supply D + S

Delivery 0.93975718 0.94941371 0.95962508 0.93719467

52.50%

57.50%

62.50%

67.50%

72.50%

77.50%

82.50%

87.50%

92.50%

97.50%

Impact on PNC within the 10 days after delivery:

• A positive and significant impact of the demand-side in-kind incentives of about 7 percentage points

• The CHW incentives are not found to have a significant effect• No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms

Control Demand Supply D + S

Timely PNC 0.12627086 0.19873905 0.10468981 0.19728536

2.50%

7.50%

12.50%

17.50%

22.50%

RESULTS ARE ROBUST TO THE FOLLOWING CHECKS:

1. Regressions with controls:• Detected impacts are not due to difference in observable

characteristics between the groups.• Randomization was done at the sector level -> controlling for

individual-level characteristics (for example: age, marital status, education level, household characteristics, distance from health center,…)

• Include characteristics which were not perfectly balanced

2. Excluding “misclassified” sectors:• 3 sectors have been misclassified during the transition to the new

computerized data system, although cooperatives were not informed

3. Difference-in difference approach:• Account for fixed unobservable characteristics

IMPACT ON FERTILITY AND FAMILY PLANNING

• The supply-side program incentivizes CHW cooperative for new and regular users of modern contraceptives

• Sample: 2,157 “baseline women”• Were pregnant or gave birth shortly before the baseline

survey• Re-interviewed in 2014• Not the optimal sample for evaluating impact on fertility

outcomes

Control Demand Supply D + S

Pregnancy 0.53006919 0.54973127 0.58538525 0.5496025

Modern 0.51592882 0.57089905 0.50192273 0.53522264

Modern* 0.68410368 0.68784647 0.63593012 0.69723052

5.00%

15.00%

25.00%

35.00%

45.00%

55.00%

65.00%

75.00%

No impact found on fertility and use of modern contraception

3 indicators:

1. Pregnancy since baseline interview

2. Ever used modern contraceptive method

3. Ever used modern contraceptive method – adjusted for baseline responses

DID THE INTERVENTION IMPACT CHW BEHAVIOR OR MOTIVATION?

We found no evidence that either of the interventions impacted the following self-reported indicators:

1. Average number of hours spent on health activities in a week

2. Number of household visited in the past month3. CHWs seek advice from peers4. Measures of satisfaction and motivation