Post on 03-Jan-2016
2 0 1 5 E A S T A F R I C A E V I D E N C E S U M M I T J U LY 8 - 9 , 2 0 1 5 | N A I R O B I , K E N YA
COMMUNITY PERFORMANCE-BASED FINANCING IMPACT EVALUATION DISSEMINATION MEETINGJEANINE UMUTESI CONDO
PRINCIPAL COLLABORATORS AND INSTITUTIONS2
MOH-RBC-MCCHFidel Ngabo Cathy Mugeni
University of Rwanda/CMHS/SPHIna R. KalisaJames HumuzaJeanine CondoVedaste Ndahindwa
The World BankGil ShapiraNetsanet W. Workie
BACKGROUND AND JUSTIFICATION 3
• 2006-2008 Health Center PBF impact evaluation results show PBF had a significant impact on:• Increasing institutional deliveries• Improving quality of prenatal care• Increasing child preventive care utilization
• Vaccination, growth monitoring
• VCT for couples and more for discordant couples
• But….PBF did not have a significant impact on:• Prenatal / post natal care utilization• Modern contraceptive use• Short term malnutrition
COMMUNITY PBF PROGRAM DESIGN4
• Model 1: Conditional in-kind incentives for women:• Aimed at increase early prenatal care utilization, facility
deliveries, and postnatal care in order to diagnose and treat preventable threats
• This demand-side model endows mothers with gifts for:• Timely antenatal consultation: first visit in the first four months of
pregnancy
• In-facility delivery
• Timely postnatal care: consultation within the ten days after delivery
COMMUNITY PBF PROGRAM DESIGN5
• Model 2: Supply-side financial incentives for CHW Cooperatives:
• Aim: (i) improve quality of data reported at the sector level, (ii) increase utilization of priority maternal and child health services, and (iii) improve motivation and behavior of CHWs.
• Pay for reporting: Quarterly payment based on the timely submission of quality data reports related to 29 indicators.• Implemented nationally Not evaluated by this study
• Pay for performance: Offers financial rewards directly to CHWs.
COMMUNITY PBF PROGRAM DESIGN6
Incentivized indicators Unit Fees (USD) (2010-2014/15)
2010 2011 2012 2013 2014
1. Deliveries: Women accompanied/referred to HC for assisted deliveries
2.73 1.37 0.99 0.99 0.99
1. Antenatal Care: Women accompanied/referred to HC for prenatal care within first 4 months of pregnancy
2.24 1.12 0.81 0.81 0.81
1. Nutrition Monitoring: % of children monitored for nutritional status (6 -59 months)
3.240.56
50.43
30.433 0.433
1. Family Planning: % of regular users using long term methods (IUD, Norplant, Surgical/NSV contraception)
2.11 1.06 0.77 0.77 0.77
1. FP: new users referred by CHWs for modern family planning methods
2.9 1.45 1.05 1.05 1.05
RESEARCH OBJECTIVES AND QUESTIONS
7
• Main Research Objective• The main objective of this study is to evaluate the effects of
demand-side (in-kind incentives) and supply-side (financial incentives) on health services utilization and health outcomes—The study attempts to isolate the causal impact of the incentives packages on health outcomes.
RESEARCH QUESTIONS8
• The primary research questions of the IE are:
• Do the demand and supply-side incentives affect:
• The number of woman who receive antenatal care during the first 4 months of pregnancy?
• The number of antenatal care visits?
• The number of facility deliveries?
• The number of woman-child pairs seen during postnatal care follow up care?
RESEARCH QUESTIONS 9
• Is there “a multiplicative effect” on outcomes when demand and supply-side incentives are combined?
• Do the supply-side incentives to CHWs increase the:
• Use of modern contraceptives services?
• Increase the time between births?
• Improve nutritional status in under-fives?
• Do the supply-side incentives to CHWs affect their motivation and behaviors?
Study Design and IE Surveys
EVALUATION DESIGN
• The study is a prospective, randomized impact evaluation
• CPBF interventions are randomly assigned at the sector level into four study arms:o D: Demand-side in-kind incentives to women
o S: Supply-side financial incentives to CHW cooperative
o DS: Demand-side incentives to women + Supply-side
incentives to CHW cooperative
o C: Comparison group
11
EVALUATION DESIGN
• Treatment group : sample of sectors where the interventions were implemented starting in 2010.
• Comparison group: sample of sectors where the interventions were not implemented until 2013
• In demand side and comparison sectors, CHW cooperatives received the average incentive payments distributed where the supply side intervention was implemented
12
SAMPLING DESIGN
Study Arm # Sectors# CHW
Cooperatives
# HH#
CHWs
D: In-kind Incentives50 50 600 1200
S: CHW incentives50 50 600 1200
DS: In-kind + CHW incentives
50 50 600 1200
C: Average financing50 50 600 1200
TOTAL200 200 2400 4800
13
Baseline Planned Sample Sizes by Study Arm
SAMPLING DESIGN14
Map of Sectors by Study Arm
SAMPLING STRATEGY AT BASELINE
• Cooperative Level• All 200 in 200 sectors
• Household Level• The household with the most recent birth in the
village (within last 4 months) was selected for the household interview for each of the 2400 randomly selected villages
• CHW individual Level• 2 CHWs selected for CHW individual questionnaire
for each of the 2400 randomly selected villages• Forcibly include MNH CHW (ASM)• Randomly select second
15
SAMPLING STRATEGY AT ENDLINE
• The endline survey covered the same sectors and villages as the baseline
• Changes from baseline survey:
• Doubling households interviewed
• Reduce by half CHWs interviewed
• Add health center assessments
16
SAMPLING STRATEGY AT ENDLINE
• CHWs cooperative• Same as baseline
• Households survey• Old sample: Tracking of households interviewed in baseline• New sample: Households with the most recent birth in each
of the villages visited during the baseline
• CHWs survey• Interview only the CHW in charge of maternal and neonatal
heath (ASM) in each village• Health facility survey
• A health facility assessment • A health worker survey (ANC and child curative care)• Patient exit interviews (ANC and child curative care)
17
CPBF IE DATA-AT BASELINE
• Baseline data collection:• Fielded from February to May 2010
Final baseline sample size
18
Study Arm # Sectors
#Households
#cooperatives
# CHWs
Demand-side 50 600 49 1,162
Supply-side 50 600 50 1,184
Demand- and Supply-side
50 600 50 1,186
Control 48 576 48 1,136
Total 198 2376 197 4,668
CPBF IE DATA-AT ENDLINE
• Endline data collection:• Fielded from November 2013 to June
2014
Final Endline sample size
19
Study Arm #Sectors
# Baseline HH
# New HH
#CHWs cooperatives
# CHWs
Demand-side
50 549 594 50 569
Supply-side 50 536 591 50 559Demand- and Supply-side
50 533 593 50 556
Control 48 539 565 47 536Total 198 2157 2343 197 2,220
CPBF IE DATA-AT ENDLINE
Quality checks:
During data collection: regular spot checks and supervision by field coordinator and research team
After data collection: 10% revisit of households (98% of households were really interviewed)
20
VALIDATION OF IE DESIGN: BASELINE BALANCE CHECK
T-tests were used to assess the difference
between each one of the three treatment arms
and the control
F-tests were used to test the hypothesis that the
variable means are identical among all 4 study
arms.
21
COMMUNITY PBF RESULT22
COMPARISON OF BASELINE AND ENDLINE SAMPLES OF WOMEN WITH RECENT BIRTHS
23
Baseline Endline T-test
Mean Mean Mean(Std.) (Std.) p-value
Number of households members 1.00 1.00 0.00 All 5.02 5.16 0.14**
(2.00) (2.00) 0.01 Under 5 1.62 1.06 -0.56***
(0.66) (0.84) 0.00Characteristics of head of household 1.00 1.00 0.00 Gender 0.94 0.93 -0.01
(0.26) (0.26) 0.37 Age 34.23 35.60 1.37***
(11.13) (11.73) 0.00 No school 0.20 0.18 -0.03**
(0.42) (0.40) 0.03 Primary school 0.64 0.69 0.05***
(0.50) (0.49) 0.00 More than primary school 0.16 0.14 -0.02*
(0.39) (0.37) 0.07Ownership of land 0.94 0.92 -0.02**
(0.27) (0.31) 0.01Characteristics of core woman 1.00 1.00 0.00 Age 28.15 28.91 0.76***
(6.25) (6.83) 0.00 Married 0.91 0.89 -0.02*
(0.31) (0.33) 0.08 No school 0.17 0.13 -0.04***
(0.39) (0.36) 0.00 Primary school 0.69 0.72 0.03**
(0.49) (0.47) 0.02 More than primary school 0.14 0.14 0.01
(0.38) (0.38) 0.59 Total number of births 3.01 3.07 0.06
(2.20) (2.19) 0.33
CHARACTERISTICS OF CHWS AT ENDLINE24
Mean Std. Obs.
Female 1.00 0.07 2204
Age 40.31 9.05 2205
From the village 0.37 0.50 2204
Number of years lived in the village 20.82 12.74 2185
Married 0.85 0.38 2205
Educational level: 1 0 2205
Primary school 0.75 0.46 2185
Post-primary school 0.10 0.32 2185
Secondary school or more 0.15 0.38 2185
Literacy 0.99 0.09 2192
Number of years worked as a CHW 4.26 2.35 2200
IMPACTS ON HEALTH SERVICE UTILIZATION OUTCOMES: TIMELY ANC, TIMELY PNC AND IN-FACILITY DELIVERY
• Indicators targeted by the Demand-side intervention• Timely PNC was not targeted in the supply-side
intervention
• Sample: 2334 women who were pregnant or gave birth shortly before the endline survey
Control Demand Supply D + S
Timely ANC 0.70958766 0.80558442 0.7412024 0.80364976
52.50%
57.50%
62.50%
67.50%
72.50%
77.50%
82.50%
87.50%
Impact on first ANC visit during the first 4 months of the pregnancy:
• A positive and significant impact of the demand-side in-kind incentives of about 10 percentage points
• The CHW incentives are not found to have a significant effect• No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms
Treatment Arm
Impact on having at least one ANC visit:
• Not targeted by the program!• Not impacted by the program but this outcome is already almost universal.
Control Demand Supply D + S
1 ANC 0.99085053 0.9881807 0.98708024 0.98712576
52.50%
57.50%
62.50%
67.50%
72.50%
77.50%
82.50%
87.50%
92.50%
97.50%
Impact on completing at least 4 ANC visits:
• Not targeted by the program!• Higher in the intervention sectors, but not statistically significant at the 10% level
Control Demand Supply D + S
4 visits 0.38929389 0.44691936 0.4273102 0.4472739
22.50%
27.50%
32.50%
37.50%
42.50%
47.50%
Impact on skilled-attended in-facility deliveries:
• No statistically significant difference between the treatment arms• Rate has increased substantially in the duration of the study for other reasons
Control Demand Supply D + S
Delivery 0.93975718 0.94941371 0.95962508 0.93719467
52.50%
57.50%
62.50%
67.50%
72.50%
77.50%
82.50%
87.50%
92.50%
97.50%
Impact on PNC within the 10 days after delivery:
• A positive and significant impact of the demand-side in-kind incentives of about 7 percentage points
• The CHW incentives are not found to have a significant effect• No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms
Control Demand Supply D + S
Timely PNC 0.12627086 0.19873905 0.10468981 0.19728536
2.50%
7.50%
12.50%
17.50%
22.50%
RESULTS ARE ROBUST TO THE FOLLOWING CHECKS:
1. Regressions with controls:• Detected impacts are not due to difference in observable
characteristics between the groups.• Randomization was done at the sector level -> controlling for
individual-level characteristics (for example: age, marital status, education level, household characteristics, distance from health center,…)
• Include characteristics which were not perfectly balanced
2. Excluding “misclassified” sectors:• 3 sectors have been misclassified during the transition to the new
computerized data system, although cooperatives were not informed
3. Difference-in difference approach:• Account for fixed unobservable characteristics
IMPACT ON FERTILITY AND FAMILY PLANNING
• The supply-side program incentivizes CHW cooperative for new and regular users of modern contraceptives
• Sample: 2,157 “baseline women”• Were pregnant or gave birth shortly before the baseline
survey• Re-interviewed in 2014• Not the optimal sample for evaluating impact on fertility
outcomes
Control Demand Supply D + S
Pregnancy 0.53006919 0.54973127 0.58538525 0.5496025
Modern 0.51592882 0.57089905 0.50192273 0.53522264
Modern* 0.68410368 0.68784647 0.63593012 0.69723052
5.00%
15.00%
25.00%
35.00%
45.00%
55.00%
65.00%
75.00%
No impact found on fertility and use of modern contraception
3 indicators:
1. Pregnancy since baseline interview
2. Ever used modern contraceptive method
3. Ever used modern contraceptive method – adjusted for baseline responses
DID THE INTERVENTION IMPACT CHW BEHAVIOR OR MOTIVATION?
We found no evidence that either of the interventions impacted the following self-reported indicators:
1. Average number of hours spent on health activities in a week
2. Number of household visited in the past month3. CHWs seek advice from peers4. Measures of satisfaction and motivation