Monitoring and Evaluation: FAMILY PLANNING PROGRAMS

Session Objectives

• Be able to apply basic M&E concepts (frameworks, indicators, etc.) to family-planning programs

• Be able to summarize the main issues in M&E of family-planning programs from a post-Cairo perspective.

• Be able to summarize the emerging issues for M&E of family-planning programs in high HIV prevalence countries.

Session Overview

• Family-planning frameworks

• M&E implications of the Cairo agenda

• Contraceptive prevalence and unmet need

• Monitoring quality of care

• Evaluating the impact of quality

• Family planning and HIV

Family Planning Frameworks

Conceptual Framework for FP Demand and Program Impact on Fertility

Development programs

Societal & individual factors

FP supply FP supply factorsfactors

Value & demand for children

Service outputs:Service outputs:

• AccessAccess

• QualityQuality

• AcceptabilityAcceptability

FP demand

• Spacing

• Limiting

Other intermediate variables

Contraceptive practice

Service Utilization

Fertility

• Wanted

• Unwanted

Other health & social improvements

Source: Bertrand, Magnani, and Rutenberg, 1996.

Conceptual framework of family planning supply factors

External Development Assistance

Political and Administrative System

• Political support

• Resource allocations

• Legal code / regulations

Larger societal & political factors

FP Organizational Structure

• Service infrastructure

• Sectoral integration

• Delivery strategies

• Public-private partnerships

Operations

• Management & supervision

• Training

• Commodity acquisition & distribution

• IEC

• Research & evaluation

Service OutputsService Outputs

• AccessAccess

• QualityQuality

• AcceptabilityAcceptability

Source: Bertrand, Magnani, and Rutenberg 1996

Applying the frameworks for FP M&E• Inputs, e.g.

– Types and levels of resources

– Qualified personnel

– Unit and total costs of program resources

• Outputs – functional areas, e.g.– People trained

– Performance of people trained– Cost per person trained

Applying the frameworks for FP M&E• Outputs – Service outputs, e.g.,

– Service delivery points providing FP services– Quality of FP services– Cost of increasing access/quality of FP services

• Outputs – Service utilization– New FP acceptors, Couple Years of Protection

(CYP)– Returning clients– Cost of increasing CYP, etc.

Applying the frameworks for FP M&E

• Outcome – intermediate outcomes– Contraceptive prevalence rate (CPR)– Unmet need– Costs associated with increased CPR

• Outcome – long term outcome– Fertility rates– Unintended pregnancy– Costs of changes in fertility, unintended

pregnancy

Indicators for FP programs

• See Bertrand and Escudero, 2002, Compendium of Indicators for Evaluating Reproductive Health Programs, 2 volumes– Indicators that crosscut program areas– Indicators for specific program areas

What is different about M&E of FP programs?

• Basic principles are the same as in other health programs

• Outcomes relatively well-defined, focused, and measurable

• Long history of data collection on FP outcomes through WFS, DHS – document global trends

• Attempts to link outcomes to program outputs - evidence of program effects

Programme of Action adopted at ICPD, Cairo 1994

Traditional (pre-Cairo) focus of FP program M&E

• Demographic impact

• Focus on married women

• Availability of services

• Contraceptive adoption (new users)

• Characteristics of women

• Cross-sectional measurement

Cairo: Objectives of FP Programs• To help couples and individuals meet their

reproductive goals• To prevent unwanted and high-risk pregnancies• To make quality FP services affordable,

acceptable, and accessible• To improve the quality of family planning IEC,

counseling, and services• To increase the participation and sharing of

responsibility of men in FP• To promote breastfeeding to enhance birth

spacing

Exercise 1

• Discuss the implications of the Cairo programme of action for M&E of FP programs. Identify 3 or more ways in which the traditional focus of FP programs listed on the earlier slide should change to respond to the Cairo agenda. What are the implications of these changes for M&E?

Contraceptive Prevalence Rate (CPR)

• Percentage of (married) women of reproductive age (15-49) who are currently using a contraceptive method.

Unmet Need for Family Planning

• Percentage of fecund women exposed to the risk of pregnancy who say they want to wait at least two years for another birth (spacing) or do not want any more children (limiting), but are not currently using a method of contraception.

Related Indicators

• Demand for FP = % (married) women using FP + % (married) women with unmet need for FP

• Percentage of demand satisfied = % (married) women using FP / % (married) women with demand for FP

Unmet Need Exercise

CPR vs Unmet Need

CPRCPR• Relatively simple to

define• Uni-dimensional• Consistency over time• Does not capture

concept of meeting needs

Unmet NeedUnmet Need• Relatively complex to

define• Multi-dimensional –

demand & use• Definition has evolved• Captures concept of

meeting need

Monitoring Quality of Care

What is Quality of Care in FP?

• General, loosely-defined concept

• Different people define quality in different

ways

• Multi-dimensional

• Appropriate standards against which to

measure quality vary

Bruce-Jain Framework

• Choice of contraceptive methods

• Information given to users

• Provider competence

• client/provider relations

• re-contact and follow-up mechanisms

• appropriate constellation of services

Indicators for QOC

• No single indicator can capture the different components of QOC

• Indicators need to be adapted to specific program context and priorities

• Shortlist of 24 QOC indicators (see Bertrand and Sullivan, Evaluation Bulletin No. 1, Table 1 page 2).

Facility Surveys for QOC Indicators

• Situation Analysis

• MEASURE Evaluation Quick Investigation of Quality (QIQ)

• MEASURE DHS+ Service Provision Assessments (SPA)

• DHS service availability modules and community surveys (SAM)

Some Data Collection Issues

• Small sample sizes for FP clients, especially in low prevalence countries

• Observation in clinics that use a client flow approach

• Sampling

• Courtesy bias and hawthorn effects

• Unit of analysis (client, provider, facility)

Case Study: QOC in Turkey

Turkey’s Strategic Framework

I n t e r m e d i a t e R e s u l t 1S t r e n g t h e n e d s u s t a i n a b i l i t y

o f F P / R H p r o g r a m

I n t e r m e d i a t e R e s u l t 2E x p a n s i o n o f h i g h q u a l i t y F P / R H s e r v i c e s

i n t h e p u b l i c a n d p r i v a t e s e c t o r s

S t r a t e g i c O b j e c t i v eI n c r e a s e d u t i l i z a t i o n o f F P / R H s e r v i c e s

The Quality Index

• Method availability

• Availability of trained personnel

• Perceived quality of FP counseling

• Adequate infection-prevention measures

• Availability of IEC materials

• Physical access to FP services

Data Source

• Istanbul Quality Surveys– Facility inventory– Client exit interviews

• Based on MEASURE Evaluation QIQ

The Quality Index

• Sum of scores from the 6 components (range 0-6)

0 2 4 6

MOHhospitals

MCH/FPcenters

Healthcenters

SSKhospitals

Privatehospitals

Method Availability

• Proportion of facilities that distribute or prescribe 3 or more modern FP methods

0 0.2 0.4 0.6 0.8 1

MOHhospitals

MCH/FPcenters

Healthcenters

SSKhospitals

Privatehospitals

Perceived Quality of FP Counseling

• Proportion of clients who report– they were seated– had sufficient time with

the provider

– clearly understood the information provided

0 0.2 0.4 0.6 0.8 1

MOHhospitals

MCH/FPcenters

Healthcenters

SSKhospitals

Privatehospitals

Adequate Infection Prevention Measures

• Proportion of facilities that meet the following standards :– Plastic bucket for CL

solution

– Unused IUD kits kept sterile

– Medical waste kept in leak-proof containers with lids

– Appropriate containers for sharp objects

0 0.2 0.4 0.6 0.8 1

MOHhospitals

MCH/FPcenters

Healthcenters

SSKhospitals

Privatehospitals

Evaluating the impact of quality of care

Quality of services •Choice•Information to users•Provider competence•Client-provider relations•Follow-up•Appropriate constellation of services

Other factors

Acceptance

Continuation

Contraceptive prevalence

Other proximate

determinants

Fertility

Known effects

Hypothesized effects

Source: Jain, 1989

Framework for links between quality of Framework for links between quality of family planning services and fertilityfamily planning services and fertility

Outcomes of interest• Intention to use• Contraceptive adoption• Contraceptive discontinuation

– Failure– Switching– Stopping

• Current contraceptive use– Contraceptive choice

• Unwanted pregnancy

Examples of impact studies

• Peru (Mensch, et al., 1996)

• Morocco (Steele, et al., 1999)

• Bangladesh (Koenig et al., 1997)

Morocco Study Design (1)

• To explore whether the service environment in which a woman resides affects adoption and continuation of the pill

• Linkage of 1995 Demographic and Health Survey calendar with 1992 DHS Service Availability Module

• Multi-level hazards models with contraceptive adoption and discontinuation as outcomes

• 862 births and 775 episodes of pill use in 107 clusters

Morocco Study Design (2)

• Explanatory factors - Individual and Community– age, education, residence, community drinking

water & toilet facilities, principle economic activity

– Contraceptive intention (discontinuation)– Breastfeeding status, last child wanted

(adoption)

• Explanatory factors – Program– Public health center <10km, pharmacy <5km,

outreach services, 3+ methods available at clinic

– Source of pills (discontinuation)

Predicted percentage of women adopting a modern contraceptive method within 12 months of giving birth by service factors

0

10

20

30

40

50

60

70

%

Yes No <3 3+

Health center <10km No. methods offered at closest set of facilities

Predicted 12-month pill discontinuation rate by reason and service factors, Morocco

0%

5%

10%

15%

20%

25%

30%

35%

Government Pharmacy orother

Yes No Yes No

Failure Desired pregnancy Side effects/ health concerns Other method-related

Health center within 10KMSource Pharmacy within 5KM

Main Findings: Morocco

• Relatively strong service effects on post-partum adoption

• Service availability associated with both adoption and discontinuation

• Number of methods available only associated with adoption

• Users of government sources have lower discontinuation

Limitations of Impact Studies

• Measures of quality inadequate (often limited to access and method choice)

• Cross-sectional designs (endogenous inputs)

• Linking individual and program data (geographic boundaries, service environment vs. individual service experience)

Emerging areas: FP/HIV linkages and integration

Context

• Considerable progress in preventing unwanted pregnancy but unmet need remains substantial

• Rapid increases in HIV in many countries

• Changing funding focus to HIV from FP

• Integrated vs. vertical programs

Synergies between FP and HIV programs

• Both are central to reproductive health

• “ABC” messages in HIV programs also relevant to FP programs

• Youth programs that encourage responsible sexual behavior prevent both HIV and teen pregnancy

• Strong RH policies support both HIV and FP programs

Dual Protection

• Abstinence

• Monogamous couples using effective contraception

• Correct and consistent condom use

FP in high HIV-prevalence countries

• Relationship between HIV and fertility desires• FP/RH needs differ for:

– HIV- concordant monogamous couple– HIV- concordant non-monogamous couples

– HIV discordant couples

– HIV+ concordant couples

• HIV counseling in FP services

FP and VCT

• FP counseling opportunity for VCT or general HIV counseling and VCT referral

• VCT services could include FP services or FP counseling and referral

• Concern over unintended consequences of integration– Provider burn-out

– Discourage FP clients

– Quality of integrated vs. vertical FP & VCT services

FP and PMTCT

• Averts child infections by preventing unintended pregnancies among HIV+ women

• PMTCT programs provide opportunity for prenatal FP counseling and post-partum contraceptive use

• Reduced breastfeeding by HIV+ mothers will lead to shorter birth intervals in the absence of FP

FP Counseling of PMTCT clients, Zambia

0

10

20

30

40

50

60

%

ANC visit 3 months PP 6 months PP

pregnant women HIV+ HIV-

Source: Rutenberg & Baek, 2004

PMTCT-Client FP Use 6 Months Post-partum, Zambia

05

1015202530354045

%

Modern Condom Sex active, nomethod

HIV+ HIV-

Source: Rutenberg & Baek, 2004

HIV Counseling in FP Sessions, Uganda

0

5

10

15

20

25

30

3540

% F

P s

essi

ons

HIV needsassessed

HIV risk discussed Dual protectionproposed

Baseline Follow-upSource: Rutenberg & Baek, 2004

Exercise 3

• Select an area of FP/HIV integration (e.g. PMTCT, VCT, HIV counseling in FP etc.). – Develop a basic input-output-outcome-impact

framework for a simple program in this area. – Suggest 3-6 indicators to monitor your program. – What data sources would you propose to collect

these indicators?