Measuring the strength of community case management implementation: Validation of mobile phone interviews with community health workers

in Malawi Elizabeth Hazel1, Agbessi Amouzou1, Lois Park1, Benjamin Banda2, Tiyese Chimuna3, Tanya Guenther4, Humphreys

Nsona5, Cesar Victora6 and Jennifer Bryce1

1Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, United States, 2National Statistics Office, Zomba, Malawi, 3Save

the Children, Lilongwe, Malawi, 4Save the Children, Washington, DC, United States,5Ministry of Health, IMCI Unit, Lilongwe, Malawi, 6Universidade Federal de Pelotas, Pelotas, Brazil

#624

Introduction Since 2008, Health Surveillance Assistants (HSAs) have provided

community case management (CCM) of childhood malaria, diarrhea and

pneumonia episodes in selected areas of Malawi. In order to gauge the

level of program implementation and provide continuous feedback for

program improvement, implementers and evaluators require high-quality

and inexpensive measures of implementation strength. However, the

routine CCM monitoring and evaluation system is still being scaled-up in

many areas. An alternative method for measuring implementation

strength is needed between now and mid-2014 to provide information on

how the program is performing on the ground.

Study Aim To test and validate a method for collecting implementation

strength data at the community level through mobile phone interviews

with HSAs.

Methods

We used selected CCM implementation strength indicators for validation.

The HSA mobile telephone responses for these indicators were validated

against routine monitoring data, supervisor/mentoring checklists and

direct observation at the HSA village clinic. We randomly selected 250

CCM-trained HSAs in two districts.

Data collection procedures:

1. Interviewers visited the catchment health center to review supervisor

records and monitoring data for the randomly selected HSAs.

2. HSAs were interviewed using a standardized questionnaire over

mobile phone. Per IRB requirement, all HSAs were consented to the

study objectives prior to interview.

3. Interviewers immediately followed-up for direct observation of the HSA

village clinic.

4. After interviews, staff worked with HSAs to investigate the reason for

any discrepancy.

Analysis: We calculated the sensitivity and specificity of the mobile phone

method for each of the key indicators.

Results We were able to reach 83% (200/241) of the HSAs by mobile phone.

Sensitivity and specificity of the mobile phone method was good (all

above >80%).

Lower for supervision and mentoring indicators.

All interviewed HSAs were working and trained in CCM. 90% had

treated a child in the previous 7 days.

One third had received any supervision/mentoring in the previous three

months.

About half had all key CCM drugs at the time of assessment and

minimum required stocks and half had no stockouts in the previous 3

months.

HSA reporting error during the telephone interview contributed to half

of errors found for drug stockouts data.

Errors on reporting forms used for validation contributed to half of

discrepancies for supervision.

Unable to determine error source for half of mentoring discrepancies.

7 USD per interview. Does not include staff time.

Airtime includes interview time and any airtime needed for logistics.

Summary of findings • Most HSAs were available for interview by mobile phone.

• Responses were accurate; slightly lower accuracy for supervision

and mentoring but still reasonable.

• Most reporting discrepancies for supervision were actually errors in

the reporting forms used to validate the HSA responses.

• Implementation strength indicators were low especially for

supervision. • Cost of method was low cost relative to conducting inspection visits.

Limitations • HSAs were informed prior to interview that their responses would be

validated.

• Data on the reasons for HSA reporting discrepancy should be

interpreted with caution.

Conclusions • Mobile phone method is inexpensive, feasible and produces highly

accurate results.

• Method is a good option for measuring community-based program

implementation strength in areas where mobile phone coverage is

adequate and the routine monitoring system is not yet scaled-up to

quality.

This study was supported by the Bill and Melinda Gates Foundation through a grant to the World Health Organization and the American people through the United States Agency for International Development (USAID) and its Translating Research into Action (TRAction). TRAction is managed by University Research Co., LLC (URC) under the Cooperative Agreement Number GHS-A-00-09-00015-00. For more information on TRAction's work, please visit http://www.tractionproject.org/.

HSA with drug box Taken with permission Source: Kate Gilroy 2009

Implementation strength indicators: reported vs. observed/validated