EVALUATION OF THE CAPACITY...

EVALUATION OF THE CAPACITY PROJECT ASSESSING PROGRESS ON HRH ISSUES

AUGUST 2008 This publication was produced for review by the United States Agency for International

Development It was prepared by Demissie Habte and William Emmet through The Global

Health Technical Assistance Project.

Front cover photograph: Clinical officer posted through the Capacity Projects Emergency Hiring Plan (center) attends to mother and child, with translator (shown at left), Lopiding Sub-District Hospital, Kenya. Credit: IntraHealth International/Capacity Project, 2006.

EVALUATION OF THE CAPACITY PROJECT ASSESSING PROGRESS ON HRH ISSUES

DISCLAIMER

The authors views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

This document (Report No. 08-001-89) is available in printed or online versions. Online documents can be located in the GH Tech website library at

www.ghtechproject.com/resources.aspx. Documents are also made available through the Development Experience Clearinghouse (www.dec.org). Additional information can be obtained from

The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100

Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901

[email protected]

This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under

USAID Contract No. GHS-I-00-05-00005-00.

Evaluation of the Capacity Project i

ACKNOWLEDGEMENTS

The Capacity Project Evaluation Team thanks the United States Agency for International Development

and the Global Health Bureau Office for Population and Reproductive Health/Service Delivery Improvement Division for commissioning this evaluation of the Capacity Project and for selecting us to undertake this challenging task. We would also like to acknowledge the inspiration provided by the Joint

Learning Initiative (JLI) as the technical and conceptual framework for the Capacity Projects terms of reference. We appreciate JLIs providing the evaluation team with permission to reproduce the diagram in Figure 4 as a clear and concise depiction of the linkage between health workers and health outcome. The

time, patience, goodwill, and depth of knowledge of the 144 persons interviewed as part of this evaluation made it possible for the team to gather information and understand the significance of the many and diverse reports and technical interventions associated with Capacity Projects multiple technical

initiatives. The willingness of Capacity Project staff in North Carolina and throughout the world to share with us their thoughts on the projects development and their perspectives on the future was of primary importance in assisting us to understand the complexities associated with addressing the linkages between

human resources for health and systems development issues. The Capacity Projects patience and good humor in fully responding to our repeated calls for clarification and information throughout the two-plus months of this evaluation made it possible for the evaluation team to feel less guilty as we increased the

intensity of our inquiries. The importance of the contribution of United States Government officials within USAID, the United States Department of State, and at overseas missions to this evaluation in terms

of framing its technical methodology and in contributing to its findings cannot be overstated. Finally, the evaluation team would like to acknowledge our appreciation for the time set aside by the many senior government officials in Rwanda and Uganda to respond to our request for interviews. Information

provided from these host country counterparts enabled the evaluation team to cut through superlatives and to focus on the reality.

As a final note, the two-consultant team employed under this assignment expresses its thanks and appreciation to the staff of GH Tech for having provided us with a positive working environment while in

Washington and for providing us with administrative assistance of the first order. While many GH Tech staff have contributed to this effort, we would like to single out Ms. Jennifer Hoeg for special notice and to thank her for having played an essential role in scheduling our interviews and in making certain that we

did not forget to show up at the appointed hours.

ii Evaluation of the Capacity Project

Evaluation of the Capacity Project iii

ACRONYMS

AA Associate award

ART Antiretroviral treatment/antiretroviral therapy

CHA Christian Health Association

CHW Community health workers

CPD Continuing professional development

DHS Demographic and Health Survey

ECSA Commonwealth Regional Health Community for East Central and Southern Africa

EHP Emergency Hiring Plan

FBO Faith-based organization

FP/RH Family planning and reproductive health

FTE Full-time equivalent

GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria

GH Bureau for Global Health

GH TECH Global Health Technical Assistance Project

GRC Global Resource Center

GHWA Global Health Workforce Alliance

HAF Human Action Framework

HCD Human Capacity Development

HIDN Office of Health, Infectious Diseases, and Nutrition (USAID)

HIV/AIDS Human Immunodeficiency Virus / Acquired Immune Deficiency Syndrome

HMIS Health management information system

HRH Human resources for health

HRIS Human resources information system

HRM Human resources management

IMA IMA World Health

IR Intermediate result

IUD Intrauterine device

JHPIEGO An affiliate of The Johns Hopkins University

JLI Joint Learning Initiative on Human Resources for Health and Development

LATH Liverpool Associates in Tropical Health

LFP Learning for Performance

LOE Level of effort

LOP Life-of-project

LWA Leader with Associate Award

MCH Maternal and child health

MSH Management Sciences for Health

NGO Nongovernmental organization

OGAC Office of the Global AIDS Coordinator, State Department

OHA Office of HIV/AIDS (USAID)

iv Evaluation of the Capacity Project

PAHO Pan-American Health Organization

PEPFAR Presidents Emergency Plan for AIDS Relief

PMP Performance Monitoring Plan

PMTCT Prevention of mother to child transmission

PRH Office of Population and Reproductive Health (USAID)

RFA Request for application

SDI Service Delivery Improvement Division (USAID)

TRG Training Resources Group

UNFPA United Nations Population Fund

USAID United States Agency for International Development

UNDP United Nations Development Programme

USG United States Government

WHO World Health Organization

Evaluation of the Capacity Project v

CONTENTS

ACKNOWLEDGEMENTS ..................................................................................................................... i

ACRONYMS..........................................................................................................................................iii

EXECUTIVE SUMMARY ................................................................................................................... vii

1. CAPACITY PROJECT OVERVIEW ...................................................................................... vii

2. CAPACITY PROJECT EVALUATION: SCOPE OF WORK AND METHODOLOGY .........vii

3. FINDINGS ................................................................................................................................viii

4. CONCLUSIONS AND RECOMMENDATIONS FOR THE FUTURE ...................................xi

EVALUATING THE CAPACITY PROJECT: ASSESSING PROGRESS ON HRH ISSUES ......... 1

1. INTRODUCTION ....................................................................................................................... 1

2. PROJECT FINDINGS AND ACCOMPLISHMENTS .............................................................. 7

CONCLUSIONS AND RECOMMENDATIONS FOR THE FUTURE ............................................. 23

vi Evaluation of the Capacity Project

Evaluation of the Capacity Project vii

EXECUTIVE SUMMARY

1. CAPACITY PROJECT OVERVIEW

In September 2004, the United States Agency for International Development (USAID) awarded the

Human Capacity Development Project (later renamed the Capacity Project), a five-year Leader with Associate Award (LWA) cooperative agreement, to IntraHealth International, the leader of a consortium of nine partners, seven of which have remained with the partnership throughout the projects initial four

years.

In recognition of the crisis facing the health workforce in underserved areas throughout the developing world, IntraHealth and its consortium are charged with improving human resources for health (HRH) to

provide quality services.

With responsibility for project administrative and technical oversight assigned to the USAID Global Health Bureaus Office of Population and Reproductive Health/Service Delivery Improvement Division (GH/PRH/SDI), the project was established with a funding ceiling (for the leader award) of $250 million,

with $30 million (largely from PRH) designated as core funding and the balance potentially available through USAID mission office field support. To date, the Presidents Emergency Plan for AIDS Relief (PEPFAR) has provided approximately 84 percent of the obligated $70,000,000 in field support.

Accordingly, USAIDs project management team includes technical advisors from both PRH/SDI and the Global Health Bureaus Office of HIV/AIDS (OHA).

The Capacity Projects objective is to improve human capacity to implement quality health programs in developing countries. In responding to this objective, the projects progress is measured by its success in

responding to three intermediate result areas and four crosscutting areas:

Intermediate Result 1: Improved workforce planning, allocation, and utilization (workforce policy and planning)

Intermediate Result 2: Improved worker skills (workforce development)

Intermediate Result 3: Strengthened systems for sustained health worker performance (workforce support).

Crosscutting Areas (as identified by the project):

Global Partnering: Facilitating international cooperation on HRH

Knowledge Management: Improving the capacity of country-specific HR planners and managers to access and use HRH-related data resources

Gender: Strengthening country-specific recognition of the importance of gender equity as an essential component in national system planning and policy for HRH

Faith-Based Organizations (FBO): Strengthening the capacity of FBOs to contribute to

solutions associated with HRH issues.

2. CAPACITY PROJECT EVALUATION: SCOPE OF WORK AND METHODOLOGY

Scope of Work

The evaluation focused on an assessment of the progress during the first four years of the projects five-year life span, and specifically to:

Review technical and programmatic strengths, weaknesses, successes, and constraints, identifying contributing factors

viii Evaluation of the Capacity Project

Assess organizational structure and management

Identify initiatives that warrant additional investment during the final year of the project, and into the future with reference to continued advancement in addressing existing and emerging challenges.

Methodology

In responding to the scope of work, the evaluation employed a multistep methodology:

A pre-evaluation team planning meeting with senior staff from PRH/SDI and OHA and from GH Tech

A document review including workplans, minutes of management meetings, study reports, and

technical briefs

A review of a self-assessment report prepared by the Capacity Project staff during early 2008 at the request of PRH/SDI and OHA

A review of an assessment by participating missions designed, executed, and compiled by PRH/SDI and OHA

A review of the various websites on HRH in which the project has been taking a lead role

Key informant interviews, either face-to-face or through teleconference, using a standardized

questionnaire (adapted to different groups) to conduct an in-depth qualitative interview with 144 persons around the world, representing a wide array of stakeholders including Capacity Project partners and international stakeholders

A two-and-a-half-day visit to Capacity Project Headquarters, including full briefings on project

activities and interviews with key project management staff

Site visits to Rwanda and Uganda, two Capacity Project participating countries selected by USAID/GH/PRH/SDI and OHA: Nine days were devoted to each of the two selected countries. During these visits, the evaluation team interviewed resident staff of the Capacity Project and the

USAID missions, senior government staff, and staff associated with USAID collaborating agencies. In addition, the evaluation team undertook extensive field trips in an effort to gain a perspective on

Capacity Project field activities and to interview key stakeholders associated with these activities.

3. FINDINGS

The evaluation team was impressed by the overwhelming consensus among respondents of appreciation

of USAID for undertaking this bold initiative on HRH, described as timely, far-sighted, and generous. USAID is now recognized as one of the global leaders in this field, and expectations are high that it will maintain this status for the next few decades.

The collaboration of GH/PRH/SDI and OHA to address HRH as a crosscutting issue represents a

management best practice. The arrangement of a centrally funded project from the Bureau for Global Health, with funds coming from two vertical programs, is a functional and effective model for future crosscutting projects. Nevertheless, given the cross-bureau importance of HRH, it is unfortunate that the

GH/Office of Health, Infectious Diseases, and Nutrition (HIDN) did not provide technical input in the management of the project and that more significant levels of field support funding were not available other than from PEPFAR. Nevertheless, funding of field activities from PEPFAR and core funding from

PRH/SDI is a bold and pragmatic exploitation of opportunities and a worthy investment that will yield benefits for service delivery programs.

The recruitment and focused use of a central headquarters staff of experts with a broad array of technical skills and a clear mission centered on providing HRH technical leadership on a global basis and within

participating countries represents a significant best practice associated with the projects management.

Evaluation of the Capacity Project ix

Despite its short life span, the project has established an organizational and operational structure with a presence in 18 countries, with eight of these having functioning country offices.

The project has developed an impressive array of tools and products to address HRH issues (Annex 3). Significant contributions in the area of workforce planning and policy include the following:

Strengthening of the human resource information system (HRIS) in participating countries

involving the development of readily available, user-friendly, open-source software to provide current and accurate data on a countrys human resources for health. When fully developed and effectively used, the HRIS will assist policymakers, HRH managers, and health service managers in identifying

and responding to critical gaps in HRH deployment, technical qualifications, and training needs of the workforce.

Development of the HRH Global Resource Center website, a searchable collection of HRH information and resources recognized as by far the best resource center for HRH in the world.

Promotion of the HRH Action Framework, an HRH planning framework useful to HRH managers

and other stakeholders as they seek to develop and implement national strategies to achieve an effective and sustainable health workforce for their respective countries.

Extensive technical assistance in Human Resource Management through leadership training, secondment of staff to government, assistance in writing HRH strategic plans, design and

management of emergency hiring plans, and consensus building among all stakeholders about HRH priorities.

The project has achieved substantial gains in implementing the specific objectives in workforce

development. More than 17 countries have requested interventions in this area during the final year of the project, and life-of-project targets have already been achieved in both pre-service and in-service system strengthening. Successful examples include the following:

- Pre-service nursing and midwifery education: In Rwanda, the Ministry of Health, in

collaboration with the Nursing and Midwifery Council, successfully rationalized the nursing cadre, resulting in the closure of the existing and ineffective schools of nursing and midwifery. With help from the Capacity Project and Belgian Technical Assistance, Rwanda opened five new

nursing and midwifery schools, using a newly developed curriculum consistent with the envisioned tasks and responsibilities of graduates.

Task shifting: The concept of an appropriate reallocation of tasks to relieve the burden on scarce human resources, or task shifting, represents a best practice implemented by the Capacity Project.

For example, in Mali, the project has shown that matrones (auxiliary midwives) can be trained to perform active management of the third stage of labor in place of scarce nurses and physicians and thereby reduce the risk of postpartum hemorrhage and potential maternal death. The Capacity

Project also documented and created implementation guidelines for a promising HRH practice developed by a Ugandan FBO, employing the concept of task shifting to address acute shortages

in human resources for health by providing a short training to degree and diploma holders from social sciences and education. In both Uganda and Rwanda, the project has been a catalyst in working with the two governments in advocating for a newly recognized professional cadre: the

human resources manager. In the future, this cadre will be able to assume some of the roles currently held by Ministry of Health physicians, thus freeing up medical doctors to focus more of their time on policy issues associated with diagnosis and care.

Continuing Professional Development: The Capacity Project has produced a brief on

strengthening health professional associations to assist participating countries in the design and development of continuing professional development initiatives. Building on this initiative, the projects program in Ukraine has focused on developing the management, finance, and operations

x Evaluation of the Capacity Project

capacity of the Ukrainian Association of Obstetricians and Gynecologists. Capacity has also provided training on communication skills to nurse trainers associated with the Ugandan Nursing and Midwifery Association. In discussions with the evaluation team, senior directors of the

association stated their belief that the assistance provided has helped improve the image of the nursing profession in Uganda.

Improving workforce support systems: The main effort in this area has been the design and implementation of studies to better understand issues that affect performance, such as retention,

productivity, and motivation of health workers as guidelines for future action. A list of these initiatives includes the following items:

A facility-based health worker study in Uganda to measure health worker satisfaction, motivation, and intent to stay in the health field.

A study on gender-based violence in the workplace in Rwanda and another on safety and health

in the workplace in Uganda, the findings of which are expected to result in policy formulation and implementation.

Relatively inexpensive initiatives, in Zanzibar and Kenya, to improve the work climate, such as improving cleanliness of health facilities; posting work schedules for information to clients; and

performance improvement schemes in Uganda and Rwanda, in which staff themselves undertake supervision based on staff-generated norms and standards, as opposed to those generated by

supervisors.

Other initiatives including interventions focused on addressing ways to improve performance, studies on worker retention, and interventions to improve the workplace climate have made initial contributions to the body of knowledge necessary to address improved workforce support systems.

Crosscutting Areas: The Capacity Project works collaboratively with the World Health Organization (WHO), the Global Health Workforce Alliance (GHWA), the Pan-American Health Organization

(PAHO), the World Bank, the Asia-Pacific Action Alliance for HRH, and the Central, Eastern, and Southern Africa Health Community. The project has achieved considerable progress in mainstreaming gender throughout the activities of the project, and the project has enlisted FBOs as

part of national efforts to address HRH issues.

Project Management and Organization: The team examined this issue from three levels: USAID/Washington, partnership management, and Capacity Project administrative management. The

team found progress at each of these levels as being satisfactory to excellent, and as contributing to the success and accomplishment of the project. The close working relationship of the two main funders of the project (PRH and OHA) has been commendable. However, the challenge of a

crosscutting project having to show service delivery output and outcomes within a short time frame has been problematic, distracting, and difficult to address. Nevertheless, the project has achieved noteworthy and significant results in both FP/RH and HIV/AIDS service delivery. These activities

and results are described in Annex 1. The apparent lack of HRH expertise in field missions has contributed to fewer countries being engaged in HRH. The partnership consortium has promoted seamless integration of technical partners at the project headquarters. Capacity Project staffing in the

country offices (Rwanda and Uganda) visited by the evaluation team was, in our opinion, skeletal and ill equipped in terms of qualifications and experience to take on the complex challenges associated with strengthening HRH systems. However, the shortage of HRH experts is a general problem

experienced by the majority of donors engaged in HRH, a finding supported by GH/PRH/SDIs survey of Capacity Projects USAID mission partners.

Annex 2 provides additional information in response to the specific questions provided to the

evaluation team in the Scope of Work.

Evaluation of the Capacity Project xi

4. CONCLUSIONS AND RECOMMENDATIONS FOR THE FUTURE

The evaluation team was impressed by the overwhelming opinion of all persons interviewed that this or a similar HRH initiative should continue, that the momentum built by the Capacity Project in addressing

HRH issues should not falter, and that there should be no gap between the current Capacity Project and its successor project. The evaluation team emphatically supports this view for the following reasons:

USAIDs involvement through the Capacity Project to address the crisis of human resources for

health was timely, far-sighted, and commendable. Over the short life span of the project, it has developed concepts, tools, and interventions to respond to the crisis, and is poised to take the lessons learned to more countries.

HRH solutions take a long time to implement and mature, and uptake by countries is slow.

The fact that the project has had just two to three years real time in which to address HRH systems

development has provided insufficient time to observe impact, and limited opportunity to scale-up activities beyond their initial development stages.

Principal Recommendations

Based on the observations in the above paragraph, the evaluation team recommends that:

A. USAID should extend the current IntraHealth Capacity Project LWA cooperative agreement by an additional 12 months through September 30, 2010, with sufficient levels of additional core funding,

an amount to be negotiated with IntraHealth based on current burn rates, obligations, and an operational workplan for the next two years. On this issue, we would recommend that USAID advise IntraHealth of this extension, if it is granted, as soon as possible following the final acceptance of this

evaluation report. In doing so, USAID will promote continued engagement of current staff throughout the extension period.

B. USAID should undertake to develop and fund a five-year follow-on project focused on building on the accomplishments of the current project and on extending USAIDs success under the current

project by introducing HRH systems development to countries not currently included in the Capacity Projects portfolio of participating countries. On this issue, we believe that a centrally funded LWA cooperative agreement has responded, as intended in the current request for application (RFA), to the

complex nature of addressing HRH as a global issue and that this mechanism should be considered as the technical base for the future project.

C. The evaluation team understands and appreciates the importance that GH/PRH and GH/OHA attach

to accounting for improvements in service delivery indicators resulting from USAIDs current and future investments in human resources for health. The evaluation team recommends that the need for data on service delivery indicators be balanced with a clear understanding and appreciation that the

driving force of health system performance is the health worker, and that strategies to alleviate the ongoing HRH crisis in the developing world will have a positive impact on all health indices. As indicated in Figure 4 of the full report, efforts to build and sustain the health workforce are predicated

on the link between the performance of health workers and health outcomes: this link is not an article of faith but grounded on solid evidence.

Based on the above three principal recommendations, the following paragraphs present a series of recommendations with reference to the focus of Capacity Projects final year(s) of implementation and

with reference to a vision for the future beyond September 2010. Since the recommendations address largely technical, managerial, and training issues, the evaluation team has attempted to identify those recommendations that most directly fall within these three domains.

xii Evaluation of the Capacity Project

Recommendations: Planning for the Next Twelve to Twenty-four Months (October 2008September 2010)

Technical Issues

In its remaining year(s), the Capacity Project, with assistance of core funding, should assign priority to an assessment of needs and to the development of guidelines, protocols, and training modules focused on addressing HRH issues associated with community health and mid-level workers. Having

developed these modules, the Capacity Project, with advocacy assistance from USAID/Washington, should work with current missions to secure field support to pilot-test and scale-up these modules. This initiative should be continued and expanded upon under the future project.

The Capacity Project should undertake, with core support, an in-depth and independent appraisal/

evaluation to identify managerial and infrastructure constraints and to recommend corrective action associated with the expansion of the HRIS in those participating countries in which rollout of the HRIS is being contemplated.

A network of countries in which Capacity Project is operating should be established in order to

promote an exchange of experience, enhance their ability to profit from lessons learned, foster cross-country collaboration, promote the creation of shared tools and databases, and provide opportunities for joint training programs.

Managerial Issues

The Capacity Project should immediately assign or recruit a suitably qualified specialist in advocacy

to develop and oversee the implementation of a plan of action whose focus will be on the periodic orientation of global partners about Capacity Project HRH initiatives. In addition, the action plan should address ways in which to promote the collaboration of international partners in responding to

those HRH development priorities (for example, infrastructure development, regional conferences, training of stakeholders) beyond the scope of a United States Government (USG)-financed project.

Training Issues

A focused effort should be directed toward enhancing the capacity of national counterparts in the analysis and development of effective policies centered on creating an enabling HRH policy

environment in the interest of addressing HRH issues such as retention, workplace safety, workplace violence, recruitment, performance-based incentives, supervision, and career development.

Recommendations: Planning for the Future (October 2009/10 and Beyond)

Technical Issues

Since the effort to develop HRH Champions in Capacity Projects participating countries has constituted a principal focus of the project, the evaluation team recommends that USAID retain this

focus as a new project extends to additional countries in Africa and beyond.

At a minimum, further development and expansion of the HRIS, the HRH Global Resource Center and the HRH Action Framework (IR1), pre-service/in-service nursing education, continuing professional development (IR2), and retention strategies, performance improvement systems, and

improvement of the workplace environment (IR3) should receive continued emphasis and support under the future project.

A future project should extend its coverage to include countries in Africa with poor HRH records, including those in Francophone and Lusophone countries. This recommendation is in recognition of

the fact that Africa has the most serious HRH crisis and a disproportionate disease burden.

Under a future project, USAID should lead a global HRH financing initiative to provide resources to resolve the health workforce crisis in the developing world similar to the Global Fund. USAID should

Evaluation of the Capacity Project xiii

also explore with the World Bank the possibility of securing a free or low-interest concessionary loan for an HRH intervention project among the Least Developed Countries.

In addition to the recommendation for leveraging international partner cooperation in addressing infrastructure development priorities, USAID should develop a mechanism to pair HRH work, under

a new project, with USAID-financed service delivery projects.

A new project should vigorously and systematically pursue an advocacy agenda focused on creating and sustaining the current momentum for HRH systems development. Under the agenda, the project should develop means of targeting different audiences and stakeholders.

A future project should assume a leadership role in establishing a Center for HRH Research, whose

mission will be to assist countries in conducting policy-relevant analysis and research on HRH, and shared learning.

A future project should develop a paradigm for the funding and implementation of research in which a commitment to the application of research findings, if found to be relevant to an identified issue, is

undertaken prior to undertaking the research itself. This commitment would include the identification of funding and the agreement of stakeholders to a timetable and workplan for implementing the research agenda and for applying its results.

Managerial Issues

Within the realm of practicality, the current USAID project management approach should be

reviewed and a visionary restructuring be considered to reconcile the current dilemma of having HRH, as a crosscutting priority, sequestered, for management purposes, within vertical programs associated largely with FP/RH and HIV/AIDS.

A future project should develop an active strategy, in collaboration with USAID/Washington, to

disseminate its work among those countries most in need of HRH assistance but within which issues associated with HRH are poorly understood. For this purpose, USAID should adopt a flexible and proactive approach to the use of core funds.

Under a future project, a concerted effort should be directed toward ensuring that the recruitment of

country staff focuses on those individuals whose set of skills and experience will equip them to address country-specific HRH development priorities.

Since the Global Resource Center (GRC) is universally recognized as a public good, USAID,

working with IntraHealth, should expend every effort to ensure continued funding and technical support for GRC.

Partner definition associated with a future project should focus on responding to the projects technical needs. While it is difficult to specify the number of partners that should be considered, it

appears that the current composition of the Capacity Project, in terms of its manageable numbers, is just about right: Any more would be unmanageable and any less would most likely be insufficient in responding to the complexities of HRH systems development.

USAID, in its design of a future project, should establish location of the project headquarters in

Washington, D.C. as a precondition for the procurements award.

USAID and the project management team of a future project should develop a clear memorandum of understanding delineating responsibilities of all parties. USAID and the implementing organization should review and, if indicated, modify, the memorandum to reflect changes in management

requirements.

xiv Evaluation of the Capacity Project

Training Issues

USAID should support a degree or competency-based certificate program in human resources

management, possibly through focused on-site training initiatives or through distance learning in human resources management to develop a critical mass of effective HRH managers.

USAID should consider supporting the training in HRH management of private sector consulting firms currently licensed in selected African countries. Such training initiatives, while possibly unique

within the nations of Sub-Saharan Africa, have shown great promise within Asia as an alternative to the provision of costly U.S.-based or European consultants.

Evaluation of the Capacity Project 1

EVALUATING THE CAPACITY PROJECT: ASSESSING PROGRESS

ON HRH ISSUES

1. INTRODUCTION

Background and Evolution of the Capacity Project: Interest in human resources for health attracted global attention when it became evident that the health workforce in the developing world was facing a crisis, and that this was affecting health service delivery and population health. The Joint Learning

Initiative on Human Resources for Health and Development (JLI), a consortium of more than 100 health leaders from around the world, undertook an exploration of the human resources for health (HRH) landscape during 200204 and identified three major aspects of an accelerating global HRH crisis:

The devastation of HIV/AIDS

An accelerated level of migration causing brain drain

A legacy of chronic underinvestment in human resources.

The JLI concluded that mobilization and strengthening of the health workforce is central to combating

health crises in some of the worlds poorest countries and for building sustainable health systems (Human Resources for Health: Overcoming the Crisis 2004).

Summary Project Description

In September 2004, the United States Agency for International Development (USAID) Bureau for Global Health/Population and Reproductive Health Office/Service Delivery Improvement Division (PRH/SDI)

awarded to IntraHealth International the Human Capacity Development Program, (later renamed the Capacity Project), a five-year (October 2004September 2009) Leader with Associate Award (LWA) cooperative agreement.

The Capacity Project was awarded to IntraHealth International, Inc., the Leader of a consortium of nine

organizations of which the following seven partners have remained with the project throughout its implementation:

IntraHealth International, the recipient of the Leader Agreement

IMA World Health (IMA)

JHPIEGO

Liverpool Associates in Tropical Health (LATH)

Management Sciences for Health (MSH)

PATH

Training Resources Group (TRG).

Drawing on the composition and diversity of its technical partnership, the Capacity Project has focused its

efforts on providing USAID/Washington Bureaus, international agencies, and USAID missions with ready access to high-quality technical assistance and support for the wide variety of technical needs associated with current and emerging HRH challenges. As structured, the Capacity Project was provided

with core funding to provide for administrative, technical, and management support. The value added of core funding is the ability to have a research agenda that fosters the projects global leadership role in promoting and strengthening HRH initiatives. The Capacity Projects technical focus and guidelines have

also made it possible for USAID missions to use traditional field support funding to obtain the services of

2 Evaluation of the Capacity Project

all partners to the project in responding to mission-specific HRH issues. Finally, the Capacity Projects design has made it possible for individual missions to develop and fund Associate Awards, which

establish a bilateral agreement between the mission and IntraHealth that provides access to the Leaders technical expertise, with the possibility of extending up to five years beyond the end of the Leader award.

The Project was established with a funding ceiling of $250 million for the leader award, with $30 million designated as core funding and the remainder as field support. As designed, project guidelines specified

that Associate Awards were not calculated as being part of the ceiling. In addition, under guidelines established as part of the cooperative agreement, Capacity Projects leader and its partners agreed to a cost-sharing arrangement equivalent to 10 percent of all core and field support expenditures.1

As illustrated in Figure 1, most of Capacity Projects core obligations to date (July 2008) are from the population account. Conversely, as illustrated in Figure 2, most of the field support/MAARD obligations to date are from the Presidents Emergency Plan for AIDS Relief (PEPFAR). Accordingly, senior

technical advisers from both GH/PRH/SDI and USAIDs Bureau for Global Health Office of HIV/AIDS (OHA) are key members of USAIDs Capacity Project Management Team.2

Stated as a general goal, the Capacity Project was designed to assist countries to build and sustain the

health workforce with a focus on ensuring that human resources in the right number with the right skills at the right place are doing the right thing. By both developing and applying proven and promising approaches to solving critical problems, the Capacity Project has focused on improving the environment

for human resources for health in an effort to promote quality, accessibility, and use of priority health care services. As illustrated by Figure 3, the Capacity Project has focused on achieving outputs and outcomes in three distinct result areas and in four crosscutting areas associated with HRH systems development.

1 As of July 2008, Capacity Project countries had achieved an internally certified cost-sharing equivalent to 9.07

percent of all core and field support expenditures, with an additional 2.96 percent under review. (See Annex 4:

Capacity Project Evaluation: Cost Share Status by Country for a detailed breakdown of Capacity Project countries

progress on cost sharing.) 2 Figures 1 and 2 represent the latest graphic update of obligations. Please see Annex 5: Capacity Project Evaluation:

Summary of Capacity Project Obligations by Funding Source, Years 15 for slight adjustments to the illustrated

breakdown based on recent mission-supported Associate Awards.


FIGURE 3: CAPACITY PROJECT RESULTS FRAMEWORK

As specified in the Capacity Projects Performance Monitoring Plan (PMP),3 technical interventions in each of the three result areas and four crosscutting areas were expected to result in:

Quality assessments of need for changes in the workforce planning process implemented

Workforce data systems developed or strengthened

Country-specific workforces realigned to better meet priority health objectives

Pre-service education for specific cadre strengthened

In-service education systems strengthened

Health professional councils increased, HRH management capacity demonstrated

Health sector human resource management systems strengthened

Worker retention systems strengthened

Global partnering on HRH developed

3 See Annex 6: Capacity Project Evaluation: Performance Monitoring Plan.


HR planners and managers use of Human Capacity Development (HCD) initiatives developed under the Capacity Project demonstrated

Increased awareness of the value of workforce gender equity in planning and policy development

promoted

Increased capacity of faith-based organizations (FBO) to respond to HRH issues developed.

Scope of Work

The purpose of this report is to summarize the findings of an evaluation focused on an assessment of the progress and accomplishments associated with the first four years of the Capacity Project, a review of the strengths and weaknesses of the projects operations and management, and an assessment of key

stakeholders perspectives with reference to the projects final year of operations and the potential for future investments associated with extending USAIDs initial support of HRH initiatives under the Capacity Project.

Accordingly, under guidance provided by USAIDs Capacity Project Management Team, the evaluation

team4 was assigned three principal tasks5:

To review the Capacity Projects technical and programmatic strengths, weaknesses, successes, and constraints, identifying contributing factors;

To assess the projects organizational structure and management; and

To identify initiatives that warrant additional investment during the final year of the project and into

the future with reference to continued advancement in addressing existing and emerging challenges associated with human resources for health.

Methodology

In responding to the scope of work outlined above, the evaluation team began its work on May 12, 2008 and concluded its analysis on August 6, 2008. The team, in collaboration with GH/PRH/SDI and OHA

staff, developed and adhered to the following methodology:

Pre-evaluation Team Planning Meeting: The evaluation team benefited from a two-day pre-evaluation planning meeting led by an external facilitator in the presence of senior staff from the PRH/SDI and OHA Capacity Project management team as well as staff from GH Tech. The evaluation team was briefed on

the project, including its genesis, its placement within the overall structure of the USAID Global Health Bureau, and its financing mechanisms. Following this meeting, the evaluation team prepared and submitted a workplan6 with milestones, deliverables, and a standardized survey instrument.7

Document Review: A significant amount of informative documents related to the Capacity Projects

technical and managerial functions and activities were made available to the evaluation team for its review and reference both at the start of the evaluation process and throughout the evaluations implementation.8

4 The evaluation team, comprised of two GH Tech consultants, was joined by a USAID GH/PRH/SDI staff members

in each of its field visits to Rwanda and Uganda. 5 See Annex 7: Capacity Project Evaluation: Scope of Work. 6 See Annex 8: Capacity Project Evaluation Workplan. 7 See Annex 9: Capacity Project Evaluation: Respondent Interview Guidelines. 8 See Annex 10: Capacity Project Evaluation: Annotated Listing of Key Capacity Project Reference Documents

Reviewed.


Self-Assessment Report Review: A comprehensive Capacity Project self-assessment report, designed by the GH/PRH/SDI and OHA staff, and consisting of responses to specific questions, was completed by

Capacity Project management staff in North Carolina in late February 2008 and made available to the evaluation team prior to the launch of the evaluation. The self-assessment9 report provided the evaluation team with an important knowledge window into Capacity Project staffs perspective of technical and

managerial issues associated with the projects implementation.

Field Mission Survey Review: In addition to the self-assessment, an email survey of USAID missions, both of those benefiting from Capacity Project interventions and of those that had not yet made use of Capacity Project resources, was designed and implemented by GH/PRH/SDI and OHA. The results of the

mission survey provided the evaluation team with important insight into the perspective of mission staff with reference to Capacity Projects contribution to their technical HRH issues. The results of the email

survey also provided the evaluation team with a greater understanding of USAID missions view of the future. Information included in the surveys referenced management issues associated with field support for centrally funded projects, bilaterals, and LWA awards.

Review of Websites: Throughout the course of the evaluation, the team made extensive use of the

Capacity Projects website (www.capacityproject.org) as an instrument for gaining an extended understanding of the depth and scope of project interventions. While, the scope of this evaluation and the expertise of the evaluation team do not provide for an expert assessment of the technical quality of the

website and of its linkages to other websites such as the HRH resource center (www.hrhresourcecenter.org) and the HRH Action Framework (www.capacityproject.org/framework,), we were impressed by the user-friendly nature of these important and informative sources of HRH-related

information.

Key Informant Interviews: Using the standardized survey instrument (Annex 9), the team carried out a qualitative, in-depth interview each lasting at least 45 minutes. Where possible, interviews were carried out face-to-face, often individually, and occasionally in groups. When not feasible, interviews were

conducted by teleconference. In total, 144 individuals were contacted, in Washington, in other U.S. cities, and overseas, and included site visits to North Carolina, Rwanda, and Uganda (see below). As illustrated

in Table 1, the audience represented a wide array of stakeholders10 including individuals associated with USAID, both in the United States and in selected missions, the Capacity Project in North Carolina and participating countries, members of the partnership consortium, HRH experts, staff attached to

international organizations, and government officials in Rwanda, Uganda, and other countries in which the Capacity Project is operating. Questionnaires were adapted for respondents and were circulated in advance to assist respondents in preparing for the discussion. At the completion of each days interview

sessions, the team met to review and summarize the days interviews, with the reviews emphasis focused on respondents contribution to the teams understanding of Capacity Projects progress to date, on management and organizational issues, and on the respondents perspective on the future of HRH

initiatives.

9 See Annex 11: Capacity Project Self-Assessment. 10 See Annex 12: Respondents.

http://www.capacityproject.org/http://www.hrhresourcecenter.org/http://www.capacityproject.org/framework


TABLE 1: CAPACITY PROJECT EVALUATION: NUMBER OF PERSONS INTERVIEWED BY AFFILIATION

USAID/ Washington

No. of Respon-dents

Capacity Project (US)

No. of Respon-dents

Participating Countries

No. of Respon-dents

International Agencies

No. of Respon-dents

GH/PRH/SDI 6 Headquarters (North Carolina)

23

Senior Government Officials Rwanda

2 WHO 2

GH/OHA 2 Capacity Project Partners

7

Senior Government Officials Uganda

12 World Bank 1

GH/PRH 7

Capacity Project Representatives and Staff in Participating Countries

Health Staff Uganda

7 PAHO 2

GH/HIDN 1 Uganda 7 Health Staff Rwanda

12 AAAHRH 1

Africa Bureau

2 Rwanda 12 Professional Health Associations

6

OGAC 1 Mali 1 NGOs 2

USAID Missions Tanzania 1 USAID CAs 6

Uganda 4 Kenya 1 Donor Agency

4

Ethiopia 1 Ethiopia 1

Rwanda 7 Consultants 2

Kenya 1 Grand Total of Respondents

144

Field Visits

Chapel Hill, North Carolina: As an essential element of the evaluation, the team spent two-and-half days in the Capacity Projects North Carolina headquarters. Key staff members of the project made formal presentations of the work of their department and provided the team with both hard and electronic copies

of materials to supplement the presentations. Individual and group meetings were also held with the Program Leadership Team and mid-level staff. The evaluation team further interacted with staff during the course of the evaluation to clarify and/or obtain information.

Rwanda and Uganda: To provide the evaluation team with an understanding of the breadth and scope of

the projects interventions in participating countries and given the time allocated to the evaluation, USAID selected two countries, Uganda and Rwanda, for site visits. In both of these countries, the team was accompanied by a USAID GH/PRH/SDI staff person who attended and monitored the majority of

interviews conducted during the two-country visits. With nine days devoted to each country visit, the team maintained a full schedule of interviews throughout the visits and travelled extensively within

Rwanda and Uganda to observe Capacity Projects field activities. During the visits, the team had opportunities to interact with Capacity staff in Rwanda and Uganda and with senior government officials, district officers, and health center and hospital staff. The team witnessed training and related service

delivery in progress, observed the state of infrastructure in training institutions and health facilities, and lent a sympathetic ear to administrators and health workers pleading for infrastructure support.


Data Analysis and Completion of Draft Report: Upon the teams return to Washington, four weeks were allocated under the scope of work for the completion of final interviews, analysis of findings, and for the

preparation and submission of the draft evaluation report.

USAID/Capacity Project Review of Draft Report and Preparation and Submission of Final Report: Following the teams presentation of a summary of the evaluations findings to USAID and Capacity Project staff, the team modified the report and submitted a draft of the report to USAID and Capacity

Project staff for their 10-day in-depth review and observations. Acting on USAID/Capacity Project observations, the team further modified the report for an August 28, 2008 submission to GH Tech for final preparation and submission to and approval by GH/PRH/SDI and OHA during September 2008.

2. PROJECT FINDINGS AND ACCOMPLISHMENTS

In the scope of work developed by

GH/PRH/SDI and OHA in collaboration with GH Tech and the evaluation team, a number of

illustrative questions were included under each of the three main tasks of the evaluation cited earlier in this report. In the following

paragraphs, the evaluation team has directed its attention to responding to issues addressed in these questions. In addition, Annex 211 of this

report provides a specific bulleted response to each of the illustrative questions. The focus of the body of this report is on highlighting

principal achievements and accomplishments, as well as challenges and constraints, associated with the Capacity Projects initial

four years of operation and the extent to which the project has met the deliverables specified in the project management plan. While the list of

accomplishments is not all-inclusive, the section identifies those initiatives whose introduction, under the Capacity Project, hold

the greatest promise for further development in addressing the critical issue of HRH as it

affects the delivery of essential health services in developing nations. Accordingly, the following paragraphs present the evaluations

findings in terms of:

Principal Achievements

Achievements by Intermediate Results

Achievements by Crosscutting Areas

Project Management and Organization Issues.

11 Annex 2: Answers to Scope of Work Questions.

Box 1: Quotes from the Field

The Capacity Project is the only act in town.

In-country implementation of Capacity

represents a happy marriage of PEPFAR

indicators with the wider needs of HRH.

USAID does not recognize that it is gaining a lot of respect for the work Capacity is doing.

Capacitys emphasis on HRH looks at the issues

of Public Health in a unique way.

Management of the Capacity team is so good

that there are lessons in it for USAID.

Capacitys IT work represents the best in

engineering practices, excellent documentation,

consistency, and the highest standards.

Capacity in Rwanda exemplifies what can be

done to be proactive in response to national

policy.

Work undertaken in Uganda exemplifies the

focus of Capacity on setting the framework for

HRH development.


Project Findings and Accomplishments: Principal Achievements

As indicated in a selection of observations offered by respondents to the evaluation team (Box 1), the initiative taken by USAID to assist developing countries in overcoming the crisis in human resources for

health (HRH) is highly regarded and appreciated as timely, far-sighted and generous. Accordingly, USAID is now recognized as a global leader in HRH and expectations are high that it will maintain this status over the coming decade.

The management model of having a central headquarters staffed with a dynamic team composed of

individuals with diverse skills and providing technical leadership to country offices has been an asset. As the Capacity Project was launched less than four years ago to address a neglected and little-understood field, it experienced an initial slow start during which staff were recruited and offices set up. However,

during what must be regarded as a brief period in which to address a complex set of issues, the project team has developed an impressive array of analytic tools and interventions to address HRH issues as they impact the quality of health programs. The space constraints of the body of this report preclude including

the full array12 of technical briefs, communications material (Voices from the Field), and tools and resources developed under each of the projects intermediate results areas. A list of key tools and interventions, many of which are discussed in this report, would, however, include: Technical Briefs on

issues ranging from Task Shifting to Strengthening Health Professional Associations to Guidelines on the Retention of Health Care Workers in low-resource settings; Voices from the Field, a series of over 15

issue-oriented communications associated with strengthening HRH in Africa; and Tools and Resources including collection and analyses of HRH strategic plans in Capacitys participating countries, intrauterine device (IUD) guidelines for family planning services, development of a Human Resource

Information System (HRIS) in, among other countries, Kenya, Rwanda, South Africa, Swaziland, and Uganda, Workforce Studies in Uganda and Zanzibar, Gender-based Violence Training Modules, development of the HRH Global Resource Center, and the development and application of the HRH

Action Framework (HAF) in Uganda. Accordingly, due in great part to the development of a foundation of HRH tools and issue-targeted interventions, the project is now well positioned and poised to roll out its initiatives and address HRH issues within a wider universe of countries in the developing world.

Based on the Capacity Projects senior staff analysis of indicators associated with the projects

performance monitoring plan,13 it would appear that the project is on track to achieve the great majority of deliverables for each of the projects 12 key indicator areas. The sole exceptions to this significant accomplishment are two targets associated with Intermediate IR 2 (Improved Workforce Development: (a)

innovative recruitment strategies, and (b) testing and implementing teaching institution accreditation systems).14 In addition, Capacity Projects tracking and documentation of country-specific activities by intermediate result with reference to family planning and HIV/AIDS,15 many of which are discussed in

this report, represents strong confirmation of the apparent breadth and depth of the projects results-oriented focus on deliverables.

12 See Annex 3: Capacity Project Evaluation: Capacity Project Products by Technical Leadership Area for a detailed

listing of Capacity Project Products. 13 See Annex 13: Capacity Project Performance Indicator Achievements. 14 Based on the teams observations and discussions with senior Capacity Project management staff, lack of progress

on Result Area 2 (Improved Workforce Development), especially in the area of pre-service training, appears to be

due to the reluctance of host countries and mission staff to dedicate the time and resources required to address this

critical area of HRH development. As will be discussed in subsequent sections of this report, the challenges

associated with workforce development will need to receive a significant degree of increased focus in future HRH

initiatives undertaken under USAID funding. 15 See Annex 1: FP/RH and HIV/AIDS Specific Activities and Results.


The project has assembled a core of competent, dedicated, and dynamic staff in Chapel Hill, North Carolina and in its eight fully participating country offices and in the 18 countries in which it has

developed significant relationships.16 The teams diverse skills in HRH and in related technical fields associated with information technology (IT), pedagogy, management, and organizational development received significant levels of positive support and recognition from the large majority of this evaluations

participating respondents.

Capacity Project staff have demonstrated that attention to HRH issues can have a significant impact on service delivery and on the provision of quality health programs through their introduction of clear and concise technical guidelines and focused technical assistance. For example:

As demonstrated by interim results from its 20072008 Demographic and Health Survey (DHS),17 the efforts of Rwandan health service providers have resulted in a significant increase in their clients use of modern contraceptive methods. According to the DHS findings, the proportion of women in

union currently using a modern contraceptive method is 27 percent, nearly three times as much as that in 2005 (10 percent). In an effort to sustain these very positive results, the Capacity Project, with USAID/Rwanda RH/FP field support, has provided in-service training to more than 300 health

workers in family planning contraceptive provision and counseling. This base of trained individuals augurs well for continued advances in Rwandas contraceptive prevalence rate.

In Kenya, the Emergency Hiring Plan18 demonstrated that effective technical assistance in HRH can result in a rapid response to a national HRH emergency. Through a subcontract with Deloitte &

Touche and in collaboration with the Ministry of Health, Capacity designed and implemented the Emergency Hiring Plan (EHP) to overcome bureaucratic personnel hiring practices. Under the EHP, a hiring plan financed through USAID/Kenya PEPFAR support, Capacity Project has hired, trained,

and placed over 830 health care workers in critically underserved health centers over a short period. While the Ministry of Health has signed an agreement indicating its commitment to integrating these health workers into the national payroll system, the EHP itself represents an innovative approach to

public/private sector collaboration in times of HRH emergencies.

In 2006, Namibias prevalence rate for HIV/AIDS among pregnant women was estimated at 19.9 percent.19 Through the Capacity Projects training of health workers, the number of mothers in five

faith-based hospitals and private health services enrolled in prevention of mother-to-child transmission (PMTCT) who accept family planning support are reported to have reached more than 90 percent of clients. In September, Capacity will undertake a review of its progress to confirm these

significant results.

The Capacity Project has developed an impressive array of internationally focused tools and products to address HRH issues:

The HRH Global Resource Center website, developed by Capacity Project, is a searchable collection of HRH information and resources recognized as by far the best resource center for HRH in the

world.

16 See Annex 14: Capacity Project Fieldwork Presence as of June 2008. 17 Government of Rwanda National Institute of Statistics, Ministry of Finance and Economic Planning, Ministry of Health, and Macro International, Inc., Rwanda Interim Demographic and Health Survey 200708 Preliminary

Report, July 2008. 18 Based on discussions with Capacity Project staff, the term emergency hiring plan has now been changed to

accelerated hiring plan in recognition of the technical focus of this important initiative. 19 Johns Hopkins Bloomberg School of Public Health, Center for Communication Programs; www.jhuccp.org.


The Human Resources Information System (HRIS), developed around the principle of readily available, user-friendly, no-cost Open Source software constitutes an important initiative focused on

providing current and accurate data on health workers for developing country HRH managers and policymakers to respond to country-specific key policy and management issues affecting service delivery. The HRH Action Framework website, developed in collaboration with the WHO and the

Global Health Workforce Alliance (with Capacity Project as the website administrator), responds to HRH manager and other key stakeholder information needs as they seek to develop and implement national strategies to achieve an effective and sustainable health workforce for their respective

countries.

The Learning for Performance (LFP) guide and toolkit for health worker training and education programs provides a process for focusing curricula on priority desired performance outcomes. In Mali

and Rwanda, the Project has used LFP to strengthen pre-service education and training approaches, and has made a measurable impact on the competence of nursing graduates.

The HRH Action Framework, in collaboration with WHO and other partners.

Focused studies to better understand HRH issues, including a retention study to identify factors that influence the decision of health workers to stay in the health sector in Uganda, a study on violence at

the workplace among health workers in Rwanda and workplace safety.

Establishment of the groundwork to help countries develop a Continuing Professional Development program for the health workforce. Capacity has produced a brief on strengthening health professional associations to assist participating countries in the design and development of continuing professional

development (CPD) initiatives. Building on this initiative, Capacitys program in Ukraine has focused on developing the management, finance, and operations capacity of the Ukrainian Association of Obstetricians and Gynecologists. Capacity has also provided training on communication skills to

nurse trainers associated with the Ugandan Nursing and Midwifery Association.

Pilot interventions focused on enhancing motivation of health workers by improving the workplace climate.

Based on responses received from international agency representatives, it is apparent that Capacity

Projects senior technical staff have made significant progress in developing meaningful and recognized working relationships with a large variety of organizations engaged in responding to HRH issues. A partial list of such principal collaborators would include the World Health Organization and its regional

offices, the Pan-American Health Organization and its country offices, the World Bank, the Global Health Workforce Alliance, the Global Fund to Fight HIV/AIDS, Tuberculosis, and Malaria (GFATM), United Nations Fund for Population Activities (UNFPA) and its country offices, and the Commonwealth

Regional Community for East, Central and Southern Africa (ECSA). The importance of these and other similar technical alliances is that they represent, through the Capacity Project, USAIDs growing position of importance and influence in advocating for an increased focus on HRH as an issue of critical

importance to improved health service delivery.

As an initiative largely financed by USAIDs GH/PRH/SDI and GH/OHA with managerial responsibility assigned to GH/PRH/SDI, USAIDs management of the Capacity Project exemplifies a good management practice as evidenced by the technical and managerial collaboration between PRH and

OHA. Nevertheless, given the cross-bureau importance of HRH, it was reasonable to have expected, in the evaluation teams opinion, a more balanced level of funding and technical input from GH/HIDN and from field support from sources other than PEPFAR.

The approved application of funding for field activities from PEPFAR represents a bold exploitation of

opportunities and a pragmatic approach to addressing an important issue. Examples of this approach would include the use of PEPFAR funds for health system strengthening, including that applied to the


training of nurses, and to the upgrading of health facilities, purchase of equipment, and support of retention strategy. It should also be noted that PRH funds have also been used for similar purposes

(training, facility upgrades, and seconding support staff). Inevitably, and as can be expected, there are occasions when tensions occur, for example, when a GH/PRH Family Planning / Reproductive Health-financed project is under pressure to show tangible impact of the investment through service-linked

indicators rather than those linked to health systems development. This emphasis on the need to report on service delivery indicators, while somewhat understandable in the context of GH/PRHs traditional focus on family planning service delivery indicators, has tended to distort the emphasis of the project from its

intended focus on HRH systems development to an emphasis on HIV/AIDS and FP/RH service delivery.

The Mission Survey was conducted by USAID/GH/PRH and OHA. The survey was sent to 13 missions in countries where Capacity has worked, over the last two to four years, with eight countries responding.

The survey consisted of 12 questions (most were open-ended) that sought the opinion of field mission staff on the work of the Capacity Project and their interest in using a centrally managed HRH project in the future. The majority appreciated the contribution of the project, and confirmed that their needs were

met and that Capacity demonstrated general responsiveness and technical proficiency. Most of the countries in which Capacity implemented core-funded activities stated that these complemented field-supported activities. Many also confirmed the benefit of a centrally managed project for reasons of

assured access to Capacitys technical expertise in HRH in a setting where most field missions suffer from a dearth of HRH expertise. With reference to the structure of a future centrally managed HRH program, the majority prefer a Leader with Associate cooperative agreement. A survey focused solely on

interest in using a centrally managed HRH project was also sent to 13 missions in countries where Capacity has not worked; only two responses were received. Only one indicated an interest in a centrally managed project.

Project Findings and Accomplishments: Achievements by Intermediate Results

As discussed in the preceding section in which principal project achievements were assessed, it would

appear that the project is on track to achieve the great majority of deliverables for each of the projects 12 key indicator areas. The following sections of the report highlight accomplishments in each of the three intermediate results areas.

IR 1: Improved workforce planning and policy

The most significant contribution under IR1 relates to the strengthening of a human resources information

system (HRIS) in Lesotho, Namibia, Rwanda, Southern Sudan, Swaziland, and Uganda. Efforts are underway to introduce HRIS in Kenya and Tanzania. While this activity is a work in progress, the Capacity Project is already recognized as a world leader in this field. Under the technical direction of

Capacity Project, development of the HRIS focused primarily on the design of no-cost, user-friendly, open source software. Once satisfied that the HRIS was technically sound in its conceptual framework, Capacity Project IT staff provided technical and financial assistance in the softwares installation,

customized the software to meet the specific needs of countries, and trained local staff to manage the system. In each country in which the HRIS was and is being introduced, a stakeholder leadership group is formed to play a key role in this endeavor, thus promoting country-specific ownership of the HRIS. After

installing the HRIS, a Data for Decision Making workshop is held in each of the client countries to promote understanding of the systems value and to enhance the appropriate use of the information once it

is collected. To date, workshops have been held in Uganda and, through the Regional HIV/AIDS Program (RHAP) Associate Award, in Swaziland. According to one expert IT respondent to this evaluation, the work that Capacity Project has undertaken in HRIS, based on the robust engineering design of the

software, on its user-friendly framework and on the fact that it has been shown to work across countries, is the best amongst 150 [similar] tool sets developed for low-resource countries.


Building on the above, the project is developing three no-cost and Open Source core software solutions, each addressing a specific HRH leadership issue: iHRIS Qualify (training, certification and licensure

database), iHRIS Manage (HR management system), and iHRIS Plan (workforce planning and modeling software). At this point in the Capacity Projects development, participating countries are at different stages in their application of the system. For example, Uganda has installed iHRIS Qualify in all four

medical professional councils to track all professionals from the time they enter school until they leave the workforce. In addition, iManage has been installed at the central level of Ugandas Ministry of Health. In Rwanda, Capacity Project has introduced iHRIS Manage and data are being imported. In

Kenya, local developers have customized iHRIS Manage and Qualify and data are being entered, and in Zanzibar iHRIS Manage has been set up. In three other countries (Southern Sudan, Namibia, and

Lesotho), the project is assisting these three countries with improving their existing health information systems. Despite the initial success associated with the introduction of the HRIS, HRIS has yet to be integrated with other information systems within ministries of health (HMIS) and with other government

ministry data systems, for example the Ugandan Public Service Commissions payroll database.

While the ultimate goal of the HRIS is to have the information available at both the national and district levels and for it to be readily accessible for planning and management purposes, this vision, at the moment, appears to be a distant goal. Stated simply, the effort to extend the HRIS nationwide in client

countries is frustrated by field conditions in all Sub-Saharan African countries, all of whose supporting IT infrastructures can be charitably characterized as abysmal. Moreover, based on findings associated with the evaluation teams admittedly limited field visits in Rwanda and Uganda, extending the HRIS beyond

the confines of each countrys capital city is going to be highly challenging. Countries and donor agencies will need to address such critical issues as the paucity of hardware and supporting software, insufficient sources of power to support high-speed/broadband connections, the marked lack of basic computer skills

among health systems managers and health service staff, and poor facilities to maintain computers and Internet connectivity. On a positive note, the Government of Uganda has already initiated measures to strengthen Internet connectivity in 43 districts and plans to roll it out to the rest of the country within two

to three years.

The Capacity Project has contributed to building leadership and management skills among individuals working on HRH in ministries of health, districts, and FBOs through technical assistance, workshops, and

in-service training. Looking to the future, Capacity has prepared the foundation for long-term HRH planning and system strengthening, through assistance provided in the development of HRH strategic plans in Lesotho, Namibia, Rwanda, and Uganda and via two HRH Action Workshops that brought

together 79 carefully chosen HRH leaders from just under 30 countries in Sub-Saharan Africa.

In Uganda, the Capacity Project promoted the HRH Action Framework (HAF) model for consensus building through the facilitation of country consultations among all stakeholders, the identification of issues, and the development of solutions. This approach to problem solving and strategic planning is still

in its final development stages where a multisectoral committee of stakeholders, including representatives of ministries of health, education, finance and planning, the Public Service Commission, professional associations, and health care providers, formed a Stakeholder Leadership Group to meet regularly to

discuss HRH issues and to develop strategies. In anticipation of central-level recommendations to apply the HAF approach at the district level, Capacity has met with nine district-level officials and district health teams to develop HRH action plans.

An illustrative example of an issue addressed through the above participatory process centered on a

perennial problem in Uganda associated with the delayed hiring of health workers and the entry of new hires into the payroll system so that health workers receive remuneration on time. In addressing this issue,

Capacity Project staff met with the payroll administrator at the Public Service Commission, a member of the Stakeholder Leadership Group, and together toured nine districts in the north to conduct a needs assessment and to find a solution for the problem. The issues were found to be relatively minor and easily


addressed (for example, providing training and per diem to payroll staff to go to the field). Other more complex issues such as incentives to attract and retain health workers in hard-to-reach areas posts are

under discussion and likely to take more time. Based on results of the application of HAF in Uganda, Capacity Project staff are considering potential application of the HAF in Vietnam, Latin America and the Caribbean, and Botswana during the projects final year.

IR 2: Improved Workforce Development

Workforce development evolved into a focus

area more slowly than anticipated, apparently due to fewer requests from USAID field missions and, as noted earlier, to a lack of

interest on the part of host countries. However, the Capacity Project has achieved substantial gains in implementing specific objectives in

workforce development in such areas as pre-service education, in-service training, strengthening professional associations, and task

shifting. The lack of anticipated requests in this area could also be a reflection of the capacity

that already exists in many countries for in-service training as well as within regional training institutions such as AMREF, CAFS,

and others. During the first few years, only Rwanda and Lesotho requested assistance in pre-service education (four more joined

subsequently), and in the fourth year of the project 12 countries (seven in Africa) have submitted requests for assistance in pre-service

education (updating curriculum) and in strengthening and rolling out a responsive in-service training program. The promotion of

Learning for Performance as an important pedagogic tool attracted interest and resulted in more requests for support in pre-service education. As noted in Capacitys Self-Assessment, there are a

number of systemic and technical issues associated with introducing change in pre-service education. Among these issues is the fact that multiple stakeholders from multiple government agencies regulate training programs, training institutions, and professional associations. In addition, curricula modification

is a time-consuming process exacerbated by the inflexibility of the senior teachers and professors and by their resistance to accept change. While not yet introduced by Capacity, engaging academics, leaders of

training institutions, and professional associations from the very beginning in a nonthreatening manner by providing incentives (infrastructure support) and encouraging them to take leadership in reviewing training programs and in implementing change might well represent a means of addressing current

professional intransigence to curriculum modification.

Despite the lack of anticipated requests in workforce development cited in the above paragraph, Capacitys experience in Rwanda, as illustrated in Box 2, represents an example of the way in which the project, when presented with the opportunity, has applied a proactive and rational approach to workforce

development, especially with reference to the repositioning of family planning. Accordingly, the following three paragraphs highlight a few examples of Capacity Projects HRH leadership in Rwanda.

Box 2. The Capacity Project in Rwanda

Producing HRH Results in a Proactive

Environment through:

Training of 193 providers in family planning

Applying Learning for Performance concepts in the adaptation of FP curricula

Providing operational and technical support on FP and HIV/AIDS to five

nursing and midwifery schools

Facilitating a situation analysis on the repositioning of an FP vasectomy initiative

Integrating gender competencies into FP curricula

Designing, implementing, and disseminating a study on health service

workplace violence

Promoting the constructive engagement of men in FP/ANC/PMTCT/VCT services

Procuring and introducing the use of IEC materials in pre-service FP and HIV/AIDS

training.


Rwandas Ministry of Health, assisted by the Capacity Project, assessed the status of the nations HRH and concluded that the quality of education provided to nurses, the nations largest group of health

workers, had been unregulated and below standard, with many of the nations current nurses having had inadequate educational backgrounds to enter nursing school. Based on the assessment, the government closed all of the existing nursing schools and, with help from Capacity Project, leveraged assistance from

Belgian Technical Aid, developed a new nursing and midwifery competency-based training curriculum, and opened five new schools of nursing, which are now training 500 students annually as registered nurses and midwives. In addition, with assistance from Capacity, the tutors in the new schools have

received teacher training. The curriculum for both teachers and students was competency-based, integrating FP/RH, HIV/AIDS, MCH, and other health care foci. Within the next two years, graduates of

the new school will have a career path to move to nursing sub-specialties (operating theatre nurse, pediatric nurse, etc.), B.Sc. in Nursing, and public health. As a final step in this process, the government and Capacity Project are developing a strategy to upgrade the skills of existing nurses through in-service,

on-the-job training.

In addition to addressing ways in which to upgrade the curricula for nurses and midwives, the Rwanda assessment facilitated by the Capacity Project revealed that poor infrastructure was a major obstacle that needed to be addressed prior to engaging in implementation of educational programs. Accordingly, with

funding provided through USAID field support to Capacity, an integral part of the implementation of pre-service nursing education included the renovation of buildings and basic infrastructure (electricity and water supply) and the purchase of IT hardware, software, library books and journals, and teaching

materials.

The Capacity Projects involvement in Rwanda also demonstrates how a systematic and targeted program of in-service training of health workers can result in the increased and relatively rapid availability of critical HRH. For example, the project has, to date, trained 547 nurses in integrated gender-sensitive

family planning/HIV clinical services: Capacity expects to train 389 additional nurses within the remaining year of the project. In addition, 18 providers have been provided with in-service training in emergency obstetric and neonatal care while six physician/nurse teams in two hospitals in Shyira and

Byumba have received training under Capacity Projects vasectomy training initiative. Based on trainee feedback provided to the Ministry of Health with reference to their vasectomy training, the government

has requested that the initiative be scaled nationally. As noted, the supply of health workers now trained or that will be trained prior to the end of the project represents a strong base with which to support future progress on Rwandas family planning and reproductive health program.

In Lesotho, Swaziland and, more recently, Ethiopia, the Capacity Project has addressed the need for

capacity development of community health workers (CHWs). However, in neither Rwanda nor Uganda, the two countries visited by the evaluation team, has the project directed attention toward CHWs. This cadre of health workers constitutes the interface between communities and the formal health system and

is recognized as necessary for the attainment of the health-related Millennium Development Goals and for equity in the provision of services. Of the nine missions responding to USAID/PRH/SDIs user survey, two of the current Capacity Project mission partners noted that the need for CHW capacity development

should have been addressed under the current project while nine of the total number of 11 missions responding to both surveys identified CHW capacity development as a priority for the future.

Capacity Project has produced a brief on strengthening health professional associations to assist participating countries in the design and development of CPD initiatives. Building on this initiative,

Capacity Projects program in Ukraine has focused on developing the management, finance, and operations capacity of the Ukrainian Association of Obstetricians and Gynecologists. The curriculum

developed under this initiative has now been integrated as part of Ukraines national standards and the Association has taken the lead, with Capacity assistance, in providing its members with continuing education related to increasing their skills and knowledge in emergency obstetric care and in family

http://www.capacityproject.org/index.php?option=com_content&task=view&id=26&Itemid=49http://www.capacityproject.org/index.php?option=com_content&task=view&id=26&Itemid=49


planning. The project has also provided training on communication skills to nurse trainers associated with the Ugandan Nursing and Midwifery Association. In discussions with the evaluation team, senior

directors of the association stated their belief that the assistance provided has helped improve the image of nursing profession in Uganda. Attempts are also being undertaken to develop a regional initiative for this purpose under ECSA.

Finally, Capacitys promotion of task shifting as a strategy to mitigate the acute and severe shortage of

fully trained health workers by training others with lower entrance requirements for specific tasks appears to hold great promise for the future. For example, in Mali, the project has shown that matrones (auxiliary midwives) in the place of nurses or physicians can perform active management of the third stage of labor

and administer uterotonic drugs according to national standards. This supports policy changes to allow this cadre, who attend most of the countrys births, to use the practice legally, which is known to reduce

the risk of postpartum hemorrhage. As another example, the project documented and created implementation guidelines for a promising HRH practice developed by a Ugandan nongovernmental org

EVALUATION OF THE CAPACITY...

Documents

Transcript of EVALUATION OF THE CAPACITY...