Final Report for the Program Evaluation of the Applied ... · CDC-SMDP CDC-Sustainable Management...

Final Report for the Program Evaluation of the Applied Leadership Development Program for District HIV & AIDS Coordination in Botswana

November 2016

Grant # 5U2GPS001617

1

Contents

Contents ........................................................................................................................................................ 1

Acronyms ...................................................................................................................................................... 2

Background Information ............................................................................................................................... 3

HIV in Botswana ........................................................................................................................................ 3

District AIDS Coordinators ........................................................................................................................ 3

Coordinating Partners ............................................................................................................................... 3

Applied Leadership Development Program for District HIV & AIDS Coordination ................................... 3

Methods ........................................................................................................................................................ 5

Data Collection Methods and Tools .......................................................................................................... 5

Ethical Considerations ............................................................................................................................... 6

Data Management and Analysis ............................................................................................................... 6

Findings ......................................................................................................................................................... 6

Overview of Participants ........................................................................................................................... 6

DAC Self-Assessment ................................................................................................................................ 6

ALDP Training Components ...................................................................................................................... 8

ALDP Training Impact ................................................................................................................................ 9

ALDP Implementation ............................................................................................................................. 10

District Program Data Review ................................................................................................................. 10

Discussion ................................................................................................................................................... 11

Limitations............................................................................................................................................... 11

Recommendations ...................................................................................................................................... 12

Appendix A: Supervisor Interview Guide .................................................................................................... 13

Appendix B: Interview Guide for MLG&RD staff ......................................................................................... 16

Appendix C: DAC Interview Guide .............................................................................................................. 17

Appendix D: Focus Group Guide for ALDP Participants .............................................................................. 19

Appendix E: Focus Group Guide for ALDP Master Trainers ........................................................................ 20

Appendix F: DAC Self-Assessment Tool ...................................................................................................... 21

Appendix G: District Data Review ............................................................................................................... 29


2

Acronyms

ADAC Assistant District AIDS Coordinators

ALDP Applied Leadership Development Program

CDC U.S. Centers for Disease Prevention and Control

CDC-SMDP CDC-Sustainable Management Development Program

DAC District HIV/AIDS Coordinator

DMSAC District Multi-Sectoral AIDS Committee

MLG&RD Ministry of Local Government and Rural Development

NACA National AIDS Control Agency

NASTAD National Alliance of State and Territorial AIDS Directors


3

Background Information

HIV in Botswana

The country of Botswana is a middle income landlocked country located in Southern Africa with an estimated population of 2.03 million (Pop census 2011). According to the 2013 UNAIDS Global AIDS Report1, Botswana is among countries in Southern Africa with the highest burden of HIV & AIDS. The Botswana AIDS Impact Survey (BAIS) of 2013 estimates that the national prevalence rate increased to 19.03% compared to the 17.6% from the 2008 BAIS III. Additionally, the BAIS estimated the incidence rate in 2013 at 2.47% compared to 2.9% for the 2008 BAIS. HIV Prevalence was found to be higher amongst females than males with 19.2% and 14.1%, respectively, suggesting that young women are at higher risk for HIV infection than their male counterparts.

District AIDS Coordinators

In Botswana, District AIDS Coordinators (DACs) are critical to the management and coordination of the district response to HIV. As secretariats to the District Multi-Sectoral AIDS Committees (DMSACs), they manage government funds allocated for HIV/AIDS, facilitate the development of a multi-sectoral annual HIV/AIDS Action Plan, support local level capacity building for implementation, mobilize resources, coordinate strategic implementation partnerships across sectors, and monitor and document district responses. Since 2000, as the national HIV/AIDS response evolved, the role of the DAC has changed significantly as the DAC’s Office adjusted to expectations on the part of different national and local stakeholders.

Coordinating Partners

The National Alliance of State and Territorial AIDS Directors (NASTAD) is an alliance of the United States’ (U.S.) chief state health agency staff with programmatic responsibility for administering HIV/AIDS and viral hepatitis health care, prevention education, and support services programs funded by the states and the federal government. NASTAD’s mission is to end the intersecting epidemics of HIV, viral hepatitis, and related conditions by strengthening domestic and global governmental public health through advocacy, capacity building, and social justice. NASTAD’s Global Program works to build the organizational, programmatic and human resource capacity of its regional/state public sector AIDS program counterparts across the world.

The Centers for Disease Control and Prevention-Sustainable Management Development Program (CDC-SMDP) worked with ministries of health, educational institutions, nongovernmental organizations, and other partners to strengthen leadership and management skills and systems to improve public health in low resource countries. CDC-SMDP strategies included integration with country public health priorities, strategic partnerships, technical assistance and training, policy and systems development, advocacy and education, and evaluation.

Applied Leadership Development Program for District HIV & AIDS Coordination

The Botswana Ministry of Local Government & Rural Development (MLG&RD) was seeking ways to better equip and support District HIV/AIDS Coordinators and other staff in responding to the dynamic national HIV epidemic. NASTAD, in collaboration with CDC-SMDP was asked by MLG&RD to support the development of the “Applied Leadership Development Program (ALDP) for District HIV & AIDS

1 Global report: UNAIDS report on the global AIDS epidemic 2013. http://www.unaids.org/sites/default/files/media_asset/UNAIDS_Global_Report_2013_en_1.pdf


4

Coordination” whose goal was to build and strengthen the identified capacity needs for DACs to be effective in addressing the HIV epidemic in Botswana.

Recognizing that a standardized program management and leadership curriculum is less likely to achieve the identified goal, the team determined to develop an applied training specifically for the DACs. Role-specific capacity needs and gaps were identified in a 360 process that included the cohort, those they supervise, and those who supervise them2. The cohort and other stakeholders were engaged in designing training content, and training was conducted via short, intensive applied training sessions interspersed with field level assignments. For sustainability, master trainers were identified from within the MLG&RD, who also acted as trainee mentors.

Figure 1. Applied Public Health Program Management Capacity Building Model

In 2012, the MLG&RD identified 10 current DACs, Assistant DACs (ADACs) and MLG&RD personnel to serve as master trainers for the development and implementation of ALDP. The master trainers were selected based on their years of service, exceptional performance in their respective roles, and availability to actively participate in all facets of ALDP.

A pilot of ALDP was conducted between October 2012 and April 2013. Each of the 10 master trainers participated in intensive one-week capacity building activities to improve facilitation skills and training content knowledge, prior to implementing ALDP with a cohort of nine DACs and ADACs. Lessons learned, recommendations and feedback from the pilot were summarized in an evaluation summary, and the results were used to finalize the ALDP curriculum content and facilitator guide.

In October 2013, ALDP was officially launched with support from MLG&RD, CDC-Botswana and NASTAD Botswana. Fifteen more DACs and ADACs participated in this second cohort, with facilitation provided by the master trainers. NASTAD continued to provide capacity support to the master trainers prior to each week of ALDP training. In May 2014 NASTAD officially “handed over” ALDP to MLG&RD, however it is unclear whether it has been institutionalized as a MLG&RD DAC mandatory training component for all current and new DACs and ADACs.

2 District AIDS Coordinator Capacity Assessment Report, April 2012 https://www.nastad.org/sites/default/files/botswana_dac_aphpmt_assessment_final.pdf.

https://www.nastad.org/sites/default/files/botswana_dac_aphpmt_assessment_final.pdf


5

Methods

The goals of the ALDP evaluation were:

1. Determine whether ALDP increased DACs competencies in the desired areas, 2. Determine whether ALDP impacted program performance, and 3. Determine whether ALDP has been integrated into the MLG&RD ongoing training of DACs.

To gather the needed information, NASTAD used a mix of qualitative and quantitative methods which included:

1) Guided interviews with the supervisors of DAC/ADACs, DAC/ADAC’s who participated in ALDP, and MLG&RD lead staff for ALDP implementation

2) Focus group discussions with ALDP participants, including master trainers 3) Completion of self-assessment by ALDP participants 4) Review of program data submitted to MLG&RD by ALDP participants for a period of four years

(2011 – 2015)

Data Collection Methods and Tools

1. Guided Conversations: Using the guides in Appendices A, B and C, NASTAD staff conducted interviews with supervisors of the training participants, ALDP training participants and MLG&RD lead staff. A total of ten conversations were conducted, six with DAC/ADAC participants, two with supervisors and two with MLG&RD lead staff.

2. Focus Group Discussions: Focus groups were conducted with both Master Trainers and training participants, using the guide in Appendices D and E. Three focus groups, with a total of 26 participants, were conducted. Two were conducted with training participants and one with Master Trainers.

3. DAC self-assessment: ALDP participants were asked to complete the same self-assessment tool that had been used to gather information prior to the development of the curriculum. This tool (Appendix F) provided additional information on the DAC’s self-perceived competency for doing their job. A total of 31 individuals completed the self-assessment.

4. District Program Data Review: NASTAD conducted a review of the data submitted by the districts to MLG. The review was conducted to determine the impact of ALDP on program performance. Data was collected on the frequency and attendance of DMSAC and TAC meetings among other variables. Data was collected for 8 districts over a four-year period to account for pre- and post-ALDP submissions.

All interviews were conducted by two NASTAD Headquarters staff who had not previously been engaged in any work in Botswana and who were not familiar to ALDP participants. One individual acted as the interviewer and one as the note taker. In addition, all interviewees were asked permission to record the interviews. In the case of the focus group discussions, a third staff member joined the team to serve as an additional note taker. All interviews and focus groups were conducted in English. Guided conversations were conducted at a site convenient to the participant, most often a private space at their workplace. One of the focus groups was conducted in the conference room at MLG&RD in Gaborone, and the two others at a local hotel in Francistown.


6

Ethical Considerations

All interview and focus group participants were asked to provide consent to participate and were encouraged not to disclose the content of their participation with others. Consent was obtained using the forms in Appendices A, B, C, and D. NASTAD study team members provided a copy of the consent form to the participant and then read it to them in its entirety. If the participant agreed to participate, s/he was then asked to consent to audio recording. Consent was documented individually both in writing and verbally for both interview and focus group participants.

For the DAC Self-Assessment, consent was obtained in two different ways. Those completing the self-assessment electronically were asked to check a box indicating that they had read the consent and were willing to participate before continuing to complete the survey. For those completing the survey in paper form, the initial page of the survey contained the consent information and they were asked to read and indicate consent prior to continuing to fill out the assessment.

All focus group participants were provided with reimbursement of any associated transportation, per diem and accommodation costs as per MLG&RD reimbursement rates. Interview participants were not given any incentive as all interviews took place during regular work hours.

Ethical clearance for this evaluation was obtained from the Health Research and Development Division of the Botswana Ministry of Health and the US Centers for Disease Control and Prevention.

Data Management and Analysis

Each interview and/or focus group was assigned a number prefaced by initial that corresponded to the type of participant (DAC, MLG, DAC_FGD, etc.) and a sequential number. This allowed the study team to log the information in the data collection log and in the database. No names were associated with any of the completed instruments.

All interview notes were entered into a Word document for each interview by the notetaker using their notes and the audio recording if one was available. Then notes taken by the interviewer or facilitator were referenced to review documentation, add additional detail, and fill any gaps in documentation.

Data from the interviews and focus groups were converted to rich text files and imported in to ATLAS.ti (Scientific Software Development, Berlin, Germany) for qualitative analysis. Standard codes were used to highlight qualitative data according to thematic areas. Data from the district data review and self-assessment were entered into Excel spreadsheets and analyzed using Excel and SPSS.

Findings

Overview of Participants

A total of 33 individuals participated in the evaluation, with some participating in a focus group, guided conversation, and completing the DAC self-assessment. Thirty of the 33 (90%) were current ADACs/DACs, and three were staff at the MLG&RD central level. Among the DAC/ADAC participants there were representatives from all the ALDP trainings conducted, 12 had been trained in 2012, 14 in 2013 and seven in 2014.

DAC Self-Assessment

Thirty-one of the 40 (77%) individuals that had participated in at least one of the ALDP trainings completed the self-assessment.


7

Decision Making, Collaborating and Partnering, and Monitoring & Evaluation were mentioned by 94 percent of the DAC/ADACs as key duties of the position, followed by Financial Planning/Budgeting (87%), Oral and Written Communication (84%), and Data Analysis, and Time Management (81%). See Figure 2 for additional information.

When asked to rank the top five job duties, 45% listed Strategic Planning as a top priority, followed by Collaborating and Partnering (19%), Project Management (13%), Coordination (10%), and Program Planning and Monitoring and Evaluation (6%). M&E and Collaborating and Partnering were the only two activities listed as ranked in the top five by all participants.

Participants were asked “How has participation in ALDP affected your skill level to perform this task?” in nine categories with the possible answers being decreased (1), no change (2), increased (3), and not part of my job (0). Table 1 below shows the mean results and 95% CI for each of the areas based on all participant’s answers that were not coded with “not part of my job”.

Table 1. Self-Assessment of Skill-level Change for ALDP Participants (n=31)

Category Mean 95% CI of the Mean

DMSAC Secretariat 2.84 (2.74, 2.94)

Planning and Priority Setting 2.73 (2.61, 2.86)

Monitoring and Evaluation 2.66 (2.54, 2.79)

Health Promotion and Disease Prevention 2.77 (2.67, 2.87)

Capacity Building Training 2.53 (2.20, 2.85)

Partner Collaboration 2.74 (2.53, 2.95)

Resource Mobilization 2.55 (2.31, 2.79)

Human Resource Management 2.58 (2.29, 2.87)

Logistics and Procurement 2.67 (2.44, 2.90)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Figure 2. Key Management Duties


8

Analysis of the categories by year of training did not show any significant differences in the responses. Based on the table above, ALDP participants felt their skills increased in all areas, with skills related to DMSAC Secretariat and Health Promotion and Disease Prevention getting higher average scores.

ALDP Training Components

Participants in guided conversations and focus groups were asked about which of the ALDP training components (curriculum, mentoring and field project) were most useful, challenging and enjoyable, as well as, which of the curriculum topics were most useful. The in-class training and field assignment were mentioned by all as being most useful, while the timeframe for field assignment implementation and the mentorship were identified as the most challenging (Figure 3). Participants thought some of the enjoyable attributes of ALDP were the fact that it was relatable to their current work, the opportunity to meet and learn from their peers, and that it provided an overview and/or refresher of topics that were important to their positions.

All participants were also asked which of the eleven topics covered during the in-class training they thought were most useful to their day-to-day work. The top three mentioned topics were Collaborating and Engaging Partners to Mobilize Resources (55%), Effective Meeting Planning and Facilitation (55%), and Data for Monitoring, Evaluation and Decision Making (64%). All topics were mentioned by at least one participant during either an interview or a focus group. (Table 2)

Figure 3. ALDP Components chosen as Most Challenging, Most Useful by Participants

Table 2. ALDP Topics Most Useful to day-to-day work (n=11)1

Topic Number (%)

Introduction to Public Health 1 (9)

HIV in Botswana 2 (18)

Implementing National Strategies and Policies 1 (9)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Curriculum

Field Assignment

Mentoring

Project Time Frame

Mentoring

Sup/Master Trainers DAC/ADAC


9

Collaborating and Engaging Partners to Mobilize Resources 6 (55)

Reprioritizing and Reallocating Resources 5 (45)

Communicating Effectively 5 (45)

Effective Meeting Planning and Facilitation 6 (55)

Preparing for Implementation 1 (9)

Building Teams and Managing Employees 4 (36)

Managing Budgets 4 (36)

Data for Monitoring, Evaluation and Decision Making 7 (64)

1Includes six DAC/ADAC interviews, 2 Supervisor Interviews, and 3 Focus Groups

ALDP Training Impact

Participants were also asked how participation in ALDP had impacted their ability to do their day-to-day work and how they were applying the skills learned during the training. Participants talked about increased confidence presentation and facilitation skills and providing feedback to stakeholders; increased skills in presenting and meeting planning and facilitation, and ongoing use of tools from the training such as the meeting checklist. Table 3 shows some selected reports in the participants’ own words.

Table 3. Impact of ALDP Training Participation as reported by Participants

Thematic Area Representative Quote

Confidence “Gained confidence in presentation skills, …, gives the ability to become well-versed/more competent in a new topic”

“Boosted my confidence.”

“Now he is more proactive…”

“Have gained more confidence in facilitating meeting, even with people who are in higher position because I can now talk to them.”

Planning and Meeting Facilitation

“Yes – effective planning of DMSAC meetings.”

“Preparing for a meeting. Communicate in advance ….. Learn to follow-up well in advance [of DMSAC meetings].”

“Facilitating meeting; the way that you write the invitation, indicating purpose, these are slight changes that make a difference; and meetings are timed”

“ALDP has helped me appreciate the differences among people, the knowledge they bring and their background, drawing out for participation those that maybe don’t talk as much.”

Meeting Minute Development

“As the Secretary to the DMSAC, minute writing is an important part of the DAC’s job. Clear difference in what I was doing before and after. Helped me know what to write and how to present the minutes, … talking about important points, …. Helpful layout.”


10

“Another tip learned was to write down the minutes shortly after the meeting…”

Communication “Helped be better leader, communication – ensuring you are following all of the steps of communication”

“Communication has also improved as one reflects on whether it should be an e-mail, letter…”

“Better communication with stakeholders in …”

Stakeholder Engagement

“Able to identify the least and most important stakeholders and knowing when to contact them.”

“He went to speak to the leaders of the organizations, ... show them the importance of attending the meetings and now they are more engaged.”

“…initially he would wait for them…Now he reaches out to them and does the necessary work…”

“Analyzing stakeholders, had a passive implementer in district … figured out a way to actively involve this partner in the district activities.”

ALDP Implementation

Participants were asked about the way ALDP is being implemented, whether there were any changes they would make to ALDP, and whether it should continue. Most participants felt strongly that ALDP was a very beneficial training and that it should continue to be offered. In fact, participants thought that ALDP should be part of a DAC/ADAC’s orientation shortly after they were assigned to the post. Other suggestions for changes were to create an advanced version of ALDP as continuing education for those who had completed the initial training, and to add additional topics to the curriculum, to make ALDP available to everyone in the DAC’s office, as well as, other partners.

In terms of changes to the current implementation, participants thought that more time was needed for development and implementation of the field assignment and that more assistance and feedback should be given as projects are developed and implemented. Several participants mentioned that the curriculum should be updated on an ongoing basis, especially the modules “HIV in Botswana” and “Implementing National Strategies and Policies”. Participants also spoke to the need for District Commissioners to be more engaged with ALDP through their role as DAC supervisors to ensure buy-in and support. Mentor assignment was another area that participants thought needed some changes. Suggestions included participants being able to write down their top choices of mentor, being able to use someone other than a Master Trainer such as a previous participant or someone in their district, and development of specific expectations for the mentor/mentee relationship.

District Program Data Review

NASTAD conducted a review of the data submitted by the districts to MLG to assess whether participation in ALDP improved program performance. Data was collected for 8 districts over a four-year period to account for pre- and post-ALDP submissions, and included information on the frequency and attendance of DMSAC and TAC meetings among other variables. However, because DACS and ADACs were re-assigned to different districts after completing the ALDP, changes in quality of data reports generated pre ALDP and post ALDP could not be attributed to changes in performance of individuals


11

completing the reports, and this data could not therefore be used to inform the evaluation of the ALDP program.

Discussion

As resources for HIV prevention have diminished in Botswana the role of the DAC/ADACs has shifted and demands have increased, understanding the extent to which ALDP provides information and tools that assist DAC/ADACs in doing their job becomes ever more important.

Goal 1: Extent to which ALDP increased DACs competencies in the desired areas.

Since the population of DACs that participated in initial competency assessments was not the same as the population that participated in the training activities, this evaluation cannot determine whether the competencies of individual DACs were increased. However, DACs participating in the training indicated that they felt that their skill level had improved in all training areas (Table 1), and that they were applying their new skills in their day to day work (Table 3).

Previous ALDP participants indicated that ALDP provided a very thorough introduction to topics and tools needed for conducting the DAC/ADACs responsibilities. Participants felt that given the diminishing resources it was important that all DAC/ADACs were exposed to ALDP, preferably soon after (within 3 months) of taking up the post. Participants felt all facets of the training, in-class curriculum, field assignment, and mentoring, were useful but needed some changes. Most of those interviewed for the evaluation thought that the timeframe for field assignment development and implementation was not sufficient.

Participants also felt strongly that it would be important for ALDP participants to have options for advanced training and ongoing supportive supervision.

Goal 2. Extent to which ALDP impacted program performance

Because district data review could not be used to draw any conclusions regarding the impact of ALDP on data quality, it is hard to determine the extent to which ALDP impacted program performance. However, ALDP participants talked about increased confidence in presentation and facilitation skills and providing feedback to stakeholders; increased skills in presenting and meeting planning and facilitation, and ongoing use of tools from the training such as the meeting checklist (Table 3), suggesting that individual DACs felt that the ALDP was positively impacting their performance.

Goal 3. Determine whether ALDP has been integrated into the MLG&RD ongoing training of DACs.

The ALDP was designed specifically to promote integration into the MLG&RD, and was successful in this effort to the extent that MLG&RD identified and trained master trainers for the development and implementation of ALDP, the master trainers conducted three cycles of training with support from NASTAD. In May 2014, NASTAD turned over the training to MLG&RD, and MLG&RD conducted at least one cycle of training entirely independent of NASTAD. However, NASTAD is unable to determine whether use of ALDP has been ongoing by MLG&RD since that time.

Limitations

There are some limitations to consider when interpreting these findings. First, all information was obtained through self-report which may lead participants to tell interviewers what they think they would want to hear. Second, while most of those participating in ALDP participated in the evaluation, only a small number participated in individual interviews and due to turnover of District Commissioners and their lack of knowledge about ALDP, evaluators were not able to conduct as many supervisory interviews as planned. Third, the tools used were not piloted prior to implementation and thus an


12

opportunity may have been missed to improve the tools and better target questions to each participant and participant group. Fourth, the district data review was not able to be used to draw any conclusions regarding the impact of ALDP on data quality given that DACs and ADACs were re-assigned to different districts after completing ALDP, and as such there was no continuity among the individual completing the data reports. Finally, while the evaluation team had not previously worked in Botswana or with ALDP, they were NASTAD staff and as NASTAD was a partner to MLG&RD it is possible that this could have introduced some response bias.

Recommendations

The findings of this evaluation provide evidence to MLG&RD that ALDP is a training that is well-thought of, useful to and appreciated by the participants, providing them with knowledge and tools necessary to do their job. Participants reported gaining skills and confidence that improved their performance. These are positive findings that show ALDP participation should continue for all DACs and ADACs.

There are however opportunities to both improve upon and expand ALDP to provide further support. Below is a list of recommendations based on the evaluation findings:

1. Field assignment timeframe: This timeframe should be extended to 12 months to provide participants the opportunity to develop and implement the improvement action and collect data on the impact of the action. This would provide participants with the opportunity to present results of the field assignment, not just speak to implementation. This would also allow for participants to go through the Monitoring and Evaluation module before presenting their projects.

2. Curriculum: The curriculum should be continuously updated to ensure up-to-date information and, if possible, consider expanding the topics of conflict resolution and reprioritization of budget resources.

3. Ongoing Opportunities: Develop an advanced version of ALDP as an opportunity for Master Trainers and those that have completed the initial ALDP to participate in as continuing education. Develop opportunities for ongoing supportive supervision.

4. Mentor/Mentee relationship: Analyze the possibilities for changing how assignments are made and allow mentees some choice. Develop clear expectations for the relationship in terms of number, frequency and timing of contact between mentor and mentee.

5. Timing of Training: Consider using ALDP as part of the orientation of DAC/ADACs to their new position and ensure participation no later than three months following appointment to the post.

DACs and ADACs play a critical role in the coordination of the HIV response at the district level and ALDP provides them with the tools and knowledge needed to do their job. As MLG&RD continue to implement and improve ALDP this report may be a useful reference.


13

Appendix A: Supervisor Interview Guide

Date: ___________________________

Name of Interviewer: ________________________

Name of Notetaker: ____________________________________

District: _________________________________________________

Interviewer: ask the following questions and explore, discuss, probe, as needed, the following topic areas. Interviewer introduces the topics that were covered during the two-week ALDP training curriculum topics:

Introduction to Public Health

HIV in Botswana

Implementing National Strategies and Policies

Collaborating and Engaging Partners to Mobilize Resources

Reprioritizing and Reallocating Resources

Communicating Effectively

Effective Meeting Planning and Facilitation

Preparing for Implementation

Building Teams and Managing Employees

Managing Budgets

Data for Monitoring, Evaluation & Decision Making

1. As mentioned, these are topics that are included with ALDP (reread list), are there any changes you can suggest for the curriculum?

2. Since participating in ALDP, are there any areas where you have seen changes in the DACs/ADACs performance? If yes, please elaborate on the changes.

Additional Probes: a. What areas have you seen the most improvement? b. Which areas do you feel more training or mentoring is needed?

3. Can you give an example where you have observed the DAC/ADAC applying skills discussed during the ALDP?

4. On a scale of 1 to 10, where one is not effective at all and 10 is very effective, how would you rate the DAC/ADAC’s planning of the DMSAC activities:

a. Prior to participation in ALDP? b. After participation in ALDP? c. Can you give an example on how the DAC/ADAC planning skills have changed?

5. On a scale of 1 to 10, where one is not effective at all and 10 is very effective, how would you rate the DAC/ADAC’s facilitation of the DMSAC meetings:


15

a. Prior to participation in ALDP? b. After participation in ALDP? c. Can you give an example on how the DAC/ADAC facilitation skills have changed?

6. On a scale of 1 to 10, where one is not effective at all and 10 is very effective, how would you rate the DAC/ADAC’s follow-up on DMSAC’s meeting action items:

a. Prior to participation in ALDP? b. After participation in ALDP? c. Can you give an example on how the DAC/ADAC follow-up on DMSAC action items has

changed?

7. On a scale of 1 to 10, where one is the lowest score and 10 is the highest score, how would you rate the following:

a. Completeness of M&E reports: i. Prior to participation in ALDP?

ii. After participation in ALDP? b. Timeliness of M&E reports?

i. Prior to participation in ALDP? ii. After participation in ALDP?

c. Can you give an example of how completeness and timeliness of the reports has changed?

8. Have there been changes in the number of queries regarding clarification of the M&E data since the DAC/ADAC participation in ALDP? Can you provide some examples?


16

Appendix B: Interview Guide for MLG&RD staff

1. Has there been another ALDP cycle (two week curriculum) planned since May 2014? a. If yes, has it begun?

b. If not, why not?

2. Can you describe plans for ongoing ALDP implementation? a. Content management and updating b. Financing c. Logistics management

3. Is there additional support needed for ongoing implementation? Please describe:

4. On a scale of one to four, where one is the lowest score and four is the highest score, how useful do you feel ALDP is to DAC performance and effectiveness in the following areas:

a. Planning b. Communication c. Data monitoring d. Data reporting and management e. Collaboration f. Can you provide some examples of how you see ALDP impacting the areas mentioned

above?

5. In your opinion, is ALDP a good investment for MLG&RD? Why or why not?


17

Appendix C: DAC Interview Guide

Date: ___________________________

Name of Interviewer: ________________________

Name of Notetaker: ____________________________________

District: _________________________________________________

Interviewer: ask the following questions and explore, discuss, probe, as needed, the following topic areas. Interviewer introduces the topics that were covered during the two-week ALDP training curriculum topics:

Introduction to Public Health

HIV in Botswana

Implementing National Strategies and Policies

Collaborating and Engaging Partners to Mobilize Resources

Reprioritizing and Reallocating Resources

Communicating Effectively

Effective Meeting Planning and Facilitation

Preparing for Implementation

Building Teams and Managing Employees

Managing Budgets

Data for Monitoring, Evaluation & Decision Making

1. As mentioned, these are topics that are included with ALDP (reread list), are there any changes you can suggest for the curriculum?

2. Since participating in ALDP, are there any areas where you have seen changes in the DACs/ADACs performance? If yes, please elaborate on the changes.

Additional Probes: a. What areas have you seen the most improvement? b. Which areas do you feel more training or mentoring is needed?

3. Can you give an example where you have observed the DAC/ADAC applying skills discussed during the ALDP?

4. On a scale of 1 to 10, where one is not effective at all and 10 is very effective, how would you rate the DAC/ADAC’s planning of the DMSAC activities:

a. Prior to participation in ALDP? b. After participation in ALDP? c. Can you give an example on how the DAC/ADAC planning skills have changed?

5. On a scale of 1 to 10, where one is not effective at all and 10 is very effective, how would you rate

the DAC/ADAC’s facilitation of the DMSAC meetings:


18

a. Prior to participation in ALDP? b. After participation in ALDP? c. Can you give an example on how the DAC/ADAC facilitation skills have changed?

6. On a scale of 1 to 10, where one is not effective at all and 10 is very effective, how would you rate the DAC/ADAC’s follow-up on DMSAC’s meeting action items:

a. Prior to participation in ALDP? b. After participation in ALDP? c. Can you give an example on how the DAC/ADAC follow-up on DMSAC action items has

changed?

7. On a scale of 1 to 10, where one is the lowest score and 10 is the highest score, how would you rate the following:

a. Completeness of M&E reports: i. Prior to participation in ALDP?

ii. After participation in ALDP? b. Timeliness of M&E reports?

i. Prior to participation in ALDP? ii. After participation in ALDP?

c. Can you give an example of how completeness and timeliness of the reports has changed?

8. Have there been changes in the number of queries regarding clarification of the M&E data since the

DAC/ADAC participation in ALDP? Can you provide some examples?


19

Appendix D: Focus Group Guide for ALDP Participants

Date: ___________________________

Name of Focus Group Facilitator: ________________________

Name of Notetaker: _______________________

No. of Participants: ___________________

1. What did you enjoy most about ALDP? Additional Probes: a) Any specific topics that were of particular interest

2. What was most challenging about ALDP?

3. In your opinion, which components of the training (e.g., training, mentoring, field assignment) have been most useful to you in your current job as a DAC/ADAC?

4. Of the topics discussed during the ALDP two-week training, which were most useful to you in your

current job as DAC/ADAC?

5. How has participating in ALDP impacted your day-to-day work?

Additional Probes:

a. How has it impacted your interaction with the DMSAC/VMSAC? b. How has it impacted data quality and reporting?

6. Please describe ways in which you have applied skills and/or topics discussed during ALDP?

7. How would you change or improve ALDP?


20

Appendix E: Focus Group Guide for ALDP Master Trainers

Date: ___________________________

Name of Focus Group Facilitator: ________________________

Name of Notetaker: _______________________

No. of Participants: _______________

1. What did you enjoy most about ALDP?

Additional Probes: a. Any specific topics that were of particular interest

2. What was most challenging about ALDP?

3. What was the experience of being a mentor to other DACs/ADACs like for you?

4. How did the capacity/skills building support that you received impact your ability to implement all the ALDP components?

5. Are there tools that you feel are missing, or that could be modified, from the ALDP facilitator guide to assist you with ongoing implementation of ALDP? If yes, what kinds of tools?

6. In your opinion, which components of the training (e.g., training, mentoring, field assignment) have been most useful to you in your current job as a DAC/ADAC?

7. How has participating in ALDP impacted your day-to-day work?

8. Please describe ways in which you have applied skills and/or topics discussed during ALDP?

9. How would you change or improve ALDP?


21

Appendix F: DAC Self-Assessment Tool

Background/Purpose:

NASTAD and the Ministry of Local Government and Rural Development (MLG&RD) are evaluating the impact of the Applied Leadership Development Program which you participated in. This survey will provide us with information that will help evaluate the impact of ALDP for you. No individual information will be shared with your supervisor or MLG&RD staff. All information collected will be aggregated and summarized into a final report.

Instructions:

Please complete the survey by filling in the blanks and circling the appropriate answers.

District:

What are your key management job duties/functions? Place a check mark only by the answers that apply.

Strategic Planning Staffing Monitoring & Evaluation Financial Planning/Budgeting Collaborating & Partnering

Problem Solving Marketing Grant Writing Decision Making Policy Development

Data Analysis Training Project Management Program Planning Curriculum Development

Time Management Supervising Team Building Oral Communication Written Communication

Please list any additional management duties/functions that are not listed above.

Please rank your top 5 job duties/functions in priority order.

1.____________________ 2.____________________ 3.____________________ 4.____________________ 5.____________________


22

CATEGORIES How has participation in ALDP affected your skill level to perform this task?

Decreased No change

Increased Not part of my job

DMSAC SECRETARIAT

Meeting Planning:

Prepare documents (agendas, minutes, etc.) to be distributed to DMSAC and TAC/TMT members for their review.

1 2 3 N/A Additional Comments:

Meeting Facilitation:

Facilitate effective meetings that align with prepared agendas.

1 2 3 N/A

Create schedules for committee/sub-committee members to submit data and content for reports.

1 2 3 N/A

Report committee/sub-committee data in a timely manner at DMSAC meetings.

1 2 3 N/A

PLANNING AND PRIORITY SETTING

Plan Development:

Translate national policies/strategies into district/sub-district HIV/AIDS response activities.

1 2 3 N/A


23


Decreased No change


Plan activities to ensure HIV/AIDS priorities are being met.

1 2 3 N/A

Use evidence-based data to develop culturally sensitive interventions.

1 2 3 N/A

Develop a comprehensive plan with key partners from sector and civil society organizations and across units.

1 2 3 N/A

Make detailed program, unit, and/or district/sub-district plans that include task assignments, scheduling, funding, and resource allocation.

1 2 3 N/A

Prioritization:

Define activities and tasks in order to prioritize program activities and workload for staff.

1 2 3 N/A

MONITORING AND EVALUATION

Data Collection:

Set clear, well-defined desired outcomes for work activities.

1 2 3 N/A


24


Decreased No change


Coordinate with Monitoring and Evaluation Officers to design data collection tools, surveillance systems, and materials.

1 2 3 N/A

Monitor:

Ensure sectors/organizations implement activities that align with pre-identified priority areas and the national goals.

1 2 3 N/A

Manage response activities by tracking progress in a database.

1 2 3 N/A

Evaluate:

Analyze epidemiologic service and program data to measure effectiveness of district/sub-district programs/activities to determine the impact on HIV/AIDS.

1 2 3 N/A

Make recommendations based on evidence and data. 1 2 3 N/A

Present results and lessons learned to stakeholders. 1 2 3 N/A

HEALTH PROMOTION AND DISEASE PREVENTION

Information Dissemination:


25


Decreased No change


Translate evidence-based information for public health decision makers and partners.

1 2 3 N/A

Make oral presentations to community groups, health professionals, and/or political groups.

1 2 3 N/A

Provide interviews or written articles to the media (TV, radio, and/or newspaper).

1 2 3 N/A

Disease Prevention:

Engage community leaders to help advocate for the availability, accessibility and utilization of HIV testing and other national programs in the community.

1 2 3 N/A

Coordinate HIV/AIDS response activities in district/sub-district.

1 2 3 N/A

CAPACITY BUILDING TRAINING

Needs Assessment:

Conduct training needs assessments to identify the skills gaps of DMSACs, NGOs, CBOs, and capacity building members.

1 2 3 N/A

Analyze available resources to determine the needed resources.

1 2 3 N/A

Curriculum Development:


26


Decreased No change


Apply adult learning theory to develop curricula, course materials, and lesson plans.

1 2 3 N/A

Implementation:

Implement a training program based on gaps identified in the needs assessment to build the capacity of the DMSAC and VMSAC/VWSAC members and other stakeholders in the community.

1 2 3 N/A

PARTNER COLLABORATION

Partnership Development:

Establish strategies for identifying and/or involving community organizations participants regarding identified needs and interest.

1 2 3 N/A

Develop partnerships with key stakeholders (colleagues, public-and private-sector representatives, and/or government officials) to achieve objectives in a synergistic fashion.

1 2 3 N/A

Coordination:

Develop operational procedures on facilitating DMSAC and sub-committee meetings and reporting for partners.

1 2 3 N/A


27


Decreased No change


Coordinate and guide stakeholders and/or district partners in compiling information for the Evidence Based Planning (EBP) district profile.

1 2 3 N/A

RESOURCE MOBILIZATION

Mobilize resources (including funding) for the district/sub-district response.

1 2 3 N/A

Mobilize communities for the formation of Village Multi-Sectoral AIDS Committees (VMSAC).

1 2 3 N/A

Facilitate the formation of VMSACs/WMSACs at the community level for the enhancement of HIV and AIDS response activities.

1 2 3 N/A

HUMAN RESOURCE MANAGEMENT

Employee Management:

Assess current and future staffing needs based on the program’s goals and budget realities.

1 2 3 N/A

Ensure roles and responsibilities are clearly defined. 1 2 3 N/A

Ensure reporting lines are clearly outlined. 1 2 3 N/A

Coordinate with staff to establish objectives and work plans, based on program goals.

1 2 3 N/A

Performance Evaluation:


28


Decreased No change


Monitor and appraise performance of staff. 1 2 3 N/A

Team Building:

Work with team members to establish and implement useful guidelines for team interactions.

1 2 3 N/A

Conflict Resolution:

Facilitate effective resolution of conflicts within the unit, community, and/or regulated parties.

1 2 3 N/A

LOGISTICS AND PROCUREMENT

Financial Management:

Manage district/sub-district budgets. 1 2 3 N/A

Advise management of budget problems identified. 1 2 3 N/A

Logistics:

Manage contracting of equipment, facilities, supplies, and services.

1 2 3 N/A


29

Appendix G: District Data Review

District: _____________________________________________________________________

Reviewer’s Name: ____________________________________________________________

Information source (circle all that apply): DAC Office MLG&RD Office

1. DMSAC Meetings (based on meeting minute review) a. Before ALDP (2012)

i. No. of meetings _____________ ii. No. of participants (by organization type) _________________

iii. List of agenda topics discussed ________________ iv. Follow-up on activities based on meeting minutes ____________

b. After ALDP i. No. of meetings _____________

ii. No. of participants (by organization type) _________________ iii. List of agenda topics discussed ________________ iv. Follow-up on activities based on meeting minutes ___________

2. District Reporting to National (BHRIMS reports)

a. Before ALDP (2012) i. No. of facilities included in the report__________

ii. Completeness of reporting by disease areas ____________ b. After ALDP

i. No. of facilities included in the report__________ ii. Completeness of reporting by disease areas ____________

Final Report for the Program Evaluation of the Applied ... · CDC-SMDP CDC-Sustainable Management...

Documents

Transcript of Final Report for the Program Evaluation of the Applied ... · CDC-SMDP CDC-Sustainable Management...