TM

PEN: CDC Guidance for Developing

Plans and Reports2011 TB ETN/PEN Conference

Awal Khan, PhD Lead, Program Evaluation Team

Field Services and Evaluation Branch (FSEB)

Division of Tuberculosis Elimination (DTBE)OID/NCHHSTP

TM

This talk – a quick overview:

Program evaluation (PE) activities

DTBE guidance on PE DTBE guidance on cohort review Developing PE plan & reporting

results Current and on-going PE

initiatives Questions

TM

Program Evaluation (PE) Activities

TM

Program Evaluation (PE) Plans Submitted

- 2010 Annual Report

TM

PE Reports Following Application Guidance

- 2010 Annual Report

TM

Program Using NTIP to Select Evaluation Focus Area, 2010

TM

Focus Area of Program Evaluation

Focus Area Interim

Annual

Contact Investigation 24 22

Known HIV Status 13 10

Sputum Culture Conversion 8 8

Completion of Treatment 6 6

Evaluation of Immigrants and Refugees

4 7

TB Case Rate 4 2

Sputum Culture Reported 3 1

Recommended Initial Therapy 2 0

Universal Genotyping 2 1

Program Evaluation/Evaluation Focal Point

1 1

TM

CDC/DTBE Guidance on Program Evaluation

TM

CDC Funding Opportunity Announcement (FOA) Requirements,

2012• Monitor their progress toward meeting

national TB objectives and performance target using NTIP reports and other data sources

• When applicable, use NTIP to report their performance on national objectives in the annual and interim progress reports for the TB cooperative agreement

• Where appropriate, grantees should continue to report and describe progress toward project-specific objectives and activities using their own data and/or NTIP.

TM

Guidance for Program Evaluation

- CDC 2012 FOA • A systematic and in-depth study

of priority program areas to enhance a program’s ability to meet its objectives and performance targets

• The focus of evaluation should be on where the program needs improvement

• Collect information that can be used to inform, enhance, and facilitate programmatic decision making process.

TM

Guidance for Program Evaluation

• Programs should monitor their progress toward meeting national TB objectives and performance target using NTIP reports and other data sources

• Identify a focus area and provide a justification

• Clearly defined objectives, purposes, and methods of the evaluation

• Uses and dissemination of findings including strategy and plan to work with local programs.

TM

Program Evaluation Focal Points

• Serve as the point of contact in their respective jurisdiction for evaluation activities

• Build program evaluation capacity

• Work closely with TB program staff

• Engage in TB PEN activities

• Share PE experiences and lessons learned with stakeholders and other programs.

TM

CDC/DTBE Guidance on Cohort Review

TM

Programs Implementing Cohort Review

TM

Guidance for Conducting Cohort Review

• Programs will be required to conduct at least one cohort review per year at the state or local levels or both

• Programs should continue ongoing case reviews as usual

• Programs are recommended to conduct cohort reviews• Semi-annually, if 0-50 cases

reported • Quarterly, if over 50 cases

reported

TM

Developing PE Plan & Reporting Results

TM

Format for Developing Evaluation Plan

A. Purpose of the evaluation:• 3 purposes/ goals including

focus area

B.Objectives: • What do you want to do?

should be “SMART”: Specific, Measurable, Achievable, Realistic, and Time-bound

C. Activities: • An objective may have multiple

activities

TM

Format for Developing Evaluation Plan – Contd.

D. Measures of activities: • Specific indicators for each

activity• Criteria for success/failure in

each activity

E. Expected outcomes / impact

F. Potential challenges and opportunities

G. Use of results.

TM

Program Evaluation Report

I. Introduction: Describe Evaluation Objectives• Describe the purpose of your

evaluation, rationale for selecting evaluation focus area, and clearly define the evaluation objectives

II. Describe data collection and

analyses • how the evaluation was

conducted (activities and timeline)

• data sources, data collected, and methods

TM

Program Evaluation Report – Contd.

III. Conclusion and Discussion of results. • Provide results of the objectives. • If not achieved, why not?  • What were the barriers to and

facilitators?

IV. Recommendations to Program Administrators for Future Program Focus and Evaluation Efforts• Describe the best and/or

evidence-based practices including identifying the next focus area.

TM

PET’s Feedback Process for Interim/Annual Reports

• PET provides feedback reports only for program evaluation • Timeline – usually May and

October

• Feedback is reviewed by program consultant and evaluation representative

• TB Control Officer and PE Focal Point receive the feedback

• Program are encouraged to contact the program consultant and PE representative to discuss the feedback.

TM

Feedback FormatFeedback Report

Program Evaluation Interim/Annual Progress Report– March 2011 Program Area:

PET Representative: _____________________ Program Consultant: __________________

IV. Recommendations Described lessons learned Listed program recommendations Identified next focus area

II. Data Sources, Collection & Analysis Described evaluation activities Presented evaluation timeline Described data sources Provided methodology for data Collection & analysis Described data collected

I. Introduction Described purpose of evaluation Identified a focus area for evaluation Provided rationale for selecting evaluation focus Used NTIP to select evaluation focus Defined clear evaluation objectives Identified expected outcomes (short, med, long)

III. Conclusion & Discussion of Results Evaluation results presented Identified challenges/barriers Identified opportunities for future evaluation Linked findings to National Objectives

Organization and Presentation Evaluation Report is organized as per guidance Yes No Evaluation Plan reported Yes No

Additional Evaluation Activities Implementation of Cohort Review Yes No Planned

TM

Conceptual model for Evaluation Focus Area “Implementation Gap”

TB Elimination

The “Implementation Gap”Some goals and objectives

are achieved and implementedand some or not

The “Implementation Gap”Use of new Tools or

Technology for evaluation activities

TM

Current and on-going PE initiatives

TM

Current and Ongoing PE Initiatives

• Identify PE technical assistance needs collaboratively with TB control programs, PE focal points, and PET representatives

• Program develops PE plan and discusses with designated PET representative for additional support and shares with the respective program consultant for collaboratively implementing the PE activities

• Field Operations and Program Evaluation Teams meet quarterly to share evaluation & programmatic perspectives for meeting the grantee’s needs

TM

Current and Ongoing PE Initiatives – Contd.

• Identify ways to provide technical assistance for sustaining PE efforts

• Developed template for contact investigation evaluation plan and report in response to requests from the program evaluation focal point

• Plan to develop evaluation plan and report templates for all national TB objectives.

TM

Number of Abstracts Received, 2009-2011

TM

Program Evaluation Focal Points

• Serve as the point of contact in their respective jurisdiction for evaluation activities

• Build program evaluation capacity

• Work closely with TB program staff

• Engage in TB PEN activities

• Share PE experiences and lessons learned with stakeholders and other programs.

TM

AcknowledgementsAl Forbes Victor BalabanGail Grant Judy Gibson Regina Gore Lakshmy MenonBruce Health Brandy PetersonAndy Heetderks Sylvia Trigoso (PHPS Fellow)Mark Miner Christina Dahlstrom (PHPS) Vic Tomlinson Lauren Polansky

Awal Khan Paul Tribble Greg AndrewsDawn Tuckey Dan RuggieroJoe Scavotto Terry Chorba

TB PEN Focal Points and TB Control Officers

TM

Thank you

TM

Good Program Evaluation Shifts Our Focus From

Did our efforts work?to

Are our efforts working?

TM

Additional Slides on Cohort Review

TM

What is Cohort Review ?

• Cohort review is a systematic review of patients with TB disease and their contacts

• The “Cohort” is a group of TB cases counted over a specific time and the review occurs after the cases are counted

• Case reviews are not cohort reviews.

TM

Purposes of Cohort Review

• A useful tool for ensuring accountability, informing and education program staff about TB protocol and national objectives

• Review of comprehensive TB case management procedures for all cases

• Opportunity for reviewing TB data for reliability and quality checking, and identifying deviations

• Opportunity for reviewing completeness of key data variables of RVCT and making corrections.

TM

Purposes of Cohort Review – Contd.

• Help program staff• Understand the challenges patients

face as they try to complete TB treatment

• Undertake activities required for program improvement

• Enable programs to verify data for national program objectives and understand activities most likely to impact program targets.