Can Feds KISS? Creating a Simple and Smart

Strategic Execution and Evaluation “System” Despite

Complex Mandates and Requirements

Environmental Evaluator’s Network Concurrent Sessions

June 23, 2011, 4-5 PMRoom 310

Liz Davenport and Thanh Vo DinhNational Oceanic and Atmospheric Administration

1

How a SMART planning, budgeting, execution, and evaluation system can better manage a complex portfolio

How better integration of performance monitoring, program evaluation, and assessing strategic progress can improve performance management

How Feds can KISS to ensure GPRA Modernization Act improves agency and government-wide performance

Network to Learn How Evaluation Can Aid an End-to-

End Management Process

2

NOAA manages about 12,800 employees, aided by about 4,000 contractors

NOAA has 6 Line Offices and 12 Staff Offices There are 127 mandates and authorities

applicable to NOAA’s mission NOAA’s responsibilities span all 50 states, Guam,

Puerto Rico, and the Pacific Island Region

Context for NOAA’s Strategy Execution and Evaluation (SEE)

Process

3

NOAA Strategic Plan

4

Here is SEE…under its first cycle of implementation...

What Is NOAA Creating in the Strategy Execution and Evaluation

Process?

5

6

1FocusAnnual

Guidance Memorandum

2Assess

Corporate Portfolio

3Request

Budget

4ReassessCorporate Portfolio

5Execute

Annual Operating

Plans

6EvaluateProgress

Request Budget to AchieveIntended Investment Outcomes

Evaluate Progress to Plan, Performance, Programs, and Strategic Priorities

Assess Portfolio to Plan Investments

Reassess Portfolio to Optimize Performance in

Operational Plans

Execute Operating Plans to Achieve Priority

Outcomes

AlignStrategic Plan

Implementation Plans

Strategy Execution & Evaluation Framework

Focus Attention to Achieve Priority Outcomes

From Compliance to Assimilation…Growing Utility of Evaluation

Compliance with performance evaluation requirements

Compliance and some performance data used to improve management

Compliance and sporadic use of requirements to “message” and “manage” performance to improve outcomes

Managers consistently use program evaluation requirements to define long-term outcomes, improve short-term success and achieve objectives

Agency success is strongly aided by assimilated program evaluation and a performance management framework that makes optimal use of performance data for management

7

Program Evaluation Requirement Driver

8

GPRA Modernization Act Authorizes Implementation to:

Foster coordinated, cross-cutting, government-wide planning ◦ Government-wide strategic goals with annual government-

wide (Federal) performance plan Engage Congress to identify management/performance

concerns◦ Agency strategic plans include Federal performance goals and

timed for release with changes in Administration Improve accountability for results and address

management weaknesses ◦ Agency and separate Federal performance plans aligned;

looks at past 5 years of performance data◦ Performance information better utilized due to greater

Administration and Congressional engagement ◦ Quarterly reviews reported at www.performance.gov

Adopt Standard, Federal Terms and Definitions Ensure Agency Clarity Regarding:

◦ Program◦ Performance Measurement◦ Performance Measures◦ Program Evaluation

Comply with New Standard:◦ Government-wide/cross-agency performance plan, with agreed

upon outcomes and accountability within a 2-year time frame

Performance Management and Evaluation “Unit” and “Paradigm”

9

Stop and Think!

10

How can we define “program” for agency performance planning and reporting and for Federal performance planning and reporting, given the need to link to and show progress in achieving Federal goals?

How can we do evaluations and use them to provide feedback to our Agency for these new requirements?

GAO Defined Types of Federal Program EvaluationsProcess (or Implementation) Evaluation – assesses extent to which a

program is operating as intended…assesses program activities’ conformance to statutory and regulatory requirements, program design, and professional standards or customer expectations

Outcome Evaluation – assesses extent to which a program achieves its outcome-oriented objectives…focuses on outputs and outcomes (including unintended effects) to judge program effectiveness but may also assess program process to understand how outcomes are produced

Impact Evaluation – (type of outcome evaluation) assesses net effect of a program by comparing program outcomes with an estimate of what would have happened in absence of the program…employed when external factors are known to influence program’s outcomes…to isolate program’s contribution to achievement of objectives

Cost-Benefit and Cost-Effectiveness Analyses – compare program’s outputs or outcomes with costs (resources expended) to produce them…when applied to existing programs, considered a program evaluation…cost-effectiveness analysis assesses the cost of meeting a single goal or objective and can be used to identify the least costly alternative for meeting that goal…aims to identify all relevant costs and benefits, usually expressed in dollar terms Performance Measurement and Evaluation, Definitions and Relationships, GAO 11-

616SP

Program Evaluation Typology

11

Can these program evaluations and performance measurements serve the new requirements at the same time serving Agency needs for management to meet strategic priorities? (National Ocean Policy…how will it be evaluated?)

Do we need anything else? Or what could be potential tools, approaches, or new paradigms for evaluation?

Stop and Think!

12

Theory of changes: Logic model for government-wide goals from collective actions

Performance accountability: www.performance.gov

Conflict resolution and coordination of capabilities between partners

Strategic evaluation in concert with performance measurement and program evaluation

Current Available Tools?

13

Thanks for letting us share SEE and speculate about GPRAMA impacts on evaluation

Thoughts, experiences , best practices on… How can we define program for agency performance planning and

reporting and for Federal performance planning and reporting given the need to link to and show progress in achieving Federal goals?

How can we do evaluations and use them to provide feedback to our Agency for these new requirements?

Can these program evaluations and performance measurements serve the new requirements at the same time serving Agency needs for management to meet strategic priorities? (National Ocean Policy…how will it be evaluated?)

Do we need anything else? Or what could be potential tools, approaches, or new paradigms for evaluation?

Discussion Questions

14

Backup Slides

15

What are your thoughts, experiences, best practices on…

1. GPRAMA requirements on your Agency’s evaluation and performance management?

2. Communicating evaluation results, using evaluation to facilitate, increase relationships/interaction with Congress and stakeholders?

3. Implementing GPRAMA requirements on evaluation within the current budget environment?

4. How are you building evaluation capacity? (Linking evaluation with performance management and evaluating success/failure of strategic goals/plans)?

Other Discussion Questions

16

• Effective compliance with evaluation requirements• Learning culture through evaluation feedback• Robust program management• Better investments through adaptive management• Valuable decision-making support tool• Transparent performance, outcomes, and results• Monitor progress toward achieving strategic goals • SMART performance targets• Improved accountability

Program Evaluation Framework Benefits

17

GPRAMA Agency Strategic Plan Changes That May Impact Evaluations

Former Process1. Revised at least every 3 years2. Covered at least a 5-year period3. Consulted with Congress4. Involved other stakeholders5. Submitted to OMB and Congress

Amended Process1. Revision every 4 years, approximately 1

year after a new Presidential term begins

2. Consultations with relevant Congressional committees at least every 2 years

3. Publication online and President and Congress notified

Amended Content4. Relationship to Federal government

priority goals5. Interagency coordination and

collaboration6. Identification of agency priority goals7. Description of incorporation of

Congressional input

18

Former Content 1. Mission statement 2. Strategic goals3. Strategies and resources4. Relationship to performance

goals5. External factors that could

significantly affect the goals6. Program evaluations

GPRAMA Agency Performance Plan Changes That May Impact Evaluations

Former Process1. Upcoming fiscal year2. No set time frame by OMB3. Submitted to OMB

Amended Process1. Covers 2 fiscal years

(current/upcoming)2. Concurrent with President’s Budget3. Available online; President and

Congress notified

19

Former Content1. Performance goals in objective,

quantifiable, measurable form2. Cover all program activities3. Strategies and resources4. Performance measures5. Basis for comparing actual

results with performance goals6. Means to verify/validate data

Amended Content1. Relationship to agency strategic and

priority goals, and Federal government performance goals

2. Coordination and collaboration3. Contributing programs and activities4. Milestones5. Goal leaders6. Balanced set of measures7. Data accuracy and reliability8. Major management challenges9. Low-priority programs

GPRAMA Agency Performance Plan Changes That May Impact Evaluations

GPRA Modernization Act…Agency Quarterly Priority Progress Reviews◦ At least quarterly, agency head, COO, and PIC

review with goal leaders progress toward each priority goal

◦ Review involves contributors in and out of Agency: Assess how programs and activities contribute to goal Categorize goals by risk of not being achieved For those at risk, identify strategies to improve

performance

20

GPRAMA Agency Performance Plan Changes That May Impact Evaluations

GPRA Modernization Act…Requires High Performance Goals with Expanded Performance Reporting◦ Set quarterly progress reviews◦ Require Congressional consultation and

engagement in performance management◦ Require a Performance Portal◦ Reduce duplicative performance reporting by

10%

21

GPRAMA Agency Performance Plan Changes That May Impact Evaluations GPRA Modernization Act Senate

Committee Suggests:◦Agency, OMB, and Relevant Congressional

Committee Review of Agency Meeting Goals

◦ If Goals Are Not Met: After 1st FY, require a Goal Performance Improvement

Plan If not met after 2nd FY, submit actions to improve

performance (statutory changes, resource transfer, etc.)

If not met after 3rd consecutive FY, must act within 60 days or face statutory changes or possible termination or reduction possible

22

GPRAMA Agency Performance Plan Changes That May Impact Evaluations

23

1. Strategic Plan must include how program evaluations were used to establish or revise strategic goals and a schedule for future evaluations

2. Performance reports must include summary findings of program evaluations completed during the fiscal year covered by the report

3. Evaluation will benefit performance planning and goal setting as well as quarterly reviews of priority goals