Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the...

Evaluation in the Government of Canada

Robert Lahey23 November 2004

PART ONE: An Overview of the Evaluation System in Canada

REL Solutions Inc. Brasilia, November 23, 2004 2

1. Background on the Canadian System2. Historical Perspective on how Evaluation

has evolved3. Government Accountability Structure4. Global View on how Evaluation Supports

Government’s Vision

OutlinePART ONE: An Overview of the Evaluation System in Canada


Population 31 million

Sovereign parliamentary democracy

Federation of 10 provinces and 3 territories

A federal Parliament and 13 provincial / territorial legislatures

Areas of federal responsibility include: national defence, criminal law, postal service, census, copyrights, trade, external relations, finance, transportation, citizenship, Indian affairs, fisheries and agriculture

Provincial / territorial governments: authority for education, hospitals, property and local government

Federal Government

Over 100 departments and agencies

Each has a Minister who reports to Parliament

Budgets approved annually

Formal Evaluation Policy in Canada for more than 30 years

Background on Canadian System of Government


Growing demand for information to support increasingly complex and costly programs

New management systems created for financial administration and planning, programming and budgeting

Evaluation as a practice not yet formalized

Historical Perspective: The 1960s


Building of key infrastructure elements in departments

Treasury Board Evaluation Policy (1977) was the first formalized evaluation policy in Canada

Evaluation Policy centre created within the new Office of the Comptroller General (1978)

• Considerable focus on building evaluation capacity in federal departments



High expectations for evaluations not always met Government-wide review led by Parliament



Evaluation capacity affected by general government downsizing

AG report (1993) indicated renewal of evaluation capacity needed

New Review Policy (1994) linked evaluation closer to internal audit

New emphasis on performance measurement / monitoring: Departmental Performance Reports to Parliament



‘Capacity gap’ for Evaluation (and Internal Audit) identified – need to reinvest in evaluation capacity within departments – both $ and skilled evaluators

New Evaluation Policy (2001): Commitment to link Evaluation to broader accountability and reporting requirements (strategic) as well as management decision-making (focused)

Emphasis on Results based Management Measuring Results Results based Management and Accountability Framework (RMAF)

Re-establishment of Policy Centre: Oversight of Evaluation practice in departments capacity building role; quality control role; a champion

Move in direction of ‘participatory’ evaluation Objectivity, not independence, emphasized Greater link to management and performance measurement / monitoring



Need for formal evaluation increases as resources become scarcer and the identification of priorities becomes more important

Necessary, but not sufficient, conditions for an effective evaluation system are: government investment and support; formalized policies and standards; and, leadership for capacity building

Evaluation quality depends on an approach that balances: timeliness, usefulness, methodological purity, client requirements and cost

Critical mass in capacity is required to ensure evaluation remains credible, relevant and strategic

Performance measurement / monitoring will typically not address ongoing issues of ‘cause and effect’; for that, ad-hoc evaluation is generally required

The priority given Evaluation is greatly enhanced if there is a perspective on modern management shared by the political level and the Public Service and evaluation is seen as integral to that vision

Historical Perspective: Lessons Learned


Canadian Government Accountability Structure

Parliament of CanadaHouse of Commons

Prime MinisterPrime Minister

Cabinet (Privy Council)Cabinet (Privy Council)

Parliamentary Committees Parliamentary Committees

TB Minister

Treasury Board Secretariat (TBS)(Secretary appointed)

Government’s management board (financial, management and human resources) Comptrollership function Centre of Excellence for Evaluation Policies and standards Capacity building Links evaluation and performance measurement

Public Servants

All Other Ministers

Federal Depts. /Agencies(DMs appointed)

Deputy Ministers (DMs) accountable for the application of Evaluation Policy

within their departments Heads of Evaluation implement policy as per TBS standards and guidelines Internal accountability and reporting to DMs External accountability and reporting to TBS and Parliament

Public Servants

Auditor General (AG)(appointed)

Conducts independent audits of government operations Produces periodic oversight reports on the conduct of evaluation Promotes accountability and best practices Reports directly to Parliament


Global Views on Public Sector Management in Canada

Vision: “Results for Canadians” – Improve government services and respond

to demands for better value and transparency

Tools and Initiatives: Applied through a wide-reaching series of initiatives Commitment to link evaluation and performance measurement /

monitoring to broader accountability and reporting requirements

Incentives and Use: Perspective on modern management shared by the political level and

the Public Service


Generate and use reliable information on results for the purposes of: Sound management, resource allocation and

decision making Accountability to Parliament and Canadians Continuous process of cultural change with no

static end state

Global Views: Goals to Achieve


4 Government Commitments

Roles & Responsibilities

Agenda for Change

ResultsResultsResponsible Responsible

SpendingSpendingValuesValues

Citizen Citizen FocusFocus

The Results for Canadians Management Framework


Learn & Adjust

StrategicAnalysis

Plan forResults

ImplementMonitor, Measure, Evaluate

Report onResults

Analysis of the current environment, past performance, emerging priorities and significant risks to achievement of desired results

Developing Strategic and Operational Plans – identify and plan to achieve key results and mitigate risk

Action taken to achieve results

Ongoing performance measurement and periodic evaluation to determine progress and allow for corrective action

Providing integrated Financial and non-financial information on: results and accountability for internal and external use

Evaluation: A Critical Link to Results-Based Management


Gain broad support for the “Vision”

Create the necessary ‘enablers’ (policies, guidelines, standards)

Invest in capacity building across the system Human capital

Tools necessary to support implementation

Provide the necessary incentives for implementation

Monitor and evaluate implementation

Learn and adjust, as necessary

Approach to Implementing the RFC Vision


Tools and Initiatives to Support the Vision

Modern Comptrollership

Broad direction for management reforms

Strengthened Evaluation and Internal Audit functions

Investment in capacity building

Functional specialists in departments (Evaluators; Auditors)

Learning for managers

Results-based Management and Accountability Frameworks (RMAFs)

Improved Performance Reporting to Parliament

Departmental Performance Reports

Canada’s Performance

Monitoring implementation

Modern Comptrollership “Capacity Check”

Managing for Results Self-Assessment Tool

Evaluation and Active Monitoring of Policy implementation

New government-wide initiatives

Expenditure and Management Review

Management Accountability Framework

Evaluation in the Government of Canada

Robert Lahey23 November 2004

PART TWO: A Detailed Examination of the Process and Conduct of

Evaluation in Federal Departments


1. Module 1: Purpose of Evaluation2. Module 2: Developing the Capacity to

‘Do’ Evaluation3. Module 3: Developing Capacity to ‘Use’

Evaluation4. Module 4: Evaluation vs. Performance

Measurement/Monitoring5. Module 5: Results based Management

and Accountability Frameworks

OutlinePART TWO: A Detailed Examination of the Process and Conduct of

Evaluation in Federal Departments


Outline of Module Definitions and Terminology Key Uses and Users of Evaluation A Logic Model for the Use of Evaluation in a

Federal Department• Performance Measures for the Evaluation

Function

MODULE ONE: Purpose of Evaluation.. Why is it important? And, to whom?


Evaluation Summative Evaluation Formative Evaluation

Performance Measurement

Performance Monitoring

Results

Results Based management

DEFINITIONS AND TERMINOLOGY


How and where do results of Evaluation get used?

Within federal departments—several users

Deputy Ministers Assistant Deputy Ministers (ADMs) Program Managers Input to Corporate functions (policy, planning)

Treasury Bard of Canada Secretariat (TBS)

Funding renewal for individual programs Expenditure and Management Reviews Management Accountability Frameworks (MAF)

Elected Officials

Departmental Performance Reports Canada’s Performance Parliamentary Committees

General Public

Access to Evaluation Reports (ATIP) Web-based increases transparency

KEY USES AND USERS OF EVALUATION


Defined four broad ways Evaluation supports senior management

Four activity areas where skill set of Evaluators and Evaluation information is used

Traditional role (conducting program and policy evaluation) represents major use of Evaluation resources (some 75%)

A LOGIC MODEL FOR THE USE OF EVALUATION

IN A FEDERAL DEPARTMENT


Informed TBS decision-making

on program funding and

renewal

Program & PolicyEvaluation

LOGIC MODEL for HRSDC EVALUATION FUNCTION

ACTIVITIES

OUTPUTS

IMMEDIATE OUTCOMES

INTERMEDIATE OUTCOMES

ULTIMATE OUTCOMES

PerformanceMeasurement

Planning & Priorities

Evaluation Reportson program/policy• relevance• success • alternative

• Briefing material• Special studies• Corporate

intelligence

• RMAF advice and sign-off

• Tools & advice on performance measurement

• Evaluation plans and priorities linked to corporate priorities

• Objective and timely evidence-based information

• Lessons learned• Action plans

• Implementation of improved performance measurement system

• OperationalizedResults-Based Measurement (RBM)

• Improved measurement and reporting of results

• Better data collection

• Integrated evaluation and corporate plans

• Information for corporate priority-setting

Better design and resourcing of programs and

policies

Improved program

management and accountability

Results-based departmental

decision-making

RBM culture in HRSD

Parliamentary reporting

• Strengthened governance & stewardship of public funds• Better accountability to Parliament

• Better results for Canadians• Effective use of resources

Support & Advice toCorporate InitiativesACTIVITIES

OUTPUTS

IMMEDIATEOUTCOMES

INTERMEDIATEOUTCOMES

ULTIMATEOUTCOMES

A LOGIC MODEL FOR THE USE OF EVALUATION

IN A FEDERAL DEPARTMENT


How well are we doing in terms of expectations?

Derive measures from Logic Model

Caution how information is used, particularly during capacity building period

PERFORMANCE MEASURES FOR THE EVALATION FUNCTION


Outline of Module Policies and Standards Champion for the Function Investment in Evaluation Capacity Recruitment, training and development of

Evaluators Vehicles to Train and Develop Educating the ‘Users’ of Evaluation Organizing an Evaluation Unit in a Federal

Department

MODULE TWO: Developing the Capacity ‘To Do’ Evaluation. A Key Role for the Central Agency


Identifies expectations, roles and responsibilities

Federal Departments given flexibility in their application

Deputy Minister (DM) accountable

Treasury Board of Canada Secretariat (TBS) oversight of individual departments…through the Centre of Excellence for Evaluation (CEE)

Periodic review by the Auditor General of Canada (AG) of how/how well Evaluation Policy is being implemented …provides system-wide oversight

POLICIES AND STANDARDS


Important to have a champion/advocate for the function

Several roles:

Setting and adjusting policy and standards for the function Developing and executing human resource (HR) strategies to train and develop Evaluators Advisory role (practical advice) to individual departments Oversight and monitoring quality of evaluation practices and products Corporate resource on Evaluation and accountability

for government officials

Establish a network for Evaluators (continuous learning)

CHAMPION FOR THE FUNCTION: THE CENTRE OF EXCELLENCE FOR EVALUATION (CEE)


Need for both $ and trained personnel

Policy not prescriptive on size of departmental Evaluation units…a decision of the DM

Accountabilities of the DM (several) serve to leverage resources for the function

In recent years, focus of TBS/CEE on building a ‘sustainable’ Evaluation function Infrastructure Product Use

With flexibility in resourcing, important that DMs understand importance and use of Evaluation

Canadian experience....periods of general downsizing (fewer resources for the function) and capacity building (new investment)

Danger of fluctuating resource levels….losing skilled/experienced Evaluators. A long-term investment to replace.

INVESTMENT IN BUILDING EVALUATION CAPACITY


Recruitment University graduates generally not ‘Evaluation-specific’ Entry into the function from many disciplines…social sciences

(Economics, Psychology, Sociology); Education; Mathematics/Statistics; Public Administration Both a strength and a weakness

Training Need to distinguish between training for ‘entry level’, ‘mid level’

and ‘senior’ Evaluators Link to the competencies required for Evaluators (Competency Profile) Various vehicles for formal training

Development Importance of ‘on the job’ training Continuous learning…link to changing/evolving role of Evaluation Communication skills particularly important for senior level Evaluators (Departmental Heads of Evaluation) Several vehicles and occasions for development No formal ‘certification’ to work as an ‘Evaluator’ in federal government

RECRUITMENT, TRAINING AND DEVELOPMENT OF EVALUATORS


Formal curriculum developed by CEE for ‘entry level’ Evaluators (Evaluator Internship Program)…not mandatory

Training needs identified for mid-level Evaluator

CEE partnering with others for training delivery…e.g. National Statistical agency (Statistics Canada); Canadian Centre for Management Development; Professional Associations (Canadian Evaluation Society (CES); department-led training; private sector trainers/course deliverers

Network of Evaluators encourages informal ‘communities of practice’ and sharing of ‘lessons learned’ across like evaluations

Link to other professional groups (e.g. Internal Auditors; Comprehensive Audit)

VEHICLES TO TRAIN AND DEVELOP EVALUATORS


‘Users’ of evaluation (in departments, central agencies, politicians)

Need to be educated on how, where, and why evaluation can/should be used/resourced

Importance of aligning ‘user’ expectations with reality of what can/will be delivered by Evaluators

Useful to use a variety of vehicles: briefing notes; presentation decks; training modules/workshops aimed at senior managers

Avoid technical jargon and deliver at appropriate level of understanding

Education an ongoing activity; not a ‘one-time’ event

Basic distinctions need to be made…e.g. difference between Internal Audit and Evaluation, an ongoing area of confusion for managers: Internal Audit: Are we doing things right? (efficiency issues) Evaluation: Are we doing the right things? (effectiveness and

performance issues)

EDUCATING THE ‘USERS’ OF EVALUATION


In Canada, federal departments determine their own arrangements

Much variation in terms of number of full-time staff hired vs use of consultants

Generally, most departments use some consultants…e.g. for surveys, specialized tasks, etc.

Important for Unit to have minimum core of full-time Evaluators

Wide range in size of Units dedicated to Evaluation…e.g. 1 person (small agency) to 25-30 f/t staff (largest department)

Typical Evaluation Unit: 10 or fewer f/t staff, plus resources to hire consultants for Evaluation

ORGANIZING AN EVALUATION UNIT


Outline of Module Central Agency-driven demand for Evaluation Steering the Evaluation function in departments Credibility building for Evaluators Developing a Comprehensive Evaluation Plan Advisory Committees Educating ‘Users’ of Evaluation Participatory Evaluation and Accountability for

Follow-up Evaluations in ‘full public view’

MODULE THREE: Developing Capacity ‘To Use’ Evaluation: Overcoming Resistance and Making Effective Use of Evaluation


CENTRAL AGENCY-DRIVEN DEMAND FOR EVALUATION

Deputy Minister in department faces variety of accountabilities requiring performance/results information

Required for good management practices and internal and external reporting

Program-specific funding renewal decisions Horizontal and vertical expenditure reviews Accountability ‘report card’ of the DM (Management

Accountability Framework (MAF) Departmental Performance Report to Parliament Ability to respond to queries of Parliamentary

Committees; issues raised by the Auditor General


Steering the Evaluation Function in Departments

Evaluation and Audit Committee, chaired by Deputy Minister Requirement of TBS for every government department Senior level committee; formal Terms of Reference; meet

minimum four times a year Review and approve annual and multi-year plan for Evaluation Senior managers identify new/changing priorities Table completed Evaluation reports and management

response Forum to alert senior managers of areas of risk to department Vehicle to inform/educate senior managers (senior ‘users’ of

evaluation


Credibility Building for Evaluators

How Evaluators see themselves. How others see Evaluation.

Two critical factors for credibility: Understanding the needs/requirements of senior

decision makers Delivering quality product on time ‘Answering the right questions at the right time’ Critical factors for success: Head of Evaluation well linked to senior management Appropriate balance between methodological rigour

and delivery in a timely fashion Responsive to changing priorities


Developing a Comprehensive Evaluation Plan

Three key questions:

i. What are the guiding principles for evaluation planning?

ii. What is the relevant ‘Evaluation Universe’?iii. How best to determine priorities and schedule

evaluations over a multi-year period?


Advisory Committees: Building trust and avoiding surprises

Useful vehicle to assist conduct of an Evaluation study

Representation from broad set of stakeholders Forum to clarify study objectives; table early

findings; review conclusions and recommendations

Creates more ‘open’ evaluation process Avoid surprises at the end Helps ensure follow-through on study

recommendations


‘Users’ of evaluation (in departments, central agencies, politicians)

Need to be educated on how, where, and why evaluation can/should be used/resourced

Importance of aligning ‘user’ expectations with reality of what can/will be delivered by Evaluators

Avoid technical jargon and deliver at appropriate level

of understanding Education an ongoing activity; not a ‘one-time’ event

EDUCATING THE ‘USERS’ OF EVALUATION


Conduct of evaluation not ‘independent’ of program managers New Policy (2001) removed ‘independence’; stressed ‘objectivity’ and ‘transparency’ Attempt to bring evaluation closer to managers…not a threat, but a tool to assist good management practices Puts onus on Managers for ongoing performance

monitoring as well as accountability for implementing evaluation recommendations

Participatory Evaluation and Accountability for Follow-up


Broad demand for greater accountability in government

Evaluation reports accessible to wide audience (via ATIP)

Web posting increases accessibility to general public

Evaluation Policy stresses ‘transparency’

Requires a culture allowing for: full disclosure of information unbiased analysis and reporting openness in government

Evaluation ‘In Full Public View’


Outline of Module Historical Context Definitions and Distinctions Departmental Performance Framework Using Logic Models to Derive Performance

Measures Results Measurement Continuum Model Roles and Responsibilities for Delivery General Strategy for Implementing M&E Systems

MODULE FOUR: Evaluation versus Performance Measurement/Monitoring: A Critical Distinction


Mid-to-late 90’s onward: Increasing government emphasis on ‘performance measurement’,

‘monitoring’ and ‘results’ Drivers: Results based management; Greater

accountability to Parliament and to citizens Put onus on managers to measure and report on

performance/results Required clarification of terminology, roles and

responsibilities in the process

Historical Context


How is Evaluation different from Performance Monitoring? How does one link to the other? Are they both necessary tools to support good

management practices and accountability?

Definitions and Distinctions


Planning Reporting and Accountability Structure (PRAS) provides broad departmental framework

Policy being updated (Fall 2004) to Management Resources and Results Structure (MRRS)

Business line structure: architecture and basis for cascading set of performance measures through organization

Brings greater focus to managers on measuring ‘results’

Evaluation specialist has the skill set to facilitate this process

Departmental Performance Framework: Basis for Ongoing Performance Monitoring


Performance indicators derive from logic model

What needs to be measured and reported?

What are the key questions that need to be answered?

How will it be used? By whom? How often? ‘Monitoring’ versus ad hoc ‘Evaluation’ Performance measurement development/implementation

Technical cost considerations An iterative process

Using Logic Models to Derive Meaningful Performance Measurement


Results Measurement Continuum Model


Managers ultimately accountable for performance measurement and reporting

Evaluation specialists support to the development and implementation of the performance measurement system through: creating PM understanding---learning events, seminars,

information packages facilitating PM development applying needed rigour and discipline---‘challenge’ guidelines Evaluation study as one option for measuring

performance, where: ad hoc study more cost-effective ‘cause and effect’ (attribution) issue important IM/IT specialist---advice re systems and data capture issues

Roles and Responsibilities for Delivery


Challenges: political; organizational; technical; resourcing

Useful to do a ‘readiness assessment’: Is there an enabling environment: political will; commitment; a champion Technical capacity, including HR skills Necessary infrastructure to produce, collect, analyze and report relevant information Necessary infrastructure/institutions to ‘use’ evaluation and monitoring information Priority needs to be established from the top Resourcing: a corporate responsibility. Needs to be linked to accountability and modern management practices

General Strategy for Implementing Evaluation and Performance Monitoring Systems


Outline of Module Why have we introduced the RMAF? RMAF Components Using the RMAF to Greatest Gain Helpful Hints for RMAF

Development/Implementation

MODULE FIVE: Results Based Management and Accountability Framework (RMAF): An Important Vehicle


Tool designed to assist the move to results measurement/reporting Broader government commitment on results reporting (Results for Canadians) Formal requirement for certain programs Facilitates development to performance measurement and establishes priority and timeframe for evaluation

Why have we introduced the RMAF?


All RMAFs have five major sections: A: Profile B: Logic Model C: Ongoing Performance Measurement Strategy D: Evaluation Strategy E: Reporting Strategy Together, describes the rationale, resources and

performance logic of program or initiative Presents the measurement, evaluation and reporting strategies that will be implemented to track progress

RMAF Components


A collaborative effort of key program stakeholders to clearly articulate:

Program objectives in terms that allow for measuring/monitoring performance/success Linkages of program activities to program goals Brings technical (Evaluator) and non-technical (Manager) to

common agreement on meaningful measures of performance and a measurement strategy Up-front investment in RMAF pays off in Clarifying program role and resourcing needs Establishment of meaningful measurement systems More systematic and disciplined monitoring of program performance Improved management information, supporting RBM Improved accountability, both to internal and external stakeholders

(senior managers; elected officials; citizens)

Using the RMAF to Greatest Gain


Accountability with RMAF development; performance monitoring; reporting…lies with Manager Carry out development and implementation as

collaborative effort (Manger; Evaluation specialist; IM/IT specialist) Flexibility, particularly in early stages:

‘acceptable’ RMAF versus perfection

performance measurement as an ‘iterative’ process Select realistic performance indicators based on what will provide the best information, not the easiest to implement

Helpful Hints for Implementing RMAFs


Evaluation and performance measurement: Important link to effective pubic sector governance Broad application across:

individual programs sectors or business lines whole organization

Not an end in itself, a means to an end Evaluation capacity building to a self-sustaining state requires:

Commitment/political will Resources (both $ and HR) Time Energy

Final Thoughts

Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the...

Documents

Transcript of Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the...