1

International Conference

Evaluation: Evidence-based Tools for Decision-making

Future Cohesion Policy: Implications for Monitoring and Evaluation

Budapest May 24-25, 2011

Dr Jim FitzpatrickManaging DirectorFitzpatrick Associates

122 Ranelagh VillageDublin 6Tel: +353 1 6280084Fax: + 353 1 6219771

2

issues in future Cohesion Policy affecting Monitoring and

Evaluation (M+E)

emerging parameters of 2014-20 period

the “new” Logical Framework

challenges in implementing new approaches

some practical suggestions

TOPICS

3

EU STRUCTURAL AND COHESION FUNDING

20 years + of multi-annual programmes

M+E central to the programming package

Objectives, priorities, targets/indicators

Agreed strategies, programmes

Regular monitoring

Formal evaluation

Gradual development of M+E practice

2014-20 will involve significant changes

                                                       1989  1990  1991  1992  1993  1994  1995  1996  1997  1998  1999   2000  2001  2002  2003  2004  2005   2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020

CEE entry

4

2014-20: INCREASING THE EFFECTIVENESS OF COHESION POLICY

the main focus of evaluations “tended to be on processes and financial implementation rather than on the actual results of programmes”, (Ex Post Evaluation

of Cohesion Policy Programme 2000-06, Synthesis Report April 2010)

a Member State-Commission National Strategic contract where Member States (or Regions) “commit themselves to quantified and verifiable objectives”. (An

Agenda for a Reformed Cohesion Policy , Barca Report, April 2009)

“specific binding conditionality in the areas directly linked to cohesion policy would be agreed with each Member State and/or Region – depending on the institutional context…” (Investing In Europe’s Future: Fifth Report on Economic, Social and

Territorial Cohesion, Nov. 2010)

Work of High Level Group on Future Cohesion Policy, Conditionality Task Force (Feb.-April 2011), Evaluation Unit/Network, Hungarian Presidency High Level Conference (March 31/April 1).

5

A CONFLUENCE of Related STRANDS

Monitoring And

Evaluation

ThematicConcentration

BindingContracts

Conditionality

Better Information

Better Evaluation/

CFA

Performance/ Results

measuring outcomes

6

provides goals

provides themes

regulatory requirements

evaluation will inform content/progress

ex ante, ongoing, ex post

EMERGING NEW ARCHITECTURE 2014-20

* Possibly for Cohesion Fund, ERDF, ESF, EAFRD, EEF

Europe 2020 Goals

Common Strategic Framework (CSF) ?*

Regulations

Partnership and Investment Contracts*

Operational Programmes

Themes/Axes (from menu)

7

STRENGTHENED CONDITIONALITY?

already is various conditionality: regulatory (inc. M+E); strategic. Infrastructure planning; institutional

types of conditionality:

“macroeconomic” – part of Stability/Growth Pact

“ex ante” – preconditions (see existing categories)

“structural reform” – structural/admin reform milestones

“performance” – Programme/EC objectives

incorporated in Partnership Contracts?

possible financial incentives/sanctions?

if implemented, could have implications for Monitoring/Evaluation

8

Other Factors

Monitoring and Evaluation

ProgrammingStrategy

Needs ThematicObjective

IntendedResult

Contribution - ImpactPolicy AllocatedINPUTS

TargetedOUTPUTS

ActualINPUTS

AchievedOUTPUTS

ActualResult

NEW LOGICAL FRAMEWORK

outcomes monitored/impacts evaluated

inputs/outputs “contribute” to outcomes

outcome indicators should be “responsive” to intervention

Source: V. Gaffey, Acting Director, Policy Development, DG Regio, Intervention Evaluation Conference, Budapest, May 24-25, 2011.

9

RE-DEFINING THE LOGIC

Source: based on DG Regio, Concepts and Ideas: Monitoring and Evaluation in Practice of European Cohesion Policy 2014+, Draft, 30 March 2011

Old: Inputs Outputs Results(Short/medium)

IMPACTS(Long term)

Monitoring Evaluation

Monitoring

New: Inputs Outputs Results/ outcomes

Evaluation

Impact

10

CRITERIA FOR A GOOD OUTCOME INDICATOR

Reasonable: capturing the essence of an outcome according to a reasonable argument about which

features of the outcome they can and cannot represent

Robust: reliable, statistically and analytically validated, and, as far as practicable, complying with

internationally recognised standards and methodologies;

Responsive to policy: linked in as direct way as possible to the policy interventions for whose

assessment they are used, while not being subject to manipulation;

Normative: having a clear and accepted normative interpretation (i.e. there must be agreement that a

movement in a particular direction or within a certain range is a favourable or an unfavourable result);

Feasible: built, as far as practicable, on available underlying data, their measurement not imposing too

large a burden on Member States, on enterprises, nor on the citizens;

Debatable: timely available to a wide public, with room being built for public debate and for their own

revision when needed and motivated.

Source: F. Barca, P. McCann, Outcome Indicators and Targets – Towards a Performance Oriented EC Cohesion Policy, High-

level Group on Future Cohesion Policy, Meeting No. 8, 15 Feb. 2011

RRR-NFD instead of SMART! The challenge is in the “Third R”

11

THE RESPONSIVENESS ISSUE: SUB-CRITERIA FOR OUTCOME INDICATORS?

Use Pilots to develop this

12

CHALLENGES REGARDING OUTCOME/RESULT INDICATORS

Conceptual: the causation problem remains? a lot of judgement?

Communication: can we be confident this will address the issue? trade-off between communication and robustness

Capacity: who is going to define the indicators? A specialist task. Will need forensic

precision

Time-lags: when will attributable outcomes arise?

Conditionality: M+E insufficiently robust to support binding conditionality

13

SOME SUGGESTIONS FOR COMMISSION/MEMBER STATES

skills, capacity, technical assistance, technical support

M+E as communication tools? Simplify language? Consistent terminology, e.g.

“outcomes” v “results”

revisit institutional context, inc. Monitoring Committees, Managing Authority

Nature of Commission Guidelines and technical support

use of regular Peer Review Groups, (“boots on the ground”) ongoing Evaluators

merge Monitoring and Evaluation function in Member States

greater role for Eurostat/National Statistical Offices (but avoid pure context

indicators)

nature of Multi-annual Evaluation plans

14

And finally…

keep the Regulations general and Guidelines detailed, specific

(not vice versa)

don’t rush the Guidelines

THANK YOU.