Council of Europe May 2005 - coe.int · a - 2 - Course objectives 1. Defining clear and informative...
Transcript of Council of Europe May 2005 - coe.int · a - 2 - Course objectives 1. Defining clear and informative...
a
- 1 -
Advanced PMM/RBB Training course
Council of EuropeMay 2005
a
- 2 -
Course objectives
1. Defining clear and informative performance indicators for preparing the 2006 PoAbudget & progress review/reporting
2. Review common difficulties with existing performance indicators
3. Introducing different types of indicators4. Ways towards using ‘common’ indicators?
a
- 3 -
Performance Indicators
What are they?� Essentially a ‘marker’;� Describing the results and effects of a
project or a programme in terms of quality and quantity over time.
� They are essentially indicators of project effects & success.
a
- 4 -
Performance indicators
Benefits to the project manager?� Agreed quality criteria for internal CoE services;
� Opportunity to build up ‘evidence’ on what does and doesn’t work – fosters organisational learning
� Enables manager to identify ‘good practices’ as well as ‘what the department is good at’;
� Provides a reference for discussions on the effectiveness of thedepartment;
� Helps defining the expected results.
a
- 5 -
Common problems in formulating performance indicators
a
- 6 -
Performance indicators too general
Often indicators are too vague, such as:
Participation of a significant number of parliamentarians and NGOs in the joint actions on Social Europe & North-South Dialogue
Data transmission method codified and transmission possible by the end of 2005 for most States which so wish
a
- 7 -
Additional information on quantities required
• There is no information on the initial percentage, therefore no comparison possible; check whether it is appropriate to indicate a date for the deadline.
• Often the quantities are just forgotten, e.g. number of hits on the Internet/Intranet portal.
• Don’t forget the quantification and the time frame if meaningful: 50% increase of hits … (compared to xxx hits in the year 2003) in the year 2004.
Be aware of the need for baseline data & for a clear definition of the scope of the intervention. Often one encounters indicators such as:80% of documents to be produced within the deadline
a
- 8 -
Quantitative but no qualitative info
Example: by 31 December 2005, the new database is operational
• There is no information which criteria this databank should comply with and for whom it is intended.
• No information on whether the introduction is successful (e.g. percentage of users being satisfied with the performance of the databank)
a
- 9 -
Performance Indicator = Activity
Is it possible to:
• specify the target group; • define the priority subjects, the general publications
and the main domains;• monitor whether the target group has received the
information.
General publications on priority subjects and the main domains of activity are made available to the citizen and target groups (expected result)A new graphic presentation is introduced; a new collection of booklets and leaflets is issued
a
- 10 -
How do you identify indicators?
Performance Indicators must correspond to certain criteria to ensure that:� Indicators are precise and relevant for the results
and the objectives of the project ( see logical framework);
� Provides information to key stakeholders;� Indicators determine what information needs to be
gathered and should, therefore, be selected on the basis of what data can be realistically and efficiently collected – or even collected at all.
a
- 11 -
Criteria 1: Relevant and Precise
• Relevant for the results and objectives they are describing in more detail;
• Precise by specifying what one intends to achieve and what that means in terms of the intended changes for specific organisations, groups of beneficiaries and situations.
Indicators are informing stakeholders about what the project intends to achieve and should therefore be:
a
- 12 -
Criteria 2: Achievable
Indicators should be based on a good knowledge of the actual situation and should therefore:Specify what is realistically achievable taking the context and the external factors into consideration (assumptions);
a
- 13 -
Criteria 3: Measurable and verifiable
• Specify the planned quantitative & qualitative changes over time with reference to the actual situation (base line);
• Ensure that such indicators are easily verifiable in practice by indicating how, where and by whomthey can be checked.
Indicators should be an easy way of checking if a project is achieving its aims and works according to plan:
a
- 14 -
Criteria 4: Selective and consistent
Monitoring can become a tedious and expensive task if not sufficient attention is paid to its manageability:
• Select only a limited number of key indicators for a number of key results;
• Whenever possible use the same key indicators to allow for a consistent, comparative analysis of results over time.
a
- 15 -
Using the LF to select indicators
Financial Resources
Performance
Indicator
Sources of
Verification
AssumptionsIntervention
Logic, PMM
Activities
Expected
Results
Project
Objective
Programme
Objective
Expected
Results
Activity Area
Objective
Strategic
Objective
Intervention
Logic, RBB
Activities
a
- 16 -
Expresses the long-term impact and the intended positive changes upon the beneficiaries (there can be multiple programme objectives).
Programme
Objective
Project
Objective
Defines the projects success and should therefore address the specific needs of the target group(s). Should be defined in terms of sustainable benefits or positive change for the target group(s).
Expected
Results
Describe the intended provision of services / products provided by the project to the target group(s).
Activities
Activities are the actions to be carried out in order to attain the Expected Results. They describe the budget in operational terms. Financial resources should be clearly attributed at this level by type of expenditure within the budgetary tables to be completed. The activities do not appear in the Logframe.
Objectives and results (PMM)
a
- 17 -
Expresses the long-term impact and positive changes upon the beneficiaries. (For Partial Agreements and administrative units with Votes I and VI, the Sub-Objective equates to the Strategic Objective)
Strategic
Objective
Activity Area
Objective
Defines the activities success. The Activity Area Objective should address the needs of the target group(s). It should be defined in terms of sustainable benefits or positive change for the target group(s).
Expected
Results
Describe the intended provision of services / products provided by the project to the target group(s).
Activities
Activities are the actions to be carried out in order to attain the Expected Results. They describe the budget in operational terms. Financial resources should be clearly attributed at this level by type of expenditure within the budgetary tables to be completed. The activities do not appear in the Logframe.
Objectives and results (RBB)
a
- 18 -
Progress chain and overlaps
Implementation of activities
Delivery of quality Services and Products
Use of Services and Products by intermediary
Satisfaction with Services and Products
Services reached all intended target groups / clients
Internal changes of the target group
Performance changes of target group
Use of services by final beneficiary
Satisfaction with services by final beneficiary
Change of behavior of final beneficiary
Benefits at general society
Results
Initial Impact
Long-termimpact
a
- 19 -
Progress organisational change
Training is taking place
Increased awareness of quality issues
Different ways of dealing with quality are compared
Some departments agreed to change and plan ahead
More departments have a clear action plan
Internal changes are being realised
Departments provide better quality services
Clients receive better products
Clients are more knowledgeable
Clients change their strategies
More citizens enjoy essential human rights
Results
Initial Impact
Long-termimpact
a
- 20 -
Types of indicators (1)
There are indicators that identify:
� Change/improvement processes�Socio-cultural, organisational, institutional�Understanding, insight, behaviour, relations
� Product development� Client benefit
� Technical, material
a
- 21 -
Identifying indicators (2)
Change (over time) can be indicated by:� Measuring and counting : numbers, %,ratios,.
� Comparison: quality benchmarking against an
accepted standard of service/product,
� Classification: yes / no of product/service availability,
suitability,....
� Qualitative descriptions of change processes
� A combination of the above.
a
- 22 -
Identifying indicators (3)
� You will usually need more than one indicator as it is rarely the case that one indicator can `prove’ anything.
� Trying to establish unequivocal causal relationships is extremely difficult and you usually need several different indicators before you can draw conclusions with confidence
a
- 23 -
Case Study Task:
1. Review the project objective and the results to be attained by the project by the end of next year (“ Progress Chain”)
2. Select and formulate indicators using the FOUR CRITERIA;
3. Present and discuss.