World Meteorological OrganizationWorking together in weather, climate and water
WMO OMM
WMO www.wmo.int
5.8 Monitoring and Evaluation of 5.8 Monitoring and Evaluation of Capacity DevelopmentCapacity Development
WMO OMM
GSN STn ERn1KOn12
KOn11
KOn13
Del
Del
Del
Act
Act
Act
Act
Budget
Budget
Budget
Budget
SP
OP (2012-15)
SECRETARIAT RBB (2012-15)
Linkage between the next SP, OP and RBB
WMO OMM
ST 1 ER 1
KOs of TC1 … TC8
Indicators, BaselinesTargets
KOs of RA I … VIIndicators, Baselines, Targets
Secretariat DepartmentsActivities and Budget
ER 2 For every ER, eachRA and TC will define specific Key Outcomes together with Indicators, Baselines and Targets; based on these plans and in close coordination the Secretariat develops deliverables, activities and budget proposals.
Framework of the Results-based WMO Operating Plan and Secretariat Budget 2012-2015
Strategic
levelOperational
level
(5) (8)
ER 1
ER 2
WMO OMMKO’s , KPI’s and Priority Areas
• ST2: Advancing scientific research and application as well as development and implementation of technology
• ER4:Enhanced capabilities of Members to access, develop, implement and use integrated and interoperable surface-based and space-based systems for weather, climate and hydrological observations, as well as related environmental observations, based on world standards set by WMO.
• KO 4.1: WMO Integrated Global Observing System (WIGOS) is implemented
• KPI– Completion of WIGOS milestones according to WIGOS implementation
plan., – Increased availability of observations for users
• Priority: WIGOS and WIS are priority areas
• Total of 24KOs, most of them have not more than 2KPIs.• Additional 7 KOs on GFCS and DRR as major initiatives
WMO OMM
1. WMO Strategic Plan : High-level statement of strategic directions and priorities expressed through Strategic Thrusts and organization-wide Expected Results and Performance Indicators addressing the Global Societal Needs;
2. WMO Operating Plan: converts the organization-wide ERs into specific outcomes/ deliverables, indicators and targets
3. Secretariat Implementation Plan and budget: converts the organization-wide ERs into Secretariat specific outcomes/ deliverables, indicators and targets and links them to the planned activities and the Secretariat budget;
4. WMO Strategic Executive summary: In a language appealing to those outside WMO and those making decisions to fund NMHSs.
and
Tool/System for Monitoring & Performance Evaluation – Tracks implementation and provides for control and corrective actions for achieving the ERs.
Strategic planning documents
WMO OMM
WMO Strategy for Capacity Development : Why?
• Establish a CD Strategy to ensure, in a holistic approach, all actors in Met/Hydro/Climate work towards the same overall objective :
Facilitating sustainable development of NMHS
Resolution Cg-XVI/B/WP 11.2• Congress noted that a coordinated and cohesive approach for capacity
development would be needed to enhance capabilities of NMHSs in developing countries to meet growing Societal Needs at different levels.
• Congress noted that capacity building is a crosscutting activity and contributes to all Expected Results (ER), especially ER 6 and Strategic Thrust 3 of the 2012-2015 Strategic Plan.
WMO OMM
Definitions and Logic of WMO strategic planning approach
WMO Strategic
Plan WMO OP
2012-2015
WMO OMM
WMO OMM
Strategic Thrust n3Expected Result n6
WMO OMMTools to Monitor and Evaluate Progress
• Types: Empirical, Qualitative, Index, breadbasket, case histories, Indicators (available economic, social or environmental)
• Example Collection tools: – Self reporting: Surveys, Visits of PR and national staff
– Country assessments– National Records– Reports: OPAGS, Regional WGs, Fellowship Evaluations, project evaluation
– Country Profile Data Base
WMO OMM
Conceptual Framework for Monitoring and Evaluation
Inputs
WMO governance decisions
TCs and RAs human resources
Financial & human resources:WMO Secretariat
WMO Joint ProgrammesMembers
Partner Organizations’ contributions
Activities
Activities asstipulated in the
WMO Operating Plan based oncontributions
of theTCs, RAand the
Secretariat
Activitiesof WMO JointProgrammes
Activities ofMembers
Deliverables
Deliverables asstipulated in the
WMO Operating Plan based oncontributions
of theTCs, RAs
and theSecretariat
Deliverables of WMO Joint
Programmes
Deliverables of Members
Expected Results
Expected Resultsas stipulated in the
WMO Strategic Plan
Expected Results of WMO Joint
Programmes
M&EM&EM&EM&E
Feedback and learning
WMO Secretariat enabling processes and coordination
Key Outcomes
Key Outcomesas stipulated in the
WMO Operating Plan
Key Outcomesof WMO JointProgrammes
WMO OMMCapacity Development – and ER 6
Expected Result 6: Enhanced capabilities of NMHSs, in particular in developing and least developed countries, to fulfil their mandates.
Looking at Key Outcomes (Kos) and Key Performance Indicators (KPIs)
How do we measure?
WMO OMMEC-WG/SOP Questionnaire
• The working group reviewed in 2011 a standardized questionnaire to assess the impacts of the achieved results.
• Such a survey could help create baselines and targets.
WMO OMM
Draft Survey on WMO Impacts of Achieved Results on Members
Survey Questions for Capacity Development, ER 6Key Outcomes 6.1-6.4
KO 6.1: NMHSs and Regional Centers are improved, particularly in developing and least developed countries
WMO OMM
KPI 6.1.1: Number of NMHSs and WMO Regional Centers with defined roles in national and regional socioeconomic development activities, in cooperation with international and regional development agencies
1. Has your NMHS been involved in national or regional socio-economic development activities in cooperation with international and regional development agencies?
Yes No 2. Does your NMHS have a defined role in the National Development Plan?
Yes No 3. If yes, please define your key role:
WMO OMM
KPI 6.1.2: Number of NMHSs with programmes and networks for collaborating on service delivery at regional levels
4. During the past two years, has your NMHS carried out programmes or become part of a network for collaborating on service delivery at regional level?
Yes No 5. Does your NMHS carry out programmes or participate in networks on service delivery at Regional level? Yes No 6. If yes, please give details:
WMO OMM
• KO 6.2: Infrastructure and operational facilities of NMHSs and Regional Centres are improved, particularly in developing and least developed countries
WMO OMM
KPI 6.2.1: Number of NMHSs with improvement in infrastructure and operational facilities to address identified deficiencies
7. During the last two years has your NMHS implemented or developed improvements to its service delivery capacity? Yes No 8. If yes,
i. In the surface observing network? Yes No
ii. In the upper-air observing network? Yes No
iii. In equipment for meteorological/environmental satellite data? Yes No
iv. In the data-processing/forecasting sector? Yes No
v. In other operational sectors of your NMHS (please specify)
WMO OMM
KPI 6.2.2: Number of NMHSs with modernized capacities to develop and disseminate products to their customers
9. What distribution systems do you use to get products and services to users?
i. Telephone of facsimile
ii. Media (printed, TV & Radio)
iii. SMS
iv. Cell Broadcast
v. Internet (web pages, ftp, email etc.)
vi. Social networking (twitter, face book etc)
vii. Dedicated telecommunication links
viii. Remote displays (electronic bill boards etc)
ix. Staff attachment to users of centres of operation
x. Remote briefing (video conference, telephone, help desk etc.)
xi. Others (please list)
WMO OMM
• KO 6.3: Education and Training Development activities at national and regional levels are improved, especially in developing and least developed countries
WMO OMM
KPI 6.3.1: The number of RTCs providing education and training support for GFCS related activities
• To be completed by WMO Secretariat using other sources.
WMO OMM
KPI 6.3.2: The degree to which Members are getting value for money from the WMO Fellowship Programme
10. Has your NMHS contributed to or had students under the WMO Fellowship Programme during the past two years?
Yes No 11. If yes,
In general terms, how would you rate the value your NMHS has been getting from the Fellowship Programme?
1= very low; 5= very high
1 2 3 4 5
WMO OMM
KPI 6.3.3: The degree to which the RTCs support the regional training demands in particular for students from developing and least developed countries
12. If you send staff to a WMO Regional Training Centre(s), please indicate by inserting ‘X’ the WMO Region(s) where the RTC(s) are located:
WMO Regions I II III IV V VI 13. From your perspective, how would you rate the degree of satisfaction with the RTC(s) you use?
1= very low; 5= very high 1 2 3 4 5
WMO OMM
• KO 6.4: Capacities of NMHSs are enhanced through cooperation and partnerships with other national and regional organizations
WMO OMM
KPI 6.4.1: Development projects and activities funded through voluntary and extra-budgetary resources
• To be completed by WMO Secretariat using other sources.
WMO OMMYour inputs…ideas?
• Questions in Survey….– Are we measuring what needs to be measured to
show the capacities of NMHS are being developed?– Are the questions
• Too simple (yes/no) to be of use?• Too difficult to answer with enough specificity to show
improvement in 4 years?• About right
– What other tools does WMO need to develop/improve (if any) to evaluate and monitor CD?
WMO OMMAdditional Sources
• Survey on the Role and Operation of Meteorological Services (2011)
• Regional Surveys• Programmatic Surveys• Information from assessments done on bilateral
basis, other development agencies or at sub-regional levels as part of project development
WMO OMM
Annex 1KPI reference
sheet to be completed
WMO Strategic Plan 2012 – 2015 Key Performance Indicator(KPI) Reference Sheet
KPI: Expected Result: Key Outcome:
UNIT OF MEASURE
PLAN FOR DATA ACQUISITION Data collection method: Data source(s): Timing/frequency of data acquisition: Estimated cost of data acquisition: Responsible department: Responsible individual(s):
DATA QUALITY ISSUES Data limitations (if any): Actions planned to address data limitations:
PLAN FOR DATA ANALYSIS, REVIEW, & REPORTING How the data will be analyzed: How data will be presented: Reporting date and to whom:
WMO OMM
Annex 2Exampleof a duly
completed KPI Reference
Sheet
WMO Strategic Plan 2012 – 2015 Key Performance Indicator (KPI) Reference Sheet 5.2.1: Increased research on operations products and services
Expected Result 5: Enhanced capabilities of Members to contribute to and draw benefits from the global research capacity for weather, climate, water and environmental science and technology development Key Outcome 5.2: Research in the prediction of high-impact weather on time scales of hours to seasons is enhanced
UNIT OF MEASURE Assessment by JSC of WWRP on progress in high-impact weather.
PLAN FOR DATA ACQUISITION Data collection method: annual meetings of WWRP JSC Data source(s): reports from WWRP working/expert groups/team Timing/frequency of data acquisition: once per year Estimated cost of data acquisition: minimal Responsible department: RES/ARE/WWR Responsible Individual(s): C/WWR
DATA QUALITY ISSUES Data limitations (if any): Actions planned to address data limitations:
PLAN FOR DATA ANALYSIS, REVIEW, & REPORTING How the data will be analyzed: review of reports by JSC members, discussion at annual meeting How data will be presented: report from annual JSC/WWRP meeting Reporting date and to whom: June each year; to CAS and WMO leadership
Top Related