Capacity Needs Assessment of Monitoring and Evaluation … · 2020-03-02 · Capacity Needs...

Capacity Needs Assessment of Monitoring and Evaluation

Systems in KwaZulu-Natal Province

By

Dr. Nelson Kamoga and Dr. Faniel Habtemichael

Final Report : 24 January 2010

Capacity Needs Assessment of Monitoring and Evaluation Systems in KwaZulu-Natal ii

Table of Contents 1. INTRODUCTION ............................................................................................................. 1

1.1. Context ............................................................................................................................... 1

1.2. Objectives of assessment ................................................................................................... 6

1.3. Methodology ...................................................................................................................... 6

1.4. Limitations of methodology ............................................................................................... 9 2. Overview of the M&E System in KZN ........................................................................... 10

2.1. M&E units in the province ................................................................................................ 10

2.1.1. Office of the Premier ........................................................................................ 12

2.1.2. Agriculture and Environmental Affairs ............................................................. 13

2.1.3. Arts & Culture ................................................................................................... 14

2.1.4. Community Safety and Liaison ......................................................................... 15

2.1.5. Economic Development & Tourism ................................................................. 16

2.1.6. Education .......................................................................................................... 17

2.1.7. Health ............................................................................................................... 18

2.1.8. Human Settlements .......................................................................................... 19

2.1.9. Local Government & Traditional Affairs ........................................................... 19

2.1.10. Public Works ..................................................................................................... 20

2.1.11. Social Development .......................................................................................... 21

2.1.12. Sports & Recreation ......................................................................................... 21

2.1.13. Transport .......................................................................................................... 22

2.1.14. Treasury ............................................................................................................ 23

2.1.15. Sisonke District Municipality ............................................................................ 24

2.1.16. Msunduzi Local Municipality ............................................................................ 24

2.1.17. Public Service Training Academy...................................................................... 25 3. OVERALL ASSESSMENT FINDINGS ................................................................................ 27

3.1. Institutional arrangements for monitoring and evaluation ............................................. 27

3.1.1. Legal framework, policies and procedures to implement an M&E systems ... 27

3.1.2. Overall system used to perform M&E tasks .................................................... 29

3.1.3. Roles and responsibilities to manage and implement M&E system ............... 29

3.1.4. Level of coordination on management and implementation of M&E system 30

3.1.5. Donors and other external stakeholder involvement in M&E capacity building30

3.1.6. Mechanisms to engage civil society, public and the private sector in M&E.... 31

3.2. Monitoring and evaluation advocacy ............................................................................... 31

3.2.1. Leadership influence on development of an M&E system .............................. 32

3.2.2. Use of M&E information for planning, resource allocation and policy making32

3.2.3. Management feedback on quality of M&E system .......................................... 33

3.2.4. Management capacity to perform and use inferential analyses on M&E data33

3.3. Monitoring and evaluation knowledge and skills ............................................................. 33

3.3.1. Capacity of the M&E system to generate and disseminate good quality data 33

3.3.2. Electronic system to collate data for the M&E system .................................... 34

3.3.3. Availability and quality of recent Monitoring and evaluation reports ............ 35

3.3.4. Availability of M&E system to generate data on the MDG indicators ............. 35

3.4. Reporting mechanisms ..................................................................................................... 36

Capacity Needs Assessment of Monitoring and Evaluation Systems in KwaZulu-Natal iii

3.4.1. Existence of an effective reporting mechanisms ............................................. 36

3.4.2. Adequacy of monitoring system to generate disaggregated data reports ...... 36

3.4.3. Availability of mechanisms to disseminate M&E reports to stakeholders ...... 37

3.4.4. Availability of mechanism to link M&E data and performance ....................... 37 4. RECOMMENDATIONS ON M&E CAPACITY BUILDING .................................................... 38

4.1. Substantially increase demand for M&E by provincial governing structures .................. 38

4.2. Elevate awareness of incentives for M&E across the province ....................................... 38

4.3. Perform detailed diagnosis of department and municipality M&E systems ................... 40

4.4. Re-arrange organizational structures to strengthen the role of M&E ............................. 40

4.5. Increase quantity and quality of M&E supply .................................................................. 42

4.6. Strengthen structures and process to support M&E systems ......................................... 44

4.6.1. Increase financial resources ............................................................................. 44

4.6.2. Develop reliable data systems ......................................................................... 44

4.6.2.1. Good quality performance indicators: ............................................................. 44

4.6.2.2. Reliable interoperable information systems .................................................... 45

4.6.2.3. Coordination with civil society and community ............................................... 47 5. CONCLUSION ............................................................................................................... 48

5.1. Way forward ..................................................................................................................... 48

5.2. Action plan ........................................................................................................................ 51 Appendix 1: M&E assessment questionnaire ........................................................................ 53 Appendix 2a: Assessment summary findings ........................................................................ 58 Appendix 2b: Assessment summary findings ........................................................................ 59 References .......................................................................................................................... 60

Tables and Figures

Page Table 1 M&E units: Year of establishment and available staff………………………………. 11 Table 2 M&E capacity building results framework..……………………………………………… 50 Table 3 Provincial action plan……………………………………………………………………………….. 51 Table 4 Public Service Training Academy action plan……………………………………………. 52 Figures 1-16 Organograms……………………………………………………………………………………………. 13 - 26 Figure 17 M&E unit restructuring – Option 1…………………………………………………………… 41 Figure 18 M&E unit restructuring – Option 2a…………………………………………………………. 42 Figure 19 M&E unit restructuring – Option 2b…………………………………………………………. 42 Figure 20 Performance indicators……………………………………………………………………………. 45 Figure 21 M&E capacity building framework …………………………………………………………… 49

Capacity Needs Assessment of Monitoring and Evaluation Systems in KwaZulu-Natal 1

1. INTRODUCTION

1.1. Context

Being a young democracy, South Africa has had to deal with the colonial and apartheid legacy of a

poor and fragmented public service. Over the last 15 years, the government of South Africa has

built institutions and developed policies of change to ensure equality and improvement in the

well-being of the previously disadvantaged majority. Although the country has made some

progress towards several intersectoral goals such as primary school enrolment and gender parity

in education, progress has been insufficient or even reversed for many of the Millennium

Development Goals (MDGs). For example, since 1994, life expectancy has fallen by more than 10

years – mainly because of the rise in HIV-related mortality. In the middle of 2009, South Africa was

hit by a wave of service delivery protests in peri-urban communities1. According to the Municipal

IQ, these protests are due to poor service delivery and due to poorly managed state structures,

one of which include performance monitoring and accountability 2. There is an increasing

acceptance by government and citizens that the objectives of delivering quality services have not

being met3. The reasons include lack of political will, inadequate leadership, management

weaknesses, inappropriate institutional design, misaligned decision rights and absence of a strong

performance culture with effective rewards and sanctions.

World-wide, a growing number of governments are working to improve their performance by

creating systems to measure the quantity, quality, and targeting of the services that the

government provides and to measure the outcomes and impacts resulting from such outputs.

Such systems also facilitate an understanding of the causes of good and poor performance. One

of the key reasons performance systems have become critical is the increasing fiscal pressure to

spend resources more effectively and efficiently as the public demands quick redress of

community needs that the government is expected to provide. In 2008 the Auditor-General4

reported that the Departments of Health and Education had inadequate reconciliations,

insufficient controls and monitoring, as well as the lack of supporting documentation.


Since 1994 there has been growing emphasis on M&E, largely to support democracy and transit

the country to a more transparent state. From 1995 there was emergence of M&E, in some

departments, but not within a coherent and overarching framework. This monitoring was largely

driven by the need to report on financial accountability, due to requirements of the Auditor-

General. However, there was widespread failure to address issues of impact or effectiveness

because the focus was more on activities and outputs, rather than outcomes. Until 2005, only

individual staff performance evaluations were institutionalised and regularly carried out in the

South African government. Policy programme monitoring and evaluation, however, were

undertaken sporadically by line function departments for purposes of their annual departmental

reports (Cloete5, 2009).

It is in this context that the government of South Africa has recently engaged in a process to

strengthen and institutionalise monitoring and evaluation (M&E) as part of the key structures to

continuously and more accurately track the process of service delivery to the people of South

Africa. In 2004, the Cabinet initiated plans for a monitoring and evaluation system for government,

and the Presidency subsequently developed the Government-wide Monitoring and Evaluation

Framework (GWMEF). The principles of the Policy Framework for the GWMEF6 are that M&E

should: contribute to improved governance (Transparency, Accountability, Participation,

Inclusion), be rights based, be development-oriented – nationally, institutionally and locally be

undertaken ethically and with integrity, be utilisation oriented, be methodologically sound and be

operationally effective.

According to Cloete7 (2009) the following considerations inter alia motivated a Cabinet decision in

2005 to develop the GWMEF:

A need for regular national government report-backs to the International UN Millennium

Goals Initiative on the progress with halving poverty in South Africa by 2014;

The fact that South Africa was the host of the World Summit on Sustainable Development

in 2002 and at that time did not have any national M&E system to assess sustainable

development as required by the Rio Convention of 1992 and reiterated at the

Johannesburg Summit where South Africa was the host country;


The undertaking by the South Africa president to regularly inform citizens about progress

with the Government’s National Programme of Action (POA);

The fact that donors are increasingly requiring systematic M&E of projects and

programmes that they fund, in order to protect their investments; and

The fact that institutionalising national M&E systems has proved to be an international

good governance practice.

The South African M&E system draws its mandate from several pieces of legislative framework

including the Constitution (Act 108 of 1996), Public Finance Management Amendment Act No 29

of 1999, Municipal Finance Management Act (2003), Public Service Act (1994 as amended), Batho

Pele principles, National Treasury guidelines and frameworks , and Cabinet decisions. The South

African Constitution calls for high standard of professional ethics; efficient, economic and effective

use of resources; development oriented public administration, provision of services in an

impartial, fair, equitable and unbiased manner; accountability; transparency; good human

resources management and career development practices to maximise human potential; and

representation. This creates clarity for departments on what is required to meet good governance

standards and helps them to focus in their efforts. The Public Finance Management Amendment

Act No 29 of 1999 requires public entities to control financial risk and promote internal control.

This act gives the National Treasury power to monitor the implementation of provincial budgets. It

also gives Accounting Officers the responsibility for the effective, efficient and transparent use of

the resources. The Municipal Finance Management Act (2003) requires municipalities to properly

monitor revenue and expenditure, indicate service delivery targets and performance indicators for

each quarter; assess the performance of the entity and monitor the operational functions of their

entities. The public service act (Amended in 1994) emphasizes the need for monitoring and

evaluation systems to ensure effective and efficient public service performance. The Eight Batho

Pele principles were developed to serve as acceptable policy and legislative framework regarding

service delivery in the public service. They lay the ground for M&E by requiring efficient and

effective utilisation of resources. These principles reinforce the need for benchmarks to constantly

measure the extent to which citizens are satisfied with the service or products they receive from


departments. They also play a critical role in the development of service delivery improvement

plans to ensure a better life for all South Africans.

National Treasury guidelines set monitoring and evaluation plan to examine whether objectives

have been met. According to the guidelines public entities, state-owned enterprises and

constitutional institutions are required to submit their expenditure estimates and related

performance outputs to inform the determination of appropriation requirements. The guidelines

require departments to critically evaluate their performance against stated targets and ensure

that outputs are achieved and measurable.

The current National Strategic Planning Green Paper8 emphasises the linkage and feedback loop

that exists between planning, and performance monitoring and evaluation. The processes and

results of monitoring and evaluation will be critical to planning and may result in modified

sequencing of programmes. The other discussion paper on improving Government Performance9

(2009) that describes the Presidency’s approach to management monitoring and evaluation

concentrates on the 10 priority outcomes of the government’s Medium Term Strategic

Framework. The paper proposes the need to use outcome and output measures to promote a

change in behaviour, and create a culture of accountability. It further states that its focus areas are

management of outcomes through ministerial accountability for improving delivery, performance,

institutionalising the Government-wide Monitoring and Evaluation system, and unblocking service

delivery.

The GWMEF draws from three data terrains namely programme performance information,

evaluations, and social, economic and demographic statistics.

In the case of the program performance information, the government, through the Department of

National Treasury, developed the Programme Performance Information Framework (PPIF) in May

2007 to monitor both financial and non-financial information about government services and

activities10. The Department of National Treasury’s mandate is informed by sections 215 and 216

of the Constitution, and other legislation such as the Public Finance Management Act (PFMA) of

1999 and the Municipal Finance Management Act (MFMA) of 2003. The Treasury’s engagement

with the GWMEF revolves around ensuring that information on inputs, activities, outputs and


outcomes underpins planning, budgeting, implementation management and accountability

reporting to promote economy, efficiency, effectiveness and equity, as well as transparency and

expenditure control. In the whole, the PPIF goal is to ensure that performance information is

available to national, provincial and local government managers at each stage of the planning,

budgeting and reporting cycle, and a results-based approach to managing and responding to

service delivery would become institutionalised. The lead institution responsible for the

implementation of the PPIF is the National Treasury. Other institutional stakeholders of the

Government-wide monitoring and evaluation system include the Department of Public Service

and Administration (To monitor staff performance evaluations), Department of Cooperative

Governance and Traditional Affairs (To assess policy programme performances of provinces and

local authorities), Department of Environmental Affairs (To assess state of the environment and

sustainable development), Public Service Commission (To promote the values and principles

governing public administration listed in section 195 of the Constitution), Statistics South Africa

(Responsible for data collection, storage and quality control), Public Administration Leadership

and Management Academy (To provide monitoring and evaluating capacity-building through

training). All national departments, provinces, and local governments are therefore expected to

develop M&E systems for performance monitoring and evaluation at their level.

The second data terrain of the GWMEF is the social, economic and demographic statistics focusing

on information generated by Statistics South Africa (Stats SA), government institutions, private

sector, research institutions and NGOs. As the agency responsible for collecting and reporting of

official statistics, Stats SA has a central role of ensuring that such statistics are fit to use. The

Statistics Act (No. 6 of 1999) mandates the Statistician General (SG) to put in place a framework to

enable evaluation of statistics generated by organs of the state. In addition, the Act empowers the

SG to designate as official any statistics produced by Stats SA or any organ of the state and

authorises the Statistician-General to comment on the quality of national statistics produced by

another organ of state; and to publish such other department’s statistics. It is in this regard that

the Stats SA has developed the South African Statistical Quality Assessment Framework

(SASQAF)11. Through SASQAF, Stats SA has enhanced transparency in data monitoring and

evaluation and thus promotes data quality maintenance within a decentralised system of statistics

production.


The third data terrain of the GWMEF is the evaluation, and focuses on the standards, processes,

and techniques of planning and conducting evaluations and communicating the results of

evaluations of government programmes and policies. The Presidency will be developing an

Evaluation Framework and other guidelines and support material to facilitate the overall

implementation of evaluation systems across the three spheres of government. The Evaluations

Framework aims to encourage government institutions to evaluate their programmes on a regular

basis, provide guidance on the general approach to be adopted when conducting evaluations and

provide for the publication of the results of evaluations.

1.2. Objectives of assessment

The overall goal of the assessment was to perform a rapid appraisal to identify the challenges

facing the institutionalization of M&E in the province of KwaZulu-Natal (KZN) and the capacity of

the Provincial Public Service Training Academy to provide relevant results based provincial M&E

training. Specific objectives of the assessment therefore included:

Determining strengths and weaknesses of the current provincial M&E systems

across departments in the province

Develop strategies to improve capacity of provincial M&E systems based on

opportunities and threats identified from the assessment

1.3. Methodology

Before the field work to assess the capacity of the M&E in the province started, a stakeholders

meeting comprising the M&E Chief Directorate of the Office of the Premier, representatives from

the M&E units of the provincial departments, two municipalities, UNDP representative, and two

consultants was held in the Premier’s Office in Pietermaritzburg, KwaZulu-Natal (KZN) on the 1st

of October 2009.

The purpose of the meeting was to initiate the assessment process, and explain to the

stakeholders the UNDP’s capacity development framework, methodology, approach of the UNDP–

commissioned task, and get an overview of the challenges and scope of the need regarding M&E


systems. It was also an opportunity to elicit important inputs from participants on the conduct of

the assessment process. The meeting identified some key challenges including sensitivities around

the current M&E systems and suggested ways of handling them during the assessment process.

Institutions that were represented at the meeting included:

i. Office of the Premier (OTP)

ii. Departments

a) Agriculture & Environmental Affairs

b) Arts & Culture

c) Community Safety & Liaison

d) Economic Development & Tourism

e) Education

f) Health

g) Human Settlements

h) Local Government & Traditional Affairs

i) Public Works

j) Social Development

k) Sports & Recreation

l) Transport

iii. Public Service Training Academy (KwaZulu-Natal)

iv. Municipalities

m) Sisonke (district)

n) Msunduzi (local)

v. UNDP representative

vi. Consultants

The OTP, UNDP, and the consultants met first with the Public Service Training Academy, and later

with the departments and municipalities. Key issues discussed included:

The OTP role in the capacity assessment was to organize and facilitate stakeholders’

meetings, validate tools of assessment, identify those who will engage in the interviews,

assist the consultants regarding some logistical issues.


The post assessment core function of the OTP will be to assist other departments to build

their M&E systems based on the assessment findings.

The UNDP engagement is based on demand for support for M&E capacity building by the

OTP. UNDP is committed to providing technical support in this regard. For example UNDP

will provide support to develop an E-Library on M&E and other development literature that

will be linked to the UN and other information sources

The role of the Provincial Public Service Training Academy will be crucial in the process of

institutionalization of the M&E system. The assessment would determine human and

institutional capacity of the Provincial Public Service Training Academy to provide relevant

M&E training to provincial departments and other public institutions. Training and other

capacity building activities by the academy will be guided by needs as identified from the

current assessment findings and future departmental M&E system assessments.

Unless there is an environment that supports the M&E system, the impact of technical

experts is minimal. For example the involvement of politicians in the M&E capacity building,

as such individuals are major drivers of demand for M&E.

The assessment involved two-week interviews with key informants responsible for M&E in

fourteen (14) provincial departments and two (2) municipalities (District and Local):

A questionnaire was designed to determine adequacy of available M&E systems. Interviewees

would respond to a question by scoring whether a specific aspect of M&E was adequate or not

using a scale of 1-5 with 5 equating to highly adequate and 1 inexistent at all. Interviewees would

elaborate on the rationale for giving a specific score thus giving clarity to the responses. See

Appendix 1 for a copy of the questionnaire. Departments were also asked to provide any

documentation depicting the relationship of the M&E unit to the rest of the department or

municipality structure.

The data from the interview findings were summarized into frequency tables as shown in

Appendix 2a and Appendix 2b. Section 2 provides a brief overview of institutional status of M&E in

departments, municipalities and the training Academy. Section 3 discusses in more detail


interviewee findings from the assessment and sections 4 and 5 provide detailed recommendations

and a summary action plan for provincial M&E capacity building.

1.4. Limitations of methodology

The assessment was based on key informants providing their perception on the current level of

adequacy of M&E systems that they use in provincial departments and local governments. This

perception may therefore be considered subjective and not truly representative of what is actually

happening on the ground. The diagnosis did not involve an audit of M&E systems to determine

quality of specific data items and information reported by departments, how it is used and the

impact (negative or positive) of using available data. Secondly, key informants were in most cases

department staff involved in generating and reporting information from the M&E systems. Thus

the system diagnostic was based on responses from suppliers of M&E and not those that demand

and therefore drive the need for M&E like politicians and higher department and local

government managers.


2. Overview of the M&E System in KZN

KwaZulu-Natal is a provincial region with an area of 94,361 square kilometers and a population of

about 10 million inhabitants (South Africa's largest provincial population, i.e. about 21%) on the

eastern sea-board of South Africa12. KZN is one of the country's most popular tourist destinations.

And its port, Durban, is the busiest in South Africa and one of the 10 largest in the world.

Agriculture is central to the economy supported by mining, vehicle-manufacturing, vehicle-

manufacturing, and tourism. The province has ten districts, one metropolitan and 61 local

municipalities. The main languages spoken are isiZulu (81%), English (13%), and Afrikaans (1.5%).

Remnants of British colonialism and a mix of Zulu, Indian and Afrikaans traditions give the province

a rich cultural diversity. Fifty four percent of this population lives in rural areas, which are

characterized by extreme poverty and poor infrastructure (roads, transport and

telecommunication). This poses a huge challenge towards ensuring effective delivery of basic

services such as health, water and sanitation. Poverty, over-consumption, poor quality of- and

poor access to education and limited livelihood options, drought and flooding (linked to climate

change), spread of HIV/AIDS (highest prevalence in South Africa) are some of the major

sustainable development challenges in the province.

According to Ndebele13 the public service in KZN, as in many other provinces is characterized by

the amalgamation of different administrations and these are the Natal Provincial Administration,

KwaZulu-Natal Government and the Mass Democratic Movement. Each of these had its own

baggage, culture and attitude with regard to the concept of public service.

Prior to the development of M&E in KZN, the provincial departmental programs contained

financial information which relate mainly to inputs such as government expenditure and income

patterns. Reporting on performance came after a reform agenda since 1999 with the introduction

of the Public finance Management Act (No 1 of 1999).

2.1. M&E units in the province

This section provides an overview of the M&E units in the province including the training Academy.

The accompanying organograms give an indication of the position of the M&E units in the


Table 1 M&E units

Year of establishment and available staff

Provincial departments and municipalities Year M&E Unit established

Number of key staff available

Agriculture & Environmental Affairs 2006 3

Arts & Culture 2007 01

Community Safety & Liaison2 2004 10

Economic Development & Tourism 2006 4

Education 20073 2

Health 2007 6

Human Settlements 20024 1

Local Government & Traditional Affairs 2007 2

Public Service Training Academy 2008 1

Public Works 20095 0

Social Development 20086 1

Sports & Recreation 2009 1

Transport 2000 3

Treasury7

Sisonke District Municipality8 1

Msunduzi Local Municipality9 2003

organizational structure especially in relation to the planning units. The organograms are not

meant to give the whole organizational structures of the departments. Most of the M&E units of

the provincial departments are only 2-3 years old as shown in Table 1. This is with the exception of

the Department of Transport which has been in existence since 2000.

1 There is only 1 manager for corporate strategy

2 Its M&E is only external, monitoring the Police service 3 It featured in the organizational only in 2008 4 Staff were in and out up to 2009 5 Policy was approved in 2009, but no staff is assigned so far 6 It was available on paper in 2007 7 It monitors other provincial departments, it does not have M&E unit for its internal activities. 8 It does not have M&E unit per se, but some of its units have performance management functions, there is no dedicated unit

performing the activity. 9 It does not have M&E unit per se, it is a Performance Management Unit that is functioning as M&E.


2.1.1. Office of the Premier

Although both the provincial and local government had made inroads in bringing services to

different communities and also facilitating poverty eradication programs, it is still not clear

whether of these initiatives had brought any improvement in the life of local people. The

Monitoring and Evaluation (M&E) Directorate in the Office of the Premier (OTP) was instituted in

2005 to provide key information for strategic decision-making and better government service

delivery based on the GWM&EF system that will collect, analyze and synthesize service delivery

data. The OTP holds an oversight role in the provincial government and as such is mandated to be

reliably informed of the status of the government. This information should create the single view

of the province—a view transcending organizational structures and creating a reliable, timely and

accurate picture of the status of the province. The M&E directorate launched the Provincial Nerve

Centre (NC) on 27 August 2008 to generate such information for the OTP. The NC provides an

automated and integrated information management system complete with reporting and analysis,

dashboards and GIS mapping to monitor and evaluate government’s key performance indicators

and to promote transparency and anti-corruption. The key objectives of the Nerve Centre include

bridging the gap between strategic provincial objectives and execution thereof. The system will

help to link objectives to tangible outcomes in terms of information on key performance indicators

and program outputs across departments and local government institutions. The NC will allow for

a single view of the province to assess progress within certain key areas of concern in the province,

along cluster requirements. It will replace the current mostly manual reporting systems, integrate

relevant data into the system, and track the progress of the KZN province growth and

development strategy, among others. Currently the Nerve Centre is operating on silo basis and

data is collected on demand, and does not regularly flow to it as planned. The Nerve Center does

evaluations with insufficient baseline data and relies on secondary sources like Statistics South

Africa. The system is expected to be full-fledged by March 2010.

Although the directorate has, with the assistance of UNDP, laid the foundation for full

implementation of the provincial M&E system there have been some challenges. Key challenges

articulated by the directorate include

Absence of a fully functional National Monitoring & evaluation system;


Lack of public sector capacity in monitoring and evaluation;

Absence of results based management paradigm;

Galvanizing consensus on impact indicators;

Lack of integrated management information system; and

Need to establish a roadmap to achieve MDG’s

Figure 1: Office of the Premier, M& unit.

2.1.2. Agriculture and Environmental Affairs

The M&E entity in the Department of Agriculture & Environmental Affairs was created in the

financial year 2006-2007 but its policies are still in the draft stage. Amendments to the M&E

framework are still outstanding department waits for central government changes to GWMEF. The

unit is better resourced in terms of finances compared to other departments. A recent audit by

the office of the auditor general determined that the unit was not well equipped to implement

M&E functions. In the whole there is still misconception about the role of M&E in the department.

The M&E unit is positioned low in the hierarchy at a sub-directorate level. However, the fact that it

falls under the Planning Directorate the potential of its reports to be used is great subject to the

quality produced.

OFFICE of the PREMIER

Chief Dirfectorate: Monitoring &

Evaluation

Directorate: Monitoring

Sub-directorate: Monitoring

Sub-directorate:

Performance Management

Directorate: Evaluation

Sub-directorate: Evaluation

Sub-directorate:

Training Coordination

Directorate: Business Unit

& GIS

Sub-directorate: Information

Management

Business Intelligence & Development

GIS Management


Figure 2: Department of Agriculture & Environmental Affairs, M& unit.

2.1.3. Arts & Culture

The so called Performance Management (M&E unit) was established in 2007 and no person was

assigned to it so far. The unit falls under the Directorate of Corporate Strategy and is temporarily

run by the Directorate of the Corporate Strategy. The fact that Policy and Planning fall under one

directorate it can make it easier for data to flow for planning purposes. There is also an M&E

committee that meets annually in the Department that oversees M&E functions. The M&E unit is

low in the hierarchy as a sub-directorate. However, the horizontal relationship with the planning

unit under one directorate is a suitable position for information sharing and usage depending on

the relationships.

HEAD OF DEPARTMENT:

AGRICULTURE & ENVIRONMENTAL AFFAIRS

Deputy Director General (DDG)

Chief Directorate: Strategic Support Services

Directorate: Policy and Planning

Sub-directorate:

Monitoring & Evaluation


Figure 3: Department of Arts & Culture, M& unit.

2.1.4. Community Safety and Liaison

There is no internal M&E in this Department. Its M&E unit is externally monitoring the provincial

police service. However, there is a need and there is plan to establish an internal M&E in April

2010. The M&E unit revises and refines its external monitoring tool every year. Accordingly it has

widened its scope to monitor procurement and budget expenditure, specialized units (like dog

units), and service excellence award to encourage police. To operationalise the excellence award

the Unit involves community structures and do survey once a year and rewards

stations/individuals/community service centers based on merit. This is a good incentive to

inculcate the culture of compliance and has to be continued on a wider scale. The M&E unit is

strategically positioned as a chief directorate which maximizes information usage, though it is

monitoring an external organization.

HEAD of DEPARTMENT: ARTS & CULTURE

Chief directorate: Corporate Governance

Directorate Corporate Strategy

Sub-directorate Policy and

Planning Sub-directorate Performance

Management (M&E ) Sub-directorate Intergovernmental Relations


Figure 4: Department of Community Safety & Liaison, M& unit.

2.1.5. Economic Development & Tourism

The M&E unit was established in 2006 but it is still staffed by one person. The unit is organized

under the Strategic Planning and Monitoring & Evaluation Management which ensures usage of its

reports and involvement in the planning process. The Unit has been under continuous process of

review and improvement and has produced a number of evaluation reports that is lacking in most

of the M&E units in the province. Its previous M&E system did not have clear outcome indicators.

The new system called MBP (managing by projects) integrates finance and human resources and is

able to track finances, and staff performance. In 2007 the Unit developed an M&E framework and

was approved by management. The Directorate is planning to establish MIS to be positioned in

the Knowledge Management unit which is under the same directorate. Though the Unit is still

monitoring project level activities, it seems to be under way to institutionalize M&E functions. It

has gained inroads to political buy-in and has positioned itself to create demands for its outputs.

The M&E and planning units are managed together under one Chief Directorate. This positioning

enables usage of the reports of the M&E unit for planning purposes.

The Unit has produced several evaluation reports.

HEAD of DEPARTMENT: COMMUNITY SAFETY &

LIAISON

Chief Directorate: M&E

Directorate: Service Delivery Evaluation

Directorate: Complaints Management


Figure 5: Department Economic Development & Tourism, M& unit.

2.1.6. Education

The M&E unit of this department was ‘established’ in 2007 but did not feature in the

organizational structure until 2008. It is staffed by 2 persons but need four more. The M&E and

Planning units fall under separate directorates which may not be conducive for coordination.

There are directorates of Quality assurance, M&E, Strategic Planning, EMIS that report to different

Chief Directorates. However, these directorates are closely interlinked and would be better if they

come under one chief directorate for ease of information sharing and usage. The M&E unit does

not track the school level performance, suggesting that it does not have any information about

learner and teacher attendance, for example. This means important information that has impact

on learning is missing from the M&E system. Such information is sent to the Education

Management Information Systems (EMIS) that is managed by a different directorate where the

M&E unit does not have access. The EMIS collects information twice a year and the M&E unit does

general annual survey. There is another web-based system called SASAMS (South African Schools

and Administration Management System) developed by the National Department of Education

that tracks students’ progress from entering to leaving the school system. The EMIS is making use

of it now. This system was rolled out in 2008 and 2500 out of 6000 schools in KZN are linked to the

system. It has modules that can track such as school finances, assets, and attendances. The

challenges of not rolling out throughout the province are electricity and connectivity problems.

HEAD of DEPARTMENT: ECONOMIC DEVELOPMENT & TOURISM

Chief Directorate: Economic Planning Program

Directorate M&E Policy Planning R&D Knowledge Management


This Unit has a plan to computerize the logical framework as a move to automate its M&E

functions.

Figure 6: Department Education, M& unit.

2.1.7. Health

The M&E unit in the Department of Health was created in 2007 and has got 3 members. M&E is

strategically positioned and directly linked to planning; they are both under one Chief Directorate

Figure 7: Department of Health, M& unit.

HEAD of DEPARTMENT: EDUCATION

Branch:

Planning and Support Services

Chief Directorate Planning

Branch:

Human Resources and Administrative Services

Chief Directorate Administrative Services

Directorate Monitoring & Evaluation

HEAD of DEPARTMENT: HEALTH

Chief Diretorate: Health Service Planning and Monitoring & Evaluation

Diretorate: Health system Monitoring & Evaluation


2.1.8. Human Settlements

The M&E unit for Department of Human Settlements was established in 2002 but does not have

any consistent M&E functions as the staff that was assigned to it was constantly relocated. The

M&E is a directorate position and no director has been appointed to date. A permanent staff in

position of deputy manager was appointed at the beginning of 2009 but the manager has other

tasks not related to M& functions. This makes it difficult for the Department’s monitoring unit to

assess the quality, quantity, cost and delivery time of new homes. The M&E unit is organized

within the office of the Head of Department and all the housing information system is linked to GIS.

Figure 8: Department of Human Settlements, M& unit.

2.1.9. Local Government & Traditional Affairs

The M&E unit came into existence in 2007 and is staffed by two persons that have clear job

descriptions. Their electronic system is Excel-based and all of their business units report in one

format. The system was piloted for two years and functions as an early warning system that

shows status of progress against targets. The Department is happy with the current system and it

is not looking for other sophisticated systems for the near future. To push down the full operation

of the system, instability of upper management is also a challenge, though the middle

management is relatively stable. There is also problem of staff retention especially for the younger

generation, the moment they start to gain experience they leave the department The M&E unit is

strategically positioned under the chief directorate, and it is headed by a director. However, the

HEAD of DEPARTMENT: HUMAN SETTLEMENTS

Directorate: Monitoring & Evaluation

Directorate: Planning


planning unit falls under a different chief directorate though it reports to the same deputy director

general.

Figure 9: Department of Local government & Traditional Affairs, M& unit.

2.1.10. Public Works

The M&E unit in the Department of Works in only in name and has no staff. It has been created

recently and placed under the Directorate of Strategic Analysis (Planning) The Director of Strategic

Analysis (Planning) is temporarily in charge of M&E activities. There is a departmental policy of

performance management information that was approved in 2009. There is a data capturing tool

called Works Information System (WIMS) that is only used in projects.

Figure 10: Department of Public Works, M& unit.

HEAD OF DEPARTMENT: LOCAL GOVERNMENT & TRADITIONAL AFFAIRS

Deputy Director General: Chief of Operations

Chief Directorate: Monitoring & Evaluation

Directorate: Monitoring Directorate: Evaluation Directorate: Systems and Reporting

Chief Directorate: Corporate Services

Directorate: Strategic Planning

HEAD of DEPARTEMNT: PUBLIC WORKS

Directorate: Strategic Analysis (Planning)

Monitoring & Evaluation (Vacant)

Chief Directorate: CFO

Chief Directorate: Corporate Services


2.1.11. Social Development

Although established in 2002, the M&E unit only became functional during the 2008-2009

financial year when it started receiving funding. A draft policy on Management of Performance

Information (financial and nonfinancial data) was developed and the document provides details on

management of source documents, collation of information, verification of information, and

submission dates. The unit is staffed by one person instead of the planned 27 at full operation. The

M&E unit is structured as a directorate reporting to the Chief Financial Officer (CFO).

Figure 11: Department of Social Development, M& unit.

2.1.12. Sports & Recreation

The M&E unit has only one member. M&E is still in its infant stage and the general awareness of

its functions is low. There is general misconception and no clear purpose of M&E. The current

M&E system is manual but it is developing an electronic system. The M&E unit’s position is low in

the hierarchy as a sub-directorate under the directorate of strategic management support.

HEAD of DEPARTMENT:

SOCIAL DEVELOPMENT

Deputy Director General: Chief Financial Officer

Directorate: Monitoring & Evaluation Chief Directorate: Financial Management

Directorate: Planning


Figure 12: Sports & Recreation, M& unit.

2.1.13. Transport

The M&E unit in the Department of Transport was the first of its kind in the province. It was

created in August 2000 and is a relatively well a organized unit, to the extent that all department

planning cannot be done without the unit’s input. The M&E unit has gone beyond the QPR’s

requirement and developed its own template of data collection and reporting called Annexure 6.

The Annexure is a basic Excel system that captures comprehensive and detailed information on

plans, activities, expenditure and outputs. The tool is able to generate routine monthly and

quarterly expenditures and output variance reports. The M&E unit does site visits to monitoring of

projects, and data quality audits. However, due to staff limitation such supervision is not optimum

at all. To partially address the shortcoming the M&E unit endeavors to train contractors on how to

complete Annexure 6.

HEAD of DEPARTEMNT: SPORTS & RECREATION

Rogramme 2

Chief Direcotrate: Corporate Governance

Directorate: Strategic Management Support

Sub-directorate: Monitoring & Evaluation

Planning & Development

Transformation & Service Delivery

Research & Innovation


Figure 13: Transport, M& unit.

2.1.14. Treasury

Treasury does not have an M&E unit for its internal functions. It does perform a monitoring

function over the provincial departments from an expenditure perspective through its Resource

Management Unit. Treasury’s M&E function over the provincial departments is not coordinated as

it is done separately through its various units like Budgeting and Supply Chain. For reasons of

security, there is no information sharing among these units and reports are submitted separately

to National Treasury and Provincial Government. The provincial Treasury’s monitoring mandate

over the province is unclear or limited. For example, the supply chain in the municipalities is not

monitored by the provincial Treasury as it is not its mandate. The QPR in municipalities also goes

directly to the National Treasury.

HEAD of DEPARTMENT: TRANSPORT

Chief Directorate: Strategic Planning and Monitoring

Directorate: Monitoring & Evaluation

Monitoring Evaluation Service delivery Improvement

Programme


2.1.15. Sisonke District Municipality

Although the municipality is aware of the urgent need to set up a functional M&E unit, none exits

due to budget constraints. Currently ‘M&E’ functions are performed through the Integrated

Development Plan and Performance Management System (IDP & PMS) unit that is staffed by one

person and uses a reporting format provided by the Department of Local Government.

Despite the requirement managers are reluctant to submit quarterly performance reports, and

apparently submit them on demand. The absence of M&E policy contributes to the irregularity of

reporting obligations and lack of information sharing among the departments in the district. The

Performance Management unit (that has functions similar to M&E) is not strategically positioned

to monitor all the activities of the District Municipality.

Figure 14: Sisonke, District Municipality, M& unit.

2.1.16. Msunduzi Local Municipality

This municipality also does not have M&E unit and the M&E function is more or less done by the

Performance Management unit. This unit was established in December 2003 and has 3 members.

The unit has been reorganized and now falls under Organizational Development and Research &

Analysis. However, the members of the Performance Management unit are still administered by

their previous head that has nothing to do with performance management. A major challenge is

the failure of managers to submit non-financial data routinely.

MUNICIPAL MANAGER

Executive Director

Manager: Operations

IDP & PMS (Performance Management

System)

Internal Audit Special programs IGR Communications


Figure 15: Msunduz, Local municipality, M& unit.

2.1.17. Public Service Training Academy

In order to mitigate the challenges of growth and development the provincial government of KZN

has instituted a Public Services Training. The academy is mandated to embark on a mission of

public service renewal by producing competent public servants that deal effectively with the

challenges facing the province including backlogs in basic services, poverty, protection of

vulnerable groups, human development, unemployment, economic performance, the natural

environment, and HIV/AIDS.

As an institutional response mechanism to the capacity challenges, the academy aims at ensuring

a coordinated and structured approach to human resource development in the province. It is a

dedicated unit in the Office of the Premier, whose exclusive function is to provide specialized

training and organizational development service to the officials of the KwaZulu-Natal Provincial

Administration. Its main function include to facilitate, develop and implement skills development

policy and program; to facilitate generic, functional and executive management development

programs; to facilitate and coordinate basic literacy and numeracy within the provincial

administration; to provide curriculum development and quality assurance, and to provide

leadership, guidance and support on skills development interventions.

As provincial trainings are organized by the Office of the Premier, PALAMA, etc. the Academy acts

as a coordinating unit in the province (fixing dates, venue, nomination, etc). As PALAMA’s trainings

have the syndrome of being generic the Academy has the responsibility of contextualizing courses.

MUNICIPAL MANAGER

Research & Analysis

Performance Management


The Public Service Training Academy has an M&E unit called Compliance, Monitoring and Impact

Assessment. Its functions include to undertake skills planning, compile and maintain a provincial

skills plan/strategy, monitor implementation of learnership and internship, facilitate and

coordinate reporting, assess the impact of human resources development (HRD) interventions.

It is a very new unit created in 2008 and is staffed by one person against the required 3. The unit

has to still develop policy guidelines for its operations.

Like most departments the M&E unit of the Academy has challenges of verifying data quality; for

example it cannot validate the training needs done by departments. The main challenges

identified in the academy for the provision of M&E training include: funding, lack of cost recovery

mechanisms, curriculum and materials development, lack of in-house M&E experts, and electronic

systems.

Figure 16: Public Service training Academy, M& unit.

For more information on intervention to capacitate the Academy see Section 5.

SENIOR GENERAL MANAGER (HEAD of the TRAINING ACADEMY)

Directorate: Skills Planning, Materials Development & Quality Assurance

Sub-directorate: Skills Planning


Mateirals Development

Training and developemnt


3. OVERALL ASSESSMENT FINDINGS

The assessment was adapted from the UNDPs framework of capacity assessment and focused on

four dimensions: Institutional arrangements and environmental situations, M&E advocacy, M&E

knowledge and skills, and Reporting mechanisms.

3.1. Institutional arrangements for monitoring and evaluation

3.1.1. Legal framework, policies and procedures to implement an M&E systems

This component of the assessment involved asking interviewees about the current monitoring and

evaluation public sector environment in which provincial departments and municipalities operate

and whether such an environment promotes M&E. The most significant aspect of the assessment

that was reviewed included adequacy, availability and implementation of current government and

or provincial policies and reforms that influence monitoring and evaluation implementation in the

province. This component further looked at adequacy of roles and responsibilities of individuals to

manage the M&E system, level of interdepartmental coordination, availability of assistance in

capacity development, and mechanisms to engage stakeholders on M&E.

In the whole, most interviewees agreed that the national legal frameworks and policies are

adequate to support M&E functions in the province. Many of them cited the Constitution (1996),

Public Finance Management Amendment Act (No 29 of 1999); the Municipal, Finance

Management Act (2003); Public Service Act (1994 as amended); Cabinet’s 2004 decision to initiate

M&E plans; Batho Pele Principle; Framework for Managing Program Performance Information14

(2007). Treasury Guidelines15 (Preparation of Expenditure Estimates for the 2009 Medium Term

Expenditure Framework), and the presidency’s development of the Government-wide Monitoring

and Evaluation System. Departments and municipalities expressed awareness of the current Green

Paper on National Strategic Planning (2009) and the discussion paper on Presidency’s approach to

Performance Management Monitoring and Evaluation (2009), and they believe these documents

can further consolidate and further contribute to strengthening their M&E activities. Interviewees

in the whole agreed that most of the new M&E policies and reforms were synergistic and had

huge potential to institutionalize M&E functions in the public sector.


In terms of institutionalizing the reforms, some of the provincial departments have used the

mandate given to provincial departments and municipalities to develop M&E systems and

frameworks and created M&E units. The typical example is the Office of the Premier (OTP) that

created the provincial Nerve centre, an automated and integrated information management

system that monitors and evaluates the performance of KZN provincial government against set

goals, targets, equity in resource allocation, effectiveness and efficiency in service delivery across

all relevant levels of spheres of government. At the time of the review 4 of the 14 departments

and municipalities interviewed had an M&E framework and 12 had an M&E unit.

However, 60% of the departments perceive the current M&E public sectors policies as

inadequately implemented mainly due to the poor appreciation of the concept of performance

monitoring management and low staff levels as well as lack of adequate integrated M&E systems

to track program inputs, outputs, outcomes, finances and data quality. Interviewees expressed a

failure of departments to operationalise the policy frameworks and there is a general lack of

readiness to implement M&E functions. A number of the departments perceived the national

guidelines as too broad and difficult to be directly implemented and that provincial and

municipality structures must adapt them to develop appropriate M&E frameworks and

implementation plans.

Another challenge identified was the general lack of understanding and acceptance of a results-

based management as interviewees indicated that many provincial staff including at management

level view performance monitoring only as a “policing” process. This leads to the general behavior

that results-based or performance monitoring means reporting for the sake of compliance and not

for continuous improvement or using data for decision making. A typical example of this situation

was the observation by the Treasury department of inadequate (10% completion) or late

completion by departments of non-financial indicators in the Quarterly Performance Reports (QPR)

from departments. This can also be attributed to the perceived greater importance of financial

data and a general lack of appreciation of outcome and quality measures by departments.


Furthermore, there are some organizational impediments characterized by absence of an

inadequately positioned M&E unit in provincial department organograms or inadequate

recognition of the role for M&E even when the unit is available. In three cases however, provincial

departments clearly recognized the role of M&E and have or are in the process of integrating M&E

unit into the planning unit to create one directorate.

In terms of institutionalization of evaluation very few departments have performed outcome or

impact evaluations in part fulfillment of the mandate to determine extent to which department or

municipality program and service objectives have been fulfilled. In addition none of the

departments interviewed had adopted the SASQAF to monitor quality of information generated at

the different service levels.

3.1.2. Overall system used to perform M&E tasks

Of the municipalities and departments interviewed 80% indicated that the current overall system

to support M&E functions was not adequate or not adequate at all. Interviewees indicated that

the current department systems did not have adequate systems to track inputs (finances and

human resources), outputs, outcomes, impacts and data quality simultaneously. A key gap in the

M&E systems is the lack of baseline data for most departments and it therefore makes it difficult

to determine progress towards goals and objectives. Two departments with relatively good

practices are Departments of Transport, and Local Government & Traditional Affairs. In both cases,

departments are able to simultaneously track inputs, outputs and progress (alerts) towards

targets. Even then, in both departments the M&E process does not include institutionalized data

quality audit system and as such no reports on data quality are routinely generated.

3.1.3. Roles and responsibilities to manage and implement M&E system

Only 43% of the departments and municipalities interviewed indicated that M&E roles and

responsibilities were adequate. The rest (57%) indicated that inadequacy in roles and

responsibilities was mainly due to failure to recognize role of M&E, failure to recruit quality staff

and failure to formalize job descriptions and roles of M&E personnel. In most cases this has led to

overlap and exhaustion of personnel and failure to focus on stewardship of M&E units by the few


personnel. M&E personnel are therefore underutilized leading to missed opportunity for input

into key provincial and municipal management areas like planning and resource allocation.

3.1.4. Level of coordination on management and implementation of M&E system

Only two departments indicated adequacy in inter-sectoral, interdepartmental and stakeholder

coordination on M&E. The majority agreed that departments were working in silos and the

opportunity to share and use inter-related information has not been adequately utilized. For

example both the Departments of Arts & Culture, and Economic Development & Tourism train

craftsmen, artists, and musicians but do not share a common database on such trainees; no

sufficient information flows from the Department of Education, and Department of Health to the

Department of Arts and Culture about the libraries in schools and hospitals; and the Department

of Education does not have real time information on private schools.

The presence of the Provincial Nerve Centre (NC) is an opportunity for inter-departmental and

inter-governmental collaboration and utilization of information. However access to the NC is

limited to MECs and Heads of Departments. Although a provincial M&E Forum is present for

networking and training of departments about the opportunity of sharing and utilization of M&E

data it has not been adequately supported.

3.1.5. Donors and other external stakeholder involvement in M&E capacity building

The majority of interviewees (86%) indicated that donor and other external support for

development of M&E systems has been inadequate. External support has been focused on

compliance and M&E primer trainings with limited technical support that leads to generation of

actual systems and process for M&E. For example, recent trainings by the World Bank and

Regenesys for the Departments of Social Development, and Agriculture & Environmental Affairs

were generic M&E primer workshops; two days training by Statistics South Africa to the

Department of Education about research methodology and statistics was not sufficient in

providing sustainable skills on evaluation; memorandum of understanding between the

Department of Local Government & Traditional Affairs for technical support by National Treasury


for M&E capacity development seems to emphasize compliance to reporting and not using data

for quality improvement.

3.1.6. Mechanisms to engage civil society, public and the private sector in M&E

Departmental and municipal respondents were asked to assess adequacy of the scope and quality

of mechanisms to engage civil society, the public and private sector, and the majority (73%)

indicated that mechanisms were inadequate. For example, respondents indicated that use of

community satisfaction surveys was minimal and that involvement of community in the

monitoring of provincial and municipality performance was inadequate or inexistent in most cases.

In the latter case this was attributed to inadequate knowledge about performance monitoring by

civil society and the community. Examples where such coordination and involvement have been

successful include collaboration between the Department of Community Safety & Liaison and

Business Against Prime, Community Police Forum, Portfolio Committee, and the Police Service in

the design of its M&E activities; Department of Agriculture & Environmental Affairs in the District

Agri-Forum for community participation; Department of Transport in the Rural Road Transport

Forum (RRTF) composed of the youth, women, church, commerce, etc to supervise commissioned

projects and ensure community voice is heard; the Public Service Training Academy also evaluates

its own services at the end of training session and report back to the concerned departments,

Office of the Premier, and to the Governance and Administration Cluster.

3.2. Monitoring and evaluation advocacy

This section assessed leadership influence (demand) on the development of an M&E system and

the extent to which the information generated is used in planning and budgeting. The assessment

would provide an inference into the provincial budget systems and how they link to the

performance monitoring systems. The ideal is that a public sector resource allocation system to be

effective requires full information on the problems and needs of the community, the best

interventions (policies) and their alternatives to solve or alleviate the problems and previous

experiences on implementation of interventions. In the case of South Africa, relevant provincial

and municipal civil servants would then use such information to provide policy, planning and

budgeting advice. Leadership and management structures would need to have the capacity to use


and promote or advocate for use of performance monitoring tools to generate data for decision

making and also have the capacity to query the quality of performance data. The overall finding

was that leadership advocacy for M&E was not adequate (71%) even when it was present.

3.2.1. Leadership influence on development of an M&E system

Until recently, most respondents indicate there was little advocacy for M&E systems in

departments and municipalities. Most recently, advocacy (pressure from MECs and Heads of

departments) for M&E across departments has been due to the recent emphasis on M&E by the

presidency. In some departments, department heads and managers have gone ahead to elevate

the role of M&E and have restructured departments to strengthen the role of M&E. For example,

the Department of Social Development has or is in the process of consolidating the M&E unit into

the Strategic Planning directorate. Support for M&E can also be seen through lobbying for training

of staff and allocation of a budget for M&E. Even with these developments, 64% (9) of the

departments and municipalities interviewed indicated that leadership influence on development

and use of M&E systems was inadequate. Interviewees indicated that signs of inadequate support

from leadership include failure to enforce timely submission of activity reports by project

managers and auditing of reports. Interviewees indicated that budget support for M&E was still

quite inadequate especially when you consider that the general recommendation for M&E budget

support is 5-10% of overall budget. For example, although the budgets for Departments of Social

Development and Health M&E units were the largest in absolute terms (R 13 million and R 19

million respectively), the budgets were actually 1% of the overall departmental budgets.

3.2.2. Use of M&E information for planning, resource allocation and policy making

Although there is general increase in M&E advocacy by leadership, 73% of interviewees indicated

that there is inadequate demand and usage of M&E reports. Interviewees suggested the main

reason is the lack of trust by both civil servants and political leaders of information generated by

the M&E systems as data is viewed as incomplete (no outcome indicators), inaccurate and un-

verifiable.


3.2.3. Management feedback on quality of M&E system

Interviewees (79%) indicated that management feedback on quality of M&E reports was

inadequate even when some feedback is given. Interviewees attributed this to inadequate

readiness by management to objectively audit data quality. In fact none of the departments had

institutionalized internal data quality monitoring as part of their M&E systems they used.

3.2.4. Management capacity to perform and use inferential analyses on M&E data

Up to 67% of interviewees perceived management capacity to perform and use of inferential

analyses as inexistent, inadequate even when present or not adequate at all. To quote one of the

interviewees “…during statistical presentations 90% of the audience gets lost…” A typical example

of this situation was described by the department of Community Safety & Liaison where the

interviewee explained that the department changed from mere submission of report to include

presentation to the police management because the latter seemed not to understand them.

Interviewees observed that there is a tendency for management to rely on narrative reports and

ignore statistical data that would otherwise be used to make generalizations of data on service

delivery.

3.3. Monitoring and evaluation knowledge and skills

In this section the assessment reviewed in more detail the supply of M&E in the form of capacity

of infrastructure available to generate M&E data. The capability to supply M&E depends on skills

of personnel, structures and processes used to generate targeted type of good quality M&E data.

In general 72% of respondent departments and municipalities perceive the current information

infrastructure as inexistent, inadequate even when present or not adequate at all.

3.3.1. Capacity of the M&E system to generate and disseminate good quality data

87% of interviewed departments and municipalities indicated that available M&E systems were

inadequate even when present or not adequate at all. Respondents maintained that current

systems do have clear documented processes for data collection, collation, analysis, reporting and

use. However, most departments did not indicate that lower level service points have standard


data collection tools commissioned from the top. In addition there are limited personnel with skills

to manage the data management processes. The majority of interviewees, for example, indicated

that they had limited skills in statistical analyses, program evaluation, cost-benefit analysis and

data quality audits. This situation increases the likelihood of generating inadequate performance

quantity and quality of data for use by provincial departments and thus compromising quality of

policy and planning decisions and yet the province has no capacity to monitor quality of data.

3.3.2. Electronic system to collate data for the M&E system

This component of the assessment was performed to determine the availability and quality of data

collation tools (databases) used by departments and municipalities including interoperability and

harmonization of tools in the event of need to share data across stakeholders. In addition

interviewees commented on capacity of staff to use such collation tools. In the whole most

departments use simple Microsoft Excel collation tools. 73% of the departments and

municipalities indicated that good quality relational databases were inexistent, inadequate even

when present or not adequate at all. A key weakness of available databases is that they are non-

relational Excel based databases with minimal or no option to perform more sophisticated

analyses including linking all performance data from program goals, objectives, inputs (finances),

outputs and outcomes. Despite Excel’s capability for sophisticated operations most of the

departments are using it as a ‘manual’ system, not more than storing data. Although most

departments indicated knowledge about usefulness of GIS, there is very minimal use of this tool by

departments as value addition to their M&E systems. Another challenge is that collation tools are

available at the provincial and municipal level yet none are available or utilized at service point

levels even when available. For example, the Department of Health has the District Health

Information System (DHIS) which is adaptable to health facilities but few of them actually use it to

collate their data. Some examples of databases in place or planned by departments include:

The Departments of Local Government &Traditional Affairs has designed a web-based

electronic system (Excel) that is approved by SITA and is ready for development. It is based

on Logical Framework and is user-friendly.


Department of Transport is comprehensively using Excel to develop reporting templates. It

also uses GIS to extract information from maps.

The Department of Agriculture & Environmental Affairs has a system called Knowledge

Bank that is used by the national Department of Agriculture Forestry & Fisheries.

The Department of Community Safety & Liaison has an internet based “Electronic

Complaints System” developed in 2006 for lodging community complaints and is linked to

cell phones. After a complaint is lodged the system generates reference number, and

acknowledgement letters, and faxes back automatically to the complainant and keeps

reminding both sides on status of follow up.

The Department of Education has a Management Information System (EMIS) that collects

information to support planning, monitoring and decision-making in the department. The

EMIS collects aggregate survey data twice a year from each school.

The provincial departments are still using PERSAL (Personnel and Salary Administration

System) and BAS (Basic Accounting System) both which have limitations and are outdated16

17 18.

3.3.3. Availability and quality of recent Monitoring and evaluation reports

Although most departments indicated that they generate great “glossy” reports most interviewees

indicated that these reports may not stand rigorous data quality audits. In addition, most

departments have no culture of performing program evaluations or cost benefit analyses. It is in

this light that the majority of interviewees (80%) indicated that such recent M&E reports are

either not adequate or do not exist at all.

3.3.4. Availability of M&E system to generate data on the MDG indicators

Although most interviewees indicated awareness about MDGs and their integration into provincial

development plan, 80% of them indicated that the M&E system was not adequately aligned to

accurately collect such data. Interviewees argued that this was mainly because MDGs were not

adequately adapted to the South African provincial setting and the indicator definitions were

unclear and no specific guidelines have been provided on how stakeholders can measure their

contribution to the MDGs. For example, one interviewee complained that “…we cannot measure


global partnership for development, which is completely out of our domain...” A key requirement

for using MDG data is a baseline and most interviewees indicated that departments and

municipalities do not have such information. Even when such information is available, it is not

readily available to all stakeholders. For example, MDG information in the Department of

Education is kept in the Education Management Information System (EMIS) where the M&E unit

does not have real-time access.

3.4. Reporting mechanisms

This aspect of the assessment reviewed adequacy of provincial M&E systems to generate reports

that are usable the extent that they generate information on targeted beneficiaries and more

importantly that ability of the systems to generate organizational and staff performance reports.

In general, the majority of interviewee (72%) perceived the current system as present but not

adequate, not adequate at all or inexistent.

3.4.1. Existence of an effective reporting mechanisms

Although 40% of the departments and municipalities indicated that reporting mechanisms were

effective in the sense that reporting templates and guidelines at respective sub-reporting levels on

what, how, whom, when to report were generally followed, written guidelines were not

necessarily available. In addition all interviewees indicated a lack of written guidelines to track and

feedback quality of reports received from different sub-reporting levels including what actions to

take in the event of data quality queries. In fact, almost all interviewees indicated a lack of

capacity to perform such data quality checks on submitted reports. None of the departments had

written data retention policies on source documents. Though all departments have standardized

reporting format (Treasury’s QPR) the Provincial Treasury does not have the power on addressing

reporting issues and enforcing compliance.

3.4.2. Adequacy of monitoring system to generate disaggregated data reports

Departments were assessed on the level of disaggregation of their data (gender, age-group,

location, socio-economic status, education, etc.). Most of the departments (67%) felt they do not


have an adequate level of disaggregated data in their reports and they attributed this to poor

design of source documents.

3.4.3. Availability of mechanisms to disseminate M&E reports to stakeholders

Interviewees were assessed for perceived adequacy of mechanisms to engage stakeholders on

M&E issues and the majority (80%) indicated such mechanisms, if present, were inadequate or not

adequate at all. Interviewees argued that high level reports generated at departmental and

municipal levels are often not simplified for the community and that the lack of a community M&E

feedback process of engagement and participation on provincial performance was needed. For

example, the Department of Community Safety & Liaison uses community-based volunteers

(Trained by the Manchester police) to collect information that is fed into the department’s M&E

system. The Department also assesses police performance through community forums.

3.4.4. Availability of mechanism to link M&E data and performance

The majority of interviewees (80%) indicated that M&E system was currently inadequate in linking

programme data to staff performance. The main reason given for this not being done is that staff

performance contracts are not linked to program activities and deliverables. In addition there is

inadequate understanding of the relationship between programme targets and individual staff

performance and lack of willingness to collaborate between human resource and M&E units. In

one department, interviewees indicated that the M&E system is able to generate performance

reports based on deliverable achieved by staff but there is no incentive to disseminate such

reports to human resource units or departmental managers.


4. RECOMMENDATIONS ON M&E CAPACITY BUILDING

There is broad agreement among experts that to successfully strengthen public sector M&E

systems a number of factors have to be in place. Below are recommendations that the province

may implement and these could be the backbone of an action plan to improve M&E systems in the

province.

4.1. Substantially increase demand for M&E by provincial governing structures

Any efforts to develop an M&E system must be intentionally created and should be based on real

demand and powerful incentives not just political statements and M&E policies. The biggest

barrier to demand is lack of knowledge about the role of M&E at the key stakeholder level where

buy-in is crucial. As this rapid assessment has shown, a number of departments expressed lack of

knowledge at both political and management levels. Even with government push for

institutionalizing M&E through the GWM&F, a number of departments are yet to develop M&E

frameworks mainly due to lack of experience and incentives to actually develop M&E systems.

To increase demand a key strategy would therefore entail increasing knowledge on usefulness of

M&E at the key stakeholder level. Such awareness should involve hands on exposure to the tools

and methods of M&E and how they are used including practical examples of public sector

institutions that have developed and used such systems to great value. In the case of KZN

departments, department stakeholders should be exposed to peer-departments with good M&E

systems within the province as well across the country. Such encounters would go a long way in

persuading department and municipality stakeholders about the value of a good M&E system. By

securing buy-in of the key stakeholders such as key departments (e.g. Treasury, Local

Government) the latter would then offer the substantive effort to create, strengthen and fund

M&E systems.

4.2. Elevate awareness of incentives for M&E across the province

Governments build M&E systems not for the sake of it but rather to use the information

generated to support budgeting, to develop plans and policies, and to promote accountability.


This is supposed to be the inherent merit for institutionalizing an M&E system and it is these

incentives that may drive demand for institutionalizing M&E. Often (as in the case of a number of

departments interviewed) the perception is that M&E is conducted primarily for accountability

purposes thus narrowing its focus. The government of South Africa has developed the GWM&F

and the core PPIF as tools to guide provinces and municipalities in developing performance based

budgeting processes. For example, in one of the departments, because of the lack of trust in its

M&E system the department continues to use an incremental budgeting system as data is

considered unreliable to force significant resource reallocation. Staff only feel obliged to report

whatever quality of data is available because they believe that “after all the budget will not

change.” There are several strategies that can be implemented by the province to increase

incentives for having good M&E systems. The strategies may take the form of:

Encouragement and rewards for conducting M&E and utilizing the findings e.g. provide

additional funding for M&E, rewards and high level recognition for departments with

best M&E practices, use of M&E experience for staff recruitment, promotion and

certification, provide free training for managers and staff on M&E to improve service

delivery and performance, provide budget-related incentives for departments to

improve performance, develop and enforce performance contracts for ALL staff.

Prompts or penalties for failing to conducting M&E and utilizing the findings e.g.

withholding part of funding or pay for departments or staff with no M&E systems or

poor performance respectively, create formal requirement for planning and budgeting

to include M&E, create formal process to report poor quality M&E systems and

adverse M&E data; develop formal system to involve civil society in review of

provincial and municipal performance including publication of poor performance,

provision of regular feedback to all stakeholders including data management teams at

all levels.

Advocacy and awareness about importance of M&E e.g. awareness raising seminars to

demystify M&E including examples of good M&E systems and their utility; pilot M&E

systems to demonstrate usefulness; high level statements within the province,

departments and municipalities endorsing merits for M&E systems; lobby for support


from donors for M&E support, eestablishment of M&E forums and networks to share

experiences, best practices and discuss strategies to resolve M&E challenges.

4.3. Perform detailed diagnosis of department and municipality M&E systems

The purpose of this diagnosis is to identify in detail how key information is collected, collated,

analysed, reported and used. In the later case, the extent of actual utilisation of data from the

M&E system would be reported. In the whole this process would help to identify specific

functional areas of the M&E system across departments and municipalities that require

strengthening or need new systems. The rapid assessment performed was able to identify some

weaknesses of departmental M&E systems. For example, almost all departments have no

documented or formal data quality control procedures to ensure that information generated from

service delivery points is measured for quality.

Good quality data is in general an intrinsic character of data and most departments will have to

improve on this component of their M&E systems. A key activity for departments and

municipalities would therefore involve performing baseline data quality audits on key

performance indicators. The audit would then be used to diagnose overall strengths and

weaknesses of departmental M&E systems and decisions can be made on developing new systems

or strengthening new ones and M&E human resource needs (including training needs) to

implement M&E systems from the service delivery point to the highest level of management.

4.4. Re-arrange organizational structures to strengthen the role of M&E

As stewards of performance monitoring department and municipal M&E units are mandated to

generate objective and accurate information to be used by the province and municipalities to

inform policy and planning. However, as highlighted earlier, demand for using M&E data is poor

and in fact M&E units in a number of departments do not have a prominent role in the

departmental organograms related to planning, financing and human resource. This is a paradox

as the objective of having an M&E system (M&E unit) is to generate information that is utilised by


the core provincial and municipal government functions i.e. policy development, planning,

budgeting and resource allocation.

In order to increase utilisation of data and to increase relevance of M&E, reorganisation of the

departmental hierarchy to bring the M&E units closer to the decision making units in the

departments would be useful. The province could consider some options of re-positioning the

M&E unit. For example, one option would be to locate the M&E, planning, finance, and human

resources units within the Office of the Head of Department at the level of chief directorate (See

Figure 17). Because of their dynamic interactions this strategic position enables them to oversee

the entire activities of all units and streamline data flow and strengthen horizontal relationships

between departments.

Another option would be that M&E and planning units must not go lower than the chief

directorate level and directly report to the Branch or Head of Department (while being outside the

Office of the Head of Department), depending on presence/(absence) of branches (See Figures 18

and 19). The twinning of human resources with M&E helps to easily integrate individual

performance with the overall organizational performance. The status quo in all departments is

that individual performance information is not flowing to M&E units.

Figure 17: M&E unit restructuring – Option 1

HEAD of DEPARTMENT

Programme 1 Programme 2 Programme 2

Chief Directorate: Montitoring & Evaluation

Chief Directorate: Planning

Chief Directorate: Finance Chief Directorate Human Resources


Figure 18: M&E unit restructuring – Option 2a.

Figure 19: M&E unit restructuring – Option 2b.

4.5. Increase quantity and quality of M&E supply

In order to perform well an M&E system needs to have highly skilled M&E personnel. Therefore

most M&E capacity-building needs to include provision of training in a range of M&E tools,

methods, approaches, and concepts. However, this should not be the emphasis of an M&E

capacity improvement response. Based on the findings of the assessment, it was clear that most

departments were understaffed and key departmental and municipal M&E activities like data

quality monitoring and evaluations were hardly done. Even the available M&E unit personnel are

not well equipped to perform such tasks. Although interviewees indicated that they had received

HEAD of DEPARTMENT

Chief Directorate: Monitoring & Evaluation

Chief Directorate: Planning Chief Directorate: Finance Human Resources

HEAD of DEPARTMENT

Branch 1

Chief Directorate:


Chief Directorate: Planning

Chief Directorate: Finance

Chief Direcdtorate: Huamn Resources

Branch 2


some training, most of such training was quite basic and most current skills needs were beyond

just M&E primer training. The preference would be to have results based training on performing

data quality audits, statistical analysis, evaluations and more specifically how to use data to advise

finance, planning and policy making departments. All these trainings should include mandatory

follow up to track significant changes in staff performance and most importantly quality of data

and data utilisation by key stakeholders. Key responses to improve M&E supply should therefore

include:

Perform M&E skills gap analysis from service delivery point to highest level

Increase resource allocation to recruit staff with evaluation and research skills

Develop appropriate job descriptions for M&E

Identify specific courses for staff to train on but not limited to program planning,

developing M&E plans, statistical analysis, program evaluation, performance

budgeting and resource allocation, data quality auditing, economic evaluations and

operational research.

The province has a Training Academy that can be used to strengthen supply of M&E through

customized departmental M&E training. However, to ensure quality training by the Academy, the

following aspects have to be considered.

The academy has to develop or outsource expertise to train on skills as deemed

necessary by departments and municipalities (See above).

All M&E training curricula have to be reviewed and appropriately certified /

accredited.

All trainings must be based on needs assessments and must involve human

resource units within departments and municipalities.

The academy should include pre-post training evaluations to assess outcome and

impact of M&E trainings and provide feedback to departments and municipalities.

The academy should consider applying cost recovery models to complement the

budget from the province.


Ensuring that the training academy’s M&E unit strategically positioned in the hierarchy

of the academy to inform decisions in the institution.

4.6. Strengthen structures and process to support M&E systems

4.6.1. Increase financial resources

In order to have a functional M&E system, adequate financial resources have to be made

available. Funds are used to recruit staff, train staff and more importantly to perform M&E

activities like field trips to collect data, perform on-site technical support, perform data quality

audits, evaluations and any other research that may be deemed necessary to generate

information for planning, policy and resource allocation decisions. The rapid appraisal revealed

that financial support for M&E units was way below national and international standards. It is

generally recommended that a budget of between 5-10% of the global budget be put aside for

M&E. None of the departments met this criterion and in fact the majority of departments

allocated less than 1% of the global budgets to M&E. Departments must be assisted to develop

budgets for M&E plans that are linked to their activities.

4.6.2. Develop reliable data systems

In order to have a reliable data system the following aspects should be considered.

4.6.2.1. Good quality performance indicators:

The challenge has always been for sector departments to collect data on a large number of

performance indicators that end up not being used and actually the data is of poor quality. The

target is to avoid an over engineered system that generates too many indicators. Departments

should have a minimum of clear useful core indicators that best inform planning and budgeting.

In developing performance indicators we have to consider the levels of relevance. For example,

these can be small number of high-level strategic indicators focusing on outputs and outcomes, for

use by senior officials; greater number of operational indicators, focusing on inputs, processes,

and outputs, for mid-level officials; and much larger number of operational indicators focusing on


processes and services, for use by line managers and their staff at facility levels relevant to that

work unit (See Figure 20).

Figure 20: Performance indicators

4.6.2.2. Reliable interoperable information systems

There should be a single reliable and interoperable information system that is coordinated within

and across departments and municipalities. The province should perform an audit of available

systems within and across departments to determine capacity of such systems to provide raw

data. The audit would involve a review of the capacity of the system to store and use data within

and across the departments. For example, the audit would specifically look at the capacity of the

system to link program objectives, activities, outputs and financial expenditure. A few

departments appeared to have developed or are in the process of developing a system with an

integrated approach to M&E data. In making a decision about what data systems to use,

departments have to remember that simple electronic data systems can still do the job of storing

and processing raw data. The Departments of Transport, and Local Government & Traditional

Affairs have Microsoft Excel-based systems that have been successfully used in storing

departmental data and skills transfer is easy to provincial staff. Whatever M&E system is


developed there should be documented standard operating procedures. Such procedures would

clearly define the following:

Definition of indicators (and level of disaggregation) including Millennium

Development Goals (MDG) indicators and how these are localized at service

delivery points

Source of primary data and how it is collected, who collects it and when it is

collected

Description of how primary source of data is collected and submitted to the next

level

Description of how data is aggregated, analyzed and reported to the next level

including how data is captured by an electronic system

A list of the potential data quality challenges and how these are dealt with,

recorded and reported

A sample of standard data collection tools and reports as used at different levels

including instructions on how they are completed

Data retention policy and procedures

A minimum of evaluation questions that should be answered to determine program

success including terms of reference on how to perform such evaluations

As a complement, intra- and inter-departmental/ sectoral coordination should be improved

through sharing of raw data generated from M&E systems. Widely accessible provincial

information sharing is relevant for synergy leading to better planning and implementation of

projects and programs. Information could be shared at planning and M&E forums and would, for

example, involve review of interrelationship and correlations between raw data items across and

departments and sectors. In the interim this level of coordination is a mandate of the Office of the

Premier (OTP) through the M&E Directorate. The M&E directorate has built and implemented a

fully functional electronic M&E system called the Nerve Centre (NC) that provides for a platform

for integration of already existing M&E in provincial departments, district and local municipalities.

The NC has been developed to be compatible with data from all provincial M&E systems. However,


one of the challenges has been, but not limited to, alignment of different sector M&E performance

indicators to the NC requirements.

4.6.2.3. Coordination with civil society and community

The rapid appraisal revealed that there is inadequate coordination and actual involvement of civil

society in M&E. Interviewees indicated that probably the best available mode of communication

with civil society is through published annual plans and reports. However, the perception is that

the detail in such documents is too much and there is very little room for community to engage on

implications of plans and reports. Civil society would play a role in actually monitoring

performance of the province in service delivery and would also help guide province in refocusing

service priorities. A number of reasons put forward included lack of knowledge and skills by civil

society and community to engage on M&E issues and lack of adequate environment to engage

provincial structure on actual performance feedback including review of challenges and reasons

why performance may be poor. In this regard therefore the province could do the following:

Develop framework to involve community and civil society in M&E through

participatory performance assessments.

Hold routine community and civil society sectoral feedback forums in which

province shares performance achievements and short comings and reasons for such

short comings.

Develop simplified versions of provincial plans and reports that are more accessible

to communities and civil society.

Sensitize communities on M&E using participatory M&E techniques.

Routinely perform and publish results on satisfaction surveys; consideration could

be made to avail funds for community driven satisfaction or performance

monitoring surveys.


5. CONCLUSION

5.1. Way forward

Experts agree that there is no best model of what a public sector M&E system should look like.

Similarly, any capacity building cannot be expected to be successful in over a short term. In fact

the most successful public sector M&E systems have taken years to get to where they are. This

does not mean that any M&E capacity building should be slow. Rather the approach should be

purposeful and focused with clear intent to build and institutionalise key components of the M&E

system as quickly as possible while champions are available.

The findings of the assessment generally indicate that the provincial demand and supply of the

M&E is relatively weak and therefore any strategies to institutionalise M&E would be two

pronged. However, the key driver of institutionalising M&E would be strengthening provincial

demand for M&E first rather than the supply. By strengthening demand at political and higher

management level, priority M&E capacity needs would be identified. Political and provincial

management leadership need to have intrinsic demand for M&E and have a clear reason for such

demand. M&E data is typically required to inform resource allocation, accountability, effectiveness

and efficiency and guide line management. Increasing M&E demand at this level would show

explicitly that the province is committed to a culture of M&E. Such commitment would therefore

include improving the quality of available M&E supply and increase utilisation of such resources.

Key strategies to improve quality of M&E supply would for example include promoting

recruitment of highly skilled M&E staff within department with clear mandates and job description

to inform provincial performance-based planning and budgeting, avail funds to implement M&E

activities as well as improve skills of available M&E staff and provide continuous feedback to M&E

service providers about the quality of services provided and current priorities.

A logical framework for a provincial M&E capacity building response is shown in Figure 21 and

provides a simple view of how any response would be expected to lead to improvements in the

province. The logic is that specific outputs (e.g. individuals trained on M&E) would lead to

intermediate outcomes (e.g. increased demand for M&E), and to final outcomes (e.g. utilisation of


M&E information by province). Such outcomes would help lead to final impacts like improved

government performance and service provision. Table 2 provides a results framework of M&E

capacity building framework in Figure 21.

Figure 21: M&E capacity building framework (Adapted from Keith Mackay, How to Build M&E Systems to Support Better Government, 2007 (IEG, World Bank)

19

Concerted activities to strengthen provincial M&E functions are undertaken by the province and donors, through provincial and local seminars on M&E systems; Detailed diagnoses of provincial/sector M&E functions; Audits of data systems; provision of M&E

training—including trainer training, officials, Conduction of evaluation are conducted on pilot/demonstration basis.

The action plan leads to the production of a range of outputs, such as number of officials trained in M&E; harmonized data systems; improved quality and focus of available monitoring indicators; improved quality and range of evaluations completed;

strengthened incentives for departments and municipalities to conduct and use M&E

Formal M&E framework or system is established by the province, leading to the systematic planning, conduct, reporting, and use of monitoring information and evaluation findings.

Strengthened demand for M&E at provincial

administrative & political levels

Strengthened supply of M&E and M&E skills at

provincial administrative & political levels

Strengthened M&E systems at provincial

administrative & political levels

Strengthened demand and supply of M&E & skills in civil society

M&E used for: provincial and local government decision making on policies, plans, and budget resource allocation; implementation and management of provincial and local government activities; monitoring of activities,

accounting of expenditures, evaluation of programs and projects; provincial and local government analysis and policy review; provincial and local government accountability.

M&E information directly supports provincial & LG

budget balancing, planning, and policy

formulation.

M&E information directly supports provincial & LG

budget balancing, planning, and policy

formulation.

Political and administrative levels in

province & LG assess and debate

performance.

Civil society assesses provincial and LG performance and inputs freely to

planning and policy debates

Improvements in provincial and departmental performance and service delivery


Table 2: M&E capacity building results framework

Objectives Activities Outputs Outcomes Impact

1. Perform detailed diagnosis of M&E systems

1. Perform provincial data management system and data quality audit

2. Develop data quality improvement action plans

1. M&E system audit reports 2. Department data management improvement

plans

1. Improved quality of M&E data

2. Increased utilization of M&E data in planning and budgeting

3. Improved

community satisfaction

1. Improved service delivery

2. Improved socio-economic wellbeing

2. Increase provincial demand for M&E

1. Develop M&E advocacy plan 2. Train political and management heads on

usefulness of M&E 3. Peer workshops on best practice M&E systems 4. Develop / review sector program performance

information frameworks

1. M&E advocacy strategic plan 2. M&E use workshops 3. M&E peer review workshops 4. M&E frameworks

3. Increase quality/quantity of supply for M&E

1. Perform M&E skills gap analysis 2. Recruit appropriate M&E personnel 3. Train M&E staff on appropriate skills 4. M&E learning tours and develop networks to

share experiences and best practices 5. Perform internal & external data quality audits 6. Increase funding for Training Academy to

support M&E capacity building (including new funding models)

1. M&E skills gap analysis reports 2. Recruit M&E staff 3. Focused M&E curricular developed 4. Train staff on M&E 5. Routine data quality audit reports 6. Increased funding for Training Academy

4. Strengthen structures and processes to support M&E systems

1. Increase financial resources for M&E 2. Develop M&E plans, policies, and frameworks 3. Select minimum performance indicators 4. Perform audit of electronic databases 5. Develop data management SOP 6. Develop or strengthen new electronic data

bases

1. Increased M&E budgets 2. M&E plans and frameworks 3. Minimum set performance indicators 4. M&E database audits

5. Re-arrange organisational structures to strengthen the role of M&E

1. Conduct M&E organizational restructuring to meet departmental needs

2. Develop new provincial M&E organograms

1. Organizational restructuring met 2. M&E units strategically positioned

6. Strengthen M&E collaboration and coordination within province and civil society

1. Conduct M&E system alignment workshops 2. Develop community M&E engagement strategy 3. Train communities on participatory M&E 4. Conduct community lead satisfaction surveys 5. Hold civil society performance forums

1. Department M&E alignment workshops 2. Community M&E strategic plan 3. Participatory M&E workshops 4. Satisfaction surveys


5.2. Action plan

An action plan is an overview of tasks the province needs to undertake to improve overall M&E

capacity. Capacity building development may be at organizational, individual (human) and systems

level within and across the province. In addition the capacity development action plan should be

based on achieving short, medium and long term outcomes as an ideal M&E system cannot be

achieved immediately. Provinces should therefore perform an options appraisal to prioritize

remedial actions. The options appraisal should be based on identifying most efficient and cost

effective actions to implement in the short term to long term period. Table 3 is a provincial action

plan showing timing and responsibility of capacity building tasks.

Table 3: Provincial action plan

Task Duration

(Term) Responsibility

Human

Recruit appropriate M&E personnel Short-Medium

Departments and Municipalities

Develop M&E advocacy plan Short OTP, and Training Academy

Train politicians and management on M&E Short-Medium OTP, and Training Academy

Conduct peer workshops on M&E best practices Short-Medium

(OTP and Department of Performance M&E

Perform M&E skills gap analysis and develop M&E training curriculum

Short OTP, Departments and Municipalities

Conduct M&E skills training Short-Medium Training Academy

M&E learning tours and develop networks Medium-Long

(OTP and Department of Performance M&E

Train communities on participatory M&E Medium-Long

Municipalities and Training Academy

Organizational

Perform detailed diagnosis of M&E systems Short OTP, and UNDP

Conduct baseline and routine data quality audits Short-Long

OTP, Departments, Municipalities & UNDP

Restructure and strategically position the M&E unit to facilitate informed decisions

Short Departments and Municipalities

Localize MDGs and incorporate in the MTSF and IDPs Short

OTP, Departments and Municipalities

Increase M&E usage Medium


Hold civil society performance forums Medium-Long OTP

Involve communities in evaluations Long


Systems

Develop M&E plan and policies frameworks Short

Departments and Municipalities

Develop relevant data management systems Short-Medium OTP and UNDP

Perform audit of electronic databases and develop interoperable information systems

Short OTP, Departments, Municipalities & UNDP

Establish performance indicators for the whole results-chain Short


Increase M&E budget up to 5-10% of budget Short-Medium Departments and


Municipalities

The training academy is expected to be engaged in extensive training exercises to improve M&E

skills among provincial officials that will be responsible for supply and demand of data from

generated by the M&E system. The OTP should lead the drive to increase allocation of extra

funding to support the training academy as an M&E capacity building partner in the province.

Table 4 is a summary of key tasks to which extra funding would be used. As a capacity building

partner with the OTP, UNDP would also be involved supporting the training academy through

improvement of information systems (as identified by UNDP ICT consultant)

Table4. Public Service Training Academy action plan

Tasks Duration (Term)

Responsibility

Human

Outsource high level skills training as deemed necessary by departments and municipalities

Short The Academy

Develop a training program in Knowledge Management

Short The academy

Train/employ to have own experts Medium The Academy and PALAMA

Organizational

Consider applying cost recovery models to complement the budget from the province

Short The Academy

Restructure and strategically position the M&E unit to facilitate informed decisions

Short The Academy

Training needs assessment that involves department and municipal human resource units

Medium (The Academy, departments, municipalities)

Systems

Review and accredit M&E curricula Short The Academy and OTP

Develop Web-services for electronic resources and publications

10

Short OTP and UNDP

Develop communication network and video streaming for distance learning

11

Short OTP and UNDP

Install computer centers in the Library to help research and access to digital libraries

12

Short OTP and UNDP

Provide UN publications and documents to create a UN section at the library (OTP, UNDP)

13

Short OTP and UNDP

Conduct pre-post training evaluations to assess outcome and impact of M&E trainings and provide feedback to departments and municipalities

Medium-Long The academy

10

Identified by Gonzalo Aramayo-Careaga (ICT/e-governance consultant at the UNDP) during recent visit to the Training Academy 11



Identified by Gonzalo Aramayo-Careaga (ICT/e-governance consultant at the UNDP) during recent visit to the Training Academy


Appendix 1: M&E assessment questionnaire

Kwa-Zulu Natal Monitoring and Evaluation Capacity Assessment Questionnaire/Discussion Guide

Background

A. The main objective of this exercise is to undertake an monitoring and evaluation (M&E) capacity needs assessment study for the Province

with a view to identifying essential gaps (human and institutional), suggesting how the capacity gap could be bridged and developing

requisite program of intervention. The information will therefore be treated confidential and will be used mainly for the foregoing purpose.

B. Name of the unit/Department...............................................................................................

C. Year the Unit was established....................................................................................

D. Number of staff in the unit..............................................................................................

E. INSTRUCTIONS: Rate the perceived or actual status of dimensions of capacity using the rating key below. When selecting the rating please consider the issues/factors suggested as critical to adequacy and or quality of the Monitoring, Evaluation and Reporting System for your unit. Write a comment on the rationale of your rating after this consideration.

5 4 3 2 1

Highly adequate

Adequate Present but not adequate

Not adequate at all Does not exist at all

Very high High Medium Low Very low Excellent Very good Average Fair Poor


A. Institutional Arrangements and environmental situations

Discussion Guide/Question Rating (1-5)

1 Adequacy and availability of legal framework, policies and procedures to manage and implement an M&E system (department or provincial level).

Please consider the following factors when rating: Availability and currency of framework/policies; Clarity of framework/policies; Suitability framework/policies; Staff and leadership Awareness about framework/policies; Extent of application at different levels; responsibility to monitor application; incentives to apply legal framework and policies; Conflict or synergy of framework/policy.

2 Adequacy of available overall system used by the organization to perform M&E tasks

Please consider the following factors when rating: Currency (changes) of system; origin and development of system; scope of system (i.e. tracks inputs, outputs, outcomes, data quality, finances) suitability of scope at different levels; auditing of M&E system.

3 Adequacy of roles and responsibilities to manage and implement M&E system

Please consider the following factors when rating: Clarity of roles to manage and implement; suitability of roles; discrepancy between formal and actual roles; barriers.

4 Level of available inter-sectoral, inter-departmental, intergovernmental and stakeholder coordination on management and implementation of M&E system(s)

Please consider the following factors when rating: Processes to support coordination; scope of coordination; suitability of scope of coordination; harmonization of data collection and reporting; integration, synergy & interoperability of ICTs used; challenges of systems; barriers.

5 Adequacy of available donors and other stakeholders in capacity building of M&E system

Please consider the following factors when rating: Scope of capacity building; quality; sustainability; capacity to meet needs; barriers.

6 Availability and adequacy of mechanisms to engage civil society, public and the private sector in the design, reporting and use of information from M&E system

Please consider the following factors when rating: current scope and quality of engagement e.g. satisfaction surveys; promotion of engagement; barriers.


B. M&E Advocacy

Discussion Guide/Questions Rating (1-5)

7 Leadership influence on development of an M&E system to generate data across key levels

Please consider the following factors when rating: Existence of influential and outspoken champions for M&E System; Advocacy within and across departments on M&E system development; Budget allocation to develop and operationalise M&E units and M&E unit staff development (e.g., what percent of the total departmental budget is allocated to the M&E unit?

8 Use of M&E information for program planning, resource allocation and policy making

Please consider the following factors when rating: Frequency of demand for M&E reports; opportunity given to M&E Unit to report; Usage of M&E reports; Readiness of management to objectively review M&E reports; Use of all data categories in M&E reports (e.g. inputs, outputs, outcomes); Barriers to M&E information use

9 Management feedback on quality of M&E system.

Please consider the following factors when rating: Frequency of feedback on M&E reports; internal and/ or external M&E system audits commissioned by management; barriers to feedback

10 Management capacity to perform and use inferential analyses on M&E data

Please consider the following factors when rating: Frequency of management’s request for clarification on statistical data


C. M&E knowledge and skills

Discussion Guide/Questions Rating (1-5)

11 Capacity of the M&E system to generate and disseminate good quality data for evidence-based decision making

Please consider the following factors when rating: Quality of data collection, collation, analysis and reporting processes; use of information for organizational learning and decision making; Contribution of M&E system to setting, targets and indicators.

12 Extent of assistance given to management for establishing indicators and target setting of departmental goals

Please consider the following factors when rating: Extent of opportunities to contribute to setting departmental goals and targets; the M&E system’s report submitted to management that contributed to setting indicators and targets (including quality of report (management reactions and follow up activities).

13 Adequacy of electronic system to collate data for the M&E system in your department

Please consider the following factors when rating: Type (operating system) of software used, scope of software, ability of staff to use software, interdepartmental /governmental harmonization of software; Harmonization and use of software with complimentary software e.g. GIS.

14 Availability and quality of recent M&E reports/evaluations/policy reviews/resource allocation/ costing data/program output data

Provide sample of most recent reports From M&E unit during interview]

15 Level of staff skill to understand and use M&E concepts to develop results (logical) frameworks, data collection and collation tools, data analyses

Please consider the following factors when rating: current number of staff relative to burden of work, current skills relative to technical inputs needed in M&E unit e.g. skills to perform data quality audits, program/ project evaluations, policy analysis, cost effectiveness/ benefit analyses, design databases.

16 Availability of M&E system to generate data on the MDG indicators

Please consider the following factors when rating: Awareness of MDG indicators, Domestication of MDG indicators, awareness of sources of data for MDG indicators; Challenges to reporting on MDG indicators.


D. Reporting mechanisms

Discussion Guide/Question Rating (1-5)

17 Existence of an effective reporting mechanisms

Please consider the following factors when rating: Availability and use of standardized reporting formats; presence of written guidelines to each sub-reporting level on what, how, whom, when to report; the feedback mechanisms to all sub-reporting levels on the quality of their reporting (i.e., accuracy, completeness, timeliness, etc); the consistency of reporting tools; Policy on retention of source documents for reports; availability of written procedure to address late, incomplete, inaccurate, and missing reports including follow-up with sub-reporting levels on data quality issue.

18 Adequacy of current M&E system to generate reports with targeted disaggregation of data showing status of reach on outputs and outcomes of priority target beneficiaries

Please consider the following factors when rating: Presence of disaggregation of data by priority e.g. Gender, age-group, location, socio-economic status, e.t.c

19 Availability of mechanisms to make M&E reports available to relevant stakeholders

Please consider the following factors when rating: Mechanisms being used to engage stakeholders on M&E issues e.g. Forums; the availability of community-based M&E system that your department is using; the regularity of dissemination of progress reports about projects/programs to relevant stakeholders; the availability of data dissemination plans.

20 Availability of a mechanism to link M&E data and overall organizational and staff performance

Please consider the following factors when rating: Existence to link program/ project outputs and outcomes to staff performance, Use of M&E data and staff performance for skills and knowledge improvement, Use of M&E data and staff performance for promotion/demotion or change in roles and responsibilities.


Appendix 2a: Assessment summary findings

KZN M&E CAPACITY ASSESSMENT: SUMMARY FINDINGS OF ADEQUACY OF M&E SYSTEM CAPACITY

CAPACITY CATEGORIES AND QUESTIONS

Loca

l G

ove

rnm

en

t &

Trad

itio

nal

Aff

airs

Spo

rts

& R

ecr

eat

ion

Agr

icu

ltu

re

&

Envi

ron

me

nta

l Aff

airs

He

alth

Soci

al D

eve

lop

me

nt

Co

mm

un

ity

Safe

ty

Art

s &

Cu

ltu

re

Eco

no

mic

De

velo

pm

en

t &

Tou

rism

Tran

spo

rt

Trai

nin

g A

cad

em

y

Siso

nke

DM

Edu

cati

on

Hu

man

Se

ttle

me

nts

Pu

blic

Wo

rks

Msu

nd

uzi

mu

nic

ipal

ity

COUNTS OF ADEQUACY SCORE

HIG

HLY

AD

EQU

ATE

AD

EQU

ATE

PR

ESEN

T B

UT

NO

T A

DEQ

UA

TE

NO

T A

DEQ

UA

TE

AT

ALL

NO

T P

RES

ENT

1. Institutional arrangements & environmental situation

QN1 3 3 3 4 3 5 4 3 4 3 1 4 3 4 3 1 5 8 0 1

QN2 4 3 3 3 2 4 4 3 3 2 2 3 3 3 2 0 3 8 4 0

QN3 4 2 3 4 3 4 4 3 4 3 2 4 2 ? 2 0 6 4 4 0

QN4 1 1 3 3 2 1 2 3 4 3 3 4 1 2 2 0 2 5 4 4

QN5 5 1 2 2 2 1 2 4 N/A 2 1 2 2 1 2 1 1 0 8 4

QN6 3 1 4 3 2 3 3 4 4 4 1 2 3 1 2 0 4 5 3 3

2. M&E Advocacy

QN7 5 2 3 4 4 3 4 3 4 3 1 2 3 N/A 2 1 4 5 3 1

QN8 2 2 2 5 2 3 4 3 4 3 3 4 3 3 1 1 3 6 4 1

QN9 3 1 3 3 3 3 4 4 4 3 1 3 2 N/A 1 0 3 7 1 3

QN10 3 1 2 4 2 4 4 3 4 4 1 2 3 3 1 0 5 4 3 3

3. M&E knowledge and skills

QN11 2 2 3 3 2 3 4 4 3 2 2 3 3 2 2 0 2 6 7 0

QN12 4 2 3 4 2 3 4 4 4 4 3 4 3 3 1 0 7 5 2 1

QN13 4 3 4 3 1 4 3 2 4 1 1 1 3 3 2 0 4 5 2 4

QN14 4 2 3 4 3 3 3 3 4 2 1 3 2 2 1 0 3 6 4 2

QN15 4 1 2 4 2 3 3 4 4 2 4 4 2 N/A 2 0 6 2 5 1

QN16 2 1 3 4 2 1 2 2 4 2 3 2 3 4 2 0 3 3 7 2

4. Reporting mechanisms

QN17 5 3 4 4 2 3 4 3 3 3 3 4 3 4 3 1 5 8 1 0

QN18 1 4 2 4 3 4 2 3 4 2 1 3 4 3 2 0 5 4 4 2

QN19 2 2 4 4 3 3 2 3 4 3 3 2 3 3 3 0 3 8 4 0

QN20 1 3 4 3 2 3 4 3 3 2 2 2 4 2 3 0 3 6 5 1

5 77 105 75 33

1.70% 26.10% 35.60% 25.40% 11.20%

27.80% 35.60% 36.60%


Appendix 2b: Assessment summary findings

KZN M&E CAPACITY ASSESSMENT: SUMMARY FINDINGS OF ADEQUACY OF M&E SYSTEM CAPACITY

CAPACITY CATEGORIES AND QUESTIONS

COUNTS OF ADEQUACY SCORE

HIG

HLY

AD

EQU

ATE

AD

EQU

ATE

PR

ESEN

T B

UT

NO

T A

DEQ

UA

TE

NO

T A

DEQ

UA

TE

AT

ALL

NO

T P

RES

ENT

HIGHLY ADEQUATE AND ADEQUATE (%)

PRESENT BUT NOT ADEQUATE (%)

NOT ADEQUATE AT ALL AND NOT PRESENT (%)

1. Institutional arrangements & environmental situation

QN1 1 5 8 0 1

25.3% 31.0% 43.7%

QN2 0 3 8 4 0

QN3 0 6 4 4 0

QN4 0 2 5 4 4

QN5 1 1 0 8 4

QN6 0 4 5 3 3

2. M&E Advocacy

QN7 1 4 5 3 1

QN8 1 3 6 4 1

23.7% 39.0% 37.3% QN9 0 3 7 1 3

QN10 0 5 4 3 3

3. M&E knowledge and skills

QN11 0 2 6 7 0

QN12 0 7 5 2 1

32.6% 32.6% 34.8%

QN13 0 4 5 2 4

QN14 0 3 6 4 2

QN15 0 6 2 5 1

QN16 0 3 3 7 2

4. Reporting mechanisms

QN17 1 5 8 1 0

QN18 0 5 4 4 2

27.4% 36.2% 36.5% QN19 0 3 8 4 0

QN20 0 3 6 5 1

5 77 105 75 33

1.70% 26.10% 35.60% 25.40% 11.20%

27.80% 35.60% 36.60%


References 1 http://www.mg.co.za/article/2009-07-22-sa-hit-servicedelivery-protests

2 http://www.municipaliq.co.za/press/press_release_service_delivery_protests.doc

3 Chabane, C. 2009. National planning, performance monitoring green papers to go before Parliament.

http://www.polity.org.za/article/national-planning-performance-monitoring-green-papers-to-go-before-parliament-2009-09-04 4 Auditor-General. 2008. “PFMA roadshows 2008.” [Online]. Available: http://www.agsa.co.za/

5 Cloete, F. 2009. Evidence-based policy analysis in South Africa: Critical assessment of the emerging Government-

Wide Monitoring and Evaluation System. Journal of Public Administration 44(2). 6 Presidency. 2007. Policy Framework for the Government-wide Monitoring and Evaluation System. The Presidency,

November 2007, Pretoria. http://www.thepresidency.gov.za/mainasp?include=learning/reference/policy/index.html. 7 Cloete, F. 2009. Evidence-based policy analysis in South Africa: Critical assessment of the emerging Government-

Wide Monitoring and Evaluation System. Journal of Public Administration 44(2). 8 Presidency. 2009. Green paper: National strategic planning. www.thepresidency.gov.za

9 Presidency. 2009. Improving government performance: Our approach.

10 The Framework for Managing Programme Performance Information. Department of National Treasury. Government

of South Africa. May 200

11 South African Statistical Quality Assessment Framework (SASQAF). 1

st Ed. Statistics South Africa, Pretoria, 2008

12 http://www.southafrica.info/about/geography/kwazulu-natal.htm

13 KwaZulu-Natal Provincial Government. 2007. SA: Ndebele: Public Service Training Academy.

http://www.polity.org.za/article/sa-ndebele-public-service-training-academy-31082007-2007-08-31

14 National Treasury. 2007. Framework for Managing Programme Performance Information, National

Treasury: Pretoria, www.treasury.gov.za/publications/guidelines/FMPI.pdf 15

National Treasury. 2008. Treasury Guidelines Preparation of Expenditure Estimates for the 2009 Medium Term Expenditure Framework. www.treasury.gov.za/publications/guidelines 16

Western Cape Provincial Treasury. 2003. “Western Cape Fiscal Policy 2003-2006: Financial Management Issues.” 17

National Treasury 2000. “In year management, monitoring and reporting.” 18

Auditor-General. 2007. “Information systems audit report on the system development life cycle followed in the implementation of the FINEST system at the Limpopo Provincial Government: Provincial Treasury.” 19

Keith Mackay, How to Build M&E Systems to Support Better Government, 2007 (IEG, World Bank)

http://www.mg.co.za/article/2009-07-22-sa-hit-servicedelivery-protests

http://www.municipaliq.co.za/press/press_release_service_delivery_protests.doc

http://www.polity.org.za/article/national-planning-performance-monitoring-green-papers-to-go-before-parliament-2009-09-04

http://www.polity.org.za/article/national-planning-performance-monitoring-green-papers-to-go-before-parliament-2009-09-04

http://www.agsa.co.za/

http://www.thepresidency.gov.za/mainasp?include=learning/reference/policy/index.html

http://www.polity.org.za/article/sa-ndebele-public-service-training-academy-31082007-2007-08-31

http://www.treasury.gov.za/publications/guidelines/FMPI.pdf

http://www.treasury.gov.za/publications/guidelines

Capacity Needs Assessment of Monitoring and Evaluation … · 2020-03-02 · Capacity Needs...

Documents

Transcript of Capacity Needs Assessment of Monitoring and Evaluation … · 2020-03-02 · Capacity Needs...