MEMORANDUM - Independent Evaluationidev.afdb.org/sites/default/files/documents/files/Tanzania -...

SCCD : K.M

AFRICAN DEVELOPMENT BANK AFRICAN DEVELOPMENT FUND

ADB/BD/WP/2016/167 ADF/BD/WP/2016/115

6 October 2016 Prepared by: IDEV Original: English

Probable Date of Presentation to the Committee

Operations/ Development Effectiveness

(CODE)

21 October 2016

FOR CONSIDERATION

MEMORANDUM TO : THE BOARDS OF DIRECTORS

FROM : Vincent O. NMEHIELLE

Secretary General

SUBJECT : TANZANIA: IMPACT EVALUATION OF THE RURAL WATER SUPPLY

AND SANITATION PROGRAMME*

Please find attached the above-mentioned Report.

Attach:

Cc: The President

* Questions on this document should be referred to:

Mr. R. NANGIA Evaluator General IDEV Extension 2041

Mrs. R. AMIRA Division Manager IDEV.1 Extension 2696

Mr. F. TURAY Chief Evaluation Officer IDEV.1 Extension 3257

AFRICAN DEVELOPMENT BANK GROUP

TANZANIA

IMPACT EVALUATION OF THE RURAL WATER SUPPLY AND SANITATION

PROGRAMME

IDEV DEPARTMENT

October 2016

Impact Evaluation

of the Rural Water Supply and Sanitation Programme,

United Republic of Tanzania

Main Report

Cover: members of the COWSO at Kona, Kiteto district. Their scheme includes a stock watering trough.

Preface

Safe domestic water supplies and appropriate sanitation and hygiene practices are vital for an adequate standard of living in rural Tanzania. With the support of their development partners, the country’s Government and citizens have been working hard towards achieving these goals, through Phase I of the Water Sector Development Programme (WSDP).

Through the Rural Water Supply and Sanitation Programme (RWSSP), the African Development Bank Group (AfDB) has been a strong supporter of these efforts. The Independent Development Evaluation of the AfDB – as it did recently in Ethiopia – has now commissioned an evaluation of the impact of the RWSSP on key aspects of rural livelihoods and the economy.

We were honoured to be asked to undertake this evaluation, and have worked with colleagues in Government, the Bank and development partners to produce the data and analysis presented below. We hope it will be a useful input to the enhancement of rural water and sanitation efforts as the WSDP begins its second phase.

The main evaluation report is presented below, together with Annexes 1 and 2 (terms of reference and a list of persons met). Volume II presents technical annexes on methodology and data, as well as the field survey instruments that were used.

We thank the Resident Representative of the African Development Bank, Dr Tonia Kandiero, and the RWSSP Task Manager in the Bank’s Country Office, Mr Sabas Marandu, for all their support during this evaluation. We are grateful, too, for all the logistical and administrative assistance provided by Mrs Halimah Kitandula of the Country Office. We acknowledge with sincere thanks the guidance and support of the Director of Rural Water Supplies in the Ministry of Water, Engineer Frida Rweyemamu, and of the Deputy Director, Engineer Jackson Mutazamba.

At the University of Dar es Salaam, the field surveys commissioned to the Centre for Economic Research and Policy were ably co-ordinated by its research co-ordinator, Dr Stephen Kirama. We acknowledge with thanks the hard work of the Centre’s survey supervisors and enumerators as they carried out the survey in 30 Local Government Authorities (LGAs) around the country. We are also grateful to one of the supervisors, Mr Shakir Msangi, for his able support to the institutional assessment mission that we undertook in October 2015.

Most importantly of all, we respectfully thank the Council Water and Sanitation Teams, members of Community-Owned Water Supply Organisations and rural water users who gave so freely of their time during our investigations. Once more, we express the hope that this evaluation will lead to stronger and more sustainable rural water and sanitation systems in rural Tanzania.

Kenneth Mdadila

Stephen Turner

Dar es Salaam and Manchester 30 March, 2016.

Contents: Volume I

Abbreviations i

Summary ii

1 Background 1

1.1 Country context 1

1.2 Programme funding and objectives 2

1.3 Theory of change 3

1.4 Summary of programme implementation 6

1.5 Impact evaluation of the RWSSP 7

1.6 Evaluation questions 8

1.7 Scope of the impact evaluation 9

1.8 Activities during the impact evaluation 9

Methodology 10

1.9 Impact evaluation design 10

1.10 Sample size and sampling procedures 11

1.11 Survey instruments and implementation of field survey 12

1.12 Water quality testing 13

1.13 Post-survey data management and checks 13

1.14 Existing empirical evidence 13

1.15 Interviews with informants 14

1.16 Limitations 14

Findings 15

1.17 Introduction 15

1.18 Policy 16

1.19 Activities 17

1.20 Outputs 18

1.20.1 Capacity development 23

1.21 Outcomes 25

1.21.1 Equity 25

1.21.2 Institutional outcomes 26

1.21.3 Institutional assumptions 28

1.22 Impact 29

1.22.1 Impact on health 29

1.22.2 Impact on girls’ education 36

1.22.3 Impact on women’s economic opportunities 38

1.22.4 Impact on economic activity 38

Sustainability and replicability 39

1.23 Technical sustainability 39

1.24 Institutional sustainability 39

Impact Evaluation of the Rural Water Supply and Sanitation Programme, United Republic of Tanzania: Volume I __________________________________________________________________________________________

1.25 Links between technical and institutional sustainability 40

1.26 Replicability 41

Conclusions, lessons and recommendations 41

1.27 Conclusions 41

1.28 Lessons 42

1.29 Recommendations 43

1.29.1 Policy recommendations 43

1.29.2 Programming recommendations 43

Annex 1. Terms of reference I

Annex 2. Persons met VIII

References XI

Tables: Volume I

Table 1. Distribution of the evaluation sample size 12

Table 2. Guide to responses to evaluation questions 15

Table 3. Primary schools' sources of drinking water 21

Table 4. Training received by 26 CWSTs, as reported to evaluation survey 24

Table 5. Subjects on which COWSOs and other user groups reported receiving training 25

Table 6. Impact of programme on incidence of diarrhoea 30

Table 7: Water quality at distribution points and at the point of use 31

Table 8: Household perceptions of changes in quality of water 31

Table 9. Water consumption per capita per day in programme and non-programme communities 32

Table 10. Availability of hand washing facility within 3m of toilet 34

Table 11. Impact estimation: availability of hand washing facility within 3m of toilet 34

Table 12. Households’ use of drinking water from improved sources 35

Table 13. Household perceptions of adequacy of water supply in dry season 35

Table 14. Households’ reported sources of water in dry and wet seasons 35

Table 15: Enrolment and dropouts at primary school 36

Table 16. Time spent by girls and women fetching water 37

Table 17: Impact on women's participation in income-generating activities 38

Figure 1. Water carts are an increasingly common means of transport 2 Figure 2. RWSSP Tanzania: theory of change 5

Figure 3. Water sellers' carts in a community with no supply: water drawn from a nearby dam 7

Figure 4. Water may be transported several kilometres to communities without a supply 18

Figure 5. A simple hand pump installed on a borehole 20

Figure 6. Community management structures at 114 water points in 'treatment' villages 22

Figure 7. Some schemes only supply water intermittently 31

i

Abbreviations AfDB African Development Bank Group BEST Basic Education Statistics BRN Big Results Now CERP Centre for Economic Research and Policy CLTS Community-Led Total Sanitation COWSO Community-Owned Water Supply Organisation CWST Council Water and Sanitation Team DFID Department for International Development DHS Demographic and Health Surveys DID Difference-in-differences DP Distribution point DPs Development Partners DWST District Water and Sanitation Team ECCT Economics of Climate Change in Tanzania EQ Evaluation question GOT Government of Tanzania IDEV Independent Development Evaluation Department IOB Policy and Operations Evaluation Department, Ministry of Foreign Affairs, The Netherlands JMP Joint Monitoring Programme LGA Local Government Authority MDG Millennium Development Goal ml Millilitre MOHSW Ministry of Health and Social Welfare MOW Ministry of Water nd No date np No page number NSC National Sanitation Campaign O&M Operation and maintenance OC Other Charges PE Personnel Emolument PIM Programme implementation manual PMO-RALG Prime Minister’s Office – Regional Administration and Local Government PSM Propensity score matching RWSS Rural water supply and sanitation RWSSP Rural Water Supply and Sanitation Programme SE Standard error TOC Theory of change UA Units of Account UNICEF United Nations Children’s Fund URT United Republic of Tanzania WASH Water, sanitation and hygiene WHO World Health Organisation WLS Weighted-least-squares WSDP Water Sector Development Programme WSP Water and Sanitation Programme

ii

Summary

The Rural Water Supply and Sanitation Programme

1. The Government of the United Republic of Tanzania (GOT), together with a number of development partners

(DPs) including the African Development Bank Group (the AfDB), World Bank and United Kingdom Department

for International Development (DFID) funded component 2 of the country’s Water and Sanitation Development

Programme (WSDP), Phase1 (2006-2015). Component 2 of the WSDP was effectively the Rural Water Supply

and Sanitation Programme (RWSSP).

2. The RWSSP (or component 2 of the WSDP, Phase1) was “to improve the health and quality of life of the rural

Tanzanians through provision of water and sanitation services on a sustainable basis”. This goal was to be

achieved by improving (i) district level capacity to implement demand-based RWSS programmes; (ii) access of

rural communities to water and sanitation services operated and maintained by capable persons; and (iii) health

and hygiene practices. The RWSSP was intended to cover all rural areas in mainland Tanzania, and to benefit

the overwhelming majority of the rural population who lacked access to safe water and adequate sanitation.

Its beneficiaries also include school children, local government authorities (LGAs) and community-based water

organisations. The RWSSP comprised three key components – (i) construction of rural water supply and school

sanitation facilities, (ii) national sanitation campaign and school WASH, and (iii) management support for LGAs.

3. The RWSSP was implemented in 2006-2015, and the AfDB’s contribution, through basket funding, was made

in two phases; 2006-2010 and 2010-2014. The total planned commitment to the RWSSP by all funders for the

period 2006-2015 was USD654.16 (or UA4223) million (m) with the AfDB accounting for about 28 percent of it.

As at the end of December 2015, the AfDB has disbursed a total of USD151.75 (UA 110.58) m in loans and USD

13.72 (UA 10) m in grant funding to the RWSSP. The other major contributors to the RWSSP funding include the

GoT, World Bank (IDA) and DFID. The total commitment to the WSDP was USD 1.67bn, of which USD 1.48bn (88

percent) had been disbursed as at 31 December 2015. The GoT’s contribution to this total commitment was

USD 369m, all of which was disbursed. The implementation of the RWSSP was to end in 2014, but had to be

extended to December 2015. With late and slow initial implementation, the RWSSP was restructured in 2011.

Construction of new rural water schemes only began at scale in 2012, picking up speed in 2013. Implementation

was accelerated by the inclusion of the water sector in the national Big Results Now (BRN) initiative (2013/14 –

2015/16) and the adoption of a goal to implement ten rural water projects in each LGA area. The LGAs were

responsible for the installation of water schemes, co-ordinated by the Ministry of Water (MOW). They

contracted water engineering firms to install the water infrastructure and appointed consultant service

providers to design and supervise construction of the schemes. In many cases, these consultants also provided

institutional facilitation and training for the Community-Owned Water Supply Organisations (COWSOs) that

were established to own and operate the schemes. The National Sanitation Campaign (NSC) resulted from the


iii

conversion of the sanitation subcomponent into a separate entity under the co-ordination of the Ministry of

Health and Social Welfare (MOHSW). This restructuring was intended to enhance the performance of the

sanitation subcomponent, which had been poor.

Impact evaluation of the RWSSP

4. The Independent development Evaluation (IDEV) of the AfDB commissioned this impact evaluation of

the RWSSP; effectively Component 2 of the WSDP, Phase I, with the purpose of:

Providing credible information on the emerging impact of the RWSSP to relevant Ministries and

agencies of the GOT, the Board of the Bank and DPs.;

Providing relevant lessons and recommendations to the GOT, Bank management (including the Water

and Sanitation Department) and DPs for sustaining the RWSSP’s benefits, and for informing the design

and implementation of Phase 2 of the WSDP;

Helping IDEV build the evidence for its 2015 Tanzania Country Strategy Paper evaluation and water

supply and sanitation thematic evaluation.

5. The evaluation questions mainly focus on the key RWSSP outputs (water, sanitation and hygiene

facilities and services, and institutional capacity) and impacts (access to and use of improved water and

sanitation; incidence of diarrhoeal disease among children under five years of age; enrolment of girls at school;

women’s participation in economic activities; self-employment in community-based WASH service delivery;

sustainability), and also on the ‘how’ and ‘why’ concerning the extent of the RWSSP impact. The study used

multiple lines of evidence to answer the evaluation questions. It responded to the impact questions by

comparing the relevant RWSSP results at the household and community levels to those of appropriate and

carefully matched non-RWSSP community groups. This approach was taken because the RWSSP was designed

and implemented without incorporating the data needs of an impact evaluation. Data were collected from

RWSSP and non-RWSSP communities (240) and households (2374) in 30 of the 79 qualified LGAs (an LGA was

qualified if it had constructed at least four water schemes under the Ten Villages initiatives by July 2014). Data

were also gathered from 168 water points, 128 primary schools, 28 Council Water and Sanitation Teams

(CWSTs) and 114 COWSOs. Additional data were also obtained secondary data sources, and interviews with

samples of officials of GOT, LGAs, health centres, DPs, NGOs, CWST and COWSOs, as well as water points. The

data analysis included propensity score matching (PSM) and difference-in-difference (DID).


iv

Key findings

6. The RWSSP delivered structures and procedures at LGA and community level for the planning, development

and operation of rural water and sanitation but the associated budgets were neither fully in place nor efficiently

coordinated. The RWSSP structures and procedures were in place and widely understood and capable of

implementation by CWSTs and COWSOs. However, fund management from central to local government was

dysfunctional thereby greatly hindering programme implementation. The RWSSP not only ensured the

provision of training to the LGAs especially the CWSTs, but also the establishment and strengthening of user

groups or COWSOs. The COWSO, a key institutional output of the RWSSP, was meant to own, manage, and

fund the operation and maintenance (O&M) of the rural water supply system. With the RWSSP support, these

bodies have developed slowly though the many formal stages meant to lead to legal registration as corporate

entities. Only 17 percent (460) of the planned COWSOs have formally been registered. Women’s membership

of these COWSOs was modest. It was between 21 percent and 50 percent in 84 percent of the user bodies, for

which the information was reported –compared to the MOW guidelines of at least 60 percent.

7. COWSOs, CWSTs and some community level operators received training from the RWSSP though the training

were uneven. Among the 114 COWSOs surveyed, 111 (97 percent) of them reported to have been trained in at

least one relevant subject area. Yet in the case of the 28 CWSTs surveyed, only 17 reported they had received

at least one training. The levels of responsibility and confidence displayed by these groups also differ widely

due in part to their exposure to variable training. Some of the longer-established groups have now been through

one or more cycles of elections, resulting in substantial (sometimes complete) turnover of membership.

Significantly, there do not appear to be systems in place for tracking the arrival of new members and arranging

their training. So far, this element of the programme has been treated as a once-off exercise, whereas in fact

institutional maintenance will be needed on a recurrent basis. The effectiveness of the RWSSP institutions was

limited by three key factors: (i) the COWSOs’ lack of operating resources and recurrent budgets resulting from

users’ limited ability to pay for safe water and improved sanitation services; (ii) irregular, untimely and uneven

disbursement of the GOT contribution; and (iii) weak coordination of water, sanitation and hygiene initiatives

at the LGA and community levels resulting from the transfer of the delivery of sanitation and hygiene outputs

to the National Sanitation Campaign (NSC).

8. The RWSSP made significant progress in installing functional rural water supply systems, but less so in the

delivery of sanitation and hygiene facilities and services, and of coordination between rural water supply and

sanitation components. By October 2015, the programme had developed 43,255 water points, compared with

a target of 31,747. These water points were estimated to have served about 10.8 million rural people relative

to the target of 8 million. A survey of this impact evaluation study reported that 82 percent of the 168 primary

water sources in programme communities were functioning, compared with 54 percent in non-programme


v

communities. Defective design was by far the most commonly mentioned reason for water sources not

functioning; over-use of the facility and improper use (nature of the water schemes – technically sophisticated

and costly; quality of the water) were other significant factors.

9. Regarding the construction of improved latrines and school water, sanitation and hygiene (WASH), only 25

percent of the improved latrines and 16 percent of the school programmes had been achieved by June 2014.

Also apparent were coordination issues at LGA and field levels with respect to rural water and sanitation

interventions, which resulted from the separate status of the NSC within the programme. In the 120 RWSSP

communities surveyed, only five percent of households reported having participated in hygiene promotion or

training events. Twenty-five percent of these households had a hand-washing facility within three metres of

their latrine – the same proportion as in the sample of non-RWSSP communities. Nor was there a significant

difference between the proportions of households with improved latrines in RWSSP communities (25 percent)

and non-RWSSP communities (20 percent). Some of the latter may have been separately served by the NSC.

(Sanitation and hygiene work in communities where the RWSSP was installing water supplies was done by the

water supply consultants, not the NSC.)

10. Almost all (66) of the 67 schools surveyed had toilet facilities for pupils, with 65 reporting that these facilities

were in use. However, only 48 percent of these 67 schools had a water supply at the toilets for hand washing.

Of the 67 surveyed, 76 percent of primary schools at or near communities served by the RWSSP had a drinking

water facility, of which 51 percent were functional.

11. The RWSSP had a statistically significant positive impact on the incidence of diarrhoeal disease among

children under five years of age and the rural population in general. This impact was higher for the population

as a whole (almost 10 percent) than for children under five (three percent). Although this impact was

significant, it was modest compared with the 20–25 percent improvement recorded by rural water and

sanitation impact evaluations in some other countries. This may be due to the fact that 29 percent of

respondents in sampled RWSSP communities reported using rainwater (mostly collected from the ground, not

collected safely from roofs) as well as, or instead of, the improved water supply during the rainy season. This

significantly reduced their expenditure on water, as well as the revenues of COWSOs. It is also notable that, in

communities where RWSSP has installed water schemes, the proportion of households using less than the

Tanzanian national standard of 25 litres per person per day (71 percent) is almost as high as the proportion in

communities where the programme has not intervened (75 percent); and that 62 percent of 285 sampled

households in RWSSP communities were found to be using water contaminated with E. Coli.

12. The RWSSP had a significant and positive impact on women’s participation in economic activities but not on

the enrolment of girls at school. These impacts were expected to accrue from time savings enjoyed by girls and

women who no longer have to walk long distances to collect water. The RWSSP generated significant time


vi

savings for girls and women fetching water. These time savings appreciably increased women’s participation in

economic activities, in particular, with respect to productive paid labour and on income-generating activities.

This improvement is estimated to amount to a rise of 9.1 percent. However it had no detectable impact on girls’

enrolment in primary schools, which was already very high.

13. The RWSSP failed to stimulate the expected significant economic growth in the local water engineering

sector across the district towns of Tanzania. The construction of so many new water supply schemes by the

programme, and the subsequent demand for maintenance services from the COWSOs operating them, were

meant to serve as the stimuli for the expected economic growth. However this has not happened, although

hardware shops and, in some cases, NGOs do make money from marketing spare parts and materials. Most

construction work was contracted to larger firms based in Dar es Salaam or other major centres. Crucially,

COWSOs’ maintenance budgets were too inadequate to inject much new money into the local economy.

14. The RWSSP made good progress in building institutions and generated positive impact but its benefits are

unlikely to be sustainable mainly because of the COWSO’s financial and technical viability, inefficient fund

management and poor coordination of water and sanitation interventions. Other contributory factors include

the failure to recognise the ongoing need for institutional maintenance at community level and the failure of

public-private sector partnership to grow as intended in the national policy.

15. The RWSSP implementation arrangements led to consultants designing schemes that often spanned several

villages and depended on comparatively high-cost pumping and reticulation approaches. These imposed

management and maintenance demands on COWSOs that were not, in fact, adequately resourced or skilled to

meet them. Furthermore, funding and fiscal management weaknesses greatly undermined the implementation

and sustainability of the RWSSP interventions.

16. The continuing segregation of water and sanitation efforts without appropriate coordination mechanism

was also an impediment to the RWSSP sustainability. The separate implementation of the NSC has impaired co-

ordination of these efforts at community level. Sanitation and hygiene indicators measured by the study show

hardly any difference between communities that have received new water supply systems and those that have

not, even though consultants for the water schemes were supposed to implement the sanitation subcomponent

in the former.

17. The technical sustainability of the WSDP’s rural water and sanitation interventions is uncertain for several

reasons. Climate change is likely to increase the challenge of obtaining enough water. The widespread use of

diesel generators to power borehole pumps means frequent breakdowns in supply, and maintenance costs that

many COWSOs cannot afford. Nor may they be able to afford the lower maintenance costs of alternative

technologies.


vii

18. The replicability of the RWSSP approaches to rural water and sanitation can be challenged by the

institutional and technical constraints detailed above. Such constraints must be overcome if the RWSSP

approaches are to become replicable. Above all, it is necessary to recognise that the development of user

institutions in the sector is not a once-off achievement – they impose ongoing support obligations for which

systems, procedures and budgets must be in place. This goes beyond the existing, and widely recognised,

challenge of inadequate recurrent budgets for LGAs, which greatly diminishes their ability to provide even

technical support to COWSOs.

Lessons

19. The efficiency and effectiveness of an otherwise sound and competent programme can be seriously

compromised by inefficient management of funds at central government level.

20. The current levels of budget subventions to CWSTs gravely compromise the sustainability of the RWSSP

achievements.

21. Without reformed financial management systems linking national and local government authorities, and

without adequate recurrent funding to CWSTs, the progress and sustainability of Phase II of the WSDP will be

as inadequate as the RWSSP.

22. For a rural water and sanitation programme to be adequately effective it is essential that the water and

sanitation components be adequately coordinated.

23. The depth of poverty in rural Tanzania is such that many people are prepared to turn away from an improved

water supply to an unimproved one when the latter is fairly readily available (in the rainy season). This has

important implications for the achievement of the health outcomes that improved rural water supplies are

expected to achieve.

24. Current conditions make it difficult for users to pay for private sector services during the operation and

maintenance phase, and the private sector is consequently poorly developed in rural Tanzania with regard to

such services. An ongoing public sector role in support of O&M must be anticipated.

25. Ongoing institutional maintenance is a prerequisite for sustainability in the sector. The public sector has a

continuing responsibility to ensure that this maintenance takes place – although elements of the required

support services could, in theory, be provided by the private sector.


viii

Recommendations

26. On the basis of its findings, the evaluation makes the policy and programming recommendations below. As

the RWSSP was funded by multiple partners including GOT, the GOT should lead the implementation of the

recommendations although the AfDB and other DPs play a facilitating/supporting role.

27. The AfDB and other DPs in the water and sanitation sector should maintain continuous and effective

dialogue with and/or support of the GOT and relevant Ministries and agencies in order to:

Expedite Government’s reforms to public finance management systems that ensure a predictable,

smooth and timely transfer of domestic and donor funding to LGAs for the development, operation and

maintenance of rural water and sanitation systems and programmes.

Ensure that Government undertakes a careful review of affordability issues as they relate to the

provision of the National Water Policy for users to pay for improved rural water supplies. While current

poverty levels persist, the effectiveness of these improved supplies in enhancing health outcomes will

remain compromised while users turn to unimproved water for financial reasons.

Ensure that government replaces the current ad hoc approach to financial support from LGAs to

COWSOs for these costs with revisions to the National Water Policy that specify what types of COWSO

maintenance and repair cost will be subsidised, and how. It should adjust budgetary subventions to

LGAs accordingly while government should recognise that, at least for the remainder of the WSDP

period (to 2025), most COWSOs will be unable to pay in full for the maintenance and replacement costs

of their water supply infrastructure.

Ensure government increases, within feasible fiscal limits, recurrent funding to CWSTs.

Ensure that the GOT’s policy and procedures for rural water supply and sanitation recognise more

explicitly that long-term institutional maintenance is vital for the sustainability of COWSOs and RWSSP

benefits. With the support of DPs, government should adjust personnel, training and recurrent budgets

to reflect this reality.

Ensure the operational integration, in WSDP Phase II, of the National Sanitation Campaign with

programmes for the development of rural water supplies, with appropriate arrangements for

coordinated management and continuing provision for the ring fencing of sanitation budgets.

Ensure that the MOW enhances its current efforts to review and develop the options of mains electricity

connection and solar technologies for rural water supply systems using groundwater.

Ensure that the MOW intensifies its efforts to upgrade field monitoring and data management for the

rural water subsector in complementing the National Sanitation Management Information System that


ix

is being developed. In consultation with the President’s Office – Regional and Local Government, MOW

should build capacity and systems for real time reporting of water supply status from COWSO level

through LGAs to central Ministry databases.

Ensure that the MOW and the MOHSW focus part of the upgraded field monitoring system on routine

collection of a limited set of indicator information that can feed into impact studies – covering, for

example, water quality, amounts of safe water consumed, the extent to which non-scheme water

sources continue to be used, the extent of adoption of sanitation and hygiene practices and time savings

and alternative time uses by women and girls.

Ensure that Government and its DPs in the rural water and sanitation sector undertake another impact

evaluation of RWSSP in five years’ time – long enough for the results and sustainability of these rural

and water sanitation initiatives to be clearly visible. The current study has been undertaken quite soon

after the implementation of the initiatives whose impact was to be tested.

Impact Evaluation of the Rural Water Supply and Sanitation Program, United Republic of Tanzania: Volume I __________________________________________________________________________________________

1

1 Background

1.1 Country context

The United Republic of Tanzania (URT) was formed in 1964 by Tanganyika and Zanzibar, both of which had recently gained independence from Britain. Over its first half century Tanzania has mostly enjoyed political stability. However, economic growth was slow and poverty remained entrenched during most of that period. Tanzania is still classified as a low-income country, ranking 159 out of 187 in the 2014 United Nations Development Programme Human Development Index with a score of 0.488 (in the bottom quartile). Kenya ranked 147 with a score of 0.535; Mozambique, 178 with a score of 0.393 (UNDP, 2015).

The most recent Human Development Report states that “with the exception of some notable progress in a few areas such as child survival (reduction of child mortality rates) and school enrolment, improvements in the overall status of human development in Tanzania are only marginal” (UNDP and GOT, 2015a: viii). The report concludes that “the Millennium Development Goal of reaching a poverty level of 18 percent by 2015 is unattainable” and notes that “poverty in Tanzania is a rural phenomenon” (UNDP and GOT, 2015a: 3, 4).

Over the past decade, however, economic growth has accelerated. Tanzania’s average gross domestic product (GDP) growth rate was 6.9 percent over the period 2004 – 2012 and reached 7.0 percent in 2014. Like Mozambique, Tanzania now has the prospect of major new revenues from large offshore gas reserves that it has started to exploit. New wealth and new equity challenges are in prospect. Meanwhile, however, while rapid economic development is evident in Dar es Salaam and some other large towns, poverty is still widespread among the estimated population of 47m (in 2014).

Approximately 28.2 percent of the population lived below the poverty line in 2012, a reduction from 34 percent in 2007. During the 2007/2012 period, there were improvements in living conditions, access to basic education, health and nutrition and, labor force participation in nonagriculture employment. Nevertheless, these benefits were not distributed equitably. Inequality has increased between urban and rural population and approximately 12 million Tanzanians are still living in poverty.

World Bank, 2015a.

In 2014, Tanzania ranked 47th out of 142 countries in the Global Gender Gap Report (World Economic Forum, 2015). But women continue to be more likely than men to be poor and illiterate; 24.7 percent of female-headed households lived below the poverty line in 2012 (GOT, 2015a: 30).

Climate change is a growing threat to livelihoods and prosperity in Tanzania (ECCT, 2015). While floods are an increasingly significant hazard, many parts of the country are currently suffering severe drought. In October 2015, the chronic national electricity shortages were exacerbated when all hydro-electric plants had to be closed due to a lack of water in their dams (BBC, 2015).

The water and sanitation situation in Tanzania mirrors the growing urban-rural divide and underscores the increasing concern about equitable development in the country. Access to safe drinking water remains a major challenge for many Tanzanians, particularly those in rural communities. According to one recent study, it has declined from 55 percent to 53 percent for rural people over the last two decades – despite high levels of development funding for the sector. Between 1995 and 2005, “Tanzania received USD 57 per beneficiary in aid flows earmarked for water but coverage fell by one percent” (Twaweza, 2014: 2, 3).

According to recent official statistics, only 8.3 percent of the rural population were using an improved sanitation facility, while 34.2 percent of the urban population did so (GOT, 2015a: 82). In 2015, 40 percent of Tanzanian schools were reported to have no water supply; 84 percent had no functioning hand washing facilities and there was an average of one latrine per 56 pupils (SHARE et al., 2015: 10).


2

The most recent assessment of countries’ progress towards the Millennium Development Goals (MDGs) showed Tanzania having made “limited or no progress” towards the targets for use of improved drinking water sources and improved sanitation facilities. For the former it showed 56 percent of the national population using improved sources (compared with 54percent in 1990), and for the latter 16 percent using improved facilities (seven percent in 1990). In urban areas in 2015, it showed 77 percent of the population using improved water sources, compared with 46 percent in rural areas - up just one percentage point over the 1990 figure. For sanitation, urban use of improved facilities was 31 percent, and rural use eight percent of the population - again, one percentage point higher than in 1990 (UNICEF and WHO, 2015: 74-75). However, open defecation is comparatively rare. In 2015, 75 percent of the rural population were estimated to use shared or other unimproved sanitation facilities, while 17 percent practised open defecation - up from 10 percent in 1990 (UNICEF and WHO, 2015: 74).

Figure 1. Water carts are an increasingly common means of transport

1.2 Programme funding and objectives

In 2006, the African Development Bank (AfDB) approved funding of USD 84m for Phase I of the Rural Water Supply and Sanitation Programme (RWSSP) in Tanzania. Phase II of AfDB funding, contributing USD 65m, ran from 2011 to 2015. The RWSSP is a contribution to Component 2 (Rural Water Supply and Sanitation) of the National Water Sector Development Programme (WSDP), 2006 – 2025. According to data provided by its Country Office, the AfDB has disbursed a total of UA 110.58m in loans and UA 10

m in grant funding (USD 151.75m and USD 13.72m at December 2015 exchange rates) to the RWSSP. The United Kingdom Department for International Development (DFID) is another major contributor to this component.

As at 31 December 2015, total revised commitments to the WSDP were USD 1.67bn, of which USD 1.48bn (88 percent) had been disbursed. The government contribution to this total commitment was USD 369m; 100 percent of this commitment had been disbursed.

Phase I of the WSDP was originally planned to end, from the GOT’s perspective, in 2014, although from the donor perspective it continues until December 2015, in order to draw on the funds they provided for that phase. The Ministry of Water (MOW) subsequently extended its implementation of WSDP Phase I to December 2015 as well.


3

As stated for Phase II of RWSSP funding, the sector goal of AfDB support to the programme was “to improve the health and quality of life of the rural Tanzanians through provision of water and sanitation services on a sustainable basis”. The programme objectives were stated to be “to improve district level capacity to implement demand-based RWSS programmes”; “to improve access of rural communities to water and sanitation services operated and maintained by capable persons”; and “to improve health and hygiene practices” (AfDB, 2010: iv).

1.3 Theory of change

In preparation for the evaluation, a theory of change (TOC) was drawn up, based on RWSSP design documentation. It should be noted that RWSSP design itself did not explicitly specify a TOC, although the results frameworks included in RWSSP appraisal reports included elements of one. The principal distinguishing feature of TOC approaches is that they identify the assumptions on which the design logic is based. This can guide a theory-based impact evaluation in its assessment of why intended final outcomes were, or were not, achieved. TOC analysis offers a critical analysis of the relationship between outputs and outcomes and can explain why, in some cases, positive outputs do not achieve the anticipated outcomes.

The discussion below identifies the key assumptions inferred from the TOC, as shown in Figure 2 below and presented in the inception report for this evaluation (Mdadila and Turner, 2015: 5-7).

The TOC focuses on the principal causal links and stages between inputs and final outcomes but adds a preliminary stage of targeting, awareness and planning that should lead to the required inputs for the RWSSP process being in place. A key assumption in this regard is that systems, procedures and budgets are indeed in place, co-ordinated and efficiently implemented. This will enable the targeting, planning and related preparatory processes that lead to the provision of inputs at scale in a manner that matches community awareness and expressed need to available resources. Crucially, the TOC assumes that the policy being implemented by the RWSSP is conducive to efficient achievement of the intended outputs and outcomes.

Whether the inputs shown in Figure 2 achieve the specified outputs depends, in turn, on some important assumptions. One is that design and construction are technically sound. Another is that the new schemes are able to deliver water that is safe for human consumption. User responses are fundamental to achieving the intended outputs. The TOC assumes that communities and households will commit to developing the required user institutions and installing domestic sanitation and hygiene facilities. Another important assumption is that the awareness raising and institutional development carried out by service providers are effective in creating competent Community-Owned Water Supply Organisations (COWSOs) or similar structures. Additionally, the TOC assumes that these user management bodies are able to establish working relations with operation and maintenance (O&M) service providers. Finally, it assumes that the Council Water and Sanitation Team (CWST)1 is competent and adequately resourced to co-ordinate planning and delivery of the intended outputs.

Furthermore, important assumptions link outputs to outcomes in the RWSSP theory of change. For example, the TOC assumes that community members actually use the improved water supply at all times; that the quality of the water remains safe; and that the system continues to deliver this water, in adequate quantities, at all times. A second major assumption is that there is no relapse in sanitation and hygiene practice, and that people continue to use the facilities and practise the enhanced hygiene that have been introduced. Similar assumptions apply to the link between school WASH outputs and the intended outcomes in Tanzanian schools. Moreover, the TOC assumes that the institutional outputs are converted into sustained institutional outcomes: COWSOs or other structures that remain in place and continue to operate and maintain water services effectively with appropriate support from LGAs. This implies an ongoing, constructive relationship of communication and support between LGAs and user institutions. Outputs and outcomes are also linked by the significant assumption that the private sector provides an efficient, effective and affordable maintenance service to water user bodies. A final and vital assumption in the TOC is that the output of an improved water supply in the

1 More commonly described during WSDP Phase I as the District Water and Sanitation Team (DWST).


4

community reduces the amount of time that girls and women must spend collecting water and bringing it to their households. If the number of households per water point is too great, or the supply at the water point is limited, this assumption may not hold true and time savings may be limited or negligible.

The final set of assumptions in the TOC links outcomes to the intended final outcomes: the results that this evaluation aims to test directly. Full health benefits are realized only when drinking water is safe at the point of use; enough safe water is available the whole year round and within a short distance from the household; there is large scale access to toilets that meet the requirement of hygienic separation of human excreta from human contact (through a slab that can be cleaned), and hygienic use of toilets; and when hands are washed with soap or a soap substitute at all critical times. Furthermore it is intended that the interventions will have economic benefits – demonstrated, for example, through a positive effect on local (self) employment in WASH service delivery, and time savings that will improve women’s employment opportunities and/or time spent on income generating activities. Both these sets of results depend on the economic context being conducive – for example, local support enterprises must be able to source inputs and to operate profitably, and there must be economic opportunities for women to exploit with the time saved when water collection is made easier.

This presentation of the RWSSP TOC is thus intended to help identify the key assumptions in the design logic of the Program. It also seeks to identify the factors and processes that the evaluation must explore to explain the extent to which the intended final outcomes (as represented by the proxy indicators shown in Figure 2) have been achieved.

Impact Evaluation of the Rural Water Supply and Sanitation Program, United Republic of Tanzania: Volume I ____________________________________________________________________________________________________________________________________________

_______________________________________________________________________________________________________________________________

5

Figure 2. RWSSP Tanzania: theory of change

Targeting, awareness and planning Inputs Outputs Outcomes Final outcomes

Communities aware, or made aware, of opportunities for - and benefits of - improved water and sanitation

Communities apply for rural water supply and sanitation (RWSS) projects

CWST appraises and prioritises village project proposals, in accordance with RWSSP Project Implementation Manual (PIM)

CWST prepares Council Water and Sanitation Plan

MOW and PMO-RALG review and approve CWSP

Community Project Agreement signed

MOW makes RWSSP funding (from AfDB and other sources) available

Community make contribution in line with policy

Consultants and contractors design and construct domestic water facilities and school WASH facilities

Households build domestic sanitation facilities

CWST service providers deliver institutional facilitation and development

Private sector spares and maintenance services (made) available

Adequate, functioning, safe domestic water supplies available to domestic users within appropriate distance

Improved and functioning domestic latrines and hand washing facilities in place

Adequate, functioning, safe school WASH systems in place

Trained and capable COWSO or other management group in place

Functioning maintenance services available

All community members have adequate supplies of safe drinking water at all times

All community members practise appropriate sanitation and hygiene

Women and girls spend less time on water collection

Girls, boys and teachers all have adequate safe drinking water supplies and appropriate sanitation and hygiene facilities (including menstrual hygiene) at schools

COWSOs or other community RWSS institutions have resources, skills and commitment to operate and maintain RWSS facilities effectively

CWSTs provide effective technical and institutional support to users

Private sector provides affordable, effective maintenance services

Reduced incidence of waterborne disease (proxy indicator: diarrhoea in children under five)

Girls less constrained by water collection and sanitation/ hygiene concerns from advancing their education (proxy indicators: girls’ enrolment and attendance at primary school)

Local employment created through WASH service delivery enterprises

Women less constrained by water collection from pursuing income generating activities


6

1.4 Summary of programme implementation

The WSDP and the RWSSP had a slow start. The WSDP - described as “probably [the] largest national water programme operating in Africa today” (GOT, 2013a: 1) - started one year late, and although a number of ‘quick win’ (mostly rehabilitation) projects that had already been identified and designed could be implemented without too much delay, a 2010 review found that “the main activities under WSDP will start only in its fourth year” (GOT, 2010a: iv). A Programme Implementation Manual was developed, but proved to be too complex for practical application (GOT, 2013a: ii). A restructuring plan was approved in June 2011, including an amended results framework, procurement plan and disbursement indicators. The Programme Implementation Manual was revised, too (GOT, 2014a: 4).

Construction of new rural water schemes only began at scale in 2012, picking up speed in 2013. This followed an extensive period of delay in developing and implementing procurement procedures for consultants to design new schemes – which, once designed, often proved to be more technically sophisticated and expensive than had been envisaged.

The 2013 evaluation of WSDP Phase I concluded that “relatively few of the rural water schemes planned under basket funding have materialised so far and this component is unlikely to make a significant impact on national access figures by 2014. The reasons for the disappointing output appear to be a combination of (a) delays in procurement, (b) irregular and somewhat unpredictable fund releases from the centre (c) limited absorptive capacity at LGA level and (d) the time taken to establish an effective role for the Prime Minister’s Office – Regional and Local Government and achieve good MOW/PMO-RALG coordination. Expenditure in this component has fallen further behind scheduled allocations to a greater extent than the other programme components” (GOT, 2013a: 28).

The water sector was one of six sectors identified for focused development through the national Big Results Now (BRN) initiative (2013/14 – 2015/16). This effort was mainstreamed within the WSDP and led to a renewed commitment to “achieve 67percent rural water coverage by 2015” (GOT, 2014b: 22) through the implementation of 10 rural water projects in each LGA area. This ‘ten villages’ campaign has helped to accelerate RWSSP implementation since 2013. It did not involve significant technical adjustments to the RWSSP approach, but did replace previous efforts at comprehensive district-wide RWSS planning with a focus on a few villages in each LGA – a development deplored in the 2013 Water Sector Status Report (GOT, 2013b: 72).

The National Sanitation Campaign (NSC) resulted from the conversion of the sanitation subcomponent of Component 2 of the WSDP into a separate entity under the coordination of the Ministry of Health and Social Welfare (MOHSW). This restructuring was intended to enhance the performance of the sanitation subcomponent, which had been poor. Since the 2011 restructuring, total investment in the NSC was USD 23.9m, of which USD 19.3m (81 percent) had been released by October 2015 (GOT, 2015c: 3). The four-year target was to construct 1.52m improved latrines and implement school water, sanitation and hygiene (WASH) programmes in 812 schools. By June 2014, only 25 percent of the improved latrines and 16 percent of the school programmes had been achieved (GOT, 2014a: 18). Through the RWSSP, the AfDB has been funding this sanitation component, along with DFID. The Water and Sanitation Programme (WSP) of the World Bank has provided technical assistance. The separate status of the NSC within the WSDP has led to coordination issues at LGA and field levels between rural water and sanitation interventions. Part of the restructuring agreement was an arrangement for the consultants already appointed to work on “ten villages” water schemes to carry out the sanitation subcomponent in those communities. Consequently, this sanitation initiative was separated from the new sanitation work undertaken elsewhere by the NSC.


7

Figure 3. Water sellers' carts in a community with no supply: water drawn from a nearby dam

1.5 Impact evaluation of the RWSSP

The Independent Development Evaluation (IDEV) of the AfDB has commissioned this impact evaluation of the RWSSP. The evaluation was intended:

to provide credible information on emerging development impact of the RWSSP to relevant Ministries and GOT agencies , the Board of the Bank and DPs;

to provide relevant lessons and recommendations to the GOT, Bank management and DPs for sustaining the RWSSP’s benefits, and for informing the design and implementation of Phase 2 of the WSDP;

to help the Bank’s IDEV build the evidence for its 2015 Tanzania Country Strategy Paper evaluation and water supply and sanitation thematic evaluation.

The TOR for this evaluation (Annex 1) gives more detail on its purpose:

The objective of the impact evaluation is to provide credible estimates of the impact of AfDB supported interventions at the level of the ultimate target groups, assess if Programme interventions work as expected, assess sustainability of results and derive lessons and recommendations that can be used to improve the effectiveness of AfDB assistance. The impact evaluation will also be used as an information source for a higher level AfDB Tanzania Country Strategy and Programme Evaluation and a Water Supply and Sanitation thematic evaluation.

The AfDB support for this sector – the RWSSP - was a basket funding contribution to Component 2 of the WSDP. There was no separate technical design for the AfDB input, and this evaluation is therefore concerned with the RWSSP as an input to Component 2 of the WSDP. It is not possible to offer a comparative attribution of impact achieved by different contributors to the basket, such as the AfDB or the World Bank. Effectively, therefore, this is an impact evaluation of Component 2 of Phase I of the WSDP.


8

1.6 Evaluation questions

The TOR also set out a tentative list of evaluation questions (EQs) that the evaluators expanded, in consultation with IDEV, during the inception phase of the assignment. As set out and subsequently approved in the inception report, the 18 EQs that this study will address are as follows. These, rather than the EQs in the TOR, are the questions on which the evaluation is required to focus.

1. What types and quantities of water supply, sanitation and hygiene outputs has the RWSSP achieved at household and school levels?

2. What types, quantities and quality of capacity development has the RWSSP achieved at LGA and community levels with regard to the design, delivery, operation, maintenance and sustainability of WASH interventions?

3. What is the quality of drinking water in beneficiary communities, households and school, at point of supply and point of use?

4. Do the quantity and proximity of improved drinking water supplies2 available to beneficiary households meet international standards?

5. What is the quality of latrines3 in beneficiary households and schools?

6. Are beneficiary households and schools using drinking water as intended by RWSSP design?

7. What time savings are accruing to girls and women as a result of RWSSP interventions?

8. To what extent are the sanitation and hygiene practices promoted by the RWSSP implemented by beneficiary households and schools?

9. Are the changes in water, sanitation and hygiene use and practice achieved by RWSSP interventions uniformly distributed between and within beneficiary communities and households?

10. What targeting, awareness and planning activities precede RWSSP interventions?

11. What are the principal features of the policy that guides RWSSP interventions?

12. What institutional outputs has the RWSSP achieved at community level?

13. What institutional outcomes has the RWSSP achieved at community level?

14. Are the institutional assumptions embodied in the RWSSP TOC valid?

15. Can changes in the incidence of diarrhoea in children aged under five in beneficiary communities – as a proxy for incidence of waterborne diseases – be attributed to RWSSP interventions?

16. Can changes in the enrolment and attendance of girls from beneficiary communities at school – a proxy for enhanced educational opportunity – be attributed to RWSSP interventions?

17. Can changes in the income generating activities practised by women from beneficiary communities – a proxy for enhanced economic opportunity – be attributed to RWSSP interventions?

18. Can changes in local (self) employment in community-based WASH service delivery be attributed to the RWSSP?

19. Are the results of RWSSP interventions technically sustainable?

2 In the framework of the WHO/UNICEF Joint Monitoring Programme (JMP) of MDG improved water and sanitation facilities, an improved water source is defined as one, that by nature of its construction and when properly used, adequately protects from outside contamination, particularly from faecal matter.

3 In the framework of the WHO/UNICEF Joint Monitoring Programme (JMP) of MDG improved water and sanitation facilities, an improved sanitation facility is defined as one that hygienically separates human excreta from human contact. These include pit latrines with slabs.


9

20. Are the results of RWSSP interventions institutionally sustainable?

21. What are the links between technical and institutional sustainability, and are they adequately reflected in the RWSSP TOC?

22. Is the RWSSP TOC replicable at the intended scale?

Table 2 below presents a summary guide to the study’s responses to the evaluation questions, as discussed in chapter 0.

1.7 Scope of the impact evaluation

Although commissioned by the AfDB as an impact evaluation of its RWSSP, the thematic scope of this study is Component 2 of the WSDP. The two programmatic titles are therefore used interchangeably in the report.

It can be seen that the evaluation questions listed above comprise a mix of descriptive, causal and evaluative questions.

Geographically, the evaluation spans mainland Tanzania. Sample communities (section 1.10 au-dessous) were drawn from the whole mainland area.

The period covered is Phase I of the WSDP, from 2006 to mid-2015, six months before the extended close of the Phase in December 2015. This spans Phases I and II of AfDB funding for the WSDP through the RWSSP. However, as explained in section Erreur ! Source du renvoi introuvable., physical implementation was limited in the early years. Implementation only began at scale in 2012. Effectively, therefore, the evaluation must assess the impact of rural water and sanitation efforts since 2012 with many of the schemes commissioned only in 2013 or 2014. As such, it complements the findings of the evaluation of WSDP Phase I carried out in 2013 when it was thought that that phase would end in 2014 (GOT, 2013a).

1.8 Activities during the impact evaluation

The evaluation began with a preparatory mission by the evaluation manager and the consultants, between 26 February and 12 March 2015. During that mission, preliminary consultations were undertaken with government (in particular the MOW), the AfDB and DPs and with potential service providers who were invited to tender to evaluate the performance of the field survey. The mission also undertook familiarisation visits to water schemes, CWSTs and COWSOs in Mkuranga and Kisarawe districts and collected and reviewed relevant documentation.

IDEV is committed to developing evaluation capacity in African governments. It has facilitated initial training on impact evaluation for Government staff, including staff members of the Tanzania MOW. The preparatory mission met two of them to arrange their participation in the evaluation. These officers subsequently took part in some of the fieldwork.

The Centre for Economic Research and Policy (CERP) at the University of Dar es Salaam won the tender for field survey services and carried these out from 3 August to 7 September 2015 after an intensive period of survey instrument design, enumerator training and piloting. Questionnaires were administered in the field in Swahili. Although the main language used during enumerator training was English, the training process involved careful cross-language consolidation and checks to ensure that all staff understood and expressed the questions in the same way in Swahili.

One of the consultants carried out a further mission in October – November 2015 to investigate institutional issues arising from the WSDP in more detail. In addition to further meetings with the MOW, AfDB and other relevant agencies in Dar es Salaam, he visited CWSTs and COWSOs in seven of the 30 LGAs that had been sampled for the evaluation field survey to observe and discuss WSDP institutional arrangements in greater depth. For this fieldwork, he was assisted by one of the CERP survey supervisors. Seven LGAs were selected,


10

within logistical limits, to represent the range of more and less successful schemes as well as technologies, institutional progress, natural conditions and demography.

Following analysis of the field survey and institutions mission data, this evaluation report has been prepared. Details of the methods used are presented in chapter 0 au-dessous.

Methodology

1.9 Impact evaluation design

EQs 15-18 (section 1.6 ci-dessus) are direct impact evaluation questions. Ideally, the ‘changes’ about which they ask would be measured over time with reference to a baseline. No baseline data are available. Differences between treatment and control communities must therefore be analysed as a proxy for the change that might be expected over time as a result of the programme intervention.

Answering the other EQs will be necessary because this is a theory-based evaluation that seeks to assess the accuracy of the assumptions in the programme’s implicit theory of change. Some of these other EQs could ideally be answered from routine performance monitoring data, but in reality not all such data are available. It was therefore necessary to answer them as fully as practicable by reference to interviews, available statistics and field observations.

The key impact evaluation question this study seeks to answer is: “what would have happened to the outcome variables of interest in communities supported by the RWSSP had the programme not been implemented?” Answering this question requires identifying and estimating the counterfactual situation. More specifically, answering it requires estimating outcomes in two states - with and without programme intervention. Estimating the former is straightforward since it is directly observed. Estimating the latter however poses a challenge because it is not directly observed. This challenge is commonly referred to as the counterfactual problem or missing data problem (Heckman et al., 1998).

The goal of an impact evaluation is to establish two groups, participants in the programme (treatment group) and non-participants in the programme (comparison group), that are statistically identical in all other aspects except programme participation/treatment. The challenge then is to identify a valid comparison group, a group with similar characteristics to the treatment group. The most robust way to measure impact would be to randomly assign some communities with programme treatment and keep the remainder as a comparison group. In a way, the average value of an outcome variable of a comparison group serves as a proxy for the counterfactual outcome of the treatment group had it not participated in the programme.

In Tanzania, RWSSP interventions were not randomly assigned. The assignment used a demand-driven approach where, among other things, communities specifically expressed getting improved water supply as their priority. A direct comparison of mean outcomes between treatment and control communities in these circumstances is most likely to produce biased results and compromise the internal validity of the impact evaluation. Nevertheless, a ‘before-and-after’ approach could also be misleading because it is most unlikely that the treatment communities experienced no other changes than the programme treatment. In the real world a number of factors that affect outcome variables of interest might be changing over time (Gertler et al., 2011). In particular, it is difficult to isolate general time trend effects from the actual programme impact (Winters et al., 2010). For these reasons the ‘before-and-after’ approach was not ideal for this impact evaluation and was not applied in this study. Instead, the study used quasi-experimental methods, which are commonly used for estimating impacts due to programme implementation. In particular, the study utilised propensity score matching (PSM) methods; and matched-difference-in-differences, a combination of PSM and difference-in-differences (DID) approaches. Estimating the impact of the programme using PSM and the matched-DID methods is potentially subject to estimation bias emerging from observable heterogeneity across communities. To minimise such a bias, pre-intervention and time invariant community characteristics were taken into account in the estimations (Heckman et al., 1998; Lokshin and Yemtsov, 2005; Smith and Todd, 2005). It should be noted


11

that baseline information was not collected at the inception of the RWSSP. This constrained utilization of some methods such as DiD, which requires at least two data points including the baseline. For this study it was possible to reconstruct baseline information and use the DiD methods for only two variables, related to time taken to fetch water and the distance covered in fetching water.

To increase the comparability of sample observations drawn from both treatment and non-treatment communities, the suggestions by Heckman et al. (1998) were followed: first, both treatment and non-treatment sample observations were drawn from the same geographic locations with similar socio-economic settings. Secondly, similar survey instruments were used to collect data from the respective two study groups. Thirdly, data from both treatment and non-treatment groups were collected around the same period using the same interviewers. To ensure consistency and quality of the collected data, training was provided to enumerators on the survey instruments; a pilot was conducted; and refinement of the instruments was undertaken. Swahili was the medium of communication during the interviews. As part of the training, enumerators and trainers went through the survey instrument and agreed on a common interpretation and translation into Swahili. This was important because the medium of exchange was Swahili and questionnaires were prepared using English. Furthermore, during the fieldwork there were close communication between supervisors and enumerators and between the teams across the zones on the survey challenges and successes. All these were aimed to ensure consistency in the data collected.

Furthermore, apart from employing the PSM and DiD techniques, the study also estimated the impacts of the RWSSP using a weighted-least-squares (WLS) regression. Under the WLS, the programme and non-programme units of observation are weighted by the inverse of estimated propensity scores to estimate impact (Hirano et al., 2003). To check for vulnerability of the estimated PSM results due to the possibility of presence of unobservable characteristics that influence the participation in and the outcomes of the programme, Rosenbaum’s sensitivity test was conducted (Rosenbaum, 2010). The study also examined the influence of the beneficiaries’ pre-existing and exogenous characteristics on the observed/estimated impact of RWSSP interventions. Table 1 in Annex 3 (Volume II) provides definitions and measurements of the outcome variables of the study.

1.10 Sample size and sampling procedures

The empirical data to estimate the impact of the RWSSP in Tanzania were collected from 240 communities and 2,400 households living within them. The sampled observation units were equally divided between treatment (RWSSP) communities and non-treatment (non-RWSSP) communities. A total of 30 LGAs (falling within ten regions of the

country) were randomly sampled and covered in the study. These LGAs were among 79 LGAs that qualified on the required sampling criteria. A LGA qualified for sampling if it managed to complete the construction of at least four

water schemes under the Ten Villages initiatives by July 20144. This approach, sampling only from LGAs with at least four schemes, enhanced the logistical feasibility of the survey. Next, an equal number of RWSSP and non-RWSSP

sample communities and households were randomly selected within each LGA.

Table 1 below summarises the distribution of the sample observations across the Regions. Similarly, Table 3 (Annex 3, Volume II) details sample distribution by LGAs under the study. The descriptions of sample size determination and sampling procedures are also given in Annex 3.

4 A total of 138 LGAs had completed at least one water scheme by July 2014.


12

Table 1. Distribution of the evaluation sample size

Region No. of LGAs

Treatment Sample Control Sample Total

Communities Households Communities Households Communities Households

Geita 3 12 120 12 120 24 240

Kagera 2 8 80 8 80 16 160

Kigoma 1 4 40 4 40 8 80

Kilimanjaro 1 4 40 4 40 8 80

Lindi 3 12 120 12 120 24 240

Manyara 1 4 40 4 40 8 80

Mara 2 8 80 8 80 16 160

Mbeya 2 8 80 8 80 16 160

Mtwara 2 8 80 8 80 16 160

Mwanza 3 12 120 12 120 24 240

Njombe 1 4 40 4 40 8 80

Pwani 2 8 80 8 80 16 160

Rukwa 1 4 40 4 40 8 80

Ruvuma 2 8 80 8 80 16 160

Singida 1 4 40 4 40 8 80

Tabora 1 4 40 4 40 8 80

Tanga 2 8 80 8 80 16 160

TOTAL

30

120

1,200 120 1,200

240 2,400

During the field survey, four LGAs that were initially included in the sample were found not to have completed four water schemes by July 2014. A replacement was done and new LGAs captured in the sample.

1.11 Survey instruments and implementation of field survey

Survey instruments were designed to reflect and explore the theory of change identified for the programme (see section 1.3 au-dessus). The main data used for this impact evaluation were captured using household, community and water point questionnaires. Additional information was collected from primary schools and health centres available within the study area. The household questionnaire, which is the main instrument of the study, captured information on socio-economic and demographic characteristics, education profile, health, usage of water and sanitation facilities, hygiene practices, ownership of assets and engagement in economic activities. Furthermore, the community questionnaire captured information on both past and present characteristics of the sampled communities. In particular, the community questionnaire was used to collect data on population density, water supply, sanitation and hygiene, public infrastructure, land use characteristics, agro-ecology features and major economic activities. The water point questionnaire was used to gather information from sampled RWSSP and non-RWSSP communities on various aspects of primary water sources. The water point questionnaire also collected information about water point management issues and community participation. The school questionnaire recorded enrolment and attendance of boys and girls, availability and use of water and sanitation facilities, hygiene practices, and school participation among boys and girls. Similarly, the health facility questionnaire recorded information on access to water and sanitation facilities, hygiene practices and diarrhoeal outpatient records at the health facility. Lastly, the WASH team questionnaire provided information on how the RWSSP activities are conducted including provision of water supply and sanitation services as well as capacity building at various stages of service delivery. The survey questions were pilot-tested, improved and administered by trained supervisors and enumerators. The survey took place between August and September 2015.

The survey instruments used during the data collection for this study are presented at Annexes 6 – 12 in Volume


13

II. Annex 12 presents semi-structured interview guides used during the October 2015 institutional assessment mission.

1.12 Water quality testing

In communities where the programme had installed water supplies, the survey teams collected water samples at distribution points (DPs) and from storage containers at household level. These samples were stored in clean containers and transported quickly to regional laboratories of the Ministry of Water, where they were tested for E. coli content (section 1.22.1.1 au-dessous).

In every sampled treatment community, three out of ten randomly sampled households were randomly selected for the water quality test. In this way, 30 percent of the treatment households underwent water quality testing. If there were no time and financial constraints, it would have been ideal to undertake water quality tests for all the sampled households, both in treatment and non-treatment communities, and to compare the outcomes. Given the constraints that did exist, water quality testing focused on treatment communities only, and more specifically on 30 percent of the households in treated communities. This sample is statistically reasonable to give credible results given the constraints. Furthermore, testing unimproved water from non-treated communities would not provide very useful information, as the chance that such water is contaminated is already high.

1.13 Post-survey data management and checks

During the field survey, supervisors carried out daily checks of the questionnaires submitted to them by survey staff, identified any gaps and inconsistencies and ensured that these were rectified.

On completion of the field survey, CERP staff captured the datasets digitally. The data were subjected to rigorous checks by CERP during and after capture and subsequently by the consultants. In the course of the capture and iterative checking processes, some coding and related errors were identified. These were rectified by referring back to the raw survey data as collected on the questionnaires.

The post-survey checking processes were protracted and thorough. Satisfactorily clean datasets were only available for analysis by the consultants at the start of November 2015. Two of the 30 LGA questionnaires (see section 1.11) are still missing at the time of writing. This is because the LGA questionnaire included some data that CWSTs were not expected to provide during the interview itself, but were asked to fill in over the following few days before returning the questionnaire to the survey team. In these two cases, the questionnaire was not returned. Although this meant that LGA questionnaire data are not available for two of the 30 LGAs, all other instruments were successfully used in these LGAs.

Only 26 sample households were missing from the final dataset, of which four households were non-response cases. Consequently, after cleaning the data we were left with 2,374 responses, which is 98.92 percent of the intended sample. Again, there were no issues with water point samples. The initial plan was to collect a 30 percent sample from both treatment and non-treatment communities, which translates to 72 sampled water points for each of these groups of communities. Consultations with AfDB suggested that we only focus on treatment water points for the collection of water samples. In other words, we randomly selected 142 water points in treatment communities and collect the samples there. We were ultimately able to sample and collect information from 168 water points, above the intended target by 18 percent.

1.14 Existing empirical evidence

As noted in section 1.11, two sources of existing empirical evidence that were used in the sample scheme areas were the records of health facilities and primary schools. A further source was the files of CWSTs, which these teams were asked to consult in providing the information requested by the LGA questionnaire (Annex 9, Volume


14

II).

At central level, reference has been made to all available documentation on the WSDP and the RWSSP, as well as the broader background of the rural water and sanitation sector in Tanzania. Documents consulted have included records of sector reviews and co-ordination meetings and WSDP/RWSSP planning, reporting and evaluation reports.

However, the extent and quality of performance indicator monitoring data up to 2015 were limited, inter alia because the MOW must rely on CWSTs (which have limited or no resources for site visits) to send it data on the status of rural water and sanitation. This meant that the impact evaluation had to rely primarily on its own field data collection, rather than existing monitoring databases.

1.15 Interviews with informants

From some perspectives, the most important informants for this evaluation are the beneficiaries and local/district personnel who provided information and opinions during the main field survey and the follow-up mission on institutional issues.

Informants in Dar es Salaam, who were interviewed during the preparatory mission and the mission on institutional issues, also provided vital information. During the two missions just mentioned, qualitative data were collected during interviews with CWSTs, COWSOs and water users at distribution points.

For this theory-based evaluation, the views of these informants in the rural water and sanitation sector were essential inputs to analysis of the assumptions in the theory of change and the explanation offered below for the findings of the study on the impact of the RWSSP.

Annex 2 presents a list of persons interviewed.

1.16 Limitations

The principal limitation on this impact evaluation is the fact that the schemes whose impact is reviewed only began construction in 2012 with some only completed in 2014. Schemes completed less than 12 months before the survey period were excluded from the sample (section 1.10). If a scheme is effective in achieving the intended outcomes and impacts, these should be measurable a year after it was installed. Conversely, of course, there is a possibility that any positive impact will have dwindled or disappeared some further years later – for example, because water supply regularity has not been maintained, on account of water quality having deteriorated or since adherence to good hygiene practices has lapsed. The findings of this impact evaluation must therefore be assessed with full recognition of the limited period that elapsed between installation of the schemes and performance of the field survey. We believe that the (limited) findings on impact set out in section 1.22 below are meaningful; nonetheless, a more conclusive analysis will only be possible after some further years have elapsed.

Lack of baseline information was a limitation for the study. By design, the RWSSP did not conduct baseline data collection. Lack of such data made it impossible to use experimental methods. Instead, the study had to rely on quasi-experimental methods, at the expense of getting less credible impact estimates. Similarly, the DiD approach could only be applied for a few questions where recall questions were possible.

It should be noted that, in the ‘programme’ or ‘treatment’ communities that were sampled, sanitation work was carried out – or should have been carried out – by the consultants contracted to design and facilitate the development of the water scheme and its COWSO. Sanitation efforts by the NSC did not take place in these sampled communities but were undertaken elsewhere, at varying distances from the sampled places – potentially also in some of the ‘control’ communities, diluting the power of our comparative analysis.

In terms of data availability, a minor limitation is the lack of two of the planned 30 LGA questionnaires (section


15

1.13 au-dessus). A broader limitation is the general low coverage and poor quality of available field monitoring data from the rural water and sanitation sector (section 1.14 above).

Methodologically, it is important to note that some key information collected during the sample survey is based on respondent estimates, for example, distances to water supplies and time taken or saved in relation to water collection. Given the scope and budget of this study it was not feasible to attempt empirical measurement of these variables.

Findings and conclusions on some issues would have been strengthened if the survey teams had been able to collect more data. For example, the number of households in which water quality samples were taken was limited. The intention to collect health data on ‘treatment’ communities from local clinics proved impracticable because these facilities normally serve many villages and it is difficult to extract data from a particular place. School attendance data were not collected as planned. This study was ambitious in its data collection plans and not all these plans could be fulfilled. This somewhat narrowed the analytical scope of the evaluation but did not invalidate its main findings.

Findings

1.17 Introduction

The presentation of findings in this chapter aims to answer the evaluation questions developed and confirmed during the inception phase of the study (section 1.6 au-dessus). Table 2 below presents a summary guide to the responses to these 22 questions, as presented in chapter 0.

Table 2. Guide to responses to evaluation questions

No. Evaluation question Section Page

1 What types and quantities of water supply, sanitation and hygiene outputs has the RWSSP achieved at household and school levels?

1.20 18

2 What types, quantities and quality of capacity development has the RWSSP achieved at LGA and community levels with regard to the design, delivery, operation, maintenance and sustainability of WASH interventions?

1.20.1 23

3 What is the quality of drinking water in beneficiary communities, households and school, at point of supply and point of use?

1.22.1.1 30

4 Do the quantity and proximity of improved drinking water supplies available to beneficiary households meet international standards?

1.22.1.2 32

5 What is the quality of latrines in beneficiary households and schools? 1.22.1.3 33

6 Are beneficiary households and schools using drinking water as intended by RWSSP design?

1.22.1.5 34

7 What time savings are accruing to girls and women as a result of RWSSP interventions?

1.22.2.1 37

8 To what extent are the sanitation and hygiene practices promoted by the RWSSP implemented by beneficiary households and schools?

1.22.1.4 33

9 Are the changes in water, sanitation and hygiene use and practice achieved by RWSSP interventions uniformly distributed between and within beneficiary communities and households?

1.21.1 25

10 What targeting, awareness and planning activities precede RWSSP interventions?

1.19 17

11 What are the principal features of the policy that guides RWSSP interventions?

1.18 16

12 What institutional outputs has the RWSSP achieved at community level? 1.20 18

13 What institutional outcomes has the RWSSP achieved at community level? 1.21.2 26

14 Are the institutional assumptions embodied in the RWSSP TOC valid? 1.21.3 28

15 Can changes in the incidence of diarrhoea in children aged under five in 1.22.1 29


16

No. Evaluation question Section Page

beneficiary communities – as a proxy for incidence of waterborne diseases – be attributed to RWSSP interventions?

16 Can changes in the enrolment and attendance of girls from beneficiary communities at school – a proxy for enhanced educational opportunity – be attributed to RWSSP interventions?

1.22.2 36

17 Can changes in the income generating activities practised by women from beneficiary communities – a proxy for enhanced economic opportunity – be attributed to RWSSP interventions?

1.22.3 38

18 Can changes in local (self) employment in community-based WASH service delivery be attributed to the RWSSP?

1.22.4 38

19 Are the results of RWSSP interventions technically sustainable? 1.23 39

20 Are the results of RWSSP interventions institutionally sustainable? 1.24 39

21 What are the links between technical and institutional sustainability, and are they adequately reflected in the RWSSP TOC?

1.25 40

22 Is the RWSSP TOC replicable at the intended scale? 1.26 41

1.18 Policy

Tanzania’s Development Vision 2025 includes “universal access to safe water” as a goal that will help achieve “a high quality livelihood for all Tanzanians” (GOT, nd5: 13). Cluster II of the strategic interventions proposed by the National Strategy for Growth and Reduction of Poverty II (MKUKUTA II) concerns ”improvement of quality of life and social well-being”. Goal 4 within this cluster is “increasing access to affordable clean and safe water; sanitation and hygiene”. Its targets include “proportion of households in rural settlements provided with improved sources of water increased from 58.7 percent in 2009 to 65 percent by 2015” and “proportion of population with access to improved sanitation facilities increased”. No specific targets are stated for sanitation (GOT, 2010b: 76-78).

Tanzania published its National Water Policy in 2002. With regard to rural water supply, it states its overall objectives is “to improve health and alleviate poverty of the rural population through improved access to adequate and safe water”. Among the stated specific objectives for this sector are the following goals: (i) “to emphasise on communities paying for part of the capital costs, full cost recovery for operation and maintenance of services as opposed to the previous concept of cost sharing”; (ii) “to depart from the traditional supply-driven to demand-responsive approach in service provision”; and (iii) “to promote participation of the private sector in the provision of goods and services” (GOT, 2002: 30). These represent three key points for this evaluation. The policy expects users to pay for operation and maintenance of their water supply systems in full and to be proactive in proposing and developing new schemes. One of the evaluation questions for this study is whether the policy’s intention of stronger private sector participation is being realized.

The rural water policy is based on a number of principles. These include “water is a basic need and right”; “use of water for human consumption shall receive first priority”; “water is an economic good”; and “improvement of health through sanitation and hygiene education”. The policy emphasises subsidiarity as one of the principles for achieving sustainability: “adopting the principle of managing water schemes at the lowest appropriate level with the beneficiaries themselves establishing, owning and managing their water schemes”. Other principles for sustainability include “ensuring full cost-recovery for operation and maintenance, and replacement”; and “recognising women as being among the principal actors in the provision of rural water supply services” (GOT, 2002: 31).

A key strategy of the policy, on the basis of these objectives and principles, is therefore the legal registration of community structures that will own and operate rural water systems. The policy hinges on two areas of strong

5 nd: no date.


17

participation: firstly, by users and the “communities” of which they are part (the concept of “community” is itself loaded with assumptions); and, secondly, by the private sector, which has traditionally been weak in rural Tanzania.

Another key element of the 2002 policy is its emphasis “on integrating water supply and sanitation services and hygiene education” (GOT, 2002: 35). However, this policy did not formally cover sanitation. A draft National Sanitation and Hygiene Policy has reportedly existed for some years, under the auspices of the MOHSW. Despite various calls for it to be finalised and implemented (for example, Thomas et al. 2013: np6), this policy remains a draft only. Consequentially, despite the water policy’s clear call for integrating water and sanitation efforts, there is a formal policy vacuum with regard to sanitation policy.

1.19 Activities

A number of targeting, awareness and planning activities are meant to precede interventions under Component 2 of the WSDP, which has been undertaken throughout mainland Tanzania. The programme as a whole has been guided by a programme implementation manual (PIM). The first edition of this manual, as noted in section Erreur ! Source du renvoi introuvable. was widely criticised as too complex. A second, simpler edition has recently come into use (GOT, 2015b). However, implementation leading to the interventions whose impact is evaluated here should mainly have been guided by the first edition of the manual, to which this evaluation therefore mainly refers.

The concept of targeting is problematic in a community-driven approach, where users should theoretically target themselves and bring their proposals for water and sanitation schemes forward to the authorities. The first edition PIM identifies a number of early steps that need to be taken at LGA level, including orientation for staff and then “promotion of demand at the community level” (GOT, 2007: 27). It was obviously impossible for CWSTs to undertake this promotion and advocacy throughout their districts. An element of top-down targeting was therefore inevitable, as CWSTs identified the communities with the greatest need for improved water supplies and undertook their stimulation of demand there. At the same time, CWSTs’ selection of communities on which to focus was influenced by messages and appeals for support reaching them from local leaders and representatives.

This promotion of demand by CWSTs (on behalf of their Councils) was meant to lead to community applications for water and sanitation schemes. Further steps for the CWST were therefore meant to be the appraisal of community project applications; the preparation of District Water and Sanitation Plans; and the annual budgeting procedures that were meant to lead to the disbursement of funds from the WSDP for scheme implementation. However, the 2013 evaluation of Phase I of the WSDP found that “few (if any) LGAS have developed district-wide water and sanitation plans” (GOT, 2013a: 29). Instead, after the period of ‘quick win’ projects (often rehabilitation of old schemes) that characterised early implementation of Component 2 of the WSDP, CWSTs came to focus on a few specific community schemes – sometimes large undertakings that developed a central water facility for several communities with lengthy reticulation to DPs scattered across comparatively large areas. This more limited focus was then reshaped as the ‘ten villages’ initiative that has driven implementation of Component 2 since 2013 (section Erreur ! Source du renvoi introuvable.).

6 np: no page number


18

As noted, the first edition of the PIM presents a lengthy exposition of the preliminary and planning steps that should be taken, notably at community level. Those steps include “self-mobilisation” and the election of what the manual called a “Watsan Committee” but was later converted to the concept of a COWSO that would be formally registered as a legal entity. Another important early task for the emerging community structure is collection of the user contribution for construction of the scheme. The PIM set this contribution as 2.5 percent of total cost for more complex schemes, including the common pumped and piped ones as well as gravity schemes; and five percent for hand pump installations. The PIM then required communities to develop “facilities and management plans”, outlining arrangements for design and construction and for O&M. The ultimate result of these processes, following CWST approval and MOW approval of the CWST’s proposed budget, would be signature of a “Community Project Agreement” for the scheme.

Figure 4. Water may be transported several kilometres to communities without a supply

In line with the national policy effort on private sector engagement, consultants played a major role in the preparatory and planning processes (but not the selection of communities to be served, which was done by LGAs on the basis of need and community requests (GOT, 2013a: 29)). Consulting firms and consortia (the latter often combining international and Tanzanian companies) were contracted by LGAs to provide both the ‘soft’ and the ‘hard’ services needed for implementation of WSDP Component 2. Consultants would facilitate the emergence and appointment of local committees (later COWSOs), and train their members. At the same time, they would design the water scheme, prepare the tender documentation and support the CWST in the tender process. Following appointment of contractors, they would go on to supervise construction on behalf of the client. As part of their social services, consultants typically undertook sanitation and hygiene awareness programmes in communities where water schemes were to be constructed. However, these were not formally linked to the NSC, which – as outlined in section Erreur ! Source du renvoi introuvable. – was split off from the WSDP. Formal NSC initiatives linked to the construction of improved latrines were not usually coordinated with support to community water projects by the WSDP.

1.20 Outputs

Different sources of data on the outputs achieved by Phase I of Component 2 of the WSDP are not fully consistent and the information has not been fully verified.


19

The 2014 Water Sector Status report – produced by the MOW at a stage when it was still thought that Phase I of the WSDP was about to end – stated that by 30 June 2014, 32,846 water points had been completed, serving 8,211,500 people. According to this report, the impetus of the Big Results Now initiative in the 2013-2014 financial year led to the installation of 16,784 water points in those 12 months (GOT, 2014c: 14). While BRN did accelerate performance in various government sectors, the increase in WSDP implementation rates was also due to the eventual achievement of broadly common understanding of systems and procedures that could feasibly and efficiently be implemented – following years of slow development of those systems and the required individual and institutional capacities (section 1.21.2 below). However, the 2014 report went on to warn that “82 projects are still not initiated because of lack of financing. This makes reaching the planned target of serving all 1,555 villages by December 2015 difficult” (GOT, 2014c: 18).

According to the draft Aide Memoire of the 15th Joint Supervision Mission for the WSDP (October 2015), LGAs reported a total of 43,255 water points developed during Phase I, that is, since 2007, compared with a target of 31,747. The Aide Memoire pointed out that these were the figures reported by LGAs, which it had not been possible to verify. On the assumption that each water point served 250 people, it was calculated that 10,813,750 rural people had been served with improved water, compared with the target of 7,990,000 (GOT, 2015c: 14). These reported achievements included the early ‘quick win’ projects, which mostly rehabilitated old water schemes rather than building completely new ones.

LGAs are supervised by the Prime Minister’s Office – Regional and Local Government (PMO-RALG)7, which produced an annual report on rural water supply and sanitation projects under the WSDP for the July 2014 – June 2015 financial year. According to this report, 1,045 projects had been completed in 1,384 villages by the end of that financial year (so some projects serve more than one village) with 237 projects awaiting completion with respect to the revised MOW-imposed deadline for the end of WSDP Phase I, 31 December 2015 (GOT, 2015d: 3).

Whatever the data source, it is clear that, over recent years, Component 2 of the WSDP made significant progress with its core output: the installation of rural water supply systems. Whether this output achieves the intended outcomes depends on a number of factors, as the discussion of assumptions in the theory of change makes clear in section 1.3 au-dessus. These include, of course, whether the water supply system actually works and whether the water it delivers is safe at the point of collection and at the point of use.

Table 13 in Volume II shows that, in programme communities surveyed for this evaluation, 82 percent of the primary water sources were reported to be functioning, compared with 54 percent in non-programme communities. According to Table 14 in Volume II, defective design was by far the most commonly mentioned reason for water sources not functioning (accounting for 58 percent of the reason given), followed by over use of the facility (13 percent) and improper use (11 percent).

Leaving aside the precise number of projects or water points achieved, it is important to consider the nature of the water schemes built by WSDP Component 2 during Phase I. Official reporting says little about this issue. The 2013 evaluation of WSDP Phase I noted what this evaluation observed during field visits:

MOW had anticipated that a substantial proportion of the designs would be for relatively low-cost schemes such as boreholes with hand pumps, but in the event most were for higher cost technologies, particularly gravity flow schemes and boreholes with motorised pumps and local distribution networks. The total cost of these schemes exceeded many times over the amount of money that MOW had budgeted and led the ministry to revise their plans, so that the initial number of schemes per LGA was reduced to an average of three, though some LGAs were allowed more.

GOT, 2013a: 29.

7 Following the October 2015 elections, this function was transferred to the President’s Office.


20

The Programme’s heavy dependence on consultants has been explained above. It is not surprising that these consultants tended to design more technically sophisticated schemes. They knew their theoretical operating advantages; they tended to overlook the local management challenges they posed; and the more complex the scheme, the higher the design and supervision revenues that would accrue to them. Very often, the technology introduced by the programme has been boreholes equipped with diesel generators running electric pumps. The water is pumped into an elevated storage tank from which water is reticulated to DPs, sometimes in more than one village.

Influenced by Big Results Now, the aim of WSDP Component 2 in its final years was to install new water schemes in ten villages per LGA. Partly because of the size and complexity of schemes, few LGAs achieved this target. In an initial sample of 30 LGAs reviewed for this evaluation, six had completed ten or more schemes.

Drawing largely on data from the 28 LGAs that returned their questionnaires (section 1.13 au-dessus), this evaluation yielded the following scheme output data.

LGAs had on average started 9.6 new projects since January 2012 with a minimum of five and a maximum of 20. This does not mean that they completed the same number of projects. Persistent funding shortfalls and delays in transfer of funds from central government to LGAs have slowed and interrupted construction of many schemes. District Water Engineers have often had to coax reluctant contractors to continue despite incomplete payment. Some have co-operated; some have slowed or abandoned construction; and some have sued LGAs. CWSTs have been instructed to complete all Phase I projects by 31 December 2015. On completion of this evaluation’s fieldwork in early November, CWSTs had received no funds since the start of the financial year on 1 July. Compliance with this instruction was therefore impossible.

Figure 5. A simple hand pump installed on a borehole

CWSTs were asked to give details of the first four projects they had started since 2012. A total number of 104 projects were reported by 28 LGAs. (Not all completed this section of the questionnaire.) The average number of DPs per project was 15 with a minimum of one and a maximum of 79. The average total scheme cost was TZS


21

370m (USD 167,972), with an average community contribution of 1.89 percent (not all CWSTs provided this information for all schemes).

Delivery of sanitation and hygiene outputs under the WSDP has been dissipated by the transfer of overall responsibility to the NSC, which was poorly coordinated with the installation of new rural water supplies. In communities that were to be served with the latter, the arrangement (according to an NSC informant) was that hygiene and sanitation advocacy should be carried out by the contractors designing the water schemes and facilitating COWSO formation. By June 2015, the sanitation component of the WSDP had reached 58percent of its target of 1.5m households with improved sanitation: 4,383,535 people out of the target of 7,600,000, assuming an average household size of 5.07 (GOT, 2015c: 14; GOT, 2015d: 2). While the same number were meant to have installed hand washing points, only 2,702,965 people were reported to have access to these (GOT, 2015d: 2). These data were as reported by LGAs, and the Joint Supervision Mission that presented them had not been able to verify them.

To date, 16,183 sub villages (3,236 villages) have signed commitments to construct improved sanitation facilities at household level. In the same period, 5,784 villages are covered by a sanitation service provider such as a hardware shop or mason against the target of 600 villages by 2012. On Sanitation and Hygiene in public institutions (schools, health facilities) and public areas - the progress indicates that 1,189 against 812 schools targeted have constructed/rehabilitated their toilets. Of these 715 have been achieved through NSC funds and 474 through other sources such as (the) council’s own source, community initiatives and development partners. In addition, 1,443 schools have active Sanitation and Hygiene clubs to ensure all the latrine facilities are maintained clean.

GOT, 2015d: 2. This evaluation’s field survey covered a total of 128 primary schools, of which 67 were at or near the sampled communities where WSDP Component 2 had been implemented. With respect to the latter group of 67 schools, 76 percent had a drinking water facility, and 51 percent (34 schools) had a functioning drinking water facility. Table 3 below shows how all the schools reported obtained drinking water for their daily use. Asked about sanitation, 66 of the 67 schools said that they had toilet facilities for the pupils and 65 reported that these facilities were in use. Separate facilities for boys and girls were provided at 96 percent of these schools with an average of 66 boys and 66 girls per toilet. A water supply at the toilet for hand washing was reported by 48 percent of the 67 ‘WSDP Component 2’ schools, and 19 percent of those schools said that they supplied soap or a soap substitute for this purpose.

Table 3. Primary schools' sources of drinking water

Source of drinking water Frequency Percent

Own improved source 25 19.53

Own non-improved source 27 21.09

Buying from tankers 1 0.78

Buying from vendors 5 3.91

Hand dug wells 3 2.34

Buying from public pump 3 2.34

Fetching at the river 5 3.91

Buying from the hospital 1 0.78

At home 5 3.91

Shallow well 3 2.34

Deep well 6 4.69

From a near water point 2 1.56

Hotel tank 1 0.78

Everyone carry his own water 6 4.69

School tans 2 1.56

Pond 1 0.78

Rainwater 1 0.78

RWSSP water source 4 3.13


22

Source of drinking water Frequency Percent

Water from public tape 1 0.78

Dams 1 0.78

No use of water services at all 1 0.78

Protected spring 3 2.34

Unprotected spring 6 4.69

Not recorded 15 11.72

Total 128 100.00

Source: impact evaluation field survey.

As shown above, the key institutional output of WSDP Component 2 was meant to be a local user institution – latterly known as a COWSO – that would own, manage and fund the O&M of the rural water supply system. Facilitated and trained in their early stages by scheme design consultants, these bodies have developed slowly through the many formal stages meant to lead to legal registration as corporate bodies. According to the 2014 Water Sector Status Report:

Out of 2,728 planned BRN COWSO establishments by June 2014, only 460 were registered (17 percent). This slowness is partly due to the poorly coordinated flow of the necessary funds to the LGAs concerned, which was allocated to them for undertaking COWSO registration also beyond the WSDP project villages. Even where funds were sent, these arrived at the LGAs without clear instructions about their purpose and intended use, which resulted in these funds being used, instead of for COWSOs registration, for other LGA activities such as monitoring, supervision, and capacity building activities.

GOT, 2014c: 20-21.

Figure 6. Community management structures at 114 water points in 'treatment' villages

Source: impact evaluation field survey.

The field survey for this evaluation covered a total of 114 water points in villages treated by the programme. Figure 6 above shows the frequency with which different types of community water and sanitation management structures were reported to exist at these water points.

0

10

20

30

40

50

60

COSWO Water usergroup

communitywater comittee

None

Percent


23

1.20.1 Capacity development

According to the most recent Joint Supervision Mission of the WSDP,

Some of the challenges identified by the sector include the lack of effective management of O&M of the projects due to weak performance on setting up of COWSOs and wherever set up, building their capacity… The involvement of the community and its representative organisation, the COWSO, should be ramped up, to start right from the beginning, and not as an after-thought. An initial period of community mobilisation, including setting up of the COWSO, will have to be mandatory; the community and the COWSO should be involved in the selection of the technical option, based on affordability (especially of the O&M) and its ability to manage the scheme, followed by its implementation and O&M.

GOT, 2015c: 15.

The 2013 evaluation of the WSDP Phase I remarked that

The amount of external technical assistance available within the programme is surprisingly light, particularly for Component 2. If the right balance is to be struck between investment and capacity development, then implementing agencies should have access not only to occasional training courses but also to ongoing technical support and guidance.

GOT, 2013a: ii.

The WSDP Phase I training manual set out proposed capacity development arrangements for LGAs and communities in some detail, identifying user groups, caretakers and operators as training targets at community level and referring to the availability of a number of training materials. It indicated that the MOW would hire training consultants who would train CWSTs and the consultants who would then undertake institutional and capacity development at community level (GOT, 2007: 65-69).

Aggregate data on capacity development activities actually implemented at these levels are not available. There is no doubt that LGAs have received some training, and that consultants responsible for rural water schemes have facilitated the appointment of user groups or COWSOs, which they have then trained. Table 4 below shows

Figure 7. Members of a new COWSO, assembled for a training session


24

what training the 28 CWSTs told the evaluation survey they had received. It is remarkable that 11 of them said they had received no training at all. This may be an exaggeration, caused by short institutional memories linked to staff turnover and/or enumerator failure to probe. Just over one third of the 28 reported receiving one or more training events on the general management of the WSDP, and associated community level capacity development. Almost as many CWSTs received training on the National Sanitation Campaign and on sanitation and hygiene issues, with specific programmes on Community-Led Total Sanitation (CLTS) also taking place and just three CWSTs reporting training focused on school WASH approaches.

Table 4. Training received by 26 CWSTs, as reported to evaluation survey

Subject area (there may have been multiple training events on the same subject)

No. of CWSTs mentioning one or more training events on this subject

No training received 11

General rural water supply program management, community capacity development

10

National Sanitation Campaign, health, hygiene 8

Community-Led Total Sanitation 8

School WASH 3

Management information system 3

Water point mapping 2

Environmental and social safeguards 1

Report writing 1

Source: evaluation field survey.

At community level, some operators received training on the generators and pumps for which they were to be responsible. Despite the detailed intentions of the PIM, the amount of training that COWSOs and previous user groups received was variable, which is one reason why the levels of responsibility and confidence displayed by these groups also differ widely. Some of the longer-established groups have now been through one or more cycles of elections, resulting in a substantial (sometimes complete) turnover of membership. There do not appear to be systems in place for tracking the arrival of new members and arranging their training.

Among the 114 COWSOs and other user groups surveyed for this evaluation in communities served by the WSDP only three were reported to have received no training. The training subjects that were delivered are shown in

Table 5 below.

During the institutional assessment mission in October 2015, one COWSO reported that they had received no training at all. They said that they had written to the District Water Engineer in 2013 requesting it but had received no reply. Another, however, said that they had received three phases of training (from consultants) before the COWSO was formally inaugurated. They said that more training was needed to keep them updated. A third was new, having replaced a previous group that had been trained. The new body had received no training and did not know, for example, how to read the meters installed at DPs. Another community now had its third user group (a COWSO), which had had some training, including on sanitation and hygiene – unlike the first user group, installed in 2009, which received no training on any subject. Another COWSO, only launched in the last 12 months, reported facilitation by the CWST’s COWSO co-ordinator. The same sort of support was reported by another COWSO, although in this case the consultants also provided training and some members were taken on a sanitation training visit to another region of Tanzania. These random examples illustrate the variability of water and sanitation capacity development at community level.


25

Table 5. Subjects on which COWSOs and other user groups reported receiving training

Training subject No. of COWSOs and other groups reporting training on this subject

General management 31

Technical and maintenance training 27

Financial management 20

Committee procedures 16

None 3

Source: evaluation field survey: ‘treatment’ villages.

1.21 Outcomes

1.21.1 Equity

The National Water Policy makes general commitments to equity, stating for example that “it is important that all members of the community including the disadvantaged groups efficiently and equitably use the water… Communities will ensure the protection and conservation of water sources as well as equitable service provision to economically disadvantaged groups within the communities” (GOT, 2002: 36). One of the principles for sustainability in rural water supply that the policy espouses is “recognizing women as being among the principal actors in the provision of rural water supply services” (GOT, 2002: 32).

National policy does not give more specific guidance on these equity issues. Nor does the first edition of the PIM, which is, however, gender sensitive and states, for example, that “women and men will have equal access and full participation in capacity building and training activities” (GOT, 2007: 60). In practice, many (but not all) COWSOs and other user groups allow severely disadvantaged residents such as the elderly and disabled to obtain domestic water free of charge and the idea of women’s participation management structures is increasingly – but not universally - accepted. Members of one COWSO interviewed by the mission said that it was difficult to get women to take part because of their husbands’ attitudes. This evaluation asked what percentage of the general membership of the local water management structures in treated communities were women, and what percentage of the leadership positions in these structures were held by women? In 84 percent of user bodies for which the information was reported, women made up between 21 percent and 50 percent of the membership of these management structures. In 77 percent of management structures, they held between 21 percent and 50 percent of the leadership positions. The evaluation team were informed that MOW guidelines require women to make up at least 60percent of COWSO membership but no document stating this has been traced.

The same questionnaire asked whether all water users were required to pay for their water, and if not, why not. The question was answered for 79 water points, at 53 percent of which all users were said to be required to pay. At water points where some users did not have to pay, the commonest category of exemption was old age (at 38 water points), followed by disability (31) and poverty (12).

Although comprehensive data are not available, there is general consensus (based on interviews with central and local government and NGO officials, and with community informants) that the benefits of changes in water, sanitation and hygiene use and practice achieved by Component 2 of the WSDP are uniformly distributed between and within beneficiary households. Indeed, in a sense there is some bias towards benefits for women and girls, because they are somewhat more likely than men and boys to enjoy time savings because of the improved services – although much of the time saved may be used for (less risky and physically arduous) domestic chores like cooking (sections 1.22.2, 1.22.3). Little has been done about one aspect of equity, however,


26

which is provision of facilities for menstrual hygiene at school sanitation facilities. Again, no comprehensive monitoring data are available but the anecdotal evidence is that few school latrines offer these facilities. A recent study identified this as a knowledge gap: “how to assist retention of girls at school through the provision of facilities for menstrual hygiene” (Thomas et al., 2013: 60).

The 2013 evaluation of the WSDP stated that:

In Component 2, one positive feature is the use of a transparent formula for the Local Government Capital Development Grant water window, ensuring that every LGA receives some level of funding; without it, funds might be reserved for a few favoured locations. Set against this is the evident bias towards high cost schemes that has arisen in the ‘ten village’ initiative, which limits the number of people who can benefit from WSDP support. On the selection of villages, it appears that pragmatic judgments are made about demand, need and opportunity in the selection of WSDP schemes, but there is no rational basis for equity considerations.

GOT, 2013a: vi.

This review in 2015 reached similar conclusions. There are anecdotal complaints and queries from District Water Engineers about alleged imbalances between districts in Component 2 funding allocations but this has been a genuinely nationwide initiative. Within districts, there are inevitably communities where natural conditions make assurance of a reliable water supply difficult. There is no early prospect of achieving full geographical equity.

1.21.2 Institutional outcomes

The 2013 evaluation of the WSDP noted the inadequacies of the (repeatedly revised) results framework for the programme, and said that “even in Component 2/RWSSP it is very difficult to find comprehensive and consistently-reported data on outputs and outcomes” (GOT, 2013a: 20). There has been no marked improvement in this situation since then. In outlining institutional outcomes, in particular, it is necessary to give a qualitative overview based on observations during the fieldwork for this evaluation.

One frequently offered argument about the early years of the WSDP is that, although they achieved very little physical progress in terms of enhanced coverage of improved rural water and sanitation, they made a major institutional contribution by developing systems, procedures and capacity for the implementation of the programme at scale. This evaluation endorses that argument. In rural water supply, the MOW, PMO-RALG and LGAs know what to do and how to do it. Although staff numbers remain inadequate and operating resources are severely lacking, CWSTs have most of the knowledge and skill that they need and include many able and committed individuals. They are partly able to “provide effective technical and institutional support to users”, as the institutional outcome is stated in the theory of change inferred for Component 2 of the WSDP (Figure 2 on page 5 above). But their effectiveness is compromised by their severe lack of operating resources and recurrent budgets.

Furthermore, one vital, related institutional outcome has not been achieved. Although much of the WSDP system works, it is massively hindered by inefficient management of funding. A study commissioned by DFID in 2013 concluded that “there exist significant shortfalls in the predictability, timeliness and completeness of intergovernmental transfers (these) are increasing and are probably the single greatest factor impeding improved LGA performance and service delivery” (Tanscott Associates, 2013: 83). The study found that:

The current practice of allocation and transfer of funds has resulted in unpredictability of both timing of receipt and amount both of which do not help LGAs in discharging their functions meaningfully.


27

The process of allocating available OC [Other Charges] resources, that is rationing, to LGAs is not transparent and responsibilities for deciding who receives how much and based on criteria, which are not defined.

Whereas there is an accepted (and fairly reliable) basis for PEs [Personnel Emoluments] allocation, there is no comparable practice for OC. This means that under conditions of resource rationing – revenues cannot meet budget requirements – there is no guidance to MOF on what is critical and should therefore be funded in full and what can be scaled down without serious negative impact on LGAs operations.

Tanscott Associates, 2013: vii.

It is perhaps understandable that LGAs’ annual budget requests are not met in full by central government (although in some years, under the influence of BRN, districts received more than they had asked for). This evaluation asked the 30 sample CWSTs how much money they had applied for in each of the last four financial years (2011-12 – 2014-15); how much they received; and how much they spent in each of those years? Not all the CWSTs provided this information, and some of those that did only gave it for some of the four financial years. Overall, however, the average amount received as a percentage of the amount requested was 64 percent in 2011-12; 81 percent in 2012-13; 123 percent in 2013-14; and 44 percent in 2014-15. These averages mask major variations with some CWSTs reportedly receiving more than twice as much as they asked for. There was a general wave of additional funding, associated with BRN, in 2013-14. The data that CWSTs provided on the proportion of funds received actually spent are less helpful because the variation is complicated by carry-overs, in some cases from one financial year to the next. Some LGAs apparently spent only 12 percent of what they received in a particular year, while others spent two or three times as much as they had received.

Planning and management of water and sanitation at LGA level are also complicated by the separation of arrangements for the two subsectors. One respondent CWST wrote that

Budget cycle for water supply and that of sanitation doesn’t match. The NSC budget is prepared by a District Health Officer together with the MOHSW in December. In February NSC budget is incorporated in the Health budget, which is combined with budget from other Departments. Whereas the Water Supply budget is prepared in January by District Water Engineer; in February it is incorporated in the District budget. In April, the District Executive Director presents the District budget containing Health and Water Supply budgets to the Ministry of Finance. Only funds are integrated, plans are always separate!

LGA Wash Team questionnaire survey response.

In general, funds are usually received only several months after the start of the financial year. Financial flows from Dar es Salaam are irregular, uneven and unpredictable. While donor disbursements into the Component 2 basket are also irregular, there are often gaps of several months between those transfers and the arrival of any new money at district level. Particularly during BRN, LGAs were encouraged to commit to more projects than they have ultimately been able to pay for, causing the severe embarrassment – and sometimes lawsuits – referred to in section 1.20 above. The irregularities are compounded by late and uneven disbursements of government’s own contributions to the basket, exacerbated over the last two years by general fiscal difficulties. Until this dysfunctional situation is comprehensively addressed, the overall institutional outcome of the WSDP Component 2 cannot be satisfactory.

In the theory of change (Figure 2 above), the core institutional outcome is “COWSOs or other community RWSS [rural water supply and sanitation] institutions have resources, skills and commitment to operate and maintain RWSS facilities effectively”. This outcome has been only partially achieved. Some members of some COWSOs (and other user bodies) do have the required skills and commitment. But, as outlined above, training has been uneven and incomplete, and there is no comprehensive system in place for the ongoing (re-)training of existing or new cadres of COWSO members. So far, this element of the programme has been treated as a once-off exercise, whereas in fact it will be needed on a recurrent basis.


28

The biggest gap in this institutional outcome concerns the resources “to operate and maintain RWSS facilities effectively”. Few COWSOs have these resources. The amounts of money that COWSOs hold in their bank accounts vary. Some actually have nothing, and live hand to mouth, collecting funds from users as and when needed to buy fuel for their generators or to pay for minor repairs. According to informants, others have over TZS 10m (USD 4,550) in the bank. For 57 COWSOs and other management bodies for which data are available from ‘treatment’ communities, the average amount held at the time of the survey was TZS 3.6m (USD 1,636), with a range from TZS 20,000 (USD 9) to TZS 6.2m (USD 2,818).

While there is general acceptance among rural Tanzanians of the principle of users paying for domestic water, these users very commonly turn to unimproved sources during the rainy season when streams, dams and depressions fill up with water. As a result, some COWSOs collect little money for two or three months a year. Some of those interviewed report collecting nothing at all during the rainy season. Average monthly revenues reported for 72 management bodies in ‘treatment’ villages were TZS 502,943 (USD 229), with a range from TZS 5,000 (USD 2) to TZS 4.1m (USD 1,863).

Even if they were collecting user fees all year round, however, many COWSOs would still be challenged by the heavy cost of buying diesel. Those with pumps connected to mains electricity enjoy lower costs and the minority using gravitational schemes are of course in the best position of all. Furthermore, the consensus across CWSTs interviewed during this evaluation is that there is no prospect of COWSOs being able to pay for anything more than minor maintenance. Typically, COWSOs report problems to the CWST and the CWST sends a technician to help with the repairs – sometimes using spares that the COWSO has been able to buy. Sometimes, also (according to CWST informants), the COWSO pays the field allowance of the visiting technician. Occasionally, they report, the CWST is able to provide the spares or other materials that are needed. If a major repair is required, the CWST includes the cost in its next annual budget – which means that the COWSO must wait many months before there is a prospect of the repair being done.

Consequentially, the institutional outcome at community level is that user bodies have some level of awareness and capacity. Yet, judged overall there is little prospect of the financial autonomy needed to carry O&M costs in the way that national policy expects. This remains the case even though most users are willing to pay for water during months when they cannot find it from unimproved sources.

A further important institutional outcome should be that water, sanitation and hygiene initiatives are coordinated at national, LGA and community levels. This outcome has not been achieved. As reported above, significant numbers of CWSTs and communities report exposure to training on sanitation and hygiene. But fully co-ordinated introduction of enhanced water supplies and sanitation and hygiene campaigns has been the exception rather than the rule – mirroring the poor co-ordination of the NSC with the rest of Component 2 of the WSDP at national level.

1.21.3 Institutional assumptions

Evaluation question 14 for this study asks whether the institutional assumptions embodied in the RWSSP theory of change are valid. The assessment here refers back to the assumptions inferred in the discussion of the TOC in section 1.3 au-dessus.

Systems, procedures and budgets are…in place, co-ordinated and efficiently implemented so that the targeting, planning and related preparatory processes lead to the provision of inputs at scale in a manner that matches community awareness and expressed need to available resources. One part of this assumption is valid so far. Systems and procedures are in place. They are understood and operated with adequate understanding, coordination and efficiency. Budgets, however, are not fully in place and are not efficiently coordinated. As explained above, recurrent funding for CWSTs is wholly inadequate. WSDP funding flows to LGA level are so slow, erratic and inadequate that the overall effectiveness of the programme is gravely compromised.

The policy being implemented by the RWSSP is conducive to efficient achievement of the intended


29

outputs and outcomes. Broadly speaking, Tanzanian authorities and citizens have understood and embraced the National Water Policy as it affects rural water and sanitation. But field research reveals an undercurrent of resentment about the perception that rural people are expected to pay proportionately more for water supplies, receiving in theory no subsidy for their O&M – compared with their urban compatriots. Besides which, the current policy is not conducive to efficient achievement of the intended outputs and outcomes for two reasons. First, poverty drives many users to turn to unimproved sources when they are available, rather than pay for safer supplies. Secondly, there is little prospect of COWSOs, funded by users, being able to fund O&M in full.

Communities and households will commit to developing the required user institutions. Field evidence shows that rural households and communities have supported the development of COWSOs. The concept of community responsibility for and management of local services is well established in rural Tanzanian life, and COWSOs have clear links to the formal structures of village government.

Awareness raising and institutional development carried out by service providers are effective in creating competent COWSOs or similar structures. The institutional facilitation and training work carried out by consultant service providers at community level have, as outlined above, provided the foundations for competent COWSOs. But the quality of this training was uneven, and competent COWSOs will depend on recurrent refresher training and orientation of new members – for which there is no formal provision at present.

User management bodies are able to establish working relations with O&M service providers. This assumption is valid in the sense that COWSOs are quite ready and willing to work with private sector service providers. It is invalid in the sense that they often lack the funds to do so.. The consequent lack of market opportunities restricts the development of private sector capacity.

The CWST is competent and adequately resourced to coordinate planning and delivery of the intended outputs. As argued above, CWSTs are generally competent but they are not adequately resourced.

COWSOs or other structures that stay in place and continue to operate and maintain water services effectively with appropriate support from LGAs. This implies an ongoing, constructive relationship of communication and support between LGAs and user institutions. As argued further in section 1.24 au-dessous, there is as yet no guarantee that COWSOS will stay in place. Much depends on the consistency and quality of support they receive over the next five years. Field observations suggest that there are good, constructive relations between COWSOs and LGAs, although the latter are typically unable to respond to the former’s problems as efficiently as they would wish.

1.22 Impact

1.22.1 Impact on health

Evaluation question 15 for this study (section 1.6 au-dessus) asks whether changes in the incidence of diarrhoea in children aged under five in beneficiary communities – as a proxy for incidence of waterborne diseases – can be attributed to RWSSP interventions.

The impact of RWSSP on reduction of the incidence of diarrhoea is notable for children under five as well as for all persons in beneficiary communities. For children under five years of age, the program achieved a significant reduction in diarrhoeal incidence, by three percent. The impact for all persons was ten percent (see Table 6).

One of the key outcomes intended from implementation of the RWSSP in Tanzania was to improve health outcomes in communities by reducing the incidence of waterborne diseases including diarrhoea. In this evaluation a comparison was done between programme and non-programme communities, using a question that asked about the occurrence of diarrhoea cases during the two weeks prior to the survey. Comparing the two categories of community, the impact of the programme on diarrhoea among the under-fives is statistically significant at the ten percent level. For the population as a whole, the results are statistically significant at the one percent level. Overall, the results of the analysis as indicated in Table 6 reveal that improving access to and


30

quality of water through the programme has exerted a positive impact on community health. Nonetheless, this impact has been considerably less than the 20 – 30percent reduction in diarrhoea revealed by other impact studies of rural water and sanitation (IOB, 2012: 31).

However, Table 12 in Volume II suggests that the average number of outpatient visitors aged five years and under was higher in health facilities within or near programme communities than those in non-programme communities. The respective percentages are 89 (programme) and 76 (non-programme). This observation is consistent with the finding that the impact of the programme on the incidence of diarrhoea is relatively small and less statistically significant on children under the age of five as compared to the population as a whole. Moreover, it could be the case that programme communities have better health infrastructure that attracts and serves outpatient visitors from non-programme communities as well as local residents. It would have been useful to track which community each of the diarrhoea outpatient visitors at the health facilities came from. However, given the format of record keeping in health facilities’ rosters, it was not possible to do so..

Table 6. Impact of programme on incidence of diarrhoea

Variable Estimated impact Standard error

All persons -0.1049*** 0.0225

Children under age of five -0.0304* 0.0147

*** Statistically significant at the 1percent level. * Statistically significant at the 10percent level.

The programme’s modest impact on the selected health indicators is reflected by the fact that the more vulnerable group – children aged under five – appear to have enjoyed a smaller reduction in diarrhoea incidence than the rest of the population. The anticipated benefits of a safe water supply have probably been compromised, as explained below, by the fact that not all the water people in ‘treatment’ communities consume is actually safe – as discussed in sections 1.22.1.1 and 1.22.1.5.

1.22.1.1 Quality of drinking water

As explained in section 1.3 au-dessus, one of the key assumptions in the theory of change for Component 2 of the WSDP (or any similar programme) is that the ‘safe’ water supply it delivers really is safe – both when it is collected and when it is consumed. Based on water quality management requirements, water is supposed to be clean at the point of collection, during transport and when it is stored in the household. Although CWSTs acknowledged in interviews with the evaluation team that they are responsible for periodic testing of water quality at all water schemes, they stated that they do not have the resources to do this on more than an occasional basis. Not only is it a major logistical challenge to visit all the sites, collect samples and deliver them to the regional laboratories of the MOW but funds must be available to pay those laboratories for each test they carry out. In many cases, the only water quality test that does occur is that required before a contractor hands a new scheme over for operation. Without a satisfactory result from this test, the contract should not be certified as complete.

In communities where the programme had installed water supplies, this evaluation tested water at 285 points of use (households) to detect E. Coli bacteria (section 1.12 above). Similarly, water was tested for E. Coli at 67 points of collection (DPs) in these communities. It would have been interesting to compare water quality in ‘treatment’ and ‘control’ communities but with limited resources it was decided to focus on the theory of change assumption with regard to the communities served by the programme.

The results in those communities reveal that water was safe for drinking in only 38 percent of households and at 58 percent of the tested water points. This suggests that unsafe storage of water within households remains a significant challenge, as does the safety of water collected from DPs installed or rehabilitated by the programme.


31

The World Health Organisation (WHO) recommends that in order for water to be safe for human drinking, it should be free of contamination, including E. Coli (WHO, 1997: 6). In order to understand the quality of water used by communities, this study conducted a sample test for 285 households and 67 water points in communities served by WSDP Component 2. The former represents the point of use and the latter represents the source of the water.

Figure 8. Some schemes only supply water intermittently

Table 7 shows the distribution of the E. Coli results by the level of contamination at both the point of use and point of collection. Out of the 285 sampled households, 62 percent used contaminated water as evidenced by exposure to E. Coli of greater than 0 per 100 ml. Similarly, about 41 percent of tested water points revealed presence of contamination in water with 4.5 percent of the water points reporting the E. Coli level of over 50 per 100 ml. Generally, these results suggest that much of the water is contaminated during transportation and/or during storage at the household.

Table 7: Water quality at distribution points and at the point of use

E. Coli concentration per ml No. of households Percent No. of DPs Percent

0 108 37.89 39 58.21

Less than 10 99 34.74 22 32.84

Greater than 10 but less than 50 53 18.60 3 4.48

Over 50 25 8.77 3 4.48

Total 285 100.00 67 100.00

Source: Computations based on household survey and water points data, 2015

Households were also asked to evaluate the quality of the water they obtain from various primary sources. About 70 percent of households in communities served by WSDP Component 2 (RWSSP) believed that the quality of water had improved over time, presumably since programme intervention, although the water quality tests quoted above suggest that this improvement is only relative.

Table 8: Household perceptions of changes in quality of water

Quality of water

Non-programme communities Programme communities

Frequency Percent Frequency Percent

Do not know 1 0.24 2 0.21


32

Improved 137 32.62 650 69.59

Remained the same 212 50.48 184 19.70

Decreased 70 16.67 98 10.49

Total 420 100.00 934 100.00

Source: Computations based on household survey data, 2015

1.22.1.2 Compliance with international standards

Evaluation question 4 (section 1.6 au-dessus) asks whether the quantity and proximity of improved drinking water supplies available to beneficiary households meet international standards. The National Water Policy states that:

The basic level of service for domestic water supply in rural areas shall be a protected, year-round supply of 25 litres of potable water per capita per day through water points located within 400 metres from the furthest homestead and serving 250 persons per outlet.

GOT, 2002: 34.

It is harder to identify agreed international standards for the quantity and proximity of domestic water than it is to find standards on drinking water quality. The Tanzanian national standards quoted above equal or exceed those quoted for various other countries. Common quantity standards are 20 – 25 litres per capita per day (IRC, 2015); the WHO-UNICEF Joint Monitoring Programme stipulates 20 litres per capita per day, of which “at least five litres should come from a safe water source, for drinking, washing food that is not cooked, and for hygiene such as hand washing” (IOB, 2012: 14). Acceptable distance standards between households and DPs vary: in Ghana, they are set at 500 metres (IRC, 2015); in Mozambique, at 1 kilometre (IOB, 2011:13). Undated AfDB guidelines state that:

The practice in many countries is a maximum of 500 metres from the user’s home to the water point. Since water quantity and quality varies widely from one country to the other, country level guidelines would be used for drinking water supply. The recommended basic water requirement for the Rural Water Supply and Sanitation Initiative should be between 20 and 35 l/c/d all year round and within a distance of about 500 metres from the user, and this requirement should be funded by the Initiative.

AfDB, nd: 14.

Table 9 below shows the extent to which WSDP Component 2 has met Tanzanian national standards. It can be seen that the proportion of households in communities where the WSDP Component 2 has installed water schemes that use less than 25 litres per person per day (71percent) is almost as high as the proportion in communities where the programme has not intervened (75percent). Table 5 in Volume II shows that households in communities served by the programme estimated a daily per capita water use of 33.6 litres, compared with 31.4 litres per capita in non-programme communities, where water may be bought from itinerant water sellers or drawn from (sometimes distant) shallow wells or watercourses. In some programme communities, including some of those sampled for this evaluation, the schemes are not functioning – which affects the aggregate consumption figures quoted here.

Table 9. Water consumption per capita per day in programme and non-programme communities

Water consumption per capita per day

Households in non-programme communities

Households in programme communities Total

<25 litres 857 832 1,689


33

(75.18%) (70.69%) (72.9 %)

>25 litres 283 345 628

(24.82%) (29.31%) (27.1%)

Total

1,140 1,177 2,317

(100%) (100%) (100%)

Source: analysis of survey data.

1.22.1.3 Latrines

The theory of change for Component 2 of the WSDP, as explained in section 1.3 au-dessus, assumes that improvements in safe water supply will be accompanied by enhancements in sanitation and hygiene practice – without which the potentially positive impact of safer domestic water on health is compromised. As noted in section 1.1, most people in rural Tanzania use latrines. It is estimated that 17 percent of the population practise open defecation. However, most latrines are of traditional, unimproved varieties. Evaluation question 5 for this study asks about the quality of latrines in beneficiary households and schools. This begs the question as to which ‘beneficiaries’ we are concerned with. Because of poor co-ordination of water and sanitation interventions, many beneficiaries of new water schemes have not been reached by advocacy programmes that might have led them to construct improved latrines. Conversely, the NSC has reached some communities where the WSDP has not installed improved water supplies.

This evaluation’s household survey included questions on sanitation facilities. Assuming residents of communities that have received WSDP water schemes to be ‘beneficiaries’ for the purposes of this evaluation question, Table 8 in Volume II shows the differences in improved latrine use between ‘beneficiaries’ in treated communities and ‘non-beneficiaries’ in control communities. Looking at various categories of latrine, the table shows that 25 percent of households in programme communities had latrines with a pit slab as compared to 20 percent in their counterparts in non-programme communities. Furthermore, the proportion in terms of pit latrines without slabs is higher in non-programme households (66 percent) than in programme households (62 percent). But in both cases the differences are small – again indicating the limited co-ordination of the NSC with WSDP Component 2 water supply programmes in the field.

1.22.1.4 Sanitation and hygiene practices

As explained earlier in this report, the co-ordination of rural water interventions with sanitation and hygiene campaigns has been inadequate under WSDP Component 2. However, the separation of the rural water sub-component from the NSC did not exclude sanitation and hygiene interventions from the rural water schemes. In the communities where these schemes were implemented, the consultants appointed for design and supervision were also expected to carry out the sanitation and hygiene campaigns, as well as facilitating institutional development of COWSOs. Although this study based its sample on LGAs and communities with water schemes, it therefore did not exclude communities where sanitation and hygiene promotion had taken place.

During this evaluation’s field surveys, a number of questions were asked in households and schools about sanitation and hygiene practices.

Households were asked whether any of their members had attended hygiene promotion or training events in the last 12 months, and whether they considered this training had been useful. Very few said yes. In fact, slightly more households in communities where the WSDP Component 2 had not intervened said they had attended such events (six percent) than did households in communities with new water schemes from the programme (five percent).

Households in ‘treatment’ and ‘control’ communities were also asked about their hand washing practices. Responses are summarised in Table 10 below. It can be seen that there is virtually no difference between the two categories of community. The WSDP Component 2 with its implementation of sanitation and hygiene campaigns through water supply consultants in ‘treatment’ communities appears to have had no impact on this variable. Table 9 in Volume II shows that most people still wash their hands without using soap, and that there


34

is little difference between programme and non-programme communities in this regard.

Hand washing is also an important element of hygiene at schools. The availability of hand washing facilities at school latrines is shown in Table 10 in Volume II. Out of 67 schools surveyed in programme areas, 65 (or 97.01 percent) had a functional toilet facility. Similarly, out of 61 schools visited in non-programme communities 57 (or 93.4 percent) had a functional toilet facility. In schools within programme communities the pupil-latrine ratio stood at 38 for boys and 38 for girls, while at schools in non-programme communities they were 59 and 61, respectively. This suggests that students (both boys and girls) in programme communities have better access to latrines compared to their counterparts in non-programme communities. The national average access to school latrines suggests ratios of 54:1 for boys and 51:1 for girls (GOT, 2012). MKUKUTA had set a target of attaining a student-pit latrine ratio of 1:25 for boys and 1:20 for girls by 2015. Presence of hand washing facilities in toilets is still a concern in schools in programme communities, where 48 percent of the sample had a water supply for hand washing – compared with 15 percent of schools in non-programme communities.

Table 10. Availability of hand washing facility within 3m of toilet

Whether household’s toilet facility has hand washing facility within 3 metres

Households in non-programme

communities Households in programme

communities Total

Not applicable 113 100 213

9.66% 8.31% 8.98%

Yes, hand washing facility is available within 3m 290 306 596

24.79% 25.44% 25.12%

Yes, hand washing facility is available but farther than 3m 114 136 250

9.74% 11.31% 10.54

No, Hand washing facility is not present 653 661 1,314

55.81% 54.95% 55.37%

Total 1,170 1,203 2,373

100.00% 100.00% 100.00%


Table 12 in Volume II shows access to latrines and hygiene behaviour at health facilities. About 96 percent of the toilets are functional in health facilities within programme communities, while 90 percent of the toilets are functional in health facilities within non-programme communities. Only 43 percent and 21 percent of toilet facilities in programme and non-programme communities, respectively, have water supply for hand washing. This suggests generally poor hygiene practices in health facilities across all communities.

Table 11 below shows the results of impact estimation for the household latrine variables. Generally, there is limited impact on ‘use of latrine’ as well as on ‘latrine ownership’ due to implementation of the programme: the results are not statistically significant at any level. This is probably due to the limited co-ordination between the water supply interventions and those concerned with enhanced sanitation and hygiene under the NSC.

Table 11. Impact estimation: availability of hand washing facility within 3m of toilet

Variable Estimated Impact SE

Use of Latrine -0.0037 0.0147

Latrine Ownership -0.0137 0.0165

1.22.1.5 Use of drinking water

Evaluation question 6 for this study (section 1.6 au-dessus) asks whether beneficiary households and schools are using drinking water as intended by RWSSP design.


35

Access to improved water differs noticeably between programme and non-programme (treatment and control) communities. While only 18.9 percent of households in non-programme communities have access to improved water, over 67 percent of those in communities served by WSDP Component 2 have access to improved water sources. Similarly, on the question of adequacy, households in programme communities seem to have sufficient water access, particularly in the dry season (see Table 13).

Table 12. Households’ use of drinking water from improved sources

Number of households Mean Standard deviation

Non-programme communities 1,170 0.189 0.385

Programme communities 1,204 0.674 0.478


Table 13. Household perceptions of adequacy of water supply in dry season

Adequacy of water from the source

Non-programme Programme

Frequency Percent Frequency Percent

Do not know 1 0.24 3 0.32

Improved 122 29.12 652 69.96

Remained the same 187 44.63 151 16.20

Decreased 109 26.01 126 13.52

Total 419 100.00 932 100.00


Despite these encouraging findings, interviews with CWSTs and COWSOs indicate that many households use unimproved sources – surface water, rivers and shallow wells – during the rainy season when these are more readily available. This significantly reduces COWSO income and viability and compromises the positive health impacts of the programme. As can be seen from Table 14, nearly 30 percent of households in communities where the programme has installed water schemes reported using rainwater during the rainy season – virtually the same proportion as those in communities without WSDP schemes. Field observation indicates that very few of these households would have obtained this rainwater from safe roof harvesting systems. As the table shows, significant proportions of households in programme communities report use of shallow wells, seep wells and hand dug wells in both dry and wet seasons.

Table 14. Households’ reported sources of water in dry and wet seasons

Variable

Dry Season Wet Season

Non-programme communities

Programme communities



N percent N percent N percent N percent

Piped water inside dwelling 15

1.19 8

0.59 15

0.99 8

0.49

Private outside standpipe/tap 21

1.67 48

3.54 18

1.19 38

2.34

Public standpipe/tap 45

3.57 526

38.79 23

1.52 419

25.78

Water vendor 48

3.81 22

1.62 12 0.79 15

0.92

Tanker service 1

0.08 1

0.07 1

0.07 0

-

Shallow well 134

10.63 153

11.28 97

6.42 138

8.49

Seep well 49

3.89 149

10.99 34

2.25 132

8.12

Hand dug well 340 195 322 181


36

Variable

Dry Season Wet Season





N percent N percent N percent N percent 26.96 14.38 21.32 11.14

Unprotected well 14

1.11 16

1.18 13

0.86 14

0.86

Protected spring 24

1.90 21

1.55 19

1.26 22

1.35

Unprotected spring 246

19.51 88

6.49 208

13.77 68

4.18

River/lake/pond 310

24.58 114

8.41 311

20.60 125

7.69

Rainwater 14

1.11 15

1.11 437

28.94 465

28.62

Total

1,261 100

1,356 100

1,510 100

1,625 100


1.22.2 Impact on girls’ education

One of the RWSSP’s goals is to improve education outcomes in communities where it intervenes by increasing girls’ enrolment and attendance in schools. Evaluation question 16 asks whether changes in the enrolment and attendance of girls from beneficiary communities at school – a proxy for enhanced educational opportunity – can be attributed to RWSSP interventions. The survey addressed this question with respect to primary school enrolment.

The evidence from impact analysis on changes to girls’ enrolment to school due to intervention by WSDP Component 2 is limited. It is important to note that in programme and non-programme communities, the average enrolment rates were above 95 percent. This is explained by the fact that the GOT has been implementing a policy that ensures enrolment chances for all school age children. Similar results are observed for the school attendance variable: the average attendance rate is 89 percent in both types of community.

The result for the impact on girls’ and all other children’s enrolment due to implementation of the programme is not statistically significant at any level (Table 15). The results are similar in the case of attendance and dropouts from primary school, as the table shows.

Furthermore, the analysis from school survey data reveals a similar picture. The school survey captured secondary data from school records about enrolment, attendance and dropouts. Enrolment in standards one to four as well as in standards five to seven was higher in schools within programme communities than in non-programme communities. On the contrary, dropouts were higher in programme than in non-programme communities (7.4 percent and 6 percent in programme and non-programme communities respectively, see Table 11 in Volume II for these results). We were unable to test for statistical significance on these differences due to the few observations in the sample.

Table 15: Enrolment and dropouts at primary school

Variable Estimated Impact S.E.

School enrolment

All Children (6 to 14 years old) -0.0050 0.0100

Boys (6 to 14 years old) -0.0060 0.0161

Girls (6 to 14 years old) -0.0093 0.0136

School attendance

All Children (6 to 14 years old) 0.0331 0.0173

Boys (6 to 14 years old) 0.0444 0.0247

Girls (6 to 14 years old) 0.0284 0.0247


37


School dropout

All Children (6 to 14 years old) 0.0064 0.0077

Boys (6 to 14 years old) 0.0079 0.0114

Girls (6 to 14 years old) 0.0059 0.0109


1.22.2.1 Time savings accruing to girls and women

Evaluation Question 7 asks what time savings are accruing to girls and women as a result of programme interventions.

By improving access to water and reducing the time it takes to collect water, household members, especially women and girls, spend less time in fetching water. Ideally, such time can be used for other productive activities but also for leisure. This assumes, of course, that the scheme provides the required amounts of water consistently and regularly. Survey data, based on household respondents’ estimates of time taken for fetching water, suggest that time has been reduced by 34 minutes - following the implementation of the programme. The results are statistically significant at the one percent confidence level, as is shown in Table 16. These results are for a single difference and do not account for differences between treatment and non-treatment communities before the intervention took place.

To further measure the impact, accounting for the differences between treatment and non-treatment communities prior to RWSSP interventions, the DID estimator, a preferable measure, is used. The bottom rows of Table 16 represent results from difference-in-differences (DID). More specifically, the table shows DID results for the variables “Distance to the water source in metres” and “Total time spent per round trip for fetching water (in minutes)”, which allowed for the use of recall questions. One of the issues with the recall questions is the ability of the respondent to accurately remember the information. In many cases this leads to a number of missing observations due to lack of response on the question. In the current study, the survey team experienced low responses with respect to the distance they used to travel in 2010 to reach the water source, creating some missing observations. This compromised the computations of the DID estimator, as the sample was substantially reduced. Therefore, the estimated impact of 41.6 metres should be viewed with caution due to the missing observations problem. Nonetheless, such results are not statistically significant up to the ten percent level. Contrary to this, the results for “Total time spent per round trip for fetching water” are statistically significant at one percent. The results suggest that the programme had an impact by reducing time taken to fetch water, per round trip, by 82.2 minutes.

Table 16. Time spent by girls and women fetching water


Access to Improved water 0.5448*** 0.0224

Distance to the water source in metres -491.5203*** 37.8343

Total time spent on per round trip for fetching water (minutes) -34.3085*** 5.8432

Daily per capita water use (in litres) (for drinking and cooking) 0.7736 1.3750

Difference in Differences Estimates

Distance to the Water Source in metres 41.6148 123.5352

Total Time Spent on per round trip for fetching water (minutes) -82.2034*** 3.5323 Source: analysis of survey data. *** Statistically significant at the 1 percent level. * Statistically significant at the 10 percent level.

Furthermore, Table 7 in Volume II shows how the saved time is used among household members. Much of the saved time (37.9 percent) is used for cooking, followed by unpaid farm labour (17.3 percent). The latter reflects


38

the dominance of subsistence agriculture in rural areas. The discussion here, in response to the evaluation question, focuses on women - although the table shows that men and boys are also responsible for water collection. Much (but not all) of the saved time is used for cooking - an activity mainly performed by girls and women in households. Comparing households in programme and non-programme communities that table suggests that there is little time saving for the latter group. A general concern is that, as much as women and girls save time from fetching water, it appears that much of it is allocated to non-income generating activities. It could be argued that the welfare of the household at large may improve as more time is available for cooking and taking care of household tasks. Nonetheless, the direct welfare of women may not improve as their participation in income generating activities is less enhanced by the saved time from fetching water. A key factor is probably the lack of opportunities for productive activities but that subject is beyond the scope of the current impact evaluation.

1.22.3 Impact on women’s economic opportunities

Evaluation question 17 asks whether changes in the income-generating activities practised by women in beneficiary communities – a proxy for enhanced economic opportunity – can be attributed to programme interventions.

The impact analysis suggests that programme interventions have significantly increased women’s participation in productive activities, in particular on productive paid labour and on income-generating activities, by 9.1 percent.

Programme interventions were expected to increase employment opportunities as well as saving women’s time from fetching water, which could then be used in to improve their welfare. The result in Table 17 is consistent with this view. Introduction of WSDP Component 2 interventions in communities has increased women’s participation in productive activities by nine percent. The results are statistically significant at the one percent level. It is important to note that despite the impact we observe, women still spend a sizeable amount of the time they save from fetching water on other household activities such as cooking and firewood fetching (see Table 7, Volume II).

Table 17: Impact on women's participation in income-generating activities


All persons (18-64 years of age) 0.0944*** 0.0109

Adult men (18-64 years of age) 0.0978*** 0.0158

Adult women (18-64 years of age) 0.0914*** 0.0151

Source: analysis of survey data. *** Statistically significant at the 1percent level. * Statistically significant at the 10percent level.

1.22.4 Impact on economic activity

Evaluation question 18 concerns potential changes in local (self) employment in community-based WASH service delivery due to the implementation of WSDP Component 2. It might be supposed that sanitation advocacy would lead to significantly stronger demand for artisan services in the construction of latrines. Likewise, the establishment of COWSOs installing, operating and maintaining new water supply schemes should stimulate the growth of locally-based engineering firms capable of taking part in scheme construction and then in providing maintenance and repair services.

According to interviews in a number of districts, there is little sign of this economic growth, although local hardware shops (and, in some cases, NGOs) do generate income from the sale of spare parts and materials. Scheme construction was contracted to larger companies based in Dar es Salaam or other major Tanzanian centres. There was relatively little opportunity for local firms to participate. In general, COWSOs’ maintenance budgets are still so inadequate that they must continue to rely heavily on funding and technical support from CWSTs. They can sometimes buy simple spare parts from local traders and can occasionally employ locally-


39

based technicians for simple, low cost maintenance work. But their combined purchasing power is still too weak to stimulate any significant degree of new economic activity in the water engineering sector. Similarly, analysis of survey data does not reveal employment creation resulting from programme interventions.

Sustainability and replicability

1.23 Technical sustainability

The technical sustainability of the WSDP’s rural water and sanitation interventions is uncertain, for several reasons.

Tanzania is confronted with potentially serious consequences of climate change (section 1.1 au-dessus). Both the groundwater supplies used by most schemes and the runoff and springs used by a minority of gravitational facilities risk lower yields in future – and will be expected to meet higher demand due to population increase and improving living standards.

The commonest technology in schemes developed by the programme is diesel generators powering electric pumps. These diesel engines are prone to breakdowns (some of them easily preventable, for example, damage caused by air locks when fuel runs dry) and have a limited operational life. Current standards of O&M, analysed above, mean that most will not last as long as they should. Current programming and procedures offer no realistic prospect of their replacement, since policy expects users to pay for this and users are plainly unable to do so. Technical sustainability thus links to financial sustainability for COWSOs. From an O&M perspective, the current model of user management and maintenance of rural water supplies is not financially sustainable. Component 2 of the WSDP has been installing schemes that, under current policy, are not financially viable in the medium to long term.

The rest of the technology in use – piping, DPs, raised storage tanks – is relatively robust but still, naturally, subject to wear and tear and in need of periodic repair or replacement. Field observations during this evaluation raise doubts as to whether COWSOs are even financially able to meet these costs.

The ability of the schemes constructed to deliver safe water is not assured and is currently not being systematically tested. Such routine testing is a prerequisite for a technically sustainable rural water supply system.

Schemes have not yet been operational long enough for the above arguments about technical sustainability to be comprehensively supported by empirical evidence. However, basic technical and institutional observation in the field is likely to lead any observer to the same conclusions.

It is understood that an analysis of criteria for support to medium and major maintenance and replacement in rural water supply schemes is being undertaken in 2016.

1.24 Institutional sustainability

Good progress has been made towards assuring the institutional sustainability of the programme’s interventions. As explained by this evaluation, the required structures, systems and procedures are largely in place. Almost a decade after the launch of the WSDP, those concerned know how the rural water and sanitation sector is supposed to work and are mostly competent and committed enough to perform their respective functions effectively. These achievements are enhanced by the steadily increasing engagement of women at all levels in the sector, although further progress is still needed. However, there are four principal constraints on this sustainability, namely:

As explained above, the principal constraint is fiscal administration. Until funds can flow smoothly, predictably and promptly from central government to LGAs, the implementation of decentralised rural


40

water and sanitation programmes will remain dysfunctional. (The option of reverting to a fully centralised model is neither practical nor desirable in the national context of Tanzania.)

In the water and sanitation sector as in other fields of administration and development, local institutions require ongoing maintenance, just like the systems for which they are responsible. The WSDP, like many similar programmes, has given inadequate attention to this longer-term challenge. Unless systems and resources are in place at LGA level for regular, routine training and advisory support to COWSOs, the sustainability of the WSDP’s substantial achievements is not assured.

Given the strong technical interconnections between effective water and sanitation outcomes, close institutional co-ordination between water and sanitation programmes is vital for sustainability. This evaluation has shown that such coordination is lacking, at central government, LGA and community levels.

Private sector capacity for servicing rural water and sanitation remains inadequate. This is because the market for these services is insufficiently developed – due, in turn, to COWSOs’ lack of resources for maintenance and repairs, and the continuing strong dependence on government budgets and technical resources. The stronger links between the public and private sectors that are envisaged by the National Water Policy and the WSDP have not yet been developed. Unless national policies were to be redirected strongly towards public sector control of the economy, this is another significant challenge to institutional sustainability.

1.25 Links between technical and institutional sustainability

Evaluation question 21 (section 1.6 au-dessus) asks what the links are between technical and institutional sustainability, and whether they are adequately reflected in the RWSSP theory of change. The technical and institutional sustainability of rural water and sanitation are interdependent. Durable technical hardware is useless without durable institutions to manage it, and vice versa. Whether a theory of change adequately reflects this depends not so much on the diagram (Figure 2, page 5), as on the assumptions underlying it (sections 1.3 and 1.21.3 above).

The theory of change inferred for the RWSSP/WSDP Component 2 does present a balance of the necessary technical and institutional outputs and outcomes. One interconnection that is not fully emphasised there is that between the type of technology in use and the sustainability of the institutions that operate it. Diesel generator technology challenges COWSOs in three ways: operating costs; the level of technical skill required for O&M; and the need to establish reliable linkages with public or private sector service providers for repairs. These three challenges make it harder for COWSOs using this technology to attain institutional sustainability. Shifting to simpler, cheaper technologies is therefore desirable, where possible: the first prize, gravitational systems, is unfortunately not an option for most communities.

In a similar way, the physical scale of a water scheme challenges the institutional viability, and ultimately sustainability, of the COWSO responsible for managing it. A scheme that spans several villages with a correspondingly bigger generator and several kilometres of pipeline to operate and potentially multiple inter-community rivalries to negotiate, is naturally a more sophisticated management proposition than a single-village scheme with five DPs. So far, the approach of the programme does not recognise this differential in the institutional orientation and support systems that it offers to communities.

Another technical-institutional linkage is implicitly present in the theory of change but deserves emphasis. From a technical perspective, the full benefits of an improved water supply are not realized unless the full benefits of enhanced sanitation and hygiene are achieved at the same time. Failures of institutional co-ordination between the water and sanitation elements of WSDP Component 2, discussed above, diminish the likelihood of achieving these full benefits. Again, the scale of a water scheme is an issue here. The larger the scheme – potentially spanning several villages – the broader the challenge of achieving appropriate sanitation and hygiene practices across a more extensive population. Although no analysis is available to prove the point, the increased effort required to achieve adequate sanitation and hygiene outcomes in a larger scheme is likely to be proportionately


41

greater than the increased design and supervision effort required for the technical elements of pumps, storage and distribution.

1.26 Replicability

The final evaluation question for this study (section 1.6 au-dessus) is whether the theory of change for the RWSSP/WSDP Component 2 is replicable at the intended scale. The answer is a qualified yes. This evaluation has shown that the national approach to rural water and sanitation that the programme has achieved is relatively straightforward and – thanks to substantial institutional effort over several years – is quite well understood. In theory, it should be perfectly feasible to replicate this approach across the many communities in Tanzania that it has not yet reached. Indeed, this is urgently necessary, given the rate of physical performance in WSDP Phase I and the levels of safe water supply and improved sanitation outlined in section 1.1 au-dessus.

However, for this theoretical replicability to be achieved through WSDP Phase II, and for the results to be sustainable, a number of elements in the theory of change, and assumptions underpinning it, must receive serious attention.

One of the inputs shown in Figure 2 on page 5 is “MOW makes RWSSP funding (from AfDB and other sources) available”. As has been argued above, it is impossible to replicate the programme efficiently unless the current dysfunctional funding arrangements are fundamentally reformed.

Replicability is challenged by an institutional approach conceived as a once-off input to establish new community structures. Unless comprehensive institutional maintenance arrangements are in place, WSDP Phase II will proceed as Phase I COWSOs collapse behind it.

If money were no object, it would be feasible to replicate the largely diesel-based technologies of WSDP Phase I across larger numbers of communities until 2025. In reality, replicability will be enhanced if maximum efforts are made to shift away from diesel generators to mains electricity (always assuming that a reliable mains supply can be provided, which is currently not the case) and solar technologies. The MOW, with World Bank support, is already accelerating the introduction of the latter (World Bank, 2015b).

Although the theory of change reflects the importance of achieving appropriate water, sanitation and hygiene outcomes, the useful replicability of the programme depends on effective integration of water scheme development and sanitation and hygiene campaigns at all levels, as argued above.

The most obvious assumption about replicability is, of course, that the necessary capital funding is available. In addition, however, one of the outcomes in the theory of change must be assured: “CWSTs provide effective technical and institutional support to users”. This outcome has not yet been achieved. Although those working in the (often understaffed) CWSTs are typically competent, their recurrent budgets are so severely underfunded that they cannot provide the required effective support, even to the communities reached by WSDP Phase I. There will be little point in replicating these rural water and sanitation approaches if the CWSTs are unable to support them.

Conclusions, lessons and recommendations

1.27 Conclusions

This evaluation concludes that Component 2 of the WSDP Phase I, supported through basket funding by the AfDB’s RWSSP, has made significant institutional and technical progress. The former was a prerequisite for the latter, although it took longer than necessary to lay the foundations for the accelerated rates of implementation that began to emerge in the latter years of Phase I. Performance was stronger at the level of outputs than of outcomes. Although significant, neither technical nor institutional progress has been adequate, nor significant enhancements are still needed in both areas before there can be a genuine prospect of sustainable outcomes, full impact, replicability and achievement of national water and sanitation targets.


42

The terms of reference for the evaluation posited that the intervention might have achieved four types of positive impact. Survey data show that two of these benefits have been achieved, and two have not.

The incidence of diarrhoea (the proxy for health benefits) has reduced significantly: to some extent among children under five years old, and more significantly among the general population in the surveyed villages where the programme has installed new water systems. This is despite the continuing widespread occurrence of E. Coli contamination; many people’s use of unimproved water sources during the rainy season; and the poor coordination of sanitation and hygiene interventions with the introduction of the water schemes – all factors that have probably constrained the positive impact on diarrhoea incidence among young children. Overall, although statistically significant, the health impact of the programme is less striking than might have been anticipated – because some of the assumptions in the programme’s implicit theory of change have not been fulfilled.

As a result of the greater proximity of the new water points, women in communities served by the programme report significant time savings, some of which they are using for economically beneficial activities, although much of it is devoted to household chores. The economic context of rural Tanzania means that the opportunities for converting saved time to higher incomes enhanced livelihoods are limited. Put another way, the opportunity cost of women’s time is very low. While shorter water collection distances are an undeniable benefit in terms of women’s wellbeing, the economic benefits that accrue are limited.

However, although girls may also save time on water collection duties for the same reason, this has not increased their enrolment at primary school, as enrolment rates were already very high.

Local economic growth in the water engineering sector has not been realized at any significant scale. Construction of new schemes was mainly contracted to larger firms based in Dar es Salaam or other major centres, and COWSOs’ maintenance budgets are so small that they cannot support much market development.

Although only two of the four potential benefits researched by this evaluation have been achieved so far, the positive results of this programme are important. Whether they are durable and replicable is far from assured. It depends on a number of climatic, technical and institutional factors. If it is to be a sound and effective investment, WSDP Phase II should take into account the lessons, and adopt the recommendations, that are outlined below.

1.28 Lessons

This impact evaluation suggests a number of lessons that arise from the implementation of the RWSSP within Component 2 of the WSDP, Phase I.

The efficiency and effectiveness of an otherwise sound and competent programme can be seriously compromised by inefficient management of funds at central government level.

A second major factor affecting the viability and sustainability of rural water and sanitation in Tanzania, as currently designed, is recurrent budgeting for LGAs. The current levels of budget subventions to CWSTs gravely compromise the sustainability of WSDP Component 2 achievements to date.

Without reformed financial management systems linking national and local government authorities, and without adequate recurrent funding to CWSTs, the progress and sustainability of Phase II of the WSDP will be as inadequate as Phase I.

For a rural water and sanitation programme to be adequately effective, it is essential that the water and sanitation components be adequately coordinated. The coordination is needed at central and local government levels, and should lead to coordinated implementation at community level.

Despite the apparent general understanding and acceptance among the public in rural Tanzania that water is an economic good for which users should pay, the depth of poverty is such that many people


43

are prepared to turn away from an improved water supply to an unimproved one when the latter is fairly readily available (in the rainy season). This has important implications for the achievement of the health outcomes that improved rural water supplies are expected to achieve.

Policy expectations about the role of the private sector in the rural water and sanitation sector can be fulfilled during the design and implementation phase of new schemes, but current conditions make it difficult for users to pay for private sector services during the O&M phase, and the private sector is consequently poorly developed in rural Tanzania with regard to such services. An ongoing public sector role in support of O&M must be anticipated.

The development of community-based institutions for the ownership and operation of rural water supplies and the continuing advocacy of appropriate sanitation and hygiene should not be viewed as a one-time event. Ongoing institutional maintenance is a prerequisite for sustainability in the sector. The public sector has an ongoing responsibility to ensure that this maintenance takes place – although elements of the required support services could, in theory, be provided by the private sector.

In the technical, institutional, economic and market conditions of rural Tanzania, the use of diesel-based technologies for rural water supply is challenging. Many COWSOs or other user institutions are incapable of operating and maintaining diesel generators efficiently for their full design life. Alternative technologies, where feasible, are likely to be preferable.

1.29 Recommendations

On the basis of the findings of this impact evaluation, we submit the following recommendations.

1.29.1 Policy recommendations

The responsible Ministries should expedite reforms to public finance management systems that ensure predictable, smooth and timely transfer of domestic and donor funding to LGAs for the development, operation and maintenance of rural water and sanitation systems and programmes.

Government should undertake a careful review of affordability issues as they relate to the provision of the National Water Policy for users to pay for improved rural water supplies. While current poverty levels persist, the effectiveness of these improved supplies in enhancing health outcomes will remain compromised while users turn to unimproved water for financial reasons.

Government should recognise that, at least for the remainder of the WSDP period (to 2025), most COWSOs will be unable to pay in full for maintenance and replacement costs of their water supply infrastructure. It should therefore replace the current ad hoc approach to financial support from LGAs to COWSOs for these costs with revisions to the National Water Policy that specify what types of COWSO maintenance and repair cost will be subsidised, and how. It should adjust budgetary subventions to LGAs accordingly.

1.29.2 Programming recommendations

Within feasible fiscal limits, government should strive to increase recurrent funding to CWSTs.

Policy and procedures for rural water supply and sanitation should recognise more explicitly that long-term institutional maintenance is vital for the sustainability of COWSOs and, therefore, for the sustainability of the programme as a whole. With the support of DPs, government should adjust personnel, training and recurrent budgets to reflect this reality.

In WSDP Phase II, government and development partners should operationally integrate the National Sanitation Campaign with programmes for the development of rural water supplies with appropriate arrangements for coordinated management and continuing provision for ring fencing of sanitation budgets.

With the support of development partners, the MOW should expedite its existing efforts to review and develop


44

the options of mains electricity connection and solar technologies for rural water supply systems using groundwater.

Complementing the National Sanitation Management Information System that is currently being developed, the MOW (with the support of DPs) should continue and intensify its efforts to upgrade field monitoring and data management for the rural water subsector. In consultation with the President’s Office – Regional and Local Government, it should build capacity and systems for real time reporting of water supply status from COWSO level through LGAs to central Ministry databases.

The MOW and the MOHSW should ensure that part of the upgraded field monitoring system comprises routine collection of a limited set of indicator information that can feed into impact studies – covering, for example, water quality, the amounts of safe water consumed, the extent to which non-scheme water sources continue to be used, the extent of adoption of sanitation and hygiene practices, and time savings and alternative time uses by women and girls.

The current study has been undertaken quite soon after the implementation of the initiatives whose impact was to be tested. Government and its development partners in the rural water and sanitation sector should undertake another impact evaluation of Component 2 of the WSDP in five years’ time – long enough for the results and sustainability of these rural and water sanitation initiatives to be clearly visible.


I

Annex 1. Terms of reference 1. General background

The Independent Development Evaluation Department (IDEV) of the African Development Bank Group (AfDB) is planning to conduct an ex post Impact Evaluation (IE) of the AfDB supported Rural Water Supply and Sanitation Program (RWSSP) in Tanzania.

The objective of the IE is to provide credible estimates of the impact of AfDB supported interventions at the level of the ultimate target groups, assess if Program interventions work as expected, assess sustainability of results and derive lessons and recommendations that can be used to improve effectiveness of AfDB assistance. The IE will also be used as an information source for a higher level AfDB Tanzania Country Strategy and Program Evaluation and a Water supply and Sanitation thematic evaluation.

The impact evaluation will be designed and conducted as per these Terms of Reference by a team of two experts, a senior evaluator with a background in program evaluation in the sector (expert I), including institutional analysis and sustainability assessment, and a quantitative impact analysis expert (expert II), preferably with relevant experience in Tanzania. The team will be responsible for preparing the methodological design of the IE, for implementing the study as well as for writing the IE draft and final report, under the overall guidance and supervision of IDEV. The terms of reference as part of the contract with Dr. Turner pertain to the services and deliverables to be specifically provided by expert I.

2. Rural Water Supply and Sanitation Program in Tanzania

Tanzania is a vast country with almost one out of every two persons having no access to safe water supply. Due to the vast geographic dispersion, rural Tanzanians often have to travel long distances, consuming many hours to fetch water. This has a huge negative impact on economic development and often results in girls dropping out of schools as they have to join their mothers in fetching potable water.

In 2006 the National Rural Water Supply and Sanitation Program (NRWSSP) was adopted in Tanzania, for the period 2006-2025. It is a long-term plan for the rural water supply and sanitation development to meet the Millennium Development Goals (MDGs) targets and beyond. It is a project supported by several Development Partners (DPs), including the African Development Bank Group (the Bank hereafter).

In 2006 the Bank approved a Rural Water Supply and Sanitation Program Phase I (RWSSP I) (2007-2010) at an amount of UA 55 million (USD 84 million), out of a total requirement of UA 223 million (USD 339 million) under the Water Sector Development Program (WSDP). In the period 2007-2009 some 3,550 water subprojects were implemented, resulting in 8,250 water points serving an estimated additional 1.9 million people. It was furthermore reported that during the same period, over 1.5 million people have been sensitized on sanitation and hygiene and about 370 demonstration latrines built, more than 1,200 artisans trained, and over 524 school sanitation clubs were formed.

The project is due to be completed by the end of 2014 under Phase II of the RWSSP. Prepared and funded jointly with the Government of Tanzania (GOT) and the DPs, the Program includes construction of rural water supply and sanitation infrastructure, national Sanitation Campaigns & School WASH, and Sector Management Support. The Phase II investment is expected to provide potable water to an additional 3.6 million people from new infrastructure and 1.0 million from rehabilitated water infrastructure. This phase has a total cost of UA 200 million (USD 304 million), and the Bank will finance UA65 million (USD 99 million) implemented over a period of four years (2011-2014). In respect to sanitation, the Phase II intended beneficiaries are villagers and school children in 264 schools. Sanitation awareness campaigns have been provided to an estimated ten million villagers in rural communities. For Phase II, the Program’s target area and population is the population of villages in the 132 Local Government Authorities (LGAs). Program villages are selected using criteria including the current water supply and sanitation coverage (the villages with the least coverage will be given highest priority) and the distance travelled to fetch water.


II

The Program is expected to empower local government and communities to provide and sustain WASH services; reduce the incidence of waterborne diseases and health related costs; save time as a result of a reduction in time adults and children spend in water collection and sanitation so that they can use time savings for productive purposes and for going to school. In this way the Program is expected to contribute towards meeting the MDG population coverage targets for access to improved water sources and sanitation and to eradicate poverty.

Key outcome variables of interest include: 1) rural water supply access and coverage; 2) use of improved water source; 3) rural sanitation access and coverage; 4) use of basic sanitation hardware; 5) water quality at source and point of use 6) hygiene and sanitation practices (safe handling of water, hand washing, sanitary conditions); 7) incidence of waterborne and washed diseases 8) travel time for collection from the nearest water point; 7) distance to nearest water point; 8) drop out of female students; 9) productive (self) employment 10) inclusiveness/ distribution of benefits between communities and households. The variables of interest to be taken up in the design of the study will be decided after the IE preparatory mission to Tanzania, planned to take place between 20 January and 15 February 2015.

3. IE questions

The tentative list of key questions that will steer the IE is as follows:

The tentative list of evaluation questions to be addressed is as follows (to be finalized after the preparatory mission):

1. What have been the types of water supply, sanitation and hygiene promotion interventions at the level of final target groups?

2. In what way was the program implemented (focus on the district, community and household level)? How did it evolve and was this in accordance with the Program Theory of Change?

3. What has been the effect on access to and use of improved water sources? 4. What has been the effect on access to and use of sanitary facilities? 5. Is the drinking water at source and point of use safe (compared to WHO norms)? 6. What has been the effect on human water consumption? 7. What has been the effect on health outcome variables of interest? 8. What has been the effect on time used for water collection and sanitary purposes and distance to the

nearest improved water source? 9. What has been the effect on productive (self) employment (inclusive of use of time savings for

productive (self) employment? 10. Have improvements been inclusive (for example between communities classified by welfare indicator,

distance to road, size)? 11. Have the interventions contributed to equal opportunities for women and girls? 12. Are the results sustainable?

a. Have institutional structures and roles been well defined and are these understood and fulfilled?

b. Have the relevant institutions the capacity to perform the required functions? c. Are the facilities provided technically sound? d. Are the facilities and services provided economically and financially viable, including

maintenance and longer term required replacement of infrastructure? e. Do the institutional arrangements provide for adequate monitoring?

13. What has been the role and proportional contribution of the AfDB, as compared to other contributors (communities, governments, DPs)?


III

4. Methodology

In line with the Network of Networks on IE (NONIE) Guidance on IE the methodological design will identify the type of interventions that are subject to IE, identify main outcome variables of interest to stakeholders, articulate the theory of change linking interventions to outcomes, address the attribution problem and use a mixed methods approach to analyse impact8. The ex post IE design will be quasi experimental. A credible comparison group needs to be identified to address the attribution question. In addition, based on a theory on how the interventions are expected to work, critical assumptions will be tested. For the key outcome variables of interest the baseline will - to the extent possible - be reconstructed. IE will include the undertaking of a survey of treated and control communities and households for a selection of regions and samples of communities and households. Statistical and econometric techniques will be used to design the evaluation and estimate net effects. The method for constructing the comparison group, the unit(s) of impact analysis, sample framework(s) and sampling strategy needs to be defined. The feasibility of Difference in Difference design (before and after and with and without comparison) will be assessed depending on possibilities to reconstruct base line information. As interventions have been applied at different periods in time and in different combinations such heterogeneity may be used for the design of the impact evaluation.

In addition, qualitative techniques (based on interviews of full range of stakeholders, observations, Group Discussions in a selection of communities) will be used for complementary analysis of factors that explain findings on effects, assessment of sustainability of program benefits, and an exploration of any pertinent issues that may arise. It will also include an assessment of the sustainability of results and a study of institutional and other determining factors. A separate mission will be fielded for the complementary analysis and assessment of sustainability.

Sources of Data

As part of the preparations the usability of available data for the impact evaluation will be ascertained. To the extent possible, available data sources will be used to estimate and assess the effects of interventions. Sample based community and household level surveys will be designed and administered as far as required to complement available data.

5. Tasks and Responsibilities of the Consultant

This TOR describes the task and responsibilities for the consultants providing services for the impact evaluation. The key roles and responsibilities for the consultant lies in designing and conducting the impact evaluation of the AfDB supported Rural Water Supply and Sanitation Program in Tanzania. The consultants should identify the relevant methodology. Among other activities, they will decide/propose on the appropriateness of the existing secondary data and the need for collecting primary data, define the sample sizes required to estimate effects as well as the sampling strategy. S/he works in consultation with the IDEV supervisory Task Manager and a national research institute/ survey firm in implementing the survey.

The scope of the assignment includes - but is not limited to - the following activities:

Preparatory mission

The IDEV Task Manager, in consultation with the AfDB Field Office, will prepare a ten day preparatory mission to Tanzania together with the consultants. The mission team will be tasked to i) consult the relevant authorities and other key stakeholders on the IE; ii) obtain a good understanding of the Program interventions and final target beneficiary target level (community household, local government) level theory of change by visiting a selection of districts WASH teams and Program sites; iii) check availability of data and develop the methodological design for the IE and iv) discuss the design, tentative work program and role of stakeholders in

8 Source: NONIE IEs and Development; NONIE Guidance on IE, 2009


IV

the IE process with key stakeholders v) identify and short list survey firms that are qualified and capable to conduct the survey if and when required.

Develop the impact evaluation design

The Consultants will develop the impact analysis design to provide credible estimates of net effects of the Bank’s supported interventions on key outcome variables of interest.

Participate in the preparatory mission to understand and properly describe the context and background

of the program, investigate availability of secondary data and its usability for the impact evaluation and

consult authorities and other DPs.

Propose the impact evaluation design and methodology for the assignment at hand.

Sampling size and Strategy S/he will define the counterfactual, sample frame, sample sizes and sampling strategy to estimate Program intervention effects. This will include:

The outcome indicator(s) that are used to determine the sample size(s).

The minimum impacts that the study should be designed to measure.

The number of communities in each of the control and treatment groups.

The number of households and individuals in each community.

Definition of the sample inclusion criteria.

Survey questionnaire development

Develop survey questionnaire for the impact evaluation.

Develop training manual for the conduct of the survey.

Provide training for the supervisors and field workers (enumerators).

Implementation of impact evaluation

Questionnaire pretesting.

Oversee the data collection process.

Provide guideline for data entry and recording.

Communication with other evaluation team members and national researchers involved.

Undertake follow up mission to further study explanatory factors and assess sustainability of the results.

Report writing

Conduct data analysis and validation.

Prepare draft impact evaluation report.

6. Deliverables and Schedule

Two reports are expected as main deliverables of this assignment.

1. Inception report: The inception report will be prepared after the preparatory mission to the country

within a period of one month. At this stage, the report has to provide a detailed description of the

Program, types of interventions and implementation and the detail of the quantitative impact analysis

design and methodology, survey questionnaires, sample designs and so forth. It also has to outline the


V

implementation plan. It is expected that the inception report provides the foundation for the

implementation of the quantitative analysis part of the impact evaluation.

2. Impact evaluation report. This report provides the results of the study of rural water supply and

sanitation, addressing the key questions and accounting for the methodology applied to address the

questions.

The inception report and draft impact analysis report will be reviewed both internally and externally to ensure a high standard of quality of the report. The peer reviewers come from IDEV, other departments in the Bank, Government of Tanzania representatives and an external expert in the field of impact evaluation. The consultants have to address comments and suggestions in revising and producing the inception report and final draft report. All deliverables must be written in English.

The assignment is scheduled to begin in January/February 2015 with the preparatory mission, and end around October/November 2015. The workload is estimated to be 70 working days for each of the consultants. The experts will need to be available and dedicated at key stages of the work. The tentative schedule is shown in the table below.

Table 1. Deliverables and tentative schedule for impact study No. Services Expert 1 Estimate of number of

days

Tentative completion

period

1 IE preparatory mission to Tanzania 10 January-February 2015

2 Inception report, including 3-5 page descriptive

chapter on the Tanzania RWSSP and AfDB

support, on methodology for sustainability and

institutional analysis and inserting contribution

of expert II

10 March-April 2015

3 Contribution to quality assurance of survey,

including contribution to review of proposed

survey and training of enumerators

5 March-April 2015

4 Field mission for sustainability assessment and

institutional analysis and holding of stakeholder

workshop

20 September 2015

5 Drafting of IE report, including inserting

contribution of expert II

20 September-October

2015

6 Communication with expert II, IDEV supervisory

Task Manager, stakeholders, peer reviewer and

survey firms and processing of comments on

inception report and draft IE report

5 February-November

2015

Total number of days 70

No. Services Expert II Estimate of number of

days

1 IE preparatory mission 10 January-February 2015

2 Inception report part on methodological design

for quantitative impact analysis and design and

testing of survey instruments

10 March-April 2015

3 Quality assurance of survey, including

contribution to piloting of questionnaires and

training of enumerators

10 May-June 2015

4 Quantitative impact analysis 20 July-September 2015


VI

No. Services Expert 1 Estimate of number of

days

Tentative completion

period

5 Draft chapter on quantitative impact evaluation 15 September-October

2015

8 Communication with expert I, IDEV Task

Manager, stakeholders, peer reviewer and

survey firms and processing of comments on

inception report and draft IE report

5 February – November

2015

Total number of days 70

7. Required skills and qualifications

The experts should meet the following requirements.

Expert I:

Advanced university degree in social sciences with strong research skills;

At least five years of experience in policy and/or program evaluations in developing countries;

Experience with sustainability assessment and institutional analysis;

Experience with conducting mixed methods IE;

Experience in African countries, in particular in Tanzania, will be a definite advantage;

Prior evaluation experience in multilateral aid organizations/financial institutions is an advantage;

Excellent report writing skills;

Excellent spoken and written proficiency in English;

Proficiency with standard MS suite, along with SPSS or Stata or equivalent.

Expert II

Advanced university degree in economics, with strong technical skills in the area of quantitative IE

techniques (statistical and econometric techniques);

At least five years of prior experience in evaluation and in statistical IE design and implementation of

surveys, of which at least two years in developing countries;

Experience in conducting IE in rural water supply and sanitation will be a definite advantage;

Experience in African countries, in particular Tanzania, will be a definite advantage;

Prior evaluation experience in multilateral aid organizations/financial institutions is an advantage;

Excellent spoken and written proficiency in English;

Proficiency with standard MS suite, namely Excel, along with Stata or equivalent.

8. Management and Reporting

This impact evaluation will be conducted under the guidance of IDEV’s Evaluator General and supervision of the assigned supervisory Task Manager. As described above, the services required from the team of expert pertain to the design, implementation and reporting of the impact evaluation results. These services entail fieldwork, data collection and analysis and report writing and reviewing. In delivering the report the experts are expected to work closely together and in consultation with the IDEV Task Manager, national research institute/ survey firm, and national staff from the concerned ministry in the government. A reference group will be established which comprises IDEV, Bank’s Research Department and Operations Department, Government of Tanzania and external IE expert. The group will comment and make suggestions


VII

on the expected inception report and draft impact evaluation report. The field work, namely the conduction of surveys if needed, will be supported by a national research institute/ survey firm. The terms of collaboration will be discussed and agreed in due time and these services will be commissioned separately.

9. Attachments

AfDB, 2005. Tanzania: Rural Water Supply and Sanitation Program Completion Report AfDB, 2010. Tanzania: Rural Water Supply and Sanitation Program Appraisal Report Ministry of Water, 2004, RWSSP implementation manual


VIII

Annex 2. Persons met

This annex lists persons met during the preparatory mission in March 2015 and the institutional assessment mission in October – November 2015.

M. Abassy National Bureau of Statistics S. Abdallah Member, COWSO, Mavuri, Kilwa District A. Adnan District Education Officer, Kilwa R. Ally Water user, Kilamba, Mkuranga District Aloyce Assistant District Water Engineer, Mtwara B. Augustino Secretary, COWSO, Kigonsera, Mbinga District A. Anthony Accountant, COWSO, Irente, Lushoto District K. Anthony Artisan, COWSO, Irente, Lushoto District L. Balige WASH Facilitator, Kihare, Kisarawe District R.J. Baseki District Water Engineer, Mkuranga Boniface District Water Engineer, Lushoto P. Chacha Director, Central Water Laboratory, MOW R. Chigodi Accountant, COWSO, Kona, Kiteto District A. Chikawe Secretary, COWSO, Kilamba, Mkuranga District C. Chonya Senior Program Co-ordinator, Integrated Water, Sanitation and Hygiene, WaterAid G. Dinda District Trade Officer, Igunga R. Eliasi COWSO, Malala, Igunga District A. Francis Chair, COWSO, Irente, Lushoto District A. Gedi Secretary, COWSO, Malala, Igunga District M. Hausi Assistant Secretary, COWSO, Kigonsera, Mbinga District J. Israel Community Development Officer, Lushoto L. Joackim Acting District Education Officer, Kiteto S. John Member, COWSO, Masoko, Kyela District T. John Member, COWSO, Masoko, Kyela District D. Jones Clinical Officer in Charge, Kilamba, Mkuranga District A. Kahn Economic Development Initiatives Ltd. T. Kandiero Resident Representative, AfDB D. Kasembe School WASH Co-ordinator, Mtwara M. Kayai Member, COWSO, Kona, Kiteto District M. Kessy District Planning Officer, Igunga I. Khalfan Member, COWSO, Mavuri, Kilwa District T. Kida Director of Programs, Economic and Social Research Foundation B. Kilama Senior Researcher, REPOA G. Kimaro Regional Water Engineer, Songea C. Kimele Chair, COWSO, Masoko, Kyela District J. Kionaumela District Water Engineer, Kiteto P. Kishiwa District Health Officer, Igunga A. Kiswaga Assistant District Water Engineer, Mbinga H. Kivunga District Health Officer, Lushoto F. Kiwanga District Health Officer, Kyela T. Kuiwite School WASH Co=ordinator, Ministry of Education and Vocational Training S. Kumburu District Education Officer, Mbinga R. Kwizela Senior Program Co-ordinator, Monitoring and Evaluation, WaterAid A. Kuligani Teacher, Kilamba, Mkuranga District A. Kumar Manager, Water and Sanitation Program, World Bank L. Kwezi Water and Sanitation Adviser, DFID L. Lilai School WASH Co-ordinator, Kilwa


IX

F. Magafu Acting District Health Officer, Mtwara K. Magai District Water Engineer, Kilwa F. Mahela District Education Standards Officer, Igunga G. Mapunda Kihunrwa Social Development Adviser, DFID S. Marandu RWSSP Task Manager, AfDB Country Office F. Masamalo District Planning Officer, Mtwara H. Masangala Chair, COWSO, Mavuri, Kilwa District M. Maulid Member, COWSO, Mavuri, Kilwa District C. Mbele Accountant, COWSO, Kigonsera, Mbinga District E. Mbena Vice Chair, COWSO, Kigonsera, Mbinga District J. Mbuya WASH Accountant, Ministry of Education and Vocational Training I. Mwaka Manager, Central Data Management Team, Ministry of Water E. Mbwille Program Co-ordinator, Integrated Water, Sanitation and Hygiene, WaterAid S. Meku National Bureau of Statistics O. Mfinanga Community Development Officer, Kiteto Mkiwa Acting District Health Officer, Mbinga B. Mkuwa Community Development Officer, Mbinga V. Mndolwa District Water Engineer, Mbinga A. Mohammed Member, COWSO, Mavuri, Kilwa District F. Mohammed Secretary, COWSO, Mavuri, Kilwa District J. Msengi District Water Engineer, Mtwara J. Msukwa Secretary, COWSO, Masoko, Kyela District M.A. Mitili District Water Engineer, Kisarawe Mtweve District Water Engineer, Kyela A. Munga Acting District Health Officer, Kiteto J.L. Mutazamba Assistant Director, Rural Water Supply, MOW A. Mwakitalima National Sanitation Campaign Co-ordinator, Ministry of Health K.B. Mwambuli Water and Sanitation Specialist, Water and Sanitation Program, World Bank E. Mwanakatwe District Crop Services Officer, Lushoto E. Nchimbi Member, COWSO, Kigonsera, Mbinga District M.O. Ndalo Chair, Water Committee, Kihare, Kisarawe District T. Ndembo Member, COWSO, Mavuri, Kilwa District N. Ndiuni Secretary, COWSO, Kona, Kiteto District Y. Ndunguru Community Development Officer, COWSO, Kigonsera, Mbinga District E. Ngonyani Scheme Attendant, COWSO, Kigonsera, Mbinga District J. Njau District Planning Officer, Lushoto R. Nomole Member, COWSO, Kona, Kiteto District S. Ntulila District Community Development Officer, Igunga G. Ntulo District Water Engineer, Igunga A.L. Nyamba Village Executive Officer, Kilamba, Mkuranga District F. Paul Senior Program Co-ordinator, Research, Policy and Advocacy, WaterAid E. Rajab Technician, COWSO, Irente, Lushoto District F.C. Rweyemamu Director, Rural Water Supply, MOW M. Selengeta Member, COWSO, Malala, Igunga District E. Sennoga Country Economist, AfDB, Rwanda H. Shaibu Member, COWSO, Kigonsera, Mbinga District H. Shauri Member, COWSO, Kona, Kiteto District A. Shemnga Acting District Education Officer, Lushoto J. Shenkawe Manager, COWSO, Irente, Lushoto District P. Shetui Member, COWSO, Irente, Lushoto District H. Shija Chair, COWSO, Kona, Kiteto District D. Solobia Acting District Planning Officer, Kyela


X

T. Temba Acting District Planning Officer, Kiteto Y. Tessema Lead Water and Sanitation Specialist, World Bank S. Tilasi Chair, COWSO, Malala, Igunga District C. Wailes Accountant, COWSO, Masoko, Kyela District


XI

References AfDB, nd. Rural Water Supply and Sanitation Initiative. Framework for implementation. Abidjan: AfDB.

AfDB, 2010. Appraisal report: Rural Water Supply and Sanitation Programme II, Tanzania. Tunis: AfDB.

BBC, 2015. Tanzania closing hydropower plants. http://www.bbc.co.uk/news/world-africa-34491984 [accessed 16 November 2015]

ECCT, 2015. The economics of climate change in Tanzania. http://economics-of-cc-in-tanzania.org/ [accessed 16 November 2015]

Gertler, P.G., Martinez, S., Premand, P., Rawlings, L.B. and Vermeersch, C. M. J., 2011. Impact evaluation in practice. Washington, DC: The World Bank.

GOT, nd. The Tanzania development vision 2025. Dar es Salaam: Planning Commission.

GOT, 2002. National water policy. Dar es Salaam: Ministry of Water.

GOT, 2007. Water Sector Development Programme 2006 – 2025. Programme implementation manual. Volume 1. Dar es Salaam: Ministry of Water.

GOT, 2010a. Water Sector Development Programme. Phase1 mid term review. Dar es Salaam: Ministry of Water and Irrigation.

GOT, 2010b. National strategy for growth and reduction of poverty II. Dar es Salaam: Ministry of Finance and Economic Affairs.

GOT, 2012. Basic education statistics. Dar es Salaam: National Bureau of Statistics.

GOT, 2013a. Final report. Water Sector Development Programme, Tanzania. Evaluation of PhaseI. Dar es Salaam: MOW.

GOT, 2013b. Water sector status report. Dar es Salaam: Ministry of Water.

GOT, 2014a. The water sector status report 2014. Marking the end of WSDP PhaseI. Dar es Salaam: Ministry of Water.

GOT, 2014b. Water Sector Development Programme PhaseII ((2014/2015 – 2018/2019). Dar es Salaam: Ministry of Water.

GOT, 2014c. The water sector status report 2014. Marking the end of WSDP phase-1. Dar es Salaam: Ministry of Water.

GOT, 2015a. Tanzania in figure [sic] 2014. Dar es Salaam: National Bureau of Statistics.

GOT, 2015b. Water Sector Development Programme 2006 – 2025. Programme Implementation Manual. http://www.maji.go.tz/?q=en; navigate to WSDP PIM [accessed 17 November 2015]

GOT, 2015c. 15th Joint Supervision and Implementation Completion and Result (ICR) Mission, Water Sector Development Programme (WSDP), October 8 – 20. Draft aide memoire. Dar es Salaam: Ministry of Water.

http://www.bbc.co.uk/news/world-africa-34491984

http://economics-of-cc-in-tanzania.org/

http://www.maji.go.tz/?q=en


XII

GOT, 2015d. 15th Joint Supervision and Implementation Completion and Result (ICR) Mission, Water Sector Development Programme (WSDP), October 8 – 20. Draft aide memoire. Annex: sanitation and hygiene services. Dar es Salaam: Ministry of Water.

GOT, 2015d. Water Sector Development Programme. Sector Coordination Department. Annual progress report. Dar es Salaam: Prime Minister’s Office – Regional Administration and Local Government.

Heckman, J., Ichimura, H. and Todd, P. 1998. Matching as an econometric evaluation estimator. Review of Economic Studies 65: 261-294.

IOB, 2011. More than water. Impact evaluation of drinking water supply and sanitation interventions in rural Mozambique. The Hague: Ministry of Foreign Affairs, Policy and Operations Evaluation Department.

IOB, 2012. From infrastructure to sustainable impact: policy review of the Dutch contribution to drinking water and sanitation (1990 – 2011). The Hague: Ministry of Foreign Affairs, Policy and Operations Evaluation Department.

IRC, 2015. Monitoring service levels. http://www.ircwash.org/news/service-levels [accessed 19 November 2015]

Mdadila, K. and Turner, S.D., 2015. Impact evaluation of the Rural Water Supply and Sanitation Programme, United Republic of Tanzania. Inception report.

SHARE, London School of Hygiene and Tropical Medicine, ICDDR-B, IIED, Shack/Slum Dwellers International and WaterAid, 2015. Research report. Process evaluation of the National Sanitation Campaign of Tanzania. Draft.

Tanscott Associates, 2013. Study on mapping of transfer of funds to Local Government Authorities (LGAs). Dar es Salaam: Ministry of Finance.

Thomas, J., Holbro, N. and Young, D., 2013. A review of sanitation and hygiene in Tanzania. Morogoro: MSABI.

Twaweza, 2014. Money flows, water trickles. Challenges of access to clean water in Tanzania. Sauti za wananchi brief 10. Dar es Salaam: Twaweza.

UNDP, 2015. International human development indicators. http://hdr.undp.org/en/countries [accessed 16 November 2015]

UNDP and GOT, 2015. Summary Tanzania human development report 2014. Dar es Salaam: Economic and Social Research Foundation.

UNICEF and WHO, 2015. Progress on sanitation and drinking water. 2015 update and MDG assessment. http://www.wssinfo.org/fileadmin/user_upload/resources/JMP-Update-report-2015_English.pdf [accessed 16 November 2015]

WHO, 1997. Guidelines for drinking-water quality. Volume 3. Surveillance and control of community supplies. Geneva: WHO.

Winters, P., Salazar, L. and Maffioli, A., 2010. Designing impact evaluations for agricultural projects. Inter-American Development Bank Office of Strategic Planning and Development Effectiveness: Impact-Evaluation Guidelines Technical Notes IDB-TN-198.

http://www.ircwash.org/news/service-levels

http://hdr.undp.org/en/countries

http://www.wssinfo.org/fileadmin/user_upload/resources/JMP-Update-report-2015_English.pdf


XIII

World Bank, 2015a. Overview: Tanzania. http://www.worldbank.org/en/country/tanzania/overview [accessed 16 November 2015]

World Bank, 2015b. Using solar energy to power water supply in Tanzania. http://www.worldbank.org/en/news/feature/2015/12/07/using-solar-energy-to-power-water-supply-in-tanzania?CID=WAT_TT_Water_EN_EXT [accessed 17 December 2015]

World Economic Forum, 2015. Global Gender Gap Report rankings, 2014. http://reports.weforum.org/global-gender-gap-report-2014/rankings/ [accessed 16 November 2015]

http://www.worldbank.org/en/country/tanzania/overview

http://www.worldbank.org/en/news/feature/2015/12/07/using-solar-energy-to-power-water-supply-in-tanzania?CID=WAT_TT_Water_EN_EXT

http://www.worldbank.org/en/news/feature/2015/12/07/using-solar-energy-to-power-water-supply-in-tanzania?CID=WAT_TT_Water_EN_EXT

http://reports.weforum.org/global-gender-gap-report-2014/rankings/

MEMORANDUM - Independent Evaluationidev.afdb.org/sites/default/files/documents/files/Tanzania -...

Documents

Transcript of MEMORANDUM - Independent Evaluationidev.afdb.org/sites/default/files/documents/files/Tanzania -...