Triangulation review and assurance: ICF · ICF makes big things possible . ICF is a global...

Triangulation review and assurance: ICFChapter 5: 3rd Party report

Document Reference: T3002

This report includes an independent review and assurance of our approach to triangulating different sources of customer research and data to produce overall conclusions on the value to customers of service performance, and of our use of these values in setting incentive rates commissioned by United Utilities and Your Voice.The overall objective was to assess whether the approach taken and judgements were reasonable and proportionate, and to verify that the customer valuations and their use are appropriate, meaningful and accurate. ICF concluded that we had taken a sound and appropriate approach to triangulating customer evidence to arrive at proposed customer valuations, performance commitments and incentives rates.

United Utilities Water Limited

Review and assurance of United Utilities plc's approach to triangulating customer evidence for PR19 Final Report

17 August 2018

ICF makes big things possible

ICF is a global consulting and technology services provider with more than 5,000 professionals focused on making big things possible for our clients. We are policy specialists, social scientists, business analysts, technologists, researchers, digital strategists and creatives. Since 1969 government and commercial clients have worked with ICF to overcome their toughest challenges on issues that matter profoundly to their success. Our five core service areas are described below. Engage with us at icf.com.

Review and assurance of United Utilities plc's approach to triangulating customer evidence for PR19 - Draft Final Report

i

Review and assurance of United Utilities plc's approach to triangulating customer evidence for PR19 Final Report

A report submitted by ICF Consulting Services Limited

Date: 17 August 2018

Job Number 30301823

Joe Sunderland

ICF Consulting Services Limited

Watling House

33 Cannon Street

London

EC4M 5SB

T +44 (0)20 3096 4800

F +44 (0)20 3368 6960

www.icf.com


i

Contents

Executive summary .................................................................................................................. 1

1 Introduction ................................................................................................................ 4

1.1 Background ............................................................................................................... 4 1.2 Structure of this document ......................................................................................... 5

2 Objectives and approach ........................................................................................... 6

2.1 Objectives .................................................................................................................. 6 2.2 Assurance framework ................................................................................................ 6 2.3 Assurance approach .................................................................................................. 8 2.4 Applying the framework ............................................................................................. 9

3 Findings ................................................................................................................... 12

3.1 Summary of assurance findings .............................................................................. 12 3.2 Initial assurance findings ......................................................................................... 16 3.3 Final assurance findings .......................................................................................... 17 3.4 Summary of improvements made by United Utilities plc ......................................... 18

4 Conclusions ............................................................................................................. 19

4.1 Concluding remarks ................................................................................................. 19

ANNEXES .............................................................................................................................. 20

Annex 1 Detailed findings ...................................................................................................... 21

Annex 2 Detailed method ...................................................................................................... 37

Annex 3 Summary of evidence sources for each performance measure .............................. 42


1

Executive summary

Background and context

United Utilities plc and YourVoice commissioned ICF to undertake an independent review and

assurance of:

■ United Utilities plc’s approach to triangulating (i.e. combining and synthesising) different

sources of customer research/data for the purposes of producing overall conclusions on

the value to customers of service performance.

■ United Utilities plc’s use of the resulting values, and other information, to set proposed

performance targets and incentive rates. This included the detailed review of five selected

performance measures.

The aim was to provide United Utilities plc and YourVoice with independent and objective

expert assurance surrounding United Utilities plc’s approach to triangulating the underlying

evidence that it uses to inform its customer valuations and its use of those values in setting

performance targets and incentive rates for PR19. The overall objective for this assurance

exercise and supporting framework was to assess whether the approach taken, and overall

judgement used by United Utilities plc, is reasonable and proportionate, and to verify that the

customer valuations United Utilities plc derived – and their subsequent use – are appropriate,

meaningful and accurate.

Assurance approach

United Utilities plc’s triangulation approach was assessed against an assurance framework to

reach an overall rating and conclusion surrounding the extent to which the customer valuations

United Utilities plc has derived – and their subsequent use – are appropriate, meaningful and

accurate.

Each stage/step in the assurance framework can be summarised broadly as follows:

■ Sources identified – this step considered whether United Utilities plc had identified and

taken account of relevant sources of customer research/data to underpin its customer

valuations.

■ Findings identified – this step considered the extent to which United Utilities plc

accurately identified the findings from each evidence source against each question it was

trying to answer.

■ Evidence assessed – this step involved reviewing whether and how United Utilities plc

weighed-up different pieces of customer research/evidence and compared and contrasted

findings.

■ Proportionality demonstrated – this step considered whether United Utilities plc took a

proportionate approach in identifying, assessing and triangulating customer evidence.

Findings

To apply the framework, United Utilities plc’s documents were assessed against a set of

questions relating to relevant sections of the framework described above.

Ratings were assigned against each question in the detailed framework, then aggregated to

provide an overall rating for each customer valuation category. The overall aggregate ratings

denoted the following assurance outcomes:


2

■ A low rating indicated a high number of – and/or material – omissions, errors or failures to

explain the rationale. It reflected an overall aggregate score (across all questions in the

framework) of 50 per cent or less of the total possible awarded marks1.

■ A medium rating indicated some omissions, errors or failures to explain the rationale, but

fewer and/or less important than those given a low rating. It reflected an overall aggregate

score (across all questions in the framework) of between 51 per cent and 80 per cent of

the total possible awarded marks.

■ A high rating indicated few or no omissions, errors or failures and was awarded only where

such omissions, where they existed, were not significant and where the overall aggregate

score was above 80 per cent of the total possible awarded marks.

The initial and final ICF assurance scores across all service performance measures reviewed

are summarised below, demonstrating the improvements made by United Utilities plc during

the course of this assurance exercise.

Table ES1.1 Assurance scores across all service performance measures

Inte

rru

pti

on

s

Le

ak

ag

e

Riv

er

qu

ality

Flo

od

ing

inc

ide

nts

Ta

ste

an

d

od

ou

r

Dis

co

lou

rati

on

Po

llu

tio

n

inc

ide

nts

Su

pp

ort

ing

vu

lne

rab

le c

st.

Wa

ter

po

ve

rty

Wa

ter

qu

ality

Initial assurance scores

Triangulation Med Med Med Low Med Med Low Low Low Low

Valuation use Med Low N/A Low Low N/A Low N/A N/A N/A

Final assurance scores

Triangulation High High High High High High High High High High

Valuation use High High N/A High High N/A High N/A N/A N/A

Conclusions

ICF’s review of United Utilities plc’s triangulation and valuation use documentation resulted in

the following conclusions:

■ United Utilities plc has a strong and varied evidence base from which to draw its customer

valuations, reducing the reliance on any single source.

■ United Utilities plc’s triangulation documentation uses clear and transparent

communication to present this evidence base.

■ United Utilities plc’s triangulation documentation and valuation use documentation

explains clearly how it has used triangulation to generate customer valuation estimates

1 Each question resulted in a rating being awarded: low (=1), medium (=2) or high (=3). If the total ‘score’ for a service performance measure across all questions in the assurance framework was less than 50 per cent of the total possible marks that could be awarded, that service performance measure was rated overall as ‘low’.


3

that reflect the evidence base, as well as using those customer valuation estimates

appropriately.

■ United Utilities plc’s incentive rates – and performance commitments – link back to the

underlying evidence and, where judgement has been used to derive a proposed incentive

rate and/or performance commitment, this is transparent.

■ United Utilities plc’s business plan content reflects a well-evidenced and clear approach

to triangulation, which largely follows ICF’s triangulation guidance developed in a previous,

separate study for CCWater.

Overall, ICF concludes that United Utilities plc has taken a sound and appropriate

approach to triangulating its customer evidence to arrive at proposed customer valuations,

performance commitments and incentives rates.


4

1 Introduction This document is the Final Report of ICF Consulting Services Limited’s (ICF’s) work

to review and assure United Utilities plc’s application of a framework/approach to

compare, contrast and triangulate multiple sources of customer evidence and data to

estimate the value its customers place on certain service improvements. This review

and assurance exercise also considered the use of those customer valuations in

setting incentives rates for United Utilities plc’s 2019 price review (PR19) business

plan.

1.1 Background

The Water Services Regulation Authority (Ofwat) uses price controls to regulate the

allowed revenues of water and wastewater companies and water-only companies in

England and Wales. Companies’ customer service and the nature and extent of their

engagement with customers forms an important part of these price controls.

Ofwat outlined its expectations for customer engagement in its 2016 customer

engagement policy statement2, setting higher-level principles for company good

practice. Companies must demonstrate that they understand their customers’ needs

and priorities and are responsible for developing customer valuations and using those

to inform their business plans based on robust, proportionate and balanced evidence.

Ofwat’s review of the 2014 price review (PR14)3 indicated that companies could

improve their customer engagement by widening the scope of customer evidence

used to support their PR19 business plans. This includes expanding the range and

application of customer valuation methods and complementing stated-preference

methods – including willingness-to-pay (WTP) evidence – with revealed preference

and other customer data and evidence.

The assurance process outlined in this report is part of United Utilities plc’s efforts to

ensure it meets these new standards.

The role of customer valuations in PR19

Customer valuations should underpin company business plans across the test areas but they hold particular importance in the tests for engaging customers, delivering outcomes for customers, securing cost efficiency and addressing affordability. Avoiding a one-size-fits-all approach means that Ofwat’s ‘Initial Assessment of Business Plans’ (IAP) tests will need to develop ways to measure the quality of the customer valuation research underpinning business plans.

PR19 continues the standard set by PR14 for improved customer engagement. It builds on the principles established in PR14 and sets a framework for a more holistic, in-depth and robust approach to customer engagement. Using a richer evidence base is at the core of this approach. Ofwat’s updated policy statement on customer engagement reflects on the scope for using alternative approaches, such as revealed preference (RP) research, for PR19. Ofwat also proposed meeting with companies during 2018 to better understand their approaches to customer engagement in advance of PR19. Anecdotal evidence suggests there is interest across the water industry to move away from traditional survey approaches

2 Ofwat (May 2016), Ofwat’s customer engagement policy statement and expectations for PR19, available at: https://www.ofwat.gov.uk/wp-content/uploads/2015/12/pap_pos20160525w2020cust.pdf 3 Ofwat (July 2015), ‘Reflections on the price review – learning from PR14’, available at: https://www.ofwat.gov.uk/publication/reflections-on-the-price-review-learning-from-pr14/

https://www.ofwat.gov.uk/wp-content/uploads/2015/12/pap_pos20160525w2020cust.pdf

https://www.ofwat.gov.uk/publication/reflections-on-the-price-review-learning-from-pr14/


5

and to pursue research that allows customer preferences to be explored in more “real world” settings.

1.2 Structure of this document

The remainder of this document is structured as follows:

■ Section 2 describes the assurance approach;

■ Section 3 presents findings; and

■ Section 4 presents conclusions.

The annexes to this report contain the following supporting information:

■ Annex 1 provides a more detailed description of the assurance findings;

■ Annex 2 presents a detailed description of the method of approach; and

■ Annex 3 lists the evidence sources triangulated by United Utilities plc for each

service performance measure.


6

2 Objectives and approach This section provides an overview of United Utilities plc’s and YourVoice’s objectives

for the work and ICF’s approach.

2.1 Objectives

United Utilities plc and YourVoice (the panel that succeeded United Utilities plc’s

Customer Challenge Group (CCG)) commissioned ICF to undertake an independent

review and assurance of:

■ United Utilities plc’s approach to triangulating (i.e. combining and synthesising)

different sources of customer research/data for the purposes of producing overall

conclusions on the value to customers of service performance. This involved a

detailed ‘deep dive’ review of five measures and a lighter review of five other

measures.

■ United Utilities plc’s use of the resulting values, and other information, to set

proposed performance targets and incentive rates. This included the detailed

review of five selected performance measures.

The aim was to provide United Utilities plc and YourVoice with independent and

objective expert assurance surrounding United Utilities plc’s approach to triangulating

the underlying evidence that it uses to inform its customer valuations and its use of

those values in setting performance targets and incentive rates for PR19. The overall

objective for this assurance exercise and supporting framework was to assess

whether the approach taken, and overall judgement used by United Utilities plc, is

reasonable and proportionate, and to verify that the customer valuations United

Utilities plc derived – and their subsequent use – are appropriate, meaningful and

accurate.

This report is intended for United Utilities plc to use in communicating the results of

this assurance review, including as supporting evidence in its PR19 business plan

submission to Ofwat.

2.2 Assurance framework

The overarching framework for reviewing and assuring United Utilities plc’s

triangulation approach was derived from the work ICF completed for the Consumer

Council for Water (CCWater) on defining and applying triangulation in the water

sector.4 That earlier work developed a framework for triangulation that set out practical

steps that water companies can take to triangulate customer data/evidence based on

good practice principles. ICF therefore used that framework to design a transparent

and robust assurance approach for this work (summarised below in Figure 2.1).

4 ICF (2017), Defining and applying 'triangulation' in the water sector: How water companies can use different sources of customer evidence in business planning, https://www.ccwater.org.uk/wp-content/uploads/2017/07/Defining-and-applying-triangulation-in-the-water-sector.pdf.

https://www.ccwater.org.uk/wp-content/uploads/2017/07/Defining-and-applying-triangulation-in-the-water-sector.pdf



7

Figure 2.1 Triangulation assurance framework – key elements

Each stage/step in the assurance framework can be summarised broadly as follows:

■ Sources identified – this step considered whether United Utilities plc had

identified and taken account of relevant sources of customer research/data to

underpin its customer valuations.

■ Findings identified – this step considered the extent to which United Utilities plc

accurately identified the findings from each evidence source against each

question it was trying to answer.

■ Evidence assessed – this step involved reviewing whether and how United

Utilities plc weighed-up different pieces of customer research/evidence and

compared and contrasted findings.

■ Proportionality demonstrated – this step considered whether United Utilities plc

took a proportionate approach in identifying, assessing and triangulating customer

evidence.

■ Conclusions tested – this step considered the extent to which United Utilities plc

engaged with external stakeholders to test the outputs of the process as a whole5.

United Utilities plc’s triangulation approach was assessed against each element of

the above framework to reach an overall rating and conclusion surrounding the extent

5 United Utilities plc’s triangulation approach was not assessed against the ‘conclusions tested’ element of the assurance framework. This is because of the point in time at which ICF’s review was undertaken, given further consumer testing was planned but not yet undertaken by United Utilities plc. This is not to say that elements of United Utilities plc’s approach were not tested – acceptability testing was undertaken in relation to selected service performance measures (and the extent to which the resulting evidence was triangulated by United Utilities plc was assessed against other elements of ICF’s assurance framework). Further, because YourVoice jointly commissioned this assurance work, they were directly engaged and involved in ICF’s work to assure United Utilities plc’s triangulation approach.


8

to which the customer valuations United Utilities plc has derived – and their

subsequent use – are appropriate, meaningful and accurate.

2.3 Assurance approach

The approach to the assurance work took into consideration the following:

■ The approach aimed specifically to provide assurance in relation to the

triangulation of evidence to identify customer valuations and the use of those

valuations to set incentive rates. This did not include reviewing the validity of the

customer research carried out and presented in the underlying evidence base.

■ The scope of work did not include an in-depth review of all customer valuations

that United Utilities plc provided. ICF took a sampling approach, based on an in-

depth review of five customer valuations, with a more high-level review of another

five customer valuations.

■ The selection of measures6 for the in-depth review was agreed between ICF,

United Utilities plc and YourVoice, with United Utilities plc and YourVoice

confirming the final selection, based broadly on the extent to which the measure

was a priority for customers, whether the resulting customer valuation was used

to inform multiple incentive rates and whether the measure was a priority area of

interest for United Utilities plc and/or YourVoice.

■ The assurance process followed an iterative, multi-review approach (shown below

in Figure 2.2), whereby ICF reviewed documentation provided by United Utilities

plc, then provided detailed feedback on how improvements could be made, before

United Utilities plc produced revised documentation for ICF to review again against

the same assurance framework. Both the triangulation documents (Step 1) and

the valuation use documents (Step 2) underwent three iterations of review and

assurance.

A detailed step-by-step description of the assurance tasks undertaken is set out in

Annex 2, while Annex 3 lists the evidence sources triangulated by United Utilities plc

for each service performance measure.

6 The following service performance measures were subjected to an in-depth (‘deep dive’) review against the triangulation assurance framework: (1) interruptions to supply; (2) leakage; (3) further improving river quality; (4) internal and external flooding incidents; and (5) taste and odour.


9

Figure 2.2 Iterative, multi-review nature of the assurance approach

2.4 Applying the framework

The following sections expand on how the framework described above was used

practically in the first step (relating to triangulation assurance) and the second step

(relating to how customer valuations were used and applied to incentive rates).

To apply the framework, United Utilities plc documents were assessed against a set

of questions relating to relevant sections of the framework described above.

Ratings were assigned against each question in the detailed framework described

below (Sections 2.4.1 and 2.4.2), then aggregated to provide an overall rating for each

customer valuation category. The overall aggregate ratings denoted the following

assurance outcomes (discussed in more detail in Annex 2):

■ A low rating indicated a high number of – and/or material – omissions, errors or

failures to explain the rationale. It reflected an overall aggregate score (across all

questions in the framework) of 50 per cent or less of the total possible awarded

marks7.

■ A medium rating indicated some omissions, errors or failures to explain the

rationale, but fewer and/or less important than those given a low rating. It reflected

an overall aggregate score (across all questions in the framework) of between 51

per cent and 80 per cent of the total possible awarded marks.

■ A high rating indicated few or no omissions, errors or failures and was awarded

only where such omissions, where they existed, were not significant and where

the overall aggregate score was above 80 per cent of the total possible awarded

marks.

Considerable variation was observed in the underlying reasons for each rating

awarded, reflecting diversity in the quality and completeness of the triangulated

evidence for each customer valuation.

7 Each question resulted in a rating being awarded: low (=1), medium (=2) or high (=3). If the total ‘score’ for a service performance measure across all questions in the assurance framework was less than 50 per cent of the total possible marks that could be awarded, that service performance measure was rated overall as ‘low’.


10

2.4.1 Triangulation assurance

In Step 1 (triangulation assurance), two sets of questions were used to assess United

Utilities plc’s documentation against the framework: one set relating to the higher-

level review, which focussed on the content of United Utilities plc’s triangulation

document only; and a second relating to the in-depth review, which incorporated an

assessment of whether the underlying evidence was appropriate. These questions –

and how they relate to the framework – are set out below (Table 2.1).

Table 2.1 Triangulation assurance questions

Framework Triangulation review question In-depth review question

Sources

identified

What evidence sources have been

identified?

What evidence sources have been

identified?

Has the triangulation document listed

the evidence sources that were

considered for inclusion?

Has the triangulation document

explained the rationale behind certain

evidence not being included in United

Utilities plc's customer valuations?

Does the evidence base used in the

triangulation document include all

relevant evidence from the evidence

sources provided?

Has the triangulation document listed

the potential estimates contained in

sources that have been considered?

Has the triangulation document

explained the rationale behind using

individual estimates from each source?

For each individual source that is

considered in the triangulation document

Findings

identified

Does the triangulation document

explain the process, calculations and

rationale behind final estimates?

Are the final estimates appropriate, given

the evidence presented?


recognise limitations in the data /

evidence and explain how these have

been overcome?

Have the limitations in the data / evidence

been overcome appropriately (and why)?

Does the triangulation document raise

any new research questions?

Are there any further research questions

raised by the evidence base /

triangulation document (over and above

any that are already raised in the

triangulation document)?

Evidence

assessed


appropriately describe the weight that

has been given to each source?


sufficiently justify those weights, where

explained or otherwise?

Are the weights ascribed to each

evidence source appropriate, given the

evidence presented in the sources?


explain whether each evidence source

is representative?


explain how these has been taken into

account?

Does United Utilities plc's estimate

appropriately weight the estimates that

are used?

Proportionality

demonstrated

Does the triangulation document assess

how important the measure is relative to

other measures - and identify an

appropriate level of effort associated

with collecting and analysing evidence?

Is the level of evidence and analysis

proportionate to the importance of

customer valuation measure in the

business plan?


11

2.4.2 Use of customer valuations

In Step 2 (use of customer valuations), a different set of questions was used to assess

the application of each customer valuation estimate (from Step 1) to United Utilities

plc’s proposed performance commitments and, in particular, incentive rates, as set

out below (Table 2.2). That is, Step 2 involved determining the extent to which United

Utilities plc’s proposed performance commitments and incentive rates reflect the

triangulation results and, if not, whether it is clear why they are different and whether

United Utilities plc’s judgement appears reasonable. To avoid duplication, questions

were not added in relation to two aspects of the framework (sources identified and

proportionality demonstrated) as United Utilities plc documents that used customer

valuations referred directly to the triangulation documents and, therefore, these

aspects of the framework had already been assessed. Further questions were added

specifically relating to how well summary documentation (a PowerPoint file and a

Word file) reflected the underlying information.

Table 2.2 Use of customer valuations assurance questions

Framework Triangulation review question

Sources

identified

n/a – already assessed for triangulation document

Findings

identified

Is there a clear link between the triangulation document and findings?

Evidence

assessed

Is the approach to setting the incentive clear and justifiable (whether based on the

customer valuation or an alternative)?

Findings

identified

n/a – already assessed for triangulation document

Overview

PowerPoint

slides

How adequately and accurately does the PowerPoint summary document reflect

the evidence and rationale in the longer document for this measure?

Overview Word

document

How adequately and accurately does the overview Word document reflect the

evidence and rationale in the longer document for this measure?


12

3 Findings This section describes the assurance findings, including a summary and further detail

of findings on each measure that was reviewed, including findings from the five

triangulation ‘deep dives’, findings from the five ‘lighter’ triangulation reviews and

findings from five reviews of United Utilities plc’s use of customer valuations.

The detailed documentation of findings against each of the questions set out earlier,

and recommendations for improving United Utilities plc’s documentation, were

provided by ICF to United Utilities plc and YourVoice in a separate Excel file at

different points throughout the assurance process.

3.1 Summary of assurance findings

Figure 3.1 and Figure 3.2 below summarise the initial, revised and final ICF assurance

scores across all service performance measures reviewed, demonstrating the

improvements made by United Utilities plc during the course of this assurance

exercise. Each figure below shows an inner ring (representing the initial assurance

score in percentage terms), a middle ring (representing the revised assurance score)

and an outer ring (representing the final assurance score). Each ring is colour-coded

according to the overall assurance score received:

■ A low rating to reflect an overall aggregate score (across all questions in the

framework) of less than 50 per cent of the total possible awarded marks.

■ A medium rating to reflect an overall aggregate score of between 51 per cent and

80 per cent of the total possible awarded marks.

■ A high rating to reflect an overall aggregate score above 80 per cent of the total

possible awarded marks.

So, for example, Figure 3.1 shows that United Utilities plc’s initial triangulation

evidence for supply interruptions scored 50%. United Utilities plc then reviewed and

addressed the comments provided by ICF on that initial evidence, for which revised

evidence was presented and scored 78%. A final set of revisions by United Utilities

resulted in a third and final iteration, which scored 89%.

It is noted that Step 1 and Step 2 each reviewed different documentation. Scoring for

the first five measures under Step 1 (in-depth reviews) and the second five measures

under Step 1 was assigned based on the two different sets of questions set out above

in Table 2.1.


13

Figure 3.1 Summary of scoring against the assessment framework – triangulation documentation (Step 1)


14

Figure 3.2 Summary of scoring against the assessment framework – valuation use documentation (Step 2)


15

The nature of the improvements made by United Utilities plc through each iteration of

the triangulation and valuation use documents is summarised below.

3.1.1 Improvements made to triangulation documentation

The following improvements to the triangulation document were observed over time

throughout the assurance process:

■ Supply interruptions – evidence was included from additional sources and the

transparency of the method and rationale was improved, as was the justification

for the final weighting of evidence.

■ Leakage – the reasons for some evidence sources not being used was elaborated,

with further explanation provided of the method and weighting of evidence

sources.

■ River quality – evidence was included from additional sources and the

transparency of the method and rationale was improved. Some elements of the

method needed further justification, and this was addressed. Some potential

limitations in the evidence were addressed and the method for weighting evidence

was clarified and justified.

■ Flooding – certain aspects of the use of the underlying evidence was clarified and

justified, while the representativeness of evidence was further assessed and

explained, which was particularly important as some queries were raised about

the representativeness of certain evidence sources relating to customers having

experienced flooding.

■ Taste and odour – greater clarity and justification were provided surrounding the

method for determining customer valuations, as well as the weighting of underlying

evidence, particularly to allay some concerns about weighting and potential

selectivity.

■ Discolouration – some source-specific queries were identified and further

methodological elaboration was provided in certain areas of the write-up.

■ Pollution incidents – the initial number of sources was low and it was not clear if

sources used had been triangulated with immersive research and other PR14

results, with further clarification provided in subsequent iterations of the

documentation.

■ Supporting vulnerable customers – acceptability testing demonstrated poor

customer understanding of what UU does to support vulnerable customers, which

was indicative of the need to consider the detail of customer evidence gathering.

■ Lifting customers out of water poverty – the relevant concerns and improvements

here were similar to those for ‘supporting vulnerable customers’.

■ Water quality compliance – the clarity of the write-up on the approach to

triangulation and the use of different evidence sources was improved, including

how they were weighted.

3.1.2 Improvements made to valuation use documentation

The following improvements to the valuation use documents were observed over time

throughout the assurance process:


16

■ Supply interruptions – further elaboration on the underlying rationale was

provided, and while findings were generally clear, some amendments were made

to clarity the link between the customer evidence and the incentive rate.

■ Leakage – inconsistencies with the triangulation documentation and some

underlying documentation were addressed, while in some areas a lack of clarity in

rationale and method were also addressed.

■ Flooding – some information that was originally omitted was included, and the

links with the underlying documentation was clarified. Further clarity was also

provided regarding the description of the method for setting the incentive rates.

■ Taste and odour – some inconsistencies with underlying incentives documentation

were found and addressed. Findings were generally well identified but were

improved by adding further detail around certain evidence sources, such as WTP.

Some shortcomings were addressed in the assessment of evidence, including

correcting calculations and adding further clarity to write-ups.

■ Pollution – the incentive rates reflected the evidence adequately and accurately,

though further work was required to link the incentives rate to the underlying

evidence.

3.2 Initial assurance findings

Headline findings from the initial assurance review of each customer valuation

measure are presented below, with further detail provided in Annex 1. A further log of

assessment against each question in Table 2.2 was sent by ICF to United Utilities plc

and YourVoice in a separate Excel file.

Table 3.1 below presents the ratings awarded after ICF’s first review of United Utilities

plc’s materials. Further detail is provided in the remainder of this section.

Table 3.1 Ratings at first review of United Utilities plc’s documentation

Inte

rru

pti

on

s

Le

ak

ag

e

Riv

er

qu

ality

Flo

od

ing

inc

ide

nts

Ta

ste

an

d

od

ou

r

Dis

co

lou

rati

on

Po

llu

tio

n

inc

ide

nts

Su

pp

ort

ing

vu

lne

rab

le c

st.

Wa

ter

po

ve

rty

Wa

ter

qu

ality

Triangulation Med Med Med Low Med Med Low Low Low Low

Valuation use Med Low N/A Low Low N/A Low N/A N/A N/A

3.2.1 Triangulation documentation

Overall, findings from the initial review of United Utilities plc’s triangulation

documentation can be summarised as follows:

■ initial reviews revealed a mixed picture of the quality of triangulation documents,

including both the documents themselves and links to the underlying evidence

sources;

■ where examined in the in-depth reviews, the underlying information used was

relevant and in most cases accurately reflected;

■ initial low ratings reflected the need to improve write-ups and explanations to

demonstrate whether and to what extent the triangulation approach taken was


17

appropriate, and a small number of customer valuations needed updating and/or

revising;

■ underlying evidence, methods and calculations appeared sound (where it was

possible to assess these based on information provided) and values that did need

updating involved relatively minor changes; and

■ the lack of clarity in some parts of the documentation meant that it was not possible

to fully assess whether use of underlying evidence was appropriate in all cases.

3.2.2 Valuation use documentation

Findings from the initial review of whether and to what extent United Utilities plc’s

valuation use documentation reflected the triangulated evidence showed a similar

picture, though with generally higher ratings than for the triangulation documentation.

The findings can be summarised as follows:

■ there was a mixed picture regarding the quality of the valuation use documents, in

parts displaying a more coherent explanation of the underlying rationale for the

approach taken than the triangulation documentation;

■ the use of underlying evidence from the triangulation document generally

appeared reasonable where this could be fully determined based on the

explanation provided; and

■ some explanations were not sufficiently clear to determine their appropriateness

or validity.

3.3 Final assurance findings

As noted above, ICF provided United Utilities plc with detailed commentary after the

initial assurance review, on the basis of which United Utilities plc updated its

documentation accordingly. This process was repeated three times for documents

reviewed in Step 1 (triangulation documentation) and for documents reviewed in Step

2 (use of customer valuations documentation).

Ratings after ICF’s final review of United Utilities plc’s triangulation and valuation use

materials are shown below (Table 3.2). Further detail is provided in the remainder of

this section.

Table 3.2 Ratings at final review of United Utilities plc’s documentation

Inte

rru

pti

on

s

Le

ak

ag

e

Riv

er

qu

ality

Flo

od

ing

inc

ide

nts

Ta

ste

an

d

od

ou

r

Dis

co

lou

rati

on

Po

llu

tio

n

inc

ide

nts

Su

pp

ort

ing

vu

lne

rab

le c

st.

Wa

ter

po

ve

rty

Wa

ter

qu

ality

Triangulation High High High High High High High High High High

Valuation use High High N/A High High N/A High N/A N/A N/A

3.3.1 Triangulation documentation

Considerable improvement was seen through this iterative process of

recommendations and revisions. With respect to United Utilities plc’s final

triangulation documentation, findings can be summarised as follows:


18

■ United Utilities plc has a strong and varied evidence base from which it has drawn

its customer valuations, which reduces reliance on any single source;

■ sources for the evidence presented are clearly identified and United Utilities plc

has demonstrated a consideration of the strength and relevance of each source;

and

■ United Utilities plc incorporated clear explanations on how it has used the

underlying evidence to inform its customer valuations.

3.3.2 Valuation use documentation

With respect to United Utilities plc’s valuation use documentation, ICF’s final

assurance findings can be summarised as follows:

■ links with the triangulation document are clear and values are consistent with the

document;

■ United Utilities plc has set out clearly its method for using triangulation evidence

to set incentives rates; and

■ the links between proposed incentive rates and the customer valuation estimates

presented in its triangulation document are explained clearly.

3.4 Summary of improvements made by United Utilities plc

United Utilities plc made considerable improvements across its triangulation and

valuation use documentation over the course of the assurance process, following

comments made by ICF at each stage. The nature of these improvements is

summarised below:

■ improvements to description of evidence, important and/or relevant considerations

with respect to each evidence source, which made the content of each document

clearer and easier to follow;

■ clearer links between the evidence base and its use of the evidence in the

triangulation documents, as well as clearer use of the triangulation evidence in the

valuation use documentation;

■ updates to calculations to improve their validity and other updates to correct minor

errors made in the first calculations that were reviewed for this assurance exercise;

and

■ additional and/or edited explanations for the use of customer valuation evidence

and for the use of customer valuations in setting incentive rates.

ICF finds that these improvements have resulted in a triangulation approach which is

consistent with ICF’s triangulation framework set out above and in an earlier paper

published by CCWater8.

8 ICF (2017), Defining and applying 'triangulation' in the water sector: How water companies can use different sources of customer evidence in business planning, https://www.ccwater.org.uk/wp-content/uploads/2017/07/Defining-and-applying-triangulation-in-the-water-sector.pdf.




19

4 Conclusions This section presents conclusions from the assurance work based on the findings

presented earlier in this report.

4.1 Concluding remarks

ICF’s review of United Utilities plc’s triangulation and valuation use documentation

resulted in the following conclusions:

■ United Utilities plc has a strong and varied evidence base from which to draw its

customer valuations, reducing the reliance on any single source.

■ United Utilities plc’s triangulation documentation uses clear and transparent

communication to present this evidence base.

■ United Utilities plc’s triangulation documentation and valuation use documentation

explains clearly how it has used triangulation to generate customer valuation

estimates that reflect the evidence base, as well as using those customer valuation

estimates appropriately.

■ United Utilities plc’s incentive rates – and performance commitments – link back

to the underlying evidence and, where judgement has been used to derive a

proposed incentive rate and/or performance commitment, this is transparent.

■ United Utilities plc’s business plan content reflects a well-evidenced and clear

approach to triangulation, which largely follows ICF’s triangulation guidance

developed in a previous, separate study for CCWater.

Overall, ICF concludes that United Utilities plc has taken a sound and appropriate

approach to triangulating its customer evidence to arrive at proposed customer

valuations, performance commitments and incentives rates.

ICF further concludes that the quality of the business plan evidence is largely

attributable to the importance that United Utilities plc and YourVoice has placed on

the process of triangulating evidence and assuring the approach taken, as evidenced

by the iterative nature of this assurance exercise and the continuous improvement in

the triangulation approach/evidence displayed throughout the process.


20

ANNEXES


21

Annex 1 Detailed findings

A1.1 Interruptions to supply

Triangulation documentation ratings Valuation use documentation ratings

Measure description (in-depth review): This performance measure focuses on the

value that United Utilities plc customers place on unplanned interruptions to their

water supply (generally measured as a period of time representing the duration of the

interruption).

For households, United Utilities plc proposes to use the "£609 immersive value as a

starting point" (for a 6 hour interruption), but then adjusts this value to bring the

estimate within the range of other results. The non-household figure has been 'scaled

according to bill size' by taking a single point on the household 'interruptions valuation

curve' (i.e. value per day vs. duration of interruption), and scaling that up according to

bill value. That scaling has been done on the basis of one statement, for which

evidence could be explained more clearly.

Performance commitment: United Utilities plc has set a performance commitment at

the expected industry upper quartile performance, as required by Ofwat in its PR19

Methodology, which is an average of 6 minutes lost per connected property per year,

from 2019/20 onward.

Below are specific findings of the assurance review of this measure, indicating main

comments, concerns and recommendations and how United Utilities plc responded

to these.

■ Sources were well identified and appropriate, but initially United Utilities plc’s

document lacked commentary on how relevant sources were chosen. United

Utilities plc’s work on the document improved clarity and introduced new

commentary on evidence of customer valuations of frequency versus duration of


22

interruptions, as well as consideration of business customers’ views. United

Utilities plc also followed through the suggestion from the initial review to add

discussion of representativeness of the sources used to inform the customer

valuation. The final iteration of the document addressed all comments.

■ Findings identified in the triangulation document initially did not appear to include

all relevant information from the underlying evidence sources that were selected.

There was also little discussion of the limitations of evidence sources and some

potential inconsistencies between evidence sources were not considered. ICF

made recommendations to improve these areas, some of which were addressed

in United Utilities plc’s second iteration of the report, including additional text on

the interpretation of different evidence sources.

ICF’s second set of recommendations comprised areas in its first

recommendations that were not updated in the second iteration, including

consideration of any additional potential sources of evidence, specifically including

how acceptability testing was taken into account, or if not, why not. The final

iteration of the document addressed the majority of final recommendations.

■ Evidence assessment initially was not justified with a clear rationale. It was not

completely clear how weightings had been considered in the customer valuation

estimate and in several cases strong statements were not supported by evidence

or rationale. Justification for the weighting of different sources was not clear.

On following ICF’s recommendations, United Utilities plc improved the clarity of

weightings and introduced discussion of potential biases in the underlying

evidence sources. The final document iteration added further explanation of

weighting of evidence sources and improved cross-referencing.

■ Proportionality appeared to have been applied in United Utilities plc’s document,

but initially was not clearly evidenced or justified. United Utilities plc followed

suggested improvements to recognise the role that proportionality played to inform

the extent of research carried out to inform this area.

Below are specific findings from the assurance review in relation to the valuation use

documentation, including main comments, concerns and recommendations and how

United Utilities plc responded to these.

■ Summary documents initially required further elaboration of the rationale for the

approach taken and further high-level information on incentives. United Utilities

plc’s second iteration of the summary documents addressed these shortcomings.

■ Findings identification was generally good in the first iteration of the valuation use

documentation. ICF suggested minor amendments to clarify the link between

triangulated evidence and the incentive rate, and the rationale behind this. ICF’s

second review of the valuation use documentation indicated that these

recommendations had been met.

■ Evidence assessment was generally clear and justified based on the evidence.

The valuation use document for this measure did not require a third iteration given the

second iteration achieved a ‘high’ overall rating, with no scope for further

improvement.


23

A1.2 Leakage


Measure description: This performance measure focuses on the value that United

Utilities plc customers place on reducing leakage (measured in cubic metres of water).

As a result of its triangulation exercise, United Utilities plc proposes to use for leakage

a figure of 70p per cubic metre, which it says is "halfway between the sliders value

and the intermediate value, and similar to the average value obtained from testing its

plan."

Performance commitment: A reduction in leakage vs 2020/21 of -15% by 2024/25,

which although greater than customers indicated was acceptable, will be achieved by

committing to a lower marginal cost than presented to customers, to be realised

through efficiency savings.



to these.

■ Sources were well identified and appropriate, but initially could have included a

more detailed description of sources considered and rationale for choosing the

sources finally chosen. ICF put forward some suggestions for additional evidence

to be identified and considered, including in relation to possible contact information

and comparison with PR14 WTP results. ICF also identified that further

explanation could be provided on possible overlaps with other categories, such as

interruptions. United Utilities plc’s updates addressed these comments. It added

information on selection of underlying evidence sources, including some relevant

background information about how supply/demand research was developed.

At the second iteration of the document, ICF provided additional suggestions for

adding further clarity to explanations, for example in relation to aggregation of

household and business customer estimates, and the final iteration of the

document addressed these recommendations.

■ Findings identification initially needed improvement, specifically in relation to

treatment of evidence on customer priorities, accuracy of descriptions of WTP

analysis and relevance and, in particular, explanations of calculations from


24

immersive research. This left significant questions about how sources were used

and the appropriateness of weightings applied to those sources (see next bullet).

On recommendation from the first review, United Utilities plc improved the clarity

of write-up in the triangulation document and clarified individual queries within the

text. The final iteration of the document addressed all recommendations.

■ Evidence assessment appeared appropriate given the information presented in

the original triangulation document. Recommendations were provided to improve

individual areas of the triangulation write-up and to any potential biases in the

underlying evidence sources. These were reflected somewhat in the second

iteration of the document, with further feedback being provided at the second

iteration. The final document iteration added further explanation of weighting of

evidence sources and improved cross-referencing.

■ Proportionality was demonstrated to some extent in United Utilities plc’s first

iteration of the triangulation document, though ICF made recommendations to set

out more clearly/comprehensively its approach to ensuring proportionate evidence

gathering. These recommendations were reflected in the second iteration of the

document. No further updates were required with respect to demonstrating

proportionality in the final iteration of United Utilities plc’s triangulation document.

The triangulation document for this measure did not require a third iteration given the

second iteration achieved a ‘high’ overall rating, with minimal scope for further

improvement.

Below are specific findings of the assurance review in relation to valuation use



■ Summary documents on first review included some significant inconsistencies and

were found not to fully reflect the underlying documents, while some figures were

not included in underlying documentation. United Utilities plc addressed most of

ICF’s comments in its second iteration, though some improvement in drafting was

recommended, so that the summary fully reflected the underlying documents.

These recommended improvements were adopted in the third iteration of the

relevant documents.

■ Findings identification was found in the first iteration to be somewhat lacking clarity

regarding the link between the performance commitment and the underlying

customer research. Some inconsistent values also needed updating.

Improvements were made to the second iteration of documentation, though

improvements in ratings were conditional upon making final changes to note that

final performance commitments are dependent upon further retesting. These

recommendations were addressed in the third and final iteration.

■ Evidence assessment initially lacked some clarity around weighting of WTP

estimates and ICF recommended that further explanation be added in other areas,

where the incentive rate was potentially conditional on further modelling. On

second review, some improvements could still be made around explanation of the

use of WTP results. These recommendations were followed in United Utilities plc’s

third iteration.


25

A1.3 Further improving river quality


Not applicable

Measure description: This measure focuses on how customers value river quality,

measured according to "% of rivers reaching a good standard" and/or "km of river

improved".



to these.

■ Sources were initially adequately identified and evidence considered was

generally appropriate. However, ICF made recommendations to improve the

clarity of the document to provide a more complete description of which research

studies were relevant for this customer valuation measure. Specific

recommendations were also made to consider whether the river quality valuation

could be informed by other valuations and to consider whether contact data might

be considered relevant. Nevertheless, these additional recommendations were

minor and did not prevent the area being rated highly at the second iteration.

■ Findings identification was initially mixed. In several areas the triangulation

document either did not take appropriate findings and/or did not clearly justify

and/or explain the findings that it did use, such as selection of WTP values from

PR14 evidence for comparison with PR19 values. For other sources of

information, a lack of clarity around explanations made it difficult to determine

whether the appropriate underlying information was used. Little consideration of

limitations of underlying evidence was included in the first document version. ICF

provided recommendations to improve these areas - and recommendations to

improve the write-up of United Utilities plc’s evidence assessment (see below)

also supported clearer identification of the relevance of evidence identified.

United Utilities plc’s second iteration of its triangulation document added

information to clarify the rationale for selecting findings, improving the document

in this area. Nevertheless, ICF made further recommendations for United Utilities

plc to improve its document further, including for explaining WTP evidence

presented, weighting of evidence sources and consideration of the strength of


26

evidence from different sources. All recommendations were followed and reflected

in final documentation.

■ Evidence assessment was generally reasonably well explained in United Utilities

plc’s first document, with some explanation of weighting of evidence, but it was

not clear that this weighting considered all evidence sources. It was also not clear

whether United Utilities plc had considered representativeness of sources when

weighting evidence. United Utilities plc’s approach appeared reasonable, but

required further explanation, which ICF made specific recommendations towards.

ICF recommended that United Utilities plc add more evidence to its consideration

of source weighting, clarified treatment of evidence sources in its calculation

descriptions and qualified a number of strong statements. United Utilities plc

added some additional explanation in response to these points, but ICF

recommended further work in these areas to improve clarity further. All

recommendations were followed and reflected in final documentation.

■ Proportionality was reasonable demonstrated in United Utilities plc’s first iteration

of the triangulation document. United Utilities plc acted on further specific

recommendations to add discussion of the proportionate approach it had taken to

selecting and analysing its evidence base.

This measure was not selected in the incentive-setting review sample.

A1.4 Internal and external flooding incidents


Measure description: This measure focuses on WTP to reduce the frequency of

flooding incidents. The final customer valuation uses the WTP value, but with a '50%

uplift' applied to the WTP value.

Performance commitment: The performance commitment for Business Plan Period

2020–2025 is to achieve upper quartile performance by the first year, in line with

Ofwat’s requirement, as set out in the PR19 methodology. The resulting target lowers

the number of internal flooding incidents to 733 by 2024/25 (documents for external

flooding were not selected for review).


27



to these.

■ Sources were initially sufficiently clearly identified and it was clear what evidence

sources had been considered. However, additional relevant information was

identified in underlying evidence sources that was not reflected in the triangulation

documentation. Little description of the process for selecting sources was

included. The second iteration of the document addressed these shortcomings by

adding additional sources, but shortcomings remained in describing the process

for selecting information. These were addressed in the third and final version of

the triangulation document.

■ Findings identification were similarly not well explained in the initial document, with

some lack of clarity in explaining interpretation of the evidence and exploring

relative strengths and limitations of the data. The second iteration added some

detail on how customer contact evidence was treated and also in relation to

evidence on repeat flooding valuations, but did not provide sufficient clarity around

calculations, in particular relating to aggregation of household and business

estimates. This additional clarity was provided in the third and final document

version.

■ Evidence assessment was initially lacking sufficient analysis of the relative

strengths and shortcomings of each source of evidence, in particular in relation to

differences between customers who have experienced service failures being

estimated and those that had not, and in relation to sample sizes. It was not clear

how these factors were taken into account in weighting evidence sources. The

second iteration of the document explained weighting of evidence more thoroughly

and clearly. The third and final version addressed remaining shortcomings by

further improving the description of weightings and referencing a new discussion

of representativeness of evidence sources (linking to the introduction section of

the document).

■ Proportionality was initially not considered. United Utilities plc introduced text in its

introduction, which partially addressed this point and in the third and final iteration

it also added some further explanation linking customer priorities to the extent of

its evidence in this area.




■ Summary documents on first review displayed some notable omissions of

information contained in underlying documentation and also included some

information where the link with underlying documentation was not clear. United

Utilities plc addressed these areas in its second iteration of the documents, though

the ‘high’ rating of the PowerPoint summary slide was contingent on addressing a

remaining inconsistency in a reference to customer contact values (which was

addressed in the third and final version of the documents).

■ Findings identification was generally relatively clear in the first iteration of the

underlying incentives documents, although some information could not be clearly

linked to the triangulation documentation. United Utilities plc noted the location of

some of this data, but to achieve a ‘high’ rating this required clearer referencing,

given that it did not link directly with the triangulation documentation. This was

addressed in the third version of the documents.


28

■ Evidence assessment and the approach to setting the incentive rates was not

clear in the first iteration of the incentives documents, so the method applied could

not be sufficiently assessed. The second iteration of the document built on useful

material in the document to provide a considerably clear explanation. Some further

potential improvements were highlighted in the second iteration, which could add

further clarity to the explanation of the method. These were addressed in the third

and final version of the documents.

A1.5 Taste and odour


Measure description: This measure focuses on WTP for 'customer contacts' relating

to taste and odour. The method uses a scaling of WTP values (unclear whether mean

or median and the scaling is unclear) to arrive at a WTP figure that appears high

compared with many of the evidence sources. An arbitrary scaling down is then

applied (75% of the WTP value previously calculated).

Performance commitment: to reduce the number of customer contacts about taste

and odour to 3.07 per 10,000 connected households per year, by 2024/25.



to these.

■ Sources were initially well identified, though some additional information was

identified in the underlying evidence that could have been used in the triangulation

documentation. The second iteration of the document included this further

evidence and added information on business acceptability testing. No further

changes were necessary in the final iteration of the document.

■ Findings identification was similarly well written up in the first iteration of the

triangulation document, with recommendations made to make minor

improvements to reflect limitations of the underlying evidence sources and to add

a comparison with PR14 customer valuations. Material was added to cover these

areas in the third and final version of the document.

■ Evidence assessment was rather unclear in the first iteration of the document. In

particular, the rationale behind the weightings and method for triangulating


29

evidence sources was not sufficiently clear, leading to the recommendation that

this weighting could be more evidence based. In response, United Utilities plc

added further detail in an annex, which was then cross-referenced in the main

document. This added further clarity, but a second review made further specific

recommendations to explain individual steps within the triangulation process.

United Utilities responded to each detailed recommendation in its third and final

version of the triangulation document.

■ Proportionality was evidenced in the initial version of the document but was not

sufficiently explained. The second and third iterations of the document added

further explanatory text on why the measure is important even though the level of

investment dependent upon the valuation is relatively low.




■ Summary documents on first review displayed several inconsistencies with the

underlying incentives documentation. These were largely corrected in the second

iteration of the document, though some additional material was recommended to

be added and one figure remained to be corrected in the last version to be

reviewed (which were subsequently addressed in a revised version).

■ Findings identification was generally good in the first version of the incentives

documentation. Recommendations were made to improve this further, by adding

further detail of WTP evidence and these were addressed in the second version

of the incentives documentation.

■ Evidence assessment initially had several shortcomings related to making the

incentives document consistent with updates made to the triangulation document

and in relation to the cost-benefit calculation in the incentives document. That

calculation needed to be corrected and also required much clearer explanation of

the rationale and assumptions behind it. These changes were made in the second

version of the document, but this also revealed a key assumption underlying the

rationale for the incentive rate set – relating to increasing customer contacts. It

was recommended that further updates be made to explain the evidence and

rationale behind that assumption, particularly given that the validity of the incentive

rate appeared to be dependent upon that assumption.


30

A1.6 Discolouration


Not applicable

Measure description: A valuation of customer contacts about taste, odour and

discolouration has been set based on WTP evidence on taste, odour and

discolouration. The value is based on customer research that identifies discolouration

as a less important issue than taste and odour.



to these.

■ Sources were initially identified correctly, but with no explanation of the process of

selection, nor consideration of sources that were not used. The second iteration

and final iteration were improved by the addition of descriptions and information

on sources, as well as the prominence of research into specific projects for

Manchester and Pennine resilience.

■ Findings identification was generally good from the outset, with appropriate

interpretation and use of the evidence. Minor improvements were suggested,

specifically around improving explanation of limitations in evidence sources. The

third and final version of the document addressed these areas and added

explanation of comparisons made with other companies’ PR19 research.

■ Evidence assessment initially appeared reasonable, but there was room for

improvement in the clarity of explanation of the evidence assessment (specifically,

in relation to the weighing up of evidence sources and consequent calculation of

weightings). The second iteration of the document did not address these

comments. In response, more specific recommendations were made, which

included further detailed description of relevant WTP research, more and clearer

explanation of weightings and further assessment of representativeness of results.

Each of these areas was addressed in the third and final version of the document.



further explanatory text on why the measure is important even though the level of

investment dependent upon the valuation is relatively low.


31


A1.7 Pollution incidents


Measure description: This performance measure focuses on the value that UU

customers place on reducing/preventing "Category 3" pollution incidents and United

Utilities plc proposed to use a figure of £330,000 per incident (which appears to be

the value that it suggests customers place on reducing Category 3 pollution incidents

by one incident, i.e. £330,000 per Category 3 incident).

Performance commitment: currently set at 120 incidents per year by 2025, but United

Utilities plc has noted that this performance commitment is still subject to change.



to these.

■ Sources were generally clearly identified and appropriate, with no obvious

omissions. However, the triangulation document did not initially include any

explanation of the process for selecting evidence for consideration. Furthermore,

questions were raised about some apparently contradictory information and the

potential inclusion of business acceptability testing. The second iteration of the

document included some further material to address these points, though some

uncertainty remained. Further explanation was added to the third and final version

of the document, which addressed these concerns.

■ Findings identification initially lacked clarity around some labelling of WTP results

and calculations relating to its immersive research. Queries were also raised with

respect to the use of means vs medians for comparison of PR14 and PR19 WTP

values. These areas where improved somewhat for the second version of the

document. But further shortcomings remained, specifically to improve the

explanation of weighing up of evidence sources and regarding the method,

rationale and workings for the final calculations behind the customer valuation,

which were not sufficiently clear. Additionally, further explanation was required in

relation to background information relevant to the comparison between PR14 and

PR19 valuations. The third and final version of the document addressed each of

these areas.


32

■ Evidence assessment generally appeared to use an appropriate weighting of

evidence and to use representative sources. However, in common with several

other measures, the rationale and process for doing so was not adequately

explained in the first version of the document. Additional explanation was added

to set out weighting of evidence sources and contextualise PR14 WTP values.

And in the third and final document clearer description of customers’ priorities

around pollution incidents and the presentation of WTP figures contributed to a

clear and well-justified assessment of the evidence.



further explanatory text, specifically in relation to customers’ priorities.




■ Summary documents on first review adequately and accurately reflected evidence

from the incentives and valuation documents, but needed to reference the

potential double counting between pollution incidents and upstream causes, as

explored in the underlying documentation. Some inconsistencies were also found

in the PowerPoint summary slide. These areas were addressed in the second

iteration of each summary document.

■ Findings identification was initially inconsistent, specifically in relation to definition

of the measure, links between the performance commitment and the underlying

customer evidence and links to evidence identified as potentially contradictory in

the underlying triangulation document. These areas were partially addressed in

the second version of the incentives document, with clear explanation of the

definitions being used. Drafting on the performance commitment was expanded

somewhat, but further work was required to fully link the performance commitment

back to the underlying customer evidence (in both the valuation use document

and the triangulation document). On cost, the second version of the incentives

document still did not appear to explain the basis for the cost information. These

comments were addressed by United Utilities plc in the third and final version of

documents.

■ Evidence assessment initially contained residual uncertainty around seemingly

contradictory evidence (see above and detailed comments) and comparison of

WTP results between sectors. Further detailed drafting comments were largely

addressed in the second and third versions of the incentives document.


33

A1.8 Supporting customers in vulnerable circumstances


Not applicable

Measure description: This performance measure focuses on supporting vulnerable

customers. United Utilities plc measures this with respect to the number of customers

that are 'helped' by being provided with 'priority services' (e.g. contacted by UU's

partners, which are Age UK, MIND, The Alzheimer's Society and Citizens Advice

Bureau).



to these.

■ Sources initially appeared relevant and appropriate, but further information was

required on how and why these sources were selected, as well as how evidence

was selected within each source. The second iteration of the document added

further write-up to this effect, which added sufficient clarity reflecting initial

comments. Further opportunities as to why other research methods were not

chosen remain, but these were not addressed in the final version of the document

and it may not be proportionate to address this recommendation.

■ Findings identification initially lacked sufficient explanation of the interpretation

and use of evidence, in particular in relation to write-up of the priorities treatment

labelling of WTP estimates and the calculation based on immersive research.

Significant questions remained about how individual sources were used. A second

version of the triangulation document partially addressed these comments, with

broadly sufficient explanation provided. One concern remained regarding

apparently circular logic given that WTP results are used to derive the level of

additional costs of providing support to customers.

■ Evidence assessment was mostly clear in the first version of the document.

Though specific recommendations were made in some areas relating to

representativeness of the sample and assessing the relative strengths and

drawbacks of using acceptability testing to generate customer valuations and

addressing biases in underlying evidence. Further explanations were added to the

second iteration of the document, which addressed these concerns.

■ Proportionality was not explicitly discussed in the first iteration of the document,

but further explanation was added to address this shortcoming in the second

version of the triangulation document.


34



improvement. This measure was not selected in the incentive-setting review sample.

A1.9 Lifting customers out of water poverty


Not applicable

Measure description: United Utilities plc’s measure of success will be the number of

unique customers lifted out of water poverty as a result of its action at any point in the

year.



to these.

■ Sources initially appeared relevant, but also limited in number. However, the

document did not include sufficient commentary on how these sources were

selected and why they were appropriate. The second iteration of the document

added research relating to social tariffs and referenced other evidence that was

considered but not incorporated. Further improvements to the third and final

iteration of the document noted that some further retesting will be undertaken and

will need to be considered.

■ Findings identification were initially somewhat clearly identified, though labelling

of WTP measures and calculations relating to immersive research were identified

for further improvement. Furthermore, clarification was required around the

document section on the priority services measure, including how the measure

relates to customer experience and potential to have done research into

customers’ valuation of clearer/further information. In response, United Utilities plc

added clarifications around details of some of the underlying research, partially

addressing these comments. Further recommendations were made on review of

that second document iteration, including further explanation of how proposed

costs have been derived and potential circularity of the logic of basing WTP results

to derive the level of additional costs of providing additional support. Further

explanation in the final versions of this section addressed these comments.

■ Evidence assessment was generally good from the outset, with only minor

comments provided with the aim of improving clarity around representativeness of


35

the sample and dealing with potential biases in the underlying evidence. Further,

as with other measures, more explanation was required in relation to the weighting

of different sources. Modifications were made that sufficiently addressed these

shortcomings in the second iteration of the document.

■ Proportionality was not explicitly discussed in the first version of the document,

though analysis and evidence did appear proportionate. The second iteration of

the document addressed this sufficiently. Finally, the third version of the document

added text noting that further testing will be undertaken in relation to this measure.


A1.10 Water quality compliance


Not applicable

Measure description: This performance measure focuses on the value that United

Utilities plc customers place on receiving water that complies with UK and European

standards. Specifically, it measures the value that customers place on receiving a

minimum proportion of water that meets the relevant UK/European standards), and

measures customer WTP/WTA an increase/decrease in the proportion of water

supplied that meets the relevant standards.



to these.

■ Sources initially appeared relevant, with relevant information having been taken

from each source. However, no write-up of the process for selection of sources

and/or evidence had been included. Queries were raised about potential for United

Utilities plc to include consideration of customer contacts and/or acceptability

testing around water quality compliance. In response, further text was added in

relation to proportionality of evidence gathering for this measure and content on

water quality in the triangulation document was split into a distinct section.

Discussion of evidence considered for inclusion was also added. Overall, these

changes addressed initial shortcomings and no further change was necessary

beyond the second iteration of the document.

■ Findings identification was initially somewhat clear, if brief. In addition, there was

scope to make the document clearer by improving labelling and

description/presentation of calculations associated with how the findings were

used. Further description of the approach to triangulation was also required. Each


36

of these areas was addressed and improved sufficiently in the second iteration of

the document.

■ Evidence assessment was rather unclear in the first version of the document. In

particular, the triangulation approach could not be understood easily and links

between the final values and underlying evidence were not clearly communicated.

In response, United Utilities plc separated out water quality from other content in

the original triangulation document, significantly improving clarity of the section on

water quality. Further, cross-references to introductory sections discussing WTP

results helped link the customer valuation to the evidence base. These changes

addressed the areas for improvement that were identified.

■ Proportionality appeared to have been appropriately applied, but as with other

measures had not been explained adequately in the documentation. This was

addressed during the second round of the document.



improvement. This measure was not selected in the incentive-setting review sample.


37

Annex 2 Detailed method This section provides a detailed description of ICF’s proposed task-by-task approach

to meet the objectives of the work. It sets out the detailed step-by-step tasks that ICF

undertook to review, assess and report on United Utilities plc’s triangulation approach

and the extent to which that triangulation approach was reflected in the way United

Utilities plc used the resulting customer valuations (for a sample of performance

measures and incentive rates).

A2.1 Overview of methodology

ICF’s methodology involved three main steps:

■ Step 1: Set-up and evidence gathering.

■ Step 2: Assessment & reporting – triangulation.

■ Step 3: Sample review, assessment & reporting – use of customer valuations.

An overview of this method is set out below in Figure A2.1.

Figure A2.1 Overview of methodology

A2.2 Step 1: Set-up & evidence gathering

The first set of tasks involved initial set-up, planning and evidence gathering. This

step involved discussions with United Utilities plc and YourVoice to verify the

objectives and priorities for the work, and to discuss the wider context and objectives

to maximise the value of the work for United Utilities plc and YourVoice. The


38

overarching objective of this step was to better understand United Utilities plc’s and

YourVoice’s objectives for this work, to review the main United Utilities plc

triangulation document and to finalise the framework/approach to the detailed

assessment under Step 2.

Task 1.1 – Start-up meeting / call

An initial start-up meeting was held involving representatives of United Utilities plc,

YourVoice and ICF. This meeting / call focused on confirming United Utilities plc’s and

YourVoice’s priorities for the work so as to clarify the scope of the work and discuss

the approach.

In particular, the following issues were discussed:

■ scope of the work, particularly United Utilities plc’s and YourVoice’s views on the

issues to be covered in the work and the relative priorities between these issues;

■ United Utilities plc’s and YourVoice’s views on the proposed assurance

framework;

■ the availability of evidence sources / documentation underpinning United Utilities

plc’s triangulation approach;

■ working arrangements between ICF, United Utilities plc and YourVoice; and

■ timing and details of deliverables and meetings.

Task 1.2 – Review United Utilities plc’s customer research triangulation document

Following the initial start-up meeting / call, the main United Utilities plc triangulation

document was reviewed by ICF in detail, including:

■ the performance measures (performance targets and incentive rates) captured by

the document and the research used to underpin these;

■ the key findings from the research relevant to the performance measures;

■ the basis on which the different evidence sources were weighed-up; and

■ the result and conclusions draw from the weighed-up evidence.

The purpose of the review was to understand the context for the assurance work and

the output(s) from United Utilities plc’s triangulation approach. Most importantly, the

review allowed ICF to understand the triangulation approach taken by United Utilities

plc in generating performance measures ahead of its PR19 business plan.

Task 1.3 – Prioritise areas / sources for in-depth review

Based on the initial review of the customer research triangulation document, ICF,

United Utilities plc and YourVoice discussed priority areas / sources for more in-depth

review. That is, although this assurance review did not include a detailed review of all

underlying customer research, ICF’s review and assurance of United Utilities plc’s

triangulation approach required more in-depth review and assessment of selected

evidence to draw conclusions as to whether United Utilities plc’s triangulation

approach is sound and credible.


39

Task 1.4 – Operationalise / finalise assurance framework

This task involved finalising and implementing the assurance framework used by ICF,

and agreeing the more detailed questions the work would answer.

As part of this task, ICF reviewed again United Utilities plc’s own publication

‘Improving Customer Research and Engagement’9 to check for any relevant

elements/questions that should be incorporated within the assurance framework.

A2.3 Step 2: Assessment & reporting – triangulation

The second set of tasks involved the main assessment and assurance work

surrounding United Utilities plc’s triangulation approach, as well as reporting findings

via a draft and final deliverable.

Task 2.1 – In-depth review of underlying evidence / sources

This task involved a more detailed review of underlying evidence/sources for the

areas/issues identified as priorities in Task 1.3. In particular, ICF scrutinised the

robustness, relevance and credibility of the underlying evidence/sources to a greater

extent to consider whether they were appropriate sources to use for the intended

purpose as well as whether they have been triangulated (with other sources) in an

appropriate manner.

Task 2.2 – Assess triangulation approach against framework – 1st iteration

This task constituted the main element of the triangulation assurance work. It involved

reviewing systematically, consistently and transparently the triangulation approach

taken by United Utilities plc against the assurance framework presented earlier. This

not only looked at United Utilities plc’s overall approach, but also whether and to what

extent it varies based on the service performance aspect (i.e. across the 10 different

customer preferences/valuations that ICF sampled to assess).

ICF reviewed United Utilities plc’s triangulation approach to each of the 10 sampled

service performance aspects against each element of the assurance framework,

based on the questions outlined earlier.

Task 2.3 – Assess triangulation approach against framework – 2nd iteration

This task involved repeating the previous task for an updated/revised version of

United Utilities plc’s documentation.

Task 2.4 – Assess triangulation approach against framework – 3rd iteration



9 United Utilities Water (2016), Improving Customer Research and Engagement, https://www.unitedutilities.com/globalassets/z_corporate-site/about-us-pdfs/looking-to-the-future/water2020-feb16-customerengagement.pdf.

https://www.unitedutilities.com/globalassets/z_corporate-site/about-us-pdfs/looking-to-the-future/water2020-feb16-customerengagement.pdf

https://www.unitedutilities.com/globalassets/z_corporate-site/about-us-pdfs/looking-to-the-future/water2020-feb16-customerengagement.pdf


40

Task 2.5 – Submit draft deliverable

ICF presented the findings of its assessment in a draft deliverable which was

submitted to United Utilities plc and YourVoice.

Task 2.6 – United Utilities plc and YourVoice feedback and comments

ICF sought feedback and comments from United Utilities plc and YourVoice on the

draft deliverable .

Task 2.7 – Submit final deliverable

Once comments on the draft deliverable were addressed, ICF submitted a final

deliverable to United Utilities plc and YourVoice.

Task 2.8 – Presentation at YourVoice Customer Engagement Subgroup meeting

ICF participated in a meeting of the YourVoice Customer Engagement Subgroup to

discuss/present the findings of the assurance work.

A2.4 Step 3: Review, assessment & reporting – use of customer valuations

The third set of tasks involved additional review and assurance work surrounding how

the resulting customer valuations (from United Utilities plc’s triangulation exercise)

were applied in practice as part of United Utilities plc’s business plan, as well as

reporting findings via a draft and final deliverable.

Task 3.1 – Meeting / call to agree approach, including sampling

ICF arranged a call involving representatives of United Utilities plc, YourVoice and

ICF to discuss and agree the scope of the additional assurance work (including

sampling/prioritising specific measures for review and assurance) and the approach.

The following issues were discussed:

■ the availability and nature of the documentation/evidence to be reviewed;

■ United Utilities plc’s and YourVoice’s proposed sample of specific measures for

review and assurance;

■ the assurance approach to be taken to assess the appropriateness of United

Utilities plc’s use of the (triangulated) customer valuations; and

■ timing and details of deliverables and meetings.

Task 3.2 – Review headline document and detailed documents for sample

This task involved:

■ reviewing a short document (~10 pages) setting out United Utilities plc’s overall

approach;

■ reviewing for up to 5 sample measures covered within that short document:


41

– a summary document (~2-3 pages) for each measure that summarises that

measure; and

– a short document (~10 pages) for each measure that sets out the measure in

detail.

Task 3.3 – Assess links to triangulated evidence – 1st iteration

This task involved identifying whether the target and incentive rates for the sample of

5 measures reviewed under the previous task (Task 3.2) reflected the triangulation

results (reviewed/assessed during Step 2) and, if not, whether it was clear why the

target and/or incentive rates were different, identifying whether and to what extent

United Utilities plc’s judgement appeared reasonable.

Task 3.4 – Assess links to triangulated evidence – 2nd iteration



Task 3.5 – Assess links to triangulated evidence – 3rd iteration



Task 3.6 – Presentation to United Utilities plc and YourVoice

ICF presented all findings from Step 1 and 2 to United Utilities plc and YourVoice.

Task 3.7 – Submit draft deliverable

ICF wrote-up the findings of its assessment in a draft deliverable (and summarised

these in this report) which was submitted to United Utilities plc and YourVoice.

Task 3.8 – United Utilities plc and YourVoice feedback and comments

Feedback and comments were sought from United Utilities plc and YourVoice on the

draft deliverable(s).

Task 3.9 – Submit final deliverable

Once comments on the draft deliverable were addressed, final deliverables were

submitted to United Utilities plc and YourVoice.


42

Annex 3 Summary of evidence sources for each performance measure

Customer valuation

Cu

sto

mer

pri

ori

ties

WT

P r

esearc

h

Su

pp

ly /

dem

an

d s

lid

ers

Su

pp

ly /

dem

an

d

cu

sto

mer

res

earc

h

Imm

ers

ive

res

earc

h

Po

st-

incid

en

t

res

earc

h

Cu

sto

mer

pan

el

An

aly

sis

of

cu

sto

mer

co

nta

cts

Cu

sto

mer

co

nta

ct

an

aly

sis

Rep

eat

flo

od

ing

res

earc

h

Acc

ep

tab

ilit

y

testi

ng

PR

14 r

esu

lts

co

mp

ari

so

n

Supply interruptions ✘ ✘ - - ✘ - ✘ ✘ - - ✘ ✘

Leakage ✘ - ✘ ✘ - - ✘ - - - ✘ -

River quality ✘ ✘ - - ✘ - - - - - ✘ ✘

Flooding ✘ ✘ - - - - - - ✘ ✘ ✘ ✘

Taste and odour ✘ ✘ - - - ✘ ✘ - ✘ - ✘ ✘

Discolouration ✘ ✘ - - - ✘ ✘ - ✘ - ✘ ✘

Pollution incidents ✘ ✘ - - - - - - - - ✘ ✘

Supporting vulnerable customers ✘ - - - - - - - - - ✘ -

Lifting customers out of water poverty ✘ - - - - - - - - - ✘ -

Water quality compliance ✘ ✘ - - - ✘ - - - - - -

Triangulation review and assurance: ICF · ICF makes big things possible . ICF is a global...

Documents

Transcript of Triangulation review and assurance: ICF · ICF makes big things possible . ICF is a global...