Price comparison website mystery shopping report for ... · comparison website.4 Today consumers...

Comparing comparison sitesPrice comparison website mystery shopping report for Consumer Focus by eDigitalResearch

Consumer Focus

About Consumer Focus

Consumer Focus is the statutory consumer champion for England, Wales, Scotland and (for postal consumers) Northern Ireland.

We operate across the whole of the economy, persuading businesses, public services and policy-makers to put consumers at the heart of what they do. We tackle the issues that matter to consumers, and give people a stronger voice. We don’t just draw attention to problems – we work with consumers and with a range of organisations to champion creative solutions that make a difference to consumers’ lives.

Following the recent consumer and competition reforms, the Government has asked Consumer Focus to establish a new Regulated Industries Unit by April 2013 to represent consumers’ interests in complex, regulated markets. The Citizens Advice service will take on our role in other markets from April 2013.

Our Annual Plan for 2012/13 is available online, consumerfocus.org.uk.

AcknowledgementsThe project was completed as part of a joint work of Consumer Focus matrix team that consisted of Justin Gutmann, Marzena Lipman and Jenni Lucas-Williams.

We would like to thank eDigitalResearch for the professional way they carried out this research, as well as Gill Wales (independent consultant) for her role in steering this project.

We are grateful to the following organisations for their comments on this report: Financial Service Authority, Ofcom, Office of Fair Trading and Ofgem.

For regular updates from Consumer Focus, sign up to our monthly e-newsletter by emailing [email protected] orfollow us on Twitter http://twitter.com/consumerfocus

consumerfocus.org.uk

mailto:enews%40consumerfocus.org.uk?subject=e-newsletter%20sing-up

http://twitter.com/consumerfocus

Foreword

With a proliferation of deals on the market and the complexity of products and services, price comparison websites have potential to empower consumers by helping them to compare a large number of offers quickly and easily. For example Money Supermarket boasts it compares some 900,000 mobile phone deals.1

Increasingly price comparison websites are becoming a key source of the information that guide consumers’ purchasing decisions. For example in 2010, the Office of Fair Trading (OFT) report Advertising of Prices showed 73 per cent of consumers had used a price comparison website; up from 62 per cent in 2009.2 Civic Consulting research reports that 81 per cent had used one in 2011.3 According to a recent Harris Poll 67 per cent of consumers who bought or renewed car insurance in the last 12 months used a price comparison website.4

Today consumers are faced with navigating through an overwhelming choice of price comparison websites. For example, for this research we investigated some 99 price comparison websites. These varied in terms of market segment covered and business models operated. For example:

● some relate to specific markets (eg energy, insurance, telecom) while others offer comparisons across a range of consumer goods and services

● some provide information only on deals and prices; and others enable purchase through links with suppliers

1 Data taken on 21 December 2012, http://bit.ly/QzVaFX2 http://bit.ly/ZfKFIq3 http://bit.ly/TWHrtc4 http://bit.ly/VjrDoA

● some get information through direct arrangements with suppliers, while others scrape information from suppliers’ websites

● some capture consumer feedback on their purchases

● some rely on advertising revenues, adverts and sponsored links, but others are paid commission for a completed sale or switch

● some are signatories of accreditation schemes (eg Consumer Focus Confidence Code, Ofcom accreditation scheme) that provide quality assurance that the information is accurate, reliable and comprehensive, others are not

The nuances of business models and commercial arrangements are not always visible and known to consumers, yet they may impact on the quality and impartiality of advice consumers will rely upon. These factors may also determine whether or not consumers end up with a better deal as a result of using the service.

That is why the issues of impartiality, reliability and accuracy of information all play such an important role in the context of price comparison websites and are increasingly the subject of close scrutiny by consumer bodies. In particular issues of accurate pricing, comprehensiveness of information, and updating and verifying user-generated reviews5 deserve greater attention. There are also transparency aspects such as financing models, contractual relationship with retailers and service providers, as well as personal data collection and sharing that need to be examined.

5 http://bit.ly/UIoYOB

http://bit.ly/QzVaFX

http://bit.ly/ZfKFIq

http://bit.ly/TWHrtc

http://bit.ly/VjrDoA

http://bit.ly/UIoYOB

4 Consumer Focus

Consumer Focus has a vested interest in this policy area as for the past four years we have managed a voluntary accreditation scheme for domestic online energy price comparison services known as the Confidence Code.6 Its objective has been to implement and maintain standards that inspire consumer trust in price comparison sites that cover the energy market.7 Also, as a consumer watchdog, we need to be able to advise consumers on the accuracy and reliability of price comparison websites.

Therefore in order to gather evidence on reliability of price comparison websites, Consumer Focus commissioned eDigitalResearch to carry out a comprehensive mystery shopping survey of price comparison websites in March 2012.

In parallel price comparison websites have also drawn attention from regulators. For example last year the OFT wrote to 100 leading price comparison websites asking them to ensure they give clear information to consumers. This followed a websweep of 55 price comparison sites which identified scope for providing greater clarity on the way search results were presented, and making it clearer for consumers to identify the business operating the sites.8 Also last year, the energy regulator Ofgem launched an investigation into switching websites’ practices after its research found that even some more active and engaged consumers found price comparison sites confusing and some were sceptical of their independence.9

6 Due to changes in consumer landscape the responsibility for management of the Confidence Code will be moving to Ofgem by April 2013.

7 http://bit.ly/p9GdJI 8 http://bit.ly/QekYZi9 http://bit.ly/UCscFS; http://bit.ly/VK0vdE

At the EU level, last year the European Commission Health and Consumers Directorate initiated a multi-stakeholder dialogue on comparison tools with the aim to ensure consumers are provided with transparent and reliable information.

Our research investigated quality of information of price comparison websites across six markets: energy, mobile phones, broadband, rail fares, package holidays, home and car insurance. In total the survey examined 99 price comparison websites with 596 mystery shopping exercises taking place.

The research indicates that not all comparison websites have the same standards when it comes to the reliability and transparency of information provided. The research found that price comparison websites were a useful platform for a basic search, displaying a high proportion of relevant search details. They were also usually accurate about products’ availability from suppliers. However, consumers should not automatically assume that a price comparison website will save them money on their purchase.

The research revealed that it was only true in 21 per cent of cases. The research also found that not all price comparison websites allowed more tailored searches and some lacked clarity on costs. For example, we found that many did not disclose in the final costs compulsory fees such as a booking fee, early contract termination charge, cost of mobile handset and handset insurance. In other instances no compulsory additional charges were automatically added to the total costs, such as airport transfers and travel insurance or emergency home cover insurance.

http://bit.ly/p9GdJI

http://bit.ly/QekYZi

http://bit.ly/UCscFS

http://bit.ly/VK0vdE

5Comparing comparison sites

Performance also varied in terms of usability. For example some price comparison websites allowed the input of more available criteria at the outset to enable a tailor-made search specific to individual consumer’s needs. Whereas, others did not, which may have impacted on accuracy of search results.

In conclusion, we recommend regulators pursue monitoring performance of price comparison websites in terms of reliability, transparency of information, and ease of use in order to increase consumer trust in price comparison websites and ensure consumers receive reliable, clear and unbiased information.

We also call on price comparison websites to step up their game and ensure that consumer trust in information they provide is well founded. That means ensuring independence, impartiality and transparency of information so that any commercial relationship between price comparison website operators and suppliers are transparent to consumers and do not influence the accuracy or consistency of the information provided. In addition we also call on price comparison websites to implement the OFT recommendations into their business practices as below:10

● Be clear about the way information is presented (by price, by popularity, by relevance, ‘best deal’ etc)

● Be clear about the nature of the search, including market coverage, frequency of updates, pricing and stock availability

10 The OFT letter to the operators of 100 leading price comparison websites, 23 November 2012; http://bit.ly/VV0d5L

● Ensure there is a clear complaint and redress process for consumers who have complaints about the comparison service they used

● Ensure to have a clear and prominently displayed privacy policies in place which gives consumers the opportunity of opt-out of data sharing

We also urge consumers to go through an accredited site if available, check information on deals isn’t being filtered out, check how often the information is updated, use a number of different price comparison sites before choosing which product or service to take out, and where it is decided to go ahead with a purchase or switch check the identity of the business operating price comparison site, including business address, not just its website address.

Marzena Lipman Policy Manager Digital Communications

http://bit.ly/VV0d5L

Price Comparison Website Mystery Shopping

Research Report

March 2012

Prepared by: Prepared for: eDigitalResearch Vanbrugh House Grange Drive Hedge End Southampton SO30 2AF P: 01489 772920 www.edigitalresearch.com

Consumer Focus

eDigitalResearch Price Comparison Website Mystery Shopping Page 2 of 264

Contents

1. Executive Summary ....................................................................................................... 5

1.1 Background ....................................................................................................................... 5

1.2 Research Objectives ........................................................................................................... 5

1.3 Methodology ..................................................................................................................... 6

1.4 Compliance with Consumer Protection from Unfair Trading Regulations ............................. 6

1.5 Accuracy ............................................................................................................................ 8

1.6 Transparency ..................................................................................................................... 9

1.7 Dynamic pricing ............................................................................................................... 10

1.8 Accreditation ................................................................................................................... 10

1.9 Big 4 ................................................................................................................................ 11

2. Consumer Focus ....................................................................................................... 12

3. Background ............................................................................................................. 13

a. Overview ............................................................................................................................ 13

b. Regulations and standards .................................................................................................. 14

c. Previous research ............................................................................................................... 14

d. Definition of the problem ................................................................................................... 15

4. Research objectives ................................................................................................. 16

5. Methodology ........................................................................................................... 17

5.1 Type of research ............................................................................................................... 17

5.2 Number of price comparison websites and mystery shops ................................................. 17

5.3 Number of scenarios ......................................................................................................... 18

5.4 Supplier website visits ...................................................................................................... 18

5.5 Repeat visits ..................................................................................................................... 18

5.6 Compliance with Consumer Protection from Unfair Trading Regulations ............................ 19

6. Changes to initial project design .............................................................................. 20

6.1 Financial products ............................................................................................................. 20

6.2 Accredited price comparison websites............................................................................... 20

7. All markets: Cross market comparison ..................................................................... 22

7.1 All markets: Number of price comparison websites and mystery shops ............................. 22

7.2 All markets: Compliance with Consumer Protection from Unfair Trading Regulations ........ 22

7.3 All markets: Accuracy ....................................................................................................... 25

7.4 All markets: Transparency ................................................................................................ 26

7.5 All markets: Chapter summary ......................................................................................... 31

8. All markets: Accreditation ........................................................................................ 34

8.1 All markets: Accreditation and Compliance with Consumer Protection from Unfair Trading Regulations ............................................................................................................................ 34

8.2 All markets: Accreditation and Accuracy ........................................................................... 37

8.3 All markets: Accreditation and Transparency .................................................................... 38

8.4 All markets: Accreditation chapter summary .................................................................... 42

9. All markets: Big 4..................................................................................................... 44


9.1 All markets: Big 4 and Compliance with Consumer Protection from Unfair Trading Regulations ............................................................................................................................ 44

9.2 All markets: Big 4 and Accuracy ........................................................................................ 46

9.3 All markets: Big 4 and Transparency ................................................................................. 47

9.4 All markets: Big 4 chapter summary ................................................................................. 51

10. Energy.................................................................................................................. 53

10.1 Energy: Number of price comparison websites and mystery shops .................................. 53

10.2 Energy: Compliance with Consumer Protection from Unfair Trading Regulations ............. 53

10.3 Energy: Accuracy ............................................................................................................ 60

10.4 Energy: Transparency ..................................................................................................... 62

10.5 Energy: Accreditation ..................................................................................................... 65

10.6 Energy: Big 4 price comparison websites versus others ................................................... 67

10.7 Energy: Chapter summary .............................................................................................. 70

11. Broadband ........................................................................................................... 72

11.1 Broadband: Number of price comparison websites and mystery shops............................. 72

11.2 Broadband: Compliance with Consumer Protection from Unfair Trading Regulations ....... 72

11.3 Broadband: Accuracy ...................................................................................................... 79

11.4 Broadband: Transparency ............................................................................................... 82

11.5 Broadband: Accreditation ............................................................................................... 84

11.6 Broadband: Big 4 price comparison websites versus others ............................................. 88

11.7 Broadband: Chapter summary ........................................................................................ 92

12. Mobile phones ..................................................................................................... 95

12.1 Mobile phones: Number of price comparison websites and mystery shops ....................... 95

12.2 Mobile phones: Compliance with Consumer Protection from Unfair Trading Regulations .. 95

12.3 Mobile phones: Accuracy .............................................................................................. 102

12.4 Mobile phones: Transparency ....................................................................................... 104

12.5 Mobile phones: Accreditation ....................................................................................... 106

12.6 Mobile phones: Big 4 price comparison websites versus others ..................................... 112

12.7. Mobile phones: Chapter summary ................................................................................ 116

13. Insurance ........................................................................................................... 117

13.1 Insurance: Number of price comparison websites and mystery shops............................. 117

13.2 Insurance: Compliance with Consumer Protection from Unfair Trading Regulations ....... 117

13.3 Insurance: Accuracy ...................................................................................................... 125

13.4 Insurance: Transparency ............................................................................................... 127

13.5 Insurance: Accreditation ............................................................................................... 129

13.6 Insurance: Big 4 price comparison websites versus others .............................................. 137

13.7 Insurance: Chapter summary 14. Package holidays ..................................................... 143

14. Package holidays ....................................................................................................... 145

14.1 Package holidays: Number of price comparison websites and mystery shops................. 145

14.2 Package holidays: Compliance with Consumer Protection from Unfair Trading Regulations ............................................................................................................................................ 145

14.3 Package holidays: Transparency ................................................................................... 152


14.4 Package holidays: Accreditation ................................................................................... 154

14.5 Package holidays: Big 4 price comparison websites versus others .................................. 158

14.6 Package holidays: Dynamic pricing ............................................................................... 161

14.7 Package holidays: Chapter summary ............................................................................. 164

15. Rail fares ............................................................................................................ 166

15.1 Rail fares: Number of price comparison websites and mystery shops ............................ 166

15.2 Rail fares: Compliance with Consumer Protection from Unfair Trading Regulations ....... 166

15.3 Rail fares: Accuracy ...................................................................................................... 172

15.4 Rail fares: Transparency ............................................................................................... 173

15.5 Rail fares: Accreditation ............................................................................................... 175

15.6 Rail fares: Big 4 price comparison websites versus others .............................................. 179

15.7 Rail fares: Dynamic pricing ........................................................................................... 182

15.8 Rail fares: Chapter summary......................................................................................... 184

APPENDIX A ................................................................................................................. 186

APPENDIX B .................................................................................................................. 254

APPENDIX C .................................................................................................................. 263

PILOT ................................................................................................................................... 264


1. Executive Summary

1.1 Background

Price comparison websites have mushroomed in the last 10 years. Dozens of online price

comparison websites now exist. Some relate to specific markets (eg energy, insurance) while others

offer comparisons on a range of consumer goods and services. As well as providing quotes and

prices, some comparison websites allow customer rating and reviews. Some allow consumers to buy

the chosen product on the spot, others (notably where the application process is quite involved)

provide supplier contact details and/or links to suppliers so consumers can submit an application

online.

Their popularity appears to be growing. In Advertising of Prices (December 2010) the OFT reports

that 73 per cent of online consumers had used a price comparison website in the year to July 2010,

although only 15 per cent had actually made a purchase through such a website: in most cases,

consumers used the website to ‘window shop’ prices.

Price comparison websites must comply with Consumer Protection Regulations. This means that

prices must be accurate, up to date and comprehensive (ie it must be clear if there are additional

costs, such as a delivery charge), site owners must not misrepresent their independence, must not

post fictitious recommendations and must be open about suppliers who have paid for prominence.

There are now a number of accreditation schemes/voluntary guides to good practice covering some

of the markets served by price comparison websites. But it is not clear what gaps remain in terms of

markets/sites with no accreditation/best practice schemes or in terms of consumer need for such

schemes. It is also not clear to what extent all accreditation schemes/voluntary guides are successful

in raising standards. There is limited information on the extent to which price comparison websites

have been checked for compliance with Consumer Protection Regulations.

1.2 Research Objectives

Consumer Focus needs to be able to advise consumers on the reliability of price comparison

websites, especially in regulated markets, and to campaign for better enforcement of regulations

and/or design of the websites where necessary.

The overall objective of the research was to provide information to assist Consumer Focus achieve

these aims. The specific research objectives were to:

assess whether price comparison websites with an accreditation are significantly more reliable (in terms of transparency and accuracy) than those without an accreditation

compare the reliability of price comparison websites with different accreditation schemes

compare the performance of the Big 4 price comparison websites against others in the market

assess the extent to which price comparison websites comply with Consumer Protection from Unfair Trading regulations

identify potential consumer problems in terms of transparency, accuracy and pressure to make hasty purchase decisions through dynamic pricing


to identify any other issues that might help demonstrate how price comparison websites could work better for consumers

1.3 Methodology

The markets included in the project were:

energy

mobile phones

broadband

rail fares

package holidays

home and car insurance

The research was undertaken by gathering information from price comparison websites operating in

the markets of interest, plus carrying out mystery shopping visits to the sites.

General information was collected about each website (eg signposting of accreditations, how often

prices were updated, indications of independence from suppliers). This was a desktop research

exercise, conducted once per site or, in the case of multi-market sites, once for each appropriate

section of the site.

The price comparison websites were also tested several times each by mystery shoppers using a

range of scenarios designed to mimic real-life consumer activity (eg searching for available eco-

friendly dual fuel tariffs for a consumer wanting to pay quarterly via cheque). A total of 596 shopper

visits were undertaken.

The 99 price comparison websites to be included in the research were selected to:

have a range of accredited and non-accredited sites for each market

include the Big 4 in all markets in which they operate

1.4 Compliance with Consumer Protection from Unfair Trading Regulations

The more specific a consumer can be about their requirements at the start of a price comparison

website search, the better tailored the list of results will be. Although not all consumers will want to

specify their needs in detail at the outset, there should be sufficient choice of non-mandatory

variables to allow a detailed specification to be created if the consumer so desires. Variables will

have different levels of relevance to different consumers; a wide range of features helps meet all

needs.

Broadly speaking, price comparison websites do well in terms of providing options to specify basic

details, but can be somewhat lacking when it comes to going beyond those. Insurance is an

exception; a wide range of information is usually asked for at the start. Broadband was the market

least likely to allow a detailed specification at the outset, often requiring little more than a postcode

to calculate a comparison. Only one in 10 broadband sites gave consumers the option of specifying

usage limits, preferred supplier or contract length. Other examples of features that it would benefit

consumers to see more frequently when inputting their requirements are inclusive minutes/texts for

mobile phones, tariff type for energy and minimum accommodation rating for package holidays.


In the energy market, some price comparison websites did not ask for postcode or current usage,

hence quotes can be indicative only, which consumers might not realise. In the broadband market it

was sometimes unclear how ‘speed’ had been defined (ie best achievable anywhere or estimated for

that location). Consumers could easily assume it is one but in reality it is the other.

On a positive note, search results were relevant a high proportion of the time ie they took into

account all the criteria entered by the shopper, and did not include products outside what had been

asked for.

Price comparison websites were not particularly good at saying how the search results had been

ordered. Although it was, in many instances, fairly obvious (and usually based on cost), identifying

the variable used avoids any consumer misunderstanding. This would be a particular help in the

mobile phone market, where 29 per cent of shoppers saw no explanation and could not work out

the ordering system.

Being able to reorder the search results is useful, as it allows consumers to view the list in a variety

of ways. If a consumer decides they do not want a particular type of product, being able to filter out

some of the results and home in on those of interest, is also useful.

There was significant variability between the markets in terms of whether these facilities were

offered. For example, only 33 per cent of shoppers on energy sites could reorder their list, compared

to 82 per cent of broadband shoppers. Being able to filter ranged from 42 per cent of energy

shoppers to 74 per cent of those on package holiday price comparison websites. On broadband sites

a common reordering and filtering option was speed, but it was not always clear which speed

definition was being used, or possible to change the definition.

As their name suggests, price comparison websites exist primarily to allow consumers to compare

the cost of the different products available in a market. Therefore it would be reasonable to expect

that the search results were either presented initially in price order or could be sorted on that

variable. In all markets this was possible a majority of the time. The worst performers were mobile

phones and broadband, where 25 per cent and 18 per cent of shoppers respectively were unable to

do so. These might be minorities, but considering the importance of price ranking they are

meaningful. It makes the price comparison task onerous if the search results are not in price order,

and consumers could easily overlook the best deals.

The price comparison websites performed well in terms of correctly ordering on cost. When a list

could be sorted on price, the search results were checked to validate the order. Accuracy was high,

at 95 per cent or more, reaching 100 per cent for energy.

The markets varied in terms of there being restrictions or limitations in the search results over and

above any specified in the search criteria. It was unlikely for insurance (six per cent) but common for

broadband (59 per cent). This undoubtedly reflects the fact that to obtain an insurance quote

required inputting a range of details, while for broadband few criteria could be specified at the start

of a search. In other markets, there were many cases when the appearance of additional restrictions

or limitations could have been avoided by having more search criteria at the start.


For example, if a choice had been offered earlier of whether or not online account management was

acceptable (for energy) and whether or not refurbished handsets should be included (for mobile

phones) at least some of the additional restrictions and limitations would have been negated.

Beyond any additional restrictions or limitations in the search results, encountering them during the

application process was less likely, but nevertheless still occurred. This was particularly true for

broadband and mobile phones. Broadband shoppers faced (amongst others) minimum contract

lengths, deals only for new or existing customers and the necessity to have or switch to a specified

landline (eg British Telecom). Examples of discoveries made by mobile phone shoppers were that a

credit check was necessary or that payment had to be via Direct Debit.

In all markets, many of the costs quoted initially were either not the final cost or had potential

future expenditure associated with them. Due to differences in the way costs were added, the

markets were split into two groups for this part of the analysis.

Broadband, mobile phones and energy

Costs were sometimes added to the original quote. For broadband, a check of the first five quotes

revealed additional payments in 62 per cent of the lists, for the likes of line rental, installation and

equipment. Half of the lists comprising the first five results for energy contained extra costs, often a

cancellation fee for early contract termination. A third of equivalent mobile phone lists had extra

costs such as the price of the handset or an amount for handset insurance.

Although additional costs were less likely to arise when making an application for one specific

product, they were still in evidence for a third of broadband and mobile phone quotes.

Train fares, package holidays and insurance Train fares were likely to have a (previously unmentioned) booking fee added at the point of

purchase. A credit/debit card processing fee was another possible addition. In the package holiday

market departure taxes/local fees were sometimes not included in the initial quote, and on other

occasions supplier failure protection (which could not be removed) raised the cost of the holiday.

For insurance, extra costs which could not be removed were less common, but included windscreen

cover and personal injury insurance.

A different type of additional cost for these three markets was one added automatically, but which

could be taken out by the purchaser. There is a risk that these are overlooked, and the consumer

pays for something they either do not need or do not want. Examples were airport transfers and

travel insurance (for package holidays) and emergency home cover (for insurance).

1.5 Accuracy

In theory, the products available on price comparison websites are also available direct from the

supplier (eg the energy company or the broadband provider). In practice, most of them are. Across

all markets other than package holidays (for which supplier sites were not visited) 85 per cent of the

products were found on the relevant supplier site. It should not, however, be assumed that the

remaining 15 per cent could not be found because they were unavailable. The lack of success in

finding them could also be explained by shortcomings in the presentation and costing of products on

the supplier site.


Consumers should not automatically assume that a price comparison website will save them money

on their purchase. This was only true in a minority of cases (21 per cent). If a consumer bought

directly from the supplier as opposed to via the price comparison site, many products would have an

identical cost. Once again, there was a high level of market variability, from 83 per cent of mobile

phones having the same price tag to only 36 per cent of rail fares. When the costs were different,

half were lower on the supplier site and half were higher. Market variability was in evidence here

also; for insurance the supplier cost was usually higher, while for rail fares it was usually lower.

1.6 Transparency

Transparency varied widely between the markets. Energy was the best overall performer, followed

by broadband and insurance. Rail fares was the least consumer-friendly in relation to transparency.

The price comparison websites were somewhat lacking when it came to providing details of how

they operate. For example, few explained how they calculate the results of a comparison, although

this ranged from none of the package holiday sites to a much better 61 per cent of those giving

energy comparisons. It would be useful for consumers to know whether factors other than cost had

been taken into account.

With the exception of energy, not many had the date of the last price update adjacent to their

quotes. In most markets it was a minority that said how often price updates took place; only

broadband and energy had reasonable proportions doing so. If this information is not made clear, it

is impossible for a consumer to know if the prices are up-to-date or not.

The price comparison sites performed better in terms of explaining how income was generated,

although again this was variable (from none of the rail fares sites to 83 per cent of the energy sites).

Some identified the number of providers compared, but few said why some providers or products

might not be included.

A Code of Conduct was rarely seen. Only one of the 99 price comparison websites contained a policy

for dealing with rogue or fraudulent traders, and only two allowed customers to report suspicious

activity of this sort directly to the site.

When the site included provider or product ratings, there was often an explanation of how these

were calculated and in many cases the ratings took customer reviews/scores into account.

In general, the sites were good at providing or signposting additional information (with the

exception of the rail fares websites). Having a glossary of terms used was extremely rare for mobile

phone sites (seven per cent) and extremely common on rail fares sites.

The presence of a customer complaints policy also depended on the market; no rail fares sites had

one but 62 per cent of package holiday sites did. There was substantial market variation in whether

customers could issue a complaint online and whether the number of days that would be taken to

respond was stated.


1.7 Dynamic pricing

There is no evidence to support dynamic pricing in the rail fares market. A low proportion of mystery

shops increased in price on one or more of the three return visits to the price comparison site but

never decreased. Additionally, all of them were amongst shoppers who cleared their cookies and

browsing history between visits (ie created conditions that should substantially decrease the

opportunity for dynamic pricing). None occurred amongst the shoppers who did not clear their

cookies and browsing history (ie the shoppers who created the optimum conditions for dynamic

pricing).

For package holidays the picture was different, and might indicate dynamic pricing in this market.

Half of the holidays found on all three return visits to the price comparison site were the same cost

each time. Although only a small proportion (eight per cent) showed the pattern of price movements

that might be expected from dynamic pricing. The incidence fell to three per cent when shoppers

cleared their cookies and browsing history but rose to 15 per cent when they did not do so. In other

words, when the conditions for dynamic pricing are present, there is a greater likelihood of

encountering a price increase than when the conditions for dynamic pricing are negated.

1.8 Accreditation

In each market there were at least two accredited price comparison websites. Looking at whether

the accredited sites within a market out-perform the non-accredited sites reveals a mixed picture. In

all markets there is variability; accredited sites are better on some parameters, while on others the

non-accredited sites come to the fore, and there were also areas of equal performance.

For energy, there were more occasions when the accredited sites were better than the non-

accredited sites. Examples include 60 per cent of accredited sites providing details about how often

prices are updated versus 30 per cent of non-accredited sites, and 80 per cent of accredited sites

saying how many providers they compared versus only 37 per cent of the non-accredited ones.

Postcode and current usage were more frequently items that could be input at the start of a search

on an accredited site, meaning an accurate quote is given instead of an estimate.

For package holidays, the reverse was true, and in many areas it was the non-accredited sites that

out-performed the accredited ones. Non-accredited sites were less likely to have departure tax/local

fees as an additional cost (ie it was included in the cost from the outset), meaning consumers are not

faced with what could be a substantial increase in the cost of the holiday once they have

commenced the booking. Non-accredited sites had more options beyond the basic requirements in

terms of specifying the criteria to be used for the search (eg board type or minimum accommodation

rating).

In rail fares there were only two non-accredited sites, and on balance they were probably worse

performers than the accredited ones.

The majority of comparison sites in the insurance market were non-accredited sites with only one

website accredited by British Insurance Brokers Association (BIBA). Overall comparisons between

accredited and non-accredited sites revealed a mixed picture although BIBA accredited status made

little difference to the usability or overall impression of the website.


Generally accredited sites are more likely to offer a wide range of filtering and reordering options as

well as providing a clear explanation about the initial ordering of results. Search results were more

likely to be found on the supplier site if the initial quote came from a non-accredited rather than an

accredited website. Although these results were not always found to be accurate with only 47 per

cent of non-accredited comparison sites advertising the same policy cost as found on supplier site.

For broadband, four websites were accredited under the Ofcom Accreditation scheme for price

comparison calculators. The remaining 12 sites were classified as non-accredited for the purposes of

this research. Generally, the accredited sites performed slightly better than the non-accredited sites

on a range of factors including the provision of clear information about the operation of the website

and explanation of terminology used. Accreditation made little difference to whether mystery

shoppers were able to find the same package at the same price on the supplier site with similar

proportions on both types of site finding the same package at the same price (from comparison site

to supplier site).

In mobile phones there were four websites also accredited under the Ofcom Accreditation scheme

for price comparison calculators. A further 11 sites were classed as non-accredited websites and

overall the comparison between the two types of websites revealed a mixed picture. Accredited

sites performed better in providing accurate results without additional costs or restrictions applied

to the initial quote. Although non-accredited sites were generally more user-friendly including a

range of filters and reordering options and were rated highly by mystery shoppers. Again,

accreditation made little difference to the validity of search results with similar proportions on both

types of site finding the same package at the same price (from comparison site to supplier site).

1.9 Big 4

There was an equally mixed picture when the Big 4 price comparison websites were compared to

the non-Big 4 in each market. In some markets the Big 4 were slightly better (eg rail fares), in others

slightly worse (eg package holidays), and in others Big 4 and non-Big 4 performed equally. Hence it

very much depends on the market that a consumer is searching as to whether they will find a Big 4

site more consumer-friendly or not.


2. Consumer Focus

Consumer Focus is the statutory consumer champion for England, Wales, Scotland and (for postal

consumers) Northern Ireland.

We operate across the whole of the economy, persuading businesses, public services and policy-

makers to put consumers at the heart of what they do.

Consumer Focus tackles the issues that matter to consumers, and aims to give people a stronger

voice. We don’t just draw attention to problems – we work with consumers and with a range of

organisations to champion creative solutions that make a difference to consumers’ lives.

Following the Government’s consumer advocacy reforms, we will continue to act in the consumer

interest across a wide range of sectors until our general advocacy role passes to the Citizens Advice

service in April 2013.

As part of the reforms, Consumer Focus will establish a new unit to identify and represent

consumers’ interests in complex, regulated sectors, including energy and postal issues and, in

Scotland, water.


3. Background

a. Overview

Price comparison websites have mushroomed in the last 10 years. Dozens of online price

comparison websites now exist. Some relate to specific markets (eg energy, insurance) while others

offer comparisons on a range of consumer goods and services. As well as providing quotes and

prices, some comparison websites allow customer rating and reviews. Some allow consumers to buy

the chosen product on the spot, others (notably where the application process is quite involved)

provide supplier contact details and/or links to suppliers so consumers can submit an application

online.

Mintel (Web Aggregators, October 2009, referenced in the Office of Fair Trading (OFT) report

Advertising of prices 1

) reported that four price comparison websites dominate in the UK:

Moneysupermarket.com, Confused.com, GoCompare.com and Comparethemarket.com. In a three

month period in 2009, these four websites accounted for 80 per cent of price comparison website

visits.

Their popularity appears to be growing. In Advertising of Prices (December 2010) the OFT reports

that 73 per cent of online consumers had used a price comparison website in the year to July 2010,

although only 15 per cent had actually made a purchase through such a website: in most cases,

consumers used the website to ‘window shop’ prices.

A slightly more recent study (based on interviews conducted December 2010-February 2011)

commissioned by the European Commission2 found that 81 per cent of online consumers in the UK

had used a price comparison website in the previous 12 months. This study also found that only a

minority of users of price comparison websites made their purchase through the website itself.

In 2009, Mintel had reported 62 per cent of internet users obtaining quotes for and/or buying a

financial product through a price comparison website.

On the face of it, price comparison websites offer a valuable service, particularly in markets where,

due to the complexity of the products or the time it would take to obtain information from

individual suppliers, it is difficult for consumers to research the markets and make informed choices.

Logically, price comparison websites that do not offer a genuine and honest service would fail to

attract repeat usage and gain a poor reputation.

Indeed, OFT’s research3 reveals that consumers trust price comparison websites for energy, travel

and insurance (car, home and travel insurances are the products most commonly bought through

price comparison websites), but regard them as less reliable for other financial products, mobile

phones and media.

1 See Chapter 9, http://www.oft.gov.uk/shared_oft/market-studies/AoP/OFT1291.pdf

2 Consumer Market Study on the Functioning of E-Commerce and Internet Marketing and Selling

Techniques in the Retail of Goods, September 2011 3 http://www.oft.gov.uk/shared_oft/market-studies/AoP/Annexe-H.pdf

http://www.oft.gov.uk/shared_oft/market-studies/AoP/OFT1291.pdf

http://www.oft.gov.uk/shared_oft/market-studies/AoP/Annexe-H.pdf


b. Regulations and standards

Price comparison websites must comply with Consumer Protection Regulations. This means that

prices must be accurate, up to date and comprehensive (ie it must be clear if there are additional

costs, such as a delivery charge), site owners must not misrepresent their independence, must not

post fictitious recommendations and must be open about suppliers who have paid for prominence.

In some markets there is an accreditation scheme for price comparison websites. Consumer Focus

operates a voluntary accreditation scheme for online domestic energy price comparison services

(Confidence Code)4; Ofcom does so for telephone and broadband services

5; National Rail for

websites providing train fare comparisons and the British Insurance Brokers’ Association (BIBA) in

relation to websites comparing insurance products. Other accreditation schemes operate across

markets, such as those run by Shopsafe6 and SafeBuy7.

The Consumer Focus Confidence Code sets standards for domestic energy price comparison services.

There is also an accreditation mark which can provide consumers with assurances about the service.

Accredited websites must commit to provide an independent, comprehensive and accurate service,

and their service is subject to an annual audit.

In 2010 the Association of British Insurers (ABI) launched a guide to good practice for insurance price

comparison sites.

In July 2011 the Financial Services Authority (FSA) warned price comparison websites covering

insurance markets that, where they provide more than introductions to insurers, they must provide

details about their relationship with the insurers, whether they have a financial interest and the

procedure for complaints.

c. Previous research

OFT has undertaken mystery shopping and consumer usability tests8 on 10 price comparison

websites in selected markets (car hire, flights, television, mobile phones, CDs and broadband). OFT

found inconsistencies in results produced by different websites for the same product criteria.

The European Commission research referred to earlier also involved a substantial mystery shopping

and price check exercise on 30 price comparison websites in the UK, focusing on mobile phones,

laptops, digital cameras, sat-nav, LCD televisions, MP3 players, perfume, video games, traditional

toys, men’s and women’s clothing, footwear, power tools, instant coffee and standard formula milk.

This research found most of the UK price comparison websites examined did not provide an

explanation of their default ranking (the rankings are not always based on price alone), why some

retailers/offers were listed but not others, or of their business model (eg whether suppliers had paid

for ranking placement).

4 http://www.consumerfocus.org.uk/get-advice/energy/confidence-code

5 http://ask.ofcom.org.uk/help/services-and-billing/pr_com_acc

6 http://www.shopsafe.co.uk/

7 http://www.safebuy.org.uk/index.html

8 Pages 172-207 http://www.oft.gov.uk/shared_oft/market-studies/AoP/Annexe-H.pdf

http://www.consumerfocus.org.uk/get-advice/energy/confidence-code

http://ask.ofcom.org.uk/help/services-and-billing/pr_com_acc

http://www.shopsafe.co.uk/

http://www.safebuy.org.uk/index.html

http://www.oft.gov.uk/shared_oft/market-studies/AoP/Annexe-H.pdf


Across Europe, many price comparison websites failed to provide full information about delivery

costs, taxes and product availability. In 80 per cent of price checks, the prices were accurate in that

they matched those advertised by the retailers themselves. However, when ordered by price the

cheapest correct product did not always appear in the top five. Most of the price comparison

websites included in the study did not provide information about how often their prices were

updated or their policy regarding fraudulent or rogue traders. Few had any accreditation trustmarks.

d. Definition of the problem

There are now a number of accreditation schemes/voluntary guides to good practice covering some

of the markets served by price comparison websites. But it is not clear what gaps remain in terms of

markets/sites with no accreditation/best practice schemes or in terms of consumer need for such

schemes.

It is also not clear to what extent all accreditation schemes/voluntary guides are successful in raising

standards.

There is limited information on the extent to which price comparison websites have been checked

for compliance with Consumer Protection Regulations.

Most of the European Commission research and also the recent work by the OFT has focused on

price comparison websites serving markets for tangible goods rather than more complex services.

Price comparison websites serving some important regulated markets (eg financial services) do not

appear to have been subjected to rigorous independent testing.

An issue that is known to have occurred within the airline industry and is suspected of becoming

more widespread is that of dynamic pricing. This is where the price of the offer changes in response

to demand and/or consumer interest. The website can detect how many times a consumer has

looked at the offer and adjust the price accordingly (eg by gradually raising it so as to encourage a

faster decision to buy).

There is speculation as to whether the practice has extended to include analysing consumer internet

browsing behaviour between visits. The price might be raised even further if a consumer is deemed

to be particularly keen by virtue of having visited websites, or carried out searches, that have a

direct association with their potential purchase. For example, if a price comparison website visitor

has looked for a package holiday to Florida, they might be classed as a more likely customer if they

have also looked on the Disney World website and searched for other Florida attractions.


4. Research objectives

Consumer Focus needs to be able to advise consumers on the reliability of price comparison

websites, especially in regulated markets, and to campaign for better enforcement of regulations

and/or design of the websites where necessary.

The overall objective of the research was to provide information to assist Consumer Focus achieve

these aims. The specific research objectives were to:

assess whether price comparison websites with an accreditation are significantly more reliable (in terms of transparency and accuracy) than those without an accreditation

compare the reliability of price comparison websites with different accreditation schemes

compare the performance of the Big 4 price comparison websites against others in the market

assess the extent to which price comparison websites comply with Consumer Protection from Unfair Trading regulations

identify potential consumer problems in terms of:

o transparency: do websites make it clear how often prices are updated; how many suppliers they feature; the basis of their relationship with suppliers; the basis of default listings, and what fees and charges will be added to the offer if the consumer decides to buy

o accuracy of offers: are the cheapest deals for a given scenario actually available from the supplier, for the same cost and with the same conditions

o pressure to make hasty purchase decisions through dynamic pricing

to identify any other issues that might help demonstrate how price comparison websites could work better for consumers

to fill gaps in the European Commission and OFT research by examining price comparison websites serving markets that these studies did not include

The markets included in the project were:

energy

mobile phones

broadband

rail fares

package holidays

home and car insurance


5. Methodology

5.1 Type of research

The research was undertaken by gathering information from price comparison websites operating in

the markets of interest, plus carrying out mystery shopping visits to the sites.

General information was collected about each website (eg signposting of accreditations, how often

prices were updated, indications of independence from suppliers). This was a desktop research

exercise, conducted once per site or, in the case of multi-market sites, once for each appropriate

section of the site.

The price comparison websites were also tested several times each by mystery shoppers using a

range of scenarios designed to mimic real-life consumer activity (eg searching for available eco-

friendly dual fuel tariffs for a consumer wanting to pay quarterly via cheque).

The forms used in the information gathering and the mystery shopping are appended (Appendix A).

The price comparison websites to be included in the research were selected to:

have a range of accredited and non-accredited sites for each market

include the Big 4 in all markets in which they operate

5.2 Number of price comparison websites and mystery shops

The tables below show summary details of the price comparison websites and shopper visits to each. In each market, the number of price comparison websites tested was a high proportion of the total. More detail is included in Appendix B. Market Total Accredited Non-accredited Big 4 Others

Energy 18 10 8 4 14 Mobile phones 15 4 11 3 12 Broadband 16 4 12 4 12 Rail fares 79 5 2 1 6 Package holidays 13 2 11 3 10 Insurance 30 2 28 8 22

Total 99 27 72 23 76

9 Maximum possible number of websites


Market Total Accredited Non-accredited Big 4 Others

Energy 116 62 54 27 89 Mobile phones 111 46 65 18 93 Broadband 122 45 77 25 97 Rail fares 56 40 16 8 48 Package holidays 77 12 65 17 60 Insurance 114 12 102 42 72

Total 596 217 379 137 459

5.3 Number of scenarios

Six scenarios were used for each market (12 in the case of insurance, split into six for each of home

insurance and car insurance). The scenarios covered a range of consumer situations, in order that

the price comparison websites were given a thorough testing. The scenarios used are outlined in

Appendix B.

5.4 Supplier website visits

Once shoppers reached the stage of having a list of quotes in price order (if possible), they were

instructed to select the nth product in the list, where n = 1st to 5th randomly assigned.

On completion of each price comparison website visit, shoppers immediately visited the supplier

website for the product they had selected, and attempted to find the same product.

For the mobile phone market, shoppers using scenarios focusing on a particular make of handset (eg

Nokia, Samsung) were instructed to visit the manufacturer’s website and shoppers using scenarios

focusing on the service provider (eg Orange or Vodafone) visited the provider’s website.

Supplier website visits were not undertaken for package holidays, due to the difficulty of

ascertaining the correct supplier sites to visit.

5.5 Repeat visits

In order to establish whether dynamic pricing was taking place in the package holiday and rail fares

markets, three return visits were made to each price comparison website over a period of 1-2 days.

Shoppers recorded whether the holiday or rail fare could still be found and, if so, noted its cost, plus

whether that was the same as, lower than or higher than the previous visit.

There could, of course, be genuine variation in the price of a holiday or rail fare which is not linked to

the price comparison website knowing the consumer is a repeat visitor. After each of the four visits

(the initial one and the three returns) half of the shoppers cleared their cookies and browsing

history. The other half did not do so, and undertook internet activity with a direct association to the

holiday or rail journey scenario being used (eg looking at the accommodation on TripAdvisor,

searching for local attractions in the area to be visited) between return visits to the price comparison

website. A comparison of any price changes between the two groups would highlight whether

dynamic pricing was taking place.


Additionally, a visit to the supplier website to obtain journey costs was undertaken after the first and

last price comparison website visits for the rail fares market.

5.6 Compliance with Consumer Protection from Unfair Trading Regulations

Consumer Protection from Unfair Trading Regulations (CPRs) outline in detail how traders should

operate when selling to consumers. Without access to the workings of each price comparison

website beyond those visible to consumers, the research had to rely on assessing compliance with

CPRs using only publicly available information, gathered from the price comparison websites and

from the information supplied when shoppers searched for a quote. In order to determine accuracy,

this information was compared to details on the relevant supplier website, where appropriate.

Compliance with CPRs was assessed for the following:

Prices must be accurate

Assessed by comparing the cost of the product given on the price comparison website against that

stated on the supplier website.

Prices must be up-to-date

Assessed by searching for information about how frequently prices were updated.

Details of the product must be comprehensive

Assessed by comparing product details on the price comparison website against those on the

supplier website.

Site owners must not misrepresent their independence

We recorded whether or not the price comparison website provided details of income-generation

(eg paid per ‘click through’, paid per successful application/switch, revenue gained through

advertising). However, sites may not have provided details of all sources of income.

Site owners must not post fictitious recommendations

We recorded whether or not the price comparison website included customer reviews/scores, but

obviously were unable to say whether these were fictitious or not.

Site owners must be open about suppliers who have paid for prominence

We recorded whether or not the price comparison website provided details of income-generation,

including whether or not any providers had paid to appear on the site. However, it cannot be

assumed that sites without this information did not include suppliers who had paid for prominence.


6. Changes to initial project design

6.1 Financial products

At the start of the project the markets to be covered were credit cards; savings accounts; personal

current accounts; and home, car and travel insurance. During the design stage, it became clear that

combining such a wide range of financial products and treating them as one market was not

possible, due to a lack of product commonality. Aggregating and analysing data about these

products would pose too many practical difficulties to be feasible.

In order to adhere to the tight deadline for reporting, the scope of the project could not be

increased to cover each financial product as an individual market. A decision had to be made on

which product to cover in the research.

Insurance was felt to be of particular interest. In July 2011, the Financial Services Authority (FSA)

warned price comparison websites covering insurance markets that, where they provide more than

introductions to insurers, they must provide details about their relationship with the insurers,

whether they have a financial interest and the procedure for complaints. In order to examine

compliance with these instructions, home and car insurance were chosen for the research. They are

products with sufficient commonality to be combined for analysis and reporting.

During fieldwork one of the price comparison websites selected for the insurance market was found

not to show any quotes. Although a consumer could search for different types of insurance, and to

some extent filter on product features, the end result was simply a list of suppliers. On clicking to

obtain more information the consumer is taken to the supplier website, where full details of their

current situation and their insurance requirements are entered and a list of policies shown. As the

modus operandi of this price comparison website is different to that of all the other price

comparison websites included in the research, a decision was taken to exclude it from the mystery

shopping element of the project. It was, however, possible to include it in the desktop research

element, which gathered operational information about the website.

6.2 Accredited price comparison websites

When initially scoping the research, it was known that the following accreditations existed within the

markets of interest:

Consumer Focus accreditation (energy)

Ofcom accreditation (mobile phone and broadband)

National Rail accreditation (train fares)

Further examination of price comparison websites revealed other accreditations:

British Insurance Brokers’ Association accreditation (insurance)

International Air Transport Association (IATA) (package holidays)

Shopsafe accreditation (multi sector)

SafeBuy accreditation (multi sector)

In addition, a plethora of other terms were found, such as ‘authorised & regulated by the FSA’,

‘approved by Ofcom’ and ‘OFT licenced’.


Price comparison websites for insurance must be authorised by the Financial Services Authority, but

while the FSA would take action against sites breaching FSA rules, the organisation does not pre-

approve an insurance price comparison website’s content.

A debate ensued about whether any of the aforementioned authorisations, approvals, etc. should be

classed as accreditations. The conclusion was that, on the basis of the following definition, price

comparison websites using such terms could not be treated as accredited:

‘Certification of competence in a specified subject or areas of expertise,

and of the integrity of an agency, firm, group or person, awarded by a duly

recognised and respected accrediting organisation’10

10

Definition from http://www.businessdictionary.com

http://www.businessdictionary.com/


7. All markets: Cross market comparison

7.1 All markets: Number of price comparison websites and mystery shops

A total of 596 mystery shops involving 99 price comparison websites were carried out across the six markets. This chapter looks at similarities and differences between the markets.

7.2 All markets: Compliance with Consumer Protection from Unfair Trading Regulations

a. All markets: Initial list of quotes

A consistently high proportion of the initial search results were based on all the criteria entered by

the shopper, with the exception of Insurance, for which the proportion was a little lower. None of

the markets were particularly good at explaining how the list had been ordered; the worst performer

was Energy, where only 30 per cent of shoppers were notified of the ranking criteria used.

There was a lot of variability between the markets in terms of consumers being given the

opportunity to reorder the initial search results, ranging from 82 per cent for Broadband to 33 per

cent for Energy. The Energy market also had fewest shoppers able to filter the initial quote list,

which was a common option for Mobile phone and Package holiday shoppers.

A majority of the initial results in all markets were either in price order or could be sorted into price

order, although less often for the two Telecoms markets. When checked, price ordering was correct

in most cases for each market, reaching 100 per cent accuracy for Energy.

Table 7A: Initial list of quotes

Total B’band Energy Insur-ance

Mobile phone

Holi-days

Rail fares

Base: All mystery shops (596) %

(122) %

(116) %

(114) %

(111) %

(77) %

(56) %

List was based on all criteria input 95 94 99 89 95 97 98

Base: All mystery shops exc. rail fares (540) %

(122) %

(116) %

(114) %

(111) %

(77) %

Was an explanation about list order 41 54 30 36 40 49

Possible to reorder initial list 57 82 33 57 52 62

Possible to filter initial list 60 59 42 58 72 74

List initially in price order or are able to sort into price order

88 82 94 96 75 95

Base: All mystery shops exc. rail fares where can get list into price order

(475) %

(100) %

(109) %

(110) %

(83) %

(73) %

List was actually in price order 97 97 100 99 94 93

Shoppers in all markets were more likely to see 10 or more quotes on the first page of results than

see fewer than 10, although less so for Broadband. Providing more than one page of quotes was

variable; high for Package holidays but lower elsewhere, especially for Insurance. The number of

pages and quotes seen does, of course, depend not only on the number of providers and products

the price comparison site compares, but also how specific the consumer’s requirements are.


Table 7B: Number of quotes


Mobile phone

Holi-days

Rail fares

Base: All mystery shops exc. rail fares (540) %

(122) %

(116) %

(114) %

(111) %

(77) %

1-9 quotes on first page 17 25 16 8 19 16 - 10+ quotes on first page 83 76 84 92 81 84 -

More than one page of quotes 47 51 40 20 60 73 -

A requirement of the Ofcom accreditation scheme for the Mobile phone and Broadband markets is that a telephone number is provided for each product, to allow consumers to make an application by phone if they would prefer. Many of the Broadband shoppers confirmed that this was the case, but an extremely low proportion did so for Mobile phones. Table 7C: Telephone number for phone purchase


Mobile phone

Holi-days

Rail fares

Base: All mystery shops exc. energy, package holidays and rail fares

(347) %

(122) %

(114) %

(111) %

Telephone number available for phone purchase

30 64 - 20 2 - -

b. All markets: Additional restrictions/limitations and costs

Shoppers looked at the first five quotes shown to ascertain whether any contained restrictions or

limitations that had not been mentioned when entering the search criteria. For Insurance, shoppers

were unlikely to discover any. For Broadband and Energy, half or more uncovered additional

restrictions or limitations.

It was less likely that additional restrictions or limitations would be uncovered when making an

application. This was particularly true for Insurance and Package holidays, but less so for Broadband

and Mobile phones.

Having additional costs in the first five quotes was lowest for Broadband and highest for Mobile

phones. Additional costs at the point of application were less common, rarely appearing at all in the

Energy market.


Table 7D: Additional restrictions/limitations and costs


Mobile phone

Holi-days

Rail fares

Base: All mystery shops exc. rail fares and package holidays

(463) %

(122) %

(116) %

(114) %

(111) %

No additional restrictions/limitations in any of first 5 quotes

62 41 47 94 67 - -


(122) %

(116) %

(114) %

(111) %

(77) %

(56) %

No additional restrictions/limitations appeared when making application

85 73 86 92 78 95 87

Base: All mystery shops exc. rail fares, package holidays and insurance

(349) %

(122) %

(116) %

(111) %

No additional costs in any of first 5 quotes

52 38 52 - 67 - -

No additional costs appear when making application

77 68 95 - 68 - -

c. All markets: Impressions of website

Shoppers were asked to give their impressions of the price comparison websites. Most did not find

anything confusing or misleading, and their overall impression was largely positive. A majority felt

they had sufficient information to feel comfortable proceeding with a purchase, had they been a real

consumer.

However, there is still room for improvement, particularly for Package holidays, which was

consistently rated lower than other markets. On the other hand, Energy was a strong performer,

with high proportions of shoppers not finding anything confusing or misleading, and almost all happy

to proceed with a purchase.

Table 7E: Impressions of website


Mobile phone

Holi-days

Rail fares


(122) %

(116) %

(114) %

(111) %

(77) %

(56) %

Nothing found confusing 85 84 93 89 78 77 80 Nothing found misleading 88 85 95 94 84 82 84

Overall impression is positive 76 76 83 76 68 69 84 Sufficient information to make purchase 87 91 95 80 84 79 95


7.3 All markets: Accuracy

a. All markets: Whether product was available on supplier website

After the shoppers had selected their assigned product from the first five in the initial list of quotes

(ie selected the nth product as per their instructions) they visited the supplier website and

attempted to find the same product. They were successful in a majority of cases, but less so for

Mobile phones in comparison to the other markets.

Table 7F: Product found on supplier website


Mobile phone

Holi-days

Rail fares

Base: All mystery shops except package holidays

(519) %

(122) %

(116) %

(114) %

(111) %

(56) %

Found product on supplier website 85 89 86 95 63 - 98

b. All markets: Cost on supplier website compared to cost on price comparison website

When a product was found on the supplier website, most were the same price for the two Telecom

markets. This was less often the case for Energy and Insurance, where around half of the supplier

costs were the same as the price comparison website quote. There was least comparability for Rail

fares, where two-thirds of the tickets had a different price.

Table 7G: How cost on supplier site compared to cost on price comparison site


Mobile phone

Holi-days

Rail fares

Base: All mystery shops except package holidays where product found on supplier site

(435) %

(108) %

(100) %

(102) %

(70) %

(55) %

Supplier site cost was…

…same as price comparison website 59 75 47 50 83 - 36 …higher than price comparison website 21 14 29 30 7 - 20 …lower than price comparison website 20 11 24 20 10 - 44

A minority of shoppers could uncover the reason for the price difference, although the proportion varied between markets. The reason was more obvious in Broadband and Rail fares compared to the other markets, but fell to a very low proportion amongst Insurance shoppers. Table 7H: Can see the reason for the cost difference


Mobile phone

Holi-days

Rail fares

Base: All mystery shops except package holidays where product found on supplier site and cost was different

(178) %

(27) %

(53) %

(51) %

(12) %

(35) %

Yes, can see the reason for the cost difference

22 41 15 6 33 - 37


c. All markets: Whether the supplier site had additional details

In all the markets where supplier site visits were undertaken, a proportion of suppliers provided

product details that were additional to those in the price comparison website. This occurred more

frequently in the Telecoms markets compared to the rest, but all markets would benefit from

improvements in this area.

Improvements could be done in two ways – by ensuring the product description is comprehensive,

and by giving the consumer more options to initially define, and then further refine, their search. For

example, a common additional restriction in the Broadband market was a minimum contract length,

which could easily be asked about upfront.

Table 7I: Supplier site had additional details


Mobile phone

Holi-days

Rail fares

Base: All mystery shops except package holidays where product found on supplier site

(435) %

(108) %

(100) %

(102) %

(70) %

(55) %

Yes, supplier site has additional details 22 31 17 19 25 - 18 No, supplier site does not have additional

details 78 69 83 81 75 - 82

7.4 All markets: Transparency

a. All markets: Information provided

In some markets there was a relatively high incidence of price comparison sites that identified the

number of providers compared (eg Energy and Insurance), while in others such sites were in a

minority (eg Rail fares). This is important information – a consumer might decide not to proceed

with a comparison if aware that the number of products or providers was low.

Unless informed otherwise, a consumer could reasonably assume that a price comparison website

covers all, or at least most, suppliers and products in that market. Across all markets, very few price

comparison sites explained why some suppliers or products might not be included. The best

performer in this respect was Insurance. The worst was Rail fares, where none of the sites provided

this information.

Transparency regarding how the results of a comparison are calculated varied between markets.

Energy was a clear leader, with many sites providing an explanation. None of the Package holiday

sites, and a very low percentage of those providing Insurance quotes, gave details of how they

compare products. This information needs to be more widespread, as all consumers need to feel

confident that results are being calculated in an equitable and comprehensive manner.


Table 7J: Information provided (1)


Mobile phone

Holi-days

Rail fares

Base: All websites (99) %

(16) %

(18) %

(30) %

(15) %

(13) %

(7) %

Identifies number of providers compared 51 37 61 60 53 38 29

Explains why some providers/products might not be included

11 12 11 17 7 8 0

Provides an explanation of how it calculates the results of a comparison

22 31 61 7 20 0 14

Around 1 in 10 price comparison websites in any market had a Code of Conduct on the site, apart

from Rail fares, where this was never present.

For consumers unfamiliar with a particular market, definitions of the key terminology would be

useful. Providing a glossary to explain the main words and phrases used ranged from a minority of

Mobile phone and Package holiday sites to a majority of those dealing with Rail fares.

In every market apart from Rail fares, most or all of the price comparison websites either provided

or signposted additional information that could be helpful.

Table 7K: Information provided (2)


Mobile phone

Holi-days

Rail fares


(16) %

(18) %

(30) %

(15) %

(13) %

(7) %

Code of Conduct on site 8 12 11 7 7 8 0 Site includes a glossary of terms used 26 37 22 27 7 8 86

Site provides or signposts additional information

90 100 89 100 80 100 57

In some markets, the price comparison websites did not appear keen to divulge details of how they

generated income. No Rail fare sites gave this information, and a low proportion of Package holiday

sites did so. Conversely, most Energy sites were open about their means of income-generation, as

were many in the Insurance and Broadband markets. Amongst those giving details, being paid on

completion of a successful application/purchase/switch was usually the main source of income.


Table 7L: Generating income


Mobile phone

Holi-days

Rail fares


(16) %

(18) %

(30) %

(15) %

(13) %

(7) %

Explains how the site generates income 56 69 83 70 40 15 0

Base: All websites explaining how the site generates income

(55) %

(11) %

(15) %

(21) %

(6) %

(2) %

(0) %

Paid on completion of a successful application/switch

82 73 93 76 100 50 -

From advertising sales 16 36 7 14 17 0 - Providers pay to appear on the

comparison site 9 18 0 14 0 0 -

Paid per 'click through' to a providers site 9 18 0 10 0 50 - Other means 11 9 7 19 0 0 -

The Energy market price comparison sites were way ahead of those operating in the other markets

when it came to letting consumers know the date of the most recent price update; 58 per cent of

shoppers saw update notification beside the list of quotes, compared to 16 per cent or less in the

other markets.

Although a minority of Insurance shoppers saw a note beside the search results saying when prices

were last updated, most of the Insurance sites contained price updating information elsewhere on

the site (63 per cent). This also applied to half of the Energy websites.

Table 7M: Price updating


Mobile phone

Holi-days

Rail fares


(122) %

(116) %

(114) %

(111) %

(77) %

(56) %

Note beside quotes to say when prices were last updated

17 16 58 4 3 6 4


(16) %

(18) %

(30) %

(15) %

(13) %

(7) %

Website says how often prices updated 30 63 50 20 20 15 0

In some markets it was quite common for provider advertising to appear on the price comparison

sites, for example Broadband and Package holidays. In others, such as Energy and Rail fares,

advertising was infrequent or non-existent. If advertising was present, it was most often found on

the page where consumers specify their requirements.


Table 7N: Advertising of provider sites


Mobile phone

Holi-days

Rail fares


(16) %

(18) %

(30) %

(15) %

(13) %

(7) %

Site includes provider site advertising 30 50 6 37 27 46 0 Base: All websites with provider advertising (30)

% (8) %

(1) %

(11) %

(4) %

(6) %

(0) %

Provider advertising appears… …on criteria pages (where details and

requirements are entered) 97 100 100 100 100 83 -

…on homepage 87 87 100 100 50 83 - …on results pages 83 100 0 82 75 83 -

…on ‘affiliates ‘page 27 62 100 9 0 17 -

Ratings for providers/products were common in some markets and rare in others. They were most

often found in Package holiday and Broadband price comparison websites, rarely appeared on those

dealing with Insurance and were never present on Rail fare comparison sites. Where they appeared

there was often an explanation of how they had been calculated.

Table 7O: Ratings for providers/products


Mobile phone

Holi-days

Rail fares


(16) %

(18) %

(30) %

(15) %

(13) %

(7) %

Site includes provider/product ratings 33 56 50 7 33 62 0 Base: All websites with provider/product ratings

(33) %

(9) %

(9) %

(2) %

(5) %

(8) %

(0) %

Provides an explanation of how ratings are calculated

70 78 78 100 60 50 -

Provider/product ratings include customer reviews/scores

64 67 56 100 60 62 -

Of the 99 price comparison websites included in the research, only one had a policy for dealing with

rogue or fraudulent traders, and only two allowed customers to report suspicious activity directly to

the site.


Table 7P: Rogue or fraudulent providers


Mobile phone

Holi-days

Rail fares


(16) %

(18) %

(30) %

(15) %

(13) %

(7) %

Site includes policy for dealing with rogue or fraudulent traders

1 0 0 0 0 8 0

Customers can report suspicions of fraudulent activity direct to website

2 0 0 3 0 8 0

Site highlights how it will respond to reports of fraudulent activity

1 0 0 0 0 8 0

A customer complaints policy was available on at least some of the sites in each market apart from

Rail fares. Its incidence ranged from 27 per cent of Mobile phone sites to 62 per cent of Package

holiday sites. Where there was a complaints policy this often meant consumers could issue a

complaint online, although this did not hold true in relation to Package holidays, where only 37 per

cent of the sites with a complaints policy allowed online complaint submission.

Many of the Energy sites with a complaints policy specified the number of days they would take to

respond. At the other end of the scale was Rail fares, where this information was more often missing

than present.

Table 7Q: Customer complaints


Mobile phone

Holi-days

Rail fares


(16) %

(18) %

(30) %

(15) %

(13) %

(7) %

Complaints policy/process available on site

45 37 50 60 27 62 0

Base: All websites with a complaint policy/process

(45) %

(6) %

(9) %

(18) %

(4) %

(8) %

(0) %

Customers can issue a complaint online 80 100 100 83 75 37 -

Specifies number of days it will take to respond to complaint

56 50 78 56 50 37 -


7.5 All markets: Chapter summary

For many of the features looked at there was wide variability across the markets,

with some performing well and others performing poorly. The variability was not

consistent; markets that were good on one parameter were not necessarily a top

performer on another

Energy

When the search results were ordered on price, they were accurate 100 per cent of

the time. Energy sites were rarely found confusing or misleading, and in most cases

shoppers would have been happy to proceed with a switch

Very few applications uncovered additional costs

A relatively high proportion of Energy sites identified the number of providers

compared, and Energy was far more likely than any other market to provide an

explanation of how the results of a comparison are calculated

Most Energy sites gave details of how they generated income, and were good at

indicating when their prices were last updated

Where an Energy site had a complaints policy, it often included the number of days

before a response could be expected

However, few of the Energy shoppers were told how the list of search results was

ordered, and being able to reorder or filter the list was less common than for other

markets

Energy shoppers were fairly likely to find product restrictions or limitations that had

not been mentioned when entering the search criteria

Only half of the supplier costs matched the price comparison website cost

Insurance

Almost all of the search results could be ordered on cost, and 99 per cent of them

were correctly ranked

Shoppers were highly unlikely to find any additional restrictions or limitations in the

quote list

Most of the products could be found on the appropriate supplier site

Insurance price comparison sites were good at identifying the number of providers

compared and a high proportion told consumers how often prices were updated

Many were open about their means of income-generation and having a complaints

policy was common

However, fewer shoppers said the search results were based on all relevant criteria

than was the case in any other market. Less than half of the results found on the

supplier website were quoted at the same price

Very few Insurance price comparison sites gave details about how the results of a

comparison were calculated


Broadband

Broadband price comparison sites were more likely than those in any other market

to explain how the initial list of quotes had been ordered, and often gave shoppers

the opportunity to reorder it

A high proportion of shoppers had sufficient information to feel comfortable

proceeding with a purchase, if they had been a real consumer

A telephone number for phone applications was often provided

When a product was found on the appropriate supplier site, more often than not it

was the same cost. If there was a difference in price, the reason was more visible

than in other markets

Many gave details of their income sources

However, Broadband was the market most likely to have additional restrictions or

limitations in the list of quotes, and additional details found on the supplier site

Broadband sites saying how many providers they compared were in a minority, as

were those with a complaints policy

Mobile phones

This market was one of the best for giving shoppers the option to filter their search

results

It had the highest level of costs matching those shown on the provider website

However, more of the suppliers gave additional product details compared to some

of the other markets

It had the lowest level of lists that were either initially in price order or could be

sorted on price

Providing a telephone number for phone applications was extremely rare, and a

minority of sites had a customer complaints policy

Mobile phone price comparison sites were least likely to give a positive overall

impression

Package holidays

Package holiday search results could often be reordered, and an option to filter the

list was available to more shoppers than any other market

A higher proportion of sites had a complaints policy compared to the other markets

However, finding something confusing or misleading was more common, and

Package holiday sites had fewest shoppers prepared to make a purchase

None of the sites gave details of their comparison process and few provided

information on price updates

It was highly unlikely that a Package holiday site explained how they generated

income


Rail fares

The price comparison websites operating in this market were those most likely to

give a positive impression, and a majority provided shoppers with sufficient detail

for them to feel comfortable making a purchase

In almost all instances the chosen ticket could be found on the train provider

website. Where the cost was different the reason was clear for a greater proportion

than was the case in other markets

However, the Rail fares market had fewest supplier costs matching those obtained

from the price comparison site

There was less operational transparency amongst Rail fares price comparison sites

compared to other markets. None explained why some providers or products might

not be included, none said how the site generated income and none had a

complaints policy

Very few of the quotes had information about when they were last updated


8. All markets: Accreditation

8.1 All markets: Accreditation and Compliance with Consumer Protection from Unfair Trading Regulations

a. All markets: Accreditation overview

Of the 99 websites included in the research, 72 were not accredited and 27 were accredited by a

range of organisations:

Consumer Focus accreditation (energy)

Ofcom accreditation (mobile phone and broadband)

National Rail accreditation (train fares)

British Insurance Brokers’ Association accreditation (insurance)

International Air Transport Association (IATA) (package holidays)

Shopsafe accreditation (multi sector)

SafeBuy accreditation (multi sector)

Mystery shoppers undertook 217 visits to accredited sites and 379 to non-accredited sites.

b. All markets: Initial list of quotes

A shopper on a non-accredited site was equally likely as one on an accredited site to be shown a list

of results that took into account all their search criteria. On non-accredited sites it was slightly less

likely that results were in price order or could be reordered on the basis of price. When checked for

accuracy, there was no difference between accredited and non-accredited sites in the proportion

that were correctly ranked in price order.

Non-accredited sites were slightly more likely to explain how the list had been ordered, and being

able to reorder it was more common compared to accredited sites. However the reverse was true of

filtering the list of results; this had a greater prevalence amongst the accredited sites.



Total Accredited Non-accredited Base: All mystery shops (596)

% (217)

% (379)

%

List was based on all criteria input 95 96 95 Base: All mystery shops exc. rail fares (540)

% (177)

% (363)

%

Was an explanation about list order 41 39 43 Possible to reorder initial list 57 54 59

Possible to filter initial list 60 64 58 Initial list in price order/can sort into price order 88 91 86 Base: All mystery shops exc. rail fares where can get list into price order

(475) %

(160) %

(314) %

List was actually in price order 97 98 97

There was no real difference between the two types of site in the number of pages of quotes shown, but having 10 or more results on the first page was more likely amongst non-accredited sites. Table 8B: Number of quotes

Total Accredited Non-accredited Base: All mystery shops exc. rail fares (540)

% (177)

% (363)

%

1-9 quotes on first page 17 19 15 10+ quotes on first page 83 80 85

More than one page of quotes 47 45 47

For the Insurance, Broadband and Mobile phone markets shoppers noted whether a telephone number was given to allow purchase by phone. Accredited sites performed better on this measure, although it was a low proportion. Table 8C: Telephone number for phone purchase

Total Accredited Non-accredited Base: All mystery shops exc. energy, package holidays and rail fares

(347) %

(103) %

(244) %


30 35 27

b. All markets: Additional restrictions/limitations

Not finding additional restrictions or limitations at any point in the selection and purchase of a

product was more often the case for non-accredited sites. However, the absence of additional costs

in the first five quotes was more common amongst accredited sites.


Table 8D: Additional restrictions/limitations

Total Accredited Non-accredited Base: All mystery shops exc. rail fares and package holidays

(463) %

(164) %

(298) %


62 56 64


(217) %

(379) %


85 81 86


(349) %

(153) %

(196) %

No additional costs in any of first 5 quotes 52 56 48 No additional costs appear when making

application 77 78 76

c. All markets: Impressions of website Finding nothing misleading or confusing was equally likely on both accredited and non-accredited

sites. However, those that were accredited gave a more positive overall impression and had a higher

proportion of shoppers feeling comfortable making a purchase.

Table 8E: Impressions of website Total Accredited Non-accredited Base: All mystery shops (596)

% (217)

% (379)

%

Nothing found confusing 85 84 85 Nothing found misleading 88 87 89

Overall impression is positive 76 79 74 Sufficient information to make purchase 87 90 85


8.2 All markets: Accreditation and Accuracy


Shoppers in all markets other than Package holidays selected a product at random from the first five

in the search results and then tried to find it on the relevant supplier website. Success in doing so

was high, regardless of whether the site was accredited or non-accredited.

Table 8F: Product found on supplier website Total Accredited Non-accredited Base: All mystery shops except package holidays (519)

% (205)

% (314)

%

Found product on supplier website 84 83 85


When a product was found on the supplier site its cost was compared to that on the price

comparison site. Cost was more often the same when the price comparison site was non-accredited,

and more often lower on the supplier site when the price comparison site was accredited.

When there was a cost difference, shoppers were more likely to see the reason if they were using an

accredited site.

Table 8G: How cost on supplier site compared to cost on price comparison site Total Accredited Non-accredited Base: All mystery shops except package holidays where product found on supplier site

(435) %

(169) %

(266) %


…same as price comparison website 59 56 61 …higher than price comparison website 21 21 21 …lower than price comparison website 20 23 18


(179)

%

(75)

%

(104)

%

Yes, can see the reason for the cost difference 22 32 15


Sometimes the products found on the supplier site had additional details; this arose with equal

frequency on both accredited and non-accredited sites.


Table 8H: Supplier site had additional details Total Accredited Non-accredited Base: All mystery shops except package holidays where product found on supplier site

(434) %

(169) %

(265) %

Yes, supplier site has additional details 22 23 22 No, supplier site does not have additional

details 78 77 78

8.3 All markets: Accreditation and Transparency


Accreditation made a big difference when it came to identifying the number of providers compared

and explaining how comparison results were calculated; substantially more of the accredited price

comparison sites provided this information. However, the non-accredited sites were better when it

came to saying why some providers/products might not be included.

Table 8I: Information provided (1) Total Accredited Non-accredited Base: All websites (99)

% (27)

% (72)

%

Identifies number of providers compared 51 67 44 Explains why some providers/products might

not be included 11 7 13


22 41 15

It might be expected that an accredited site would contain the Code of Conduct relating to the

accreditation. However, only 11 per cent included one, perhaps feeling that it is sufficient simply to

state that they are accredited.

There can, of course, be cases where the accreditation is not directly relevant to operating a price

comparison site (e.g. the IATA accreditation on package holiday sites relates to the airlines providing

the flights, not how the site itself provides holiday comparisons). In these situations there is perhaps

less need to have the Code of Conduct on the site.

A small proportion of the non-accredited sites contained a Code of Conduct. These five sites covered

a range of sectors (energy, insurance, mobile phones, broadband and package holidays). Despite not

being accredited by an independent organisation, these sites appear keen to let consumers know

the standards of operation that can be expected.

Accredited sites were slightly more likely to provide a glossary of terms. Accreditation made no

difference to whether the site provided or signposted additional information.


Table 8J: Information provided (2) Total Accredited Non-accredited Base: All websites (99)

% (27)

% (72)

%

Code of Conduct on site 8 11 7 Site includes a glossary of terms used 26 30 25


90 92 91

Accredited sites were more transparent about their income-generation, and were never paid per

‘click through’ to a provider’s site or had providers paying to appear. Accredited sites were more

likely to generate revenue upon completion of a successful application/purchase/switch or via

advertising sales.

Table 8K: Generating income Total Accredited Non-accredited Base: All websites (99)

% (27)

% (72)

%

Explains how the site generates income 56 67 51 Base: All websites explaining how the site generates income

(55) %

(18) %

(37) %


82 100 73

From advertising sales 16 17 16 Providers pay to appear on the comparison site 9 0 14

Paid per 'click through' to a providers site 9 0 14 Other means 11 0 16

Accredited sites out-performed the non-accredited ones in relation to providing information about

price updates. More of the accredited site shoppers saw a note beside the quotes to say when it last

took place. Additionally, a greater proportion of accredited sites told consumers how often prices

were updated.

Table 8L: Price updating Total Accredited Non-accredited Base: All mystery shops (596)

% (217)

% (379)

%


17 26 12


(27) %

(72) %

Website says how often it updates its prices 30 37 28

Although accredited sites were more often generating income through advertising sales, seeing

advertising specifically related to a provider was more common amongst non-accredited sites.


Table 8M: Advertising of provider sites

Total Accredited Non-accredited Base: All websites (99)

% (27)

% (72)

%

Site includes advertising for provider sites 30 18 35 Base: All websites with provider advertising (30)

% (5) %

(25) %


requirements are entered) 97 80 100

…on homepage 87 100 84 …on results pages 83 80 84

…on ‘affiliates’ page 27 20 28

Showing ratings for providers/products was more often a feature of non-accredited sites and non-

accredited sites more often included customer reviews/scores in these ratings. However, when it

came to explaining how the provider/product ratings were calculated, accredited sites were better

performers.

Table 8N: Ratings for providers/products Total Accredited Non-accredited Base: All websites (99)

% (27)

% (72)

%

Site includes ratings for providers/products 33 26 36 Base: All websites with provider/product ratings (33)

% (7) %

(26) %


70 86 65


64 57 65

Only two sites had information relating to rogue or fraudulent traders, and both were non-

accredited.

Table 8O: Rogue or fraudulent providers Total Accredited Non-accredited Base: All websites (99)

% (27) %

(72) %


1 0 1


2 0 3


1 0 1


Being accredited did not make it more likely that there was a complaints policy on the site, as might

be expected; more non-accredited sites had a complaints policy compared to the accredited sites.

However, accredited sites with a complaints policy more frequently allowed customers to issue a

complaint online. There was no difference between accredited and non-accredited sites when it

came to specifying the number of days taken to respond.

Table 8P: Customer complaints Total Accredited Non-accredited Base: All websites (99)

% (27) %

(56) %

Complaints policy/process available on site 45 32 61 Base: All websites with a complaint policy/process (45)

% (9) %

(34) %

Customers can issue a complaint online 80 89 79 Specifies number of days it will take to respond

to complaint 56 56 56


8.4 All markets: Accreditation chapter summary

There are 35 key variables on which accredited price comparison websites have

been compared against non-accredited sites. For 11 of these there was no

difference in performance; the sites were equally good (or poor) regardless of

whether they had an accreditation.

The equal performance areas were:

o Search results based on all criteria entered

o When checked, search results actually in price order

o More than one page of quotes

o No additional costs appear on application

o Nothing on website found confusing

o Nothing on website found misleading

o Site includes policy for dealing with rogue or fraudulent traders

o Complaints policy specifies number of days to respond

o Provides or signposts additional information

o *Able to find product on supplier site

o Supplier site does not have additional product details

(* As noted elsewhere in the report, not being able to find the product, or not finding it at

the same price, could be due to supplier site differences or inadequacies)

The remaining 24 variables were more likely to show accredited sites as better

performers (16 accredited better versus 8 non-accredited better)

Additionally, the magnitude of difference when the accredited price comparison

websites outperformed the non-accredited sites was often greater; on seven

occasions the accredited sites were 10-26 percentage points ahead of the non-

accredited sites. When a non-accredited site outperformed an accredited one, the

percentage points difference was almost always under 10

The areas where accredited sites performed better were:

o Search results initially in price order or can sort into price order

o Possible to filter search results

o Telephone number available for phone purchase (Insurance, Mobile phones

and Broadband markets)

o No additional costs appear in first five quotes

o Positive overall impression

o Sufficient information provided to feel happy making a purchase

o Can see why the supplier site cost is different

o Identifies the number of providers compared

o Site has explanation of how provider/product ratings are calculated

o Code of Conduct on site

o Provides an explanation of how the results of a comparison are calculated

o Explains how the site generates income


o Note beside quotes to say when prices last updated

o Site identifies how often prices are updated

o Customers can issue a complaint online

o Site includes glossary of terms used

The areas where non-accredited sites performed better were:

o Provides an explanation about order of search results

o Possible to reorder search results

o 10 or more quotes on first page

o No additional restrictions or limitations in any of the first five quotes

o No additional restrictions or limitations appeared when making application

o *Supplier site cost is same as price comparison site cost

o Explains why some providers/products might not be included

o Complaints policy available on site


9. All markets: Big 4

9.1 All markets: Big 4 and Compliance with Consumer Protection from Unfair Trading Regulations

a. All markets: Big 4 overview

At least one of the Big 4 websites operates in all of the six markets included in the study. All Big 4

sites cover insurance, broadband and energy. Three give price comparisons for mobile phones, three

for package holidays and one for rail fares.

Where a Big 4 site covered two or more markets, details about the site were collected separately for

each market, from the appropriate part of the site. Of the 99 websites about which details were

collected, 23 were Big 4 (or a specific part of a Big 4 site) and 76 were non-Big 4.

Mystery shoppers made 137 visits to Big 4 site sections and 459 visits to non-Big 4 sites.

This chapter looks at whether there were any differences between the Big 4 price comparison sites

and other price comparison sites.

b. All markets: Initial list of quotes

It was equally common for Big 4 and non-Big 4 sites to provide search results that were based on all

the criteria entered by the shopper, and for shoppers to be able to filter the list. However, the non-

Big 4 sites were more likely to provide an explanation about the order of the search results, and also

to allow reordering.

Having search results that were presented in price order or that could be reordered on price was

more likely for shoppers on Big 4 sites. When checked for accuracy, both Big 4 and non-Big 4 had the

same proportion of lists that were indeed in price order.


Total Big 4 Non-Big 4 Base: All mystery shops (596)

% (137)

% (459)

%

List was based on all criteria input 95 97 95 Base: All mystery shops exc. rail fares (540)

% (129)

% (411)

%

Was an explanation about list order 41 37 43 Possible to reorder initial list 57 51 59

Possible to filter initial list 60 60 60 List initially in price order or are able to sort into

price order 88 93 87

Base: All mystery shops exc. rail fares where can get list into price order

(475) %

(120) %

(355) %

List was actually in price order 97 97 97


The Big 4 had a slightly higher incidence of providing more than one page of quotes. In addition,

shoppers on Big 4 sites were more likely to see 10 or more quotes on the first page compared to

shoppers on non-Big 4 sites.

Table 9B: Number of quotes

Total Big 4 Non-Big 4 Base: All mystery shops exc. rail fares (540)

% (129)

% (411)

%

1-9 quotes on first page 17 9 19 10+ quotes on first page 83 91 81

More than one page of quotes 47 50 46

For the Insurance, Mobile phones and Broadband markets, a telephone number was provided for phone applications on more occasions in Big 4 sites compared to non-Big 4. Table 9C: Telephone number for phone purchase

Total Big 4 Non-Big 4 Base: All mystery shops exc. energy, package holidays and rail fares

(347) %

(85) %

(262) %


30 35 28

c. All markets: Additional restrictions/limitations

Not encountering additional restrictions or limitations (either in the first five quotes in the search

results or when a specific quote is selected and an application started) was more common on Big 4

sites. However, it was the non-Big 4 sites that more commonly did not have any additional costs in

the first five quotes. At the point of making an application, additional costs were equally unlikely to

appear on both Big 4 and non-Big 4 sites.

Table 9D: Additional restrictions/limitations

Total Big 4 Non-Big 4 Base: All mystery shops exc. rail fares and package holidays

(463) %

(112) %

(351) %


62 71 59


(137) %

(458) %


85 92 82


(349) %

(70) %

(279) %

No additional costs in any of first 5 quotes 52 41 54 No additional costs appear when making

application 77 77 77


d. All markets: Impressions of website

Both Big 4 and non-Big 4 sites were equally unlikely to be found misleading by shoppers. Slightly

more Big 4 shoppers did not find anything confusing compared to those on the non-Big 4 sites.

Big 4 sites also out-performed the non-Big 4 in relation to giving a positive impression and shoppers

having enough information to feel comfortable making a purchase.

Table 9E: Impressions of website Total Big 4 Non-Big 4 Base: All mystery shops (596)

% (137)

% (459)

%

Nothing found confusing 85 87 84 Nothing found misleading 88 88 88

Overall impression is positive 76 83 73 Sufficient information to make purchase 87 89 86

9.2 All markets: Big 4 and Accuracy


Shoppers in all markets other than Package holidays selected a product at random from the first five

in the search results and then tried to find it on the relevant supplier website. Most products could

be found, regardless of whether the price comparison website was in the Big 4 or not.

Table 9F: Product found on supplier website Total Big 4 Non-Big 4 Base: All mystery shops except package holidays (519)

% (119)

% (392)

%

Found product on supplier website 85 85 85


The cost of the product on the supplier site was the same as the price comparison site in many

cases. An identical cost was more common when the product came initially from a non-Big 4 site.

The supplier cost being higher was more likely when the product had been selected from a Big 4 site.

When the cost of the product was different, shoppers were asked whether they could ascertain why

this was. A minority could see the reason for the price difference; this was particularly true for the

non-Big 4 sites.


Table 9G: How cost on supplier site compared to cost on price comparison site Total Big 4 Non-Big 4 Base: All mystery shops except package holidays where product found on supplier site

(435) %

(101) %

(334) %


…same as price comparison website 59 54 60 …higher than price comparison website 21 29 19 …lower than price comparison website 20 17 21


(178)

%

(46)

%

(133)

%

Yes, can see the reason for the cost difference 22 26 21


Additional details in the supplier site were equally unlikely to be found regardless of whether the

shopper had started off in a Big 4 or a non-Big 4 price comparison website.

Table 9H: Supplier site had additional details Total Big 4 Non-Big 4 Base: All mystery shops except package holidays where product found on supplier site

(435) %

(101) %

(333) %

Yes, supplier site has additional details 22 22 22 No, supplier site does not have additional

details 78 78 78

9.3 All markets: Big 4 and Transparency


The Big 4 websites were better at identifying how many providers they compared and why

some providers/products might not be included. The reverse was true when it came to

explaining how the results of a comparison were calculated; on this occasion the non-Big 4

were the better performers.

Table 9I: Information provided (1) Total Big 4 Non-Big 4 Base: All websites (99)

% (23)

% (76)

%

Identifies number of providers compared 51 57 49 Explains why some providers/products might

not be included 11 17 9


22 9 26

The Big 4 were substantially more likely to have a Code of Conduct. They also out-performed the

non-Big 4 in terms of having a glossary and providing or signposting additional information.


Table 9J: Information provided (2) Total Big 4 Non-Big 4 Base: All websites (99)

% (23)

% (76)

%

Code of Conduct on site 8 22 4 Site includes a glossary of terms used 26 43 21


90 98 89

The Big 4 were more transparent than the non-Big 4 about how they generate income. For both

types of site this was mainly via payment on completion of each successful

application/purchase/switch. Big 4 sites never generated income from advertising sales, providers

paying to appear on the site or per ‘click through’ to a provider site.

Table 9K: Generating income Total Big 4 Non-Big 4 Base: All websites (99)

% (23)

% (76)

%

Explains how the site generates income 56 70 51 Base: All websites explaining how the site generates income

(55) %

(16) %

(39) %


82 75 85

From advertising sales 16 0 23 Providers pay to appear on the comparison site 9 0 13

Paid per 'click through' to a providers site 9 0 13 Other means 11 25 5

The non-Big 4 were better at informing consumers about price updates. More of the non-Big 4

shoppers saw a note beside the quotes to say when prices were last updated, and a greater

proportion of the non-Big 4 websites carried information about how often this took place.

Table 9L: Price updating Total Big 4 Non-Big 4 Base: All mystery shops (596)

% (137)

% (459)

%


17 14 18


(23) %

(76) %

Website says how often it updates its prices 30 22 33

Although the Big 4 did not get income from advertising sales, that did not mean they were

completely devoid of provider site adverts. However, their incidence was lower than amongst the

non-Big 4 sites.


Table 9M: Advertising of provider sites

Total Big 4 Non-Big 4 Base: All websites (99)

% (23)

% (76)

%

Site includes advertising for provider sites 30 13 36 Base: All websites with provider advertising (30)

% (3) %

(27) %


requirements are entered) 97 100 96

…on homepage 87 33 93 …on results pages 83 67 85

…on ‘affiliates’ page 27 0 30

Provider/product ratings were present on more Big 4 sites than non-Big 4, and the Big 4 were more likely to provide an explanation of how they were calculated. Compared to non-Big 4 provider/product ratings, those on Big 4 sites were substantially more likely to include customer reviews/scores. Table 9N: Ratings for providers/products Total Big 4 Non-Big 4 Base: All websites (99)

% (23)

% (76)

%

Site includes ratings for providers/products 33 43 30 Base: All websites with provider/product ratings (33)

% (10)

% (23)

%


70 90 61


64 90 52

Only two price comparison websites had information relating to rogue or fraudulent traders, and both of these were non-Big 4 sites.


Table 9O: Rogue or fraudulent providers Total Big 4 Non-Big 4 Base: All websites (99)

% (23) %

(76) %


1 0 1


2 0 3


1 0 1

Big 4 sites were three times more likely to have a complaints policy compared to non-Big 4 sites, plus more often allowed customers to issue a complaint online. Specifying the number of days it would take to respond was also more prevalent amongst the Big 4 sites. Table 9P: Customer complaints Total Big 4 Non-Big 4 Base: All websites (99)

% (23) %

(76) %

Complaints policy/process available on site 45 96 30 Base: All websites with a complaint policy/process (45)

% (22) %

(23) %

Customers can issue a complaint online 80 91 70 Specifies number of days it will take to respond

to complaint 56 68 43


9.4 All markets: Big 4 chapter summary

There are 35 key variables on which Big 4 price comparison websites have been

compared against non-Big 4 sites. For 8 of these there was no difference in

performance; the sites were equally good (or poor) regardless of whether they were

Big 4 or not

The equal performance areas were:

o Search results based on all criteria entered

o Possible to filter search results

o When checked, search results actually in price order

o No additional costs appear on application

o Nothing on website found misleading

o *Able to find product on supplier site

o Supplier site does not have additional product details

o Site includes policy for dealing with rogue or fraudulent traders

Out of the remaining 27 variables Big 4 sites were far more often the better

performers (20 Big 4 better versus seven non-Big 4 better)

The areas where Big 4 sites performed better were:

o Search results initially in price order or can sort into price order

o More than one page of quotes

o 10 or more quotes on first page

o Telephone number available for phone purchase (Insurance, Mobile phones

and Broadband markets)

o No additional restrictions or limitations in any of the first five quotes

o No additional restrictions or limitations appeared when making application

o Nothing on website found confusing

o Positive overall impression

o Sufficient information provided to feel happy making a purchase

o Can see why the supplier site cost is different

o Identifies the number of providers compared

o Explains why some providers/products might not be included

o Code of Conduct on site

o Site includes glossary of terms used

o Provides or signposts additional information

o Explains how the site generates income

o Site has explanation of how ratings are calculated

o Complaints policy available on site

o Customers can issue a complaint online

o Complaints policy specifies number of days to respond




The areas where non-Big 4 sites performed better were:

o Provides an explanation about order of search results

o Possible to reorder search results

o No additional costs appear in first five quotes

o *Supplier site cost is same as price comparison site cost

o Provides an explanation of how the results of a comparison are calculated

o Note beside quotes to say when prices last updated

o Site identifies how often prices are updated




10. Energy

10.1 Energy: Number of price comparison websites and mystery shops

A total of 18 price comparison websites were included in the energy market part of the research.

Mystery shoppers made 116 site visits to pose scenarios on these sites.

10.2 Energy: Compliance with Consumer Protection from Unfair Trading Regulations

a. Energy: Obtaining a list of quotes

In order to obtain a list of quotes, it was always necessary for mystery shoppers to enter a range of

details, most commonly current supplier, current means of payment and whether looking for a dual

or single fuel quote. However, not all were asked for an address or postcode or allowed to enter

details of their current usage. Both of these are essential to obtain a personalised and accurate

quote. Without them, any costs quoted can only be indicative.

In two-thirds of cases the mystery shopper was asked if they were an Economy 7 customer, and the

same proportion could specify their preferred future means of payment. Some were offered scope

to select tariff type (eg fixed, variable or online) or given the option to include ‘green’ tariffs in their

search.

For a minority the search included a choice of seeing only tariffs available through the price

comparison website or viewing a full list of options, with those that would require an application

directly to the supplier highlighted. This is a useful option, alerting the consumer to the fact that the

price comparison website does not offer switching for a full list of suppliers.

Table 10A: Details can enter when starting search Total Base: All mystery shops (116)

%

Can input details on… …current supplier 97

…current means of payment 95 ..whether want dual or single fuel 93

…current tariff 90 …energy usage (kWh or £) 89

…address or postcode 86 …Economy 7 customer 67

…future means of payment 67 …tariff type (eg fixed, variable, online) 41

…whether to include ‘green’ tariffs 37 …only products available on the site or all products 21

Once details had been entered, mystery shoppers were satisfied that the list of quotes shown took

into account all their search criteria for all except one of the price comparison website visits. The

aberration involved a scenario specifying a ‘green’ tariff, none of which were available in that area.


In most instances there was no explanation of how the list of quotes had been ordered. However,

where this was the case it was generally obvious how it had been done. Overall, most mystery

shoppers were either told or could easily see how the list had been ordered. Being in ascending cost

order was the most common situation. However, a few lists were ordered on cost saving as opposed

to actual cost.

Table 10B: Whether there was an explanation of how initial list had been ordered Total Base: All mystery shops (116)

%

Yes (total) 30 Yes, explanation that was ordered on price 26

Yes, explanation that was ordered in some other way 4 No (total) 70

No, but could see it was ordered on price 53 No, but could see it was ordered some other way 8

No, and was not obvious how ordered 9

b. Energy: Ability to reorder and filter initial list of quotes

Once faced with a list of possibilities, it would sometimes be useful for consumers to be able to

reorder it on a variable other than that used initially (eg to reorder on tariff type or supplier).

Similarly, having the ability to shorten the list by filtering out some of the tariffs could be a useful

tool.

Despite their potential benefit, these options were only available in a minority of cases. When

available, the most prevalent reordering factors were price and supplier. These were also present as

filtering options. A wider range was available for filtering, including tariff type, dual or single fuel and

payment type.

Mystery shoppers were instructed to sort the list of quotes into price order. A minority of lists were

either not in price order initially or could not be reordered on price (6 per cent). Amongst the

remaining 94 per cent, the first 2-3 pages of options were checked and in all cases were found to be

sorted correctly (ie they were, indeed, in order of ascending price).


Table 10C: Reordering and filtering options Total Base: All mystery shops (116)

%

Able to reorder 33 Able to filter 42

Base: All mystery shops where can reorder list (38 ) %

Can reorder on… … price 68 …supplier 47

…customer ratings 8 …supplier’s service rating 7

Base: All mystery shops where can filter list (49) %

Can filter on… …tariff type 69 …dual or single fuel 47

…payment type 41 …price 27

…supplier 16 …supplier’s service rating 14 …annual or monthly cost 14

…only products available via site or all products 10

c. Energy: Ease of viewing the list of quotes

Most mystery shoppers saw 10 or more quotes on the first page. When they only saw 1-9 this was

often because there were no more to show. In fact, for 60 per cent of all mystery shops there were

no quotes beyond the first page. Amongst the remainder, there was a mix of being able to view all

the quotes at once and moving forward page by page, with over half having both options available.

Table 10D: Number of tariffs per page and options for viewing Total Base: All mystery shops (116)

%

Number of tariffs per page 1-9 16 10+ 84

Is there more than one page of quotes? Yes 40 No 60

Base: All mystery shops with more than one page of quotes (46) %

There is an option to… …view all quotes 72 … move to next set or page of results 87

…both of these 59


d. Energy: Accessing a summary of the current tariff

Being able to access the details consumers enter about their current tariff facilitates comparison and

was easily available to most mystery shoppers, either in the form of a link or by being visible near

the list of quotes. The remainder could access a summary, but not in either of these ways.

The summaries were generally comprehensive. Information on current expenditure was shown in a

majority, and many also gave usage in kWh per annum. Current supplier, tariff and method of

payment were often listed. However, with only 42 per cent showing the discounts that apply to the

current tariff, or noting that no discounts are applicable, there is scope to increase its inclusion.

Although Confidence Code restrictions mean that only certain types of discount can be included, it

would nevertheless be useful for consumers to see in the summary whether or not any discounts

had been applied.

Table 10E: Summaries Total Base: All mystery shops (116)

%

On the first page of quotes there is… …a summary of the current tariff 49

…a link to access a summary of the current tariff 37 …neither of these 14

Summary shows… …current spend 92

…current supplier’s name 83 …current tariff name 70

…current payment method 68 …current usage (kWh) 67

…unit rate 50 …discounts included in current tariff or a note to say no discounts apply 42

e. Energy: Restrictions and limitations not previously mentioned

When a consumer reads the details of the quotes shown it is doubtless frustrating if any are found to

contain restrictions or limitations that could have been avoided by a more tailored initial selection

by the price comparison website. Additionally, if a consumer does not fully read the details, or

assumes that the quotes shown must be appropriate for them, they could end up trying to switch to

an inappropriate product.

The first five quotes in the list of options generated after inputting details were examined, and half

contained some form of additional restriction or limitation. One or more quotes in a third of all the

lists required online account management, and the same proportion necessitated paperless billing.

Some contained quotes for which payment by Direct Debit was mandatory.


Few additional restrictions or limitations were discovered further on in the process, at the point of

making an application to switch; most mystery shoppers found none. Being required to pay via

Direct Debit was a revelation for some, a few found paperless billing was a necessity and one

shopper had selected a quote available only to homeowners.

Table 10F: Quote restriction and limitations Total Base: All mystery shops (116)

%

Did any of first 5 quotes have additional restrictions/limitations? Yes 53 No 47

Additional restrictions/limitations (first 5 quotes in initial list) Online account management 34

Paperless billing 34 Must pay by Direct Debit 22

Need to pass a credit check 6

Did quote chosen to switch to have any additional restrictions/limitations? Yes 14 No 86

Additional restrictions/limitations (at point of making application) Must pay by Direct Debit 7

Paperless billing 3 Only available to homeowners 1

f. Energy: Additional costs not previously mentioned

Another potential source of consumer confusion or frustration is discovering that the amount

quoted is not the full cost, because additional items need to be added. When the first five quotes

were examined for additional costs, only half of the lists did not contain any. The most common

extra cost was a cancellation fee; although this is not an immediate addition, if it is overlooked it

could cause upset at a later stage if the contract is cancelled for any reason.

A small proportion of lists contained a quote(s) which charged extra for paper bills. Other additional

costs did not arise often, but included a quote(s) requiring payment of a fee to the Independent Gas

Transporter (IGT) network, and a quote(s) necessitating an additional amount for the standing

charge. Both of these would unavoidably increase the cost.

At the stage where a quote was selected as the one to switch to, a majority of the mystery shoppers

did not find any additional costs. A small proportion uncovered the fact that there was a cancellation

charge and a few found that there was an extra cost for paper bills.


Table 10G: Additional costs Total Base: All mystery shops (116)

%

Did any of first 5 quotes have additional costs? Yes 48 No 52

Additional costs (first 5 quotes in initial list) Cancellation fee 47

Paper bills 5 Independent Gas Transporter fee 2

Standing charge 2

Did quote chosen to switch to have any additional costs? Yes 5 No 95

Additional costs (at point of making application) Cancellation fee 4

Paper bills 3

g. Energy: Whether anything on the price comparison website was confusing or misleading

Most of the mystery shops did not result in any confusion (93 per cent). Comments from shoppers

who felt there was a lack of clarity in some area included:

‘I could not seem to apply for over half of the results returned to me by my search using this website. It merely stated that I needed to call them in order to switch my supply to them.’

‘Results that appeared to be the cheapest, but the website could not offer the

application.’ NB This was due to the supplier ‘not supporting the service’ of the price comparison website for that tariff.

‘I found it a bit confusing that tariffs that cost more than the existing tariff are

given in red as a minus figure. It would be more logical to display them as a plus figure.’

‘The button to make an application was called 'More details', which I found

confusing, as I assumed it meant they would provide me with more details, rather than I would provide them’


Similarly, most mystery shoppers did not feel anything on the price comparison website was

misleading (95 per cent). Comments from those who thought the opposite included:

‘I couldn't understand why one of the results stated that I would be saving £0.01 per year.’

‘I said I paid on receipt of my bill, yet when the quotes were first shown they

assumed I was going to pay by Direct Debit, and I had to change it back to paying on receipt of bill to get a true price.’

‘Cancellation charges not upfront on the initial search results.’

h. Energy: Overall impressions

The majority of mystery shoppers gained a positive impression of the price comparison website. The

few with a negative view gave reasons ranging from the use of a small font size to a rather dull

layout.

If they were a real customer, most would have felt comfortable proceeding with an application,

being satisfied that they had sufficient information about the quote. The minority who would need

more information gave a range of requirements:

‘The cost of kWh, and how long the contract is.’ ‘I don't know much about the company, and I don't know anything about my right

to cancel the contract, or if there is a period that I can't cancel during.’ ‘There did not seem to be prices per unit displayed on the comparison website.’ Table 10H: Overall impressions Total Base: All mystery shops (116)

%

Is your overall impression of this website…. …very positive 39

…quite positive 45 …neither positive nor negative 12

…quite negative 5 …very negative 0

Do you at this stage have sufficient information to make an application? Yes 95 No 5


10.3 Energy: Accuracy

a. Energy: Whether tariff was available on supplier website

In 86 per cent of cases, the same tariff was found on the supplier website. It should not, however, be

assumed that the remaining tariffs could not be found because they are unavailable from the

supplier. The lack of success could potentially be due to shortcomings in the presentation of tariffs

on the supplier site. Examples where this might be the case included:

‘Although it is possible to compare prices for just electricity, it is not possible to compare prices for just gas, as the scenario requests. When comparing dual fuel it is a combined cost and no separate one for gas is shown. This means it is not possible to obtain a quote for just gas.’

‘I Googled E.on and from their home page searched under 'find the right deal for

you', then searched under 'green plans'. I requested more details and quote, and put in my address and postcode, after which I got the message stating “we don't have any products available via our website for your metering”.’

‘I was only given a quote for how much I could pay monthly, an estimate of

£429.49 annually. There was no tariff mentioned, and it was very confusing.’ ‘I clicked the compare prices options and entered all my details and the only result

that was returned was the Standard (variable) tariff.’ ‘I found the Sainsbury's Online Price Freeze June 2013, but was unable to get a

quote for electricity only. It seems on the supplier website this tariff is only available to dual fuel customers, whereas on the comparison site electricity only customers could apply.’

b. Energy: Cost of tariff on supplier website compared to cost on price comparison website

It would be reasonable for a consumer obtaining a quote from a price comparison website to expect to get the same quote if they visited the supplier website. When the tariff could be found on the supplier site it was offered at the same price in half of the comparisons. The other half was a mix of higher and lower prices. The supplier cost was more likely to be different when a postcode could not be entered as an initial search criterion (71 per cent compared to 51 per cent when a postcode could be entered 11). The level of price difference ranged from £1 to £351.

11

Low base size (14)


Table 10I: Tariff costs – price comparison website versus supplier website Total Base: All mystery shops (116)

%

Could find tariff on supplier site? Yes 86 No 14

Base: All mystery shops where tariff could be found on supplier site (100) %

Supplier site cost was… …same as price comparison website 47

…higher than price comparison website 29 …lower than price comparison website 24

Where there was a price difference, for most there was no obvious reason. The explanation for the

rest (which amounts to a total of eight mystery shops) was often due to differences in the way the

price had been calculated:

‘For the supplier I had to input my actually monthly fee, whereas the comparison site took an average for my type of property.’

‘I had to use annual consumption in KWh on the comparison site and cost per month on the supplier site.’

‘The comparison site asks if you know how much energy you use and gives you an option to say "you are not sure". The results are based on this as an annual estimate. The npower website asked for an annual spend, which was difficult to judge, so I estimated approx £600 for each of gas and electricity.’

‘Having given them exactly the same details for my current supplier and usage in £, they worked out it would be different (higher) in units than the comparison site.’

When the supplier cost was lower and a reason could be seen this was sometimes because the

supplier was offering a discount. This could have been an introductory or prompt pay discount,

neither of which, under the Confidence Code, are allowed to be included in the price comparison

website’s calculation.


Table 10J: Reasons for the price difference Total Base: All mystery shops where tariff could be found on supplier site and cost was different (53)

%

Can see the reason for the price difference? Yes 15 No 85

Reasons for price difference Difference in way prices were calculated 9

Supplier was offering a discount 6

c. Energy: Whether the supplier site had additional details

In most cases the supplier site did not have any details that were additional to those in the price

comparison website. Where additional information was found, it centred on paperless billing, a

cancellation fee or being required to manage the account online.

Table 10K: Whether supplier site had additional details Total Base: All mystery shops where tariff could be found on supplier site (100)

%

Supplier site had additional details? Yes 17 No 83

What were the additional details? Paperless billing 4 Cancellation fee 4

Online account management 3

10.4 Energy: Transparency

a. Energy: Information provided

The price comparison websites covering the energy market were sometimes lacking when it came to

providing various types of information. Of the 18 price comparison websites looked at:

none showed a policy for dealing with rogue or fraudulent providers, nor highlighted how they would respond to reports of such activity, nor provided consumers with a direct means of bringing it to their attention

only 11 per cent had details about a Code of Conduct on the website

only 11 per cent gave an explanation about why some suppliers or tariffs might not be included

only 22 per cent provided a glossary of the terms used (eg kWh, Direct Debit, prepayment meter)

only 50 per cent published details of their complaint process or policy, which in most cases was difficult to find

only 50 per cent said how frequently they updated their prices, although for most of these it was easy to find (78 per cent)


Moving into areas where transparency was better, and a majority of the 18 price comparison

websites provided information:

61 per cent explained how they calculated the results of a comparison, which in 64 per cent of cases was easy to find

61 per cent identified how many providers they compared, which was easily found in 72 per cent of cases

amongst the 50 per cent that included provider or product ratings, 78 per cent explained how these were calculated

83 per cent stated how they generated income, and for 60 per cent of these it was easy to find

89 per cent signposted consumers to other sources of information, and for 62 per of these it was easy to find

On most of the first page of quotes reached by the mystery shoppers there was a note saying when

the prices were last updated, albeit not always clearly displayed. Nevertheless, there was still a

sizeable proportion of shoppers who had no idea how up-to-date the quotes they were looking at

were.

Some of the first pages of quotes had a clearly displayed note about VAT being included in the

prices, while for others it was present but easily overlooked. In a few instances the note contained

information about VAT being excluded from the quoted costs. For half of the mystery shops there

was no notification of whether VAT was included or excluded.

Table 10L: Information about price updating and VAT Total Base: All mystery shops (116)

%

Note on first page of quotes about price updating? Yes, and it was clearly displayed 33

Yes, although it was not very clearly displayed 25 No, there was not a note 42

Note on first page of quotes about VAT being included in prices? Yes, and it was clearly displayed 24

Yes, although it was not very clearly displayed 19 No, there was a note about VAT being excluded 6

No, there was not a note 51


b. Energy: Details of the information provided

For all of the nine sites with a complaints process or policy, consumers could submit their complaint

online. Most specified the number of days it would take before a response was received.

When a price comparison website gave details about how often prices were updated, this ranged

from the specific (eg ‘more than daily’ or ‘daily’) to the nebulous (eg ‘regularly’ or ‘when the supplier

tells us’).

For the sites with information about income generation, this was mainly via a supplier payment per

successful switch. Only one price comparison website said they were getting income from

advertising.

The most common additional information provided or signposted was about energy efficiency. The

Consumer Focus website was also often mentioned, while many pointed the consumer in the

direction of other external agencies (eg the Energy Saving Trust).

Table 10M: Operational information Total Base: All websites with a complaints policy (9)

%

Can submit complaint online? Yes 100 No 0

Specify number of days before receive a response? Yes 78 No 22

Base: All websites with information about income generation

(15)

%

How does the site generate income? Comparison site paid on completion of successful switch 93

From advertising sales 7 Base: All websites providing or signposting additional information

(16)

%

Provide or signpost information about… …energy efficiency advice 94

…Consumer Focus website 75 …contact details for external agencies eg Energy Saving Trust 69

...grant schemes eg Warm Front 44


10.5 Energy: Accreditation

a. Energy: Overview

Of the 18 energy price comparison websites, 10 are accredited by Consumer Focus under the

Consumer Focus Confidence Code. One non-accredited website stated that its price comparison

calculator was accredited by Consumer Focus, and another that it had Ofgem accreditation. Only the

10 price comparison websites accredited under the Consumer Focus Confidence Code are taken as

‘accredited’ for the purpose of this research. In fact, all non-accredited sites are running off an

accredited site’s calculator and database – therefore prices should be the same as the accredited

site, but the non-accredited may choose to omit tariffs that a consumer cannot switch to.

When the results for accredited price comparison websites are compared to those for sites without

accreditation, the picture is not clear-cut. On many parameters accredited websites perform better

than non-accredited sites, but for a few features appear to be less consumer-friendly than the non-

accredited sites. On other occasions there are no differences between the two types of site.

b. Energy: Where accredited price comparison websites sites perform better than those that are not accredited

Accredited sites were more likely to require the consumer to input key items of information at the

start of their comparison, namely address or postcode, and current energy usage. They were also

more likely to offer the consumer a choice of seeing only tariffs available via the price comparison

website or all available tariffs, regardless of whether the consumer could switch through the price

comparison website itself. Being able to filter the initial list of quotes was more common amongst

accredited sites.

Table 10N: Details can enter when starting search

Total Accredited Non-accredited


(62) %

(54) %

Can input details on… …postcode 86 90 81

…current usage 90 94 83 ..see all tariffs or restrict to those available via site 21 29 11

Can filter initial list of tariffs 42 47 37

The chance of quote restrictions or limitations being found partway through the comparison process

was slightly lower for accredited price comparison websites (50 per cent accredited versus 57 per

cent not accredited).

In terms of providing information about how the site operates, accredited price comparison

websites outperformed those that have no accreditation. They were better at explaining how they

calculate the results, more often provided or signposted additional information, more often gave

price updating details, and were more likely to have a complaint process. More of the accredited

sites informed consumers about their income generation methods, and showed how many providers

they include, compared to the non-accredited sites.


Table 10O: Provision of information



(10) %

(8) %

Provide or signpost additional information 89 100 83 Provide details about how often prices are updated 50 60 33

Have a complaint process 50 60 37 Provide income generation details 83 100 62

State how many providers are compared 61 80 37 Explain how results are calculated 61 70 50

In comparing quote details to those provided on the supplier site, there were more frequently

additional details when the quote had been sourced from an accredited price comparison website

compared to a non-accredited one (21 per cent and 15 per cent respectively).

Additionally, accredited price comparison websites more often provided information about VAT

inclusion in their prices compared to non-accredited sites (49 per cent of website visits to accredited

sites versus 37 per cent of the non-accredited website visits).

c. Energy: Where accredited price comparison websites do not perform as well as those that are not accredited

On a few parameters the price comparison websites accredited under the Consumer Focus

Confidence Code did not perform as well as the non-accredited websites. Being able to reorder the

initial list of quotes was less common, as was providing an explanation of how the original list had

been sorted.

It was more likely that the supplier website cost was different to the price comparison website cost

when the quote came from an accredited website, and concomitantly more likely that the costs

were the same when the quote originated from a non-accredited site. This may well be due to

accredited websites being more stringent in applying the Confidence Code requirement to exclude

certain types of discount.

Table 10P: Areas where accredited sites do not perform as well as non-accredited sites



(62) %

(54) %

Can reorder initial list of tariffs 33 19 48 Provide explanation of how initial list is sorted 30 24 37

Base: All mystery shops where tariff was found on supplier site

(100)

%

(53) %

(47) %

Supplier site cost different to price comparison site 53 58 47


d. Energy: Where accredited and non-accredited price comparison websites perform equally

There was little difference between accredited and non-accredited price comparison websites on a

range of factors, including the number of quotes shown on the first page, whether additional costs

arose partway through the application process, and being able to find the tariff on the supplier

website.

Accredited and non-accredited price comparison websites were similar in terms of giving a positive

impression and not being confusing or misleading in any respect.

A Code of Conduct was equally unlikely to be found on an accredited site compared to a non-

accredited one, and both types of website were equally poor at explaining why some suppliers or

tariffs might not be included.

Table 10Q: Areas where accredited and non-accredited sites perform equally



(62) %

(54) %

Show 1-9 tariffs on first page 16 18 13 Show 10+ tariffs on first page 84 82 87

No additional costs appear partway through application 52 53 50 Able to find tariff on supplier site 86 85 87

Gives a positive impression 84 84 83 Not found confusing 93 92 94

Not found misleading 95 92 98 Base: All websites

(18) %

(10) %

(8) %

Code of Conduct present 11 10 12 Explains why some suppliers/tariffs not included 11 1012 12

10.6 Energy: Big 4 price comparison websites versus others

a. Energy: Overview

All of the Big 4 price comparison websites provide energy-related comparisons. Two have the

Consumer Focus Confidence Code accreditation for the energy part of the website.

Comparing the Big 4 against other price comparison websites in the energy market shows them to

be on a par with each other for some aspects of their operation. For other aspects the situation

varies – sometimes the Big 4 are better and sometimes the reverse is true.

b. Energy: Where Big 4 price comparison websites sites perform better than non-Big 4

The Big 4 price comparison websites perform better than non-Big 4 in the following areas:

offering an option to include all tariffs not just those available via the price comparison website

12

Price comparison websites accredited under the Consumer Focus Confidence Code must include all suppliers


giving a positive impression to the website visitor

providing information about VAT being included in prices

having a Code of Conduct

having a complaint policy

explaining why some suppliers / tariffs might not be included

explaining how the website generates income

providing or signposting additional information

Table 10R: Areas where Big 4 sites perform better


% (27) %

(89) %

Option to see all tariffs or only those available via site 21 41 15 Gives a positive impression 84 89 81

Providing information about VAT being included 43 63 37 Base: All websites

(18) %

(4) %

(14) %

Code of conduct present 11 25 7 Have a complaint process 50 100 36

Explains why some suppliers/tariffs not included 11 25 7 Provide income generation details 83 100 79

Provide or signpost additional information 89 100 86

c. Energy: Where Big 4 price comparison websites do not perform as well as non-Big 4

The Big 4 price comparison websites do not perform as well as non-Big 4 in the following areas:

explaining how the initial list of quotes was ordered

being able to reorder the initial list of quotes

no additional costs being added partway through the application process

finding the same tariff on the supplier website

no additional details about the tariff on the supplier website

explaining how the results of a comparison are calculated

identifying how many providers are compared


Table 10S: Areas where Big 4 sites do not perform as well as non-Big 4


% (27) %

(89) %

Provide explanation of how initial list is sorted 30 19 34 Can reorder initial list of tariffs 33 22 36

No additional costs appear partway through application 52 41 55 Able to find tariff on supplier site 86 74 90

No additional details on supplier site 83 75 85 Base: All websites

(18) %

(4) %

(14) %

Explain how results are calculated 61 25 71 State how many providers are compared 61 25 71

d. Energy: Where Big 4 and non-Big 4 price comparison websites perform equally

The Big 4 and non-Big 4 price comparison websites perform equally in the following areas:

being able to input current energy usage and postcode as search criteria

being able to filter the initial list of quotes

number of quotes shown on first page of results

no additional restrictions or limitations being found partway through an application

not being confusing or misleading to consumers

providing information about when prices were last updated

Table 10T: Areas where Big 4 and non-Big 4 sites perform equally


% (27) %

(89) %

Can input postcode 86 89 85 Can input details of current energy usage 89 86 90

Can filter initial list of tariffs 42 44 42 Show 1-9 tariffs on first page 16 15 16 Show 10+ tariffs on first page 84 85 84

No restrictions/limitations found partway through 47 48 46 Not found confusing 93 93 93

Not found misleading 95 93 96 Information about when prices were last updated 57 59 57


10.7 Energy: Chapter summary

Although a majority of mystery shoppers were asked for postcode and current

usage before providing a quote (86 per cent and 89 per cent respectively), the

accuracy of prices shown to consumers would be improved if these were mandatory

Transparency regarding the suppliers and tariffs available via the price comparison

website would be better if all sites provided an option for consumers to include

products that would require a direct application to the supplier (only 21 per cent of

shoppers had this choice)

Although in most cases it was obvious that the initial list of quotes was ordered on

price, or on occasion by cost saving, this was not often explicitly stated (30 per cent)

Where the initial list was in price order, or could be reordered on this variable, a

check of the first 2-3 pages showed the order to be correct in all cases

A minority of the initial lists could be reordered or filtered (33 per cent and 42 per

cent respectively), both of which are useful tools, and could be adopted by more of

the sites

Summaries of the current tariff were generally comprehensive, although more of

them could show whether or not any discounts apply (only seen in 42 per cent of

site visits)

Additional restrictions or limitations were present in the details of tariffs for 53 per

cent of site visits. If online account management, paperless billing and payment by

Direct Debit were always choices when inputting details at the start of a search, this

proportion could be substantially decreased

Similarly, 47 per cent of site visits uncovered a cancellation fee in the tariff details.

The ability to include or exclude such tariffs at the start of a search would negate

this

Providing information about when prices were last updated could be improved (43

per cent of the initial tariff lists did not show this). The same was true for

information about VAT (51 per cent of the initial tariff lists did not show this)

Energy price comparison websites could, in some areas, provide more details of how

they operate. Examples include their policy for dealing with rogue or fraudulent

suppliers (which none of the sites provided), having a Code of Conduct (which only

11 per cent provided) and explaining why some suppliers or tariffs might not be

included (again only provided by 11 per cent)

There was also scope for improvement amongst the 50 per cent not supplying

details of their complaint process

On a more positive note, 83 per cent explained how they generated income and 89

per cent provided, or signposted consumers to, additional energy-related

information


Most selected tariffs could be found on the supplier website (86 per cent). Not

being able to find the tariff was, on some occasions, potentially due to

inadequacies in the supplier site, rather than the price comparison website

showing a non-existent product

When the tariff was found on the supplier site, it was the same price in only 47 per

cent of comparisons, the remainder being split 29 per cent higher on the supplier

site and 24 per cent lower

In most cases the reason for the price discrepancy could not be ascertained (85

per cent). For the remainder, possible explanations included differences in the

way the cost calculation had been done, and a supplier discount not present on

the price comparison website

Whilst those price comparison websites accredited by Consumer Focus were more

consumer-friendly than non-accredited sites in a range of areas, there were also a

few instances when the accredited websites performed less well than the non-

accredited ones

The Big 4 energy price comparison websites were equivalent to other price

comparison websites in the energy market for some aspects of their operation.

For other aspects the Big 4 were better performers, but the reverse situation also

held true


11. Broadband

11.1 Broadband: Number of price comparison websites and mystery shops

A total of 16 price comparison websites were included in the Broadband market part of the research.

Mystery shoppers made 122 site visits to pose scenarios on these sites.

11.2 Broadband: Compliance with Consumer Protection from Unfair Trading Regulations

a. Broadband: Obtaining a list of quotes

In the majority of cases, mystery shoppers were required to enter their postcode in order to obtain a

list of quotes. In a third of all cases, a postcode was the only criteria requested and therefore any

results reflected little more than availability based on location.

This broad approach may suit the needs of consumers with limited awareness of their current

package and/or future needs. But for those consumers looking to compare their current plan against

the wider market, the basic search facility does not permit enough criteria to be included initially to

produce results for a like for like comparison.


%

Can input details on… …postcode 88

… looking for broadband only or broadband package 42 ..minimum speed 16

… monthly cost 13 … provider (current or preferred) 11

…phone number 11 …contract length 10

… usage limit 10 … inclusive calls 5

…annual cost 2 additional features (eg online security) 2

…email address 1

When presented with the list of results, the majority of mystery shoppers were satisfied that these

quotes took into account all their search criteria. In the few instances where the results ignored

search criteria, the requirement most commonly missed was whether the quotes should include

broadband only or broadband packages.

In most cases, it was easy to see how the list was ordered. In over half of cases, mystery shoppers

could find a clear explanation with results being in ascending cost order (1st year costs) the most

common situation.


Table 11B: Whether there was an explanation of how initial list had been ordered

Total Base: All mystery shops (122)

%

Yes (total) 54 Yes, explanation that was ordered on 1st year costs 32 Yes, explanation that was ordered on monthly costs 7

Yes, explanation that was ordered on speed 2 Yes, explanation that was ordered on special offer 1


No, but could see it was ordered on 1st year costs 20 No, but could see it was ordered on monthly costs 7

No, but could see it was ordered on contract length 1 No, but could see it was ordered on speed 1

No, but could see it was ordered in some other way 2 No, and was not obvious how ordered 15

b. Broadband: Ability to reorder and filter initial list of quotes

Following receipt of the results, mystery shoppers were able to reorder the possibilities on a wide

range of different variables. The most prevalent reordering factors were monthly cost and

broadband speed. These were also commonly provided as filtering options. Contract length, provider

and usage limits were offered as filters in over half of cases.

The ability to personalise the results at this stage increases in importance due to the limited criteria

permitted during the initial search. The results returned are generally very broad, based almost

entirely on availability by location and therefore the facility to tailor and personalise the list becomes

essential.


either not in price order initially or could not be reordered on price (18 per cent). Amongst the

remaining 82 per cent, the first 2-3 pages of options were checked and in all cases were found to be

sorted correctly (ie they were, indeed, in order of ascending price).



%


Base: All mystery shops where can reorder list

(100 )

%

Can reorder on… …monthly costs 97 …speed 85 …usage 74

…1st year costs 73 …contract length 72

…provider 62 …activation fee (set up costs) 20

...special offers 12 …popularity 12

…customer ratings 9 …reliability 7

Base: All mystery shops where can filter list

(72 )

%

Can filter on… …contract length 74 …monthly costs 71

…speed 68 …provider 64

…usage 63 …additional features 24

…inclusive calls 24 …1st year costs 21

…activation fee 10 …TV channels 4

c. Broadband: Speed

Broadband speed was commonly offered as a filtering option (68 per cent) however it is not always

clear what type of speed is used to filter the results and therefore results could be misleading.

Across the sites visited, there was not a consistent approach to the type of broadband speed shown

and therefore a clear explanation of the type of speed displayed is essential in order to provide

consumers with accurate and relevant results.

In the majority of cases, the type of broadband speed used is labelled. That said, a quarter of

mystery shoppers were unsure about whether the speed quoted was actually achievable at their

location as there was no definition provided.

The ability to change the type of broadband speed used was also provided in some cases (where

reordering was possible) allowing for results to be made more accurate and personalised.


69 per cent of websites provide an online checker for consumers to test their broadband speed. This

information was generally easy to find and may help to ease confusion for consumers.

Table 11D: Speed Total Base: All mystery shops where broadband speed was included in the results (114)

%

Is there a note about broadband speed being an estimate based on location or the best achievable?

Yes, and it was clearly labelled 51 Yes, although it wasn’t very clearly labelled 23

No, there was not a note about what ‘speed’ is displayed 26

Total Base: All mystery shops that could be reordered by speed (85)

%

Is it possible to change the definition of speed and reorder the options? Yes 45 No 55

Total Base: All mystery shops that could be filtered on speed (52)

%

What types of speed can be used to filter the initial list… …by the estimated broadband speed for your area 23

…by the ‘best’ possible speed achievable 38 …options can be filtered by speed but it is not clear which type is used 42

Total Base: All websites (16)

%

Does the site provide… …broadband speed checker 69 Information was easy to find 88

d. Broadband: Ease of viewing the list of quotes Most mystery shoppers saw 10 or more quotes on the first page. When they only saw 1-9 this was

often because there were no more to show. In fact, for half of all mystery shops there were no

quotes beyond the first page. Amongst the remainder, there was a mix of being able to view all the

quotes at once and moving forward page by page.


Table 11E: Number of products per page and options for viewing Total Base: All mystery shops (122)

%

Number of products per page 1-9 25 10+ 75



There is an option to … …to view all quotes 26 …move to next set or page of results 35

…both of these 11

e. Broadband: Accessing a summary of search criteria

Most mystery shoppers were able to see and/or edit their initial search criteria from the results

page.

Table 11F: Summaries Total Base: All mystery shops (122)

%

On the first page of quotes there is… …a summary of the search criteria 50

…a link to access and edit the search criteria 27 …neither of these 39

f. Broadband: Restrictions and limitations not previously mentioned

In order for a consumer to make an informed decision about a package’s suitability, it is necessary

for the quotes shown to include a range of details and specifications.

When faced with a list of possibilities, it is frustrating and misleading if any quotes are found to


by the price comparison website. Additionally, if a consumer does not fully read the details, or

assumes that the quotes shown must be appropriate for them, they could end up trying to buy or

switch to an inappropriate product.

The first five quotes in the list of options generated after inputting details were examined. Over half

the mystery shops found these quotes contained some form of additional restriction or limitation.

One or more quotes in a third of all the lists specified a minimum contract length with one in every

four mystery shops seeing one or more packages requiring a specified landline service (eg BT, Sky).


In some cases restrictions or limitations were only discovered at the point of application

with minimum length of contract and existing/new customers being the most commonly

found issue. However, in most cases mystery shoppers were happy that the application

process did not contain any additional restrictions or limitations.

Table 11G: Quote restriction and limitations Total Base: All mystery shops (122)

%

Did any of the first 5 quotes have additional restrictions/limitations? Yes 59 No 41

Additional restrictions/limitations (first 5 quotes in initial list) Minimum length of contract 31

Must have/switch to specified landline (eg BT) 23 Usage restrictions 21

Existing customers/new customers only 18 Must pay by Direct Debit 5

Not available in the specified area 2 Home visit required for installation 1

Did quote chosen to switch to have any additional restrictions/limitations (at point of application)

Yes 27 No 73

Additional restrictions/limitations (at point of making applications) Minimum length of contract 11

Existing customers/new customers only 11 Must have/switch to specified landline (eg BT) 9

Usage restrictions 6 Must pay by Direct Debit 3


g. Broadband: Additional costs not previously mentioned

In over a half of cases, mystery shoppers discovered that the initial amount quoted was not the full

cost, because additional items were added. When the first five quotes were examined, one or more

packages in a third of all the lists included an extra cost for line rental which was not part of the

original quotation.

Applying costs at this stage may increase confusion and frustration for consumers who are therefore

unable to compare prices accurately from the initial quotes provided.

As with the additional restrictions or limitations, most additional costs were included prior to the

application stage. However, in a minority of cases additional costs particularly for line rental were

only highlighted as consumers proceeded through their application.


Table 11H: Additional costs Total Base: All mystery shops (122)

%

Did any of the first 5 quotes have additional costs? Yes 62 No 38

Additional costs (first 5 quotes in initial list) Line rental 37

Installation 22 Equipment 20

Cancellation fee 13 Activation fee (set up costs) 11

Delivery 11

Did quote chosen to switch to have any additional costs (at point of application)

Yes 32 No 68

Additional costs (at point of making applications) Line rental 16

Installation 6 Activation fee (set up costs) 6

Equipment 5 Cancellation fee 3

Delivery 3

h. Broadband: Whether anything on the price comparison website was confusing or misleading


who were not convinced that the search facilities available to them would identify the best packages

for their needs:

‘I felt that the search facility was more like a filter so I could not enter personal specifics to get the results I wanted this was slightly off putting. I found it time consuming and difficult to find the type of contract that I wanted.’

‘I don't see how entering my postcode only can bring up the best choices for me.’

‘To find where to specify my search, was more complicated then it needed to be, the website was packed with lots of information that didn't apply to me. The website also included very few search results.’


misleading (84 per cent). Comments from those who thought the opposite included:


‘Some results had hidden installation charges, which I found a bit misleading when viewing results in price order, as they really weren't as cheap as they were originally thought to be.’

‘Most of the quoted prices were for O2 customers, and the prices for non-O2

customers were considerably higher – it felt more like an ad for becoming a customer with O2 than a broadband comparison site.’

i. Broadband: Overall impressions


few with a negative view were mostly concerned with the website’s limited or confusing search

facility.



more information gave a range of requirements:

‘Is the package available in my area?’ ‘There is no 'More info’ or 'Further information' link for any of the packages

displayed, only a link to take you to the package site to buy it. I'd need to find out more information, such as exactly what is included, how the billing works, cancellation charges etc.’

Table 11I: Overall impressions Total Base: All mystery shops (122)

%

Is your overall impression of this website? …very positive 31




11.3 Broadband: Accuracy

a. Broadband: Whether package was available on supplier website

In 89 per cent of cases, the same package was found on the supplier website. Of the remainder, in a

small proportion of cases this was due to the supplier website reporting the service was unavailable

for that postcode. This inconsistency meant that mystery shoppers were able to apply for packages

via comparison websites that would not be available for their postcode directly from the supplier

site. Examples where this was the case included:


‘I put in my postcode and it said that my broadband speed was very low and the package wasn't available.’

’The package on the supplier's website was cheaper, however, it could not be

provided for my postcode, which was not identified till I was re-directed to the supplier's website.’

‘After finding the same package I entered my phone number and postcode only to

be told that the service is not available in my area.’

b. Broadband: Cost of package on supplier website compared to cost on price comparison website

It would be reasonable for a consumer obtaining a quote from a price comparison website to expect

to get the same quote if they visited the supplier website.

When the package could be found on the supplier site it was offered at the same price in three in

out of four comparisons. The other quarter was a mix of higher and lower prices. The level of price

difference ranged from £4 to over £173.

Table 11J: Package costs – price comparison website versus supplier website Total Base: All mystery shops (122)

%

Could find package on supplier site? Yes 89 No 11

Base: All mystery shops where package could be found on supplier site (108)

%



In over half of cases where there was a price difference, there was no obvious reason. The

explanation for the rest (which amounts to a total of 13 mystery shops) was often due to the cost of

line rental being more than previously quoted or included as a new additional cost:

‘Because I had to choose a landline package at the same time. I never had a choice to opt out of this.’

‘The price was considerably more when I clicked through to the supplier site, as the price for the phone line was £3.00 a month more than stated on the comparison site.’

‘The BT site was charging me for line rental as well as the broadband, the price comparison site was charging me only for broadband. I could not see how to get broadband without line rental on the BT site.’


‘Line rental is not included on the comparison site but is included on the supplier site.’

When the supplier cost was lower and a reason could be seen this was sometimes because the

supplier was offering a discount.

Table 11K: Reasons for the price difference Total Base: All mystery shops where package could be found on supplier site and cost was different

(27) %


Total Base: All mystery shops where package could be found on supplier site and cost was different with a clear reason why

(13) %

Reasons for price difference

Added on line rental charges 54 Special offers and discounts 23

Activation charges 15

c. Broadband: Whether the supplier site had additional details


comparison website. Where additional information was found, it centred mostly on whether you

were an existing or new customer.

Table 11L: Whether supplier site had additional details Total Base: All mystery shops where package could be found on supplier site (108)

%


What were the additional details? Existing customers/new customers only 24

Minimum length of contract 15 Must have/switch to specified landline (eg BT) 15

Activation fee (set up costs) 15 Usage restrictions 6


Must pay by Direct Debit 3


11.4 Broadband: Transparency

a. Broadband: Information provided

The price comparison websites covering the broadband market were sometimes lacking when it

came to providing various types of information. Of the 16 price comparison websites looked at:


only 13 per cent had a Code of Conduct on the website

only 13 per cent gave an explanation about why some providers or packages might not be included

only 31 per cent explained how they calculated the results of a comparison

only 38 per cent identified how many providers they compared, although in 68 per cent of cases this information was easy to find

only 38 per cent provided a glossary of the terms used (eg wireless router, megabyte, kilobyte, gigabyte, internet service provider)

only 38 per cent published details of their complaint process or policy, which in most cases was difficult to find

Moving into areas where transparency was better:

amongst the 56 per cent that included provider or product ratings, 78 per cent explained how these were calculated

62 per cent said how frequently they updated their prices, which was easy to find in most cases (60 per cent)

64 per cent included a clear breakdown of what is included in the 1st year cost of the broadband package

69 per cent stated how they generated income, and on all of these websites it was easy to find

69 per cent included a broadband speed tester and all the websites signposted consumers to the latest sector news

In most cases, the first page of quotes reached by the mystery shoppers did not include any

information about when prices were last updated. A minority of mystery shoppers were able to find

a note about when prices are updated, although this was not always clearly displayed.

Table 11M: Information about price updating Total Base: All mystery shops (122)

%


Yes, although it wasn’t very clearly displayed 10 No, there was not a note about when the prices were last updated 84


b. Broadband: Details of the information provided

For all of the six sites with a complaints process or policy, consumers could submit their complaint

online. Half of these also specified the number of days it would take before a response was received.

When a price comparison website gave details about how often prices were updated, this ranged

from the specific (eg ‘daily’ or ‘monthly’) to the nebulous (eg ‘regularly’). Several sites referred to a

combination of two approaches (eg ‘monthly’ and ‘when a provider tells us things have changed’) to

make sure that prices remained up to date and accurate.


successful switch. Advertising sales was used to generate income in four comparison sites. Two

comparison sites were paid per click through to a supplier site. For two sites, providers paid directly

to be included in the searches but names were not specified in either case.

Sector news was the most commonly provided piece of additional information and was included on

all websites. Over half of websites provided consumers with guidance about how to improve their

broadband connection speed and/or contact details for external agencies (eg Ofcom).

Table 11N: Operational information Total Base: All websites with a complaints policy NB Low base (6)

%



Base: All websites with information about income generation (11) %


From advertising sales 36 Provider pay to appear on the site 18

Comparison site paid per click through to supplier site 18 Base: All websites (16)

%

Provide or signpost information about… Latest sector news 100

Broadband speed tester 69 Contact details for external agencies (eg Ofcom) 56

Information about improve the connection speed 56 Best practice in comparison site 6


11.5 Broadband: Accreditation

a. Broadband: Overview

Of the 16 Broadband price comparison websites, four are accredited by Ofcom under the

Accreditation scheme for price comparison calculators. One other site states that their cost

comparison calculator is approved by Ofcom. Only the four price comparison websites specified as

accredited under the Ofcom scheme are taken as ‘accredited’ for the purpose of this research.

In general, the performance of price comparison websites with accreditation is slightly better than

those for sites without accreditation. There are several parameters where they perform better than

non-accredited sites, but for a few features appear to be less consumer-friendly than the non-

accredited sites. On other occasions there are no differences between the two types of site.

b. Broadband: Where accredited price comparison websites sites perform better than those that are not accredited

Accredited sites were more likely to allow the consumer to include criteria other than their postcode

at the start of their comparison, namely minimum speed, usage limit or monthly cost.

When used correctly, the extra details mean accredited sites should be able to provide more

personalised search results than the non-accredited sites. However, this additional information is

not always used effectively as 14 per cent of lists from accredited sites included packages that did

not match initial search criteria. Being able to filter the initial list of quotes was more common

amongst accredited sites.

Table 11O: Initial search criteria


Base: All mystery shops (122)

%

(45)

%

(77)

%


…minimum speed 16 27 10 …monthly cost 13 18 10

…contract length 11 16 8 …usage limit 10 18 5

Can filter initial list of packages 59 78 48 List included packages that did not match my criteria 7 14 3

In general, mystery shoppers felt more positive about the accredited sites in comparison to the non-accredited sites (89 per cent compared to 69 per cent respectively).


Table 11P: Areas where accredited sites perform better than non-accredited sites



(45) %

(77) %

Overall impression of the website was positive 76 89 69 It was easy to enter information on the website 83 93 77

Do results include a phone number to apply over the phone?

Yes, there is a telephone number 64 69 61

Additional costs added (initial 5 quotes) 62 53 68 Base: All mystery shops where broadband speed is included in the results

(114)

%

(39)

%

(75)

%

Is there a note about which broadband speed is used? Yes, and it was clearly labelled 51 59 47

Base: All mystery shops that could be reordered on speed of broadband

(85) %

(31)

%

(45)

%

Yes, it is possible to change the type of speed and reorder the results

45 61 35

Accredited sites were generally clear about the type of broadband speed (location based estimate or

best advertised) used and in over half of cases allowed mystery shoppers to change the type of

speed and reorder the results.

Accredited sites were more likely than non-accredited sites to include a telephone number alongside

the various packages for consumers to apply over the telephone rather than online. This is in line

with the accreditation guidelines published by Ofcom.

The likelihood of extra costs being added on to the initial quotes is also greater among non-

accredited sites (68 per cent) than accredited sites (53 per cent).

In terms of providing information about how the site operates, accredited price comparison

websites outperformed those that have no accreditation in some distinct areas. They were better at

explaining how the site generates income and notifying consumers when prices were last updated.

More accredited sites explained to consumers about what is included in 1st year costs compared to

the non-accredited sites. Mystery shoppers were also more likely to be able to edit and/or see a

summary of their criteria from the first page of results on accredited sites than non-accredited sites.


Table 11Q: Provision of information



(45) %

(77) %

On the first page of packages, there was a… note about when the prices were last updated 16 36 4

summary of criteria and/or a link to edit criteria provided on the first page of packages

61 76 52

clear breakdown of what is included in 1st year costs 64 78 56



(4) %

(12) %

There is information to explain how the site generates income

69 75 67


Base: All websites who specify how they generate income (11) %

(3) %

(8) %

The comparison site is paid per successful switch 73 100 63 From advertising sales 36 67 25

c. Broadband: Where accredited price comparison websites do not perform as well as those that are not accredited

On a few parameters the price comparison websites accredited under the Ofcom accreditation

scheme did not perform as well as the non-accredited websites. Accredited sites were more likely to

return packages that did not match the initial search criteria. This may be due to the wider range of

criteria permitted and therefore these sites have a greater risk of returning an incorrect package.

Accredited sites were less likely to be explicit about the type of broadband speed used to filter

results than non-accredited site.

In comparing quote details to those provided on the supplier site, extra details were added more

frequently when the quote had been sourced from an accredited price comparison website (38 per

cent) compared to a non-accredited one (26 per cent).

Non-accredited sites were generally found to be less confusing and/or misleading than accredited

sites.

A Code of Conduct was more likely to be found on an accredited site compared to a non-accredited

one, as was a complaint policy with all sites offering the ability to submit your complaint online.


Table 11R: Areas where accredited sites do not perform as well as non-accredited sites



(45) %

(77) %

The packages can be filtered by speed but it is not clear which type of speed is used

42 48 35

List included packages that did not match my criteria 7 14 3

There are additional restrictions/limitations not mentioned previously (supplier site)

31 38 26

Yes, the website contained information that was confusing

16 18 12

Yes, the website contained information that was misleading

15 18 13



(4) %

(12) %

Yes, there a complaints policy/process on the website? 38 0 50 Yes, there a code of conduct on the website 13 0 17

d. Broadband: Where accredited and non-accredited price comparison websites perform equally

There was little difference between accredited and non-accredited price comparison websites on a

range of factors, including the option to view all quotes, and reordering the results.

Both accredited and non-accredited price comparison websites both clearly displayed package in

price order (following reordering where necessary) and provided enough information for mystery

shoppers to feel comfortable in making an application.

Mystery shoppers were equally likely to be able to locate the same package on a supplier site

regardless of the source of the initial quotation (eg from an accredited or non-accredited website).

Both types of sites were also equally accurate in the pricing of these packages as the majority could

be found at the same cost on the supplier site.


Table 11S: Areas where accredited sites and non-accredited sites perform equally



(45) %

(77) %

It was possible to reorder the initial list 82 80 83 On the first page of packages…

...option to view all 26 27 26

I was able to find the same package on the supplier site 89 87 90


Base: All mystery shops where mystery shoppers were able to find the same package

(108) %

(39) %

(69) %

The supplier site cost was exactly the same as the price comparison

75 77 74



(45) %

(77) %

Yes, I have sufficient information to make an application 91 91 91 Yes, the packages were all in price order 97 95 98

11.6 Broadband: Big 4 price comparison websites versus others

a. Broadband: Overview

All of the Big 4 price comparison websites provide broadband-related comparisons. None of these

sites are accredited under the Ofcom accreditation scheme for price comparison calculators.

Comparing the Big 4 against other price comparison websites in the broadband market reveals a

mixed picture. Overall, using a comparison from the Big 4 does not mean that the information will be

better or more accurate. There are areas where the Big 4 outperform lesser used sites including

search accuracy and overall impression but equally there are factors which are more successful on

the non Big 4 sites.

b. Broadband: Where Big 4 price comparison websites sites perform better than non-Big 4


offering an option to include the type of package required in the initial search criteria

results list appears to be based on all my criteria

explanation that the list was order on 1st year costs

reordering the initial list

filtering by usage, contract length and broadband speed was more common on Big 4 sites than non-Big 4 websites

accredited sites are more likely to be explicit about the type of broadband speed used for comparison although this may not always be easy to find

more likely to explain which type of speed is being used and allow filtering on that criteria


more likely to outline what is included in the 1st year cost

initial quote is more likely to reflect the cost shown in the application stage on Big 4 sites compared to non-Big 4 sites however this accuracy is not continued when comparing quotes to supplier sites

overall impression of Big 4 websites is more positive than non-Big 4 sites

Table 11T: Areas where Big 4 price comparison websites perform better than non-accredited sites


% (25)

% (97)

%

Can input details on… …type of package 42 64 36

Yes, list appeared to be based on all my criteria 93 100 91 Yes, there was an explanation that the list was ordered

on 1st year costs 32 68 23

Yes, it is possible to reorder the initial list 82 92 79 Can reorder on….

…1st year costs 73 91 68 …provider 62 70 60

…activation fee (set up costs) 20 30 17 Can filter on…

…usage 63 75 60 …contract length 74 83 72

…speed 68 75 67 Yes, all packages were sorted into price order (1st year

costs) 97 96 97

Yes, there was a note about the type of broadband speed used but it was not clearly labelled

23 40 16

No, there was not a note about what speed was displayed

15 12 28

The packages can be filtered by speed but it is not clear which type of speed is used

42 20 48

10 or more quotes displayed on the first page of packages

75 92 71

Yes, there is a clear breakdown of what is included in the 1st year cost

64 76 61

No, there are no additional restrictions/limitations (from the comparison site to application)

73 88 69

No, there are no additional costs (from the comparison site to application)

68 72 67

Overall impression of the website is positive 76 80 75


c. Broadband: Where Big 4 price comparison websites do not perform as well as non-Big 4


limited range of search criteria. Big 4 sites were less likely to include minimum speed, usage limit, provider, monthly cost, and contract length in their initial criteria

being able to filter the initial list of quotes

offering an easy to find explanation about the type of broadband speed used for comparison

allowing the type of speed to be changed and options reordered on that criteria

displaying details about how regularly prices are updated

including additional costs after the initial quotation

package was more likely to cost the same on a supplier site if the quotation was sourced from a non-Big 4 comparison website

Table 8U: Areas where Big 4 price comparison websites do not perform better than non-Big 4 sites


% (25)

% (97)

%

Can input details on… …minimum speed 17 4 20

…usage limit 12 4 11 …provider 11 4 13

…monthly cost 13 4 15 …contract length 11 8 11

Yes, it is possible to filter the initial list 59 48 62 Yes, there was a note about the type of broadband

speed used and it was clearly labelled 51 48 57

Yes, it is possible to change the type of speed and reorder the results

45 24 50

On the first page of packages… …there was nothing to say when prices were last

updated 84 92 82

…a summary of your search criteria 50 44 52 Yes, there is a telephone number clearly displayed

against the various packages 64 60 65

No, there are no additional costs (from the initial results) 38 16 43

The supplier site cost was exactly the same as the price comparison site cost

75 59 79


d. Broadband: Where Big 4 and non-Big 4 price comparison websites perform equally


being able to input postcode as search criteria

being able to edit your search criteria using a link from the results page

packages did not have additional restrictions/limitations not mentioned in the initial quotations

providing a sufficient level of information to make an application

Table 11V: Areas where Big 4 price comparison websites perform equally to non-Big 4 price comparison websites Total Big 4 Non-Big 4 Base: All mystery shops (122)

% (25)

% (97)

%


On the first page of packages… …a link for you to access and edit your search criteria 27 28 27

…does not include a summary or link to the search criteria

39 40 39

No, there are no additional restrictions/limitations (not mentioned in the search criteria)

41 44 40

Yes, I have sufficient information to make an application 91 92 91

I was able to find the same package on the supplier site 89 88 89 Yes, I was able to see the reason for the cost difference 41 44 39

Yes, the supplier site has additional details 31 32 30


11.7 Broadband: Chapter summary

In the majority of cases, mystery shoppers were required to enter their postcode in

order to obtain a list of quotes. In a third of all cases, a postcode was the only criteria

requested. The type of package (eg broadband only, broadband and telephone) was the

next most popular option appearing on 42 per cent of searches

Accredited sites are more likely to ask for a wider range of options such as minimum

speed and usage limit however this information is not always used effectively with 14

per cent of searches bringing back inaccurate results which were not based on the initial

criteria

The predominance of postcode based searches mean that methods of personalising and

tailoring the results once delivered become of greater importance to the consumer. 82

per cent provided the ability to reorder the results with 59 per cent offering filters. The

most commonly provided factors being monthly cost and contract length

Overall, using a price comparison site to compare broadband packages is generally seen

as a positive experience. Only a minority of shoppers reported the sites containing any

misleading or confusing information. Among those who did there was a lack of belief

that searches based almost entirely on a postcode would actually result in the most

suitable deals

Websites do not consistently use one type of broadband speed to offer comparison. In

most cases there is an explanation provided as to which type of speed is displayed and

45 per cent of sites allow consumers to change the definition and reorder based on this

factor

Most sites allowed consumers to view, edit and resubmit their search criteria. Most

were able to easily move from one page of results to the next and/or chose to view all

suitable packages on one page

Additional restrictions occurred in 59 per cent of the initial search results. Minimum

length of contract was the most common restriction and was generally applied at both

the initial results and application stages

Additional costs were present in 62 per cent of all mystery shops. Line rental was the

most prevalent extra cost affecting over a third of cases at the initial search stage and a

further 16 per cent of shops while making an application

Price comparison websites generally returned broadband packages which could be

found on supplier sites. For 75 per cent of cases, the package was offered at the same

price and where differences occurred this was often due to line rental being added as

an additional cost

Information about how frequently prices are updated, what is included in 1st year costs

and how the site generates its income are regularly found on comparison sites.


Most price comparison sites within the broadband market request a consumer’s

postcode as part of the initial search process. 88 per cent of cases included this factor

with the type of package (eg broadband only, broadband and telephone) as the next

most popular option appearing on 42 per cent of searches

Accredited sites are more likely to ask for a wider range of options such as minimum

speed and usage limit however this information is not always used effectively with 14

per cent of searches bringing back inaccurate results which were not based on the initial

criteria

The predominance of postcode based searches mean that methods of personalising and

tailoring the results once delivered become of greater importance to the consumer. 82

per cent provided the ability to reorder the results with 59 per cent offering filters. The

most commonly provided factors being monthly cost and contract length

Overall, using a price comparison site to compare broadband packages is generally seen

as a positive experience. Only a minority of shoppers reported the sites containing any

misleading or confusing information. Among those who did there was a lack of belief

that searches based almost entirely on a postcode would actually result in the most

suitable deals

Websites do not consistently use one type of broadband speed to offer comparison. In

most cases there is an explanation provided as to which type of speed is displayed and

45 per cent of sites allow consumers to change the definition and reorder based on this

factor

Most sites allowed consumers to view, edit and resubmit their search criteria. Most

were able to easily move from one page of results to the next and/or chose to view all

suitable packages on one page

Additional restrictions occurred in 59 per cent of the initial search results. Minimum

length of contract was the most common restriction and was generally applied at both

the initial results and application stages

Additional costs were present in 62 per cent of all mystery shops. Line rental was the

most prevalent extra cost affecting over a third of cases at the initial search stage and a

further 16 per cent of shops while making an application

Price comparison websites generally returned broadband packages which could be

found on supplier sites. For 75 per cent of cases, the package was offered at the same

price and where differences occurred this was often due to line rental being added as

an additional cost

Information about how frequently prices are updated, what is included in 1st year costs

and how the site generates its income are regularly found on comparison sites.

Complaints process was less common included by only 38 per cent of sites and in most

cases it was difficult to find


Complaints process was less common included by only 38 per cent of sites and in most

cases it was difficult to find

Additional information such as a broadband speed tester was provided by over two

thirds of site

Overall, accredited sites perform slightly better than non-accredited sites with clearer

guidance about how they generate income, how often prices are updated and the

general feel and appearance of the website

Accreditation made little difference to either the accuracy or availability of packages as

mystery shoppers were equally likely to find packages on supplier sites regardless of

where the quotes originated from (accredited/non-accredited)

Comparison between the Big 4 and non-Big 4 websites revealed a mixed picture. There

were a number of areas such as accuracy of results, explanation of the type of

broadband speed used and overall impression of the website where the Big 4

outperform the lesser known sites. However non-Big 4 sites provided shoppers a wider

range of search criteria, were less likely to include additional costs and were more

accurate in their quotations with 79 per cent of packages offered at the same price as

on the supplier site


12. Mobile phones

12.1 Mobile phones: Number of price comparison websites and mystery shops

A total of 15 price comparison websites were included in the mobile phone market part of the

research. Mystery shoppers made 111 site visits to pose scenarios on these sites.

12.2 Mobile phones: Compliance with Consumer Protection from Unfair Trading Regulations

a. Mobile phones: Obtaining a list of quotes Site search criteria tended to focus initially on the handset and then the provider and type of

contract. Expected or existing monthly costs were only asked for in 36 per cent of cases. Contact

information was rarely asked for as part of the initial search.


%

Can input details on… ...brand of phone 75 ...model of phone 67

...network provider 65 ...type of contract 58

...inclusive minutes/texts 45 ...monthly cost 36

...data allowance 32 …contract length 30

...colour of phone 17 ...incentives 13

...additional features (eg insurance) 7 ...features and specification of phone 5

…name 1 ...postcode 1


In four out of ten cases, there was an explanation describing how the initial list has been ordered. A

further 32 per cent of cases did not offer an explanation but mystery shoppers were able to easily

identify which factor had been used to order the results. However, in 29 per cent of cases it was not

clear how the list was organised. In those cases, the lack of clarity may mean that consumers assume

that the most suitable options are at the top of the list and therefore risk missing deals that may be

cheaper or more relevant to their needs.


%

Yes (total) 40 Yes, explanation that was ordered on cheapest monthly cost 22

Yes, explanation that was ordered on most popular 11 Yes, explanation that was ordered in some other way 5

Yes, explanation that was ordered on cost of the handset 3 No (total) 32

No, but could see it was ordered on cheapest monthly cost 23 No, but could see it was ordered on cost of the handset 4

No, but could see it was ordered in some other way 3 No, but could see it was ordered on most popular 2

No, there was not an explanation and it was not obvious how the list had been ordered

29

b. Mobile phones: Ability to reorder and filter initial list of quotes

After the initial search, half of the mystery shoppers were able to reorder the possibilities on a wide

range of different variables. In two thirds of cases, options could be reordered on cheapest monthly

cost and inclusive minutes. Filters were more commonly provided by the comparison sites with

network provider as the most frequently included option (80 per cent). Monthly cost was also

offered as a filter but only by 53 per cent of cases.

Mystery shoppers were instructed to sort the list of quotes into price order. One or more quotes in a

quarter of all lists were either not in price order initially or could not be reordered on price (25 per

cent). Amongst the remaining 75 per cent, the first 2-3 pages of options were checked and in the

majority of cases (94 per cent) were found to be sorted correctly (ie they were, indeed, in order of

ascending price).



%



(58 )

%

Can reorder on… …cheapest monthly cost 79 …most inclusive minutes 71

…most inclusive texts 67 …largest data allowance 55

…contract length 40 …handset price 38 …most popular 39

…brand 31 …incentives 26

…customer ratings 3 Base: All mystery shops where can filter list

(80 )

%

Can filter on… …network provider 80

…cheapest monthly cost 65 …contract length 63

…inclusive minutes 59 …inclusive texts 51

…data allowance 35 …brand 33

…handset price 26 …incentives 23

...features and specification of phone 13 …most popular 5

c. Mobile phones: Ease of viewing the list of quotes

Most mystery shoppers saw 10 or more quotes on the first page. 38 per cent of mystery shoppers

who saw 1-9 products did so because there were no more options to show. Amongst the remainder,

most sites allowed mystery shoppers to move forward page by page rather than all at once.


Table 12D: Number of products per page and options for viewing Total Base: All mystery shops (111)

%




There is an option to… …view all quotes 5 …move to next set or page of results 56

…both of these 0

d. Mobile phones: Accessing a summary of search criteria


page. A quarter of shoppers were also able to access a summary of their chosen handset’s key

features. Comparison sites who offer this type of summary information may help to improve

consumers’ awareness and understanding of the suitability of a particular handset and lessen the

risk of confusion due to wide range of handsets available.

Table 12E: Summaries Total Base: All mystery shops (111)

%


…a link to access and edit the search criteria 18 …a summary of the key features of your chosen handset 28

...none of these 27


e. Mobile phones: Restrictions and limitations not previously mentioned

Whilst summary information about a handset’s features may help to support consumers in deciding

on the right phone, additional details about the nature of the overall package should also be

provided to help consumers make an informed choice about which option to apply for.



by the price comparison website. Additionally if a consumer does not fully read the details or

assumes that all packages shown must be appropriate for them, they may try to apply for a package

that they are ineligible for.

67 per cent of cases included additional details or limitations that were not mentioned in the initial

search criteria with minimum length of contract as the factor most often omitted.

Table 12F: Quote restriction and limitations Total Base: All mystery shops (111)

%


Additional restrictions/limitations (first 5 quotes in initial list)

Minimum length of contract 24 Refurbished phone (not new) 11

Usage restrictions 6 Must pay by Direct Debit 4

Subject to credit check 4 Existing customers/new customers only 1

Did quote chosen have any additional restrictions/limitations Yes 21 No 79

Additional restrictions/limitations (at point of making applications) Refurbished phone (not new) 7

Subject to credit check 6 Must pay by Direct Debit 4

Minimum length of contract 3 Usage restrictions 2


h. Mobile phones: Additional costs not previously mentioned

Additional costs not included in the initial quotation were not found as often as the additional

restrictions/limitations mentioned above. However, costs were added in 33 per cent of cases and a

further 32 per cent of packages were subject to extra charges at the application stage. The most

frequently added costs were for the handset itself (12 per cent) and handset insurance (20 per cent

of packages at application stage).

Table 12G: Additional costs Total Base: All mystery shops (111)

%

Did any of the first 5 quotes have additional costs? Yes 33 No 67

Additional costs (first 5 quotes in initial list) Cost of handset (phone) 12

Handset insurance 11 Credit (top up for pay as you go) 5

Technical helpline 5 Delivery 4

Paper bill 1

Did quote chosen have any additional costs Yes 32 No 68

Additional costs (at point of making applications) Handset insurance 20 Technical helpline 11

Credit (top up for pay as you go) 5 Paper bill 4

Delivery 3 Cost of handset (phone) 1

g. Mobile phones: Whether anything on the price comparison website was confusing or misleading

Most of the mystery shops did not result in any confusion (78 per cent). Most comments were from

shoppers who felt the search facilities, including the reordering and filtering options, were not

flexible or comprehensive enough for their needs:

‘I couldn't really see clearly how to filter the results shown, it wasn't explained well and I ended up clicking on the column headings, which sort of worked – it didn't always produce the results in the right order, though.’

‘Too many choices, would make it far too difficult to decide.’

‘There seemed to be no order to the handsets once I searched. I also found the search to be really limiting, as I could only input one criterion in it.’



misleading (84 per cent). Those who felt the opposite were mostly concerned with the additional

restrictions and/or costs which were applied at various stages with insurance and reconditioned

handset being used as examples:

‘The phone insurance was offered free at the time of purchasing and then it said I would be charged after the first month. There was a statement saying I could cancel it, but it wasn't possible to opt out of the insurance at the time of purchasing, which I found misleading.’

‘All the low cost phones were refurbished, which is misleading, but also irritating if

you don't want a refurbished phone, because you have to go through all these before you get to the new phones. The addition of airtime top ups to the quoted price was also misleading.’

h. Mobile phones: Overall impressions


few with a negative view were mostly concerned with the limitations of the search facilities and the

inclusion of misleading information as mentioned previously.

‘It is poorly laid out, hard to find what you want, and you cannot search for what you want. It also doesn't show you many options of phones, it is very limited.’ ‘I do not feel that this site offered enough search options in the area I was looking for which is value for money and I felt that the results were very misleading with regards to upfront costs and condition of the handset.’



more information focussed on the details of the contract, what was included or excluded and

whether there would be any additional charges:

‘I don't have very much information at all about the charges for texts, calls etc and I can't find it. Or whether this is only available with a monthly Direct Debit? ’

‘There was no information about the tariff and/or charges until I pressed the link

which took me to T Mobile and then I had to look around to find out the information I wanted.’


Table 12H: Overall impressions Total Base: All mystery shops (111)

%





12.3 Mobile phones: Accuracy

a. Mobile phones: Whether package was available on supplier website

In 63 per cent of cases, the same package was found on the supplier website. Mystery shoppers

reported a wide range of explanations as to why the same handset and contract package could not

be found. The reasons provided were similar regardless of whether the shoppers were looking on a

phone manufacturer or a network provider site:

‘I searched the supplier site via tariff. There were several at the same monthly price, but these had different names and different amounts of included minutes.’

’The handset was not available. I also checked other handsets to see if I could

locate the actual deal/package and I couldn't.’ ‘I was not able to find exactly the same contract on the supplier site. I could find a

tariff which was the same in terms of cost, minutes and texts, but also included 500mb of internet, which the tariff on the comparison site did not.’

b. Mobile phones: Cost of package on supplier website compared to cost on price comparison website


to get the same quote if they visited the supplier website.

In 83 per cent of cases where the package could be found, it was available at the same price as

advertised on the price comparison website. In the small proportion of cases where there was a

discrepancy this was a mixture of prices on the supplier site being lower or higher to that on the

price comparison site (seven per cent versus ten per cent). The level of price difference ranged from

£4 to £75.


Table 12I: Package costs – price comparison website versus supplier website

Total Network provider

Phone manufacturer


(56) (55)

Could find package on supplier site? Yes 63 75 51 No 37 25 49

Base: All mystery shops where package could be found on supplier

site (70) (42)

(38)

Supplier site cost was… …same as price comparison website 83 79 89

…higher than price comparison website 10 14 4 …lower than price comparison website 7 7 7

In most cases where there was a price difference, there was no obvious reason. The explanation for

the rest (which amounts to a total of four mystery shops) was often due to the cost of handset or

data allowance being more than previously quoted:

‘The comparison site listed the standard internet 750 data allowance as £9, but on the supplier site it was £5.’

‘The supplier site has a higher cost as they are charging more for the handset.’

Table 12J: Reasons for the price difference Total Network

provider Phone

manufacturer Base: All mystery shops where package could be found on supplier site and cost was different

(12) %

(9)

(3)

Can see the reason for the price difference? Yes 33 44 0 No 67 56 100

c. Mobile phones: Whether the supplier site had additional details


comparison website. Where additional information was found, it usually specified additional usage

restrictions.


Table 12K: Whether supplier site had additional details Total Network

provider Phone

manufacturer Base: All mystery shops where package could be found on supplier site

(9) %

(9) %

(3) %

Supplier site had additional details? Yes 25 24 25 No 75 76 75

What were the additional details? Usage restrictions 41 40 43

Subject to a credit check 24 30 14 Minimum length of contract 18 10 29

Refurbished phone (not new) 12 20 0 Must pay by Direct Debit 6 0 14

12.4 Mobile phones: Transparency

a. Mobile phones: Information provided

The type of information provided by price comparison websites covering the mobile phone market

was generally limited and not always easy to find. Of the 15 price comparison websites looked at:


just one site out of 15 (seven per cent) had a Code of Conduct on the website and/or provided a glossary of the technical terms used on the site

only seven per cent (one website) explained why some providers/products might not be included


a complaints policy and/or process was only present on one in every four sites and in most cases it was difficult to find. Although in most cases consumers were allowed to issue their complaint online

33 per cent included ratings for the providers and products that were included on the site. However the majority of these sites were clear about how these ratings had been calculated and in 60 per cent of cases the ratings included customer generated information such as a reviews

40 per cent of sites explained how they generate income

Moving into areas where transparency was better; a majority of the 15 price comparison websites

provided information:

80 per cent of sites signposted consumers to more relevant details such as latest sector news (73 per cent) and/or contact details for external agencies such as Ofcom (27 per cent). This information was easy to find in 83 per cent of cases

53 per cent identify the number of providers compared by the comparison website and in most cases this information is easy to find


The first page of quotes reached by the mystery shoppers rarely contained any information about

when prices were last updated. A small minority of mystery shoppers were able to find a note about

when prices are updated, which was generally displayed clearly.

Special offers and/or deals were offered in 61 per cent of cases with 56 per cent of those

automatically adding any savings into the monthly cost.

Table 12L: Information about prices and special offers Total Base: All mystery shops (111)

%


Yes, although it wasn’t very clearly displayed 0 No, there was not a note about when the prices were last updated 97

Yes, the package included a special offer/deal 61

Total Base: All mystery shops that included a package with a special offer/deal (45)

%

Yes, this special offer/deal was included in the overall monthly cost 56

b. Mobile phones: Details of the information provided

For three out of the four sites with a complaints process or policy, consumers could submit their

complaint online. Half of these also specified the number of days it would take before a response

was received.

On the whole, price comparison sites (80 per cent) did not publish details about how often prices

were updated. Only one site stated a specific timing (eg ‘daily’) with the remaining two sites using

more general terms such as ‘constantly updated’. This information was easy to find in all of these

cases.

A minority of websites explained how they generate income. All of these sites used a payment

model which pays them per successful application. Advertising sales were used to generate income

in one of the tested comparison sites. This information was easy to find on all but one of the six

comparison sites that published these details.


73 per cent of sites. One in every four sites provided consumers with contact details for external

agencies (eg Ofcom) and one in five offered information about how to recycle an old mobile phone.


Table 12M: Operational information Total Base: All websites with a complaints policy NB Low base (4)

%



Base: All websites with information about income generation (6) %


From advertising sales 17

Base: All websites providing of signposting additional information (12) %

Provide or signpost information about… Latest sector news 73

Contact details for external agencies (eg Ofcom) 27 Information about recycling an old mobile phone 20

12.5 Mobile phones: Accreditation

a. Mobile phones: Overview

Of the 15 mobile phone price comparison websites, three were accredited by Ofcom under the

Accreditation scheme for price comparison calculators. Shopsafe and SafeBuy also accredited one

site each. Six sites referenced other approvals and/or awards including website design and consumer

choice awards as well as independent recommendations by media and industry experts. These sites

were included in the non-accredited sample for the purpose of this research. Only the four sites with

formal accreditation (Ofcom, Shopsafe and/or SafeBuy) were classified as ‘accredited’ for the

purposes of this research.

The comparison of price comparison websites with accreditation against those without accreditation

is a variable picture. There are several areas where they perform better than non-accredited sites,

but for a few features appear to be less consumer-friendly than the non-accredited sites. On other

occasions there are no differences between the two types of site.


Table 12N: Accreditations Total Base: All websites (15)

%

Whether accredited or not Ofcom accreditation 20

Shopsafe accreditation 6 SafeBuy accredited 6

Not accredited 73 Other memberships/protections

Ofcom approved 6

b. Mobile phones: Where accredited price comparison websites sites perform better than those that are not accredited

Accredited sites were more likely to focus on the brand and model of handset in the initial search

criteria. Once results were returned, accredited sites were generally clearer about the factor used to

order the initial list than non-accredited sites.

Mystery shoppers were more likely to be able to reorder the lists on accredited sites whereas non-

accredited sites tended to offer a filtering option. In 13 per cent of accredited shops, shoppers were

unable to reorder based on monthly cost. This figure was much higher for non-accredited sites (38

per cent) and meant that the cheapest offer was not always present at the top of the results table

where consumers may expect it to be. Where filters were provided, 85 per cent of sites provided a

filter for network provider.

Table 12O: Initial search criteria



(46) %

(65) %

Can input details on… …brand of phone 75 78 72 …model of phone 67 72 63

There was not an explanation and it is not clear how the list has been ordered

29 15 38

Can reorder the initial list 52 65 43 Can reorder on… ...largest data allowance 55 67 43

…contract length 40 53 25 …incentives 26 30 21

…customer ratings 3 7 0 Can filter on… …network provider 80 85 77

…contract length 63 70 57 …incentives 23 27 19


Summarising and displaying information for future reference was generally more popular on

accredited sites than non-accredited with 61 per cent including a summary of search criteria and 43 per cent including details about the chosen handset. This function helped to remind consumers of their initial requirements and make any necessary amendments. Table 12P: Areas where accredited sites perform better than non-accredited sites



(46) %

(65) %

On the first page of packages, Is there a summary of your search criteria 58 61 55

Is there a summary of the key features of your chosen handset

28 43 17

I have the package list in price order 75 87 66

There were additional costs added on to the initial quote 32 30 35


There were additional limitations and/or restrictions added to the package at the application stage

21 16 25


Base: All mystery shops where the same package could be found (70) %

(28) %

(42) %

The supplier site has additional details (to the comparison site)

25 18 29

The chance of extra costs being added on to the initial quote was slightly greater among non-

accredited sites (35 per cent) than accredited sites (30 per cent). More shoppers (on accredited

sites) felt they were given sufficient information to make an application (87 per cent) in comparison

to non-accredited sites (82 per cent).

In most cases, accredited sites were more upfront about restrictions than non-accredited sites with

only 16 per cent highlighting additional limitations at the application stage. Searches on the

supplier’s sites also revealed that more accredited sites (than non-accredited) had included all the

necessary restrictions with only 18 per cent of cases highlighting additional limitations.

In terms of providing information about how the site operates, there were areas where accredited

price comparison websites outperformed those that have no accreditation. They were better at

explaining how the site generates its income, how it calculates the comparison results and more

often specified when prices were last updated.


Table 12Q: Provision of information



(4) %

(11) %

Yes, it is clear how the site generates income 40 75 27 The site specifies how regularly it updates its prices 20 50 9

The site explains how it calculate the results of a comparison

20 50 9

c. Mobile phones: Where accredited price comparison websites do not perform as well as those that are not accredited

Overall non-accredited sites were more likely to allow for details relating to the type, cost and length

of contract to be entered as part of the initial search criteria.

Non-accredited sites tended to provide consumers with a range of filters to shorten the list of

possible packages. These filters included monthly cost, inclusive texts and inclusive minutes. Where

reordering was available, three in every four non-accredited sites were offered the function to

reorder on monthly cost.

These sites were also less likely to add extra costs on to the initial cost than accredited sites with the

price of the handset being the most common cost added on by an accredited site (15 per cent of

cases).

In terms of information provided on the site, non-accredited sites were more likely to display their

complaints policy online (36 per cent of sites) and three out of four sites allowed consumers to issue

a complaint online. Information about when prices were last updated was only found in a small

minority of cases all of which were non-accredited sites than accredited sites.





(46) %

(65) %

Can input details on… …network provider 65 56 71

…type of contract 58 50 63 …inclusive minutes/texts 45 39 49

…monthly cost 36 20 48 …contract length 30 11 43

…incentives 13 2 20 …additional features (eg insurance) 7 2 11

Can reorder on… …cheapest monthly cost 79 73 86 …handset price 38 33 43

…brand 31 17 46

Can filter the initial list 72 72 83 Can filter on… …inclusive minutes 59 45 68

…inclusive texts 51 42 57 …brand 33 18 43

…handset price 26 21 30

The site includes information about when the prices were last updated

3 0 5

Overall impression of the website was positive 68 63 71



(4) %

(11) %

The website has a complaints policy available online 27 0 36

d. Mobile phones: Where accredited and non-accredited price comparison websites perform equally

There are areas where both accredited and non-accredited sites perform equally. This shared

performance does not always equate to being consumer friendly. For example, neither type of site

consistently provided a contact number for consumers to apply via the telephone as recommended

in the Ofcom accreditation guidelines.

Accreditation made little difference to the accuracy or availability of packages as results were based

on all of the search criteria and in most cases, packages could be found on the supplier site at the

same price as advertised on the comparison website.

On the whole, both accredited and non-accredited sites were easy to use and did not contain

misleading or confusing information.


Table 12S: Areas where accredited sites and non-accredited sites perform equally



(46) %

(65) %

Can input details on… …data allowance 32 35 31

…colour of phone 17 15 18

Yes, list appeared to be based on all of my criteria 96 93 97 The first page included a link to access and edit search

criteria 18 20 17

There were telephone numbers displayed against the available packages

2 2 2

There were additional restrictions/limitations added on to the initial package

33 33 34

Yes, the website contained information that was confusing

22 22 22

Yes, the website contained information that was misleading

16 15 17

It was easy to enter information on the website 82 80 83

I was able to find the same package 63 61 65 Base: All mystery shops where reordering on monthly cost was possible


(83) %

(40) %

(43) %

Yes, all packages were in price order (following reordering)

94 95 93


Base: All mystery shops where the same package could be found (70) %

(28) %

(42) %


83 82 83


12.6 Mobile phones: Big 4 price comparison websites versus others

a. Mobile phones: Overview

All of the Big 4 price comparison websites provide mobile phone-related comparisons. None of these

sites are accredited under the Ofcom accreditation scheme for price comparison calculators.

Comparison between the Big 4 and other price comparison websites in the mobile phone market

reveals mixed performance. Overall, using a comparison from the Big 4 does not mean that the

information will be better or more accurate. There are areas where the Big 4 outperform lesser used

sites including search accuracy and overall impression but equally there are factors which are more

successful on the non-Big 4 sites.

b. Mobile phones: Where Big 4 price comparison websites sites perform better than non-Big 4


majority of mystery shoppers were impressed by the Big 4 websites and felt comfortable that they had enough information to make an application

less likely to include packages that did not match initial search criteria

filtering options were often provided to help consumers shorten and tailor the list of available packages. Commonly found filters included contract length, network provider and brand of handset

less likely to include additional restrictions and/or limitations at any stage following the initial search including when located on the supplier site

three out of four mystery shoppers were able to locate the same package on the supplier site although not always at the same price

Big 4 were generally clearer about how they generate income (67 per cent versus 33 per cent of non-Big 4 sites) with all sites specifying that they earn a fee for each successful application

all of the Big 4 websites publish their complaints policy and allow consumers to issue a complaint online


Table 12T: Areas where Big 4 price comparison websites perform better than non-accredited sites


% (18)

%

(93)

%

Can input details on… …contract length 30 44 27 …features and specification of phone 5 11 4

The initial list appeared to be based on all of my criteria 96 100 95

Reorder on… …most inclusive minutes 71 78 69 …brand 31 56 27

It was possible to filter the initial list 72 83 70 Filter on… …network provider 80 87 78

…contract length 63 73 60 ...brand 33 47 29

…cost of handset (phone) 26 33 25

On the first page of packages, there is a summary of the key features of your chosen handset

28 44 25

There were additional restrictions/limitations (first 5 quotes in initial list)

33 22 35

There were additional costs (first 5 quotes in initial list) 33 22 35


There were additional restrictions/limitations (at application stage)

21 11 23

Overall impression of the website was positive 68 83 65 I was able to find the same package on the supplier site 63 72 61

Yes, the supplier site had additional details 25 8 29 …usage restrictions 41 0 44

…subject to a credit check 24 0 25

c. Mobile phones: Where Big 4 price comparison websites do not perform as well as non-Big 4

The Big 4 price comparison websites do not perform as well as non-Big 4 in the following:

offered a wider range of initial search criteria including brand, model and colour of phone, inclusive minutes/texts and data allowance

non-Big 4 sites were generally clearer about how the initial list was ordered and more flexible and accurate about subsequent reordering or filtering based on a range of criteria including price order

only 20 per cent of non-Big 4 cases were found to contain confusing information versus 28 per cent of Big 4 cases.

14 per cent of non-Big 4 cases contained misleading information versus 28 per cent of Big 4

mystery shoppers were more likely to find the package at the same cost if the initial quotation was sourced from a non-Big 4 site (86 per cent) than if it was found on a Big 4 site (69 per cent)


25 per cent of non-Big 4 sites explained how they calculated the results of the comparison. None of the Big 4 sites were clear in how their results are determined

Provider advertising is more commonly found on the Big 4 sites (67 per cent) than on the non-Big 4 websites (17 per cent)

Table 12U: Areas where Big 4 price comparison websites do not perform better than non-Big 4 sites


% (18)

% (93)

%

Can input details on… …brand of phone 75 50 80 …model of phone 67 61 68

…inclusive minutes/texts 45 33 47 …data allowance 32 22 34

…colour of phone 17 11 18 …incentives 13 6 14

…additional features (insurance) 7 0 9

There was no explanation and it was not obvious how the list had been ordered

29 50 25

Reorder on… …most popular 38 11 43 …cost of handset (phone) 38 22 41

…largest data allowance 55 33 59 …contract length 40 0 41

Filter on… …inclusive minutes 59 53 60 …features and specification of phone 13 7 14

…most popular 4 0 5

On the first page of packages, there is a summary of your search criteria

58 28 63

The package list is not in price order and I cannot reorder it based on the monthly cost

25 39 23

Yes, all pages were in price order (following reordering) 94 82 96

There were additional costs (at application stage) 32 44 29

Yes, the site contained information that was confusing 22 28 20 Yes, the site contained information that was misleading 16 28 14


83 69 86

Total Big 4 Non-Big 4 Base: All websites (15)

% (3) %

(12) %

The site provides an explanation of how it calculates results

20 0 25

Provider advertising is included on the website 27 67 17


d. Mobile phones: Where Big 4 and non-Big 4 price comparison websites perform equally


information about when prices were last updated was missing from both types of sites

some filtering and/or reordering options such as network provider and type of contract were provided consistently by a proportion of non-Big 4 and Big 4 sites

in almost all cases, the initial list reflected all search criteria

Table 12V: Areas where Big 4 price comparison websites perform equally to non-Big 4 price comparison websites


% (18)

% (93)

%

Can input details on... …network provider 65 67 65 …type of contract 58 61 57

…monthly cost 36 39 35

Yes, list appeared to be based on all of my criteria 96 100 97 Yes, there was an explanation that the initial list was

reordered on cheapest monthly cost 22 22 22

No, there was no explanation but I could see that the list was ordered on cheapest monthly cost

23 22 24

It was possible to reorder the initial list 52 50 53 Reorder on… …cheapest monthly cost 79 78 80

…most inclusive texts 67 67 67 Filter on… ….monthly cost 65 67 65

…inclusive texts 51 53 51 …data allowance 35 33 35

…incentives 23 20 23

There was a note about when the prices were last updated

3 0 3


12.7. Mobile phones: Chapter summary

Price comparison websites in the mobile phone market offer a range of search criteria

for consumers to use in tailoring their requirements. These tend to focus initially on the

brand and model of handset. Contract details appeared in over half of cases with

monthly costs available in a third of websites

Overall, comparison websites in this market are viewed as easy to use with a minority of

cases reporting confusing and/or misleading information. Mystery shoppers were

generally more impressed with non-accredited sites and/or the Big 4 sites both of which

were rated positively

There is a lack of easy to find information explaining how the site operates, how results

are calculated and how consumers can complain if their expectations are not met. The

sites are better at providing consumers with additional information such as latest sector

news and/or contact details for external agencies

In most cases, it was easy to identify how the results had been ordered and most sites

offered mystery shoppers the ability to personalise and/or tailor their results through a

range of reordering and filtering options

Additional restrictions and limitations were frequently added following the initial search

results. The most common restriction was minimum length of contract which affected

24 per cent of cases. Refurbished phones were also added to a small proportion of

cases and often resulted in mystery shoppers feeling mislead by the website overall.

Extra costs were found in a third of all cases with cost of the handset being the most

commonly added affecting 12 per cent of cases following the initial search stage. 32 per

cent of applications were subject to additional charges with the majority of these cases

reporting handset insurance as the added cost

Most comparison sites were found to accurately present packages with most mystery

shoppers able to locate the same package on a supplier site although this was easier on

network provider site

Generally, the cost for these packages were found to be the same from comparison to

supplier site with phone manufacturer sites performing better than network provider

websites. Accredited status also slightly improved the likelihood of finding an accurate

price on the comparison website

Accredited sites perform better than non-accredited websites in some areas such as

providing details about how the site generates income, calculates its results and

offering summaries of key information such as search criteria. There were other areas

where non-accredited sites were the top performers such as the overall impression of

the site and accuracy of the package information (excluding costs)

There are some differences between the Big 4 comparison sites and the non-Big 4.

These are mainly connected to the usability of the site such as filtering options and

overall confidence in the site. However, fewer non-Big 4 sites were reported as

including confusing or misleading information which suggests that shoppers’ familiarity

of the Big-4 may play a role


13. Insurance

13.1 Insurance: Number of price comparison websites and mystery shops

The insurance market section of this research consisted of scenarios and searches on both home and car insurance sites. A total of 15 individual price comparison websites were included in the insurance market part of the research. The same sites were visited for both car and home insurance giving us a total sample of 30 websites. Mystery shoppers made a total of 114 site visits to pose scenarios on these sites.

13.2 Insurance: Compliance with Consumer Protection from Unfair Trading Regulations

a. Insurance: Obtaining a list of quotes

Most comparison sites within the insurance market ask for a core set of initial criteria such as name, address, employment and date of birth. Outside of this standard set, the majority of home and car insurance sites asked for a range of specific information about previous claims, type of property, mileage and value of contents or car. Table 13A: Details can enter when starting search Total Base: All mystery shops (114)

%

Can input details on… …name and address 100

…date of birth 100 …employment 100

…phone number 98 …email address 96 …marital status 94

Total Base: All home insurance mystery shops (57)

%

Can input details on… …property details 100 …property tenure 98 …property alarms 98

…type of cover required 95 …policy holder personal details (do you smoke, declared bankrupt) 95

…locks on doors/windows 95 ..whether property is in a neighbourhood watch area 93

...use of property 91 …previous claims 89

...voluntary excess 84 …length of stay at property 82

…detail and value of contents 81 …renewal date 74

…estimated rebuild costs 74 …value of property 70

…existing insurance provider 41


…renewal price 32

Total Base: All car insurance mystery shops (57)

%

Can input details on… …details about driving license 100

…vehicle details 100 …claims and convictions 98

…additional drivers 98 …no claims discount 98

…commencement date 98 ...number of cars in the household 95

…expected mileage 95 …type of cover required 93

…voluntary excess 91 ...preferred payment option 67

…existing insurance provider 40 …renewal price 35

In the majority of cases, it was clear the initial list had been ordered on the annual premium cost.

Mystery shoppers were unsure of the factor used to order the list in just four per cent of cases.

Table 13B: Whether there was an explanation of how initial list had been ordered Total Base: All mystery shops with more than one quote (114)

%

Yes (total) 37 Yes, explanation that was ordered on annual premium 34

Yes, explanation that was ordered on monthly premium 3 No (total) 60

No, but could see it was ordered on annual premium 55 No, but could see it was ordered on monthly premium 3

No, but could see it was ordered on customer ratings 1 No, but could see it was ordered in some other way 1



b. Insurance: Ability to reorder and filter initial list of quotes

Half of all cases allowed mystery shoppers to reorder their results based on a range of options

including annual and monthly premium, voluntary excess amount and provider. Both sub markets

(home and car insurance) also provided options tailored to the additional factors within that market

such as courtesy car, new for old replacement and legal cover. Filters were also available in half of

cases and based on similar factors.


either not in price order initially or could not be reordered on price (four per cent). Amongst the

remaining 96 per cent, the first 2-3 pages of options were checked and in all cases (except one) were

found to be sorted correctly (ie they were, indeed, in order of ascending price).

Table 13C: Reordering options Total Base: All mystery shops (114)

%

Able to reorder 57 Possible to compare a small number of quotes in more detail 48


(64)

%

Can reorder on… ...annual premium 84

…monthly premium 78 …voluntary excess amount 25

…provider 14 Base: All mystery shops where can reorder list (home insurance options only)

(30)

%

Can reorder on… …content excess amount 60

…buildings excess amount 50 …additional policy features (such as new for old) 7

Base: All mystery shops where can reorder list (car insurance options only)

(34)

%

Can reorder on… …courtesy car 44

…legal cover 44 …total cost of monthly instalments 15

…breakdown cover 12 …windscreen cover 9


Table 13D: Filtering options Total Base: All mystery shops (114)

%

Able to filter 58 Base: All mystery shops where can filter list

(65)

%

Can filter on… ...total excess (voluntary/compulsory) 55

…voluntary excess 23 …annual premium 12

…monthly premium 12 …provider 1

Base: All mystery shops where can filter list (home insurance)

(29)

%

Can filter on… …additional policy features (such as new for old) 48

Base: All mystery shops where can filter list (home insurance)

(36)

%

Can filter on… …courtesy car 83

…legal cover 37 …windscreen cover 26 …breakdown cover 23

c. Insurance: Ease of viewing the list of quotes

Most mystery shoppers saw 10 or more quotes on the first page. When they only saw 1-9 this was

often because there were no more to show. In fact, for all of these mystery shops there were no

quotes beyond the first page. Amongst the remainder, there was a mix of being able to view all the

quotes at once and moving forward page by page.

Table 13E: Number of products per page and options for viewing Total Base: All mystery shops (114)

%



There is an option to… …view all quotes 12 …move to next set or page of results 8

…both of these 0


d. Insurance: Accessing a summary of search criteria


page.


%


…a link to access and edit the search criteria 49 …neither of these 25

e. Insurance: Restrictions and limitations not previously mentioned

In order for a consumer to make an informed decision about whether the insurance policy covers

their requirements, it is necessary for the results to provide a detailed list of inclusions, exclusions

and any additional restrictions.


include invalid cover which could have been avoided by a more tailored initial selection by the price

comparison website. Additionally, if a consumer does not fully read the details, or assumes that the

policies shown must be appropriate for them, they could end up trying to apply for an inappropriate

policy.

Due to the comprehensive list of criteria requested as part of the initial search, very few mystery

shops presented additional restrictions or limitations at the results stage.


Table 13G: Quote restriction and limitations Total Base: All mystery shops (114)

%


Additional restrictions/limitations (first 5 quotes in initial list) Limited payment options 2

Must not have made a claim 2 Existing customers/new customers only 1

Base: All mystery shops for home insurance (57) %

Restrictions on type/use of property 4

Did quote chosen have any additional restrictions/limitations Yes 8 No 92

Additional restrictions/limitations (at point of making applications) Must not have made a claim 1

Limited payment options 1 Base: All mystery shops for home insurance (57)

%

Restrictions on type/use of property 5 Restrictions on location 5

Restrictions regarding fencing repair/replacement 2 Restrictions regarding sheds/outbuildings 2

f. Insurance: Additional costs not previously mentioned

Within the insurance market, extra costs could be added in three distinct ways including automatic

charges which could not be removed, automatic charges which consumers can choose to remove

and optional extras which can only be added by the consumer. In the majority of mystery shops, the

latter option applied however there were instances where this was not the case as outlined below.

Charges which cannot be removed are slightly more prevalent within the car insurance market with

windscreen cover (12 per cent) and personal injury insurance (11 per cent) as examples. These costs

only occur in a small proportion of cases but the inability to remove these costs means that

consumers may be purchasing additional cover that they neither wanted nor needed.

In 37 per cent of home insurance shops, emergency home cover was added onto policy costs and

whilst consumers had the option to remove it, this relies heavily on them noticing the additional

charge and understanding how to take it off.


Table 13H: Additional costs: First five quotes

Base: All mystery shops

Not an additional cost

Additional cost that was…

…automatically added and I was unable to remove

…automatically added but I had a choice to remove

…not automatically added, which I could choose to add

Legal cover (114) % 30 11 7 53

Garden cover (57) % 0 2 12 86 Accidental damage (57) % 68 5 4 23 Replacement locks (57) % 0 7 16 77 Alternative accommodation (57) % 0 4 12 84 New for old replacement (57) % 0 4 11 86 Emergency helpline (57) % 89 2 2 7 Emergency home cover (57) % 0 4 37 60 Food in fridge or freezer (57) % 0 4 11 86 Courtesy car (57) % 68 7 5 19 Breakdown cover (57) % 28 7 5 60 Windscreen cover (57) % 61 12 4 23 Personal injury insurance (57) % 69 11 2 19

Table 13I: Additional costs: chosen quotation

Base: All mystery shops






Legal cover (114) % 27 4 18 51

Garden cover (57) % 0 16 84 0 Accidental damage (57) % 0 5 30 65 Replacement locks (57) % 72 2 4 23 Alternative accommodation (57) % 0 5 95 0 New for old replacement (57) % 0 5 2 93 Emergency helpline (57) % 0 4 2 95 Emergency home cover (57) % 47 2 5 46 Food in fridge or freezer (57) % 0 14 86 0 Courtesy car (57) % 69 4 5 23 Breakdown cover (57) % 35 5 5 54 Windscreen cover (57) % 70 9 4 18 Personal injury insurance (57) % 60 2 12 26


g. Insurance: Whether anything on the price comparison website was confusing or misleading


suggest that the majority of their confusion was related to the extra features and/or cover offered:

‘I was confused that the comparison site said that personal accident cover was included, when the Bell [insurance supplier] site said that this was an additional £29.95.’

‘When I initially went to the website, the premium at the top of the page matched what I had been told on the price comparison website. However, the premium at the bottom was higher and it took me a minute or two to work out that was because they'd added legal assistance on, which I then took off.’

‘Although it looked like legal cover was included in the quote, when you held the mouse over it, it said it could be added for an extra cost.’


misleading (94 per cent). Comments from those who thought the opposite also centred on the

additional costs for cover that was not requested as part of the initial search:

‘Not all details are presented on the comparison site, although that is to be expected, as not all details can be provided there. More details on excesses are here rather than the other [comparision] site.’

‘I think that it was misleading to say that personal injury cover was included when,

in fact, there is an additional charge for this.’

h. Insurance: Overall impressions


few with a negative view were mostly concerned the amount of information that needed to be

entered in order to receive a quotation.

‘It took a long time to enter the initial information. Some of the answers I didn't know and had to look up or enquire and come back to the form. This took a long time. Once the results came up I found there was very little policy information in the summary and had to read through the full policy to get specific details.’



more information gave a range of requirements mainly around the details of the policy’s cover:

‘There is little information about the insurance on this website. I would need to see the written policy to read and understand what I am actually covered for.’

‘For policy wording it said check with insurer, and it was the same for the

windscreen excess. So I would like some more info on this.’


Table 13J: Overall impressions Total Base: All mystery shops (114)

%





13.3 Insurance: Accuracy

a. Insurance: Whether package was available on supplier website

In 95 per cent of cases, the same package was found on the supplier website. The minority which

were unable to find the same package offered a range of reasons including being denied insurance

cover:

‘I was advised at the end of filling in numerous questions which did not appear on the original application that the company were 'unable to quote' and suggested to go to confused.com.’

b. Insurance: Cost of package on supplier website compared to cost on price comparison website


to get the same quote if they visited the supplier website. When the package could be found on the

supplier site it was offered at the same price in half of all comparisons. This proportion was similar

across both types of insurance. The other half was a mix of higher and lower prices. The level of

price difference ranged from £1 to £179.


Table 13K: Package costs – price comparison website versus supplier website Total Base: All mystery shops (107)

%

Could find package on supplier site? Yes 95 No 5

Base: All mystery shops where package could be found on supplier site (102)

%



In most of the cases where there was a price difference, there was no obvious reason. The

explanation for the rest (which amounts to a total of three mystery shops) was generally due

additional cover being added.

Table 13L: Reasons for the price difference Total Base: All mystery shops where package could be found on supplier site and cost was different

(51) %


c. Insurance: Whether the supplier site had additional details


comparison website. Where additional information was found, it generally about additional cover or

extra policy features such as new for old replacement or breakdown cover.

Table 13M: Whether supplier site had additional details Total Base: All mystery shops where package could be found on supplier site (102)

%


What were the additional details? Inclusive extras (new for old, breakdown cover) 47

Amount of excess (voluntary/compulsory) 37 Use of car (eg social only) 27 Limited payment options 11


13.4 Insurance: Transparency

a. Insurance: Information provided

The price comparison websites covering the insurance market were sometimes lacking when it came

to providing various types of information. Of the 30 price comparison websites looked at:


just seven per cent had a Code of Conduct on the website and/or provided a glossary of the technical terms used on the site

only seven per cent included ratings for the providers and products that were included on the site. However both of these sites were clear about how these ratings had been calculated and included customer generated information such as a reviews

only seven per cent of sites explained how they calculated the results of a comparison

only 20 per cent made any reference to how frequently the websites update their prices

a glossary of terms was only present on 27 per cent of sites but generally included a good range of terminology where it was available

none of the sites who stated that providers pay a fee to be included in the result actually named these providers

Moving into areas where transparency was better, and a majority of the 30 price comparison

websites provided information:

87 per cent of sites signposted consumers to more relevant details such as latest sector news, advice for insurance for specific groups and/or needs (57 per cent) and/or contact details for external agencies (40 per cent). This information was easy to find in 83 per cent of cases

70 per cent of sites explained how they generate income

60 per cent identified the number of providers compared by the comparison website and in most cases this information was easy to find

a complaints policy is available on 60 per cent of websites, 83 per cent of which allow consumers to submit complaints online

In most cases, the first page of quotes reached by the mystery shoppers did not include any

information about when prices were last updated. A minority of mystery shoppers were able to find

a note about when prices are updated, although this was not always clearly displayed. Details about

insurance premium tax were also missing from the majority of websites. Almost as many sites stated

that their quotations excluded the insurance premium tax as included it.


Table 13N: Information about price updating Total Base: All mystery shops (114)

%


No, there was not a note about when prices were last updated 96

Note on first page of quotes about insurance premium tax being included? Yes, and it was clearly displayed 10

Yes, although it wasn’t very clearly displayed 4 No, there was not a note about insurance premium tax being included 78

No, there was a note about insurance premium tax being excluded 9

b. Insurance: Details of the information provided

In most cases, sites with a complaints policy available from their website generally also allowed

consumers to submit their complaint online. Half of these also specified the number of days it would

take before a response was received.


successful switch. Advertising sales was used to generate income in 14 per cent of comparison sites

and an additional 14 per cent of sites were paid directly by providers wishing to be included in the

searches but names were not specified.


all websites. Over half of websites provided consumers with advice aimed at specific groups/needs,

and 40 per cent gave contact details for external agencies - eg British Insurance Broker’s Association

(BIBA).


Table 13O: Operational information Total Base: All websites with a complaints policy (18)

%



Base: All websites with information about income generation (21)

%


From advertising sales 14 Provider pay to appear on the site 14

Comparison site paid per click through to supplier site 10 Base: All websites providing of signposting additional information (30)

%

Provide or signpost information about… …latest sector news 87

…advice about insurance for specific groups and/or needs 57 …contact information for external agencies 40

c. Insurance: Complaints

At the point of choosing a policy from a list of suitable options, only one per cent of shoppers found

any information about whether the policy provider has received any formal complaints from

consumers.

13.5 Insurance: Accreditation

a. Insurance: Overview

One of the comparison websites operating in the insurance market is accredited by the BIBA for both

home and car insurance. This is the only formal accreditation used for the purposes of this research.

Table 13P: Accreditations Base: All websites Total

(30) %

Whether accredited or not British Insurance Brokers Association (BIBA) 7

Not accredited 93 Other memberships/protections

Financial Services Authority (FSA) authorised & regulated 87 OFT licenced or registered 7


On some parameters the BIBA accredited website was more consumer-friendly than the non-

accredited ones, but in many other areas it was the non-accredited websites that outperformed

those with an accreditation. The variability is well illustrated in the following table showing

additional costs.

Table 13Q: Additional costs: initial list Base: All mystery shops






A=accredited, NA=non-accredited A NA A NA A NA A NA Legal cover A (12)

NA (102) % 33 29 0 12 17 6 50 53

Garden cover A (6) NA (51)

% 0 0 17 0 33 10 50 90

Accidental damage A (6) NA (51)

% 33 73 0 6 0 4 67 18

Replacement locks A (6) NA (51)

% 0 0 0 8 17 16 83 76

Alternative accommodation

A (6) NA (51)

% 0 0 0 4 17 12 83 84

New for old replacement

A (6) NA (51)

% 0 0 0 4 17 10 83 86

Emergency helpline A (6) NA (51)

% 83 90 0 2 0 2 17 6

Emergency home cover

A (6) NA (51)

% 0 0 0 4 33 37 67 59

Food in fridge or freezer

A (6) NA (51)

% 0 0 0 4 17 10 83 86

Courtesy car A (6) NA (51)

% 67 69 17 6 0 6 17 20

Breakdown cover A (6) NA (51)

% 33 27 0 8 0 6 67 59

Windscreen cover A (6) NA (51)

% 67 61 17 12 0 4 17 24

Personal injury insurance

A (6) NA (51)

% 67 69 17 10 0 2 17 20


Table 13R: Additional costs: chosen quotation Base: All mystery shops






A=accredited, NA=non-accredited A NA A NA A NA A NA Legal cover A (12)

NA (102) % 25 27 8 4 25 17 42 52

Garden cover A (6) NA (51)

% 0 0 33 13 67 86 0 0

Accidental damage A (6) NA (51)

% 0 0 17 4 50 27 33 69

Replacement locks A (6) NA (51)

% 67 73 0 2 0 4 33 22


A (6) NA (51)

% 0 0 0 6 100 94 0 0


A (6) NA (51)

% 0 0 17 4 0 2 83 94

Emergency helpline A (6) NA (51)

% 0 0 0 4 0 2 100 94

Emergency home cover

A (6) NA (51)

% 67 45 0 2 17 4 17 49


A (6) NA (51)

% 0 0 33 12 67 88 0 0

Courtesy car A (6) NA (51)

% 83 67 17 2 0 6 0 25

Breakdown cover A (6) NA (51)

% 17 37 17 4 0 6 67 53

Windscreen cover A (6) NA (51)

% 83 69 17 8 0 4 0 20


A (6) NA (51)

% 50 61 17 0 0 14 33 25

b. Insurance: Where accredited price comparison websites sites perform better than those that are not accredited

Searches using an accredited site generally requested a wider range of details than non-accredited

sites.


Table 13S: Initial search criteria


Base: All home insurance mystery shops (57) %

(6) %

(51) %

Can input details on… …locks on doors/windows 95 100 94

…details about use of property 91 100 90 …made any previous claims 89 100 88 …length of stay at property 82 100 80

…estimated rebuild costs 74 83 73 ...renewal date 74 83 73

…value of property 70 83 69 …renewal price 32 67 27


Base: All car insurance mystery shops (57) %

(6) %

(51) %

Can input details on… …number of cars in the household 95 100 94

…type of insurance 93 100 92 …preferred payment option 67 83 65

…renewal price 35 67 31

Accredited sites were more likely to explain how the original list was ordered with the most common

factor being annual premium. Non-accredited sites also used this factor but were less likely to be

explicit about its use.

Reordering and filtering on annual and/or monthly premium was more commonly found on the

accredited than on non-accredited websites. The availability of this function allowed mystery

shoppers on accredited sites to easily focus their results on the cheapest price.


Table 13T: Areas where accredited sites performed better than non-accredited sites



(12) %

(102) %

There was an explanation that the list was ordered on annual premium

33 67 29

There was no explanation but I could see that the list was order on annual premium

54 25 58

It was possible to reorder the original list 57 67 56 Reorder on… …annual premium 84 100 82

…monthly premium 78 88 77 Filter on… …annual premium 12 40 10

…monthly premium 12 40 10

It was possible to compare a small number of quotes in detail

48 83 44

Yes, there is a telephone number clearly displayed against the various quotes

20 33 19

Yes, there are additional restrictions/limitations (first five quotes)

6 0 7

Yes, I have sufficient information to make an application 80 92 78 Yes, the site contained information that was misleading 6 0 7


Base: All mystery shops which could find the same policy on the supplier site

(102) %

(10) %

(92) %

The supplier cost was exactly the same as the price comparison site

50 80 47


c. Insurance: Where accredited price comparison websites do not perform as well as non-accredited sites

Some search criteria such as the type of home insurance required were more likely to appear on

non-accredited sites than accredited sites. Most of the essential information was collected by all

sites regardless of whether they were accredited or not.

Table 13U: Areas where accredited sites do not perform as well as non-accredited sites


Base: All home insurance mystery shops (57) %

(6) %

(51) %

Can input details on… …type of cover required 95 83 96

…policy holder details (do you smoke, declared bankrupt)

95 83 96

…whether the property is in a neighbourhood watch area

93 83 94


Base: All car insurance mystery shops (57) %

(6) %

(51) %

Can input details on… …additional drivers 98 83 100 ...expected mileage 95 67 98

…voluntary excess amount 91 83 92 …existing insurance provider 40 17 43

Following the initial results, non-accredited sites general presented the list in price order although

this was not always explained to mystery shoppers. In 60 per cent of cases, filters were offered to

help shorten and tailor the list of results on non-accredited sites. The most commonly available

filters included total excess as well as a range of extra features such as courtesy car and legal cover.

Policies found on non-accredited sites were more likely to be available directly from the supplier site

than those sourced from accredited sites. However, 53 per cent of these policies were priced

differently to the quote shown on the supplier site versus 20 per cent of prices quoted on accredited

websites.


Table 13V: Areas where accredited sites do not perform as well as non-accredited sites



(12) %

(102) %

It was possible to filter the original list 58 42 60 Filter on… …total excess (voluntary/compulsory) 55 0 60

…courtesy car 83 50 81 …legal cover 37 50 64

Option to view all quotes 12 8 13 I was able to find the same policy 95 83 97


Base: All mystery shops where the same policy could be found on the supplier site

(102) %

(10) %

(92) %

The supplier site cost was higher/lower than the price comparison site cost

50 20 53


d. Insurance: Where accredited and non-accredited price comparison websites perform equally

A BIBA accreditation made no difference to whether or not the initial list of quotes was based on all

the criteria input by shoppers, or whether it was possible to get the list into price order.

Not all of the factors where the sites performed similarly were consumer friendly. For example,

neither type of site clearly displayed information about when prices were last updated.

Table 13W: Areas where accredited and non-accredited sites perform equally



(12) %

(96) %

Can input details on… …name and address 100 100 100

…date of birth 100 100 100 ...employment 100 100 100

…phone number 98 100 98 …email address 96 100 96 …marital status 94 92 94

Yes, the list appeared to be based on all of my criteria 89 92 89

On the first page of packages, is there…

There was a note about when the prices were last updated

4 8 4

A summary of the search criteria 33 33 33

There were additional restrictions/limitations at the application stage

8 8 8

The site contained information that was confusing 11 8 11 It was easy to enter information on the site 81 83 80 Overall impression of the site was positive 76 75 76


13.6 Insurance: Big 4 price comparison websites versus others

a. Insurance: Overview

All of the Big 4 price comparison websites provide insurance-related comparisons.

Comparing the Big 4 against other price comparison websites in the insurance market reveals a

mixed picture. Several areas where the Big 4 perform better than non-Big 4 sites would improve the

consumer’s ability to integrate and/or alter their results. There is also variation between the sites in

terms of the type of information requested as part of the initial search and the extra costs included

in a policy without approval. In these areas, Big 4 sites are not always more consumer friendly than

non-Big 4 sites. Similar performance was reported in areas such as the amount of confusing and

misleading information included on the site.

b. Insurance: Where Big 4 price comparison websites sites perform better than non-Big 4


certain search criteria are requested from Big 4 sites more regularly than non-Big 4 sites

more likely to allow users to reorder and/or filter the initial list although the range of options available was more extensive on non-Big 4 sites

comparison of a smaller number of quotes was offered in twice as many cases on Big 4 sites than on the non-Big 4

options to view the policy’s summary information and/or edit this information

a provider’s telephone number was provided in a third of cases on Big 4 sites versus 11 per cent of cases on non-Big 4 websites

a Big 4 site was the only site to offer information about whether providers had received any formal complaints

confidence in making an application and the general impression of the website was higher on the Big 4 in comparison to the other sites

more quotes from Big 4 sites were found to be priced the same on supplier sites

Table 13X: Initial search criteria

Total Big 4 Non-Big 4 Base: All home insurance mystery shops (57)

% (21)

% (36)

%

Can input details on… …use of property 91 100 86 …previous claims 89 95 86

…length of stay at the property 82 90 78 Total Big 4 Non-Big 4 Base: All car insurance mystery shops (57)

% (21)

% (36)

%

…number of cars in the household 95 100 92 …preferred payment option 67 86 56

…renewal price 35 43 31


Table 13Y: Additional costs: initial list Base: All mystery shops






B=Big 4, NB=non-Big 4 B NB B NB B NB B NB Legal cover B (42)

NB (72) % 31 29 10 11 5 8 55 51

Garden cover B (21) NB (36)

% 0 0 5 0 19 8 76 92

Accidental damage B (21) NB (36)

% 67 69 0 8 0 6 33 17

Replacement locks B (21) NB (36)

% 0 0 10 6 10 19 81 75


B (21) NB (36)

% 0 0 5 3 10 14 86 83


B (21) NB (36)

% 0 0 10 0 10 11 81 89

Emergency helpline B (21) NB (36)

% 81 94 5 0 0 3 14 3

Emergency home cover B (21) NB (36)

% 0 0 5 3 29 42 67 56

Food in fridge or freezer B (21) NB (36)

% 0 0 5 3 14 8 81 89

Courtesy car B (21) NB (36)

% 71 67 5 8 0 8 24 17

Breakdown cover B (21) NB (36)

% 14 36 5 8 0 8 81 47

Windscreen cover B (21) NB (36)

% 67 58 10 14 0 5 24 22


B (21) NB (36)

% 61 72 10 11 0 3 29 14


Table 13Z: Areas where Big 4 price comparison websites perform better than non-accredited sites


% (42)

% (72)

%

Yes, the list appeared to be based on all of my criteria 89 93 88

It was possible to reorder the initial list 57 62 54 Reorder on… …annual premium 84 92 79

…monthly premium 78 85 74

It was possible to filter the initial list 58 64 54 Filter on… …breakdown cover

…windscreen cover It was possible to compare a small number of quotes in

detail 48 79 30

On the first page of packages, there was a summary of the search criteria

33 38 31

On the first page of packages, there was a link to access and edit the search criteria

49 57 44

There is an option to view all packages 12 26 4 Yes, there is a telephone number clearly displayed

against the various quotes 20 36 11

There are additional restrictions and/or limitations (initial 5 quotes)

6 2 8

Yes, I have sufficient information to make an application 80 88 75 Overall impression of the website was positive 76 83 72


50 59 44

c. Insurance: Where Big 4 price comparison websites do not perform as well as non-Big 4


asking about the type of insurance cover required ahead of a full search. By providing this information initially, consumers may feel more confident that any results will focus on the type of cover that they need

non-Big 4 sites offered more reordering options in comparison to Big 4 sites and were more likely to provide filters for both annual and monthly premiums

displaying information about insurance premium tax


Table 13AA: Initial search criteria


% (21)

% (36)

%

Can input details on… …type of cover required 95 90 97

…policy holder details (do you smoke, declared bankrupt)

95 90 97

…renewal date 74 67 78 …value of the property 70 67 72

…existing insurance provider 41 24 53 Total Big 4 Non-Big 4 Base: All car insurance mystery shops (57)

% (21)

% (36)

%

…expected mileage 95 90 97 …type of cover required 93 81 100

…voluntary excess amount 91 81 97 …existing insurance provider 40 14 56

Table 13AB: Reordering and filtering options

Total Big 4 Non-Big 4 Base: All home insurance mystery shops where reordering was possible

(30) %

(12) %

(18) %

Reorder on… …contents excess amount 60 33 78 …buildings excess amount 50 25 67

Total Big 4 Non-Big 4 Base: All car insurance mystery shops where reordering was possible

(34) %

(20) %

(14) %

Reorder on… …courtesy car 55 29 55 …legal cover 44 21 60

Total Big 4 Non-Big 4 Base: All mystery shops where filtering was possible (65)

% (27)

% (38)

%

Filter on… …annual premium 12 7 16 …monthly premium 12 7 16

Table 13AC: Areas where Big 4 site do not perform as well as non-Big 4 websites


% (42)

% (72)

%

Yes, there is a clearly displayed note about insurance premium tax

10 0 15


d. Insurance: Where Big 4 and non-Big 4 price comparison websites perform equally

The Big 4 and non-Big 4 price comparison websites perform equally in the majority of areas:

asking consumers to input a set of standard personal details including employment status, name, address and date of birth before the initial search

the majority of additional criteria are regularly requested across both Big 4 and non-Big 4 comparison sites

the ability and accuracy of reordering options based on annual premiums

neither Big 4 nor non-Big 4 sites provided clear information about when their prices were last updated

initial ordering is generally based on annual premium and the amount of sites who explain the ordering to consumers is consistent across both types of sites

using a Big 4 website is as easy as using another comparison site. The amount of misleading and confusing information is generally the same across all sites

being able to find the same policy on the provider site

Table 13AD: Areas where Big 4 site perform as well as non-Big 4 websites


% (42)

% (72)

%

Can input details on… …name and address 100 100 100

…date of birth 100 100 100 …employment 100 100 100

…phone number 98 98 99 …marital status 94 93 94


% (21)

% (36)

%

Can input details on… …property details 100 100 100 …property alarms 98 100 97 …property tenure 98 100 97

…locks on windows/doors 95 95 94 …whether the property is in a neighbourhood watch

area 93 91 94

… details and value of contents 81 81 81

Total Big 4 Non-Big 4 Base: All car insurance mystery shops (57)

% (21)

% (36)

%

Can input details on… …details of the vehicle 100 100 100

…driving licence details 100 100 100 …no claims discount 98 100 97

…commencement date 98 100 97 …claims and convictions 98 95 100

…additional drivers 98 95 100


Table 13AE: Areas where Big 4 price comparison websites and

non-Big 4 sites perform equally


% (42)

% (72)

%

Yes, there was an explanation that the list was ordered on annual premium

33 33 33

No, there was not an explanation but it was obvious that the list was order on annual premium

54 54 54

There was a note about when prices were last updated 4 2 6 Yes, all pages are in price order 99 98 100

There are additional restrictions and/or limitations at the application stage

8 5 9

Yes, the site contained information that was confusing 11 10 11 Yes, the site contained information that was misleading 6 5 7

It was easy to enter information on the website 81 83 79

I was able to find the same package on the supplier site 95 95 95 The supplier site does have additional details 19 18 19


13.7 Insurance: Chapter summary

Price comparison sites in the insurance market asked for an extensive range of search

criteria as part of the initial search process. Although there is some variation across the

different sites and within the two sub markets of car and home insurance, most sites

followed the same list of criteria and details including name, address, details of the

property or vehicle and type of cover

Overall sites were found to be easy to use and generally did not include confusing

and/or misleading information. Where confusion did arise it was almost always linked

to the additional cover and charges that this may incur

The majority of sites ordered the results on annual premium costs with only four per

cent of cases unclear of the factor used to initially order the results

Only a small proportion of policies returned following the initial search were found to

include additional restrictions or limitations which included limited payment options,

claim history and specifying whether mystery shoppers were existing or new customers.

In comparison, additional costs were a slightly more prevalent issue within the

insurance market and more specifically found within the car insurance market. Where

charges could not be removed, the most frequently found cost was for windscreen

cover (12 per cent) and personal injury insurance (11 per cent). Emergency home cover

was included in 37 per cent of cases where costs were added but could be removed by

the consumer

Almost all policies returned following the initial search were based on all the criteria

specified but half of all cases identified a price difference between the comparison site

and the supplier site. The reason for a price difference was rarely explicit but in the

small minority that were able to provide reasons highlighted additional cover such as

breakdown service as the cause of the discrepancy

General information about the website and how it operates was lacking on most sites.

Only seven per cent explained how they calculated results and a further seven per cent

published a code of conduct

Information about how the site generates income was easier to find and was located on

70 per cent of sites. A complaints policy was available on 60 per cent of websites with

83 per cent allowing consumer to submit their complaint online

Links to the latest sector news was actively provided by the majority of websites

Comparisons between accredited and non-accredited sites revealed a mixed picture

with some variation in the type and frequency of additional costs added on to the

policy.

Generally accredited sites are more likely to offer a wide range of search criteria,

filtering and reordering options as well as providing a clear explanation about the initial

ordering of results

Price comparison sites in the insurance market asked for an extensive range of search

criteria as part of the initial search process. Although there is some variation across the

different sites and within the two sub markets of car and home insurance, most sites

followed the same list of criteria and details including name, address, details of the

property or vehicle and type of cover

Overall sites were found to be easy to use and generally did not include confusing

and/or misleading information. Where confusion did arise it was almost always linked

to the additional cover and charges that this may incur

The majority of sites ordered the results on annual premium costs with only four per

cent of cases unclear of the factor used to initially order the results

Only a small proportion of policies returned following the initial search were found to

include additional restrictions or limitations which included limited payment options,

claim history and specifying whether mystery shoppers were existing or new customers

In comparison, additional costs were a slightly more prevalent issue within the

insurance market and more specifically found within the car insurance market. Where

charges could not be removed, the most frequently found cost was for windscreen

cover (12 per cent) and personal injury insurance (11 per cent). Emergency home cover

was included in 37 per cent of cases where costs were added but could be removed by

the consumer

Almost all policies returned following the initial search were based on all the criteria

specified but half of all cases identified a price difference between the comparison site

and the supplier site. The reason for a price difference was rarely explicit but in the

small minority that were able to provide reasons highlighted additional cover such as

breakdown service as the cause of the discrepancy

General information about the website and how it operates was lacking on most sites.

Only seven per cent explained how they calculated results and a further seven per cent

published a code of conduct

Information about how the site generates income was easier to find and was located on

70 per cent of sites. A complaints policy was available on 60 per cent of websites with

83 per cent allowing consumer to submit their complaint online

Links to the latest sector news was actively provided by the majority of websites

Comparisons between accredited and non-accredited sites revealed a mixed picture

with some variation in the type and frequency of additional costs added on to the

policy. Although BIBA accredited status made little difference to the usability or overall

impression of the website

Generally accredited sites are more likely to offer a wide range of search criteria,

filtering and reordering options as well as providing a clear explanation about the initial

ordering of results. Search results were more likely to be found on the supplier site if

the initial quote came from a non-accredited rather than an accredited website


BIBA accreditation made no difference to the accuracy of search results or the overall

impression of the site

Using a Big 4 website did not mean that results would be more accurate or easier to

understand

Big 4 websites were found to be more appealing and mystery shoppers generally felt

more comfortable that they had the right level of information when comparing results

on a Big 4 website than a non-Big 4 site

There were areas where non-Big 4 sites outperformed Big 4 sites such as filtering

options and providing information about insurance premium tax

The cost of this policy was generally found to be more accurate if originally sourced

from an accredited website

Using a Big 4 website did not mean that results would be more accurate or easier to

understand

Big 4 websites were found to be more appealing and mystery shoppers generally felt

more comfortable that they had the right level of information when comparing results

on a Big 4 website than a non-Big 4 site

There were areas where non-Big 4 sites outperformed Big 4 sites such as filtering

options and providing information about insurance premium tax


14. Package holidays

14.1 Package holidays: Number of price comparison websites and mystery shops

A total of 13 price comparison websites were included in the package holiday market part of the

research. Mystery shoppers made 77 site visits to pose scenarios on these sites.

14.2 Package holidays: Compliance with Consumer Protection from Unfair Trading Regulations

a. Package holidays: Obtaining a list of quotes

As the starting point for obtaining a list of package holidays, mystery shoppers usually had the

option of specifying a destination, trip duration (or dates of departure and return), preferred

departure airport and how many people would be in the party. Beyond these basics, some of the

price comparison websites offered further refinements in the form of board type, minimum

accommodation rating and room type. Being able to specify these at the outset is advantageous, as

it means the list of holidays produced will be more tailored to each individual consumer’s needs and

preferences.

Most shoppers (97 per cent) felt that the list of holidays was based on all of the criteria they input, ie

only 3 per cent were shown holidays that ignored their specifications for one or more of destination,

duration, etc.

Table 14A: Details can enter when starting search and whether list shown takes all of these into account Total Base: All mystery shops (77)

%

Can input details on… …destination 100

…departure date 97 …number of adults 96 …departure airport 94

…number of children 87 …duration 73

…return date 61 …number of rooms 48

…board type (eg all inclusive, half board) 47 …minimum accommodation rating 39

…ages of children 38 …flexibility (eg +/- 3 days) 13

…room type 10

Was the initial list based on all the criteria input? Yes 97 No 3

Shoppers were equally likely to have a list that they were told was ranked in a particular way as to

have a list that had no explanation of the ranking system used. Although many lists were ordered on

price (71 per cent) a variety of other variables were used, such as relevance and popularity.



%

Yes (total) 49 Yes, explanation that was ordered on price 29


No, but could see it was ordered on price 42 No, but could see it was ordered some other way 1


b. Package holidays: Ability to reorder and filter initial list of quotes

Many lists could be reordered, and even more could be filtered in some way. Ideally, these tools

should always be available, as they are helpful in terms of refining the list, making comparisons

between holidays of the same type, and selecting the one which most closely meets the consumer’s

needs.

There was a range of reordering and filtering options, centred mainly on price and accommodation

star rating. Being able to sort or filter on the name of the resort or hotel was also fairly

commonplace. Some shoppers could reorder on customer ratings, and some could filter on

resort/hotel facilities.


%


Base: Can reorder list (48) %

Base: Can filter list (57) %

Can reorder on… Can filter on… …price 94 …accommodation star rating 75

…accommodation star rating 58 …board type 61 …hotel/apartment/resort name 44 …price 58

…customer ratings 21 …resort/location 58 …popularity/bestsellers/website’s

favourites 17 …hotel name

…departure airport 36 35

…aircraft departure time 13 …facilities (pool, laundry, kids club) 21 …departure airport 6 …room type 19


Most holiday lists were either in price order from the outset, or could be reordered on price.

Checking through the first 2-3 pages of quotes revealed that most of them were, indeed, in order of

ascending cost; only seven per cent of shoppers found their list had some or all quotes out of order.

Table 14D: Price order Total Base: All mystery shops (77)

%

List in price order (either initially or by reordering)? Yes 95 No 5

Base: All mystery shops where could get list into price order (73) %

Whether list was really in price order Yes 93 No 7

c. Package holidays: Ease of viewing the list of quotes

In terms of supplying sufficient choice, most shoppers had two or more pages of quotes, and there

were usually 10+ holidays on each.

The most common option for viewing the holidays was page by page, rather than seeing all of them

at once.

Table 14E: Number of holidays per page and options for viewing Total Base: All mystery shops (77)

%

Number of holidays per page 1-9 16 10+ 84


Base: All mystery shops with more than one page of quotes (56) %

There is an option to… …view all quotes 9 … move to next set or page of results 95

…both of these 4


A summary of the search criteria was shown on the first page of quotes in a majority of cases,

enabling the consumer to remind themselves, if necessary, of the criteria they used at the start of

their search.


%


…a link to access a summary of the search criteria 5 …neither of these 10

Shoppers were instructed to select the nth holiday from the list and to proceed with booking it to

the point of entering payment details.

Holiday costs were three times more likely to be quoted as ‘per person’ than in total, which could

potentially lead to confusion. On a few occasions the cost was provided in both ways. There was also

a fairly high incidence of costs given as ‘from £x’, implying further refinements and choices had to be

made before a final price was reached.

Table 14G: How cost is quoted Total Base: All mystery shops (77)

%

Cost is quoted as… …per person/adult 64 …in total 21

…both per person and in total 12 …other (eg per room, per night) 4

Whether cost is ‘from £x’ Yes 39 No 61

d. Package holidays: Restrictions and limitations not previously mentioned

Very few of the holidays had restrictions or limitations that appeared at some point during the

booking process. The four such instances found involved a no pets rule, sharing a bathroom in the

accommodation, a requirement to use an e-ticket, and a luggage limit of 15kg.

Table 14H: Quote restriction and limitations Total Base: All mystery shops (77)

%

Did selected holiday have additional restrictions/limitations? Yes 5 No 95


e. Package holidays: Additional costs

A variety of additional costs appeared as shoppers moved through the booking process; only 16 per

cent of the shoppers did not encounter any at all. However, where additional costs appeared they

were more often optional as opposed to mandatory. The main exceptions to this were departure

taxes/local fees and supplier failure protection. Although the first is unavoidable, the second should

surely be discretionary, and not a cost that is added automatically and cannot be removed. There is

also an argument for showing departure taxes/local fees in the initial quote; these can be quite

substantial and later addition might increase the holiday cost beyond the consumer’s budget.

Other automatic additions that could not be taken out included hold luggage and a fee for paying by

credit card, although neither arose for very many of the shoppers.

The most prevalent optional additional costs were travel insurance, airport parking and transfers.

Others included hotel/board upgrades and hold baggage. In most cases shoppers were given a

choice of whether or not to add the cost, but in some cases the cost was added automatically and

had to be removed by the shopper if they did not want it. This was particularly true of airport

transfers and travel insurance. In these situations there is a risk that the consumer does not notice

the additions, and ends up paying for something that they either have already or do not want.

Although 17 per cent of the mystery shops used a scenario where the traveller was holidaying on

their own, a single person supplement never arose as an additional charge.

Table 14I: Additional costs Base: All mystery shops






Single person supplement (77) % 100 0 0 0 Supplier failure protection (77) % 77 17 1 5 Departure taxes/local fees (77) % 77 21 1 1 Hotel/board upgrades (77) % 70 0 1 29 Cost of hold luggage/bags (77) % 68 6 6 19 Transfers to/from airport (77) % 61 0 13 26 Airport parking (77) % 61 0 0 39 Travel insurance (77) % 39 1 21 39 Base: All mystery shops (77)

%

Were any other costs added automatically that could not be removed? Yes 12 No 88

What were these? Charge for credit card payment 4 5kg additional luggage allowance per person 3

Extra cost for half board 1 ATOL Trust protection fee 1


f. Package holidays: Whether anything on the price comparison website was confusing or misleading

Although most shoppers felt there was nothing on the price comparison website that was confusing

or misleading, a significant minority felt there was. The reasons given often focused on costs:

‘It gives a price per person on the home page, then when you click into the price (which does not say 'from £') it gives higher costs when listing the hotels. It should not tell you the cost is £562, if it then goes up to £773 per person’

‘The headline price for hotel rooms was quoted for single rooms, which was not entirely obvious’

‘The price it showed on the initial search results was only the flight price, not the total price including the hotel – this wasn't added until the next page where you selected your hotel. It wasn't clear on the first page with the flight price whether it was per person or the total price’ ‘The “from” prices seemed rather misleading to me. In reality the holiday prices were a lot higher’ ‘The prices changed between the search results table and the booking page, apparently due to availability changes’ ‘Cheap flight cost, but it did not state the baggage allowance charge, which made the price not cheap compared to other destinations and hotels’ ‘I chose an all inclusive holiday. I would have thought that it included transfers, but it didn't’

Table 14J: Whether anything on website was confusing or misleading Total Base: All mystery shops (77)

%

Any on website confusing? Yes 23 No 77

Anything on website misleading? Yes 18 No 82


g. Package holidays: Overall impressions

Impressions of the price comparison websites were largely positive, and most shoppers felt they had

sufficient information to feel comfortable booking the holiday.

‘All the information I needed was clearly given to me. I could amend my booking before confirming it and booking, so I knew what features I was getting and the exact cost’ ‘The website was easy to use, and it produced a wide range of accurate and relevant search results. The costs were clearly explained, and there were no surprises added on at the final payment stage’ ‘This website is easy to navigate and well laid out. Easy to follow instructions, with no hidden extras to give you a nasty surprise at the total stage’ ‘I liked the amount of information available, and the fact that you can see the best deal is by being searched for on more than one website’ ‘I quite enjoyed searching for my holiday on this website. The initial search options were varied enough to take my details, and the results were displayed nicely. Pricing was nice and clear’ ‘The site was easy to read and follow. My search came up with a number of holiday options, which were easy to read, filter and compare’

Some shoppers wanted more information before making a booking, most commonly clarification as

to whether or not transfers were included, or details about the resort/hotel:

‘It states that transfers are not included unless shown, but then does not give you the option to add these, so I am now unsure if they are there or not’ ‘More information on accommodation – pictures, reviews, map, facilities etc’

Table 14K: Overall impressions Total Base: All mystery shops (77)

%

Is your overall impression of this website… …very positive 29 …quite positive 40

…neither positive nor negative 19 …quite negative 9 …very negative 3

Do you at this stage have sufficient information to make a booking? Yes 79 No 21


14.3 Package holidays: Transparency

a. Package holidays: Information provided

The price comparison websites covering the package holiday market were often lacking when it

came to providing various types of information. In most cases a minority provided consumers with

information about how they operate. Of the 13 price comparison websites looked at:

none explained how they calculated the results of a comparison

only 8 per cent had a Code of Conduct on the website

only 8 per cent provided a glossary of the terms or criteria used

only 8 per cent showed a policy for dealing with rogue or fraudulent providers, highlighted how they would respond to reports of such activity, and provided consumers with a direct means of bringing it to their attention

only 8 per cent gave an explanation about why some suppliers or holidays might not be included

only 15 per cent were clear about how they generated income

only 15 per cent said how frequently they updated their prices, and only 6 per cent of mystery shoppers saw price update details on the first page of quotes

only 38 per cent identified how many providers they compared

only 38 per cent of sites with a complaint process or policy specified how long it would take to respond

amongst the 62 per cent that included provider or product ratings, only 50 per cent explained how these were calculated

Table 14L: Information about price updating Total Base: All mystery shops (77)

%


Yes, although it was not very clearly displayed 1 No, there was not a note 94

Moving into areas where transparency was better, and a majority of the 18 price comparison websites provided information:

62 per cent published details of their complaint process or policy, although for 50 per cent this was difficult to find

100 per cent signposted consumers to other sources of information


b. Package holidays: Details of the information provided

The price comparison websites with a complaint policy did not always allow the consumer to submit

a complaint online. A minority specified how many days it would be before the complainant received

a reply.

Half of the websites included provider advertising, which could be situated anywhere from the

homepage to the results page.

The sites were very likely to provide destination information (or signpost where this could be

obtained) and often gave contact details for external agencies such as Association of British Travel

Agents (ABTA) and Air Travel Organisers’ Licencing (ATOL). Relevant news items and travel advice

also featured on some sites.

Amongst those who gave provider/product ratings, these often included customer reviews or scores.

Only two sites said how they generated income; for one this was per ‘click through’ and for the other

it was per successful consumer purchase.

Table 14M: Operational information Total Base: All websites with a complaints policy (8)

%


Unsure 12 Specify number of days before receive a response?

Yes 37 No 62

Base: All websites providing or signposting additional information

(13)

%

Provide or signpost information about… …visitor information for destination 77

…contact details for external agencies (eg ABTA, ATOL) 62 …latest sector news 54

…travel advice for specific groups (eg minors alone) 31

Base: All websites providing income generation information

How income is generated

(2) %

Paid per 'click through' to a providers site 50 Paid on the completion of a successful application/purchase by the customer 50

Base: All websites price update information

How often prices are updated

(2) %

More than daily 50 Daily 50

Base: All websites

(13)

%


Whether site includes advertising for provider sites Yes 46 No 54

Base: All websites with provider site advertising

(6) %

Where the provider advertising appears Homepage 83

On criteria pages (where enter details and requirements) 83 On results pages 83

On 'affiliates' page 17 Base: All websites with provider/product ratings

Whether provider/product ratings include customer reviews/scores

(8) %

Yes 63 No 25

Not sure 12

14.4 Package holidays: Accreditation

a. Package holidays: Overview

Accreditation in the package holiday market was, for two of the price comparison websites, provided

by the International Air Transport Association (IATA). A further two were members of IATA, but not

accredited by them. Other consumer assurances were being ATOL protected and having ABTA

membership.

Table 14N: Accreditations Total Base: All websites (13)

%

Whether accredited or not IATA accredited 30 Not accredited 69

Other memberships/protections IATA membership 23

ATOL protected 54 ABTA membership 38

On some parameters the two IATA accredited websites were more consumer-friendly than the non-

accredited ones, but in many other areas it was the non-accredited websites that outperformed

those with an accreditation. The variability is well illustrated in the following table showing

additional costs. Non-accredited sites were less likely to have departure tax as an additional cost (ie

it was included in the quote from the outset), meaning consumers are not faced with what could be

a substantial increase in the cost of the holiday once they had commenced a booking.

The non-accredited sites sometimes added costs such as transfers and travel insurance, relying on

the consumer to remove them if they are not wanted; accredited sites never did this, they always

gave the consumer a choice to add them rather than take them out.


Table 14O: Additional costs Base: All mystery shops






A=accredited, NA=non-accredited A NA A NA A NA A NA Single person supplement A (12)

NA (65) % 100 94 0 3 0 0 0 3

Supplier failure protection A (12) NA (65)

% 92 74 0 20 0 2 8 5

Departure taxes/local fees A (12) NA (65)

% 58 80 42 17 0 2 0 2

Hotel/board upgrades A (12) NA (65)

% 75 69 0 0 0 2 25 29

Cost of hold luggage/bags A (12) NA (65)

% 83 65 0 8 0 8 17 20

Transfers to/from airport A (12) NA (65)

% 75 58 0 0 0 15 25 26

Airport parking A (12) NA (65)

% 58 62 0 0 0 0 42 38

Travel insurance A (12) NA (65)

% 17 43 0 2 0 25 83 31

b. Package holidays: Where accredited price comparison websites sites perform better than those that are not accredited

An advantage for the mystery shoppers on the accredited sites was more often being able to reorder

and filter the initial list of quotes compared to shoppers on the non-accredited sites.

They were also less likely to encounter additional restrictions or limitations in the holidays shown,

and more likely to have a list that was in price order.

Information about the number of providers compared was more common on accredited sites.

Finally, shoppers rated the accredited sites more highly in terms of providing a positive impression.


Table 14P: Areas where accredited sites perform better



(12) %

(65) %

Can reorder initial list of quotes 62 83 58 Can filter initial list of quotes 74 92 71 Show 10+ tariffs on first page 84 100 82

No additional restrictions in first 5 quotes in list 95 100 94 Gives a positive impression 69 83 66

Cost of selected quote is ‘from £x’ 39 33 40

Base: All mystery shops where could get list into price order (73) %

(11) %

(62) %

Whether list was really in price order Yes 93 100 92 No 7 0 8


(2) %

(11) %

States how many providers are compared 38 50 36

c. Package holidays: Where accredited price comparison websites do not perform as well as non-accredited sites

Non-accredited sites had more options beyond the basic requirements in terms of specifying the

criteria to be used for the holiday search eg board type or minimum accommodation rating.

Table 14Q: Details can enter when starting search



(12) %

(65) %

Can input details on… …number of adults 96 100 95 …departure airport 94 92 94

…number of children 87 92 86 …duration 73 17 83

…return date 61 100 54 …number of rooms 48 50 48

…board type (eg all inclusive, half board) 47 8 54 …minimum accommodation rating 39 8 45

…ages of children 38 42 37 …flexibility (eg +/- 3 days) 10 0 15

…room type 10 8 11


Shoppers less often saw an explanation of how the initial list of quotes had been ordered when the

visit was to an accredited site, and were less often able to access a summary of their search criteria

from the first page of quotes.

Although a minority of the shoppers found something confusing or misleading, a greater proportion

of the mystery shops on accredited sites resulted in the shopper saying either of these.




(12) %

(65) %

Explained how initial list had been ordered 50 42 51 Could access a summary of search criteria from first page of quotes

90 83 91

Anything on websites was confusing 23 33 22 Anything on website was misleading 18 25 17

In terms of transparency, neither of the accredited sites provided information across a range of

areas, including income generation, price updating and why some providers/products might not be

included. Neither of them had a Code of Conduct on the site nor a policy for dealing with rogue or

fraudulent traders.

Table 14S: Areas where accredited sites do not perform as well as non-accredited sites



(2) %

(11) %

States why some providers/products might not be included

8 0 9

Clear how the site generates income 15 0 18 Says how often prices are updated 15 0 18

Has a policy for dealing with rogue or fraudulent providers

8 0 9

Customers can report suspicions of fraudulent activity directly to the site

8 0 9

Has a complaint policy available on the site 62 50 64 Customer can issue a complaint online 23 0 27

Has a Code of Conduct on the site 8 0 9 Base: All websites including provider/product ratings (8)

% (1) %

(7) %

Explains how ratings for providers/products are calculated

50 0 57


d. Package holidays: Where accredited and non-accredited price comparison websites perform equally

An IATA accreditation made no difference to whether or not the initial list of quotes was based on all

the criteria input by shoppers, nor whether it was possible to get the list into price order.

Table 14T: Areas where accredited and non-accredited sites perform equally



(12) %

(65) %

Initial list of quotes appears to be based on criteria input 97 100 97 Possible to get initial list of quotes into price order 95 92 95

14.5 Package holidays: Big 4 price comparison websites versus others

a. Package holidays: Overview

Three of the Big 4 price comparison websites operate in the package holiday market. None of them

are accredited in relation to this market. On some occasions they are better than non-Big 4 sites, but

on more occasions they are worse.

b. Package holidays: Where Big 4 price comparison websites sites perform better than non-Big 4


providing a (link to a) summary of the search criteria on the first page of quotes

being able to get the initial list into price order

the initial list actually being in price order

providing the quote as a total cost rather than per person, or giving both

including departure taxes/local fees in the initial quote and not adding them later

giving consumers the choice to add extras such as transfers and travel insurance, as opposed to adding them automatically

not being confusing for consumers

explaining how the website generates income

having a complaint policy


Table 14U: Areas where Big 4 sites perform better


% (17) %

(60) %

Providing a (link to ) a summary of the search criteria on the first page of quotes

90 100 87

Can get initial list into price order 95 100 93 Initial list actually was in price order 93 100 91

Provided quote as total cost 21 29 18 Provided quote both as total cost and per person 12 24 8 Departure taxes/local fees not an additional cost 77 82 75

Transfer costs automatically added 13 6 15 Travel insurance automatically added 22 0 29

Not found confusing 77 82 75 Base: All websites

(13) %

(3) %

(10) %

Have a complaint policy 62 100 50 Provide income generation details 15 33 10

c. Package holidays: Where Big 4 price comparison websites do not perform as well as non-Big 4

The Big 4 price comparison websites do not perform as well as non-Big 4 in the following: not making supplier failure protection a mandatory addition

explaining how the initial list of quotes was ordered

being able to reorder and filter the initial list of quotes

providing information about price updating, and showing when prices were last updated on the first page of quotes

providing a definite cost as opposed to ‘from £x’


explaining why some providers/products might not be included

having a policy for dealing with rogue or fraudulent traders and allowing consumers to report suspicions of such activity directly to the site

allowing complaints to be submitted online and specifying the number of days it will take to respond

having a Code of Conduct

having a glossary of the terms used


Table 14V: Areas where Big 4 sites do not perform as well as non-Big 4


% (17) %

(60) %

Supplier failure protection is a mandatory addition 17 24 15 Explanation of how initial list of quotes was ordered 50 24 57

Able to reorder initial list 62 12 77 Able to filter initial list 74 65 77

On first page of quotes shows when prices were last updated

6 0 8

Provides a cost ‘from £x’ 39 76 28 Base: All websites

(13) %

(3) %

(10) %

Provides price update information 15 0 20 Identifies how many providers are compared 38 0 50

Explains why some providers/products might not be included

8 0 10

Has policy for dealing with rogue or fraudulent traders 8 0 10 Can report suspicions of such activity directly to the site 8 0 10

Has a Code of Conduct 8 0 10 Has a glossary of terms used 8 0 10

Provides information about price updating 15 0 20 Base: All websites with a complaint policy

(8) %

(3) %

(5) %

Customer can issue a complaint online 37 33 40 Specifies number of days will take to respond 37 33 40

d. Package holidays: Where Big 4 and non-Big 4 price comparison websites perform equally

initial list of quotes being based on all the consumer’s criteria

no additional restrictions/limitations being found in the initial list

providing sufficient information such that shoppers felt comfortable making a booking

not being misleading to consumers



explaining how provider/product ratings are calculated


Table 14W: Areas where Big 4 and non-Big 4 sites perform equally


% (17) %

(60) %

Initial list of quotes based on all the consumer’s criteria 97 100 97 No additional restrictions/limitations found in initial list 95 94 95

Feel comfortable making an application 79 76 80 Not found misleading 82 82 82

Gives a positive impression 69 71 68 Base: All websites

(13) %

(3) %

(10) %

Explains how results of a comparison are calculated 0 0 0 Base: All websites with provider/product ratings

(8) %

(2) %

(6) %

Explains how provider/product ratings are calculated 50 50 50

14.6 Package holidays: Dynamic pricing

In order to ascertain whether or not dynamic pricing was taking place, mystery shoppers revisited

the price comparison website three times and searched for the holiday using the same criteria. Half

of the shoppers cleared their cookies and browsing history straight after each visit, while the

remainder carried out some holiday-related searches or browsing activity between visits.

Most of the shoppers were able to find the same holiday on all three return visits. This was more

common when the between-visit activity was to clear all trace of their price comparison website

visit, and less often the outcome when doing holiday-related searches/browsing. However, this

could be due to chance rather than being linked to dynamic pricing.

Table 14X: Whether or not found holiday on return visits

What did between visits… Total …cleared

browser/ cookies

…holiday related

searches Base: All mystery shops (77)

% (38) %

(39) %

Found holiday on all 3 return visits 77 84 69 Found holiday on 1-2 return visits 16 13 18

Could not find holiday on any return visits 8 3 13

Half of the holidays that could be found on all three return visits had the same overall price each

time. There were 17 permutations to the pattern of price movement amongst the other half.


Those permutations where the price got higher on one or more return visit (perhaps also remaining

the same at some point but never getting lower) are shown in the table below in bold italics. These

are the price movement combinations that could indicate dynamic pricing. They comprised only

eight per cent of the mystery shops for which the holiday could be found on all return visits.

These permutations were less likely to occur when the shoppers were clearing their cookies and

browsing history compared to when they were carrying out holiday related searching and browsing.

In other words, when the conditions that would negate dynamic pricing are created, there is less

chance of seeing its characteristic price movements. When the conditions that would allow dynamic

pricing are present, there is more chance of seeing its characteristic price movements.

Table 14Y: Price of holiday compared to previous visit

Between visits…

Total …cleared browser/ cookies

…holiday related

searches Base: All mystery shops where could find holiday on all 3 return visits (59)

% (32)

% (27)

%

Whether price is same, higher or lower than previous visit

2nd visit 3rd visit 4th visit

Same Same Same 51 56 44 Higher Higher Lower 7 12 0

Same Lower Same 5 3 7 Same Higher Same 5 3 7

Higher Lower Higher 5 0 11 Higher Same Lower 3 3 4 Lower Same Higher 3 3 4 Lower Higher Lower 3 3 4 Same Same Higher 2 0 4 Same Higher Lower 2 3 0

Higher Lower Lower 2 3 0 Higher Lower Same 2 0 4 Higher Same Higher 2 0 4 Lower Lower Higher 2 0 4 Lower Same Same 2 3 0 Lower Same Lower 2 3 0 Lower Same Same 2 3 0 Lower Same Lower 2 0 4


Out of the 60 return visits that resulted in finding the same holiday at a different cost, there were

only seven instances where the shopper could see the reason for it. These were mainly due to a

price movement in the flight or hotel cost.

Table 14Z: Whether shopper could see the reason for the cost difference Total Base: All return visits where same holiday was found at a different cost (60)

%

Could see the reason for the cost difference? Yes 12 No 88

Reasons for cost difference Flight more expensive 5 Hotel more expensive 2

Flight cheaper 2 Hotel cheaper 2

Flight has been changed 2


14.7 Package holidays: Chapter summary

Beyond the basics of destination, dates of travel and how many people the booking

was for, some shoppers found they could specify details such as room type, board

type and minimum accommodation rating. These could be offered more widely; the

more details that can be input at the start the more relevant the initial list of quotes

will be

In most cases, the holidays shown had taken into account all the criteria the

shopper was able to input; only three per cent found some of the criteria had been

ignored

Showing how the initial list of holidays had been ordered could be improved, as only

half of the shoppers were provided with an explanation. This is particularly

pertinent because price was not always the ordering factor, but it could be the

assumption of some that it was

In 29 per cent of cases the initial list was sorted on a variable other than price, such

as relevance or popularity

Many of the initial lists could be reordered (62 per cent) or filtered (74 per cent), but

a more universal adoption of these tools would make the choice process more

consumer-friendly

Price and accommodation star rating were common reordering or filtering options.

Some lists could be ordered on customer ratings (21 per cent) or the somewhat

vague terms ‘popularity’, ‘bestsellers’ or ‘our favourites’ (17 per cent)

62 per cent of the websites used some form of rating system for the holiday and/or

the provider, but only half of them had an explanation of how these had been

calculated. This should be more widespread, as without knowing the basis for the

ratings it is impossible for a consumer to decide whether they are of value

A majority of lists could be put into price order (if they were not in that order to

start with). Where this was possible, 93 per cent of shoppers were satisfied that the

ranking was correct

Holiday cost was substantially more often quoted as ‘per person’ rather than in total

(64 per cent versus 21 per cent respectively), which could potentially lead to

confusion

A minority of holidays were found to include additional restrictions or limitations

(five per cent)

The picture was different for additional costs, which arose fairly frequently. In most

instances they were not added automatically; the consumer had a choice of

whether or not they wanted to add the likes of travel insurance, transfers, airport

parking or an upgrade of hotel/board


However, 21 per cent of shoppers had travel insurance added automatically and 13

per cent found the same for transfers. Although they were able to remove these

costs, there is a danger that some consumers will not notice the additions, and end

up paying for something they either did not need or did not want. It would be

preferable if optional extras were always added by the consumer

Two types of cost were added automatically that shoppers were unable to remove,

namely departure taxes/local fees (21 per cent) and supplier failure protection (17

per cent). Although supplier failure protection may be of benefit to the holiday-

maker, it is not a mandatory requirement and should be presented as an optional

extra

Whilst departure tax has to be paid by everyone, it could perhaps be included in the

initial quotes as opposed to at a later stage, as its addition can make a substantial

difference to the price of a holiday, perhaps taking it out of the consumer’s budget

Some shoppers felt that going through the process of choosing and booking a

holiday had been confusing, or they had been mislead in some way (23 per cent and

18 per cent respectively). The reasons given were often related to costs, indicating a

need for increased clarity in this area

There was a widespread lack of information on the websites about how they

operated. None explained how they calculated the results of a comparison, a mere

15 per cent said how often prices were updated and only 38 per cent identified how

many providers they compared, few had a Code of Conduct (8 per cent) and only 15

per cent provided information on how they generated income

On a more positive note, 62 per cent published details of their complaint process.

The other 38 per cent could be encouraged to follow suit

Although the two sites accredited by IATA outperformed the non-accredited sites in

some areas, it was commonly the non-accredited sites that were more consumer-

friendly

The three Big 4 sites were better than the non-Big 4 sites in some respects, but on

more parameters were worse

If the definition of dynamic pricing is taken as the cost of a holiday rising on at least

one out of three return visits to the website made within 1-2 days of getting the

original quote, and never becoming lower, then eight per cent of the package deals

found on all three return visits fell into that category

Although this is a minority, the incidence of dynamic pricing was three per cent

when shoppers cleared their browsing history/cookies between visits and 15 per

cent when they did not do so and, as an alternative, made holiday-related searches

between visits

In other words, when the conditions for dynamic pricing are present, there is a

greater likelihood of encountering a price increase than when the conditions for

dynamic pricing are negated


15. Rail fares

15.1 Rail fares: Number of price comparison websites and mystery shops

A total of seven price comparison websites were included in the rail fares market part of the

research. These are all the rail fare websites known at the time of the research. Mystery shoppers

made 56 site visits to pose scenarios on these sites.

15.2 Rail fares: Compliance with Consumer Protection from Unfair Trading Regulations

a. Rail fares: Obtaining a list of quotes

In order to view an initial list of options, shoppers most commonly were able to enter information

about where they were travelling from and to, the date(s) and time(s) of travel and how many

passengers needed a ticket. They were often asked if a Railcard was being used, and could specify an

open return or one-way ticket. There was less scope to specify a journey that took a particular route,

the class of travel and whether they only wanted to see trains with no changes. All of these help

make the initial list of trains and tickets more closely match what the consumer wants.

In only one instance did the initial list not match the criteria input by the shopper. The explanation

was:

‘I requested an open return, but the search facility informed me I had to enter a return date, it didn’t allow me to leave this open. Therefore my results were limited to returning on the next day, rather than on an open return basis’

Table 15A: Details can enter and whether list shown takes all of these into account Total Base: All mystery shops (56)

%

Can input details on… …journey to 100 …date(s) of travel 100

…journey from 98 …time(s) of travel 98

…no. of adult passengers 95 …no. of child passengers 89

…whether have a Railcard 86 …whether want an open return 82 …whether want one way ticket 80

…whether want to go via a particular place 62 …class of travel 52

…how many Railcards have 32 …whether only want to see trains with no changes 32

Was the initial list based on all the criteria input? Yes 98 No 2


b. Rail fares: Ability to further refine search

Once presented with an initial list, most shoppers could further refine it. Other than changing the

time of travel, common refinements were contracting the list to see only the cheapest tickets or

expanding the list and viewing all tickets for that route.

In a minority of cases it was possible to view only a certain class of travel, only trains with no

changes, slower routes and fares with more flexibility. These are useful options; wider availability

would help ticket comparison and choice of the most suitable option for a consumer’s needs.

Table 15B: Whether was possible to further refine search on page with initial list Total Base: All mystery shops (56)

%

Possible to further refine search? Yes 82 No 18

Refinements possible Change time of departure and/or arrival 48

View all tickets/fares 48 View only cheapest tickets 48

View only certain class of travel 27 View/check for slower routes 23

View only trains with no changes 14 View fares with more flexibility 14

c. Rail fares: Whether journey summary and cost shown when make final selection

On making the final selection of trains and tickets, the cost was immediately shown in most cases.

All shoppers also saw a summary of their journey at this point, which (apart from showing dates,

times, departure and destination stations) often included details such as duration of journey,

whether a change of train was required and the number of people travelling.

The journey summary was less likely to include items such as whether any travel restrictions applied,

ticket validity information, whether changes prior to travel were allowed, and whether or not they

could obtain a refund if they had to cancel. It would be helpful for all consumers to have these

included in the summary, as this information might lead to them rethinking their journey choices

prior to making the booking.


Table 15C: Whether shown cost and summary of journey Total Base: All mystery shops (56)

%

Cost of journey shown when made final selections? Yes 98 No 2

Shown a journey summary? Yes 100 No 0

What was shown in summary Duration of journey 71

Any train changes needed 67 No. of passengers 67

Ticket validity details 44 Whether a Railcard had been used 35

(Non) refund details 29 Travel restrictions 24

Changes allowed prior to travel for a fee 13

d. Rail fares: Restrictions and limitations not previously mentioned

Not many of the shoppers found that additional restrictions or limitations applied to their chosen

ticket (ie factors that had not been mentioned earlier in the booking process). Where found, these

mainly related to ticket validity, travel restrictions, a train change being necessary or that the ticket

was non-refundable. Earlier mention of these factors would be advantageous.

Table 15D: Whether selected ticket had additional restrictions/limitations Total Base: All mystery shops (56)

%

Did selected ticket have any additional restrictions/limitations? Yes 13 No 87

Additional restrictions/limitations Ticket validity 7

Travel restrictions/allowances 7 Train change needed 5

Non-refundable 4 Changes prior to travel incur a fee 2


e. Rail fares: Additional costs

When making the booking a range of optional additional costs were shown, such as seat booking,

travel insurance, ticket delivery options and whether to add PlusBus. These were mainly shown as

costs which the traveller could choose to add or ignore.

A high proportion of shoppers had a booking fee added to the ticket cost, and sometimes there was

a charge for paying by credit or debit card. Where a booking fee appeared it had never been

previously mentioned.

Table 15E: Additional costs Base: All mystery shops






Seat booking (56) % 82 0 0 18 Travel insurance (56) % 59 0 0 41 PlusBus (56) % 39 0 0 61 Ticket delivery (56) % 29 2 2 68 Booking fee (56) % 43 57 0 0 Credit/debit card fee (56) % 59 37 2 2 Base: All mystery shops (56)

%

Were any other costs added automatically that could not be removed? Yes 0 No 100

f. Rail fares: Whether anything on the price comparison website was confusing or misleading

Although most shoppers did not find anything confusing about the price comparison website, a

significant minority did. A range of reasons was given, from not being told the name of the train

operator to difficulties using the journey planner:

‘It did not tell me the train company I would be travelling with. I had to hunt around for this information’ ‘I was unable to find the train companies I would be travelling with on this site’ ‘I don't fully understand all the ticket type explanations, they seem too general’ ‘Why I had to register before getting to the final stages of a price/booking’ ‘The journey planner options did not work, i.e. via, avoid or stop at’ ‘When I entered the station name I was travelling from I entered Nottingham, but it did not recognise this station until I actually clicked upon the station map which popped up and clicked on NOTTINGHAM – it had to be in all capital letters to proceed’


‘I could not select an "open return" ticket without putting in my return date and approximate time, which I found confusing’

If the price comparison website was misleading in some respect, this was usually in relation to

charging a booking fee:

‘I do not feel it was made clear anywhere that I would be charged a booking fee on completion of my order’ ‘Price is not the actual price until you get to the end and booking fees and card fees are added on, which are not mentioned at the beginning of the process’ ‘The booking fee and payment card fee are hidden until the very end’ ‘Booking fee and debit card fee added at the last steps’ ‘No booking fee or credit card fee stated at outset’

Table 15F: Whether anything on website was confusing or misleading Total Base: All mystery shops (56)

%

Any on website confusing? Yes 20 No 80

Anything on website misleading? Yes 16 No 84

g. Rail fares: Overall impressions

Impressions of the price comparison websites were largely positive, and most shoppers felt they had

sufficient information to feel comfortable booking the ticket.

‘I found that the search facility was very quick and easy to use. Furthermore, results loaded quickly and they were relevant to the criteria which I had input. I liked that it was very simple to make changes to my travel details, and the costs were clearly shown’ ‘I found the booking very quick and easy to complete. All information was very clearly written in an understandable manner’ ‘I found the website easy to use, and I was impressed with the prices. I would definitely visit the site again in my personal capacity’ ‘Overall, a very good website. It was quick and easy to input search criteria. Results were returned swiftly and were easy to review. Following the selection and booking process was clear and concise’


‘I think that the search is easy to use and any restrictions are made clear. There were no hidden costs’ ‘It's easy to find and book cheap tickets, but I don't like hidden costs that don't become visible until the point of purchase’ ‘Easy to navigate and well laid out. The only thing that was disappointing was that there was no mention of the booking fee or the additional fee to pay by credit card’ ‘It is easy to enter information, and there is a good variety of services available. Information regarding train service you are travelling on is not clearly visible. Booking fee should appear a bit earlier in the process’

Three shoppers wanted more information before making a booking:

‘I would like to find out their cancellation policy’ ‘About any restrictions on the ticket’ ‘Name of the train operator’

Table 15G: Overall impressions Total Base: All mystery shops (56)

%

Is your overall impression of this website… …very positive 30 …quite positive 54

…neither positive nor negative 12 …quite negative 4 …very negative 1

Do you at this stage have sufficient information to make a booking? Yes 95 No 5


15.3 Rail fares: Accuracy

a. Rail fares: Whether ticket was available on supplier website and, if so, whether was same cost

A majority of shoppers were able to find the same ticket on the train company website, although not

often at the same price. Where there was a difference, it was twice as likely to be that the cost was

lower if bought directly from the provider rather than higher. The level of price difference ranged

from £1 to £52.

Table 15H: Ticket costs – price comparison website versus supplier website Total Base: All mystery shops (56)

%

Could find ticket on supplier site? Yes 98 No 2

Base: All mystery shops where ticket could be found on supplier site (55) %



Most shoppers could not ascertain the reason for the price difference. Amongst those who could, it

was usually because the train company was not charging a booking fee or not charging a fee for card

payment. In a few instances the train operator was offering a lower cost ‘advance’ ticket, which had

not been shown on the price comparison website.

On the one occasion when the train operator cost was higher and the shopper could see the reason

why, it was due to the price comparison website offering a £5 discount for an immediate booking.

Table15I: Reasons for the price difference Total Base: All mystery shops where ticket could be found on supplier site and cost was different (35)

%


Reasons for price difference No booking fee 26

No card fee 14 Supplier provided an ‘advance’ ticket 9


b. Rail fares: Whether the supplier site had additional details

Some of the train operator websites gave additional information about the ticket. These related to

ticket validity, travel restrictions and whether changes could be made (for a fee) prior to travel. It is

important that consumers are clear on all these points prior to purchase, which is not always the

case when the ticket is chosen from a price comparison website.

Table 15J: Whether supplier site had additional details Total Base: All mystery shops where ticket could be found on supplier site (55)

%


What were the additional details? Ticket validity details 20

Changes prior to travel allowed for a fee 14 Travel restrictions/allowances 6

15.4 Rail fares: Transparency

a. Rail fares: Information provided

In many areas the rail fare price comparison websites were lacking when it came to providing

information about the way they operated. Of the seven price comparison websites looked at:

none published details of their complaint policy/process

none had a Code of Conduct on the website


none explained why some suppliers or tickets might not be included

none explained how they generated income

none provided supplier ratings

none said how frequently they updated their prices, and in only 4 per cent of mystery shops could the shoppers see when the prices were last updated


only 29 per cent said how many providers they compared

On only two occasions did a mystery shopper see a note on the homepage informing potential customers that a booking fee would, or might, be charged. In both cases a booking fee was not added to their ticket price.


Most shoppers did not see notification about whether VAT was included in the prices

quoted.

Table 15K: Information provision Total Base: All mystery shops (56)

%

Any mention on rail travel homepage that a booking fee may/will be charged?

Yes 4 No 96

Note anywhere to say VAT included? No, was a note about VAT being excluded 5

No, was not a note about VAT 95

Areas where transparency was better amongst the seven sites were:

46 per cent of shoppers had been alerted to the fact that two singles might be cheaper than a return

57 per cent of sites provided or signposted additional information

86 per cent of sites provided a glossary of the terms used (eg anytime, off peak, super off peak)

b. Rail fares: Details of the information provided

None of the price comparison websites comparing rail fares had provider advertising on the site.

Of the three sites that provided or signposted additional information, all gave travel advice for

specific groups (eg minors travelling alone, wheelchair users).

Table 15L: Details of the information provided Total Base: All websites (7)

% Whether site includes advertising for provider sites

Yes 0 No 100

Base: All websites providing or signposting additional information (3) %

Provide or signpost information about… …visitor information for destination 33

…travel advice for specific groups 100


15.5 Rail fares: Accreditation

a. Rail fares: Overview

Five of the rail fare price comparison websites were National Rail accredited. Two did not have this

accreditation (one of these sites being National Rail itself). There was variability between the

accredited sites and the non-accredited sites; on some parameters the accredited ones performed

better but in other areas were not as good as the non-accredited sites. This variability is well

illustrated in the table below. Accredited sites were more likely than non-accredited sites to add a

booking fee. In terms of adding a credit/debit card processing fee, the situation was reversed, and it

was more common amongst non-accredited sites.

Table 15M: Additional costs Base: All mystery shops






A=accredited, NA=non-accredited A NA A NA A NA A NA Seat booking A (40)

NA (16) % 80 87 0 0 0 0 20 12

Travel insurance A (40) NA (16)

% 63 50 0 0 0 0 37 50

PlusBus A (40) NA (16)

% 28 69 0 0 0 0 72 31

Ticket delivery A (40) NA (16)

% 25 37 2 0 2 0 70 62

Booking fee A (40) NA (16)

% 40 50 60 50 0 0 0 0

Credit/debit card fee A (40) NA (16)

% 63 50 33 50 2 0 2 0

b. Rail fares: Where accredited price comparison websites sites perform better than those that are not accredited

All of the shoppers on accredited sites were shown a list of options that was based on all the criteria

they input, which was not always the case for non-accredited site shoppers. When the final journey

selections were made, the shoppers on accredited sites were more likely to see the total journey

cost compared to those on non-accredited sites. Not finding anything confusing was more common

on the accredited sites, which were also more likely to provide or signpost additional information.

Accredited sites were also better at providing information about the number of providers compared

and how the results of a comparison are calculated.


Table 15N: Areas where accredited sites perform better



(40) %

(16) %

Initial list based on all criteria input 98 100 94 Journey cost shown when made final selections 98 100 94

Nothing confusing on website 80 85 69 Base: All websites (7)

% (5) %

(2) %

States how many providers are compared 29 40 0 Provides or signposts additional information 57 60 50

Explains how the results of a comparison are calculated 14 20 0

The accredited sites were substantially more likely to allow further refinements, such as viewing all

tickets/fares or only the cheapest ones.

Table 15O: Whether was possible to further refine search on page with initial list



(40) %

(16) %

Possible to further refine search? Yes 82 87 69 No 18 13 31

Refinements possible Change time of departure and/or arrival 48 55 31

View all tickets/fares 48 57 25 View only cheapest tickets 48 50 44

View only certain class of travel 27 27 25 View/check for slower routes 23 27 12

View only trains with no changes 14 15 12 View fares with more flexibility 14 15 12

In relation to the details that could be input at the start of a search, shoppers on accredited sites had

more opportunity to enter factors such as wanting a one-way ticket or an open return, whether they

had a Railcard and, if so, how many.


Table 15P: Details can enter when starting search



(40) %

(16) %

Can input details on… …date(s) of travel 100 100 100

…journey from 98 100 94 …time(s) of travel 98 97 100

…no. of adult passengers 95 100 81 …no. of child passengers 89 100 62

…whether have a Railcard 86 95 62 …whether want an open return 82 95 50 …whether want one way ticket 80 87 62

…whether want to go via a particular place 62 55 81 …class of travel 52 47 62

…how many Railcards have 32 40 12 …whether only want to see trains with no changes 32 30 37

c. Rail fares: Where accredited price comparison websites do not perform as well as non-accredited site

The table above shows that non-accredited sites more often allowed shoppers to input class of

travel as a search variable, and also whether they wanted to go via a particular station.

Shoppers on non-accredited sites more often found there were no additional restrictions/limitations

appearing partway through the purchase of their selected ticket. On all visits to a non-accredited site

shoppers felt they had been given sufficient information to make a booking; this was not the case for

all of the accredited site visits. Shoppers were more likely to say that nothing was misleading on a

non-accredited site, and to gain a positive overall impression.


Table 15Q: Areas where accredited sites do not perform as well as non-accredited sites



(40) %

(15) %

No additional restrictions/limitations in selected ticket 87 85 94 Have sufficient information to feel comfortable making

booking 95 92 100

Nothing misleading on website 84 82 88 Positive impression of website 84 80 94


(5) %

(2) %

Site includes glossary of terms used 86 80 100

d. Rail fares: Where accredited and non-accredited price comparison websites perform equally

Being accredited or not made no difference to whether the site provided or signposted additional

information. Both types of site were equally likely to alert shoppers to the possible cheaper cost of

two single tickets versus a return.

Table 15R: Areas where accredited and non-accredited sites perform equally



(40) %

(16) %

Provides or signposts additional information 50 50 50 Suggests at some point that two singles possibly cheaper

than return 46 47 44


15.6 Rail fares: Big 4 price comparison websites versus others

a. Rail fares: Overview

One of the Big 4 price comparison websites operates in the rail fares market. The other six sites

included in the research were non-Big 4. Although there was some variation in terms of performance

(sometimes the Big 4 site outperformed the rest, sometimes the reverse was true) on balance the

Big 4 site was the more consumer-friendly. This is well illustrated by the figures in the table below,

showing the details that could be entered at the start of a search. Many, but not all, of the options

were more common for the Big 4 mystery shops.

Table 15S: Details can enter when starting search


% (8) %

(48) %

Can input details on… …journey to 100 100 100

…date(s) of travel 100 100 100 …journey from 98 100 98

…time(s) of travel 98 100 98 …no. of adult passengers 95 100 94 …no. of child passengers 89 100 87

…whether have a Railcard 86 100 83 …whether want an open return 82 100 79 …whether want one way ticket 80 100 77

…whether want to go via a particular place 62 37 67 …class of travel 52 25 56

…how many Railcards have 32 12 35 …whether only want to see trains with no changes 32 0 37

b. Rail fares: Where Big 4 price comparison site performs better than non-Big 4

The Big 4 price comparison website performs better than the non-Big 4 in the following areas:


having a glossary of the terms used

not adding a booking fee or credit/debit card fee

allowing further refinement of the initial list of trains/tickets

alerting the consumer to the fact that two singles might be cheaper than a return

not being confusing or misleading for consumers



Table 15T: Areas where Big 4 site performs better


% (8) %

(48) %

Booking fee added automatically and cannot be removed

57 0 67

Credit/debit card fee added automatically and cannot be removed

37 0 44

Can further refine initial list of trains/tickets 82 100 79 Alerting consumer that two singles might be cheaper

than a return 46 62 44

Nothing confusing on website 80 87 79 Nothing misleading on website 84 100 81 Positive impression of website 84 100 81

Base: All websites

(7) %

(1) %

(6) %

Identifies how many providers are compared 29 100 17 Has a glossary of terms used 86 100 83

c. Rail fares: Where Big 4 price comparison website does not perform as well as non-Big 4

The Big 4 price comparison website does not perform as well as non-Big 4 in the following:

showing a wide range of items in the journey summary

providing sufficient information such that shoppers felt comfortable making a booking

providing a note to say when prices were last updated


providing or signposting additional information

Table 15U: What was shown in summary


% (8) %

(48) %

What was shown in summary Duration of journey 71 75 70

Any train changes needed 67 62 68 No. of passengers 67 62 68

Ticket validity details 44 37 45 Whether a Railcard had been used 35 25 36

(Non) refund details 29 37 28 Travel restrictions 24 25 23

Changes allowed prior to travel for a fee 13 0 15


Table 15V: Areas where Big 4 site does not perform as well as non-Big 4


% (8) %

(48) %

Have sufficient information to make a booking 95 75 98 Provide a note to say when prices were last updated 4 0 4

Base: All websites

(7) %

(1) %

(6) %

Explains how the results of a comparison are calculated 14 0 17 Provides or signposts additional information 57 0 67

d. Rail fares: Where Big 4 and non-Big 4 price comparison websites perform equally

The Big 4 website has an equal performance to the non-Big 4 websites for a few items:

initial list of quotes being based on all the consumer’s criteria

cost of journey being shown when final selections are made

no additional restrictions/limitations being found when final selections are made

Table 15W: Areas where Big 4 and non-Big 4 sites perform equally


% (8) %

(48) %

Initial list of quotes based on all the consumer’s criteria 98 100 98 Cost of journey shown when make final selections 98 100 98

No additional restrictions/limitations found when final selections made

87 87 87


15.7 Rail fares: Dynamic pricing

Within 1-2 days of their first visit to the price comparison website, shoppers made three return visits

(ie four visits in total) and looked for the same ticket then noted its price. The shoppers were split

into two equal-sized groups:

cleared cookies and browsing history straight after each visit (ie created conditions which would reduce the opportunity for dynamic pricing)

did not clear cookies or browsing history, plus made a range of rail travel related searches in between price comparison site visits (ie created the optimum dynamic pricing conditions)

The shoppers were able to find the same ticket on all of their return visits. In most cases the ticket

price was identical on all return visits.

If the presence of dynamic pricing is taken as the cost increasing on one or more visit, but never

decreasing, a low proportion of the mystery shops fell into this category. Additionally, all of them

were amongst shoppers who cleared their cookies and browsing history. Hence there was no

evidence of dynamic pricing in the rail fares market.

Table 15X: Price of ticket compared to previous visit

Between visits…

Total …cleared browser/ cookies

…rail related

searches Base: All mystery shops (56)

% (28)

% (28)

%

Whether price is same, higher or lower than previous visit

2nd visit 3rd visit 4th visit

Same Same Same 87 89 86 Lower Same Same 5 4 7

Higher Same Same 4 7 0 Higher Lower Lower 2 0 4 Higher Lower Same 2 0 4

Additionally, the ticket cost from the final price comparison website visit was compared to the cost

quoted on the train provider website at that point in time, and any variation noted (ie whether the

supplier cost was higher, lower or the same). The results were then compared to those obtained

from the supplier visit undertaken after the first price comparison website visit. If dynamic pricing

had occurred, for the final visit comparison there should be a greater proportion of supplier site

costs that were lower than the price comparison site cost. This was not the case; the two

proportions were identical.


Table 15Y: Ticket costs – price comparison website versus supplier website (compared after last of three return visits to price comparison website)

Total Cost on supplier website after first price comparison

website visit Base: All mystery shops (56)

% (56)

%

Could find ticket on supplier site? Yes 98 98 No 2 2

Base: All mystery shops where ticket could be found on supplier site (55) %

(55) %

Supplier site cost was… …same as price comparison website 42 36

…higher than price comparison website 16 20 …lower than price comparison website 44 44

Where there was a cost difference and the shopper could see the reason for it, the explanations

centred on the price comparison website charging a booking fee and/or payment card processing

fee. Occasionally the cost was lower from the supplier due to their offering an ‘advance’ ticket.

Table 15Z: Whether shopper could see the reason for the cost difference Total Base: All return visits where same ticket was found at a different cost (32)

%

Could see the reason for the cost difference? Yes 37 No 63


15.8 Rail fares: Chapter summary

There is scope to increase the amount of detail that can be entered about journey

requirements at the start of a search. Although the basic details (dates, times,

number of passengers and departure/arrival stations) were almost universal, others

were less common. Being able to input information such as class of travel, travelling

via a particular place and excluding trains requiring a change are all useful but not

necessarily currently available in a majority of cases

The same is true of being able to further refine the list presented initially; the

availability of each option could be improved. Allowing refinement so that the

consumer can view only the cheapest tickets, or fares with more flexibility, or only

trains with no changes would assist the comparison and choice process

In all but one case the mystery shoppers were satisfied that the initial list of quotes

was based on all the criteria they had entered

Although all shoppers saw a summary of the journey before making their booking, it

could often be expanded to include important details such as travel restrictions,

ticket validity and (non) refund information. The consumer would then either be

reassured that they had chosen the ticket most appropriate to their needs, or

alerted to the fact that their selection was not quite what they were looking for

For almost half the shoppers, the fact that two single tickets might be cheaper than

a return was highlighted. Without knowing the incidence of journeys to which this

can apply, it is impossible to determine whether such notification is always

appearing when needed, or only being shown in some cases

When moving through the booking process a minority of shoppers found

restrictions or limitations that had not been mentioned at the point of ticket

selection (13 per cent). Those found included information about ticket validity,

travel restrictions and train changes. Their increased inclusion in both the

refinement options and the summary details would help negate their later discovery

Many shoppers found that a booking fee appeared in their final cost (57 per cent),

despite not being mentioned anywhere on the price comparison website as a

possible or definite addition. A small proportion of shoppers (four per cent) saw

mention on the website homepage that a booking fee might be charged, but in none

of these cases was it actually applied to the booking

Another automatically added additional cost was a credit/debit card processing fee,

albeit encountered by fewer shoppers than a booking fee (37 per cent). It should be

made absolutely clear from the outset that these (might) apply, so that consumers

are able to make an informed decision about proceeding

In all but one case, shoppers found the same ticket on the train provider website,

although it was not often the same price (36 per cent). The cost was twice as likely

to be lower on the provider site as opposed to higher (44 per cent versus 20 per

cent respectively)


Most of the shoppers who saw a price difference were unable to ascertain the

reason (63 per cent). Amongst those who could, it was generally that the price

comparison cost was higher due to charging a booking fee or an amount for

credit/debit card payment

Occasionally, the train operator cost was lower because they were offering an

‘advance’ ticket which had not been available on the price comparison website

Some of the supplier sites gave additional details about the ticket (18 per cent), such

as its validity, any travel restrictions and whether changes were allowed prior to the

date of travel (for a fee). These need better visibility in the price comparison

websites

The price comparison websites performed poorly in relation to the provision of

information about how they operate. None explained why some train

operators/tickets might not be included, none had details of their complaint policy,

none said how frequently they updated their prices and none explained how they

generated income. Only a minority showed how they calculated the results of a

comparison (14 per cent) or said how many train operators they compared (29 per

cent)

The sites accredited by National Rail were sometimes better than the non-

accredited sites, but on other parameters the reverse was true. Shoppers on non-

accredited sites more often found there were no additional restrictions/limitations

appearing partway through the purchase of their selected ticket. Conversely,

shoppers using the accredited sites could usually further refine their search once

presented with an initial list of possibilities; this was not as often the case for non-

accredited sites

Accredited sites were more likely than non-accredited sites to add a booking

fee. In terms of adding a credit/debit card processing fee, the situation was

reversed, and it was more common amongst non-accredited sites

Differences were seen between the one Big 4 site operating in the rail travel market

and the non-Big 4 sites, most notably that the Big 4 site never added a booking fee

or a charge for credit/debit card payment. The Big 4 site identified how many

providers it compared, more often alerted shoppers to the fact that two singles

might be cheaper than a return and was never found confusing or misleading by

shoppers

In other areas the non-Big 4 sites were better (eg explaining how the results of a

comparison are calculated). However, on balance the Big 4 site was more consumer-

friendly than the non-Big 4 websites

There was no evidence of dynamic pricing. In most cases the ticket cost was

identical on all return visits (87 per cent) regardless of whether shoppers cleared

cookies and browsing history or not


APPENDIX A

Desktop research and mystery shopper form

187 of 264

INTRO TEXT

Consumer Focus Price Comparison Website Mystery Shopping Version for Desktop review

# Question Code Answer Options Routing Q Type

Q1a Please specify which site you are looking at. All Open

Q1b What is the 3 digit number of the price comparison website you are to visit?

Enter 3 digit code number All From allocation

Q2

Which market are you looking at on this site? N.B. When you look at the site you should look at the homepage and the page(s) that relate specifically to this market

1 Broadband

All From allocation Single code

2 Energy

3 Car insurance

4 Home insurance

5 Mobile phones

6 Package holidays

7 Rail fares

Q3a

Is this a “Big 4” website (i.e. is it confused.com, gocompare, moneysupermarket or comparethe market)?

1 Yes, Big 4

All Single code 2 No, not Big 4

Q3b

Is this an accredited website? N.B. There may well be (additional) accreditation, regulation, authorisation, etc. that we are currently unaware of. You will be searching for these in during your visit.

1 Accredited by Consumer Focus


2 Accredited by Ofcom

3 Author of ABI‟s Good Practice Guide

4 Accredited by National Rail

5 Not accredited

Look for any accreditations, memberships, authorisations, approvals, protections, codes of good practice, etc. that would give peace of mind to potential customers in terms of the quality, transparency, accuracy, accessibility, etc. of the products and services available, and their ability to get recompense if anything went wrong. You should look on the homepage and the page(s) that relate to the specific market you are shopping. The details you need are often:

Logos / symbols at the top or bottom of the page (but also sometimes in the body of the page) Written out in the Terms & Conditions or in a section such as “About Us” / “About this site”

Look both for the accreditations we think are appropriate (i.e. those you have just recorded in the previous question) and any others.

Q4

Tick all applicable from the list below. Write in details of any other accreditations, memberships, authorisations, approvals, protections, codes of good practice, etc. that you come across, giving as much detail as possible.

1 Ofcom accredited

All Multi code other than last item

2 Ofcom approved

3 National Rail accredited

4 Consumer Focus accredited – in relation to cost calculator, price comparison tool

5 Consumer Focus accredited - whole site or this part of the site (as opposed to just cost calculator)

6 Ofgem accredited

7 Financial Services Authority (FSA) authorised & regulated

8 British Insurance Brokers Association (BIBA) accredited

9 British Insurance Brokers Association (BIBA) membership

10 IATA accredited

11 IATA membership

12 ATOL protected

13 ABTA membership

188 of 264

14 Shopsafe accredited

15 Shopsafe member

16 SafeBuy accredited

17 OFT licenced or registered

18 OFT any other mention

19 Award winner/runner-up/commended (e.g. Online Business of the Year, Best Online Retailer, National Business Award)

20 Other (specify)

21 Cannot find any of this type of information

22 No accreditation

Q6 Does the site identify how many providers it compares on behalf of the customer?

1 Yes

All Single code 2 No

3 Don‟t know/Not sure

Q7 How many? Q6 (1)

Open

Q8 Using the following scale, please rate how easy or difficult it was to find information about the number of providers?

1 Very easy

Q6 (1) Single code

2 Quite easy

3 Neither easy or difficult

4 Quite difficult

5 Very difficult

Q9 Does the site explain why some providers or products may not be included in search results

1 Yes



Q10 Please describe briefly the reasons why some providers/products may not be included in search results

Q9 (1)

Open

Q11 Is how the site generates income made clear?

1 Yes



Q12 Which of the following statements best describes how the site generates income?

1 Providers pay to appear on the comparison site

Q11 (1)

Multiple code

2 The comparison site is paid per „click through‟ to a providers site

3 The comparison site is paid each time a customer provides their email address or telephone number

4 The comparison site is paid on the completion of a successful application/switch by the customer

5 The comparison site charges a „sign up‟ fee to the customer

6 The comparison site does not generate income (not for profit)

7 From advertising sales

189 of 264

8 Other

Q13 Please describe briefly how the comparison site generates income Q12 (8)

Open

Q17

Which of the following statements best describes how often the website updates the prices (for the product/tariff/packages) it features on the site? Please only record what the website says about updating its prices. If this is not mentioned then please select 'doesn't specify'. Please do not guess at the frequency.

1 More than daily

All Single code

2 Daily

3 Weekly

4 Monthly

5 „Regularly‟ (no time specified)

6 When the provider tells the comparison site that things have changed

7 Doesn‟t specify

8 Other

Q18 Please describe briefly how often the website updates its prices Q17 (7)

Open

Q19 Using the following scale, please rate how easy or difficult it was to find information about prices on the site?

1 Very easy

Q17 (NOT 6)

Single code

2 Quite easy


4 Quite difficult

5 Very difficult

Q20 Does the comparison site include any advertising for any provider sites?

1 Yes



Q21 Where does the advertising appear?

1 Homepage

Q20 (1)

Multi code

2 On criteria pages (where you enter your details and requirements)

3 On results pages

4 On „affiliates‟ page

5 Other

Q14 Does the site identify the names of providers who pay to appear on the comparison site?

1 Yes

Q12 (1)

Single code 2 No


Q15 How many providers are identified? Q14 (1)

Open

Q16 Using the following scale, please rate how easy or difficult it was to find information about how the site generates income?

1 Very easy

Q11 (1)

Single code

2 Quite easy


4 Quite difficult

5 Very difficult

190 of 264

Q22

Does the site provide an explanation for how it calculates the results of a comparison? How does it determine which offer is most relevant/appropriate to you?

1 Yes



Q23 Using the following scale, please rate how easy or difficult it was to find information about the calculations used?

1 Very easy

Q22 (1)

Single code

2 Quite easy


4 Quite difficult

5 Very difficult

Q24

Does the comparison site include any ratings (star ratings, satisfaction scores) for the provider and/or product? Please note, this information may only be displayed on the results page and may need you to conduct a search in order to view the information.

1 Yes

All Single code

2 No


Q25 Does the site provide an explanation for how it calculates these ratings?

1 Yes

Q24 (1)

Single code 2 No


Q26a Do the ratings include customer reviews/scores

1 Yes

Q24 (1)

Single code 2 No


191 of 264

Q26b

Using the following scale, please rate how easy or difficult it was to find information about how the site calculates the service ranking?

1 Very easy

Q25 (1)

Single code

2 Quite easy


4 Quite difficult

5 Very difficult

Q27 Does the site have a policy for dealing with rogue or fraudulent providers?

1 Yes



Q28 Can customers report any suspicions of fraudulent activity directly to the comparison site?

1 Yes



Q30

Using the following scale, please rate how easy or difficult it was to find information about the site‟s approach to rogue or fraudulent activity?

1 Very easy

Q29 (1)

Single code

2 Quite easy


4 Quite difficult

5 Very difficult

Q29 Does the site highlight how it will respond to reports of fraudulent or rogue activity?

1 Yes



Q31 Is the complaints process clear and easy to find?

1 Yes



Q32 Can customer issue a complaint online?

1 Yes

Q31 (1)

Single code 2 No


Q33 Does the site specify the number of the days it will take to respond to customer complaints?

1 Yes

Q31 (1)

Single code 2 No


Q34 How many? All Open

192 of 264

Q37

Does the site include a glossary of terms or criteria used? (e.g. payment options, flexible rate, capped or fixed rates)

1 Yes



Q38 Which of the following criteria/terminology is included?

1

Q37 (1)

Multi code

2

3

4

5

Q38 is market specific, please see additional sheet for codes.

There is no Q40

Q41

Does the site provide or signpost you to any of the following types or sources of additional information? Please tick all that apply.

1

All Multi code

2

3

4

5

6

Q41 is market specific, please see additional sheet for codes.

Q35

Using the following scale, please rate how easy or difficult it was to find information about the site‟s approach to customer complaints?

1 Very easy

Q31 (1)

Single code

2 Quite easy


4 Quite difficult

5 Very difficult

Q36 Can you find a „code of conduct‟ on the site?

1 Yes



Q39 Using the following scale, please rate how easy or difficult it was to find the glossary of terms?

1 Very easy

Q37 (1)

Single code

2 Quite easy


4 Quite difficult

5 Very difficult

Q42 Using the following scale, please rate how easy or difficult it was to find this additional information?

1 Very easy Q41 (NOT 6)

Single code

193 of 264

CODES FOR Q38 AND Q41 (MARKET SPECIFIC)

Q38 Which of the following criteria/terminology is included?

1

Q37 (1)

Multi code

2

3

4

5

ENERGY

Monthly Direct Debit

Quarterly Direct Debit

Prepayment meter

Payment by cash, cheque, debit or credit card (quarterly)

Kilowatt hour (kWh)

None of the above

BROADBAND

Wireless broadband

Measures of data/speed (megabyte, kilobyte, gigabyte)

Router

Internet service provider (ISP)

None of the above

MOBILE PHONE

Android/Smart phone

Network provider/coverage

Pay as you go (pre pay)/Contract

Other technical terminology

None of the above

CAR INSURANCE

No claims bonus

Insurance excess (voluntary/compulsory)

Policy exclusions

Insurance premium tax

None of the above

HOME INSURANCE

Accidental damage

Insurance excess (voluntary/compulsory)

Rebuild costs

Insurance premium tax

None of the above

RAIL FARES

Explanation of types of tickets (anytime, off peak, super off peak)

None of the above

PACKAGE HOLIDAYS

Catering types (all inclusive, full board, half board etc.)

Departure taxes/local fees

Supplier failure protection

None of the above

194 of 264

Q41

Does the site provide or signpost you to any of the following types or sources of additional information? Please tick all that apply.

1

All Multi code

2

3

4

5

ENERGY

Energy efficiency advice

Best practice in comparison sites

Contact details for external agencies (such as Energy Saving Trust)

Information about grants schemes (e.g. Warm Front)

Consumer Focus website

None of the above

BROADBAND

Broadband speed tester


Contact details for external agencies (such as Ofcom)

Information about speeding up connections

Latest sector news

None of the above

MOBILE PHONE

Information about recycling old mobile phones


Contact details for external agencies (such as Ofcom)

Latest sector news

None of the above

CAR INSURANCE

Car maintenance tips/hints


Contact details for external agencies (such as ABI)

Advice about buying a car (vehicle checks)

Advice about insurance for specific groups (e.g. young drivers, additional drivers)

Latest sector news

None of the above

HOME INSURANCE

Home security tips


Contact details for external agencies (such as ABI)

Advice about insurance for specific types of property (e.g. holiday homes, rented)

Latest sector news

None of the above

RAIL FARES

Visitor information for destinations


Contact details for external agencies (such as ABTA, ATOL)

Advice about rail travel for specific groups (e.g. minors travelling alone, with cycles)

Latest sector news

None of the above

PACKAGE HOLIDAYS

Visitor information for destinations


Contact details for external agencies (such as ABTA, ATOL)

Advice about travel for specific groups (e.g. minors travelling alone)

Latest sector news

None of the above

195 of 264

CONSUMER FOCUS PRICE COMPARISON WEBSITES RESEARCH: ENERGY MYSTERY SHOPPER FORM


QA What is the market for your mystery shop?

1 Broadband

All

From allocation Single code QA to QG will be common for all shops

2 Energy

3 Car insurance

4 Home insurance

5 Mobile phones

6 Package holidays

7 Rail fares

QBi Tick the price comparison website you are to visit.

1 TheEnergyShop.com

All From allocation

2 Ukpower

3 Uswitch

4 Simply Switch

5 Energy Helpline

6 Fuel Switch

7 Money Supermarket

8 Confused.com

9 Unravelit

10 Which Switch

11 Compare and save

12 GoCompare

13 Compare the Market

14 Energy Choices

15 Spot Energy

16 Whos Cheapest

17 Home Advisory Service

18 Save on your bills

QBii What is the 3 digit number of the price comparison website you are to visit for this shop?


QCi Is this an accredited website?






5 Not accredited

QCii Is this a Big 4 website? 1 Yes

All From allocation Single code 2 No

QD

At a certain point you will be instructed to select one product from the first 5 that are listed. Tick the one that you are to select for this particular shop.

1 First product


2 Second product

3 Third product

4 Fourth product

5 Fifth product

QE

Please tick what you are doing after each visit. You have separate instructions on what each entails.

1 Straight after my initial search and straight after each of my 3 repeat searches I will clear my browsing data QA

(6,7)

From allocation Single code 2

At some point between each of my 3 repeat searches I will do at least two sector related searches

QF 1 Network provider website (e.g. Orange, O2) QA Single code

196 of 264

Which type of supplier site are you visiting?

2 Phone manufacturer website (e.g. Nokia, Apple) (5)

QG What is the 3 digit number of the scenario you are to follow for this shop?


From the website homepage, find where you can get comparative quotes for different energy tariffs. Input appropriate details for the scenario you have been assigned then answer the following question. N.B. Make a note of the details you enter so that you can replicate them when you visit the supplier website.

Q1

What information was it possible for you to input in order to get an initial list of quotes? You should tick all the items available, even if you did not use them. *N.B. If this is an option you should tick that you don‟t want to restrict the search only to products available via the price comparison site/only to products you can switch to today

1 Name

All Multi-code

2 Postal address

3 E-mail address

4 Whether looking for gas only, electricity only, or both

5 Energy usage (£ or kWh)

6 Whether currently an Economy 7 customer

7 Current supplier(s)

8 Current tariff(s)

9 Current means of payment (e.g. Direct Debit)

10 Future means of payment (i.e. for the new account)

11 Whether want to include green/environmental tariffs in search results

12 Other tariff type preferences (e.g. fixed, variable, online)

13

*Whether want to restrict the search only to products available via the price comparison site / whether wanted to see only products that you can switch to today

14 Other (specify)

The next few questions are about the initial list of quotes that appears once you have submitted your details.

Q2

Did the initial list of tariffs appear to be based on the criteria you input, or did it include tariffs that ignored those requirements? For example, if you asked for a particular tariff type (e.g. fixed price or green/environmental) was the list restricted to this type only.

1 Yes, list appeared to be based on all of my criteria

All Single code

2 No, list included tariffs that did not match my criteria

Q3 Please give more information on how some of the tariffs did not match your criteria. Q2 (2)

Open

Q4

Was there an explanation as to how the initial list of quotes had been ordered? For example, it might say they were ordered on price.

Yes, there was an explanation that the list was ordered….

All Single code

1 …on price

2 …on customer ratings

3 …in some other way (specify)

No, there was not an explanation but I could see that the list was ordered…

4 …on price



No, there was not an explanation…

7 …and it was not obvious how the list had been ordered

Q5

Was it possible to do each of the following?

All Radio grid. Single code for each

a) Reorder the initial list of tariffs (e.g. sort them on price or some other factor)

1 Yes

197 of 264

b) Filter the original list (e.g. on payment type, annual or monthly cost)

2 No

Q6

What factors could the original list be reordered on, i.e. rearranged into a sequence different to the one you saw initially? N.B. We are not talking here about filtering the list, we are asking about reordering it.

1 List could be reordered on price

Q5a (1)

Multi code

2 List could be reordered on supplier

3 List could be reordered on supplier‟s service rating

4 List could be reordered on customer ratings

5 List could be reordered on other factors (specify)

Q7 What factors could the original list be filtered on, i.e. cut down to include only specific types of tariff?

1 List could be filtered on price

Q5b (1)

Multi code

2 List could be filtered on supplier

3 List could be filtered on supplier‟s service rating

4 List could be filtered on payment type

5 List could be filtered on annual or monthly cost

6 List could be filtered on tariff type

7 List could be filtered on dual fuel or single fuel

8

List could be filtered on whether or not want to restrict the search only to products available via the price comparison site / whether want to see only products that you can switch to today

9 List could be filtered on other factors (specify)

Q8 On the first page of tariffs, is there a note anywhere to say that VAT is included in the prices?

1 Yes, and it was clearly displayed

All Single code

2 Yes, although it wasn‟t very clearly displayed

3 No, there was not a note about VAT being included

4 No, there was a note about VAT being excluded

Q9 On the first page of tariffs, is there a note anywhere to say when the prices were last updated?


All Single code 2 Yes, although it wasn‟t very clearly displayed

3 No, there was not a note about when the prices were last updated

Q10 On the first page of tariffs, how many options are listed?

1 1 to 9 All Single code

2 10 or more

Q11 On the first page of tariffs, is there….

1 …a summary of your current tariff i.e. the details you put in earlier

All Single code 2

…a link for you to access a summary of your current tariff

3 Neither of these

Q12

Did your scenario involve looking for dual fuel tariffs (i.e. gas and electricity) or a single fuel tariff (i.e. gas or electricity only)?

1 Dual fuel

All Single code

2 Single fuel

Q13

Are each of the following included in the summary of your current tariff? Take care – sometimes the “current spend” is not in the summary box, but stated separately above or below the list of tariffs. N.B. You can only tick the “not applicable” box for “Consumption threshold”

Q12 (2)

Radio grid. Single code for each. Code 3 is only allowed for the 4th item

Current supplier‟s name 1 Yes

Current tariff name 2 No

Unit rate (pence per kWh) 3 Not applicable to my current tariff

Consumption threshold (this is only applicable if there is more than one unit rate for the gas/electricity)

Current payment method

All discounts included in the calculation of the current tariff or a note to say no discounts apply

Current spend (£) per annum (can be estimated)

Current usage in kWh per annum (can be

198 of 264

estimated)

Q14

Are each of the following included in the summary of your current electricity tariff? Take care – sometimes the “current spend” is not in the summary box, but stated separately above or below the list of tariffs. N.B. You can only tick the “not applicable” box for “Consumption threshold”

Q12 (1)





Consumption threshold (this is only applicable if there is more than one unit rate for the electricity)



Current spend (£) per annum (can be estimated and can also be combined with gas spend to give one figure)

Current usage in kWh per annum (can be estimated but should be shown separately for gas and electricity – if not, tick “no”)

Q15

Are each of the following included in the summary of your current gas tariff? Take care – sometimes the “current spend” is not in the summary box, but stated separately above or below the list of tariffs. N.B. You can only tick the “not applicable” box for “Consumption threshold”

Q12 (1)





Consumption threshold (this is only applicable if there is more than one unit rate for the gas)



Current spend (£) per annum (can be estimated and can also be combined with electricity spend to give one figure)

Current usage in kWh per annum (can be estimated but should be shown separately for gas and electricity – if not, tick “no”)

Q16 At the end of the list of tariffs on the first page, are there any of the following?

1 Option to “view all”

All Multi code other than last option

2 Option to move to next set or page of results

3 Neither of these is applicable as there are no more tariffs to show

Q17

If the list is ordered by price leave it like that. If it isn‟t already in price order and you can reorder on that variable, please do so now. Please code which situation applies.

1 I have the tariff list in price order

All Single code

2 The tariff list is not in price order and I cannot reorder it on that variable

Q18

Scroll down the first *2-3 pages of tariffs. Are they all in price order? *If there‟s only one page just look at the tariffs on it

1 Yes, all were in price order Q17 (1)

Single code

2 No, some or all were not in price order

Browse the details provided for the first 5 tariffs in the list. This often requires you to click a button such as “Read more” or “Further information” for each one. N.B. You should do this for all 5, not just the one you will select later.

Q19

Did any of the first 5 tariffs in the list have any restrictions / limitations that were not mentioned in the search criteria when you first put in your details?

1 Paperless billing is a necessity

All Multi-code other than last option

2 Need to pass a credit check

199 of 264

N.B. We are not asking here about any additional costs – they are covered in the next question Please read the list below carefully and tick all that apply. If there are no additional restrictions / limitations please tick the last box.

3 Must manage account online

4 Not available until 6+ weeks time

5 Must pay by Direct Debit

6 Other restriction / limitation (specify)

7 No, no additional restrictions / limitations

Q20

Did any of the first 5 tariffs in the list have any additional costs that were not mentioned earlier? N.B. You should tell us both about costs that are unavoidable for someone choosing that tariff and those which are possible additions dependent on what they choose / how they act in the future (e.g. paying extra for paper bills, or having a cancellation fee). Please read the list below carefully and tick all that apply. If there are no additional costs please tick the last box.

1 Extra cost for paper bills


2 Cancellation fee / exit fee

3 Other additional cost (specify)

4 No, no additional costs

Check which of the first 5 tariffs in the list you are to select and look at the details provided for that particular tariff. If it is a tariff that you cannot apply for via the cost comparison website then choose either the one above or the one below it. You can spot a tariff that you cannot apply for because the “apply” button will be greyed out. This can be the case when you have asked the site to search all available tariffs, including those that are not available via the price comparison site or that you can‟t apply for today.

Q21a Was the tariff you selected greyed out? 1 Yes


Q21b

Who is the supplier for your chosen *tariff? Make a separate note of this, to remind you which supplier website to visit. *Remember, if the one you had to select was greyed out you should take the one above or below it

1 British Gas

All Single code

2 eDF

3 E.ON

4 nPower

5 Scottish Power

6 Southern Electric

7 Other (specify)

Q22

Write in the name of the tariff. Make a separate note of this, to remind you which tariff you will be looking for on the supplier website.

All Open

Q23 Write in the (estimated) annual or monthly cost for this tariff. Make a separate note of this, to compare to the price on the supplier website.

All Open

Q24

If you were a “real” customer, do you at this stage have sufficient information about the tariff (cost, features, restrictions, etc.) to feel comfortable making an application?

1 Yes, I have sufficient information to make an application

All Single code

2 No, I would need more information before making an application

Q25 What additional information would you need? Q24 (2)

Open

200 of 264

Now click on the “apply” button for your chosen tariff. N.B. We are not going to go through the whole of the application process.

Q26 Which of the following best describes what happens when you click on the application button?

1 Stay within the price comparison website

All Single code 2 Redirected to the supplier website

3 Redirected to another price comparison website

4 Other outcome (specify)

Q27

In the first screen of the application process, were there any restrictions / limitations that had not been mentioned previously? N.B. We are not asking here about any additional costs – they are covered in the next question Please read the list below carefully and tick all that apply. If there are no additional restrictions / limitations please tick the last box.









Q28

In the first screen of the application process, were there any additional costs that had not been mentioned previously? N.B. You should tell us both about costs that are unavoidable for someone choosing that tariff and those which are possible additions dependent on what they choose / how they act in the future (e.g. paying extra for paper bills, or having a cancellation fee). Please read the list below carefully and tick all that apply. If there are no additional costs please tick the last box.






Q29 Now some questions about the website as a whole. Was there anything on the website that you found confusing?

1 Yes All Single code

2 No

Q30 What was that? Q29 (1)

Open

Q31 Was there anything on the website that you found misleading?


2 No


Open

Q33 How easy or difficult was it to enter search information on this website?

1 Very easy

All Single code

2 Quite easy

3 Neither easy nor difficult

4 Quite difficult

5 Very difficult

Q34 From your visit today, is your overall impression of this website positive or negative?

1 Very positive

All Single code

2 Quite positive

3 Neither positive nor negative

4 Quite negative

5 Very negative

Q35 Why do you say that? All Open

201 of 264

Q36 If there is anything else you would like to tell us about this price comparison website and your visit today please do so in the box below.

All Open. Optional

Open a new window and go to the supplier‟s website for the tariff you selected (e.g. British Gas, E.ON, etc.). Act as if you were a potential customer looking to see what tariffs are available from this supplier. Go to the appropriate section of the website in order to get a quote for switching. Enter the details for the scenario in exactly the same way as you did on the price comparison website. Look through the list of results until you find the tariff you selected in the price comparison website.

Q37 What was the outcome of looking for the tariff on the supplier‟s website?

1 I was able to find the same tariff All Single code

2 I was not able to find the same tariff

Q38

Write in details of what you did and what the outcome was (e.g. if you found tariffs with similar names write them in, if you found the tariff finished several weeks ago let us know). There are no more questions to complete.

Q37 (2)

Open

Q39 Write in the (estimated) annual or monthly cost for this tariff. Q37 (1)

Open

Q40

How does the supplier site cost compare to the price comparison site cost? N.B. If one was given as a monthly sum and one was an annual sum, multiply or divide by 12 to make them comparable *If the costs are only different by £1 or less you can still use this code

1 *The supplier site cost was exactly the same as the price comparison site cost

Q37 (1)

Single code 2 The supplier site cost was lower than the price comparison site cost

3 The supplier site cost was higher than the price comparison site cost

Q41 Can you see the reason for the cost difference?

1 Yes Q40 (2,3)

Single code 2 No

Q42 Write in why the supplier site cost is lower than the price comparison site cost.

Q40 (2) + Q41 (1)

Open

Q43 Write in why the supplier site cost is higher than the price comparison site cost.

Q40 (3) + Q41 (1)

Open

Q44

Read through the tariff details on the supplier site and compare them to the tariff details on the price comparison website. Does the supplier site have any additional details about the tariff?

1 Yes, supplier site has additional details

Q37 (1)

Single code

2 No, supplier site does not have additional details

Q45 What are the additional details on the supplier site?


Q44 (1)

Multi code





6 Cancellation fee


8 Other (specify)

Q46 If there is anything else you would like to tell us about how the tariff is presented and described on the supplier site versus the price comparison site please do so in the box below.

Q37 (1)

Open. Optional

202 of 264


CONSUMER FOCUS PRICE COMPARISON WEBSITES RESEARCH: BROADBAND MYSTERY SHOPPER FORM


1 Broadband

All


2 Energy

3 Car insurance

4 Home insurance

5 Mobile phones

6 Package holidays

7 Rail fares


1 Broadband Choices

All From allocation

2 Broadband.co.uk

3 Cable.co.uk

4 HomePhone Choices

5 Simply Digital

6 Money Supermarket

7 Confused.com

8 GoCompare


10 Broadbandgenie

11 Money Saving Expert

12 Tesco Compare

13 Money.co.uk

14 Uswitch

15 Broadband Expert

16 Compare Broadband









5 Not accredited



QD


1 First product


2 Second product

3 Third product

4 Fourth product

5 Fifth product

QE



(6,7)



QF


1 Network provider website (e.g. Orange, O2) QA (5)

Single code 2 Phone manufacturer website (e.g. Nokia, Apple)

203 of 264



From the website homepage, find where you can get comparative quotes for different broadband packages. Input appropriate details for the scenario you have been assigned then answer the following question. N.B. Make a note of the details you enter so that you can replicate them when you visit the supplier website.

Q1

What information was it possible for you to input in order to get an initial list of quotes? You should tick all the items available, even if you did not use them.

1 Name

All Multi-code

2 Postcode

3 E-mail address

4 Phone number

5 Whether looking for broadband only or broadband package (e.g. with home phone and/or T.V)

6 Minimum speed

7 Usage limit

8 Provider (current or preferred)

9 Monthly cost

10 Annual cost

11 Additional features (such as online security)

12 Inclusive calls

13 Contract length

14 Other (specify)


Q2

Did the initial list of packages appear to be based on the criteria you input, or did it include packages that ignored those requirements? For example, if you asked for a particular package type (e.g. broadband only or broadband and home phone) was the list restricted to this type only.


All Single code

2 No, list included packages that did not match my criteria

Q3 Please give more information on how some of the packages did not match your criteria. Q2 (2)

Open

Q4



All Single code

1 …on 1st year costs

2 …on monthly costs

3 …on reliability

4 …on contract length

5 …on speed

6 …on usage

7 …on special offer


9 …on set up fee

10 ..on popularity



…on 1st year costs

…on monthly costs

…on reliability

…on contract length

…on speed

…on usage

…on special offer

…on customer ratings

…on set up fee

…in some other way (specify)



204 of 264

Q5



c) Reorder the initial list of package (e.g. sort them on 1st year costs, speed, or some other factor)

1 Yes

d) Filter the original list (e.g. on monthly cost, provider, usage limit)

2 No

Q6


1 List could be reordered on 1st year costs

Q5a (1)

Multi code

2 List could be reordered on monthly costs

3 List could be reordered on reliability

4 List could be reordered on provider

5 List could be reordered on speed

6 List could be reordered on usage

7 List could be reordered on contract length

8 List could be reordered on special offers


10 List could be reordered on one off costs

11 List could be reordered on popularity


Q7 What factors could the original list be filtered on, i.e. cut down to include only specific types of packages?

1 List could be filtered on 1st year costs

Q5b (1)

Multi code

2 List could be filtered on monthly costs

3 List could be filtered on provider

4 List could be filtered on usage

5 List could be filtered on additional features

6 List could be filtered on contract length

7 List could be filtered on inclusive calls

8 List could be filtered on speed

9 List could be filtered on set up costs

10 List could be filtered on tv channels


Q8

Is it clear whether the broadband speed displayed is an estimate (specific to your location) or the „best‟ achievable speed advertised?

1 Yes, and it was clearly labelled

All Single code

2 Yes, although it wasn‟t very clearly labelled

3 No, there was not a note about what „speed‟ is displayed

4 Broadband speed was not included in the results

Q9 Is it possible to change the definition of „speed‟ and reorder the options?

1 Yes, it is possible to change the type of „speed‟ and reorder the results Q6 (5) Single code

2 No, the site only displays one type of „speed‟

Q10 Which type of „speed‟ can be used to filter the packages

1 The packages can be filtered by the estimated broadband speed for your area

Q7 (8) Multi code 2 The packages can be filtered by the „best‟ possible broadband advertised (up to...)

3 The packages can be filtered by speed but it is not clear which type of speed is used.

Q11 On the first page of packages, is there a note anywhere to say when the prices were last updated?




Q12a On the first page of packages, how many options are listed?


2 10 or more

Q12b On the first page of packages, is there….

1 …a summary of your search criteria i.e. the details you put in earlier All Multiple code

2 …a link for you to access and edit your search

205 of 264

criteria

3 Neither of these

Q13

Did your scenario involve looking for broadband only, broadband + home phone + tv, or a mobile broadband package?

1 Broadband only

All Single code 2 Broadband and home phone and tv

3 Mobile broadband

Q14 At the end of the list of packages on the first page, are there any of the following?




3 Neither of these is applicable as there are no more packages to show

Q15

If the list is ordered by „1st year cost‟ (annual cost) leave it like that. If it isn‟t already in 1st year cost order and you can reorder on that variable, please do so now. Please code which situation applies.

1 I have the package list in price order (1st year cost)

All Single code

2 The package list is not in price order and I cannot reorder it based on 1st year costs

Q16

Scroll down the first *2-3 pages of packages. Are they all in price order? *If there‟s only one page just look at the packages on it


Single code


Q17

Do the results (except those offered by Tesco) include a telephone number for customers to apply over the phone?

1 Yes, there is a telephone number clearly displayed against the various packages

All Single code

2 No, there are no telephone numbers displayed against the available packages

Q18 Does the site include a breakdown of what is included in the 1st year costs?

1 Yes, there is a clear breakdown of what is included in the 1st year cost

All Single code

2 No, there is no breakdown of what costs are included in the 1st year cost

Browse the details provided for the first 5 packages in the list. This often requires you to click a link under „package details‟ or a button such as “More info” or “Further information” for each one. N.B. You should do this for all 5, not just the one you will select later.

Q19

Did any of the first 5 packages in the list have any restrictions / limitations that were not mentioned in the search criteria when you first put in your details? N.B. We are not asking here about any additional costs – they are covered in the next question Please read the list below carefully and tick all that apply. If there are no additional restrictions / limitations please tick the last box.

1 Must have/switch to specified landline (e.g. BT)


2 Not available in your area

3 Usage restrictions

4 Home visit required for installation


6 Existing customers/new customers only

7 Minimum length of contract



Q20 Did any of the first 5 packages in the list have any additional costs that were not

1 Extra cost for line rental All Multi-code other than last option

206 of 264

mentioned earlier? N.B. You should tell us both about costs that are unavoidable for someone choosing that packages and those which are possible additions dependent on what they choose / how they act in the future (e.g. paying extra for paper bills, or having a cancellation fee). Please read the list below carefully and tick all that apply. If there are no additional costs please tick the last box.


3 Equipment costs (router)

4 Installation costs

5 Delivery costs

6 Activation charge



Check which of the first 5 packages in the list you are to select and look at the details provided for that particular package. If it is a package that you cannot apply for via the cost comparison website then choose either the one above or the one below it. You can spot a package that you cannot apply for because the “apply” button will be greyed out. This can be the case when you have asked the site to search all available packages, including those that are not available via the price comparison site or that you can‟t apply for today.

Q21a Was the package you selected greyed out?


2 No

Q21b

Who is the supplier for your chosen *package? Make a separate note of this, to remind you which supplier website to visit. *Remember, if the one you had to select was greyed out you should take the one above or below it

1 AOL

All Single code

2 British Telecom

3 O2

4 Orange

5 Talk Talk

6 Plusnet

7 Sky

8 Tesco

9 Virgin

10 BE Broadband

11 Tiscali

12 Other (specify)

Q22

Write in the name of the package. Make a separate note of this, to remind you which package you will be looking for on the supplier website.

All Open

Q23 Write in the 1st year or monthly cost for this package. Make a separate note of this, to compare to the price on the supplier website.

All Open

Q24

If you were a “real” customer, do you at this stage have sufficient information about the package (cost, features, restrictions, etc.) to feel comfortable making an application?


All Single code


207 of 264


Open

Now click on the “apply” button for your chosen package. N.B. We are not going to go through the whole of the application process.






Q27


1 Must have/switch to specified landline (e.g. BT)








8 Other restriction/limitation


Q28

In the first screen of the application process, were there any additional costs that had not been mentioned previously? N.B. You should tell us both about costs that are unavoidable for someone choosing that package and those which are possible additions dependent on what they choose / how they act in the future (e.g. paying extra for paper bills, or having a cancellation fee). Please read the list below carefully and tick all that apply. If there are no additional costs please tick the last box.

1 Extra cost for line rental



3 Equipment costs (router)

4 Installation costs

5 Delivery costs

6 Activation charge



Q29

Now some questions about the comparison website as a whole. Was there anything on the website that you found confusing?

1 Yes



Open

Q31 Was there anything on the comparison website that you found misleading?


2 No


Open

Q33 How easy or difficult was it to enter 1 Very easy All Single code

208 of 264

search information on this website? 2 Quite easy


4 Quite difficult

5 Very difficult


1 Very positive

All Single code

2 Quite positive


4 Quite negative

5 Very negative



All Open. Optional

Open a new window and go to the supplier‟s website for the broadband package you selected (e.g. British Telecom, Sky etc.). Act as if you were a potential customer looking to see what packages are available from this provider. Go to the appropriate section of the website in order to get a quote. Enter the details for the scenario in exactly the same way as you did on the price comparison website. Look through the list of results until you find the package you selected in the price comparison website.

Q37 What was the outcome of looking for the package on the supplier‟s website?

1 I was able to find the same package All Single code

2 I was not able to find the same package

Q38

Write in details of what you did and what the outcome was (e.g. if you found packages with similar names write them in, if you found the package finished several weeks ago let us know). There are no more questions to complete.

Q37 (2)

Open

Q39 Write in the 1st year or monthly cost for this package. Q37 (1)

Open

Q40



Q37 (1)




1 Yes Q40 (2,3)

Single code 2 No


Q40 (2) + Q41 (1)

Open


Q40 (3) + Q41 (1)

Open

Q44

Read through the package details on the supplier site and compare them to the package details on the price comparison website. Does the supplier site have any additional details about the package?


Q37 (1)

Single code


209 of 264


1 Must have BT landline

Q44 (1)

Multi code







8 Activation charge

9 Other (specify)

Q46 If there is anything else you would like to tell us about how the package is presented and described on the supplier site versus the price comparison site please do so in the box below.

Q37 (1)

Open. Optional

210 of 264

CONSUMER FOCUS PRICE COMPARISON WEBSITES RESEARCH: CAR INSURANCE MYSTERY SHOPPER FORM



1 Broadband

All


2 Energy

3 Car insurance

4 Home insurance

5 Mobile phones

6 Package holidays

7 Rail fares


1 Beatthatquote

All From allocation

2 Tesco Compare

3 Money Supermarket

4 Confused.com

5 GoCompare


7 Money facts.co.uk

8 Uswitch

9 Quotezone

10 Compare and save

11 Know your money

12 Moneynet.co.uk

13 Money Expert

14 Love Money

15 Fairinvestment.co.uk









5 Not accredited



QD


1 First product

All From allocation

2 Second product

3 Third product

4 Fourth product

5 Fifth product

QE



(6,7)



QF




211 of 264



From the website homepage, find where you can get comparative quotes for car insurance. Input appropriate details for the scenario you have been assigned then answer the following question. N.B. Make a note of all the details you enter so that you can replicate them when you visit the supplier website.

Q1


1 Name and address

All Multi-code

2 Date of birth

3 E-mail address

4 Phone number

5 Marital status

6 Employment

7 Driving license details (type, how long have you had it)

8 Number of cars in the household

9 Car details (reg. number, modifications)

10 Claims and convictions

11 Additional drivers

12 Type of insurance required

13 No claims bonus

14 Expected mileage

15 Commencement date

16 Voluntary excess amount

17 Preferred payment option

18 Existing insurance provider

19 Renewal price

20 Other (specify)

As you work through your visit keep an eye out for any notification that a particular provider has had formal complaints against it (which might be beside quotes from that supplier or as a note somewhere else on the page). At the end of the shop you will be asked to record whether or not you saw any warnings of this type. The next few questions are about the initial list of quotes that appears once you have submitted your details.

Q2

Did the initial list of results appear to be based on the criteria you input, or did it include quotes that ignored those requirements? For example, if you asked for a particular insurance type (e.g. comprehensive only) was the list restricted to this type only.


All Single code

2 No, list included results that did not match my criteria

Q3 Please give more information on how some of the results did not match your criteria. Q2 (2)

Open

Q4



All Single code

1 …on annual premium

2 …on monthly premium

3 …on voluntary excess amount

4 …on inclusive extras (breakdown cover, legal cover)






9 …on voluntary excess amount

10 …on inclusive extras (breakdown cover, legal cover)




212 of 264


14 Only got one quote

Q5


All except code 14 at Q4

Radio grid. Single code for each

e) Reorder the initial list of quotes (e.g. sort them on annual premium, voluntary excess or some other factor)

1 Yes

f) Filter the original list (e.g. sort them on annual premium, voluntary excess or some other factor)

2 No

g) Compare a small number of quotes in more detail

Q6


1 List could be reordered on annual premium

Q5a (1)

Multi code

2 List could be reordered on monthly premium

3 List could be reordered on voluntary excess amount


5 List could be reordered on breakdown cover

6 List could be reordered on courtesy car

7 List could be reordered on legal cover

8 List could be reordered on windscreen cover


10 List could be reordered on total cost of monthly instalments


Q7 What factors could the original list be filtered on, i.e. cut down to include only specific types of policies?

1 List could be filtered on voluntary excess

Q5b (1)

Multi code

2 List could be filtered on provider

3 List could be filtered on total excess (compulsory and voluntary)

4 List could be filtered on breakdown cover

5 List could be filtered on courtesy car

6 List could be filtered on legal cover

7 List could be filtered on windscreen cover


Q8 On the first page of quotes, is there a note anywhere to say that insurance premium tax is included in the prices?


All Single code


3 No, there was not a note about insurance premium tax being included

4 No, but there was a note about being insurance premium tax excluded

Q9 On the first page of quotes, is there a note anywhere to say when the prices were last updated?




Q10 On the first page of quotes, how many options are listed?


2 10 or more

N.B. There is no Q11

Q12 On the first page of quotes, is there….

1 …a summary of your search criteria i.e. the details you put in earlier

All Multiple code 2

…a link for you to access and edit your search criteria

3 Neither of these

213 of 264

Q13 At the end of the list of results on the first page, are there any of the following?




3 Neither of these is applicable as there are no more policies to show

Q14 Did your scenario involve looking for a comprehensive or third party, fire and theft policy?

1 Comprehensive insurance

All Single code

2 Third party, fire and theft

Q15

If the list is ordered by annual premium leave it like that. If it isn‟t already in price order and you can reorder on that variable, please do so now. Please code which situation applies.

1 I have the results list in price order (annual premium)

All Single code

2 The results list is not in price order and I cannot reorder it based on annual premium costs

Q16

Scroll down the first *2-3 pages of quotes. Are they all in price order? *If there‟s only one page just look at the quotes on it

1 Yes, all were in price order

Q15 (1)

Single code 2 No, some or all were not in price order

Q17

Do the results include a telephone number for customers to apply over the phone?

1 Yes, there is a telephone number clearly displayed against the various quotes

All Single code

2 No, there are no telephone numbers displayed against the available quotes

Browse the details provided for the first 5 policies in the list. This may require you to select the first five policies and click a link to „”Compare all” or a button such as “More info” or “Further information” for each quote. N.B. You should do this for all 5, not just the one you will select later.

Q18

Did any of the first 5 policies in the list have any restrictions / limitations that were not mentioned in the search criteria when you first put in your details? N.B. We are not asking here about any additional costs – they are covered in the next question Please read the list below carefully and tick all that apply. If there are no additional restrictions / limitations please tick the last box.



2 Must have specified no claims bonus

3 Use of car (e.g. only social use, social and commuting)

4 Limited payment options (only annual/monthly)

5 Age restrictions



Q19

For each of the following, please say whether or not it was an additional cost and, if so, what type it was.


Courtesy car 1 Additional cost that was automatically added and I was unable to remove it

Breakdown cover 2 Additional cost that was automatically added but I had a choice to remove it

Legal cover 3 Additional cost that was not automatically added, which I could choose to add to the cost

Windscreen cover 4 Not an additional cost that appeared at all / was no extra cost for this


Check which of the first 5 policies in the list you are to select and look at the details provided for that particular policy.

214 of 264

If it is a policy that you cannot apply for via the cost comparison website then choose either the one above or the one below it. You can spot a policy that you cannot apply for because the “apply” button will be greyed out.

Q20a Was the policy you selected greyed out? 1 Yes


Q20b

Who is the supplier for your chosen *policy? Make a separate note of this, to remind you which supplier website to visit. *Remember, if the one you had to select was greyed out you should take the one above or below it

1 Admiral

All Single code

2 AA

3 Bell

4 Churchill

5 Co-operative insurance

6 Diamond

7 Elephant

8 Endsleigh

9 Lloyds TSB

10 Privilege

11 RAC

12 SAGA

13 Shelia‟s Wheels

14 Swinton

15 Tesco

16 Other (specify)

Q21

Write in the monthly premium for this policy or, if your scenario involved you wanting to pay the full cost up front, write in the full amount for the year. Make a separate note of this, to compare to the price on the supplier website.

All Open

Q22 Write in the details of the compulsory excess on the policy (including those applied to specific repairs such as windscreen repairs).

All Open

Q23a Can you pay in monthly instalments if you want to?

1 Yes


3 Don‟t know/doesn‟t specify

Q23b

What additional information is shown for monthly instalments? Please tick all that apply.

1 Number of instalments

Q23a (1)

Multi code

2 Amount due at each instalment

3 APR payable

4 Total cost of paying monthly

5 Difference between paying monthly or as an annual premium

6 Deposit

7 Other details (please specify)

Q24 Can you pay the cost for the year in a lump sum up front if you want to?

1 Yes



Q25

If you were a “real” customer, do you at this stage have sufficient information about the policy (cost, level of cover, restrictions, etc.) to feel comfortable making an application?


All Single code



Open

Now click on the “apply” button for your chosen policy. N.B. We are not going to go through the whole of the application process.






215 of 264

Q28







5 Age restrictions



Q29



Courtesy car 1 Additional cost that was automatically added and I was unable to remove it

Breakdown cover 2 Additional cost that was automatically added but I had a choice to remove it

Legal cover 3 Additional cost that was not automatically added, which I could choose to add to the cost

Windscreen cover 4 Not an additional cost that appeared at all / was no extra cost for this


Q30a At any point during your visit, did you see notification that a particular provider had had formal complaints against it?


2 No

Q30b


1 Yes



Open



2 No


Open


1 Very easy

All Single code

2 Quite easy


4 Quite difficult

5 Very difficult


1 Very positive

All Single code

2 Quite positive


4 Quite negative

5 Very negative


Q37 If there is anything else you would like to tell us about this price comparison website and your All Open.

216 of 264

visit today please do so in the box below. Optional

Open a new window and go to the supplier‟s website for the car insurance policy you selected (e.g. Churchill, AA etc.). Act as if you were a potential customer looking to see what policies are available from this provider. Go to the appropriate section of the website in order to get a quote. Enter the details for the scenario in exactly the same way as you did on the price comparison website.

Q38 What was the outcome of looking for the policy on the supplier‟s website?

1 I was able to find the same policy All Single code

2 I was not able to find the same policy

Q39

Write in details of what you did and what the outcome was (e.g. if you were unable to get insurance cover). There are no more questions to complete.

Q38 (2)

Open

Q40 Write in the monthly premium for this policy or, if your scenario involved you wanting to pay the full cost up front, write in the full amount for the year.

Q38 (1)

Open

Q41



Q38 (1)




1 Yes Q41 (2,3)

Single code 2 No


Q41 (2) + Q42 (1)

Open


Q41 (3) + Q42 (1)

Open

Q45

Read through the summary policy details on the supplier site and compare them to the policy details on the price comparison website. Does the supplier site have any additional details about the policy?


Q38 (1)

Single code




Q45 (1)

Multi code




5 Age restrictions

6 Amount of excess (voluntary or compulsory)

7 Inclusive extras (breakdown cover, legal cover)

8 Other (specify)


Q38 (1)

Open. Optional

217 of 264

CONSUMER FOCUS PRICE COMPARISON WEBSITES RESEARCH: HOME INSURANCE MYSTERY SHOPPER FORM



1 Broadband

All


2 Energy

3 Car insurance

4 Home insurance

5 Mobile phones

6 Package holidays

7 Rail fares


1 Beatthatquote

All From allocation

2 Tesco Compare

3 Money Supermarket

4 Confused.com

5 GoCompare


7 Money facts.co.uk

8 Uswitch

9 Quotezone

10 Compare and save

11 Know your money

12 Moneynet.co.uk

13 Money Expert

14 Love Money

15 Fairinvestment.co.uk









5 Not accredited



QD


1 First product

All From allocation

2 Second product

3 Third product

4 Fourth product

5 Fifth product

QE



(6,7)



QF




218 of 264



From the website homepage, find where you can get comparative quotes for home insurance. Input appropriate details for the scenario you have been assigned then answer the following question. N.B. Make a note of all the details you enter so that you can replicate them when you visit the supplier website.

Q1


1 Name and address

All Multi-code

2 Date of birth

3 E-mail address

4 Phone number

5 Marital status

6 Employment

7 Type of cover required (buildings only, contents only or building and contents)

8 Details about use of property

9 Ownership of property

10 Length of stay at property (when did you buy it?)

11 Policy holder details (additional names, do you smoke, declared bankrupt?)

12 Property details (e.g. no. of rooms, construction, when built)

13 Whether or not made a previous home insurance claim

14 Value of the property

15 Estimated cost of rebuilding the property

16 Voluntary excess

17 Value and details of contents (be to covered by policy)

18 Existing insurance provider

19 Renewal price

20 Renewal date

21 Whether area has Neighbourhood Watch

22 Property alarms (smoke, burglar)

23 Locks on doors / windows

24 Other (specify)

As you work through your visit keep an eye out for any notification that a particular provider has had formal complaints against it (which might be beside quotes from that supplier or as a note somewhere else on the page). At the end of the shop you will be asked to record whether or not you saw any warnings of this type. The next few questions are about the initial list of quotes that appears once you have submitted your details.

Q2

Did the initial list of results appear to be based on the criteria you input, or did it include quotes that ignored those requirements? For example, if you asked for a particular insurance type (e.g. comprehensive only) was the list restricted to this type only.


All Single code

2 No, list included results that did not match my criteria

Q3 Please give more information on how some of the results did not match your criteria. Q2 (2)

Open

Q4



All Single code



3 …on buildings excess

4 …on contents excess

5 …on inclusive extras (alternative accommodation, new for old replacement)



No, there was not an explanation but I could see that the

219 of 264

list was ordered…



10 …on buildings excess

11 …on contents excess

12 …on inclusive extras (alternative accommodation, new for old replacement)





16 Only got one quote

Q5


All except code 16 at Q4


h) Reorder the initial list of quotes (e.g. sort them on annual premium, excess or some other factor)

1 Yes

i) Filter the original list (e.g. sort them on annual premium, excess or some other factor)

2 No

j) Compare a small number of quotes in more detail

Q6


1 List could be reordered on annual premium

Q5a (1)

Multi code

2 List could be reordered on monthly premium

3 List could be reordered on buildings excess amount

4 List could be reordered on contents excess amount



7 List could be reordered on additional features (such as alternative accommodation or new for old)


Q7 What factors could the original list be filtered on, i.e. cut down to include only specific types of policies?

1 List could be filtered on annual premium

Q5b (1)

Multi code

2 List could be filtered on monthly premium

3 List could be filtered on total excess (voluntary/compulsory)

4 List could be filtered on additional features (such as alternative accommodation or new for old)


Q8 On the first page of quotes, is there a note anywhere to say that insurance premium tax is included in the prices?


All Single code


3 No, there was not a note about insurance premium tax being included

4 No, there was a note about being insurance premium tax excluded

Q9 On the first page of quotes, is there a note anywhere to say when the prices were last updated?




Q10 On the first page of quotes, how many options are listed?


2 10 or more

N.B. There is no Q11

Q12 On the first page of quotes, is there…. 1 …a summary of your search criteria i.e. the details you put in earlier

All Multiple code

220 of 264

2 …a link for you to access and edit your search criteria

3 Neither of these

Q13 At the end of the list of results on the first page, are there any of the following?




3 Neither of these is applicable as there are no more policies to show

Q14 Did your scenario involve looking for buildings and contents insurance, buildings only or contents only?

1 Buildings and contents insurance

All Single code 2 Buildings only

3 Contents only

Q15

If the list is ordered by annual premium leave it like that. If it isn‟t already in price order and you can reorder on that variable, please do so now. Please code which situation applies.

1 I have the results list in price order (annual premium)

All Single code

2 The results list is not in price order and I cannot reorder it based on annual premium costs

Q16

Scroll down the first *2-3 pages of quotes. Are they all in price order? *If there‟s only one page just look at the quotes on it


Q15 (1)


Q17

Do the results include a telephone number for customers to apply over the phone?

1 Yes, there is a telephone number clearly displayed against the various quotes

All Single code

2 No, there are no telephone numbers displayed against the available quotes

Browse the details provided for the first 5 policies in the list. This may require you to select the first five policies and click a link to „”Compare all” or a button such as “More info” or “Further information” for each quote. N.B. You should do this for all 5, not just the one you will select later.

Q18

Did any of the first 5 policies in the list have any restrictions / limitations that were not mentioned in the search criteria when you first put in your details? N.B. We are not asking here about any additional costs – they are covered in the next question Please read the list below carefully and tick all that apply. If there are no additional restrictions / limitations please tick the last box.



2 Must not have made a claim


4 Restrictions on location (e.g. are subject to flooding, subsistence)

5 Restrictions on type / use of property

6 Restrictions regarding sheds / outbuildings

7 Restrictions regarding fencing repair / replacement



Q19

For each of the following, please say whether or not it was an additional cost and, if so, what type it was. All


Legal cover 1 Additional cost that was automatically added and I

221 of 264

was unable to remove it

New for old replacement 2 Additional cost that was automatically added but I had a choice to remove it

Emergency helpline 3 Additional cost that was not automatically added, which I could choose to add to the cost

Alternative accommodation 4 Not an additional cost that appeared at all / was no extra cost for this

Home emergency cover

Replacement locks

Accidental damage

Garden cover


Check which of the first 5 policies in the list you are to select and look at the details provided for that particular policy. If it is a policy that you cannot apply for via the cost comparison website then choose either the one above or the one below it. You can spot a policy that you cannot apply for because the “apply” button will be greyed out.

Q20a Was the policy you selected greyed out? 1 Yes


Q20b

Who is the supplier for your chosen *policy? Make a separate note of this, to remind you which supplier website to visit. *Remember, if the one you had to select was greyed out you should take the one above or below it

1 AXA

All Single code

2 AA

3 Churchill

4 Co-operative insurance

5 Endsleigh

6 Lloyds TSB

7 LVE

8 Marks and Spencers

9 More than

10 Post Office

11 Privilege

12 SAGA

13 Shelia‟s Wheels

14 Swinton

15 Tesco

16 Zurich

17 Other (specify)

Q21

Write in the monthly premium for this policy or, if your scenario involved you wanting to pay the full cost up front, write in the full amount for the year. Make a separate note of this, to compare to the price on the supplier website.

All Open

Q22 Write in the details of the compulsory excess on the policy. All Open

Q23a Can you pay in monthly instalments if you want to?

1 Yes



Q23b

What additional information is shown for monthly instalments? Please tick all that apply.

1 Number of instalments

Q23a (1)

Multi code

2 Amount due at each instalment

3 APR payable

4 Total cost of paying monthly

5 Difference between paying monthly or as an annual premium

6 Deposit

7 Other details (please specify)

Q24 Can you pay the cost for the year in a lump sum up front if you want to?

1 Yes



222 of 264

Q25

If you were a “real” customer, do you at this stage have sufficient information about the policy (cost, level of cover, restrictions, etc.) to feel comfortable making an application?


All Single code



Open

Now click on the “apply” button for your chosen policy. N.B. We are not going to go through the whole of the application process.






Q28




2 Must not have made a claim


4 Restrictions on location (e.g. are subject to flooding, subsistence)

5 Restrictions on type/use of property

6 Restrictions regarding sheds / outbuildings

7 Restrictions regarding fencing repair / replacement



Q29



Legal cover 1 Additional cost that was automatically added and I was unable to remove it

New for old replacement 2 Additional cost that was automatically added but I had a choice to remove it

Emergency helpline 3 Additional cost that was not automatically added, which I could choose to add to the cost

Alternative accommodation 4 Not an additional cost that appeared at all / was no extra cost for this

Home emergency cover

Replacement locks

Accidental damage

Garden cover


Q30a At any point during your visit, did you see notification that a particular provider had had formal complaints against it?


2 No

Q30b Now some questions about the comparison website as a whole. Was there anything on the website that you


2 No

223 of 264

found confusing?


Open



2 No


Open


1 Very easy

All Single code

2 Quite easy


4 Quite difficult

5 Very difficult


1 Very positive

All Single code

2 Quite positive


4 Quite negative

5 Very negative



All Open. Optional

Open a new window and go to the supplier‟s website for the car insurance policy you selected (e.g. Endsleigh, AA etc.). Act as if you were a potential customer looking to see what policies are available from this provider. Go to the appropriate section of the website in order to get a quote. Enter the details for the scenario in exactly the same way as you did on the price comparison website.

Q38 What was the outcome of looking for the policy on the supplier‟s website?

1 I was able to find the same policy All Single code

2 I was not able to find the same policy

Q39

Write in details of what you did and what the outcome was (e.g. if you were unable to get insurance cover). There are no more questions to complete.

Q38 (2)

Open

Q40 Write in the monthly premium for this policy or, if your scenario involved you wanting to pay the full cost up front, write in the full amount for the year.

Q38 (1)

Open

Q41



Q38 (1)




1 Yes Q41 (2,3)

Single code 2 No


Q41 (2) + Q42 (1)

Open

Q44 Write in why the supplier site cost is higher than the price comparison site cost. Q41 (3) +

Open

224 of 264

Q42 (1)

Q45

Read through the summary policy details on the supplier site and compare them to the policy details on the price comparison website. Does the supplier site have any additional details about the policy?


Q38 (1)

Single code




Q45 (1)

Multi code



4 Restrictions on location

5 Restrictions on type/use of property

6 Amount of excess (voluntary or compulsory)

7 Inclusive extras (alternative accommodation, new for old)

8 Other (specify)


Q38 (1)

Open. Optional

225 of 264

CONSUMER FOCUS PRICE COMPARISON WEBSITES RESEARCH: MOBILE PHONE MYSTERY SHOPPER FORM



1 Broadband

All


2 Energy

3 Car insurance

4 Home insurance

5 Mobile phones

6 Package holidays

7 Rail fares


1 Best mobile contracts

All From allocation

2 Bill Monitor

3 Buy mobile phones.net

4 Compare mobile deals

5 Confused.com

6 GoCompare

7 Mobile Checker

8 Mobile Choices

9 Mobilife

10 Money Supermarket

11 Omio

12 One compare.com

13 One stop phone shop

14 Recombu

15 Urmob









5 Not accredited



QD


1 First product


2 Second product

3 Third product

4 Fourth product

5 Fifth product

QE



(6,7)



QF




226 of 264



From the website homepage, find where you can get comparative quotes for different mobile phone packages. Input appropriate details for the scenario you have been assigned then answer the following question. N.B. Make a note of the details you enter so that you can replicate them when you visit the supplier website.

Q1


1 Name

All Multi-code

2 Postcode

3 E-mail address

4 Phone number

5 Brand of phone

6 Model of phone

7 Colour of phone

8 Type of contract (monthly contract, PAYG, SIM only)

9 Network provider (Orange, 3, Vodaphone)

10 Monthly cost

11 Inclusive minutes/texts

12 Additional features (such as insurance)

13 Data allowance (internet)

14 Contract length

15 Incentives

16 Phone features

17 Other (specify)


Q2

Did the initial list of packages appear to be based on the criteria you input, or did it include packages that ignored those requirements? For example, if you asked for a particular model of phone (e.g. Apple iPhone) or network provider (e.g. Orange) was the list restricted to this type only.


All Single code

2 No, list included packages that did not match my criteria

Q3 Please give more information on how some of the packages did not match your criteria. Q2 (2)

Open

Q4



All Single code

1 …on most popular

2 …on cheapest monthly cost

3 …on cost of handset

4 …on most inclusive texts

5 …on most inclusive minutes

6 …on largest data allowance (internet data)




9 …on most popular

10 …on cheapest monthly cost

11 …on cost of handset

12 …on most inclusive texts

13 …on most inclusive minutes

14 …on largest data allowance (internet data)





Q5 Was it possible to do each of the following? All Radio grid. Single code for each

227 of 264

k) Reorder the initial list of package (e.g. sort them on cheapest monthly cost, free handset or some other factor)

1 Yes

l) Filter the original list (e.g. on cheapest monthly cost, free handset or some other factor)

2 No

Q6


1 List could be reordered on most popular

Q5a (1)

Multi code

2 List could be reordered on cheapest monthly cost

3 List could be reordered on handset price

4 List could be reordered on most inclusive minutes

5 List could be reordered on most inclusive texts

6 List could be reordered on largest data allowance

7 List could be reordered on contract length


9 List could be reordered on brand

10 List could be reordered on incentives


Q7 What factors could the original list be filtered on, i.e. cut down to include only specific types of tariff?

1 List could be filtered on monthly cost

Q5b (1)

Multi code

2 List could be filtered on inclusive minutes

3 List could be filtered on inclusive texts

4 List could be filtered on contract length

5 List could be filtered on network provider

6 List could be filtered on data allowance

7 List could be filtered on handset price

8 List could be filtered on most popular

9 List could be filtered on brand

10 List could be filtered on incentives

11 List could be filtered on phone features


Q8 On the first page of packages, is there a note anywhere to say when the prices were last updated?




Q9 On the first page of packages, how many options are listed?


2 10 or more

Q10 On the first page of packages, is there….


All Multiple code 2

…a link for you to access and edit your search criteria

3 …a summary of the key features of your chosen handset

4 None of these

Q11 Did your scenario involve looking for monthly contract phone, PAYG or the „best deal‟?

1 Monthly contract

All Single code 2 Pay as you go

3 „Best deal‟

N.B. There are no Q12 – Q13

Q14 At the end of the list of packages on the first page, are there any of the following?




3 Neither of these is applicable as there are no more packages to show

228 of 264

Q15

If the list is ordered by „monthly cost‟ leave it like that. If it isn‟t already in „monthly cost‟ order and you can reorder on that variable, please do so now. N.B. If you are doing a PAYG scenario „monthly cost‟ is not applicable. You might be able to reorder the list on “best deals” but quite often no reordering is possible Please code which situation applies.

1 I have the package list in price order (monthly cost / “best deals”)

All Single code

2 The package list is not in price order and I cannot reorder it based on the monthly cost / “best deals”

Q16

Scroll down the first *2-3 pages of packages. Are they all in price order? *If there‟s only one page just look at the packages on it


Single code


Q17

Do the results (except those offered by Tesco) include a telephone number for customers to purchase over the phone?

1 Yes, there is a telephone number clearly displayed against the various packages

All Single code

2 No, there are no telephone numbers displayed against the available packages

Browse the details provided for the first 5 packages in the list. This may require you to click a link under contract or package details or a button such as “More info” or “Further information” for each one. N.B. You should do this for all 5, not just the one you will select later.

Q18

Did any of the first 5 packages in the list have any restrictions / limitations that were not mentioned in the search criteria when you first put in your details? N.B. We are not asking here about any additional costs – they are covered in the next question Please read the list below carefully and tick all that apply. If there are no additional restrictions / limitations please tick the last box.

1 Refurbished phone (not new)



3 Usage restrictions (inclusive minutes only to same network or at certain times)



6 Subject to credit check



Q19

Did any of the first 5 packages in the list have any additional costs that were not mentioned earlier? N.B. You should tell us both about costs that are unavoidable for someone choosing that packages and those which are possible additions dependent on what they choose / how they act in the future (e.g. paying extra for paper bills, or having a cancellation fee). Please read the list below carefully and tick all that apply. If there are no additional costs please tick the last box.

1 Handset cost


2 Paper bill

3 Delivery costs

4 Handset insurance

5 Technical assistance (helpline)

6 Top up



229 of 264

Check which of the first 5 packages in the list you are to select and look at the details provided for that particular package. If it is a package that you cannot apply for via the cost comparison website then choose either the one above or the one below it. You can spot a package that you cannot apply for because the “apply” button will be greyed out. This can be the case when you have asked the site to search all available packages, including those that are not available via the price comparison site or that you can‟t apply for today.

Q20a Was the package you selected greyed out?


2 No

Q20b

Who is the network provider for your chosen *package? Make a separate note of this, to remind you which supplier website to visit. *Remember, if the one you had to select was greyed out you should take the one above or below it

1 Orange

All Single code

2 T-Mobile

3 3

4 Vodaphone

5 O2

6 Talk mobile

7 Tesco mobile

8 Virgin mobile

9 Other (specify)

Q21

Who is the phone manufacturer? Make a separate note of this, to remind you which supplier website to visit.

1 Apple

All Single code

2 Nokia

3 LG

4 Blackberry

5 Motorola

6 HTC

7 Samsung

8 Sony Ericsson

12 Other (specify)

Q22

Write in the name of the package. Make a separate note of this, to remind you which package you will be looking for on the supplier website.

All Open

Q23 Write in the monthly cost for this package. Make a separate note of this, to compare to the price on the supplier website.

All Open

Q24

Is the package subject to any special offers or deals? Please tick all that apply.

1 Cashback offer


2 Free line rental (limited months)

3 Free gift (cinema tickets, iPad)

4 Additional SIM card

5 Other special offers

6 No special offers / deals

Q25 Were these offers included in the overall monthly cost (e.g. cashback amount used to reduce the monthly rental costs)

1 Yes

Q24 (1-5)

Single code 2 No

3 Not sure/Don‟t know

Q26

If you were a “real” customer, do you at this stage have sufficient information about the package (cost, special offers, restrictions, etc.) to feel comfortable making an application?


All Single code



Open

Now click on the “apply” button for your chosen package. N.B. We are not going to go through the whole of the application process.

230 of 264



All Single code

2 Redirected to the network provider website

3 Redirected to the phone manufacturers website


5 Redirected to a retail shop (e.g. phones4u)


Q29


1 Refurbished phone (not new)







7 Other restriction/limitation


Q30

In the first screen of the application process, were there any additional costs that had not been mentioned previously? N.B. You should tell us both about costs that are unavoidable for someone choosing that package and those which are possible additions dependent on what they choose / how they act in the future (e.g. paying extra for paper bills, or having a cancellation fee). Please read the list below carefully and tick all that apply. If there are no additional costs please tick the last box.

1 Handset cost


2 Paper bill

3 Delivery costs

4 Handset insurance

5 Technical assistance (helpline)

6 Top up


8 No, no additional cost

Q31


1 Yes



Open



2 No


Open


1 Very easy

All Single code

2 Quite easy


4 Quite difficult

5 Very difficult

Q36 From your visit today, is your overall impression of this website positive or

1 Very positive All Single code

2 Quite positive

231 of 264

negative? 3 Neither positive nor negative

4 Quite negative

5 Very negative



All Open. Optional

Open a new window and go to the supplier‟s website for the mobile phone package you selected. You will be visiting either the network provider website (e.g. Orange, O2 etc.) or the phone manufacturer‟s website (e.g. Nokia, Apple) depending what your allocation specifies. Act as if you were a potential customer looking to see what packages are available from this provider. Go to the appropriate section of the website in order to get a quote. Enter the details for the scenario in exactly the same way as you did on the price comparison website. Look through the list of results until you find the package you selected in the price comparison website. N.B. There is no Q39

Q40 What was the outcome of looking for the package on the supplier‟s website?

1 I was able to find the same package All Single code

2 I was not able to find the same package

Q41

Write in details of what you did and what the outcome was (e.g. if you found the monthly call package with similar name write them in or the handset was available but not on the same network or if you found the package finished several weeks ago let us know). There are no more questions to complete.

Q40 (2)

Open

Q42 Write in the monthly cost for this package. Q40 (1)

Open

Q43



Q40 (1)




1 Yes Q43 (2,3)

Single code 2 No


Q43 (2) + Q44 (1)

Open


Q43 (3) + Q44 (1)

Open

Q47

Read through the package details on the supplier site and compare them to the package details on the price comparison website. Does the supplier site have any additional details about the package?


Q40 (1)

Single code


Q48 What are the additional details on the 1 Refurbished phone (not new) Q47 Multi code

232 of 264

supplier site? 2 Minimum length of contract (1)





7 Other (specify)


Q40 (1)

Open. Optional

233 of 264

CONSUMER FOCUS PRICE COMPARISON WEBSITES RESEARCH: RAIL FARES MYSTERY SHOPPER FORM



1 Broadband

All


2 Energy

3 Car insurance

4 Home insurance

5 Mobile phones

6 Package holidays

7 Rail fares


1 Travel Supermarket (Money Supermarket)

All From allocation

2 TheTrainLine

3 Railsaver

4 Redspottedhanky

5 Raileasy

6 Mytrainticket

7 National Rail Enquiries









5 Not accredited



QD


1 First product

All From allocation

2 Second product

3 Third product

4 Fourth product

5 Fifth product

QE

Please tick what you are doing after each visit. There are further instructions later on what each entails.


(6,7)



Before you start your initial search, you need to make sure that your browser is set up to receive cookies from the various sites. You do this in the following way: Go to the „Tools‟ menu on your browser window (for Google Chrome, this is shown as a spanner). You will need to find the option to enable cookies. Google Chrome: Tools menu (spanner); Options; Under the hood; Content settings; „allow local data to be set‟ must be selected and nothing else. Internet Explorer: Tools menu; Internet Options; Privacy; Settings (set to „accept all cookies‟) Mozilla FireFox: Tools Menu; Options; Privacy; History; Firefox will „use custom settings for history‟ (selected from drop down) and make sure „accept cookies from sites‟ is selected.

Instruction

234 of 264

QF






From the website homepage, find where you can get quotes for rail journeys (which may well be from the homepage itself).

Q1a On this page, is there any mention that a booking fee will / might be charged?


2 No

Input appropriate details for the scenario you have been assigned then answer the following question. N.B. Make a note of the details you enter so that you can replicate them when you visit the supplier website(s).

Q1b

What information was it possible for you to input in order to get an initial list of train times and tickets? You should tick all the items available, even if you did not use them.

1 Journey from

All Multi-code

2 Journey to

3 Date(s) of travel

4 Time(s) of travel

5 No. of adult passengers

6 No. of child passengers

7 Class of travel (1st, standard, etc.)

8 Whether want to view only trains with no changes

9 Whether want to go via a particular place

10 Whether want to avoid going via a particular place

11 Whether want an open return

12 Whether want a one way ticket

13 Whether have a Railcard

14 How many Railcards have

15 Other (specify)

The next few questions are about the initial list of times and ticket prices that appears once you have submitted your details.

Q2

Did the initial list of times and ticket types appear to be based on the criteria you input, or did it include quotes that ignored those requirements? For example, if you asked for a particular journey type (e.g. 1st class, after 10am) was the list restricted to this type only.


All Single code

2 No, list included trains/tickets that did not match my criteria

Q3 Please give more information on how some of the trains/tickets did not match your criteria. Q2 (2)

Open

Q4

On the page with the initial list, was it possible for you to further refine your search in any way, if you wanted to? Tick all that are applicable or “no further refinement possible”.

1 View / check for slower routes


2 View only cheap(est) tickets

3 View all tickets / fares

4 View only 1st class / standard class

5 View only trains with no changes

6 View fares with more flexibility

7 Change time of departure / arrival

8 List could be refined on other factors (specify)

9 No further refinement possible

Make the final selections for your journey (i.e pick the exact train(s) you want to travel on).

Q5a At this point, was the cost of the journey shown?


2 No

235 of 264

Q5b At any point in the selection process, were you alerted to the fact that two single tickets might be cheaper than a return?


2 No

Q6

Was a summary of your journey shown (e.g. from, to, time, date)? N.B. You might have to click forward one page to see this.

1 Yes

All Single code

2 No

Q7a Which of the following were shown in the summary?

1 Ticket validity details (e.g. only valid on Virgin trains, only valid on date of purchase)

Q6 (1) Multi code

2 (Non) refund details

3 Travel restrictions / allowances (e.g. only after 11am, any route allowed)

4 Changes allowed prior to travel for a fee

5 Whether a Railcard had been used

6 No. of passengers

7 Any train changes needed

8 Duration of journey

9 Other (apart from time, date, to, from) (specify)

Q7b Is there a note anywhere to say that VAT is included in the prices?


All Single code


3 No, there was not a note about VAT being included

4 No, there was a note about VAT being excluded

Q7c Is there a note anywhere to say when the prices were last updated?




Move through the booking pages until you get to the final one, where you have to enter payment details (we won‟t be asking you to do this). Keep a close watch for:

Additional costs that are automatically added and you are unable to remove them Additional costs that are automatically added but you have a choice to remove them Additional costs that are not automatically added, which you have to choose to add to the cost

You should not choose any additional costs unless they are relevant to your scenario, or they cannot be removed. Also look at the details provided as you move through the booking process, and note any restrictions / limitations that were not mentioned when you first chose the journey (e.g. travel restrictions, etc.).

Q8

Were there any restrictions / limitations that had not been mentioned when you made your final journey selections? N.B. We are not asking here about any additional costs – they are covered in the next question Please read the list below carefully and tick all that apply. If there were no additional restrictions / limitations please tick the last box.

1 Train change needed



3 Non-refundable

4 Travel restrictions / allowances (e.g. only after 11am, only a particular route allowed)




Q9


All Radio grid. Single code for each Seat booking 1

Additional cost that was automatically added and I was unable to remove it

Travel insurance 2 Additional cost that was automatically added but I

236 of 264

had a choice to remove it

PlusBus 3 Additional cost that was not automatically added, which I could choose to add to the cost

Ticket delivery via post or Special Delivery 4 Not an additional cost that appeared at all / was no extra cost for this

Booking fee

Credit / debit card fee (might not appear until you reach the payment page and choose a card type to pay with)

Q10

Were there any costs other than those in the previous question which were added automatically and you were unable to remove?

1 Yes

All Single code

2 No

Q11 What were these costs for? Q10 (1)

Open

Q12 At any point when you were making your journey choices, was a booking fee mentioned?

1 Yes Q9 (1) for booking fee

Single code

2 No

Q13 Which of the following best describes what happened when you clicked to book your journey?

1 Stayed within the price comparison website




Q14a

Which rail company or companies will you be travelling on? Make a separate note of this, to remind you which supplier website(s) to visit.

1 South West Trains

All Multi code

2 Southeastern Trains

3 Southern

4 First TransPennine Express

5 CrossCountry

6 East Coast

7 First Great Western

8 East Midlands Trains

9 Northern Rail

10 London Midland

11 First ScotRail

12 Other (specify)

Q14b Write in the cost of your journey. Make a separate note of this, as you will be comparing it against the cost quoted on your return visits and on the supplier website(s).

All Open

Q15

If you were a “real” customer, do you at this stage have sufficient information about the journey (cost, restrictions, etc.) to feel comfortable booking it?

1 Yes, I have sufficient information to make a booking

All Single code

2 No, I would need more information before making a booking


Open



2 No


Open



2 No

Q20 What was that? Q19 Open

237 of 264

(1)


1 Very easy

All Single code

2 Quite easy


4 Quite difficult

5 Very difficult


1 Very positive

All Single code

2 Quite positive


4 Quite negative

5 Very negative



All Open. Optional

Open a new window and go to the supplier‟s website for the journey you selected (e.g. First Great Western, Virgin Trains, etc.). If your journey involved more than one supplier you will need to go to both websites, one after the other. Go to the appropriate section of the supplier website for journey costs. Enter the details for the scenario in exactly the same way as you did on the price comparison website. Pick the same trains on the same days at the same times as you did before.

Q25 What was the outcome of looking for the ticket on the supplier website(s)?

1 I was able to find the same ticket(s) All Single code

2 I was not able to find the same ticket(s)

Q26 Write in details of what you did and what the outcome was. Q25 (2)

Open

Q27

Write in the cost of your journey. This may involve adding together the cost of each leg from the different supplier sites. Make a separate note of the total, as you will be comparing it against the cost quoted on your return visit to the supplier website(s).

Q25 (1)

Open

Q28

How does the total supplier site cost compare to the price comparison site cost? *You should only use this if the figures are identical to the last penny


Q25 (1)




1 Yes Q28 (2,3)

Single code 2 No


Q28 (2) + Q29 (1)

Open


Q28 (3) + Q29 (1)

Open

Q32 Read through the journey details on the supplier site(s) and compare them to the journey details on the price comparison

1 Yes, supplier site has additional details Q25 (1)

Single code

238 of 264

website. Do(es) the supplier site(s) have any additional details about the journey?




Q32 (1)

Multi code


3 Non-refundable



6 Other (specify)

Q34 If there is anything else you would like to tell us about how the journey is presented and described on the supplier site(s) versus the price comparison site please do so in the box below.

Q25 (1)

Open. Optional

You are to make 3 return visits to this price comparison website over the next 1-2 days, to repeat the search (so you will be doing 4 visits to the site in total). Leave at least an hour between return visits. On the 4th (last) visit you will also look at the supplier site(s) again. Do not save any of these searches, start afresh each time.

All Instruction

At some point between each of your 3 repeat searches you will do at least two rail journey related searches. Examples of rail journey related searches are:

Searching to see what there is to see and do in the place you‟re visiting Looking at train company websites (in particular those involved in your journey) Visiting websites such as rail.co.uk

QE (2) Instruction

Straight after your initial search and straight after each of your 3 repeat searches you will clear your browsing data. Go to the „Tools‟ menu on your browser window (for Google Chrome, this is shown as a spanner). You will need to find the option to clear your browser history and select all options with the exception of passwords/active log in (make sure that „cookies‟ are selected) to delete. If you have an option to select a time range, please make sure if covers your shop e.g. in the past day. Google Chrome: Tools menu (spanner); Tools; Clear browsing data Internet Explorer: Tools menu; Internet Options; Browsing History; Delete Mozilla FireFox: Tools Menu; Clear recent history (check cookies are selected under „details‟)

QE (1) Instruction

The details to be recorded for your 1st visit are now finished, so you can close the form and come back later when you are doing your 2nd visit.

All Instruction

VISIT 2

Q35 Did you close your browser between the first visit and this one?

1 Yes

QE (2) Single code

2 No

Q36 What rail journey related searches did you carry out after your first visit to the price comparison website?

QE (2) Open

Visit the price comparison website for the second time and, using the same scenario / selections as your first visit, work your way through to the payment page.

Q37 Write in the final cost for the journey. Make a separate note of this, as you will be comparing it against the cost quoted on your next visit.

QE (2) Open

239 of 264

Q38 Is this cost the same as, higher than or lower than the cost you ended up with at the end of your first visit?

1 This cost is the same as the first visit

QE (2) Single code 2 This cost is higher than the first visit

3 This cost is lower than the first visit

Q39

Is it obvious to you why the cost is higher? To answer “yes” there should be a definite reason that you can see clearly. You should not answer “yes” if you are simply guessing at the reason.

1 Yes Q38 (2)

Single code

2 No

Q40 Why is the cost higher?

Q38 (2) + Q39 (1)

Open

Q41

Is it obvious to you why the cost is lower? To answer “yes” there should be a definite reason that you can see clearly. You should not answer “yes” if you are simply guessing at the reason.

1 Yes Q38 (3)

Single code

2 No

Q42 Why is the cost lower?

Q38 (3) + Q41 (1)

Open

Q43 Please confirm that you cleared your browsing data straight after your first visit to the price comparison website

1 Yes, I did

QE (1) Single code

2 No, I didn‟t

Please contact the office immediately for instructions

Q43 (2)

Instruction



Q43 (1)

Open



Q43 (1)

Single code 2 This cost is higher than the first visit


Q46


1 Yes Q45 (2)

Single code

2 No


Q45 (2) + Q46 (1)

Open

Q48


1 Yes Q45 (3)

Single code

2 No

Q49 Why is the cost lower? Q45 (3) + Q48

Open

240 of 264

(1)

The details to be recorded for your 2nd visit are now finished, so you can close the form and come back later when you are doing your 3rd visit.

All Instruction

VISIT 3


1 Yes

QE (2) Single code

2 No

Q51 What rail journey related searches did you carry out after your second visit to the price comparison website?

QE (2) Open

Visit the price comparison website for the third time and, using the same scenario / selections as your first visit, work your way through to the payment page.


QE (2) Open

Q53

Is this cost the same as, higher than or lower than the cost you ended up with at the end of your second visit? N.B. Note that the comparison is between the cost you got on this visit and the cost you got on the previous one

1 This cost is the same as the second visit

QE (2) Single code 2 This cost is higher than the second visit

3 This cost is lower than the second visit

Q54


1 Yes Q53 (2)

Single code

2 No


Q53 (2) + Q54 (1)

Open

Q56


1 Yes Q53 (3)

Single code

2 No


Q53 (3) + Q56 (1)

Open

Q58 Please confirm that you cleared your browsing data straight after your last visit to the price comparison website

1 Yes, I did

QE (1) Single code

2 No, I didn‟t


Q58 (2)

Instruction



Q58 (1)

Open

241 of 264

Q60



Q58 (1)

Single code 2 This cost is higher than the second visit


Q61


1 Yes Q60 (2)

Single code

2 No


Q60 (2) + Q61 (1)

Open

Q63


1 Yes Q60 (3)

Single code

2 No


Q60 (3) + Q63 (1)

Open

The details to be recorded for your 3rd visit are now finished, so you can close the form and come back later when you are doing your 4th visit.

All Instruction

VISIT 4


1 Yes

QE (2) Single code

2 No

Q66 What rail journey related searches did you carry out after your third visit to the price comparison website?

QE (2) Open

Visit the price comparison website for the fourth (and last) time and, using the same scenario / selections as your first visit, work your way through to the payment page.

Q67 Write in the final cost for this rail journey. QE (2) Open

Q68

Is this cost the same as, higher than or lower than the cost you ended up with at the end of your third visit? N.B. Note that the comparison is between the cost you got on this visit and the cost you got on the previous one

1 This cost is the same as the third visit

QE (2) Single code 2 This cost is higher than the third visit

3 This cost is lower than the third visit

Q69


1 Yes Q68 (2)

Single code

2 No


Q68 (2) + Q69 (1)

Open

242 of 264

Q71


1 Yes Q68 (3)

Single code

2 No


Q68 (3) + Q71 (1)

Open


1 Yes, I did

QE (1) Single code

2 No, I didn‟t


Q73 (2)

Instruction


Q74 Write in the final cost for the journey. Q73 (1)

Open

Q75



Q73 (1)

Single code 2 This cost is higher than the third visit


Q76


1 Yes Q75 (2)

Single code

2 No


Q75 (2) + Q76 (1)

Open

Q78


1 Yes Q75 (3)

Single code

2 No


Q75 (3) + Q78 (1)

Open

Open a new window and go to the supplier‟s website for the journey you selected (e.g. First Great Western, Virgin Trains, etc.). If your journey involved more than one supplier you will need to go to both websites, one after the other. Go to the appropriate section of the supplier website for journey costs. Enter the details for the scenario in exactly the same way as you did on the price comparison website. Pick the same trains on the same days at the same times as you did before.

Q80 What was the outcome of looking for the ticket on the supplier website(s)?

1 I was able to find the same ticket(s) All Single code

2 I was not able to find the same ticket(s)

Q81 Write in details of what you did and what the outcome was. Q80 (2)

Open

243 of 264

Q82 Write in the cost of your journey. This may involve adding together the cost of each leg from the different supplier sites.

Q80 (1)

Open

Q83

How does the total supplier site cost compare to the price comparison site cost that you saw on this visit? *You should only use this if the figures are identical to the last penny


Q80 (1)




1 Yes Q83 (2,3)

Single code 2 No


Q83 (2) + Q84 (1)

Open


Q83 (3) + Q84 (1)

Open

Q87

Read through the journey details on the supplier site(s) and compare them to the journey details on the price comparison website. Do(es) the supplier site(s) have any additional details about the journey?


Q80 (1)

Single code




Q87 (1)

Multi code


3 Non-refundable



6 Other (specify)

Q89 If there is anything else you would like to tell us about how the journey is presented and described on the supplier site(s) versus the price comparison site please do so in the box below.

Q80 (1)

Open. Optional

You have now completed the entire shop and can submit your form.

All Instruction

244 of 264

CONSUMER FOCUS PRICE COMPARISON WEBSITES RESEARCH: PACKAGE HOLIDAYS MYSTERY SHOPPER FORM



1 Broadband

All


2 Energy

3 Car insurance

4 Home insurance

5 Mobile phones

6 Package holidays

7 Rail fares


1 Travel Supermarket (Money Supermarket)

All From allocation

2 Confused.com

3 GoCompare

4 Expedia

5 Lastminute.com

6 Skyscanner

7 ebookers

8 Opodo

9 Cheap flights

10 Travel republic

11 Lowcostholidays

12 Kelkoo

13 onthebeach.co.uk









5 Not accredited



QD


1 First product

All From allocation

2 Second product

3 Third product

4 Fourth product

5 Fifth product

QE

Please tick what you are doing after each visit. There are further instructions later on what each entails.


(6,7)



Before you start your initial search, you need to make sure that your browser is set up to receive cookies from the various sites. You do this in the following way: Go to the „Tools‟ menu on your browser window (for Google Chrome, this is shown as a spanner). You will need to find the option to enable cookies.

Instruction

245 of 264

Google Chrome: Tools menu (spanner); Options; Under the hood; Content settings; „allow local data to be set‟ must be selected and nothing else. Internet Explorer: Tools menu; Internet Options; Privacy; Settings (set to „accept all cookies‟) Mozilla FireFox: Tools Menu; Options; Privacy; History; Firefox will „use custom settings for history‟ (selected from drop down) and make sure „accept cookies from sites‟ is selected.

QF






From the website homepage, find where you can get details of package holidays (which might also be called “bargain holidays”, “cheap holidays” or some other name). A number of sites only have one “holiday” section (i.e. it is not possible to find a specific package holiday area). Follow your scenario, inputting appropriate details as necessary, then answer the following question.

Q1


1 Departure airport

All Multi-code

2 Destination

3 Departure date

4 Return date

5 Trip duration/no. of nights

6 No. of adults

7 No. of children

8 Board type (all inclusive, self-catering, half board, full board, B&B)

9 No. of rooms needed

10 Minimum hotel rating (2 star, 3 star, etc.)

11 Room type (single, double, family, etc.)

12 Age(s) of children

13 Other (specify)

The next few questions are about the initial list of holidays that appears once you have submitted your details.

Q2

Did the initial list of holidays appear to be based on the criteria you input, or did it include holidays that ignored those requirements? For example, if you asked for a particular type of holiday was the list restricted to this type only.


All Single code

2 No, list included holidays that did not match my criteria

Q3 Please give more information on how some of the holidays did not match your criteria. Q2 (2)

Open

Q4

Was there an explanation as to how the initial list of holidays had been ordered? For example, it might say they were ordered on price.


All Single code

1 …on price




4 …on price





Q5 Was it possible to do each of the following?


m) Reorder the initial list of holidays (e.g. 1 Yes

246 of 264

sort them on price, popularity or some other factor)

n) Filter the original list (e.g. on price, star rating, board basis)

2 No

Q6


1 List could be reordered on price

Q5a (1)

Multi code

2 List could be reordered on hotel/apartment/resort name

3 List could be reordered on accommodation star ratings



Q7 What factors could the original list be filtered on, i.e. cut down to include only specific types of holiday?

1 List could be filtered on price

Q5b (1)

Multi code

2 List could be filtered on accommodation star ratings

3 List could be filtered on departure airport

4 List could be filtered on resort / location

5 List could be filtered on hotel name

6 List could be filtered on room type

7 List could be filtered on board basis


Q8a On the first page of holidays, is there a note anywhere to say when the prices were last updated?




Q8b On the first page of holidays, how many options are listed?


2 10 or more

Q9 On the first page of holidays, is there….


All Single code 2

…a link for you to access a summary of your search criteria

3 Neither of these

Q10 At the end of the list of holidays on the first page, are there any of the following?




3 Neither of these is applicable as there are no more holidays to show

Q11

If the list is ordered by price leave it like that. If it isn‟t already in price order and you can reorder on that variable, please do so now. Please code which situation applies.

1 I have the holiday list in price order

All Single code

2 The holiday list is not in price order and I cannot reorder it on that variable

Q12

Scroll down the first *2-3 pages of holidays. Are they all in price order? *If there‟s only one page just look at the holidays on it


Q11 (1)


Check which of the first 5 holidays in the list you are to select. Take a screen shot or print the page so that you know exactly which holiday it is that you have to go back to when you make your return visits.

Q13 Write in the cost shown at this point for the holiday. All Open

Q14 Is the cost quoted as per person/adult, or in total, or both?

1 Per person/adult

All Multi code apart from last item

2 In total

3 Both

4 Other situation (specify)

Q15 Is the cost quoted as “from £xxx”? 1 Yes All Single code

247 of 264

2 No

Now click on your selected holiday to get further into the booking process.

Q16 Which of the following best describes what happens?





Make any final selections that are needed (e.g. exact flights required, exact rooms). If the cost quoted was “from £xxx” try to keep to this total (e.g. don‟t upgrade rooms, choose the flights that will keep you at that price). Move through the booking pages until you get to the final one, where you have to enter payment details (we won‟t be asking you to do this). Keep a close watch for:

Additional costs that are automatically added and you are unable to remove them Additional costs that are automatically added but you have a choice to remove them Additional costs that are not automatically added, which you have to choose to add to the cost

You should not choose any additional costs unless they are relevant to your scenario, or they cannot be removed. Also look at the details provided as you move through the booking process, and note any restrictions / limitations that were not mentioned when you looked at the holiday in the initial list (e.g. no children allowed, no wheelchair access, etc.).

Q17

Were there any restrictions / limitations that had not been mentioned in the holiday details in the initial list? N.B. We are not asking here about any additional costs – they are covered in the next question Please read the list below carefully and tick all that apply. If there were no additional restrictions / limitations please tick the last box.

1 No children allowed


2 No wheelchair access

3

4

5



Q18



Transfers to / from airport at destination 1 Additional cost that was automatically added and I was unable to remove it

Travel insurance 2 Additional cost that was automatically added but I had a choice to remove it

Departure taxes / local fees 3 Additional cost that was not automatically added, which I could choose to add to the cost

Cost of hold luggage / bags 4 Not an additional cost that appeared at all / was no extra cost for this

Supplier failure protection

Airport parking

Hotel or board upgrades

Single person supplement

Q19

Were there any costs other than those in the previous question which were added automatically and you were unable to remove?

1 Yes

All Single code

2 No

Q20 What were these costs for? Q19 (1)

Open

248 of 264

Q21 Write in the final cost for this holiday. This should be the full cost for the entire party, not a per person cost. Make a separate note of this, as you will be comparing it against the cost quoted on your return visits.

All Open

Q22

If you were a “real” customer, do you at this stage have sufficient information about the holiday (cost, accommodation, what‟s included/not included, etc.) to feel comfortable booking it?

1 Yes, I have sufficient information to make a booking

All Single code

2 No, I would need more information before making a booking


Open



2 No


Open



2 No


Open


1 Very easy

All Single code

2 Quite easy


4 Quite difficult

5 Very difficult


1 Very positive

All Single code

2 Quite positive


4 Quite negative

5 Very negative



All Open. Optional

For package holidays there are no searches on supplier websites. However, you are to make 3 return visits to this price comparison website over the next 1-2 days, to repeat the search (so you will be doing 4 visits to the site in total). Leave at least an hour between return visits. Do not save any of these searches, start afresh each time. There is a separate section of this form for each return visit.

All Instruction

At some point between each of your 3 repeat searches you will do at least two holiday related searches. Examples of holiday related searches are:

Searching to see what there is to see and do in the place you‟re visiting Searching on a tour operator/travel agent website for similar holidays Looking up the accommodation from your selected holiday in TripAdvisor

QE (2) Instruction

Straight after your initial search and straight after each of your 3 repeat searches you will clear your browsing data. You do this in the following way:

QE (1) Instruction

249 of 264

Go to the „Tools‟ menu on your browser window (for Google Chrome, this is shown as a spanner). You will need to find the option to clear your browser history and select all options with the exception of passwords/active log in (make sure that „cookies‟ are selected) to delete. If you have an option to select a time range, please make sure if covers your shop e.g. in the past day. Google Chrome: Tools menu (spanner); Tools; Clear browsing data Internet Explorer: Tools menu; Internet Options; Browsing History; Delete Mozilla FireFox: Tools Menu; Clear recent history (check cookies are selected under „details‟)

The details to be recorded for your 1st visit are now finished, so you can close the form and come back later when you are doing your 2nd visit.

All Instruction

VISIT 2


1 Yes

QE (2) Single code

2 No

Q33 What holiday related searches did you carry out after your first visit to the price comparison website?

QE (2)

Open


Q34 Could you find the same holiday or not?

1 Yes

QE (2) Single code

2 No

Q35 Write in the final cost for this holiday. This should be the full cost for the entire party, not a per person cost. Make a separate note of this, as you will be comparing it against the cost quoted on your next visit.

Q34 (1)

Open



Q34 (1)



Q37


1 Yes Q36 (2)

Single code

2 No


Q36 (2) + Q37 (1)

Open

Q39


1 Yes Q36 (3)

Single code

2 No


Q36 (3) + Q39 (1)

Open

Q41 Please confirm that you cleared your browsing data straight after your first

1 Yes, I did QE (1) Single code

250 of 264

visit to the price comparison website 2 No, I didn‟t


Q41 (2)

Instruction



1 Yes Q41 (1)

Single code

2 No


Q42 (1)

Open



Q42 (1)



Q45


1 Yes Q44 (2)

Single code

2 No


Q44 (2) + Q45 (1)

Open

Q47


1 Yes Q44 (3)

Single code

2 No

Q48 Why is the cost lower? Q4 (3) + Q47

(1) Open

The details to be recorded for your 2nd visit are now finished, so you can close the form and come back later when you are doing your 3rd visit.

All Instruction

VISIT 3

Q49 Did you close your browser between the second visit and this one?

1 Yes

QE (2) Single code

2 No

Q50 What holiday related searches did you carry out after your second visit to the price comparison website?

QE (2)

Open


Q51 Could you find the same holiday or not? 1 Yes QE (2) Single code

251 of 264

2 No


Q51 (1)

Open

Q53



Q51 (1)



Q54


1 Yes Q53 (2)

Single code

2 No


Q53 (2) + Q54 (1)

Open

Q56


1 Yes Q53 (3)

Single code

2 No


Q53 (3) + Q56 (1)

Open


1 Yes, I did

QE (1) Single code

2 No, I didn‟t


Q58 (2)

Instruction



1 Yes Q58 (1)

Single code

2 No


Q59 (1)

Open

Q61



Q59 (1)



252 of 264

Q62


1 Yes Q61 (2)

Single code

2 No


Q61 (2) + Q62 (1)

Open

Q64


1 Yes Q61 (3)

Single code

2 No


Q61 (3) + Q64 (1)

Open

The details to be recorded for your 3rd visit are now finished, so you can close the form and come back later when you are doing your 4th visit.

All Instruction

VISIT 4

Q66 Did you close your browser between the third visit and this one?

1 Yes

QE (2) Single code

2 No

Q67 What holiday related searches did you carry out after your third visit to the price comparison website?

QE (2) Open



1 Yes

QE (2) Single code

2 No

Q69 Write in the final cost for this holiday. This should be the full cost for the entire party, not a per person cost.

Q68 (1)

Open

Q70



Q68 (1)



Q71


1 Yes Q70 (2)

Single code

2 No


Q70 (2) + Q71 (1)

Open

253 of 264

Q73


1 Yes Q70 (3)

Single code

2 No


Q70 (3) + Q73 (1)

Open


1 Yes, I did

QE (1) Single code

2 No, I didn‟t


Q75 (2)

Instruction



1 Yes Q75 (1)

Single code

2 No

Q77 Write in the final cost for this holiday. This should be the full cost for the entire party, not a per person cost.

Q76 (1)

Open

Q78



Q76 (1)



Q79


1 Yes Q78 (2)

Single code

2 No


Q78 (2) + Q79 (1)

Open

Q81


1 Yes Q78 (3)

Single code

2 No


Q78 (3) + Q81 (1)

Open

You have now completed the entire shop and can submit your form.

All Instruction

254 of 264

APPENDIX B

Price comparison websites and scenarios used

255 of 264

Market Price comparison website

PCW code number Accredited or not Big 4 or not Pilot

Broadband http://www.broadbandchoices.co.uk/ 004 Accredited by Ofcom Not Big 4 Broadband http://www.broadband.co.uk/ 006 Approved by Ofcom Not Big 4 Broadband http://www.cable.co.uk/ 009 Accredited by Ofcom Not Big 4

Broadband http://www.homephonechoices.co.uk/partner-ns~guk-homephonechoices/ 023 Accredited by Ofcom Not Big 4

Broadband http://www.simplifydigital.co.uk/ 051 Accredited by Ofcom Not Big 4

Broadband http://www.moneysupermarket.com/broadband/ 035 Not accredited Big 4

Broadband http://www.gocompare.com/broadband/ 021 Not accredited Big 4

Broadband http://www.comparethemarket.com/broadband/ 014 Not accredited Big 4

Broadband http://www.broadbandgenie.co.uk/ 007 Not accredited Not Big 4 √ Broadband

http://www.moneysavingexpert.com/phones/cheap-broadband 034 Not accredited Not Big 4

Broadband http://broadband.consumerchoices.co.uk/?partner=tesco~tescocompare 055 Not accredited Not Big 4

Broadband http://www.money.co.uk/broadband.htm 036 Not accredited Not Big 4 Broadband http://www.uswitch.com/broadband/ 064 Not accredited Not Big 4 Broadband http://www.broadband-expert.co.uk/ 005 Not accredited Not Big 4 Broadband http://www.comparebroadband.co.uk/ 012 Not accredited Not Big 4 Broadband http://www.confused.com/broadband 015 Not accredited Big 4



Mobile Phones http://www.billmonitor.com/ 003 Accredited by Ofcom Not Big 4 Mobile Phones http://www.mobilife.com/home 031 Accredited by Ofcom Not Big 4 Mobile Phones http://www.mobilechoices.co.uk/ 030 Accredited by Ofcom Not Big 4 Mobile Phones

http://www.moneysupermarket.com/mobile-phones/ 035 Not accredited Big 4

Mobile Phones http://www.confused.com/mobile-phones 015 Not accredited Big 4 Mobile Phones

http://www.gocompare.com/mobile-phones/ 021 Not accredited Big 4

http://www.broadbandchoices.co.uk/

http://www.broadband.co.uk/

http://www.cable.co.uk/

http://www.homephonechoices.co.uk/partner-ns~guk-homephonechoices/

http://www.homephonechoices.co.uk/partner-ns~guk-homephonechoices/

http://www.simplifydigital.co.uk/

http://www.moneysupermarket.com/broadband/

http://www.moneysupermarket.com/broadband/

http://www.gocompare.com/broadband/

http://www.comparethemarket.com/broadband/

http://www.comparethemarket.com/broadband/

http://www.broadbandgenie.co.uk/

http://www.moneysavingexpert.com/phones/cheap-broadband

http://www.moneysavingexpert.com/phones/cheap-broadband

http://broadband.consumerchoices.co.uk/?partner=tesco~tescocompare

http://broadband.consumerchoices.co.uk/?partner=tesco~tescocompare

http://www.money.co.uk/broadband.htm

http://www.uswitch.com/broadband/

http://www.broadband-expert.co.uk/

http://www.comparebroadband.co.uk/

http://www.confused.com/broadband

http://www.billmonitor.com/

http://www.mobilife.com/home

http://www.mobilechoices.co.uk/

http://www.moneysupermarket.com/mobile-phones/

http://www.moneysupermarket.com/mobile-phones/

http://www.confused.com/mobile-phones

http://www.gocompare.com/mobile-phones/

http://www.gocompare.com/mobile-phones/

256 of 264

Mobile Phones http://www.buymobilephones.net/ 008

Accredited by Shopsafe & SafeBuy Not Big 4

Mobile Phones http://www.onestopphoneshop.co.uk/ 042 Not accredited Not Big 4 Mobile Phones http://www.onecompare.com/ 041 Not accredited Not Big 4 Mobile Phones http://recombu.com/ 048 Not accredited Not Big 4 Mobile Phones http://www.best-mobile-contracts.co.uk/ 002 Not accredited Not Big 4 Mobile Phones http://www.urmob.co.uk/ 063 Not accredited Not Big 4

Mobile Phones http://www.omio.com/ 040 Not accredited Not Big 4 √ Mobile Phones http://www.comparemobiledeals.com 013 Not accredited Not Big 4 Mobile Phones http://www.mobilechecker.co.uk/ 029 Not accredited Not Big 4



Energy https://www.theenergyshop.com 056 Accredited by Consumer Focus Not Big 4

Energy http://www.ukpower.co.uk/ 061 Accredited by Consumer Focus Not Big 4

Energy http://www.uswitch.com/gas-electricity/comparison 064

Accredited by Consumer Focus Not Big 4

Energy http://www.simplyswitch.com/ 052 Accredited by Consumer Focus Not Big 4

Energy http://www.energyhelpline.com/ 018 Accredited by Consumer Focus Not Big 4

Energy http://fuelswitch.com/ 020 Accredited by Consumer Focus Not Big 4

Energy http://www.moneysupermarket.com/gas-and-electricity/ 035

Accredited by Consumer Focus Big 4

Energy http://www.confused.com/gas-electricity 015 Accredited by Consumer Focus Big 4

Energy http://www.unravelit.com/ 062 Accredited by Consumer Focus Not Big 4 √

Energy http://www.which.co.uk/switch/ 065 Accredited by Consumer Focus Not Big 4

Energy http://www.compareandsave.com/gas-electric/ 011 Not accredited Not Big 4

Energy http://www.gocompare.com/energy/ 021 Not accredited Big 4

Energy http://www.comparethemarket.com/utilities/ 014 Not accredited Big 4

http://www.buymobilephones.net/

http://www.onestopphoneshop.co.uk/

http://www.onecompare.com/

http://recombu.com/

http://www.best-mobile-contracts.co.uk/

http://www.urmob.co.uk/

http://www.omio.com/

http://www.comparemobiledeals.com/

http://www.mobilechecker.co.uk/

https://www.theenergyshop.com/

http://www.ukpower.co.uk/

http://www.uswitch.com/gas-electricity/comparison

http://www.uswitch.com/gas-electricity/comparison

http://www.simplyswitch.com/

http://www.energyhelpline.com/

http://fuelswitch.com/

http://www.moneysupermarket.com/gas-and-electricity/

http://www.moneysupermarket.com/gas-and-electricity/

http://www.confused.com/gas-electricity

http://www.unravelit.com/

http://www.compareandsave.com/gas-electric/

http://www.compareandsave.com/gas-electric/

http://www.gocompare.com/energy/

http://www.comparethemarket.com/utilities/

http://www.comparethemarket.com/utilities/

257 of 264

Energy http://www.energychoices.co.uk/ 017 Not accredited Not Big 4 Energy http://www.spotenergy.co.uk/ 054 Not accredited Not Big 4 Energy http://www.whoscheapest.co.uk/ 066 Not accredited Not Big 4 Energy http://www.homeadvisoryservice.co.uk/ 022 Not accredited Not Big 4 Energy http://www.saveonyourbills.co.uk/ 050 Not accredited Not Big 4

Market Price comparison website PCW

code number

Accredited or not Big 4 or not Pilot

Home insurance http://www.beatthatquote.com/insurance/ 001 Not accredited Not Big 4 Home insurance http://www.tescocompare.com/ 055 Not accredited Not Big 4 Home insurance

http://www.moneysupermarket.com/money/ 035 Not accredited Big 4

Home insurance http://www.confused.com/money 015 Not accredited Big 4 Home insurance http://www.gocompare.com/money/ 021 Accredited by BIBA Big 4 Home insurance

http://www.comparethemarket.com/money/ 014 Not accredited Big 4

Home insurance http://moneyfacts.co.uk/ 033 Not accredited Not Big 4 Home insurance http://www.uswitch.com/money 064 Not accredited Not Big 4 Home insurance http://www.quotezone.co.uk/ 045 Not accredited Not Big 4 Home insurance http://www.money.co.uk 011 Not accredited Not Big 4 Home insurance http://www.knowyourmoney.co.uk/ 025 Not accredited Not Big 4

Home insurance http://www.moneynet.co.uk/ 037 Not accredited Not Big 4 √ Home insurance http://www.moneyexpert.com/ 032 Not accredited Not Big 4 Home insurance http://www.fairinvestment.co.uk/ 058 Not accredited Not Big 4 Home insurance http://www.lovemoney.com/ 027 Not accredited Not Big 4

Market Price comparison website PCW

code number


Car insurance http://www.beatthatquote.com/insurance/ 001 Not accredited Not Big 4 Car http://www.tescocompare.com/ 055 Not accredited Not Big 4

http://www.energychoices.co.uk/

http://www.spotenergy.co.uk/

http://www.whoscheapest.co.uk/

http://www.homeadvisoryservice.co.uk/

http://www.saveonyourbills.co.uk/

http://www.beatthatquote.com/insurance/

http://www.tescocompare.com/

http://www.moneysupermarket.com/money/


http://www.confused.com/money

http://www.gocompare.com/money/

http://www.comparethemarket.com/money/


http://moneyfacts.co.uk/

http://www.uswitch.com/money

http://www.money.co.uk/

http://www.knowyourmoney.co.uk/

http://www.moneynet.co.uk/

http://www.moneyexpert.com/

http://www.fairinvestment.co.uk/

http://www.lovemoney.com/

http://www.beatthatquote.com/insurance/

http://www.tescocompare.com/

258 of 264

insurance Car insurance

http://www.moneysupermarket.com/money/ 035 Not accredited Big 4

Car insurance http://www.confused.com/money 015 Not accredited Big 4 Car insurance http://www.gocompare.com/money/ 021 Accredited by BIBA Big 4 Car insurance

http://www.comparethemarket.com/money/ 014 Not accredited Big 4

Car insurance http://moneyfacts.co.uk/ 033 Not accredited Not Big 4 Car insurance http://www.uswitch.com/money 064 Not accredited Not Big 4 Car insurance http://www.quotezone.co.uk/ 045 Not accredited Not Big 4 Car insurance http://www.money.co.uk 011 Not accredited Not Big 4

Car insurance http://www.knowyourmoney.co.uk/ 025 Not accredited Not Big 4 √ Car insurance http://www.moneynet.co.uk/ 037 Not accredited Not Big 4 Car insurance http://www.moneyexpert.com/ 032 Not accredited Not Big 4 Car insurance http://www.fairinvestment.co.uk/ 058 Not accredited Not Big 4 Car insurance http://www.lovemoney.com/ 027 Not accredited Not Big 4



Rail Fares http://www.travelsupermarket.com/c/train-tickets/ 060

Accredited by National Rail Big 4

Rail Fares http://www.thetrainline.com/buytickets/? 057 Accredited by National Rail Not Big 4

Rail Fares http://railsaver.co.uk/ 047 Not accredited Not Big 4

Rail Fares http://www.redspottedhanky.com/ 049 Accredited by National Rail Not Big 4 √

Rail Fares http://www.raileasy.co.uk 046 Accredited by National Rail Not Big 4

Rail Fares http://www.mytrainticket.co.uk/ 038 Accredited by National Rail Not Big 4

Rail Fares http://www.nationalrail.co.uk/ 039 Not accredited Not Big 4



http://www.confused.com/money

http://www.gocompare.com/money/



http://moneyfacts.co.uk/

http://www.uswitch.com/money

http://www.money.co.uk/

http://www.knowyourmoney.co.uk/

http://www.moneynet.co.uk/

http://www.moneyexpert.com/

http://www.fairinvestment.co.uk/

http://www.lovemoney.com/

http://www.travelsupermarket.com/c/train-tickets/

http://www.travelsupermarket.com/c/train-tickets/

http://www.thetrainline.com/buytickets/?

http://railsaver.co.uk/

http://www.redspottedhanky.com/

http://www.raileasy.co.uk/

http://www.mytrainticket.co.uk/

http://www.nationalrail.co.uk/

259 of 264

Market Price comparison website PCW code number


Package Holidays

http://www.travelsupermarket.com/holidays/ 060 Not accredited Big 4

Package Holidays http://travel.confused.com/ 015 Not accredited Big 4 Package Holidays

http://www.gcholidayrewards.com/holidays.asp 021 Not accredited Big 4

Package Holidays http://www.expedia.co.uk/ 019 Not accredited Not Big 4 Package Holidays

http://www.lastminute.com/?skin=engb.lastminute.com 026 Not accredited Not Big 4

Package Holidays http://www.skyscanner.net/ 053 Not accredited Not Big 4 Package Holidays http://www.ebookers.com/ 016 IATA accredited Not Big 4 Package Holidays http://www.opodo.co.uk/ 044 Not accredited Not Big 4 Package Holidays http://www.cheapflights.co.uk/ 010 Not accredited Not Big 4

Package Holidays http://www.travelrepublic.co.uk 059 IATA accredited Not Big 4 √ Package Holidays http://lowcostholidays.com/ 028 Not accredited Not Big 4 Package Holidays http://www.kelkoo.co.uk/ 024 Not accredited Not Big 4 Package Holidays http://www.onthebeach.co.uk 043 Not accredited Not Big 4

http://www.travelsupermarket.com/holidays/

http://www.travelsupermarket.com/holidays/

http://travel.confused.com/

http://www.gcholidayrewards.com/holidays.asp

http://www.gcholidayrewards.com/holidays.asp

http://www.expedia.co.uk/

http://www.lastminute.com/?skin=engb.lastminute.com

http://www.lastminute.com/?skin=engb.lastminute.com

http://www.skyscanner.net/

http://www.ebookers.com/

http://www.opodo.co.uk/

http://www.cheapflights.co.uk/

http://www.travelrepublic.co.uk/

http://lowcostholidays.com/

http://www.onthebeach.co.uk/

260 of 264

SCENARIOS

Broadband You want broadband only and are browsing to see what's available in terms of deals/special offers. Try to avoid inputting any information about what you want, it's a general "Best deals" or "February's specials" that we want.

Broadband You are looking for broadband (only) at a speed of 24Mb or more and are searching to see what's available for your phone number/postcode. Make sure you note any details you have to put in (e.g. whether you have a BT line, etc.)

Broadband You are looking for broadband (only) and would like to go with O2. What's available from them? You will have to input your postcode and home phone number. Make sure you note any details you have to put in (e.g. whether you have a BT line, etc.)

Broadband You want broadband only on a contract no longer than 3 months (i.e. either monthly or 3 monthly). You’ll have to input your postcode and home phone number. Make sure you note any details you have to put in (e.g. whether you have a BT line, etc.)

Broadband You are looking for a home phone + broadband + tv package - you don't mind who supplies it. You will have to input your postcode and home phone number. Make sure you note any details you have to put in (e.g. whether you have a BT line, etc.)

Broadband You're interested in mobile broadband and are browsing to see what's available in terms of deals and special offers. For this search try to avoid inputting any information about what you want, it's a general "Best deals" or "February's specials" that we want. Note that it's mobile broadband

Mobile phones You want an Apple iPhone 4 32GB on a contract with any mobile network

Mobile phones You want a contract with Vodafone and are looking to see the different tariffs available. You're not bothered about the make of phone

Mobile phones You are browsing to see what the website is offering in terms of deals and special offers

Mobile phones You'd like a PAYG package and are looking for the cheapest deals

Mobile phones You want a Nokia phone but aren't bothered about the model. You don't mind who the network supplier is, but you want a contract not PAYG

Mobile phones You want a contract with T Mobile and are looking to see the different tariffs available. You're not bothered about the make of phone

Energy You're thinking about changing your energy supplier. Look at dual fuel (gas and electricity) prices for your address.

Energy You're thinking about changing your energy supplier. Look at dual fuel (gas and electricity) prices for your address. Ideally you would like a fixed or capped tariff

Energy You're thinking about changing your electricity supplier (but not your gas). Look at electricity prices for your address.

Energy You're thinking about changing your energy supplier. Look at dual fuel (gas and electricity) prices for your address. Ideally you would like a green or environmental tariff

Energy You're thinking about changing your gas supplier (but not your electricity). You want to pay quarterly by cash/cheque/credit card/debit card (i.e. you don't want to use Direct Debit). Look at gas prices for your address.

261 of 264

Energy You're thinking about changing your energy supplier. Look at dual fuel (gas and electricity) prices for your address. You want to pay by Direct Debit

Car insurance Your car insurance is due for renewal exactly 4 weeks from the date of your search and you want to compare quotes. You want fully comprehensive insurance and a voluntary excess of £250. The insurance is for you only. You want to pay monthly.

Car insurance Your car insurance is due for renewal exactly 3 weeks from the date of your search and you want to compare quotes. You want 3rd Party Fire & Theft. The insurance is for you and your partner. You want to pay monthly.

Car insurance Your car insurance is due for renewal exactly 3 weeks from the date of your search and you want to compare quotes. You want fully comprehensive insurance. The insurance is for you and your partner and your annual mileage is around 25,000 miles. You want to pay monthly.

Car insurance Your car insurance is due for renewal exactly 4 weeks from the date of your search and you want to compare quotes. You want fully comprehensive insurance for yourself only. You don't have any no claims bonus. You want to pay monthly.

Car insurance Your car insurance is due for renewal exactly 2 weeks from the date of your search and you want to compare quotes. You want 3rd Party Fire & Theft. The insurance is for you and your partner and you want to pay a lump sum up front as opposed to monthly instalments.

Car insurance Your car insurance is due for renewal exactly 3 weeks from the date of your search and you want to compare quotes. You want fully comprehensive insurance and a protected no claims bonus. The insurance is for you only. You want to pay the whole amount in one go.

Home insurance

You are looking for contents insurance for your home and want to compare quotes. Your current policy expires exactly 2 weeks from the date of your search. You have never made a previous building or contents insurance claim. The value of your contents is £50,000 and you'd like to insure your laptop when its out of the home as well (value £500). You want to pay monthly.

Home insurance

You are looking for building & contents insurance for your home and want to compare quotes. Your current policy expires exactly 3 weeks from the date of your search. You have never made a previous building or contents insurance claim. You want to pay monthly.

Home insurance

You are looking for buildings insurance for your home and want to compare quotes. You made a £2,000 claim on your current policy a year ago because there was a small fire. Your current policy expires on 30th March. You want to pay monthly.

Home insurance

You are looking for building & contents insurance for your home and want to compare quotes. You want accidental damage cover to be included. Your current policy expires exactly 3 weeks from the date of your search. You want to pay monthly.

Home insurance

You are looking for building & contents insurance for your home and want to compare quotes. You have a tv worth £1,800 but nothing else worth £1,000 or over. Your current policy expires exactly 4 weeks from the date of your search. You want to pay a lump sum for the year up front.

Home insurance

You are looking for contents insurance for your home and want to compare quotes. Your current policy expires on 2nd April. You want to pay a lump sum for the year up front. Your contents are worth £40,000.

262 of 264

Rail fares You’re planning a day trip from Birmingham New Street to Blackpool Pleasure Beach on 18th April. You will be travelling with your partner and want a train around 10am, returning sometime later the same day.

Rail fares You're going from Brighton to Edinburgh, and because you don't want to go on the tube you need a train that goes from Brighton to St Pancras so that you can walk to Kings Cross for the Edinburgh train. You will leave Brighton about 8am on a date that's within 7 days of the day you do the initial shop, and you will leave Edinburgh the following day around 8am as well. When you do your repeat visits use the same date.

Rail fares You're looking for a cheap single ticket from London to Leeds. Choose a date that's within 7 days of the day you do your initial shop. When you do your repeat visits use the same date.

Rail fares You and 2 children are going to York from Glasgow Queen Street and you want to go via Edinburgh Waverley. You're travelling on 16th April and will be getting an open return.

Rail fares You are travelling from Cardiff Central to Bristol Temple Meads in mid April (you're not too bothered about the exact date) You want a train leaving around noon and will return exactly one week later at about the same time of day.

Rail fares You and a friend are looking for tickets from Nottingham to London St Pancras (return). You will travel before 7am on 22nd May and return between 5pm and 6pm the following day.

Package holiday

You are looking for an all-inclusive deal somewhere in Spain for you and your partner, for a fortnight in July, flying from whatever your most convenient airport is.

Package holiday

You are looking for a holiday in Bangkok in September for you and a friend. You want 7 nights bed & breakfast and will share a room. You will fly from whatever your most convenient local airport is. (If no deals are available for Bangkok choose another long-haul destination)

Package holiday

You are browsing to see what the website is offering in terms of deals and special offers. You don't mind when or where you go and can stay for anything up to 2 weeks. You will fly from whatever your most convenient local airport is.

Package holiday

You fancy a City Break and want to see what deals are available for April. You want a hotel for 3-4 nights. You will be going on your own. You will fly from whatever your most convenient local airport is.

Package holiday

You want a week in Sharm El Sheik in Egypt half-board for you, your partner and 2 friends (who are a couple and will hence share a room). You want a hotel that's 3 stars or above. You will fly from whatever your most convenient local airport is. (If no deals are available for Sharm El Sheik choose another destination e.g. Turkey, Dubai)

Package holiday

You want to take your children aged 5 and 10 to Disney World in Florida and are looking for a deal that allows you to stay near the park. You will be flying to Orlando from whatever your most convenient local airport is. You will be going in August for 2 weeks and want self-catering accommodation.

263 of 264

APPENDIX C

Pilot survey and amendments made

264 of 264

PILOT

The pilot comprised seven desktop searches, one for each of:

energy

broadband

mobile phones

car insurance

home insurance

rail fares

package holidays

It also included two mystery shops per market ie 14 in total.

During the pilot, the following amendments were made:

1. One of the scenarios for insurance said it was needed from a point 5 weeks ahead, but the site only

allowed 30 days in advance. The scenario was changed to 3 weeks.

2. One of the package holiday scenarios was to book a long-haul trip to Bangkok, but none were

available on the website. The scenario was amended to be any long-haul destination if Bangkok was

unavailable. Similarly, the scenario involving Sharm El Sheik was expanded to include a substitute

destination such as Turkey or Dubai

3. Filtering and reordering by brand was added as an option to the mobile phones questionnaire

4. PAYG mobile phone deals are a one-off cost, while contracts are monthly payments. Hence the

instruction to ‘Write in the monthly cost for this package’ was altered to be ‘Write in the monthly cost

for this package if it is a contract. If it is a PAYG deal write in the one-off cost shown’

Consumer Focus Victoria House Southampton Row London WC1B 4AD

t: 020 7799 7900f: 020 7799 7901e: [email protected]: consumerfocus.org.uk

Media team: 020 7799 8004/8006

Published: February 2013

If you have any questions or would like further information about our research, please contact Marzena Lipman by telephone on 020 7799 7981 or via email [email protected]

For regular updates from Consumer Focus, sign up to our monthly e-newsletter by emailing [email protected] or follow us on Twitter http://twitter.com/consumerfocus

If you require this publication in Braille, large print or on audio CD please contact us.

Deaf, hard of hearing or speech-impaired consumers can contact Consumer Focus via Text Relay:

From a textphone, call 18001 020 7799 7900

From a telephone, call 18002 020 7799 7900

consumerfocus.org.uk

Price comparison website mystery shopping report for ... · comparison website.4 Today consumers...

Documents

Transcript of Price comparison website mystery shopping report for ... · comparison website.4 Today consumers...