11.02.2011

1

The Influence of Commercial Intent of Search Results on Their Perceived Relevance

Dirk Lewandowski Hamburg University of Applied Sciences, Germany

Introduction

•  Ongoing discussion about results quality –  Did Google’s results really get worse?

–  Do search engine results (and especially, Google’s) get more and more commercial?

•  Discussion on biased search results / “search neutrality” –  Does Google prefer its own services in its rankings?

•  Objective –  To find out whether commercial results have a negative effect on relevance as

perceived by the users.

11.02.2011

2

Search engine market share (Germany)

Previous work

•  Retrieval effectiveness tests –  Lots of work done

–  Main objective is to compare results quality of commercial Web search engines

–  Tests are highly standardized

•  Commercial intent –  Some studies on commercial intent of the searchers

–  No studies on the commercial intent of the documents

11.02.2011

3

Research questions

•  RQ1: How effective are web search engines in retrieving relevant results?

•  RQ2: What ratio of the top 10 search engine results are pages with a commercial intent?

•  RQ3: Are results with a commercial intent comparably relevant to non-commercial results?

•  RQ4: Do banner or textual ads have a negative effect on the perceived relevance?

Test design

•  Retrieval effectiveness test –  To collect relevance judgments

•  Results classification –  Classify each result according to its commercial intent

•  Commercial intent of the page/website itself

•  Commercial intent as expressed through advertisements on the page

11.02.2011

4

Retrieval effectiveness test design

•  50 queries –  “Real life queries” from students asked for their last query entered into a Web

search engine

–  Mainly simple queries without an explicit commercial intent

•  3 search engines –  Google, Microsoft, Yahoo (German user interfaces)

•  10 results per query

•  Binary relevance judgments

•  50 undergraduate students as jurors

Retrieval effectiveness test: Data collection

•  Results from all search engines collected

•  Results –  were made anonymous

–  were randomized

•  Each juror evaluated the complete results set for one query

11.02.2011

5

Results classification (1)

•  Commercial 1.  A product is sold on the page

2.  Webpage of a company

•  Non-commercial 3.  Page of a public authority, government office, administration, etc.

4.  Page of a non-governmental organization, association, club, etc.

5.  Private page

6.  Page of an institution of higher education

•  Other

Results classification (2)

•  Number of banner ads

•  Number of text-based ads (“sponsored links”)

11.02.2011

6

Limitations

•  Results cannot be generalised –  Commercial intent of results does not have a negative influence on their

perceived relevance when asking undergraduate students.

•  Relatively small dataset –  500 URLs per search engine (total of 1,500 URLs)

Results

11.02.2011

7

Results

1.  Results from the classification task

2.  Results from the retrieval effectiveness test

3.  Combined results: Influence of commercial intent on the perceived relevance

Results

1.  Results from the classification task

11.02.2011

8

Results with a commercial intent account for

–  82% (Google) –  75% (Microsoft) –  78% (Yahoo)

Results from the classification task (1)

Results from the classification task (2)

•  Lowest ratio of results with a commercial intent on the first position.

11.02.2011

9

Results from the classification task (3)

•  Google has a significantly higher ratio of results with sponsored links. –  Google: 23.7% of all results –  Microsoft: 12.3%

–  Yahoo: 17.7%

•  No significant difference between the ratios of results with banner ads. –  Google: 21.3% of all results –  Microsoft: 23.0%

–  Yahoo: 22.0%

Results

•  Results from the retrieval effectiveness test

11.02.2011

10

Macro precision Number of queries answered best by an individual search engine

Mean Average Precision All search engines, all results

11.02.2011

11

Results

•  Combined results: Influence of commercial intent on the perceived relevance

Relevance of results with a commercial intent

•  Relevance of results with commercial intent is comparable to non-commercial results

–  Google: 59.4% of commercial results relevant (vs. 58.7%)

–  Microsoft: 34.6% vs. 36.5%

–  Yahoo: 51.1% vs. 51.7%

11.02.2011

12

Relevance of results with advertising

•  Relevance of results with text-based ads is comparable to results without text-based ads

–  Google: 59.8% vs. 59.3%

–  Microsoft: 40.0% vs. 33.9%

–  Yahoo: 52.7% vs. 52.0%

•  Relevance of results with banner ads is comparable to results without banner ads

–  Google: 58.5% vs. 59.8%

–  Microsoft: 33.3 vs.35.1% –  Yahoo: 55.8% vs. 49.1%

Discussion

11.02.2011

13

Discussion

•  Most striking result is that Google has significantly more results with text-based ads.

•  Search Engine Optimization (SEO) may be a reason –  Many pages are highly optimised for Google.

•  Wouldn’t it be clever to prefer pages carrying ads from the search engine’s own ads system?

–  However, Google says it doesn’t do this.

Google‘s advertising model

11.02.2011

14

Conclusion & further research

Conclusion

1.  In terms of retrieval effectiveness, Google outperforms the other two search engines (RQ1).

2.  There is only a slight variation in the ratio of commercial results from search engine to search engine (RQ2).

3.  The results classified as “commercial” are comparably relevant to the results classified as “non-commercial” (RQ3).

4.  Textual or banner ads do not have a negative effect on the perceived relevance of the corresponding result (RQ4).

11.02.2011

15

Further research

1.  Study should be repeated using other user groups –  Relevance judgments came from a comparably homogeneous user group.

–  Data already collected for other user groups (another student group; librarians; crowd-sourcing).

2.  Large ratio of results with text-based ads in Google deserves further investigation.

–  Currently, we conduct a large-scale study where we automatically detect text-based ads on search results, comparing Google and Bing.

•  30,000 queries, 2 search engines, 30 results per query 1.8 Million URLs.