BiasTrust: Teaching Biased Users About Controversial Topics

V.G.Vinod Vydiswaran, ChengXiang Zhai, Dan RothUniversity of Illinois at Urbana-Champaign

Peter PirolliPalo Alto Research Center

CIKM 2012 Research Poster: Information Retrieval TrackResearch Question Interface variantsExposing conflicting viewpoints

Acknowledgments This research was supported by the Multimodal Information Access and Synthesis (MIAS) Center at the University of Illinois at Urbana-Champaign, part of CCICADA, a DHS Science and Technology Center of Excellence, and grants from Boeing, as part of the Information Trust Institute. Some part of this research was done at Palo Alto Research Center, with grant support from the Office of Naval Research.

Contact details

vgvinodv@illinois.edu, czhai@illinois.edu, danr@illinois.edu, Peter.Pirolli@parc.com

Examples of Controversial Topics

Conclusions

BiasTrust: User Study details

User Study findings

Pre-survey

Post-survey

Expertise

Source

Evidence

AgreementNovelty

Bias

Study phase

Show similar Show contrast Quit

| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |

Contrastive

Single

Document position

Rea

ders

hip

(in %

)

A. Contrastive display improves readership

Area Under Curve Single display

Contrastive display

Readership change Absolute Relative

Top 10 pairs 45.00 % 64.44 %

Contrast docs only 22.00 % 64.44 %

Expertise rating (in “stars”)

Rea

ders

hip

(in %

)

Expertise rating

Documents rated uniformly at random Documents rated 1 or 3

Milk contains nine essential nutrients… Dairy products add significant amounts of cholesterol and saturated fat to the diet... The protein in milk is high quality, which

means it contains all of the essential amino acids or 'building blocks' of protein.

Milk proteins, milk sugar, and saturated fat in dairy products pose health risks for children and encourage the development of obesity, diabetes, and heart disease...

Drinking of cow milk has been linked to iron-deficiency anemia in infants and children

It is long established that milk supports growth and bone development

One outbreak of development of enlarged breasts in boys and premature development of breast buds in girls in Bahrain was traced to ingestion of milk from a cow given continuous estrogen treatment by its owner to ensure uninterrupted milk production.

rbST [man-made bovine growth hormone] has no biological effects in humans. There is no way that bST [naturally-occurring bovine growth hormone] or rbST in milk induces early puberty.

Yes No

Every coin has two sides

BiasTrust Task setup

Understand how human biases affect learning of controversial topics

Study user interface factors that help in such learning

People tend to be biased

They may be exposed to only one side of the story

Presence of Confirmation bias

Effects of filter bubble

Subjects learn more about a “controversial” topic Subjects are shown quotes

(documents) from experts on the topic Expertise varies, is subjective

Subjects are asked to judge if quotes are biased, informative, interesting

Pre- and post-surveys measure extent of learning

Is milk good for you? Is organic milk healthier? Does milk cause early puberty?

Are alternative energy sources viable? Nuclear? Solar? Clean coal?

Additional findings

Does contrastive display help or hinder learning?

Do multiple documents per page affect learning?

Does sorting results by topic help? What is the effect of display of source

expertise on readership?which documents subjects consider

biased, novel, or agree with?

40 study sessions from 24 participants Average age of subjects: 28.6 ± 4.9 years Time to complete one study session: 45 min

Challenges in claim verification

ClaimSource

Data

Users

Evidence

ClaimVerifier

What kind of data can be utilized?

How to find relevant pieces of evidence ?

Are sources trustworthy? How to

present evidence?

How to assign truth values to textual claims?

How to address user bias?

How to build trust models that make use of evidence?

Data/Language Understanding

Algorithmic/Computational Issues

HCI Issues

[KDD-DMH’11]

[KDD’11, ECIR’12][C

IKM’12, A

SIS&T’12]

B. Readership higher for expert documents C. Learning improves with contrastive display

Higher expertise documents tend to be read more. Lower expertise documents are read less if ratings

were shown than when ratings were not shown.

Showing multiple documents per page increases readership.

Subjects learned more about the topics they did not know about.

Particulars Overall Milk Energy

Number of documents read 18.6 20.1 17.1

Number of documents skipped 12.6 13.0 12.1

Time spent (in min) 26.5 26.5 26.6

Particulars Total Change

Milk 9 7 2 +12.3 % *

Energy 13 8 5 + 3.3 %

Particulars Total Change

Milk 11 2 9 - 31.0 % *

Energy 7 2 5 - 27.9 % *

Significant improvement (at p = 0.05 level) in mean knowledge rating for Milk domain

Significant improvement (at p=0.05 level) in bias rating for both domains

Subjects tend to not read contrastive viewpoint, if it is not shown by default.

Subjects read more when both viewpoints are shown.

User study helped demonstrate need to show multiple viewpoints for claim verification.

Knowledge of expertise rating helps users.

Showing contrastive viewpoints helps subjects reduce strongly-held biases.

This study helped us design better claim verification systems.

See additional results in Vydiswaran, V., Zhai, C., Roth, D., and Pirolli, P., ASIS&T 2012

(A) Single document, with option to view contrasting viewpoint

(B) Single document, with contrasting viewpoint

(C) Single vs. multiple documents; each with contrasting viewpoints

Factors studied in BiasTrust

This work address the challenge of how users perceive credible information.

Studying user interactions will help us design better claim verification systems.

Is it healthy to drink milk?