![Text Retrieval and Mining Lecture 12 [Borrows slides from Viktor Lavrenko and Chengxiang Zhai]](https://static.fdocuments.net/doc/165x107/56649f3f5503460f94c6042b/text-retrieval-and-mining-lecture-12-borrows-slides-from-viktor-lavrenko-and.jpg)
Text Retrieval and Mining Lecture 12 [Borrows slides from Viktor Lavrenko and Chengxiang Zhai]
BiasTrust: Teaching Biased Users About Controversial Topics V.G.Vinod Vydiswaran, ChengXiang Zhai,...
1
BiasTrust: Teaching Biased Users About Controversial Topics V.G.Vinod Vydiswaran, ChengXiang Zhai, Dan Roth University of Illinois at Urbana-Champaign Peter Pirolli Palo Alto Research Center CIKM 2012 Research Poster: Information Retrieval Track Research Question Interface variants Exposing conflicting viewpoints Acknowledgments This research was supported by the Multimodal Information Access and Synthesis (MIAS) Center at the University of Illinois at Urbana-Champaign, part of CCICADA, a DHS Science and Technology Center of Excellence, and grants from Boeing, as part of the Information Trust Institute. Some part of this research was done at Palo Alto Research Center, with Contact details vgvinodv@illinois .edu, [email protected] u, [email protected] , Examples of Controversial Topics Conclusions BiasTrust: User Study details User Study findings Pre-survey Post-survey Expertise Source Evidence Agreement Novelty Bias Study phase Show similarShow contrast Quit | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Contras tive Sing le Document position Readership (in %) A. Contrastive display improves readership Area Under Curve Single display Contrasti ve display Readership change Absolute Relative Top 10 pairs 45.00 % 64.44 % Contrast docs only 22.00 % 64.44 % Expertise rating (in “stars”) Readership (in %) Expertise rating Documents rated uniformly at random Documents rated 1 or 3 Milk contains nine essential nutrients… Dairy products add significant amounts of cholesterol and saturated fat to the diet... The protein in milk is high quality, which means it contains all of the essential amino acids or 'building blocks' of protein. Milk proteins, milk sugar, and saturated fat in dairy products pose health risks for children and encourage the development of obesity, diabetes, and heart disease... Drinking of cow milk has been linked to iron-deficiency anemia in infants and children It is long established that milk supports growth and bone development One outbreak of development of enlarged breasts in boys and premature development of breast buds in girls in Bahrain was traced to ingestion of milk from a cow given continuous estrogen treatment by its owner to ensure uninterrupted milk production. rbST [man-made bovine growth hormone] has no biological effects in humans. There is no way that bST [naturally- occurring bovine growth hormone] or rbST in milk induces early puberty. Yes No Every coin has two sides BiasTrust Task setup Understand how human biases affect learning of controversial topics Study user interface factors that help in such learning People tend to be biased They may be exposed to only one side of the story Presence of Confirmation bias Effects of filter bubble Subjects learn more about a “controversial” topic Subjects are shown quotes (documents) from experts on the topic Expertise varies, is subjective Subjects are asked to judge if quotes are biased, informative, interesting Pre- and post-surveys measure extent of learning Is milk good for you? Is organic milk healthier? Does milk cause early puberty? Are alternative energy sources viable? Nuclear? Solar? Clean coal? Additional findings Does contrastive display help or hinder learning? Do multiple documents per page affect learning? Does sorting results by topic help? What is the effect of display of source expertise on readership? which documents subjects consider biased, novel, or agree with? 40 study sessions from 24 participants Average age of subjects: 28.6 ± 4.9 years Time to complete one study session: 45 min Challenges in claim verification Claim Sourc e Data User s Evidence ClaimVerifier What kind of data can be utilized? How to find relevant pieces of evidence ? Are sources trustworth y? How to present evidence ? How to assign truth values to textual claims? How to address user bias? How to build trust models that make use of evidence? Data/Language Understanding Algorithmic/Computational Issues HCI Issue s [KDD-DMH’11] [KDD’11, ECIR’12] [C IKM’12, ASIS&T’12] B. Readership higher for expert documents C. Learning improves with contrastive display Higher expertise documents tend to be read more. Lower expertise documents are read less if ratings were shown than when ratings were not shown. Showing multiple documents per page increases readership. Subjects learned more about the topics they did not know about. Particulars Overall Milk Energy Number of documents read 18.6 20.1 17.1 Number of documents skipped 12.6 13.0 12.1 Time spent (in min) 26.5 26.5 26.6 Particul ars Tota l Change Milk 9 7 2 +12.3 % * Energy 13 85 + 3.3 % Particul ars Tota l Change Milk 11 2 9 - 31.0 % * Energy 7 2 5 - 27.9 % * Significant improvement (at p = 0.05 level) in mean knowledge rating for Milk domain Significant improvement (at p=0.05 level) in bias rating for both domains Subjects tend to not read contrastive viewpoint, if it is not shown by default. Subjects read more when both viewpoints are shown. User study helped demonstrate need to show multiple viewpoints for claim verification. Knowledge of expertise rating helps users. Showing contrastive viewpoints helps subjects reduce strongly- held biases. This study helped us design better claim verification systems. (A) Single document, with option to view contrasting viewpoint (B) Single document, with contrasting viewpoint (C) Single vs. multiple documents; each with contrasting viewpoints Factors studied in BiasTrust This work address the challenge of how users perceive credible information. Studying user interactions will help us design better claim verification systems. Is it healthy to drink milk?
-
Upload
jocelyn-brown -
Category
Documents
-
view
219 -
download
0
Transcript of BiasTrust: Teaching Biased Users About Controversial Topics V.G.Vinod Vydiswaran, ChengXiang Zhai,...
BiasTrust: Teaching Biased Users About Controversial Topics
V.G.Vinod Vydiswaran, ChengXiang Zhai, Dan RothUniversity of Illinois at Urbana-Champaign
Peter PirolliPalo Alto Research Center
CIKM 2012 Research Poster: Information Retrieval TrackResearch Question Interface variantsExposing conflicting viewpoints
Acknowledgments This research was supported by the Multimodal Information Access and Synthesis (MIAS) Center at the University of Illinois at Urbana-Champaign, part of CCICADA, a DHS Science and Technology Center of Excellence, and grants from Boeing, as part of the Information Trust Institute. Some part of this research was done at Palo Alto Research Center, with grant support from the Office of Naval Research.
Contact details
[email protected], [email protected], [email protected], [email protected]
Examples of Controversial Topics
Conclusions
BiasTrust: User Study details
User Study findings
Pre-survey
Post-survey
Expertise
Source
Evidence
AgreementNovelty
Bias
Study phase
Show similar Show contrast Quit
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
Contrastive
Single
Document position
Rea
ders
hip
(in %
)
A. Contrastive display improves readership
Area Under Curve Single display
Contrastive display
Readership change Absolute Relative
Top 10 pairs 45.00 % 64.44 %
Contrast docs only 22.00 % 64.44 %
Expertise rating (in “stars”)
Rea
ders
hip
(in %
)
Expertise rating
Documents rated uniformly at random Documents rated 1 or 3
Milk contains nine essential nutrients… Dairy products add significant amounts of cholesterol and saturated fat to the diet... The protein in milk is high quality, which
means it contains all of the essential amino acids or 'building blocks' of protein.
Milk proteins, milk sugar, and saturated fat in dairy products pose health risks for children and encourage the development of obesity, diabetes, and heart disease...
Drinking of cow milk has been linked to iron-deficiency anemia in infants and children
It is long established that milk supports growth and bone development
One outbreak of development of enlarged breasts in boys and premature development of breast buds in girls in Bahrain was traced to ingestion of milk from a cow given continuous estrogen treatment by its owner to ensure uninterrupted milk production.
rbST [man-made bovine growth hormone] has no biological effects in humans. There is no way that bST [naturally-occurring bovine growth hormone] or rbST in milk induces early puberty.
Yes No
Every coin has two sides
BiasTrust Task setup
Understand how human biases affect learning of controversial topics
Study user interface factors that help in such learning
People tend to be biased
They may be exposed to only one side of the story
Presence of Confirmation bias
Effects of filter bubble
Subjects learn more about a “controversial” topic Subjects are shown quotes
(documents) from experts on the topic Expertise varies, is subjective
Subjects are asked to judge if quotes are biased, informative, interesting
Pre- and post-surveys measure extent of learning
Is milk good for you? Is organic milk healthier? Does milk cause early puberty?
Are alternative energy sources viable? Nuclear? Solar? Clean coal?
Additional findings
Does contrastive display help or hinder learning?
Do multiple documents per page affect learning?
Does sorting results by topic help? What is the effect of display of source
expertise on readership?which documents subjects consider
biased, novel, or agree with?
40 study sessions from 24 participants Average age of subjects: 28.6 ± 4.9 years Time to complete one study session: 45 min
Challenges in claim verification
ClaimSource
Data
Users
Evidence
ClaimVerifier
What kind of data can be utilized?
How to find relevant pieces of evidence ?
Are sources trustworthy? How to
present evidence?
How to assign truth values to textual claims?
How to address user bias?
How to build trust models that make use of evidence?
Data/Language Understanding
Algorithmic/Computational Issues
HCI Issues
[KDD-DMH’11]
[KDD’11, ECIR’12][C
IKM’12, A
SIS&T’12]
B. Readership higher for expert documents C. Learning improves with contrastive display
Higher expertise documents tend to be read more. Lower expertise documents are read less if ratings
were shown than when ratings were not shown.
Showing multiple documents per page increases readership.
Subjects learned more about the topics they did not know about.
Particulars Overall Milk Energy
Number of documents read 18.6 20.1 17.1
Number of documents skipped 12.6 13.0 12.1
Time spent (in min) 26.5 26.5 26.6
Particulars Total Change
Milk 9 7 2 +12.3 % *
Energy 13 8 5 + 3.3 %
Particulars Total Change
Milk 11 2 9 - 31.0 % *
Energy 7 2 5 - 27.9 % *
Significant improvement (at p = 0.05 level) in mean knowledge rating for Milk domain
Significant improvement (at p=0.05 level) in bias rating for both domains
Subjects tend to not read contrastive viewpoint, if it is not shown by default.
Subjects read more when both viewpoints are shown.
User study helped demonstrate need to show multiple viewpoints for claim verification.
Knowledge of expertise rating helps users.
Showing contrastive viewpoints helps subjects reduce strongly-held biases.
This study helped us design better claim verification systems.
See additional results in Vydiswaran, V., Zhai, C., Roth, D., and Pirolli, P., ASIS&T 2012
(A) Single document, with option to view contrasting viewpoint
(B) Single document, with contrasting viewpoint
(C) Single vs. multiple documents; each with contrasting viewpoints
Factors studied in BiasTrust
This work address the challenge of how users perceive credible information.
Studying user interactions will help us design better claim verification systems.
Is it healthy to drink milk?
![CS276A Text Retrieval and Mining Lecture 12 [Utilizando slides de Viktor Lavrenko e Chengxiang Zhai]](https://static.fdocuments.net/doc/165x107/552fc132497959413d8d6a63/cs276a-text-retrieval-and-mining-lecture-12-utilizando-slides-de-viktor-lavrenko-e-chengxiang-zhai.jpg)
