Post on 14-Apr-2017
Revealing the Hidden Patterns of News Photos: Analysis of Millions of News Photos through GDELT and Deep Learning-based Vision APIs
Haewoon Kwak Jisun An
Qatar Computing Research InstituteHamad Bin Khalifa University
2
3
4
Roles of News Photos
● Influence people’s perception● Enhance reader’s memory● Deliver emotion otherwise hard to be
conveyed
5
6
7
Why was this photo not picked?(source: https://www.donaldjtrump.com)
8
To Understand Messages of Photos
● We need to know○ What are shown in the photos○ How they are portrayed
9
Challenges in News Photo Analysis
● Text mining has been a useful tool for analyzing news text
→ What is the appropriate tool forexamining news photos?
10
Conventional Tool for Photo Analysis
● Manual coding … hard to scale
11
Deep Learning for Image Recognition
12
Object Recognition
13
Emotion Detection
https://www.microsoft.com/cognitive-services/en-us/emotion-api 14
Deep learning enables us to study news photos in large-scale
15
Goal of This Work
● To offer a general understanding of news photos ○ What are shown in the photos?○ How are people portrayed?
■ From the perspective of emotion■ From the perspective of gender
● Case study: Portrayal of politicians
16
● We can crawl photos from news websites and analyze them
● But, setting the deep learning framework and training it take time/money/...
Data Collection
17
18
GDELT Visual GKG (VGKG)
● Collects news articles around the world● Extract photos from the articles● Calls Google Cloud Vision API to analyze
photos
● VGKG is available since 1 Jan 2016 http:
//blog.gdeltproject.org/announcing-the-new-gdelt-visual-global-knowledge-graph-vgkg/
19
Example of VGKG
20160101004500 http://www.bbc.co.uk/news/uk-35205943 http://ichef.bbci.co.uk/news/1024/cpsprodpb/B89F/production/_87436274_87436273.jpg profession<FIELD>0.95780987<FIELD>/m/063km<RECORD>person<FIELD>0.85714287<FIELD>/m/01g317<RECORD>close up<FIELD>0.82379222<FIELD>/m/02cqfm<RECORD>bishop<FIELD>0.78259438<FIELD>/m/01b7b<RECORD>bishop<FIELD>0.71334475<FIELD>/m/027k49j<RECORD>diocesan bishop<FIELD>0.64282793<FIELD>/m/09sgrf<RECORD>auxiliary bishop<FIELD>0.57118613<FIELD>/m/05mx3n<RECORD>clergy<FIELD>0.57113737<FIELD>/m/0db79 -2<FIELD>-2<FIELD>-2<FIELD>-2 0.95443642<FIELD>3.199043<FIELD>12.419704<FIELD>-7.179338<FIELD>0.621747<FIELD>433,215;575,215;575,357;433,357<FIELD>-2<FIELD>-2<FIELD>-2<FIELD>2<FIELD>-2<FIELD>-2<FIELD>-2
20
Date, Document identifier (URL), Image URL, Labels (description, confidence score, unique id), Geographic Landmarks, Logos, Safe Search, Faces (Angle, Emotion, etc.), OCR
(Potential) Limitations of GDELT
● List of news sources is not explicitly announced (also, growing) - coverage bias might exist
● Our work of comparing GDELT with another news dataset will be presented in the poster session
Two Tales of the World: Comparison of Widely Used World News Datasets - GDELT and EventRegistry Haewoon Kwak and Jisun AnICWSM'16: The 10th International Conference on Web and Social Media (poster), 2016 21
Our Dataset - Full
● GKG and VGKG in January 2016● Popularity measured by Alexa.com
22
Our Dataset - 7 Popular News Media
● Top 30 & > 1K records
23
Data Preprocessing
● Keep labels whose confidence score ≥ .8
http://i2.cdn.turner.com/cnnnext/dam/assets/160116174054-kerry-handshake-zarif-large-169.jpg
Person 0.84957772Business 0.59667766
24
What Are Shown in the Photos?Common Objects in News Photos
25
News Topics and Relevant Photos
● News photos should relate with topics of news articles→ Common objects might be different across topics
● CNN has ‘section’ info. in its URLhttp://edition.cnn.com/2016/04/07/travel/japan-best-of-wakayama/index.htmlhttp://edition.cnn.com/2016/05/05/politics/paul-ryan-donald-trump-republican-resistance/index.html 26
Person is the Most Common Object
27
But, in Travel, Person is Uncommon
28
Region-related Sections
29
● Why does this matter?
Western Media and the Third World
● Golan reports that western mass media strengthen the portrayal of the third world by reporting war, poverty, famine, conflicts, violence and conflicts and lead to negative perception (Golan 2008).
30
How CNN deals with MENA region?
31
How Are People Portrayed? From the Perspective of Emotion
32
Classification of Emotions
33https://articulation360.wordpress.com/2011/08/26/emotions-memory-game/
Google API Can Detect 4 Emotions
34https://articulation360.wordpress.com/2011/08/26/emotions-memory-game/
SURPRISE
SORROW ANGER
JOY
Neutral (75%) or Joy (24%)
● Among 11,127 faces (in 7 popular media), 2,740 faces (24.6%) have one of emotions
● Most of them (2,665 faces) express joy
35
Nonverbal & Verbal Communication
● Happy faces accelerate the cognitive processing of positive words and slow down that of negative words (Stenberg, Wiking, and Dahl 1998)
36
We Use Microsoft Face API
● Measures smiling intensity (0.0~1.0)
37
0.998 0.0 (baby)
https://www.microsoft.com/cognitive-services/en-us/face-api
Smile Comes with Positive Text
● Positive correlation between smile intensity and tone (sentiment) of the text
⍴=0.225
38
How Are People Portrayed? From the Perspective of Gender
39
Previous Studies on News Media
1. Men outnumber women2. Men and women are associated with
particular roles3. More women than men were depicted
as happy and calm.
→ We’ll verify this in large-scale
40
Again, We Use Microsoft Face API
41https://www.microsoft.com/cognitive-services/en-us/face-api
● Measures Gender and Age
Unequal Gender Representation
0.5
42
Stereotyping: Women in “Living”
0.5
43
Women Smile More Than Men
44
Younger Women, Older Men
45
Case StudyPortrayal of Politicians
46
Smiling Politicians
● Goodnow (2010) found that Obama smiles more than Clinton in photos in Time magazine
● Smile gives a positive, non-threatening impression to viewers (Goffman 1979)
47
Bias of CNN Toward Sanders?
(Smiling faces / All faces)
* CNN even uses “Sorrow” faces for Sanders
48
Pro-Clinton Media Behave Similarly
49
Summary and Future Work
50
Key Findings
● What are shown in the news photos○ People commonly appear (≥ 40.5% @top500)
● How they are portrayed○ People are neutral (75%) or smiling (24%)○ Gender representation is unequal○ Gender role stereotyping is found○ Women smile more and look younger than men
● Clinton smiles more than Sanders in some media
→We demonstrate the great potential of deep learning for computational journalism
51
Deeper Analysis on Text and Photos
● Headline and photos?● Topic and photos?● Keywords and photos?
52
Building PhotoBiasMeter.org
● Showing the preference of media outlets toward candidates over time
● Challenges○ Modeling complex dimension of
preference - “Smile” is only one dimension
53
@haewoonFull paper is available via
http://arxiv.org/abs/1603.04531
54