Extracting Brand Perceptions from Consumer CreatedImages: A Machine Learning Approach

Liu Liu1, Daria Dzyabura1, Natalie Mizik2

1New York University - Stern School of Business

2University of Washington - Foster School of Business

2016 Stanford GSB Digital Marketing Conference

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 1 / 27

Visual Content on the Rise

“3.8 trillion photos were taken in all of human history until mid-2011, but1 trillion photos were taken in 2015 alone...”(Kane & Pear, 2016)

New successful social media platforms emphasize visual content

e.g., Instagram users add an average of 95M photos/videos daily 1

1https://www.instagram.com/press/Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 2 / 27

Brands Embrace Visual Marketing

Companies develop visual stimuli to shape customers’ perceptions ofbrands

One-third of total annual marketing budgets was earmarked forcreating, producing, and promoting visual content in 2016 (Gujral,

2015).

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 3 / 27

Consumer-Created Brand Images (i.e., #brand)

Consumers post millions of photos online to share their experiences andcommunicate their feelings, thoughts, and attitudes.

They often hashtag brands and depict their interactions with brands

49,580,574 posts on Instagram with #nike (retrieved Nov. 2016)

#eddiebauerrugged

#pradaglamorous

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 4 / 27

Overview

Consumers share many images on social media, which contain brands

These images may contain valuable information about consumers’brand perceptions

Propose a method for extracting meaningful information about brandperceptions

Apply it to Instagram data for Apparel, Bevergages, and Industrialcorporations

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 5 / 27

Related Literature

Visual Design: color, shape, texture as fundamental elements of design(Hashimoto & Clayton, 2009; Dondis, 1974; Arnheim, 1954)

Computer Vision: extract quantifiable features (Shapiro & Stockman, 2001)

Visual Marketing: visual stimuli impact consumer behavior and perceptions(see (Wedel & Pieters, 2007) for a review)

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 6 / 27

Outline of the Talk

Methodology

Collect training dataExtract image featuresTrain and validate classifier out-of-sample

Application

Compare consumer and firm-created images to consumer brandperceptions measured in survey

Summary

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 7 / 27

1. Collect Images Labeled with Perceptual Attributes

Brand perceptual attributes:

{glamorous, rugged, fun, healthy, reliable, trustworthy}

Query Flickr: search for perceptual attributes and antonyms (Karayev et al.,

2013; Zhang, Korayem, Crandall, & LeBuhn, 2012; Dhar, Ordonez, & Berg, 2011;

McAuley & Leskovec, 2012)

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 8 / 27

glamorous drab rugged gentle

healthy unhealthy fun dull

reliable unreliable trustworthy untrustworthy

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 9 / 27

1. Collect Perceptual Attribute Labeled Images

Brand perceptual attributes:

{glamorous, rugged, fun, healthy, reliable, trustworthy}

Query Flickr: search for perceptual attributes and antonyms (Karayev et al.,

2013; Zhang et al., 2012; Dhar et al., 2011; McAuley & Leskovec, 2012)

About 4,000 images per perceptual attribute (half positive and halfnegative) and 23,404 in total

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 10 / 27

2. Extract Visual Features

(a) rugged (b) gentle

Color dark, high contrast light, warmerShape jagged corners rounded cornersTexture coarse, uneven smooth

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 11 / 27

2. Extract Visual Features

Colore.g., hue, saturation,

brightness

Shapee.g., line, corner,

edge/gradient direction

Texturee.g., local binary patter,

gabor filter

Color, shape, and texture are fundamental visual design elements(Hashimoto & Clayton, 2009)

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 12 / 27

List of Features by Feature Type

Feature Type Feature

ColorRGB color histogramHSV color histogramL*a*b color histogram

Shape

Line: number of straight linesLine: percentage of parallel linesLine: histogram of line orientations & distancesLine: histogram of line orientationsCorner: percentage of global cornersCorner: percentage of local cornersEdge Orientation HistogramHistogram of Oriented Gradients (HOG)

TextureLocal Binary Patter (LBP)Gabor

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 13 / 27

3. Train Classifier on Brand Perceptual Attributes

Input: {(xi , yi ), i = 1, ...,Np}

Classification function:

fp(xi ;wp, bp) = wTp xi + bp

s.t. yi f (xi ;wp, bp) > 0, i = 1, ...,Np

(1)

Support Vector Machine (SVM)

minwp ,bp ,

1

2wTp wp + C

Np∑i=1

ξi

s.t. yi (wTp xi + bp) ≥ 1− ξi , ξi ≥ 0, i = 1, ...,Np,

(2)

p: perceptual attributexi : D-dimensional visual feature vector for image iyi ∈ {−1,+1}: class labels

ξi : slack variables

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 14 / 27

Classification Performance

Train SVM with single type of feature and feature combinations (80%train and 20% test)

Perceptual AttributeAccuracy (holdout sample)

Best Model Color Shape Texture

glamorous 0.741 0.695 0.700 0.709rugged 0.733 0.656 0.700 0.672trustworthy 0.702 0.702 0.678 0.652fun 0.653 0.604 0.573 0.556healthy 0.634 0.634 0.560 0.514reliable 0.574 0.562 0.567 0.530

Visual Expressiveness: to what extent perceptual attribute can becaptured with features relating design elements

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 15 / 27

Feature Composition of Best Classifier

Perceptual Attribute Color Shape Texture

glamorous 1 1 1

rugged 1 1 0

trustworthy 1 0 0

fun 1 1 0

healthy 1 0 0

reliable 1 1 1

1 = feature included in best classifier, 0 = feature not include in best classifier

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 16 / 27

glamorous drab rugged gentle

healthy unhealthy fun dull

reliable unreliable trustworthy untrustworthy

Color histogram (RGB) computed from top 25 images that are most representative ofeach perceptual attribute and its antonymLiu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 17 / 27

Outline of the Talk

Methodology

Collect training dataExtract image featuresTrain and validate classifier out-of-sample

Application

Compare consumer and firm-created images to consumer brandperceptions measured in survey

Summary

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 18 / 27

Consumer-Created and Firm-Created Brand Images

Consumers: photos on Instagram(#brand)

Firms: photos on official accounts on Instagram

68 brands from 3 product categories: Apparel, Beverages, andIndustrial Corporations

About 2,000 consumer hashtagged photos per brand and 137,982photos in total74,907 photos in total from brands’ official accounts.

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 19 / 27

Brand Perceptual Attributes Expressed in Images

Images of brand j : I j = {I j1, ..., IjNj}

Classifier of perceptual attribute p: fp(x ;wp, bp)

Compute the ratio of brand j images that express the perceptualattribute

F{j , p} =

∑Nj

i=1 1(fp(xi ;wp, bp) > 0)

Nj, (3)

where Nj is number of photos of brand j , xi is the visual featurevector extracted from the i th image.

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 20 / 27

Example: Percentage of Images Expressing PerceptualAttribute

Prada vs. Eddie Bauer

Prada Eddie Bauer

glamorous 0.607 0.471rugged 0.343 0.406

P-value < 0.0001

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 21 / 27

Compare Consumer and Firm Images to Brand PerceptionSurvey

Young and Rubicams Brand Asset Valuator (BAV) (Lovett, Peres, & Shachar,

2014)

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 22 / 27

Pearson’s Correlation: Consumer vs. BAV, Consumer vs.Firm, Firm vs. BAV

ProductCategory

PerceptualAttribute

Consumer Imagevs.

BAV

Consumer Imagevs.

Firm Image

Firm Imagevs

BAV

Apparelglamorous

0.491**(p=0.0034)

N = 29

0.818**(p=3e-8)N = 29

0.581**(p=0.0005)

N = 29

rugged0.400*

(p=0.0157)N = 29

0.782**(p=3e-7)N = 29

0.430**(p=0.010)

N = 29

Beverageshealthy

0.451**(p=0.0090)

N = 27

0.218(p=0.1533)

N = 24

0.314(p=0.0673)

N = 24

fun0.346*

(p=0.0387)N = 27

0.554**(p=0.0025)

N = 24

0.228(p=0.1422)

N = 24

glamorous0.198

(p=0.1608)N = 27

0.595**(p=0.0011)

N = 24

0.364*(p=0.0404)

N = 24

rugged0.400*

(p=0.0195)N = 27

0.4403*(p=0.0156)

N = 24

0.388*(p=0.0304)

N = 24

IndustrialCorporation

rugged0.412

(p=0.0918)N = 12

0.689*(p=0.0200)

N = 9

0.527(p=0.0726)

N = 9

(*p < 0.05, **p < 0.01)

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 23 / 27

Summary

Photos consumers share on social media contain valuable brandinformation

Extracting this information requires new tools

Develop methodology for extracting brand perceptions from images,and demonstrate that some brand perceptual attributes can berepresented with basic elements of visual design

Demonstrate that for some perceptual attributes, photos consumerspost online represent their perception of the brand

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 24 / 27

References I

Arnheim, R. (1954). Art and visual perception: A psychology of thecreative eye. Univ of California Press.

Dhar, S., Ordonez, V., & Berg, T. L. (2011). High level describableattributes for predicting aesthetics and interestingness. In Computervision and pattern recognition (cvpr), 2011 ieee conference on (pp.1657–1664).

Dondis, D. A. (1974). A primer of visual literacy.Gujral, R. (2015). Industry report: In visual marketing, it’s scale to win.

Retrieved 2016-09-12, fromhttp://digiday.com/sponsored/chutesoti-117240/

Hashimoto, A., & Clayton, M. (2009). Visual design fundamentals: adigital approach. Nelson Education.

Kane, G. C., & Pear, A. (2016, January). The rise of visual contentonline. Sloan Management Review.

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 25 / 27

References IIKarayev, S., Trentacoste, M., Han, H., Agarwala, A., Darrell, T.,

Hertzmann, A., & Winnemoeller, H. (2013). Recognizing imagestyle. arXiv preprint arXiv:1311.3715.

Lovett, M., Peres, R., & Shachar, R. (2014). A data set of brands andtheir characteristics. Marketing Science, 33(4), 609–617.

McAuley, J., & Leskovec, J. (2012). Image labeling on a network: usingsocial-network metadata for image classification. In Europeanconference on computer vision (pp. 828–841).

Shapiro, L., & Stockman, G. C. (2001). Computer vision. 2001. ed:Prentice Hall.

Wedel, M., & Pieters, R. (Eds.). (2007). Visual marketing. New York:Lawrence Erlbaum Associates.

Zhang, H., Korayem, M., Crandall, D. J., & LeBuhn, G. (2012). Miningphoto-sharing websites to study ecological phenomena. InProceedings of the 21st international conference on world wide web(pp. 749–758).

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 26 / 27

Thank you

Liu Liu (lliu@stern.nyu.edu)Daria Dzyabura (ddzyabur@stern.nyu.edu)

Natalie Mizik (nmizik@uw.edu)

Liu Liu, Daria Dzyabura, Natalie Mizik Brand Perception 2016 Stanford 27 / 27