Panos Ipeirotis New York University Opinion Mining using Econometrics A Case Study on Reputation...
-
Upload
ernest-casey -
Category
Documents
-
view
226 -
download
1
Transcript of Panos Ipeirotis New York University Opinion Mining using Econometrics A Case Study on Reputation...
Panos IpeirotisPanos Ipeirotis
New York UniversityNew York University
Opinion Mining using Econometrics Opinion Mining using Econometrics A Case Study on Reputation SystemsA Case Study on Reputation Systems
Joint work with Anindya Ghose Joint work with Anindya Ghose and Arun Sundararajanand Arun Sundararajan
Comparative Shopping in e-MarketplacesComparative Shopping in e-Marketplaces
Are Customers Irrational?Are Customers Irrational?
Are Customers Irrational?Are Customers Irrational?
$11.04
$18.28
-$0.61
-$9.00
-$11.40
-$1.04
Price Premiums
Price premiums @ Amazon Price premiums @ Amazon
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
-100 -75 -50 -25 0 25 50 75 100
Price Premium
Nu
mb
er
of
Tra
ns
ac
tio
ns
Irra
tiona
l (?)
Average price premiums @ AmazonAverage price premiums @ Amazon
0
200
400
600
800
1000
1200
-100 -75 -50 -25 0 25 50 75 100
Average Price Premium
Nu
mb
er
of
Tra
ns
ac
tio
ns
Irra
tiona
l (?)
Why not buying the cheapest?Why not buying the cheapest?
You buy more than a product
Customers do not pay only for the product
Customers also pay for a set of fulfillment characteristics
Delivery
Packaging
Responsiveness
…
Reputation Matters!
Reputation SystemsReputation Systems
Facilitate electronic commerce
Integral part of online marketplaces
Provide information about unobserved fulfillment characteristics (most of which we take for granted in traditional commerce)
Reputation in ecommerce is complex
Different buyers value different fulfillment characteristics
Sellers have varying abilities on these characteristics
Example of a reputation profileExample of a reputation profile
Reputation profiles: ObservationsReputation profiles: Observations
Reputation profile capture more than “averages”
Well beyond “average score” and “lifetime”
Rich textual content: information about a seller on a variety of dimensions (or fulfillment characteristics).
How the seller’s performance (potentially on each of these characteristics) has evolved over time
Buyer-seller networks
Reputation in ecommerce is complex
Different buyers value different fulfillment characteristics
Sellers have varying abilities on these characteristics
Previous work studied only effect of “average score” and “lifetime”
Our research agendaOur research agenda
What are the dimensions of online reputation?
What characteristics comprise the important parts of a seller’s overall reputation? (politeness? packaging? delivery?)
How do these dimensions affect pricing power?
Does a better reputation enable a seller to charge a higher price?
Which dimensions affect this pricing power most significantly?
Average numerical ratings?
Number of prior successful transactions?
Assessments of ability on specific fulfillment characteristics?
Do competitors with better reputations limit a seller’s pricing power?
Can prior reputation predict marketplace outcomes?
Given a set of sellers, their reputations, and their prices, can one predict which seller will successfully make the sale?
DataData
Overview
Panel of 280 software products sold by Amazon.com
Data on all “secondary” market transactions
Amazon Web services facilitate capturing transactions
Complete reputation profile for all sellers who completed one or more transactions during this period
Summary
280 products X 180 days
1,078 sellers, of which 122 transacted
12,232 transactions
107,922 “observations” (seller-competitor pairs)
Data: TransactionsData: Transactions
Sales of (mostly new) software
Data: TransactionsData: Transactions
Capturing transactions and “price premiums”
Data: TransactionsData: Transactions
Seller ListingItem Price
When item is sold, listing disappears
Capturing transactions and “price premiums”
Data: TransactionsData: Transactions
While listing appears, item is still available
time
1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10
Capturing transactions and “price premiums”
Data: TransactionsData: Transactions
While listing appears, item is still available
time
1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10
Item still not sold on 1/7
Capturing transactions and “price premiums”
Data: TransactionsData: Transactions
When item is sold, listing disappears
time
1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10
Item sold on 1/9
Data: Variables of InterestData: Variables of Interest
Regular Price Premium
Difference in the price charged by a seller and the listed price of a competing seller at the time the transaction occurred
(Seller Price – Competitor Price)
Calculated for each seller-competitor pair, for each transaction
Each transaction therefore generates N observations, where N is the number of competing sellers
Average Price Premium
Difference in the price charged by a seller and the average price of all competing sellers at the time the transaction occurred
(Seller Price – Avg. (Competitor Price) )
Calculated for each transaction
Each transaction generates 1 observation
Price premiums @ Amazon Price premiums @ Amazon
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
-100 -75 -50 -25 0 25 50 75 100
Price Premium
Nu
mb
er
of
Tra
ns
ac
tio
ns
Average price premiums @ AmazonAverage price premiums @ Amazon
0
200
400
600
800
1000
1200
-100 -75 -50 -25 0 25 50 75 100
Average Price Premium
Nu
mb
er
of
Tra
ns
ac
tio
ns
The dimensions of reputationThe dimensions of reputation
How reputation affects price premiums?
Decomposing reputationDecomposing reputation
Is reputation just a scalar metric?
Previous studies assumed a “monolithic” reputation.
We break down reputation in individual components
Sellers characterized by a set of n fulfillment characteristics
What are these characteristics (valued by consumers?)
We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”)
We scan the textual feedback to discover these dimensions
seller lifeseller ranking
Data: Reputation ProfilesData: Reputation Profiles
Decomposing and scoring reputationDecomposing and scoring reputation
Decomposing and scoring reputation
We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”)
The sellers are rated on these dimensions by buyers using modifiers (adjectives or adverbs), not numerical scores
“Fast shipping!”
“Great packaging”
“Awesome unresponsiveness”
“Unbelievable delays”
“Unbelievable price”
Dimensions from text: ExampleDimensions from text: Example
ij
Parsing the feedback
P1: I was impressed by the speedy delivery! Great Service!
P2: The item arrived in awful packaging, and the delivery was slow
…Identified modifier-dimension pairs
P1: “speedy – delivery”, “great – service”
P2: “awful – packaging”, “slow – delivery”
…Reducing textual feedback to a n X p matrix
Dimensions: 1-delivery, 2-packaging, 3-service
ij
11 12 13" ", " ", " "speedy NULL great
21 22 23" ", " ", " "slow awful NULL Postings
Decomposing and scoring reputationDecomposing and scoring reputation
Scoring reputation
“Fast shipping!”
“Great packaging”
“Awesome unresponsiveness”
“Unbelievable delays”
“Unbelievable price”
How can we find out the meaning of these adjectives?
The dimensions of reputationThe dimensions of reputation
We assume that each modifier assigns a “score” to each dimension
:score associated with appearing as the modifier for the k-th dimension
ri: weight of posting that appears on the i-th position (weight down old posts)
wi: weight assigned to the i-th dimension
Thus, the overall (text) reputation score Π(i) is:
11 1 1
1 2
1
( ,1) ... ( , )
( ) , ,...,
( ,1) ... ( , )
i in
pi ip pn n
a a n w
i r r r
a a n w
( , )a k
scores forfirst dimension
scores forn-th dimension
,
( ) ( ( , )) ( , )i j ji j
i w a i R i Sum of ri weights in which
j modifies dimension i
estimated coefficients
scores forfirst posting
The dimensions of reputationThe dimensions of reputation
Scoring the dimensions
Use price premiums as “true” reputation score
Use regression to assess scores (coefficients) for each dimension-modifier pair
Regressions
Control for all variables that affect price premiums
Control for all numeric scores of reputation
Examine effect of text: E.g., seller with “fast delivery” has premium $10 over seller with “slow delivery”
,
( ) ( ( , )) ( , )i j ji j
i w a i R i estimated coefficients
Some indicative dollar valuesSome indicative dollar values
Positive
Negative
Natural method for extraction of sentiment strength and polarity
ResultsResults
Some dimensions that matter
Delivery and contract fulfillment (extent and speed)
Product quality and appropriate description
Packaging
Customer service
Price (!)
Responsiveness/Communication (speed and quality)
Overall feeling (transaction)
ResultsResults
Further evidence
Classifier (aka choice model) that predicts sale given set of sellers
Binary decision between seller and competitor
Naïve Bayes and Decision Trees (SVM’s forthcoming)
Only prices and characteristics: 53%
+ numerical reputation, lifetime: 74%
+ encoded textual information: 89%
Other applicationsOther applications
Summarize and query reputation data
Give me all merchants that deliver fast
SELECT merchant FROM reputation
WHERE delivery > ‘fast’
Summarize reputation of seller XYZ Inc.
Delivery: 3.8/5
Responsiveness: 4.8/5
Packaging: 4.9/5
Pricing reputation
Given the competition, merchant XYZ can charge $20 more and still make the sale (confidence: 83%)
SummarySummary
Key contributions
New technique that automatically scores “sentiment” based on economic data
Validation by multiple methods (estimating an econometric model, building classifiers)
New evidence of the extent to which interdisciplinary research can be fun and distracting
Broader contribution
Economic data is abundant and there is rich literature on how to handle such data
Economic data can be used for training for MANY applications
Moving aheadMoving ahead
Extensions of current work
Dimensionality reduction, grouping dimensions topics that might correspond more closely to the “true” dimensions of reputation
Latent Dirichlet Allocation, (probabilistic) Latent Semantic Analysis, Non-negative Matrix Factorization, Tensors
Identifying weights for dimensions, using normalized scores
“Correct” game theoretic model of market competition
Exploiting network structure
Exploring connection with the “trustrank” literature
Network position as an additional dimension of seller reputation
Buyers as seller/category specific “authorities”
Thank you!Thank you!
http://economining.stern.nyu.edu
Prior studies of reputation Prior studies of reputation
Positive feedback significant, negative not
Ba and Pavlou (2002) for CD’s, software, electronics; Bajari and Hortacsu (2003) for collectible coins?
Negative feedback significant, positive not
Lee et al. (2000) for computer equipment, Reiley et al. (2000) for collectible coins
Nature of price: winning online auction bid (usually eBay)Measure of reputation: average numerical score, # of transactions
Both positive and negative feedback significant
Dewan and Hsu (2004) for rare stamps, Melnik and Alm (2002) for gold coins, Houser and Wooders (2005) for Pentium chips