Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People ›...
Transcript of Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People ›...
![Page 1: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/1.jpg)
Generalizing Dependency Features for Opinion Mining
Mahesh Joshi1 and Carolyn Rosé1,2
1 Language Technologies Institute2 Human-Computer Interaction Institute
Carnegie Mellon University
ACL-IJCNLP 2009 Short Papers Track
![Page 2: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/2.jpg)
One Slide Summary• Goal: utilize structure in language to improve “opinion
vs. not-opinion” classification in product reviews
• Key idea: “Back off” the head word in a dependency relation to it’s part-of-speech tag, use these “back-off” features– amod(camera,great) => amod(NN,great)
• Result: It works!– Improvement in accuracy from 0.652 to 0.679
• yes, it’s significant!
8/4/2009 Generalizing Dependency Features for Opinion Mining 2
![Page 3: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/3.jpg)
One Slide Summary• Goal: utilize structure in language to improve “opinion
vs. not-opinion” classification in product reviews
• Key idea: “Back off” the head word in a dependency relation to it’s part-of-speech tag, use these “back-off” features– amod(camera,great) => amod(NN,great)
• Result: It works!– Improvement in accuracy from 0.652 to 0.679
• yes, it’s significant!
8/4/2009 Generalizing Dependency Features for Opinion Mining 2
![Page 4: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/4.jpg)
Task Description• Opinion mining in product reviews
• Given a sentence from a product review– Predict whether or not it is an opinion sentence
• Opinion Sentence: “If a sentence contains one or more product features and one or more opinion words, then the sentence is called an opinion sentence.” [Hu and Liu; SIGKDD 2004]
8/4/2009 Generalizing Dependency Features for Opinion Mining 3
![Page 5: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/5.jpg)
Examples• Opinion sentences
– It is very light weight and has good signal strength.– Player works and looks great – if you can get the dvd’s to
play.
• Non-opinion sentences– Had it for a week.– If this doesnt bring back the picture, try pressing this
button without playing a dvd.
8/4/2009 Generalizing Dependency Features for Opinion Mining 4
![Page 6: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/6.jpg)
Research Question
How can we better utilize structure in language for opinion classification?
8/4/2009 Generalizing Dependency Features for Opinion Mining 5
![Page 7: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/7.jpg)
Dependency Relations
8/4/2009 Generalizing Dependency Features for Opinion Mining 6
![Page 8: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/8.jpg)
Dependency Relations
8/4/2009 Generalizing Dependency Features for Opinion Mining 6
it is a fantastic camera and well worth the price.
![Page 9: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/9.jpg)
Dependency Relationsnsubj(camera-5,it-1)cop(camera-5,is-2)det(camera-5,a-3)amod(camera-5,fantastic-4)advmod(worth-8,well-7)det(price-10,the-9)amod(price-10,worth-8)conj_and(camera-5,price-10)
8/4/2009 Generalizing Dependency Features for Opinion Mining 6
it is a fantastic camera and well worth the price.
(Stanford parser output) ==>
![Page 10: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/10.jpg)
Dependency Relationsnsubj(camera-5,it-1)cop(camera-5,is-2)det(camera-5,a-3)amod(camera-5,fantastic-4)advmod(worth-8,well-7)det(price-10,the-9)amod(price-10,worth-8)conj_and(camera-5,price-10)
8/4/2009 Generalizing Dependency Features for Opinion Mining 6
it is a fantastic camera and well worth the price.
(Stanford parser output) ==>Relation
![Page 11: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/11.jpg)
Dependency Relationsnsubj(camera-5,it-1)cop(camera-5,is-2)det(camera-5,a-3)amod(camera-5,fantastic-4)advmod(worth-8,well-7)det(price-10,the-9)amod(price-10,worth-8)conj_and(camera-5,price-10)
8/4/2009 Generalizing Dependency Features for Opinion Mining 6
it is a fantastic camera and well worth the price.
(Stanford parser output) ==>
![Page 12: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/12.jpg)
Dependency Relationsnsubj(camera-5,it-1)cop(camera-5,is-2)det(camera-5,a-3)amod(camera-5,fantastic-4)advmod(worth-8,well-7)det(price-10,the-9)amod(price-10,worth-8)conj_and(camera-5,price-10)
8/4/2009 Generalizing Dependency Features for Opinion Mining 6
it is a fantastic camera and well worth the price.
(Stanford parser output) ==>
Head Word
![Page 13: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/13.jpg)
Dependency Relationsnsubj(camera-5,it-1)cop(camera-5,is-2)det(camera-5,a-3)amod(camera-5,fantastic-4)advmod(worth-8,well-7)det(price-10,the-9)amod(price-10,worth-8)conj_and(camera-5,price-10)
8/4/2009 Generalizing Dependency Features for Opinion Mining 6
it is a fantastic camera and well worth the price.
(Stanford parser output) ==>
![Page 14: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/14.jpg)
Dependency Relationsnsubj(camera-5,it-1)cop(camera-5,is-2)det(camera-5,a-3)amod(camera-5,fantastic-4)advmod(worth-8,well-7)det(price-10,the-9)amod(price-10,worth-8)conj_and(camera-5,price-10)
8/4/2009 Generalizing Dependency Features for Opinion Mining 6
it is a fantastic camera and well worth the price.
(Stanford parser output) ==>
Modifier Word
![Page 15: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/15.jpg)
Dependency Relations
8/4/2009 Generalizing Dependency Features for Opinion Mining 7
![Page 16: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/16.jpg)
Dependency Relations
• amod(camera-5,fantastic-4)– adjectival modifier relationship between camera (head,
noun) and fantastic (modifier, adjective)
8/4/2009 Generalizing Dependency Features for Opinion Mining 7
![Page 17: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/17.jpg)
Dependency Relations
• amod(camera-5,fantastic-4)– adjectival modifier relationship between camera (head,
noun) and fantastic (modifier, adjective)
• advmod(worth-8,well-7)– adverbial modifier relationship between worth (head,
adjective) and well (modifier, adverb)
8/4/2009 Generalizing Dependency Features for Opinion Mining 7
![Page 18: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/18.jpg)
Previous Work
8/4/2009 Generalizing Dependency Features for Opinion Mining 8
![Page 19: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/19.jpg)
Previous Work• Success in using dependency relations:
– [Gamon; COLING 2004], [Matsumoto et al.; PAKDD 2005]• Different task: predicting customer satisfaction ratings, and polarity
(positive / negative) for movie reviews respectively– [Wilson et al.; AAAI 2004]
• Different task: predicting strength of subjective language– Use full set of dependency relations
8/4/2009 Generalizing Dependency Features for Opinion Mining 8
![Page 20: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/20.jpg)
Previous Work• Success in using dependency relations:
– [Gamon; COLING 2004], [Matsumoto et al.; PAKDD 2005]• Different task: predicting customer satisfaction ratings, and polarity
(positive / negative) for movie reviews respectively– [Wilson et al.; AAAI 2004]
• Different task: predicting strength of subjective language– Use full set of dependency relations
• We propose multiple generalization approaches– one of our approaches was used by Gamon as well as
Wilson et al.
8/4/2009 Generalizing Dependency Features for Opinion Mining 8
![Page 21: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/21.jpg)
Previous Work
8/4/2009 Generalizing Dependency Features for Opinion Mining 9
![Page 22: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/22.jpg)
Previous Work• Dependency relations not found to be useful:
– [Dave et al.; WWW 2003], [Ng et al.; ACL 2006]• Different task: polarity prediction in product reviews and movie
reviews respectively– Both use a subset of dependency relations
• manually chosen grammatical relations
8/4/2009 Generalizing Dependency Features for Opinion Mining 9
![Page 23: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/23.jpg)
Previous Work• Dependency relations not found to be useful:
– [Dave et al.; WWW 2003], [Ng et al.; ACL 2006]• Different task: polarity prediction in product reviews and movie
reviews respectively– Both use a subset of dependency relations
• manually chosen grammatical relations
• We use the full set of dependency relations
8/4/2009 Generalizing Dependency Features for Opinion Mining 9
![Page 24: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/24.jpg)
Recent Work - not in paper
8/4/2009 Generalizing Dependency Features for Opinion Mining 10
![Page 25: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/25.jpg)
Recent Work - not in paper• Transformation of dependency relations
– [Greene and Resnik; NAACL 2009]– Given dependency relations of form: relation(head_word,modifier_word)
• Create features of form: head_word-relation & relation-modifier_word
8/4/2009 Generalizing Dependency Features for Opinion Mining 10
![Page 26: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/26.jpg)
Recent Work - not in paper• Transformation of dependency relations
– [Greene and Resnik; NAACL 2009]– Given dependency relations of form: relation(head_word,modifier_word)
• Create features of form: head_word-relation & relation-modifier_word
8/4/2009 Generalizing Dependency Features for Opinion Mining 10
![Page 27: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/27.jpg)
Recent Work - not in paper• Transformation of dependency relations
– [Greene and Resnik; NAACL 2009]– Given dependency relations of form: relation(head_word,modifier_word)
• Create features of form: head_word-relation & relation-modifier_word
• Dependency relations chosen if they contain “domain relevant” verbs and/or nouns
• Most closely related to our approach
8/4/2009 Generalizing Dependency Features for Opinion Mining 10
![Page 28: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/28.jpg)
Lesson from Past Work
8/4/2009 Generalizing Dependency Features for Opinion Mining 11
![Page 29: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/29.jpg)
Lesson from Past Work• Using full set of dependency relations works better
8/4/2009 Generalizing Dependency Features for Opinion Mining 11
![Page 30: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/30.jpg)
Lesson from Past Work• Using full set of dependency relations works better
• However, there might be overfitting in scarce data scenarios– common to NLP where annotated data is expensive!
• Need features that can generalize well
8/4/2009 Generalizing Dependency Features for Opinion Mining 11
![Page 31: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/31.jpg)
Motivating Our Approach• Consider two opinion sentences:
This is a great camera!
Despite its few negligible flaws, this great mp3 player won my vote.
8/4/2009 Generalizing Dependency Features for Opinion Mining 12
![Page 32: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/32.jpg)
Motivating Our Approach
• Both have the dependency relation amod with different pair of words participating:
amod(camera,great)amod(player,great)
• We can see the structural similarity of these features
• A machine learning algorithm can’t!
8/4/2009 Generalizing Dependency Features for Opinion Mining 13
![Page 33: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/33.jpg)
Motivating Our Approach
• Both have the dependency relation amod with different pair of words participating:
amod(camera,great)amod(player,great)
• We can see the structural similarity of these features
• A machine learning algorithm can’t!
8/4/2009 Generalizing Dependency Features for Opinion Mining 13
![Page 34: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/34.jpg)
So Lets “Back Off!”• “Back off” the head word to its part-of-speech tag
amod(camera,great) => amod(NN,great)amod(player,great) => amod(NN,great)
• Now the algorithm can see that these are similar features (we made them identical)
8/4/2009 Generalizing Dependency Features for Opinion Mining 14
![Page 35: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/35.jpg)
Advantages of Backing Off
8/4/2009 Generalizing Dependency Features for Opinion Mining 15
![Page 36: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/36.jpg)
Advantages of Backing Off• Stronger evidence of association of a generalized
feature with the opinion category
8/4/2009 Generalizing Dependency Features for Opinion Mining 15
![Page 37: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/37.jpg)
Advantages of Backing Off• Stronger evidence of association of a generalized
feature with the opinion category
• New test sentence: “this is a great phone”– amod(phone,great): may not be useful because we might
have never seen it!– amod(NN,great): still valid
8/4/2009 Generalizing Dependency Features for Opinion Mining 15
![Page 38: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/38.jpg)
Backing Off / Generalizing
8/4/2009 Generalizing Dependency Features for Opinion Mining 16
![Page 39: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/39.jpg)
Backing Off / Generalizing• Composite Back-off Features
– Head word => its part-of-speech tag• amod(camera,great) => amod(NN,great)
8/4/2009 Generalizing Dependency Features for Opinion Mining 16
![Page 40: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/40.jpg)
Backing Off / Generalizing• Composite Back-off Features
– Head word => its part-of-speech tag• amod(camera,great) => amod(NN,great)
8/4/2009 Generalizing Dependency Features for Opinion Mining 16
![Page 41: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/41.jpg)
Backing Off / Generalizing• Composite Back-off Features
– Head word => its part-of-speech tag• amod(camera,great) => amod(NN,great)
– Modifier word => its part-of-speech tag• amod(camera,great) => amod(camera,JJ)
8/4/2009 Generalizing Dependency Features for Opinion Mining 16
![Page 42: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/42.jpg)
Backing Off / Generalizing• Composite Back-off Features
– Head word => its part-of-speech tag• amod(camera,great) => amod(NN,great)
– Modifier word => its part-of-speech tag• amod(camera,great) => amod(camera,JJ)
8/4/2009 Generalizing Dependency Features for Opinion Mining 16
![Page 43: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/43.jpg)
Backing Off / Generalizing• Composite Back-off Features
– Head word => its part-of-speech tag• amod(camera,great) => amod(NN,great)
– Modifier word => its part-of-speech tag• amod(camera,great) => amod(camera,JJ)
• Full Back-off Features– Head word => its part-of-speech tag; Also modifier word
=> its part-of-speech tag• amod(camera,great) => amod(NN,JJ)• used by [Gamon; COLING 2004] and [Wilson et al.; AAAI 2004]
8/4/2009 Generalizing Dependency Features for Opinion Mining 16
![Page 44: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/44.jpg)
Backing Off / Generalizing• Composite Back-off Features
– Head word => its part-of-speech tag• amod(camera,great) => amod(NN,great)
– Modifier word => its part-of-speech tag• amod(camera,great) => amod(camera,JJ)
• Full Back-off Features– Head word => its part-of-speech tag; Also modifier word
=> its part-of-speech tag• amod(camera,great) => amod(NN,JJ)• used by [Gamon; COLING 2004] and [Wilson et al.; AAAI 2004]
8/4/2009 Generalizing Dependency Features for Opinion Mining 16
![Page 45: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/45.jpg)
Backing Off / Generalizing• Composite Back-off Features
– Head word => its part-of-speech tag• amod(camera,great) => amod(NN,great)
– Modifier word => its part-of-speech tag• amod(camera,great) => amod(camera,JJ)
• Full Back-off Features– Head word => its part-of-speech tag; Also modifier word
=> its part-of-speech tag• amod(camera,great) => amod(NN,JJ)• used by [Gamon; COLING 2004] and [Wilson et al.; AAAI 2004]
8/4/2009 Generalizing Dependency Features for Opinion Mining 16
![Page 46: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/46.jpg)
Experiments
8/4/2009 Generalizing Dependency Features for Opinion Mining 17
![Page 47: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/47.jpg)
Experiments• 2,200 sentences (Randomly sampled from
Amazon.com and Cnet.com reviews for 11 products, 200 per product)– Subset of dataset released by [Hu and Liu; SIGKDD 2004]
8/4/2009 Generalizing Dependency Features for Opinion Mining 17
![Page 48: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/48.jpg)
Experiments• 2,200 sentences (Randomly sampled from
Amazon.com and Cnet.com reviews for 11 products, 200 per product)– Subset of dataset released by [Hu and Liu; SIGKDD 2004]
• Support Vector Machine classifier, linear kernel
8/4/2009 Generalizing Dependency Features for Opinion Mining 17
![Page 49: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/49.jpg)
Experiments• 2,200 sentences (Randomly sampled from
Amazon.com and Cnet.com reviews for 11 products, 200 per product)– Subset of dataset released by [Hu and Liu; SIGKDD 2004]
• Support Vector Machine classifier, linear kernel
• Chi-squared feature selection– used significant features at α = 0.05
8/4/2009 Generalizing Dependency Features for Opinion Mining 17
![Page 50: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/50.jpg)
Evaluation• 11-fold cross-validation, sentences for each product
used in test fold once
• Reporting average accuracy, Cohen’s kappa
8/4/2009 Generalizing Dependency Features for Opinion Mining 18
![Page 51: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/51.jpg)
Baselines
8/4/2009 Generalizing Dependency Features for Opinion Mining 19
![Page 52: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/52.jpg)
Baselines• Several standard feature sets
– ngrams (n = 1, 2, 3)– Part-of-Speech ngrams (n = 2, 3)– Dependency relations (no back-off)
8/4/2009 Generalizing Dependency Features for Opinion Mining 19
![Page 53: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/53.jpg)
Baselines• Several standard feature sets
– ngrams (n = 1, 2, 3)– Part-of-Speech ngrams (n = 2, 3)– Dependency relations (no back-off)
• Back-off ngrams (n = 2, 3) similar to [McDonald et al.; ACL 2007]– back off words in an ngram to POS tags– create features using all possible (2n-1) back-off
combinations
8/4/2009 Generalizing Dependency Features for Opinion Mining 19
![Page 54: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/54.jpg)
Feature Set Accuracy KappaUnigrams 0.652 (±0.048) 0.295 (±0.049)Uni.+Bigrams 0.657 (±0.066) 0.304 (±0.089)Uni.+Trigrams 0.655 (±0.062) 0.306 (±0.077)Uni.+Back-off Bigrams 0.650 (±0.056) 0.299 (±0.079)Uni.+Back-off Trigrams 0.647 (±0.051) 0.287 (±0.075)Uni.+POS Bigrams 0.676 (±0.057) 0.349 (±0.083)Uni.+POS Trigrams 0.661 (±0.050) 0.317 (±0.064)Uni.+Dep. Lex 0.639 (±0.055) 0.268 (±0.079)Uni.+Dep. Head-Back-off 0.679 (±0.063) 0.351 (±0.097)Uni.+Dep. Mod-Back-off 0.657 (±0.056) 0.308 (±0.063)Uni.+Dep. Head-Mod-Back-off 0.670 (±0.046) 0.336 (±0.065)
Results
8/4/2009 Generalizing Dependency Features for Opinion Mining 20
![Page 55: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/55.jpg)
Feature Set Accuracy KappaUnigrams 0.652 (±0.048) 0.295 (±0.049)Uni.+Bigrams 0.657 (±0.066) 0.304 (±0.089)Uni.+Trigrams 0.655 (±0.062) 0.306 (±0.077)Uni.+Back-off Bigrams 0.650 (±0.056) 0.299 (±0.079)Uni.+Back-off Trigrams 0.647 (±0.051) 0.287 (±0.075)Uni.+POS Bigrams 0.676 (±0.057) 0.349 (±0.083)Uni.+POS Trigrams 0.661 (±0.050) 0.317 (±0.064)Uni.+Dep. Lex 0.639 (±0.055) 0.268 (±0.079)Uni.+Dep. Head-Back-off 0.679 (±0.063) 0.351 (±0.097)Uni.+Dep. Mod-Back-off 0.657 (±0.056) 0.308 (±0.063)Uni.+Dep. Head-Mod-Back-off 0.670 (±0.046) 0.336 (±0.065)
Results
8/4/2009 Generalizing Dependency Features for Opinion Mining 20
![Page 56: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/56.jpg)
Feature Set Accuracy KappaUnigrams 0.652 (±0.048) 0.295 (±0.049)Uni.+Bigrams 0.657 (±0.066) 0.304 (±0.089)Uni.+Trigrams 0.655 (±0.062) 0.306 (±0.077)Uni.+Back-off Bigrams 0.650 (±0.056) 0.299 (±0.079)Uni.+Back-off Trigrams 0.647 (±0.051) 0.287 (±0.075)Uni.+POS Bigrams 0.676 (±0.057) 0.349 (±0.083)Uni.+POS Trigrams 0.661 (±0.050) 0.317 (±0.064)Uni.+Dep. Lex 0.639 (±0.055) 0.268 (±0.079)Uni.+Dep. Head-Back-off 0.679 (±0.063) 0.351 (±0.097)Uni.+Dep. Mod-Back-off 0.657 (±0.056) 0.308 (±0.063)Uni.+Dep. Head-Mod-Back-off 0.670 (±0.046) 0.336 (±0.065)
Results
8/4/2009 Generalizing Dependency Features for Opinion Mining 20
![Page 57: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/57.jpg)
Feature Set Accuracy KappaUnigrams 0.652 (±0.048) 0.295 (±0.049)Uni.+Bigrams 0.657 (±0.066) 0.304 (±0.089)Uni.+Trigrams 0.655 (±0.062) 0.306 (±0.077)Uni.+Back-off Bigrams 0.650 (±0.056) 0.299 (±0.079)Uni.+Back-off Trigrams 0.647 (±0.051) 0.287 (±0.075)Uni.+POS Bigrams 0.676 (±0.057) 0.349 (±0.083)Uni.+POS Trigrams 0.661 (±0.050) 0.317 (±0.064)Uni.+Dep. Lex 0.639 (±0.055) 0.268 (±0.079)Uni.+Dep. Head-Back-off 0.679 (±0.063) 0.351 (±0.097)Uni.+Dep. Mod-Back-off 0.657 (±0.056) 0.308 (±0.063)Uni.+Dep. Head-Mod-Back-off 0.670 (±0.046) 0.336 (±0.065)
Results
8/4/2009 Generalizing Dependency Features for Opinion Mining 20
![Page 58: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/58.jpg)
Feature Set Accuracy KappaUnigrams 0.652 (±0.048) 0.295 (±0.049)Uni.+Bigrams 0.657 (±0.066) 0.304 (±0.089)Uni.+Trigrams 0.655 (±0.062) 0.306 (±0.077)Uni.+Back-off Bigrams 0.650 (±0.056) 0.299 (±0.079)Uni.+Back-off Trigrams 0.647 (±0.051) 0.287 (±0.075)Uni.+POS Bigrams 0.676 (±0.057) 0.349 (±0.083)Uni.+POS Trigrams 0.661 (±0.050) 0.317 (±0.064)Uni.+Dep. Lex 0.639 (±0.055) 0.268 (±0.079)Uni.+Dep. Head-Back-off 0.679 (±0.063) 0.351 (±0.097)Uni.+Dep. Mod-Back-off 0.657 (±0.056) 0.308 (±0.063)Uni.+Dep. Head-Mod-Back-off 0.670 (±0.046) 0.336 (±0.065)
Results
8/4/2009 Generalizing Dependency Features for Opinion Mining 20
![Page 59: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/59.jpg)
Feature Set Accuracy KappaUnigrams 0.652 (±0.048) 0.295 (±0.049)Uni.+Bigrams 0.657 (±0.066) 0.304 (±0.089)Uni.+Trigrams 0.655 (±0.062) 0.306 (±0.077)Uni.+Back-off Bigrams 0.650 (±0.056) 0.299 (±0.079)Uni.+Back-off Trigrams 0.647 (±0.051) 0.287 (±0.075)Uni.+POS Bigrams 0.676 (±0.057) 0.349 (±0.083)Uni.+POS Trigrams 0.661 (±0.050) 0.317 (±0.064)Uni.+Dep. Lex 0.639 (±0.055) 0.268 (±0.079)Uni.+Dep. Head-Back-off 0.679 (±0.063) 0.351 (±0.097)Uni.+Dep. Mod-Back-off 0.657 (±0.056) 0.308 (±0.063)Uni.+Dep. Head-Mod-Back-off 0.670 (±0.046) 0.336 (±0.065)
Results
8/4/2009 Generalizing Dependency Features for Opinion Mining 20
![Page 60: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/60.jpg)
Discussion• “Head-Back-off” features
– significantly better than unigrams-only baseline– represent a better way to use dependency relations
• Generalizing them further (full back-off features) worsens performance
8/4/2009 Generalizing Dependency Features for Opinion Mining 21
![Page 61: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/61.jpg)
Head-Back-off Successes• It is perhaps too small.
– cop(JJ,is)
• The lens retracts and has its own metal cover so you don't need to fuss with a lens cap.– det(NN,the), poss(NN,its), neg(VB,n’t)
• The panorama setting is unbelievable!– cop(JJ,is), det(NN,the)
8/4/2009 Generalizing Dependency Features for Opinion Mining 22
![Page 62: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/62.jpg)
Head-Back-off Successes• Even with the waterproof housing it is small.
– cop(JJ,is), nsubj(JJ,it), det(NN,the)
• The auto color balance is often fooled by the clouds.– det(NN,the)
• First, I have to say that I have NEVER had the slightest problem with this camera or the software.– det(NN,this)
8/4/2009 Generalizing Dependency Features for Opinion Mining 23
![Page 63: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/63.jpg)
Error Analysis• Sentences that are opinions, but not about the
product’s features– Really like them, they work well and the macro function of
the 2500 really helps my Ebay biz.• “Really like them” is about two other cameras the reviewer owns,
not the camera being reviewed.
– I have had this little gem for four months now.• About the product as a whole (not any particular explicitly
mentioned feature)
8/4/2009 Generalizing Dependency Features for Opinion Mining 24
![Page 64: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/64.jpg)
Error Analysis• Few misclassifications due to Head-Back-off
features “misfiring”– Some people, in their reviews, complain about its small
size, and how it doesn’t compare with larger cameras.• Misclassified as “opinion sentence”• poss(NN,its), neg(VB,n’t)
– The closest competitor is the SONY DSC-P1 (3.3mp).• Misclassified as “opinion sentence”• cop(NN,is), det(NN,the)
8/4/2009 Generalizing Dependency Features for Opinion Mining 25
![Page 65: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/65.jpg)
Conclusions• “Head-Back-off” features are a sweet spot
– not too specific, not too general
• Future work:– Explore relation to supervised domain adaptation
8/4/2009 Generalizing Dependency Features for Opinion Mining 26
![Page 66: Generalizing Dependency Features for Opinion Mining › afs › cs.cmu.edu › Web › People › maheshj › pubs › ...• Opinion mining in product reviews • Given a sentence](https://reader033.fdocuments.net/reader033/viewer/2022060504/5f1d466e9acea934930bd3b0/html5/thumbnails/66.jpg)
Questions?
8/4/2009 Generalizing Dependency Features for Opinion Mining 27