AINL 2016: Kravchenko
-
Upload
lidia-pivovarova -
Category
Science
-
view
172 -
download
4
Transcript of AINL 2016: Kravchenko
![Page 1: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/1.jpg)
Solution for workshop
AINL FRUCT: Artificial Intelligence and Natural
Language Conference
10-12 NOVEMBER 2016SAINT-PETERSBURG
http://ainlconf.ru/
![Page 2: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/2.jpg)
Paraphrase Detection using Semantic Similarity Algorithms
Dmitry Kravchenko
Ben-Gurion University of the Negev
![Page 3: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/3.jpg)
Tasks description
Input:
2 files with list of pairs of sentences in Russian in XML format:
a) training set
b) test set
Output:
Task 1:
Algorithm should classify each pair into one of three classes: Non-paraphrase, Near-paraphrase, Precise-paraphrase
Task 2:
Algorithm should classify each pair into one of three classes: Non-paraphrase, Paraphrase
![Page 4: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/4.jpg)
Algorithm Data-Flow
SEMILAR Toolkit
DKPro Similarity
Python difflib
NLTK WordNet
Swoogle
BLEU algorithms
Yandex
MicrosoftGradient BoostingClassifier
Input
substitution of acronymsusing onlinedictionary:
wiktionary.org
Output
![Page 5: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/5.jpg)
Classification algorithm● GradientBooster Classifier
● Task 1:– Feature vector which contain 77 features:
● 18 features: 6 scores of SEMILAR toolkit * 3 translation engines
● 39 features: 13 scores of DKPro Similarity toolkit * 3 translation engines
● 3 features: 1 python difflib similarity score * 3 translation engines
● 6 features: 2 scores of sentence similarity scores (Yuhua Li, David McLean, etc. et al) * 3 translation engines
● 3 features: 1 score of Swoogle comparator * 3 translations● 8 BLEU scores on source sentences (in Russian)
![Page 6: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/6.jpg)
Classification algorithm● Task 2:
– Feature vector which contain 69 features:● 18 features: 6 scores of SEMILAR toolkit * 3 translation
engines● 39 features: 13 scores of DKPro Similarity toolkit * 3
translation engines● 3 features: 1 python difflib similarity score * 3 translation
engines● 6 features: 2 scores of sentence similarity scores (Yuhua Li,
David McLean, etc. et al) * 3 translation engines● 3 features: 1 score of Swoogle comparator * 3 translations
● (without BLEU scores)
![Page 7: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/7.jpg)
6 scores of SEMILAR toolkit
● greedyComparerWNLin● optimumComparerLSATasa ● dependencyComparerWnLeskTanim● cmComparer● bleuComparer● lsaComparer
![Page 8: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/8.jpg)
greedyComparerWNLin
This score refers to a sentence to sentence similarity method which greedily aligns words between given sentences. The word alignment method used is WordNet based method proposed by Lin in 1998: article name is “An information-theoretic definition of similarity”.
Please refer to:
A Comparison of Greedy and Optimal Assessment of Natural Language Student Input Using Word-to-Word Similarity Metrics
http://www.aclweb.org/website/old_anthology/W/W12/W12-20.pdf#page=175
![Page 9: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/9.jpg)
optimumComparerLSATasa
Similar to greedyComparerWNLin, but the words are aligned optimally (similar to job assignment problem) and the word-to-word similarity method
Article name is: Latent Semantic Analysis Models on Wikipedia and TASA
http://deeptutor2.memphis.edu/Semilar-Web/public/downloads/LSA-Models-LREC014/LSAModelsOnWikipediaAndTASADanEtAl-LREC014.pdf
![Page 10: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/10.jpg)
dependencyComparerWnLeskTanim
Please see:● https://www.aaai.org/ocs/index.php/FLAIRS/200
9/paper/viewFile/55/298.
The word-to-word similarity method used.
It is WordNet based method proposed by Lesk and Tanim
![Page 11: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/11.jpg)
cmComparer
Method proposed by Corley and Mihalcea. (article name is: SEMILAR: The Semantic Similarity Toolkit)
![Page 12: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/12.jpg)
lsaComparer
LSA based word representation are summed up for each sentence and the similarity is calculated using the resultant representation.
● (resultant Vector based method is described in the article: NeRoSim: A System for Measuring and Interpreting Semantic Textual Similarity http://alt.qcri.org/semeval2015/cdrom/pdf/SemEval030.pdf)
![Page 13: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/13.jpg)
Word-to-word Similarity score
Article: NeRoSim: A System for Measuring and Interpreting Semantic Textual Similarity
![Page 14: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/14.jpg)
13 scores of DKPro Similarity toolkit
● CosineSimilarity,
● ExactStringMatchComparator,
● GreedyStringTiling2-gram, GreedyStringTiling 4-gram,
● JaroSecondStringComparator,
● JaroWinklerSecondStringComparator,
● normalized LevenshteinComparator,
● LongestCommonSubsequenceNormComparator,
● SubstringMatchComparator,
● WordNGramContainmentMeasure,
● WordNGram-JaccardMeasure 2-gram, WordNGramJaccardMeasure 3-gram, WordNGramJaccardMeasure 4-gram
![Page 15: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/15.jpg)
Four rest Toolkits
● Python difflib comparator ● NLTK WordNet. Sentence similarity scores
(Yuhua Li, David McLean, etc. et al) ● Swoogle comparator ● BLEU scores (for Russian language, no need
for English translation): bleu def 1-gram, bleu def 2-gram, bleu def 3-gram, bleu def 4-gram, bleu lin 1-gram, bleu lin 2-gram, bleu lin 3-gram, bleu lin 4-gram
![Page 16: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/16.jpg)
Results on Test Set
Task number Accuracy F1 macro Place
First Task Standard 0.5695 0.5437 4 out of 11
Second Task Standard 0.7153 0.7853 6 out of 10
![Page 17: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/17.jpg)
Which impact Toolkits gave?
SEMILAR DKPro Similarity Swoogle NLTK WordNet Python difflib66.00
68.00
70.00
72.00
74.00
76.00
78.00
80.00
82.00
80.1379.52
78.94 78.76
75.92
77.02
75.78
75.03 75.02
71.36
Accuracy F1 macro
5-fold cross validation results on the Training Set Second Task
![Page 18: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/18.jpg)
Which Translation Engine is Better?5-fold cross validation results on the Training Set Second Task
Symbols for Toolkit on X axis: 1: SEMILAR 2: DKPro Similarity 3: Python difflib 4: NLTK WordNet 5: Swoogle 6: All 5 Toolkits together
![Page 19: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/19.jpg)
Conclusion
By using this algorithm we can detect semantic similarity not only for Russian language, but for
any other language, which translation is available via translation engines.
![Page 20: AINL 2016: Kravchenko](https://reader031.fdocuments.net/reader031/viewer/2022021919/58749deb1a28abfc5f8b6acd/html5/thumbnails/20.jpg)
Thank you!