Collocation Extraction Using Monolingual Word Alignment

MethodZhanyi Liu, Haifeng Wang, Hua

Wu, Sheng Li

EMNLP 2009

Collocation

• Two words– Consecutive ("by accident")– Interrupted ("take ... advice")

• Other examples– Proper noun ("New York")– Compound nouns ("ice cream")– Correlative conjunction ("either ... or")

Previous Works

• Co-occurring word pairs– Word pairs in a given window size

• Association measures– Frequency, log-likelihood, mutual information ...

• Disadvantage– Long-span collocation

• "either ... or", "because ... so"

• Limited by window size

– False collocation• Any word pairs in window size

Monolingual Word Alignment

• Bilingual word alignment (BWA)– Source-target sentence pairs

• Monolingual Word Alignment (MWA)– Source-source sentence pairs

– Replicate the corpus

Monolingual Word Alignment (2)

Bilingual

Monolingual

A word never collocates with itself

MWA Model

• Sentence with l words S ={w1,...,wl}

• Alignment A = {(i,ai) | i [1,l]}∈

A = {(2,3), (3,2), (4,7), (6,7)...}

MWA Model (2)

• Adapt IBM Model 3 to MWA

• EM training algorithm, produce 3 probability– Word collocation probability

– Position collocation probability• d(4|7,12)• Prob that 4th collocates with 7th word in a 12-word sentence

– Fertility probability

• Prob that wi is collocate with Φi words

Collocation Extraction

• Extract and rank. Filter when freq(wi,wj)<5

• Symmetric assumption– (wi, wj) = (wj, wi)

Initial Experiment

• Chinese

• Training data– LDC2007T03 Tagged Chinese Giga Word

– Xinhua portion, 28M words

• Gold set– Handcrafted collocation dictionaries

– 56888 collocations

Initial Experiment (2)

• Precision

• Baseline– Frequency, log-likelihood, mutual information

– Log-likelihood achieves the best performance

Initial Experiment (3)Observation

Precision is lowSmall gold set (57K/200K = 28%)

Low precision when N < 20K

ObservationFrequency vs. Probability vs. PrecisionPrecision curve

Lower freq --> lower precisionAlignment probability curve

Lower freq --> higher probability

Observation (2)

• Conclusion– What causes lower precision of top 20K?

– Collocation with low freq but high probability

Improved MWA Method

• Add a penalization function y=f(x), x=freq(w1,w2)– When x is small, y approaches 0 (penalize)– When x is large, y approaches 1 (do not penalize)

• y = e-b/x (b is tuned to 25)• New ranking score

Further Evaluation

• Automatic evaluation– Greatly outperforms the best baseline– For top 1K, 20.6% vs. 11.7%– Exponential function plays a key role

Further Evaluation (2)

• Human Evaluation– Top 1K collocations– For each collocation, tag "True" or "False"

• 4 "False" cases– A: two semantically related words

• (醫生 , 護士 )

– B: a part of multi-word collocation(>= 3 words)• (自我 , 機制 ) in (自我 , 約束 , 機制 )

– C: high frequency bigram• (他 , 說 ), (這 , 是 ), (很 , 好 )

– D: two words co-occurring frequently• (北京 , 月 ), (和 , 為 )

Further Evaluation (3)

• True collocations are much more than baseline• False collocation

– A: semantically related, not distinguishable by MWA– B: only two-word collocation is extracted.

• Few collocations have >=3 words

– C: frequent bigram, not distinguishable by MWA– D: much less than baseline

Further Evaluation (3) cont.

• MWA are able to produce long-span collocations• 48 extracted collocations with span > 6

– 33 are tagged "True"• ("處於 ", "狀態 "), ("由於 ", "因此 ")

– 69% precision

Fertility vs. Precision

• Manually label 100 sentences and observe fertility– 78% words collocate with 1 word– 17% words collocate with 2 words– 95% words have fertility <= 2

• Limit Φmax

Conclusion

• Main contribution– Successfully adapt BWA to MWA– Propose a ranking method

• Alignment probability + Exponential penalty function

• Initial failure are well discussed• Future work

– Improving Statistical Machine Translation with Monolingual Collocation, ACL 2010

– Improve alignment, phrase table