ABC--- A Phrase-to-Phrase Alignment Method Integrating monolingual and bilingual information in sub...

ABC---A Phrase-to-Phrase Alignment Method

Integrating monolingual and bilingual information in sub sentential phrase alignment

Ying Zhang (Joy) Joy+@cs.cmu.edu

ISL, Carnegie Mellon Univ.

July 08, 2002

07/09/2002Copyright. Joy, joy@cs.cmu.edu2

Overview

The advantage of phrase to phrase alignment Existing methods Algorithm Integrating bilingual information with

monolingual information Experiments and results Discussion and future work

SMT and sub-sentential alignment

Statistical Machine Translation (SMT) system is based on the noise channel model

)()|(maxarg

)|(maxarg

tgtPtgtsrcP

srctgtPnTranslatio

Translation Model

Language Model

SMT and sub-sentential alignment (Cont.)

Through sub-sentential alignment, we are training the Translation Model (TM)

In our system, TM contains word to word, or phrase to phrase transducers. E.g.

Why phrases?

Mismatch between languages

Why phrases? (Cont.)

Phrases encapsulate the context of words– Tense: e.g.

Word to word alignment

Phrase to phrase alignment

Local reordering– E.g. Relative clauses in Chinese

Which still needs global reordering, which is our future work

For languages need word segmentation, such as Chinese– The word segmenter can not segment the sentence

perfectly, due to the incomplete coverage of word list and segmentation ambiguity

– Previous work (Zhang 2001) tries to identify phrases in the corpus using only monolingual information and augment the word list with new phrases found

Precision: Hard to decide on phrase boundary Prediction: Phrase identified may not occur in the future

testing data

Example of using phrases to soothe word segmentation failure

Some alignment algorithms

IBM models (Brown 93) HMM alignment: phrase to phrase (Vogel 96) Competitive links: word to word (Melamed 97) Flow network (Gaussier 98) Bitext Map (Melamed 01)

Algorithm

Given a sentence pair (S,T),

S=<s1,s2,…,si,…sm>

T=<t1,t2,…,tj,…,tn>, where si/tj are src/tgt words.

Given an m*n matrix B, where

B(i,j)= co-occurrence(si,tj)=

N=a+b+c+d;

)()()()(

dcbadbca

bcadNts ji

tj ~tj

si a b

~si c d

Algorithm (Cont.)

Goal: find a partition over matrix B, under the constraint that one src/tgt word can only align to one tgt/src word or one tgt/src phrase (adjacent word sequence)

Legal segmentation, imperfect alignment Illegal segmentation, perfect alignment

Algorithm (Cont.)

While(still has row or column not aligned){

Find cell[i,j], where B(i,j) is the max among all available(not aligned) cells;

Expand cell[i,j] with similarity sim_thresh to region[RowStart,RowEnd; ColStart,ColEnd]

Mark all the cells in the region as aligned

Output the aligned regions as phrases

Algorithm (Cont.)

Expand cell[i,j] with sim_threshcurrent aligned region: region[RowStart=i, RowEnd=i; ColStart=j, ColEnd=j]

While(still ok to expand){

if all cells[m,n], where m=RowStart-1, ColStart<=n<=ColEnd, B(m,n) is similar to B(i,j) then RowStart = RowStart --; //expand to north

if all cells[m,n], where m=RowEnd+1, ColStart<=n<=ColEnd, B(m,n) is similar to B(i,j) then RowStart = RowStart ++; //expand to south

… //expand to east

… //expand to west

Define similar(x,y)= true, if abs((x-y)/y) < 1-similarity_thresh

Algorithm (Cont.)

Expand to North

Expand to South

Expand to EastExpand to West

Find the best similarity threshold

Simlarity_threshold is critical in this algorithm

The algorithm described above used ONE Simlarity_threshold value for all region expansions in the matrix, and the same ONE value for all sentence pairs

Ideally, it is better to use different threshold values for each region and find the global best segmentation for one matrix

– A search tree, combinational explosion

Find the best similarity threshold (Cont.)

One practical solution:For one matrix B:

For(st=0.1;st<=0.9;st+=0.1){

find segmentation of B given similarity_threshold = st;

Select the solution with the highest performance(solution)

lsAlignedCelji

solutionePerformanc

Integrating monolingual information

Motivation:– Use more information in the alignment– Easier for aligning phrases– There is much more monolingual data than bilingual

Pittsburgh Los AngelesSomerset

Union town Santa Monica

Santa Clarita

Corona

Integrating monolingual information (Cont.)

Given a sentence pair (S,T),

S=<s1,s2,…,si,…sm> and T=<t1,t2,…,tj,…,tn>, where si/tj are src/tgt words.

Construct an m*m matrix A, where A(i,j) = collocation(si, sj); Only A(i,i-1) and A(i,i+1) have values

Construct an n*n matrix C, where C(i,j) = collocation(ti, tj); Only C(j-1,j) and A(j+1,j) have values

Construct an m*n matrix B, where

B(i,j)= co-occurrence(si, tj).

Normalization:– Assign self2self value α(si) A(i,i), 0<=α(si)<=1

– Assign self2self value β(tj) C(j,j), 0<= β(tj)<=1– Normalize A so that:

))(1()1,()1,(

)1,()1,(' isiiAiiA

iiAiiA

))(1()1,()1,(

)1,()1,(' isiiAiiA

iiAiiA

)(),(' isiiA

jiA 1),('

– Normalize C so that:

– Normalize B so that:

jiC 1),('

))(1(),1(),1(

),1(),1(' jtjjCjjC

jjCjjC

))(1(),1(),1(

),1(),1(' jtjjCjjC

jjCjjC

)(),(' jtjjC

jiB1 1

Calculating new src-tgt matrix B’

OK. That’s it! Yes, that’s the whole story!

''''' CBAB

Example

With pure bilingual information:

After integration with monolingual information:the development of shanghai's pudong

shanghai 0.134 0.053 0.059 7.016 10.196pudong 0.076 0.061 0.029 9.963 38.152development 2.993 5.482 1.851 1.954 8.213

the development of shanghai's pudongshanghai 0.225 0.056 0.061 7.477 0.413pudong 0.047 0.021 0 0.261 53.949development 0.116 9.641 0.024 0.014 0.106

Visualization

Left: Using pure bilingual information

Right: Integrated with monolingual information

What Is the Self2self Value?

Take a look at:

What actually happens is:

stands for how much word Si should “make use of” its neighbours’ relation with the target words.

For content words, self2self value should be higher, and for function words, it should be lower.

jiijiiiii

iiiji BB

AB ,,1

1,, )1()1('

How To Set the Self2self Values

Well, this is tricky Before June evaluation I set α = 0.6 for all src

words and β = 0.48 for all tgt words– Not good– “the” should have lower self2self value and

“Pittsburgh” should have a higher self2self value

Calculating Self2self Values

Observation: Source language content words tend to align to a few target words with high scores while function words tend to align to many target words with low scores

“has”“the”

“beijing”“computer”“bus”

“in”

Calculating Self2self Values (Cont.)

Calculating the entropy of a word over the distribution of normalized co-occurrence scores

– Given word si, for all the possible co-occurred word tj, their co-occurrence score is C(i,j),

– Let

– Define

Map the score linearly to a value between 0~1 Better map the scores to a range narrower than 0~1. E.g.

0.45~0.85, why?

ij jiC

jiCstprob

),()|(

))|((log)|()( ijj

iji stprobstprobsH

A Modification to the Segmentation Algorithm

Original algorithm calculates A*B*C only once In the modified version:

– Set B[i,j] to 0 for all aligned cells when a new aligned region is found

– Re-calculate A*B*C Motivation:

– Since we found an aligned region, the boundary of this phrase is known. It should not affect the unaligned neighbors

More computationally expensive Experiments showed better performance

Updating Bilingual Information by Iteration

Using EM to update the bilingual co-occurrence scores– Doesn’t help too much

Results

The Dev-test on small data track (3540 sentence pair training data + 10K glossary)

NIST Bleu

Baseline(IBM1+Gloss) 6.0097 0.1231

Original Algorithm 6.3673 (+5.9%) 0.1478 (+20.0%)

Modified Algorithm 6.4310 (+7.0%) 0.1507 (+22.4%)

After LM-fill NIST Bleu

Baseline(IBM1+Gloss)+LM 6.3775 0.1417

Original Algorithm+LM 6.6754(+4.7%) 0.1611(+13.7%)

Modified Algorithm+LM 6.7987(+6.6%) 0.1712(+20.8%)

Results (Cont.)

No LM-fill With LM-fill

NIST Bleu NIST Bleu

Baseline(IBM1+Gloss) 6.0097 0.1231 6.3775 0.1417

HMM+IBM1+Gloss 6.1802 0.1305 6.4750 0.1459

ARV+IBM1+Gloss 6.3636 0.1473 6.7405 0.1681

JOY+IBM1+Gloss 6.4310 0.1507 6.7987 0.1712

ARV+JOY+IBM1+Gloss 6.5117 0.1569 6.8790 0.1776

Conclusion

Simple Efficient

– Unlike stochastic bracketing (Wu 95) which is O(m3n3), the algorithm of segmenting the matrix is linear O(min(m,n)). The construction of A*B*C is O(m*n);

Effective– Improved the translation quality from baseline (NIST=6.0097,

Bleu=0.1231) to (NIST=6.4310, Bleu=0.1507) on small data track dev-test

Future work

Find a better segmentation algorithm (dynamic threshold)

Find a method which is mathematically more sound for self2self values

Investigate the possibility of using trigram or distance bi-gram monolingual information

References

Peter F. Brown, Stephen A. Della Pietra, Vin-cent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machinetranslation: Parameter estimation. Computa-tional Linguistics, 19 (2) :263-311.

Gaussier, E. (1998) Flow Network Models for Word Alignment and Terminology Extraction from Bilingual Corpora. In Proceedings of COLING-ACL-98, Montreal, pp. 444-450.

I. Dan Melamed. "A Word-to-Word Model of Translational Equivalence". In Procs. of the ACL97. pp 490--497. Madrid Spain, 1997.

I. Dan Melamed (2001). Empirical Methods for Exploiting Parallel Texts MIT Press. Stephan Vogel, Hermann Ney, and Christoph Till-mann. 1996. HMM-based word alignment in statistical

translation. In COLING '96: The 16th Int. Conf. on Computational Linguistics, pages 836-841, Copenhagen, August.

Dekai Wu, An Algorithm for Simultaneously Bracketing Parallel Texts by Aligning Words, ACL, June 1995 Ying Zhang, Ralf D. Brown, Robert E. Frederking and Alon Lavie. "Pre-processing of Bilingual Corpora for

Mandarin-English EBMT". MT Summit VIII, Sep. 2001.

Acknowledgement

I would like to thank Stephan Vogel, Jian Zhang, Jie Yang, Jerry Zhu, Ashish and other people for their valuable advice and suggestions during this work.

Questions and Comments

ABC--- A Phrase-to-Phrase Alignment Method Integrating monolingual and bilingual information in sub...

Documents

Transcript of ABC--- A Phrase-to-Phrase Alignment Method Integrating monolingual and bilingual information in sub...

JOY · pampers. JOY HAPPINESs LION

Statistical Phrase Alignment Model Using Dependency Relation Probability

Phonological phrase boundaries constrain lexical access ...titan.cog.brown.edu/research/infant_lab/phonological phrase boundaries.pdf · phonological phrase boundaries constrain lexical

1. Possesor Phrase Possessee Noun (Genitive Noun phrase) (Head Noun) 2.Possessee Noun Possesor Phrase (Head Noun) (Genitive Noun phrase) GENITIVE PHRASE.

HMM Word and Phrase Alignment for Statistical Machine ...mi.eng.cam.ac.uk/~wjb31/ppubs/hltemnlp05wtoptalk.pdf · Introduction HMM HMM-based Word Alignment Models (Vogel et al, ’96)

Phrase Alignment Models for Statistical Machine …denero/research/denero_phd...Abstract Phrase Alignment Models for Statistical Machine Translation by John Sturdy DeNero Doctor of

Research Article A Relationship: Word Alignment, Phrase ...

Phrase structure grammar and phrase structure parsingstp.lingfil.uu.se/~nivre/master/5LN710/2012-12-06.pdf · Phrase structure grammar and phrase structure parsing Natural Language

LYRICS-TO-AUDIO ALIGNMENT AND PHRASE-LEVEL … · LYRICS-TO-AUDIO ALIGNMENT AND PHRASE-LEVEL SEGMENTATION USING INCOMPLETE INTERNET-STYLE CHORD ANNOTATIONS Matthias Mauch Hiromasa

German Dictionary Phrase Book 3 Rough Guide Phrase Books

Improving SMT with Phrase to Phrase Translations

· Web viewappositive phrase, infinitive phrase, participial phrase, gerund phrase plagiarism - the use of ideas or writings of another author, representing them as one’s original

#JOY# $JOY$ *JOY MANGANO* @MANGANO@

Improving Semantic Textual Similarity with Phrase Entity Alignment · 2017. 6. 18. · Received: January 7, 2017 193 International Journal of Intelligent Engineering and Systems,

111 The Impact of Statistical Word Alignment Quality and Structure in Phrase-based Statistical Machine Translation A doctoral dissertation by: Francisco.

Linguistically-motivated Tree-based Probabilistic Phrase Alignment Toshiaki Nakazawa, Sadao Kurohashi (Kyoto University)

CSE 517 Natural Language Processing Winter 2013...Learning a Phrase Translation Table • Task: learn the model from a parallel corpus • Three stages: – word alignment: using IBM

BIA: a discriminative phrase alignment toolkit · BIA: a Discriminative Phrase Alignment Toolkit Patrik Lambert1 and Rafael Banchs2 1. LIUM (Computing Laboratory) University of Le

has been in the United States for 70 years Prepositional Phrase Verb Phrase Prepositional Phrase.

Phrase simple et phrase complexe (CM2)

#JOY# $JOY$ JOY MANGANO @MANGANO@