MIXED SYNTAX TRANSLATION Hieu Hoang MT Marathon 2011 Trento.
Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model ...
-
Upload
aileen-blankenship -
Category
Documents
-
view
215 -
download
0
Transcript of Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model ...
![Page 1: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/1.jpg)
Syntax for MT
EECS 767
Feb. 1, 2006
![Page 2: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/2.jpg)
Outline
Motivation Syntax-based translation model
FormalizationTraining
Using syntax in MTUsing multiple featuresSyntax-based features
![Page 3: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/3.jpg)
The IBM Models
Word reorderingSingle words, not groupsConditioned on position of words
Null-word insertionUniform across position
![Page 4: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/4.jpg)
The Alignment Template Model
Word ReorderingPhrases can be reordered in any way, but
tend to stay in same order as source.Reordering within phrases defined by
templates Word Translations
Must match up = No null
![Page 5: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/5.jpg)
Implied Assumptions
Word OrderSimilar to source sentence
TranslationNear 1-1 correspondence
![Page 6: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/6.jpg)
What goes wrong?
We see many errors in machine translation when we only look at the word level Missing content words
MT: Condemns US interference in its internal affairs. Human: Ukraine condemns US interference in its internal
affairs.
Verb phrase MT: Indonesia that oppose the presence of foreign troops. Human: Indonesia reiterated its opposition to foreign military
presence.
WS 2003 Syntax for Statistical Machine Translation Final Presentation
![Page 7: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/7.jpg)
What goes wrong cont.
Wrong dependencies MT: …, particularly those who cheat the audience
the players. Human: …, particularly those players who cheat
the audience.
Missing articles MT: …, he is fully able to activate team. Human: …, he is fully able to activate the team.
WS 2003 Syntax for Statistical Machine Translation Final Presentation
![Page 8: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/8.jpg)
What goes wrong cont.
Word salad: the world arena on top of the u . s . sampla compet
itors , and since mid – july has not appeared in sports field , the wounds heal go back to the situation is very good , less than a half hours in the same score to eliminate 6:2 in light of the south african athletes to the second round .
WS 2003 Syntax for Statistical Machine Translation Final Presentation
![Page 9: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/9.jpg)
How can we improve? Relying on language model to produce more ‘accurate’
sentences is not enough Many of the problems can be considered ‘syntactic’ Perhaps MT-systems don’t know enough about what is
important to people So, include syntax into MT
Build a model around syntax Include syntax-based features in a model
WS 2003 Syntax for Statistical Machine Translation Final Presentation
![Page 10: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/10.jpg)
A New Translation Story
You have a sentence and its parse tree The children at each node in the tree are rearranged New nodes may be inserted before or after a child node These new nodes are assigned a translation Each of the leaf lexical nodes is then translated
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 11: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/11.jpg)
A Syntax-based model
Assume word order is based on a reordering of source syntax tree.
Assume null-generated words happen at syntactical boundaries.
(For now) Assume a word translates into a single word.
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 12: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/12.jpg)
Reorder
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 13: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/13.jpg)
Insert
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 14: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/14.jpg)
Translate
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 15: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/15.jpg)
Parameters
Reorder (R) – child node reorderingCan take any possible child node reorderingDefines word order in translation sentenceConditioned on original child node orderOnly applies to non-leaf nodes
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 16: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/16.jpg)
Parameters cont.
Insertion (N) – placement and translationLeft, right, or noneDefines word to be insertedPlace conditioned on current and parent labelsWord choice is unconditioned
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 17: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/17.jpg)
Parameters cont. Translation (T) – 1 to 1
Conditioned only on source wordCan take on null
Translation (T) – N to NConsider word fertility (for 1-to-N mapping)Consider phrase translation at each nodeLimit size of possible phrasesMix phrasal w/ word-to-word translation
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 18: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/18.jpg)
Formalization
Set of nodes in parse tree
Total probability
Assume node independence
Assume random variables areIndependent of one another andonly dependent on certain features
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 19: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/19.jpg)
Training (EM)1. Initialize all probability tables (uniform)
2. Reset all counters
3. For each pair in the training corpusA) Try all possible mappings of N,R, and T
B) Update the counts as seen in the mappings
4. Normalize the probability tables with the new counts
5. Repeat 2-4 several times
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 20: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/20.jpg)
Decoding Modify original CFG with new reordering and their pro
babilities Add in VP->VP X and X -> word rules from N Add lexical rules englishWord->foreignWord Use the noisy-channel approach starting with a transla
ted sentence Proceed through the parse tree using a bottom-up bea
m search keeping an N-best list of good partial translations for each subtree
Yamada&Knight A Decoder for Syntax-based Statistical MT 2002
![Page 21: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/21.jpg)
Decoding cont.
Yamada&Knight A Decoder for Syntax-based Statistical MT 2002
![Page 22: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/22.jpg)
Performance (Alignment)
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 23: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/23.jpg)
Performance (Alignment) cont. Counting number of individual alignments Perfect means all alignments in a pair are
correct
Yamada A Syntax-Based Statistical Translation Model Thesis 2002
![Page 24: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/24.jpg)
Performance cont.
Chinese-English BLEU scores
Yamada&Knight A Decoder for Syntax-based Statistical MT 2002
![Page 25: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/25.jpg)
Do we need the entire model to be based on syntax? Good performance increase Large computational cost
Many permutations to CFG rules (120K non-lexical)
How about trying something else?Add syntax-based features that look for more
specific things
![Page 26: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/26.jpg)
Using Syntax in MT
Multiple FeaturesFormalizationBaselineTraining
Syntax-based FeaturesShallowDeep
![Page 27: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/27.jpg)
Multiple Features (log-linear)Calculate probability using a variety of features parameterized by an associated ‘weight’
Find the translated sentence which maximizes the feature function with your foreign sentence
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 28: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/28.jpg)
Baseline System
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 29: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/29.jpg)
Baseline System
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 30: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/30.jpg)
Baseline Features
Alignment template featureUses simple counts
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 31: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/31.jpg)
Baseline Features Word selection feature
Uses lexicon probability estimated by relative frequency
Additional feature capturing word reordering within phrasal alignments
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 32: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/32.jpg)
Baseline Features
Phrase alignment feature Measure of deviation from monotone alignment
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 33: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/33.jpg)
Baseline Features Language model feature
Standard backing-off trigram probability
Word/Phrase penalty feature Feature counting number of words in translated sentence Feature counting number of phrases in translated sentence
Alignment lexicon feature Feature counting the number of time something from a
given alignment lexicon is used
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 34: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/34.jpg)
A possible training method
Line optimization methodJHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 35: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/35.jpg)
Use reranking of N-best lists
Feature functions do not need to be integrated in dynamic programming search
A feature function can arbitrarily condition itself on any part of English/Chinese sentece/parse tree/chunks
Provides a simple software architecture Using a fixed set of translations allows feature functions to be a vect
or of numbers You are limited to improvements you see within the N-best lists
WS 2003 Syntax for Statistical Machine Translation Final Presentation
![Page 36: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/36.jpg)
Syntax-based Features Shallow
POS and Chunk Tag countsProjected POS language model
DeepTree-to-stringTree-to-treeVerb arguments
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 37: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/37.jpg)
Shallow Syntax-Based Features
POS and chunk tag count Low-level syntactic problems with baseline system. Too many ar
ticles, commas and singular nouns. Too few pronouns, past tense verbs, and plural nouns.
Reranker can learn balanced distributions of tags from various features
Examples Number of NPs in English Difference in number of NPs between English and Chinese Number of Chinese N tags translated to only non-N tags in E
nglish.
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 38: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/38.jpg)
Shallow Syntax-Based Features
Projected POS language model Use word-level alignments to project Chinese POS ta
gs onto the English words Possibly keeping relative position within Chinese phrase Possibly keeping NULLs in POS sequence Possibly using lexicalized NULLs from English word
Use the POS tags to train a language model based on POS N-grams
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 39: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/39.jpg)
Deep Syntax-Based Features
Tree to string Uses the Syntax-based model we saw previously Reduces computational cost by limiting size of reorder
ings Add in a feature for probability as defined by the mod
el and the probability of the viterbi alignment defined by the model
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 40: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/40.jpg)
Deep Syntax-Based Features
Tree to Tree Uses tree transformation functions similar to those in
the tree-to-string model The probability of transforming a source tree into a
target tree is modeled as a sequence of steps starting from the root of the target tree down.
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 41: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/41.jpg)
Tree to Tree cont. At each level of the tree:
1. At most one of the current node’s children is grouped with the current node into a single elementary tree with its probability conditioned on the current node and its children.
2. An alignment of the children of the current elementary tree is chosen with its probability conditioned on the current node an the children of child in the elementary tree. This is similar to the reorder operation in the tree-to-string model, but allows for node addition and removal.
Leaf-level parameters are ignored when calculating probability of tree-to-tree.
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 42: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/42.jpg)
Verb Arguments
Idea: A feature that counts the difference in the number of arguments to the main verb between the Chinese and English sentences
Perform a breadth-first search traversal of the dependency trees Mark the first verb encountered as the main verb The number of arguments is equal to the number of its children
JHU WS 2003 Syntax for Statistical Machine Translation Final Report
![Page 43: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/43.jpg)
Performance Some things helped, some things didn’t Is syntax useful? Necessary?
![Page 44: Syntax for MT EECS 767 Feb. 1, 2006. Outline Motivation Syntax-based translation model Formalization Training Using syntax in MT Using multiple.](https://reader030.fdocuments.net/reader030/viewer/2022032604/56649e695503460f94b661e4/html5/thumbnails/44.jpg)
References K. Yamada and K. Knight. 2001. A syntax-based statistical translation model. I
n ACL-01. K. Yamada. 2002. A Syntax-Based Statistical Translation Model. Ph.D. thesis,
University of Southern California. Yamada, Kenji and Kevin Knight. 2002. A decoder for syntaxbased MT. In Pro
c. of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA.
Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alex Fraser, Shankar Kumar, Libin Shen, David Smith, Katherine Eng, Viren Jain, Zhen Jin, and Dragomir Radev. A smorgasbord of features for statistical machine translation. In Proceedings of the Human Language Technology Conference.North American chapter of the Association for Computational Linguistics Annual Meeting, pages 161-168, 2004. MIT Press.
Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alex Fraser, Shankar Kumar, Libin Shen, David Smith, Katherine Eng, Viren Jain, Zhen Jin, and Dragomir Radev. Final Report of the Johns Hopkins 2003 summer workshop on Syntax for Statistical Machine Translation.
Philipp Koehn, Franz Josef Och, and Daniel Marcu. Statistical phrase-based translation. In Proceedings of the Human Language Technology Conference/North American Chapter of the Association for Computational Linguistics Annual Meeting, 2003. MIT Press.