5/28/031 Data Intensive Linguistics Statistical Alignment and Machine Translation.
-
Upload
elvin-parrish -
Category
Documents
-
view
233 -
download
0
Transcript of 5/28/031 Data Intensive Linguistics Statistical Alignment and Machine Translation.
5/28/03 1
Data Intensive Linguistics
Statistical Alignment and Machine Translation
5/28/03 2
Overview MT is very hard: by any reasonable
standard translation programs available today do not perform very well. But they are still used and useful.
Most MT systems are a mix of probabilistic and non-probabilistic components, though there are a few completely statistical translation systems.
5/28/03 3
Overview (Cont’d) A large part of implementing an MT
system is not specific to MT. Nonetheless, parts of MT that are
specific to it are: text alignment and word alignment.
Definition: In the sentence alignment problem, one seeks to say that some group of sentences in one language corresponds in content to some other group of sentences in another language. Such a grouping is referred to as a bead of sentences.
5/28/03 4
Overview of the Lecture Text Alignment Word Alignment Fully Statistical Attempt at MT
5/28/03 5
Text Alignment: Aligning Sentences and Paragraphs Text alignment is useful for bilingual
lexicography, MT, but also as a first step to using bilingual corpora for other tasks.
Text alignment is not trivial because translators do not always translate one sentence in the input into one sentence in the output, although they do so in 90% of the cases.
Another problem is that of crossing dependencies, where the order of sentences are changed in the translation.
5/28/03 6
Different Approached to Text Alignment Length-Based Approaches: short
sentences will be translated as short sentences and long sentences as long sentences.
Offset Alignment by Signal Processing Techniques: these approaches do not attempt to align beads of sentences but rather just to align position offsets in the two parallel texts.
Lexical Methods: Use lexical information to align beads of sentences.
5/28/03 7
Length-Based Methods I: General Approach
Goal: Find alignment A with highest probability given the two parallel texts S and T: arg maxA P(A|S, T)=argmaxA P(A, S, T)
To estimate the above probabilities, the aligned text is decomposed in a sequence of aligned beads where each bead is assumed to be independent of the others. Then P(A, S, T) k=1
K P(Bk). The question, then, is how to estimate the
probability of a certain type of alignment bead given the sentences in that bead.
5/28/03 8
Length-Based Methods II: Gale and Church, 1993
The algorithm uses sentence length (measured in characters) to evaluate how likely an alignment of some number of sentences in L1 is with some number of sentences in L2.
The algorithm uses a Dynamic Programming technique that allows the system to efficiently consider all possible alignments and find the minimum cost alignment.
The method performs well (at least on related languages). It gets a 4% error rate. It works best on 1:1 alignments [only 2% error rate]. It has a high error rate on more difficult alignments.
5/28/03 9
Length-Based Methods II: Other Approaches Brown et al., 1991: Same approach as
Gale and Church, except that sentence lengths are compared in terms of words rather than characters. Other difference in goal: Brown et al. Didn’t want to align entire articles but just a subset of the corpus suitable for further research.
Wu, 1994: Wu applies Gale and Church’s method to a corpus of parallel English and Cantonese Text. The results are not much worse than on related languages. To improve accuracy, Wu uses lexical cues.
5/28/03 10
Lexical Methods of Sentence Alignment I: Kay & Röscheisen, 1993
Assume the first and last sentences of the texts align. These are the initial anchors.
Then, until most sentences are aligned:1. Form an envelope of possible alignments.2. Choose pairs of words that tend to co-occur
in these potential partial alignments.3. Find pairs of source and target sentences
which contain many possible lexical correspondences. The most reliable of these pairs are used to induce a set of partial alignments which will be part of the final result.
5/28/03 11
Lexical Methods of Sentence Alignment II: Chen, 1993 Chen does sentence alignment by
constructing a simple word-to-word translation model as he goes along.
The best alignment is the one that maximizes the likelihood of generating the corpus given the translation model.
This best alignment is found by using dynamic programming.
5/28/03 12
Lexical Methods of Sentence Alignment III: Haruno & Yamazaki, 1996
Their method is a variant of Kay & Röscheisen (1993) with the following differences: For structurally very different languages,
function words impede alignment. They eliminate function words using a POS Tagger.
If trying to align short texts, there are not enough repeated words for reliable alignment using Kay & Röscheisen (1993). So they use an online dictionary to find matching word pairs
5/28/03 13
Word Alignment A common use of aligned texts is the
derivation of bilingual dictionaries and terminology databases.
This is usually done in two steps: First, the text alignment is extended to a word alignment. Then, some criterion, such as frequency is used to select aligned pairs for which there is enough evidence to include them in the bilingual dictionary.
Using a 2 measure works well unless one word in L1 occurs with more than one word in L2. Then, it is useful to assume a one-to-one correspondence.
Future work is likely to use existing bilingual dictionaries.
5/28/03 14
Fully Statistical MT Suppose we are translating F -> E We conceptualize this by imagining that
the French has arisen by a stochastic process from an underlying but unknown English version.
We need to guess the most likely such version
5/28/03 15
Stochastic MT Model P(E|F) as P(F|E)P(E) / P(F) P(F) is constant, all we want is to choose E P(F|E)P(E) is a noisy channel model, where
P(F|E) is a distorting channel that converts E into F
5/28/03 16
Why not just model P(E|F)? It seems circular to model P(E|F) by an
approach that requires P(F|E)
5/28/03 17
Why not just model P(E|F)? It seems circular to model P(E|F) by an
approach that requires P(F|E) P(F|E) is the translation specialist P(E) is the English specialist Both can be sloppy
5/28/03 18
Scene of the crime To be guilty
You have to have a motive You have to have the means The CSI person can ignore motive The police psychologist can ignore means
To be a good translation You have to convey the message You have to be plausible English P(F|E) can score translation while ignoring
many aspects of English P(E) doesn’t have to know anything about
French
5/28/03 19
Summa cum laude “Topmost with praise” scores high on P(L|
E) but poorly on P(E) “Suit and tie” scores much higher on P(E)
but badly on P(L|E) “With highest honors” maximizes P(E)P(L|
E) (we hope)
5/28/03 20
What are the models Source model p(E) could be trigram model
Guarantees semi-fluent English Channel model p(F|E) or p(L|E) could be finite-state
transducer Stochastically translates each word + allows a little
random rearrangement – with high prob, words stay more or less put
Maximizing p(F|E) would give really lousy French translation of English
Random word translation is stupid – need word sense from context
Random word rearrangement is stupid – phrases rearrange!
This channel has no idea what fluent French looks like But maximizing p(E)*p(F|E) gives a better English
translation of French because p(E) knows what English should look like.
5/28/03 21
IBM models 1,2,3,4,5 Models for P(F|E) There is a set of English words and the
extra English word NULL Each English word generates and places 0
or more French words Any remaining French words are deemed
to have been produced by NULL
5/28/03 22
IBM models 1,2,3,4,5 In Model 2, the placement of a word in the
French depends on where it was in the English
5/28/03 23
IBM models 1,2,3,4,5 In model 3 we model how many French
words and English word can produce, using a concept called fertility
5/28/03 24
IBM models 1,2,3,4,5 In model 4 the placement of later French
words produced by an English word depends on what happened to earlier French words generated by that same English word
5/28/03 25
Alignments The basic idea in all 5 is to build models
of translation on top of models of the alignment.
We really want P(F|E), which we could obtain from direct models of the joint distribution P(F,E)
Instead we introduce probabilistic models of the alignment A, and work with P(F,A,E)
5/28/03 26
Alignments
The dog ate my homework
Le chien a mangé mes devoirs
5/28/03 27
IBM models 1,2,3,4,5 In model 5 they do non-deficient
alignment. To understand what is at stake, you need to know a little more about the internals of the models
5/28/03 28
Alignments All of models 1 through 5 are actually
models of how alignments come to be Conceptually, we form the probability of
a pair of sentences by summing over all alignments that relate these sentences.
In practice, the Candide team sometimes preferred to approximate by assuming that the sentence pair having the single best alignment was also the best sentence pair.
5/28/03 29
Why all the models We don’t start with aligned text, so we
have to get initial alignments from somewhere.
Model 1 is words only, and is relatively easy to train with technology similar to the forward-backward algorithm for HMMs. This is the EM algorithm
5/28/03 30
Why all the models We are working in a space with many
parameters and many local minima. EM guarantees only local optimality, so it
pays (as in Pereira and Schabes’ work on grammar) to start from a place which is already good.
5/28/03 31
Why all the models The purpose of model 1 is to find a good
place in which to start training model 2 Model 2 feeds model 3 And so on
5/28/03 32
Maximum entropy Going beyond the 5 models, there is a
systematic use of log-linear maximum entropy models
These refine the probability models using features like French word is “en”, English word is
“in”, neighboring English word is a month name
5/28/03 33
Practicalities Training for 2 million Hansard sentences
takes 3600 processor hours on a farm of 15 1993 vintage IBM PowerPC processors.
My guess is that we could do this in a week (8*2* hours
5/28/03 34
Performance Rated on fluency and adequacy by
independent raters (see Candide paper for details) Systran gets a score of .743 on adequacy,
Candide gets .670 Candide gets a score of .580 on fluency,
where Systran gets only .540 Both are significantly less good than
Transman, which is human-in-the-loop system
5/28/03 35
Prospects More elaborate translation models, with
serious notions of phrase, word-sense, etc. are in the works.
Nobody really knows what level of quality is attainable.
Nobody really knows what level of quality is required for wider adoption.
For specific tasks, MT is clearly useful
5/28/03 36
Prospects Many other tasks can be coerced to look
like translation Summarization (E -> Summary) Information retrieval (E -> Keywords) Title generation (E -> Title) Paraphrasing (E -> Paraphrase)
There’s a large and growing literature using this idea.
5/28/03 37
Where to read moreM&S chapter 13 and references therein
Kevin Knight Automating Knowledge Acquisition for Machine translation AI Magazine 18(4) 1997
http://www-2.cs.cmu.edu/~aberger/mt.html
Especially the overview paper on Candide and the longer paper on Maximum Entropy with Berger as first author
5/28/03 38
Offset Alignment by Signal Processing Techniques I : Church, 1993 Church argues that length-based methods work
well on clean text but may break down in real-world situations (noisy OCR or unknown markup conventions)
Church’s method is to induce an alignment by using cognates (words that are similar across languages) at the level of character sequences.
The method consists of building a dot-plot, i.e., the source and translated text are concatenated and then a square graph is made with this text on both axes. A dot is placed at (x,y) when there is a match. [Unit=4-grams].
5/28/03 39
Offset Alignment by Signal Processing Techniques II: Church, 1993 (Cont’d)
Signal processing methods are then used to compress the resulting plot.
The interesting part in a dot-plot is called the bitext maps. These maps show the correspondence between the two languages.
In the bitext maps, can be found faint, roughly straight diagonals.
A heuristic search along this diagonal provides an alignment in terms of offsets in the two texts.
5/28/03 40
Offset Alignment by Signal Processing Techniques III: Fung & McKeown, 1994
Fung and McKeown’s algorithm works: Without having found sentence boudaries. In only roughly parallel text (with certain
sections missing in one language) With unrelated language pairs.
The technique is to infer a small bilingual dictionary that will give points of alignment.
For each word, a signal is produced, as an arrival vector of integer numbers giving the numver of words between each occurrence of the word at hand.