Adaptor Grammars
description
Transcript of Adaptor Grammars
1
Adaptor Grammars
Ehsan Khoddammohammadi
Recent Advances in Parsing TechnologyWS 2012/13
Saarland University
2
Outline
• Definition and motivation behind unsupervised grammar learning
• Non-parametric Bayesian statistics• Adaptor grammars vs. PCFG• A short introduction to Chinese Restaurant
Process• Applications of Adaptor grammar
3
Unsupervised Learning
• How many categories of objects?• How many features does an object have?• How many words and rules are in a language?
4
Grammar Induction
Goal:– study how a grammar and parses can be learnt
from terminal strings alone
Motivation:– Help us understand human language acquisition– Inducing parsers for low-resource languages
5
Nonparametric Bayesian statistics
• Learning the things people learn requires using rich, unbounded hypothesis spaces
• Language learning is non-parametric inference, no (obvious) bound on number of words, grammatical, morphemes.
• Use stochastic processes to define priors on infinite hypothesis spaces
6
Nonparametric Bayesian statistics
• Likelihood: how well grammar describes data• Prior: Encode our knowledge or expectation of
grammars before seeing the data– Universal Grammar (very specific)– Shorter Grammars (general constraints)
• Posterior: Shows uncertainty of learner about which grammar is correct (distribution over grammars)
Posterior Likelihood Prior
7
Is PCFG good enough for our purpose?
• PCFG can be learnt through Bayesian framework but …
• Set of rules is fixed in standard PCFG estimation
• PCFG rules are “too small” to be effective units of generalization
How can we solve this problem?
8
1. let the set of non-terminals grow unboundedly:– Start with un-lexicalized short grammar– Split-Join of non-terminals
2. let the set of rules grow unboundedly:– Generate new rules when ever you need– Learn sub-trees and their probabilities ( Bigger units
of generalization)
Two Non-parametric Bayesian extensions to PCFGs
9
Adaptive Grammar
• CFG rules is used to generate the trees as in a CFG
• We have two types of non-terminals:– Un-adapted (normal) non-terminals• Picking a rule and recursive expanding its children as in
PCFG– Adapted non-terminals• Picking a rule and recursive expanding its children• Generating a previously generated tree (proportional to
number of times that it is already generated)We have a Chinese Restaurant Process for each
adapted non-terminal
10
The Story of Adaptor Grammars• In PCFG, rules are applied independently from each other.• The sequence of trees generated by an adaptor grammar
are not independent.• if an adapted sub-tree has been used frequently in the
past, it's more likely to be used again.• An un-adapted nonterminal expands Using with probability
proportional to • An adapted nonterminal expands:
– to a sub-tree rooted in with probability proportional to the number of times was previously generated
– Using with probability proportional to – is prior.
11
Properties of Adaptor grammars
• In Adaptor grammars:– The probability of adapted sub-trees are learnt
separately, not just product of probability of rules.
– “Rich get richer” (Zipf distribution)
– Useful compound structures are more probable than their parts.
– there is no recursion amongst adapted non-terminals (an adapted non-terminal never expands to itself)
12
The Chinese Restaurant Process
13
The Chinese Restaurant Process
P(zi j | z1,...,zi 1)
n ji 1
existing table j
i 1
next unoccupied table
• n customers walk into a restaurant, choose tables zi with probability
• Defines an exchangeable distribution over seating arrangements (inc. counts on tables)
14
CRP
15
CRP
16
CRP
17
CRP
18
CRP
19
Application of Adaptor grammars
No usage for parsing! Because grammar induction is hard.
1. Word Segmentation2. Learning concatenative morphology3. Learning the structure of NE NPs4. Topic Modeling
20
Unsupervised Word Segmentation
• Input: phoneme sequences with sentence boundaries
• Task: identify words
21
Word segmentation with PCFG
22
Unigram word segmentation
23
Collocation word segmentation
24
Performance
Generalization Accuracy
Unigram 56%
+ collocations 76%
+ syllable structure 87%
• Evaluated on Brent corpus
25
Morphology
• Input: raw text• Task: identify stems and morphemes and
decompose a word to its morphological components
• Adaptor grammars can just be applied for simple concatenative morphology.
26
CFG for morphological analysis
27
Adaptor grammar for morphological analysis
Generated Words:1. cats2. dogs3. cats
28
Performance
• For more sophisticated model:
• 116,129 tokens: 70% correctly segmented
• 7,170 verb types: 66% correctly segmented
29
Inference
• distribution of adapted trees are exchangeable : Gibbs sampling
• Variational Inference method is also provided for learning adaptor grammars.
Covering this part is beyond the objectives of this talk.
30
Conclusion
• We are interested in inducing grammars without supervision for two reasons:– Language acquisition – Low-resource languages
• PCFG rules are too much small for bigger generalization
• Learning the things people learn requires using rich, unbounded hypothesis spaces
• Adaptor grammars using CRP to learn rules from this unbounded hypothesis spaces
31
References• Adaptor Grammars: A Framework for Specifying Compositional
Nonparametric Bayesian Models, M. Johnson et al. , ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS, 2007
• Using adaptor grammars to identify synergies in the unsupervised acquisition of linguistic structure, Mark Johnson, ACL-08, HLT , 2008
• Inferring Structure from Data, Tom Griffith, Machine Learning summer school, Sardinia, 2010
32
Thank you for your attention!