Post on 28-Dec-2015
Computational Linguistics
What is it and what (if any) are its
unifying themes?
2
Computational linguistics
3
I often agree with XKCD…
4
mo
re rig
oro
us
less rig
oro
us
mo
re flakey
literarycriticism
physics biologychemistry psychology
computational linguistics
neuropsychology
linguistics?
5
What defines the rigor of a field?
• Whether results are reproducible
• Whether theories are testable/falsifiable
• Whether there are a common set of methods for similar problems
• Whether approaches to problems can yield interesting new questions/answers
6
Linguistics
7
mo
re rig
oro
us
less rig
oro
us
literarycriticism
engineering sociologylinguistics
8
mo
re rig
oro
us
less rig
oro
us
other areas of sociolinguistics(e.g. D
eborah Tannen)
“theoretical” linguistics(e.g. m
inimalist syntax)
some areas of sociolinguistics
(e.g. Bill Labov)
The true situation with linguistics
psycholinguistics
experimental phonetics
historical linguistics
“theoretical” linguistics(e.g. lexical-functional gram
mar)
9
Okay enough alreadyWhat is computational linguistics
• Text normalization/segmentation• Morphological analysis• Automatic word pronunciation prediction• Transliteration• Word-class prediction: e.g. part of speech tagging• Parsing• Semantic role labeling• Machine translation• Dialog systems• Topic detection• Summarization• Text retrieval• Bioinformatics• Language modeling for automatic speech recognition• Computer-aided language learning (CALL)
10
Computational linguistics
• Often thought of as natural language engineering
• But there is also a serious scientific component to it.
11
Why CL may seem ad hoc
• Wide variety of areas (as in linguistics)
• If it’s natural language engineering, the goal is often just to build something that works
• Techniques tend to change in somewhat faddish ways…– For example: machine learning approaches
fall in and out of favor
12
13
14
15
16
Machine learning in CL
• In general it’s a plus since it has meant that evaluation has become more rigorous
• But it’s important that the field not turn into applied machine learning
• For this to be avoided, people need to continue to focus on what linguistic features are important
• Fortunately, this seems to be happening
17
Some interesting themes…
• Finite-state methods:– Many application areas– Raises interesting questions about how much of
language is “regular” (in the sense of “finite state”)
• Grammar induction:– Linguists have done a poor job at their stated goal of
explaining how humans learn grammar
• Computational models of language change:– Historical evidence for language change is only
partial. There are many changes in language for which we have no direct evidence.
18
Finite state methods
• Used from the 1950’s onwards
• Went out of fashion a bit during the 1980’s
• Then a revival in the 1990’s with the advent of weighted finite-state methods
19
Some applications
• Analysis of word structure – morphology
• Analysis of sentence structure– Part of speech tagging– Parsing
• Speech recognition
• Text normalization
• Computational biology
• …
20
Regular languages
• A regular language is a language with a finite alphabet that can be constructed out of one or more of the following operations:– Set union– Concatenation– Transitive closure (Kleene star)
21
Finite state automata: formal definition
Every regular language can be recognized by a finite-state automaton.Every finite-state automaton recognizes a regular language. (Kleene’s theorem)
22
Representation of FSA’s: State Diagram
23
Regular relations: formal definition
24
Finite-state transducers
25
An FST
26
Composition
• In addition to union, concatenation and Kleene closure, regular relations are closed under composition
• Composition is to be understood here the same way as composition in algebra:– R1oR1 means take the output of R1 and feed it
to the input of R2
27
Composition: an illustration
28
R1 as a transducer
29
R2 as a transducer
30
R1○R2
31
Some things you can do with FSTs
• Text analysis/normalization– Word segmentation– Abbreviation expansion– Digit-to-number-name mappingsi.e. mapping from writing to language
• Morphological analysis• Syntactic analysis
– E.g. part-of-speech tagging
• (With weights) pronunciation modeling and language modeling for speech recognition
32
That’s fine for engineering but…
• Does it really account for the facts?– Is morphology really regular?– Is the mapping between writing and speech
really regular?
33
What is morphology?
• scripsērunt is third person, plural, perfect, active of scrībō (`I write’)
• Morphology relates word forms– the “lemma” of scripsērunt is scrībō
• Morphology analyzes the structure of word forms– scripsērunt has the structure scrīb+s+ērunt
34
Morphology is a relation
• Imagine you have a Latin morphological analyzer comprising:– D: a relation that maps between surface form
and decomposed form– L: a relation that maps between decomposed
form and lemma
• Then:– scripsērunt ○ D = scrīb+s+ērunt– scripsērunt ○ D ○ L = scrībō
35
English regular plurals
• cat + s = cats /s/
• dog + s = dogs /z/
• spouse + s = spouses /Əz/
• This can be implemented by a rule that composes with the base word, inserting the relevant form of the affix at the end
36
Templatic affixes in Yowlumne
Transducer for each affix transforms base into required templaticform and appends the relevant string.
37
Subtractive morphology
Transducer deletes final VC of the base…
38
Bontoc infixation
• Insert a marker “>” after the first consonant (if any)
• Change “>” into the infix –um-
39
Side note … infixation in English
Kalamazoo
f*****g
40
Reduplication: Gothic
Problem: mapping w to ww is not a regular relation
41
Factoring Reduplication
• Prosodic constraints
• Copy verification transducer C
42
Non-Exact Copies
• Dakota (Inkelas & Zoll, 1999):
43
Non-Exact Copies
• Basic and modified stems in Sye (Inkelas & Zoll, 1999):
“they will fall all over”
44
Morphological Doubling Theory(Inkelas & Zoll, 1999)
• Most linguistic accounts of reduplication assume that the copying is done as part of morphology
• In MDT:– Reduplication involves doubling at the
morphosyntactic level – i.e. one is actually simply repeating words or morphemes
– Phonological doubling is thus expected, but not required
45
Gothic Reduplication under Morphological Doubling Theory
46
Summary
• If Inkelas & Zoll are right then all morphology can be computed using regular relations
• This in turn suggests that computational morphology has picked the right tool for the job
47
Another Example: Linguistic analysis of text
• Maps between the stuff you see on the page – e.g. text written in the standard orthography of a language – into linguistic units (words, morphemes, phonemes…)
• For example:– I ate a 25kg bass– [aI εIt Ə twεnti faIv kIlƏgræm bæs]
• This can be done using transducers– But is the mapping between writing and language
really regular (finite-state)?
48
Linguistic analysis of text
• Abbreviation expansion
• Disambiguation
• Number expansion
• Morphological analysis of words
• Word pronunciation
• …
49
A transducer for number namesConsider a machine that maps between digit strings and their reading as number names in English.
30,294,005,179,018,903.56 → thirty quadrillion, two hundred and ninety four trillion, five billion, one hundred seventy nine million, eighteen thousand, nine hundred three, point five six
50
Mapping between speech and writing
It seems obvious on the face of it that the mapping between speech and its written form is regular. After all, the words are ordered in the same way as speech. Even the tend to be ordered in the same lettersway as the sounds they represent.
51
Some examples where it isn’t…
twt`nx
jmn
nb
xpr
w
r`
‘honorific inversion’
52
Finite state methods
• In morphology they seem almost exactly correct as characterizations of the natural phenomenon
• In the mapping from writing to language, again, finite-state models seem almost exactly correct
53
Grammar induction
The common “nativist” view in linguistics…From Gilbert Harman's review of Chomsky's New Horizons in the Study of Language and Mind (published in Journal of Philosophy, 98(5), May 2001):
Further reflection along these lines and a great deal of empirical study of particular languages has led to the "principles and parameters" framework which has dominated linguistics in the last few decades. The idea is that languages are basically the same in structure, up to certain parameters, for example, whether the head of a phrase goes at the beginning of a phrase or at the end. Children do not have to learn the basic principles, they only need to set the parameters. Linguistics aims at stating the basic principles and parameters by considering how languages differ in certain more or less subtle respects. The result of this approach has been a truly amazing outpouring of discoveries about how languages are the same yet different.
54
Similarly…
Children come equipped with a set of principles of grammar construction (i.e. Universal Grammar (UG)). The principles of UG have open parameters. Specific grammars arise once values for these open parameters are specified. Parameter values are determined on the basis of [the primary linguistic data]. A language specific grammar, then, is simply a specification the values that the principles of UG leave open.
Cedric Boeckx and Norbert Hornstein. 2003. “The Varying Aims of Linguistic Theory.”
55
My “challenge” with Shalom Lappin
56
…
57
Automatic induction of grammars from unannotated text
• Klein, Dan and Manning, Christopher. 2004. “Corpus-based induction of syntactic structure: models of dependency and constituency”. Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics
• Lots of subsequent work…
58
Different syntactic representations
59
Dependency Model with Valence (DMV)
• Each head generates a set of non-STOP arguments to one side, then a STOP argument; then similarly on the other side
• Trained using expectation maximization
60
Performance
61
Improvements
• Constituent structure can be induced in a similar way to inducing word classes (e.g. parts of speech) – by considering the environments in which the putative constituent finds itself.
• In Klein & Manning’s constituent-context model (CCM) probability of a bracketing is computed as follows:
62
Combined DMV+CCM
Subsequent work – e.g. Rens Bod’s 2006 Unsupervised Data Oriented Parsing – report F-scores close to 83.0
For comparison, the best supervised parsers get about 91.0
63
Some objections … and a synopsis
• Children do not learn grammars from unannotated text corpora: they get a lot of guidance from the environmental situation– Sure
• Performance of automatic induction algorithms is still far from human performance so they do not constitute evidence that we can do away with (nativist) linguistic theories of language acquisition– They do not show this. But the argument would have more
weight if nativist theories had already been demonstrated to contribute to a working model of grammar induction
• But Computational Linguistics is starting to make some serious contributions to this 50-year-old debate
64
The evolution of complex structure in language
Examples from: Stump, Gregory (2001) Inflectional Morphology:A Theory of Paradigm Structure. Cambridge University Press.
65
Evolutionary Modeling (A tiny sample)
• Hare, M. and Elman, J. L. (1995) Learning and morphological change. Cognition, 56(1):61--98.
• Kirby, S. (1999) Function, Selection, and Innateness: The Emergence of Language Universals. Oxford
• Nettle, D. "Using Social Impact Theory to simulate language change". Lingua, 108(2-3):95--117, 1999.
• de Boer, B. (2001) The Origins of Vowel Systems. Oxford
• Niyogi, P. (2006) The Computational Nature of Language Learning and Evolution. Cambridge, MA: MIT Press.
66
A multi-agent simulation• System is seeded with a grammar and small number of agents
– Each agent randomly selects a set of phonetic rules to apply to forms– Agents are assigned to one of a small number of social groups
• 2 parents “beget” child agents.– Children are exposed to a predetermined number of training forms combined
from both parents• Forms are presented proportional to their underlying “frequency”
– Children must learn to generalize to unseen slots for words– Learning algorithm similar to:
• David Yarowsky and Richard Wicentowski (2001) "Minimally supervised morphological analysis by multimodal alignment." Proceedings of ACL-2000, Hong Kong, pages 207-216.
• Features include last n-characters of input form, plus semantic class– Learners select the optimal surface form to derive other forms from (optimal =
requiring the simplest resulting ruleset – a Minimum Description Length criterion)• Forms are periodically pooled among all agents and the n best forms are
kept for each word and each slot• Population grows, but is kept in check by “natural disasters” and a quasi-
Malthusian model of resource limitations– Agents age and die according to reasonably realistic mortality statistics
67
Final states for a given initial state
68
Another example
• Kirby, Simon. 2001. “Spontaneous evolution of linguistic structure: an iterated learning model of the emergence of regularity and irregularity.” IEEE Transactions on Evolutionary Computation, 5(2):102--110.
• Assumes two meaning components each with 5 values, for 25 possible words
• Initial “speaker” randomly selects examples from the 25, producing random strings for each, and “teaches” them to the “hearer”
• Not all of the slots are filled, thus producing a “bottleneck”: the hearer must compute forms for the missing slots
69
The basic algorithm produces results that are too regular
Initial state
Final state
70
A more realistic result…• Addition of other
constraints, including – a random tendency for
“speakers” to omit symbols,
– a frequency distribution over the 25 possible meaning combinations
71
Summary
• Evolutionary modeling is evolving slowly– We are a long way from being able to model
the complexities of known language evolution
• Nonetheless, computational approaches promise to lend insights into how complex social systems such as language change over time, and complement discoveries in historical linguistics
72
Final thoughts
• Language is central to what it means to be human.
• Language is used to:– Communicate information– Communicate requests– Persuade, cajole…– (In written form) record history– Deceive
• Other animals do some or most of these things (cf. Anindya Sinha’s work on bonnet macaques)
• But humans are better at all of these
73
Final thoughts
• So the scientific study of language ought to be more central than it is
• We need to learn much more about how language works– How humans evolved language– How languages changed over time– How humans learn language
• Computational linguistics can contribute to all of these questions.
74