Distributional models Katrin Erk. You can get an idea of what a word means from observing it in...

Distributional models

Katrin Erk

You can get an idea of what a word means from observing it

in context• He filled the wampimuk, passed it around,

and we all drank some• We found a little hairy wampimuk sleeping

behind a tree. • (examples by Marco Baroni)

• Distributional modeling:• Represent the meaning of a word through the

contexts in which it is observed• Similar words appear in similar contexts

2

What words can appear in these contexts?

Word 1: drown, bathroom, shower, fill, fall, lie, electrocute, toilet, whirlpool, iron, gin

Word 2: eat, fall, pick, slice, peel, tree, throw, fruit, pie, bite, crab, grate

Word 3: advocate, overthrow, establish, citizen, ideal, representative, dictatorship, campaign, bastion, freedom

Word 4: spend, enjoy, remember, last, pass, end, die, happen, brighten, relive

What words can appear in these contexts?


Word 2: eat, fall, pick, slice, peel, tree, throw, fruit, pie, bite, crab, grate



bathtub apple

democracyday

What can you say about word number 5?


Word 2: eat, fall, ripe, slice, peel, tree, throw, fruit, pie, bite, crab, grate



bathtub apple

democracy

day

Word 5: eat, paint, peel, apple, fruit, juice, lemon, blue, grow

What can you say about word number 5?


Word 2: eat, fall, ripe, slice, peel, tree, throw, fruit, pie, bite, crab, grate



bathtub apple

democracy

day

Word 5: eat, paint, peel, apple, fruit, juice, lemon, blue, grow

orange

Describing meaning through context

• Similar words appear in similar contexts• apple, orange

• Measure similarity in meaning as similarity in contexts

• Caveat: If a word has multiple meanings, like “orange”, it will appear in a mixture of contexts

• How to describe the contexts of a word?• Count other words nearby

Counting context words for “apple”

• They picked up red apples that had fallen to the ground

• Eating apples is healthy

Word count, 3-word context window, lemmatized

• She ate a red apple

• Pick an apple.

a be eat fall have

healthy

pick

red that

up

2 1 2 1 1 1 2 2 1 1

How can we compare two context word counts?

eat fall ripe slice

peel

tree

throw

fruit

pie bite

crab

794 244 47 221 208 160

145 156 109 104 88

Count how often “apple” occurs close to other words in a large text collection (corpus):

Interpret counts as coordinates:fall

eat

appleEvery context wordbecomes a dimension.


eat fall ripe slice

peel

tree

throw

fruit

pie bite

crab

794 244 47 221 208 160

145 156 109 104 88

Count how often “apple” occurs close to other words in a large text collection (corpus):

Do the same for “orange”:

eat fall ripe slice

peel

tree

throw

fruit

pie bite

crab

265 22 25 62 220 64 74 111 4 4 8


eat fall ripe slice

peel

tree

throw

fruit

pie bite

crab

794 244 47 221 208 160

145 156 109 104 88

Then visualize both count tables as vectors in the same space:

eat fall ripe slice

peel

tree

throw

fruit

pie bite

crab

265 22 25 62 220 64 74 111 4 4 8

fall

eat

apple

orange

Similarity betweentwo words as proximity in space

What do we mean by “similarity” of vectors?

Euclidean distance (a dissimilarity measure!):

orange

apple

Problem with Euclidean distance: very sensitive to word frequency!

Braeburn

apple

What do we mean by “similarity” of vectors?

Cosine similarity:

orange

apple

Use angle between vectorsinstead of point distanceto get around word frequency issues

Using distributional models

• Predicting word similarity (WordSim353):• doctor nurse 7.00• professor doctor 6.62• student professor 6.81• smart student 4.62• smart stupid 5.81

• Doing the TOEFL test:• provisions

1. stipulations2. interrelations3. jurisdictions4. interpretations

Similar:• synonyms• antonyms• topically related words

Using distributional models

• Predicting • priming effects• free associations

• Finding (near-)synonyms: automatically building a thesaurus

• Related: use distributional similarity of documents (containing similar words) in Information Retrieval

Corpora in which to count words

• Corpus = text collection• What works best for computing

a distributional model?• “Moby Dick”• 2 years of the Wall Street Journal • A collection of dating ads• A collection of webpage texts

Corpora in which to count words

• Need to be available electronically• Best possible match for today’s English

language in general• Mixture of genres• Mixture of authors• Spoken and written

• Larger is better

Corpora in which to count (English) words

• Brown Corpus• 1 million words• Balanced corpus, mixture of genres

• British National Corpus• 100 million words• Balanced corpus, mixture of genres, spoken

and written

Corpora in which to count (English) words

• English Gigaword corpus• 1 billion words• Short news articles

• Wikipedia dump• 2 billion words

• UKWaC (UK web as corpus)• 2 billion words• Collection of webpages ending in .uk

What can we do with our word counts?

• … blackboard

Background in philosophy of language: Wittgenstein,

“meaning” as “use”

• Man kan fur eine große Klasse von Fallen der Benutzung des Wortes ‘Bedeutung’ – wenn auch nicht fur alle Falle seiner Benutzung – dieses Wort so erklaren: Die Bedeutung eines Wortes ist sein Gebrauch in der Sprache. -- Wittgenstein, Philosophical Investigations

• For a large class of cases – though not for all – in which we employ the word ‘meaning’ it can be explained thus: the meaning of a word is its use in the language. (translation: Anscombe/Stokhof)

Background in linguistics: Harris and Firth

• Zellig Harris (1957): Classify linguistic units by observing the contexts they occur in• phonemes• morphemes• phrase types: noun phrase, verb phrase…• Not specifically about semantics

• John Firth (1957): “collocations”• identify senses of a word by looking at groups of

contexts in which it appears• “You shall know a word by the company it keeps”

Background in psychology

• Landauer/Dumais 1998: “A solution to Plato’s problem”

• How come you know so many words?• “A typical American seventh grader knows the

meaning of 10-15 words today that she did not know yesterday. She must have acquired most of them as a result of reading because (a) the majority of English words are used only in print, (b) she already knew well almost all the words she would have encountered in speech, and (c) she learned less than one word by direct instruction.”


• Many phenomena to do with word meaning can be simulated with distributional models• Word similarity ratings (WordSim353):• doctor nurse 7.00• professor doctor 6.62• student professor 6.81• smart student 4.62• smart stupid 5.81• How would you simulate this with a distributional

model?


• Many phenomena to do with word meaning can be simulated with distributional models• Priming• Hearing one word makes you react faster to a

related word• Hodgson data:• election – vote• dove – peace• vase – flower• NOT: election – peace

• How would you simulate that with a distributional model?


• Are our mental representations of words, of concepts distributional?

• Do they represent the contexts in which a word has been encountered?

• Does the meaning of a word depend on the contexts in which it is used?


• Concepts as distributional:• Yes (Landauer and others):• How else would we learn so many words?• Text as one of the main ways in which we encounter

words• No (Barsalou and others):• Perception is central to how we represent concepts

• Distributional and perceptual (Andrews/Vigliocco/Vinson and others)• Both types of information seems to be relevant• Simulations: Combining both makes for a better model

Distributional models Katrin Erk. You can get an idea of what a word means from observing it in...

Documents

Transcript of Distributional models Katrin Erk. You can get an idea of what a word means from observing it in...