CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN Massimo Poesio Uni Trento,...
-
Upload
jadyn-plumb -
Category
Documents
-
view
218 -
download
1
Transcript of CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN Massimo Poesio Uni Trento,...
CONCEPTUAL KNOWLEDGE: CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, EVIDENCE FROM CORPORA, THE MIND, AND THE BRAINTHE MIND, AND THE BRAIN
Massimo PoesioMassimo PoesioUni Trento, Center for Mind / Brain SciencesUni Trento, Center for Mind / Brain Sciences
Uni Essex, Language & ComputationUni Essex, Language & Computation
(joint work with A. Almuhareb, (joint work with A. Almuhareb, E. Barbu, M. Baroni, B. Murphy)E. Barbu, M. Baroni, B. Murphy)
MOTIVATIONSMOTIVATIONS
Research on conceptual knowledge is Research on conceptual knowledge is carried out in Computational Linguistics, carried out in Computational Linguistics, Neural Science, and PsychologyNeural Science, and Psychology
But there is limited interchange between But there is limited interchange between CL and the other disciplines studying CL and the other disciplines studying concepts concepts – Except indirectly through the use of WordNetExcept indirectly through the use of WordNet
This work: use data from Psychology and This work: use data from Psychology and Neural Science to evaluate (vector-space) Neural Science to evaluate (vector-space) models produced in CLmodels produced in CL
OUTLINEOUTLINE
Vector space representations Vector space representations A `semantic’ vector space modelA `semantic’ vector space model How to evaluate such models How to evaluate such models Attribute extraction and Feature Attribute extraction and Feature
normsnorms Category distinctions and Brain dataCategory distinctions and Brain data
CONCEPTUAL SEMANTICS IN CONCEPTUAL SEMANTICS IN VECTOR SPACEVECTOR SPACE
LEXICAL ACQUISITION IN LEXICAL ACQUISITION IN CORPUS / COMP LINGCORPUS / COMP LING
Vectorial representations of lexical Vectorial representations of lexical meaning derived from IRmeaning derived from IR
WORD-BASED vector models: WORD-BASED vector models: – vector dimensions are words vector dimensions are words – Schuetze 91, 98; HAL, LSA, Turney, Schuetze 91, 98; HAL, LSA, Turney,
RappRapp GRAMMATICAL RELATION models:GRAMMATICAL RELATION models:
– vector dimensions are pairs <Rel,W>vector dimensions are pairs <Rel,W>– Grefenstette 93, Lin 98, Curran&Moens, Grefenstette 93, Lin 98, Curran&Moens,
Pantel, Widdows, Pado & Lapata, …..Pantel, Widdows, Pado & Lapata, …..
FEATURES IN VECTOR SPACE FEATURES IN VECTOR SPACE MODELSMODELS
GRAMMATICAL RELATIONS
WORDS
STRENGHTS OF THIS STRENGHTS OF THIS APPROACH: CATEGORIZATIONAPPROACH: CATEGORIZATION
LIMITATIONS OF THIS WORKLIMITATIONS OF THIS WORK
Very simplistic view of concepts Very simplistic view of concepts – In fact, typically extract lexical In fact, typically extract lexical
representations for WORDS (non-representations for WORDS (non-disambiguated)disambiguated)
Limited evaluationLimited evaluation– Typical evaluation: judges’ opinions Typical evaluation: judges’ opinions
about correctness of distances / about correctness of distances / comparing with WordNetcomparing with WordNet
Most work not connected with work Most work not connected with work on concepts in Psychology / Neural on concepts in Psychology / Neural ScienceScience
OUR WORKOUR WORK
Acquire richer, more semantic-Acquire richer, more semantic-oriented concept descriptions by oriented concept descriptions by exploiting relation extraction exploiting relation extraction techniquestechniques
Develop task-based methods for Develop task-based methods for evaluating the resultsevaluating the results
Integrate results from corpora with Integrate results from corpora with results from psychology & neural results from psychology & neural sciencescience
THIS TALKTHIS TALK
Acquire richer, more semantic-Acquire richer, more semantic-oriented concept descriptions by oriented concept descriptions by exploiting relation extraction exploiting relation extraction techniquestechniques
Develop task-based methods for Develop task-based methods for evaluating the resultsevaluating the results
Integrate results from Integrate results from corpora with results from corpora with results from psychology & neural sciencepsychology & neural science
OUTLINEOUTLINE
Vector space representations Vector space representations A `semantic’ vector space modelA `semantic’ vector space model How to evaluate such models How to evaluate such models Attribute extraction and Feature Attribute extraction and Feature
normsnorms Category distinctions and Brain dataCategory distinctions and Brain data
MORE ADVANCED THEORIES MORE ADVANCED THEORIES OF CONCEPTSOF CONCEPTS
In Linguistics:In Linguistics:– Pustejovsky Pustejovsky
In AI:In AI:– Description LogicsDescription Logics– Formal ontologiesFormal ontologies
In Psychology:In Psychology:– Theory Theory (Murphy, 2002)Theory Theory (Murphy, 2002)– FUSS (Vigliocco Vinson et al)FUSS (Vigliocco Vinson et al)
SEMANTIC CONCEPT DESCRIPTIONSSEMANTIC CONCEPT DESCRIPTIONSPUSTEJOVSKY (1991, 1995)PUSTEJOVSKY (1991, 1995)
Lexical entries have a QUALIA STRUCTURE Lexical entries have a QUALIA STRUCTURE consisting of four ‘roles’consisting of four ‘roles’– FORMAL role: what type of object it is (shape, FORMAL role: what type of object it is (shape,
color, ….)color, ….)– CONSTITUTIVE role: what it consists of (parts, CONSTITUTIVE role: what it consists of (parts,
stuff, etc.)stuff, etc.) E.g., for books, chapters, index, paper ….E.g., for books, chapters, index, paper ….
– TELIC role: what is the purpose of the object TELIC role: what is the purpose of the object (e.g., for books, READING)(e.g., for books, READING)
– AGENTIVE role: how the object was created AGENTIVE role: how the object was created (e.g., for books, WRITING)(e.g., for books, WRITING)
BEYOND BUNDLES OF ATTRIBUTES: BEYOND BUNDLES OF ATTRIBUTES: DESCRIPTION LOGICS, THEORY THEORYDESCRIPTION LOGICS, THEORY THEORY
We know much more about concepts than We know much more about concepts than the fact that they have certain attributes:the fact that they have certain attributes:– We know that cars have 4 wheels whereas We know that cars have 4 wheels whereas
bicycles have 2bicycles have 2– We don’t just know that people have heads, We don’t just know that people have heads,
bodies and legs, but that heads are attached in bodies and legs, but that heads are attached in certain positions whereas legs are attached in certain positions whereas legs are attached in other onesother ones
– Facts of this type can be expressed even in the Facts of this type can be expressed even in the simplest concept description languages, those simplest concept description languages, those of description logicsof description logics
BEYOND SIMPLE RELATIONS: BEYOND SIMPLE RELATIONS: DESCRIPTION LOGICSDESCRIPTION LOGICS
Bear (and Animal ( 4 Paw) …)
Strawberry (and Fruit (fills Color red) … )
Female (and Human ( Male))
WORD SENSE WORD SENSE DISCRIMINATIONDISCRIMINATION
The senses of The senses of palmpalm in WordNet in WordNet1.1. the inner surface of the hand from the the inner surface of the hand from the
wrist to the base of the fingerswrist to the base of the fingers2.2. a linear unit based on the length or a linear unit based on the length or
width of the human handwidth of the human hand3.3. any plant of the family Palmae having any plant of the family Palmae having
an unbranched trunk crowned by large an unbranched trunk crowned by large pinnate or palmate leavespinnate or palmate leaves
4.4. an award for winning a championship an award for winning a championship or commemorating some other eventor commemorating some other event
CONCEPT ACQUISITION MEETS CONCEPT ACQUISITION MEETS RELATION EXTRACTIONRELATION EXTRACTION
– We developed methods to identify We developed methods to identify SEMANTICSEMANTIC properties of concepts properties of concepts (`Deep lexical relations’)(`Deep lexical relations’)ATTRIBUTES and their VALUES ATTRIBUTES and their VALUES
– Almuhareb & Poesio 2004, 2005Almuhareb & Poesio 2004, 2005Extracting QUALIA Extracting QUALIA
– Poesio & Almuhareb 2005Poesio & Almuhareb 2005Letting relations emerge from the data: Letting relations emerge from the data:
STRUDELSTRUDEL– Baroni et al, Cognitive Science to appearBaroni et al, Cognitive Science to appear
Extracting Wu & Barsalou –style relationsExtracting Wu & Barsalou –style relations– Poesio Barbu Giuliano & Romano, 2008Poesio Barbu Giuliano & Romano, 2008
We showed that for a variety of tasks such We showed that for a variety of tasks such conceptual descriptions are ‘better’ than conceptual descriptions are ‘better’ than word-based or grammatical function-based word-based or grammatical function-based descriptionsdescriptions
ALMUHAREB & POESIO 2005: ALMUHAREB & POESIO 2005: USING A PARSERUSING A PARSER
LOOKING ONLY FOR (POTENTIAL) ATTRIBUTES AND THEIR VALUES BETTER THAN USING ALL GRS
EVEN IF ATTRIBUTES OBTAINED USING TEXT PATTERNS (“THE X OF THE Y” )
ATTRIBUTES AND VALUES VS. ATTRIBUTES AND VALUES VS. ALL CORPUS FEATURESALL CORPUS FEATURES
DescriptionDescription Vector sizeVector size
500500 15221522 30443044 47534753 49694969
Values onlyValues only 64.86%64.86% 94.59%94.59% -- - - 94.59%94.59%
Attributes onlyAttributes only 97.30%97.30% 97.30%97.30% 97.3097.30%%
--
Attributes (1522) Attributes (1522) & Values (1522)& Values (1522)
-- -- 100%100% -- --
SUPERVISED EXTRACTION OF SUPERVISED EXTRACTION OF CONCEPT DESCRIPTIONS CONCEPT DESCRIPTIONS
Using a theory of attributes merging Using a theory of attributes merging ideas from Pustejovsky and Guarino ideas from Pustejovsky and Guarino (Poesio and Almuhareb, 2005)(Poesio and Almuhareb, 2005)
Using Wu and Barsalou’s theory of Using Wu and Barsalou’s theory of attributes (Poesio Barbu Romano & attributes (Poesio Barbu Romano & Giuliano, 2008)Giuliano, 2008)
SUPERVISED EXTRACTION OF SUPERVISED EXTRACTION OF CONCEPT DESCRIPTIONS CONCEPT DESCRIPTIONS
Using a theory of attributes merging Using a theory of attributes merging ideas from Pustejovsky and Guarino ideas from Pustejovsky and Guarino (Poesio and Almuhareb, 2005)(Poesio and Almuhareb, 2005)
Using Wu and Barsalou’s theory of Using Wu and Barsalou’s theory of attributes (Poesio Barbu Romano & attributes (Poesio Barbu Romano & Giuliano, 2008)Giuliano, 2008)
THE CLASSIFICATION SCHEME FOR THE CLASSIFICATION SCHEME FOR ATTRIBUTES OF POESIO & ALMUHAREB ATTRIBUTES OF POESIO & ALMUHAREB
20052005 PART PART
– (cfr. Guarino’s non-relational attributes, Pustejovsky’s (cfr. Guarino’s non-relational attributes, Pustejovsky’s constitutive roles)constitutive roles)
RELATED OBJECT RELATED OBJECT – Non-relational attributes other than parts, relational Non-relational attributes other than parts, relational
rolesroles QUALITYQUALITY
– Guarino’s qualities, Pustejovsky’s formal rolesGuarino’s qualities, Pustejovsky’s formal roles ACTIVITY ACTIVITY
– Pustejosvky’s telic and agentive rolesPustejosvky’s telic and agentive roles RELATED AGENTRELATED AGENT NOT AN ATTRIBUTE (= everything elseNOT AN ATTRIBUTE (= everything else))
A SUPERVISED FEATURE A SUPERVISED FEATURE CLASSIFIERCLASSIFIER
We developed a supervised feature We developed a supervised feature classifier that relies on 4 types of classifier that relies on 4 types of informationinformation– Morphological info (Dixon, 1991)Morphological info (Dixon, 1991)– Question patternsQuestion patterns– Features of featuresFeatures of features– Feature useFeature use
Some nouns used more commonly as features than Some nouns used more commonly as features than as concepts: i.e., “the F of the C is” more frequent as concepts: i.e., “the F of the C is” more frequent than “the * of the F is”than “the * of the F is”
(These last four methods all rely on info (These last four methods all rely on info extracted from the Web)extracted from the Web)
THE EXPERIMENTTHE EXPERIMENT We created a BALANCED DATASETWe created a BALANCED DATASET
– ~ ~ 400400 concepts concepts – representing representing all 21 WordNetall 21 WordNet classes, including both classes, including both
ABSTRACT and CONCRETE conceptsABSTRACT and CONCRETE concepts– balanced as to balanced as to ambiguityambiguity and and frequencyfrequency
We collected from the Web We collected from the Web 20,00020,000 candidate candidate features of these concepts using patternsfeatures of these concepts using patterns
We hand-classified We hand-classified 1,1551,155 candidate features candidate features We used these data to train We used these data to train
– A binary classifier (feature / non feature)A binary classifier (feature / non feature)– A 5-way classifierA 5-way classifier
OUTLINEOUTLINE
Vector space representations Vector space representations An example of `Semantic-based’ An example of `Semantic-based’
vector space modelvector space modelEvaluating such modelsEvaluating such models Attribute extraction and Feature Attribute extraction and Feature
normsnorms Category distinctions and Brain dataCategory distinctions and Brain data
EVALUATIONEVALUATION
– Qualitative:Qualitative:Visual inspectionVisual inspectionAsk subjects to assess correctness of the Ask subjects to assess correctness of the
classification of the attributesclassification of the attributes
– Quantitative:Quantitative:Use conceptual descriptions for CLUSTERING Use conceptual descriptions for CLUSTERING
(CATEGORIZATION)(CATEGORIZATION)
VISUAL EVALUATION: VISUAL EVALUATION: TOP 400 FEATURES OF DEER TOP 400 FEATURES OF DEER
ACCORDING TO OUR CLASSIFIERACCORDING TO OUR CLASSIFIER
ClassClass AttributesAttributes
Parts & Parts & Related Related ObjectsObjects
antlers, leg, carcass, head, eyes, skin, body, blood, track, neck, antlers, leg, carcass, head, eyes, skin, body, blood, track, neck, horns, flesh, meat, legs, hide, loin, chest, throat, tongue, heart, horns, flesh, meat, legs, hide, loin, chest, throat, tongue, heart, horn, coat, trail, tail, bones, ears, scent, home, coverts, nose, horn, coat, trail, tail, bones, ears, scent, home, coverts, nose, feet, shoulder, stomach, foot, sight, rack, skull, hair, intestines, feet, shoulder, stomach, foot, sight, rack, skull, hair, intestines, necks, line, brain, belly, tendons, step, heads, mind, entrails, necks, line, brain, belly, tendons, step, heads, mind, entrails, skins, hooves, cells, bell, cavity, skins, hooves, cells, bell, cavity, picturepicture, testicles, , testicles, photophoto, , forehead, genitals, knees, innards, rump, butt, fur, face, shank, forehead, genitals, knees, innards, rump, butt, fur, face, shank, brains, brains, imageimage, ear, , ear, statuestatue, path, corpse, jaw, bladder, muzzle, , path, corpse, jaw, bladder, muzzle, calf, hoofs, abdomen, calf, hoofs, abdomen, hillhill, quarters, shit, senses, paths, wound, , quarters, shit, senses, paths, wound, stream, feces, varieties, protector, hoof, stream, feces, varieties, protector, hoof, picspics, nostrils, portrait, , nostrils, portrait, liver, flanks, pen, forest, vitals, side, hips, garden, food, muscle, liver, flanks, pen, forest, vitals, side, hips, garden, food, muscle, muscles, guts, droppings, bodies, veil, footprints, wounds, muscles, guts, droppings, bodies, veil, footprints, wounds, hearts, homes, teeth, underside, breast, turn, haunches, hearts, homes, teeth, underside, breast, turn, haunches, forelegs, brow, hip, figure, torso, village, spring, chin, tails, forelegs, brow, hip, figure, torso, village, spring, chin, tails, organs, enclosure, lips, hindquarters, valley, cave, flank, figures, organs, enclosure, lips, hindquarters, valley, cave, flank, figures, tissues, spot, insides, dick, backs, skeleton, barktissues, spot, insides, dick, backs, skeleton, bark
VISUAL EVALUATION: VISUAL EVALUATION: QUALITIESQUALITIES
ClassClass AttributesAttributes
QualityQuality
deathdeath, age, beauty, sex, form, sense, reaction, cry, terror, curiosity, , age, beauty, sex, form, sense, reaction, cry, terror, curiosity, health, fleetness, appetite, health, fleetness, appetite, survivalsurvival, condition, thirst, life, ways, , condition, thirst, life, ways, swiftness, plight, whereabouts, swiftness, plight, whereabouts, fatefate, grace, gender, , grace, gender, sicknesssickness, , need, perspective, need, perspective, slaughterslaughter, , capturecapture, modesty, ecology, , modesty, ecology, preservationpreservation, detriment, agility, heartbeat, greed, gentleness, , detriment, agility, heartbeat, greed, gentleness, behaviour, behavior, aggressiveness, screams, favor, predicament, behaviour, behavior, aggressiveness, screams, favor, predicament, genetics, honours, elegance, propensity, reactions, harvest, genetics, honours, elegance, propensity, reactions, harvest, rescuerescue, curse, mercy, gaze, sustainability, intelligence, lives, , curse, mercy, gaze, sustainability, intelligence, lives, thoughtsthoughts
QUANTITATIVE EVALUATIONQUANTITATIVE EVALUATION ATTRIBUTES ATTRIBUTES
– PROBLEM: can’t compare against WordNetPROBLEM: can’t compare against WordNet– Precision / recall against hand-annotated Precision / recall against hand-annotated
datasetsdatasets– Human judges (ourselves):Human judges (ourselves):
We used the classifiers to classify the top 20 features We used the classifiers to classify the top 20 features of 21 randomly chosen conceptsof 21 randomly chosen concepts
We separately evaluated the resultsWe separately evaluated the results
CATEGORIES:CATEGORIES:– Clustering of the balanced datasetClustering of the balanced dataset– PROBLEM: PROBLEM: The WordNet category structure is The WordNet category structure is
highly subjectivehighly subjective
ATTRIBUTE CLASSIFICATIONATTRIBUTE CLASSIFICATION
DescriptionDescription Cross-ValidationCross-Validation Human Judge Human Judge (AA)(AA)
Human Judge Human Judge (MP)(MP)
Correctly Correctly ClassifiedClassified
Instances Instances (Accuracy)(Accuracy)
928 (out of 1155)928 (out of 1155)80.35%80.35%
244 (out of 365)244 (out of 365)66.85%66.85%
182 (out of 260)182 (out of 260)70.00%70.00%
MeasureMeasure PP RR FF PP RR FF PP RR FF
ActivityActivity 0.8220.822 0.8780.878 0.8490.849 0.8330.833 0.5930.593 0.6930.693 0.8750.875 0.6220.622 0.7270.727
Part/Related-Part/Related-ObjectObject
0.8420.842 0.8820.882 0.8620.862 0.8320.832 0.7120.712 0.7670.767 0.7930.793 0.8020.802 0.7980.798
QualityQuality 0.7990.799 0.8210.821 0.8100.810 0.7670.767 0.6290.629 0.6910.691 0.8640.864 0.6460.646 0.7390.739
Related-AgentRelated-Agent 0.9300.930 0.9700.970 0.9500.950 0.8150.815 0.9170.917 0.8630.863 0.8260.826 0.8260.826 0.8260.826
Not-AttributeNot-Attribute 0.6020.602 0.4870.487 0.5380.538 0.3390.339 0.6270.627 0.4400.440 0.2970.297 0.6330.633 0.4040.404
CLUSTERING WITH 2-WAY CLUSTERING WITH 2-WAY CLASSIFIERCLASSIFIER
AllAllCandidate Candidate
AttributesAttributesHeuristic filteringHeuristic filtering Filtering by Filtering by
classificationclassification
PurityPurity 0.6570.657 0.6720.672 0.6930.693
EntropyEntropy 0.3350.335 0.319 0.319 0.302 0.302
Vector sizeVector size 24,17824,178 4,2964,296 3,8243,824
Clustered Clustered ConceptsConcepts 402402 402402 401401
CLUSTERING: ERROR ANALYSISCLUSTERING: ERROR ANALYSIS
ANIMAL bear, bull, camel, cat, cow, deer, dog, elephant, horse, kitten, lion, monkey, puppy, rat, sheep, tiger, turtle
CLUSTERING: ERROR ANALYSISCLUSTERING: ERROR ANALYSIS
ANIMAL bear, bull, camel, cat, cow, deer, dog, elephant, horse, kitten, lion, monkey, puppy, rat, sheep, tiger, turtle
EDIBLE FRUIT
apple, banana, berry, cherry, fig, grape, kiwi, lemon, lime, mango, melon, olive, orange, peach, pear, pineapple, strawberry, watermelon, (pistachio, oyster)
CLUSTERING: ERROR ANALYSISCLUSTERING: ERROR ANALYSISANIMAL bear, bull, camel, cat, cow, deer, dog, elephant,
horse, kitten, lion, monkey, puppy, rat, sheep, tiger, turtle
EDIBLE FRUIT
apple, banana, berry, cherry, fig, grape, kiwi, lemon, lime, mango, melon, olive, orange, peach, pear, pineapple, strawberry, watermelon, (pistachio, oyster)
ILLNESS acne, anthrax, arthritis, asthma, cancer, cholera, cirrhosis, diabetes, eczema, flu, glaucoma, hepatitis, leukemia, malnutrition, meningitis, plague, rheumatism, smallpox, (superego, lumbago, neuralgia, sciatica, gestation, menopause, quaternary, pain)
CLUSTERING: ERROR ANALYSISCLUSTERING: ERROR ANALYSISANIMAL bear, bull, camel, cat, cow, deer, dog, elephant,
horse, kitten, lion, monkey, puppy, rat, sheep, tiger, turtle
EDIBLE FRUIT
apple, banana, berry, cherry, fig, grape, kiwi, lemon, lime, mango, melon, olive, orange, peach, pear, pineapple, strawberry, watermelon, (pistachio, oyster)
ILLNESS acne, anthrax, arthritis, asthma, cancer, cholera, cirrhosis, diabetes, eczema, flu, glaucoma, hepatitis, leukemia, malnutrition, meningitis, plague, rheumatism, smallpox, (superego, lumbago, neuralgia, sciatica, gestation, menopause, quaternary, pain)
IN WORDNET: PAIN
LIMITS OF THIS TYPE OF LIMITS OF THIS TYPE OF EVALUATIONEVALUATION
No way of telling how complete / No way of telling how complete / accurate are our concept accurate are our concept descriptionsdescriptions– Both in terms of relations and in terms Both in terms of relations and in terms
of their relative importance of their relative importance No way of telling whether the No way of telling whether the
category distinctions we get from category distinctions we get from WordNet are empirically foundedWordNet are empirically founded
BEYOND JUDGES / EVALUATION BEYOND JUDGES / EVALUATION AGAINST WORDNETAGAINST WORDNET
Task-based evaluationTask-based evaluation
Evidence from other areas of Evidence from other areas of cognitive sciencecognitive science
(ESSLLI 2008 Workshop - Baroni / (ESSLLI 2008 Workshop - Baroni / Evert / Lenci )Evert / Lenci )
TASK-BASED (BLACK-BOX) TASK-BASED (BLACK-BOX) EVALUATIONEVALUATION
Tasks requiring lexical knowledge:Tasks requiring lexical knowledge:– Lexical tests:Lexical tests:
TOEFL test (Rapp 2001, Turney 2005)TOEFL test (Rapp 2001, Turney 2005)
– NLP tasks:NLP tasks:Eg, anaphora resolution (Poesio et al 2004)Eg, anaphora resolution (Poesio et al 2004)
– Actual applicationsActual applicationsE.g., language models (Mitchell & Lapata E.g., language models (Mitchell & Lapata
ACL 2009, Lapata invited talk)ACL 2009, Lapata invited talk)
EVIDENCE FROM OTHER AREAS EVIDENCE FROM OTHER AREAS OF COGNITIVE SCIENCEOF COGNITIVE SCIENCE
Attributes: evidence from psychologyAttributes: evidence from psychology– Association lists (priming)Association lists (priming)
E.g., use results of association tests to evaluate E.g., use results of association tests to evaluate proximity (Lund et al, 1995; Pado and Lapata, 2008)proximity (Lund et al, 1995; Pado and Lapata, 2008)
Comparison against feature norms: Schulte im Walde, Comparison against feature norms: Schulte im Walde, 2008)2008)
– Feature normsFeature norms
Category distinctions: evidence from Category distinctions: evidence from neural scienceneural science
OUTLINEOUTLINE
Vector space representations Vector space representations An example of `Semantic-based’ An example of `Semantic-based’
vector space modelvector space model How to evaluate such models How to evaluate such models Attribute extraction and Feature Attribute extraction and Feature
normsnorms Category distinctions and Brain dataCategory distinctions and Brain data
FEATURE-BASED REPRESENTATIONS IN FEATURE-BASED REPRESENTATIONS IN PSYCHOLOGYPSYCHOLOGY
Feature-based concept representations assumed Feature-based concept representations assumed by many cognitive psychology theories (Smith by many cognitive psychology theories (Smith and Medin, 1981, McRae et al, 1997)and Medin, 1981, McRae et al, 1997)
Underpin development of prototype theory (Rosch Underpin development of prototype theory (Rosch et al)et al)
Used, e.g., to account for semantic priming Used, e.g., to account for semantic priming (McRae et al, 1997; Plaut, 1995)(McRae et al, 1997; Plaut, 1995)
Underlie much work on category-specific defects Underlie much work on category-specific defects (Warrington and Shallice, 1984; Caramazza and (Warrington and Shallice, 1984; Caramazza and Shelton, 1998; Tyler et al, 2000; Vinson and Shelton, 1998; Tyler et al, 2000; Vinson and Vigliocco, 2004)Vigliocco, 2004)
FEATURE NORMSFEATURE NORMS
Subjects produce lists of features for Subjects produce lists of features for a concepta concept
Weighed by number of subjects that Weighed by number of subjects that produce themproduce them
Several existing (Rosch and Mervis, Several existing (Rosch and Mervis, Garrard et al, McRae et al, Vinson Garrard et al, McRae et al, Vinson and Vigliocco)and Vigliocco)
Substantial differences in collection Substantial differences in collection methodology and resultsmethodology and results
SPEAKER-GENERATED FEATURES SPEAKER-GENERATED FEATURES (VINSON AND VIGLIOCCO)(VINSON AND VIGLIOCCO)
COMPARING CORPUS FEATURES COMPARING CORPUS FEATURES WITH FEATURE NORMS WITH FEATURE NORMS
(Almuhareb et al 2005, Poesio et al 2007)(Almuhareb et al 2005, Poesio et al 2007)
35 concepts in common between the Almuhareb & 35 concepts in common between the Almuhareb & Poesio dataset and the dataset produced by Poesio dataset and the dataset produced by Vinson and Vigliocco (2002, 2003)Vinson and Vigliocco (2002, 2003)– ANIMALS: bear, camel, cat, cow, dog, elephant, horse, ANIMALS: bear, camel, cat, cow, dog, elephant, horse,
lion, mouse, sheep, tiger, zebralion, mouse, sheep, tiger, zebra– FRUIT: apple, banana, cherry, grape, lemon, orange, FRUIT: apple, banana, cherry, grape, lemon, orange,
peach, pear, pineapple, strawberry, watermelonpeach, pear, pineapple, strawberry, watermelon– VEHICLE: airplane, bicycle, boat, car, helicopter, VEHICLE: airplane, bicycle, boat, car, helicopter,
motorcycle, ship, truck, vanmotorcycle, ship, truck, van
We compared the features we obtained for these We compared the features we obtained for these concepts with the speaker-generated features concepts with the speaker-generated features collected by Vinson and Viglioccocollected by Vinson and Vigliocco
RESULTSRESULTS Best recall: ~ 52% (using all attributes and Best recall: ~ 52% (using all attributes and
values)values) Best precision: ~ 19%Best precision: ~ 19% But: high correlation But: high correlation (ro=.777) (ro=.777) between between
the distances between concept the distances between concept representations obtained from corpora, representations obtained from corpora, and the distances between the and the distances between the representations for the same concepts representations for the same concepts obtained from subjects (using the cosine obtained from subjects (using the cosine as a measure of similarity)as a measure of similarity)
DISCUSSIONDISCUSSION
– Substantial differences in features and Substantial differences in features and overlap, but correlation similaroverlap, but correlation similar
– Problems:Problems:Each feature norm slightly differentEach feature norm slightly differentThey have been normalized by hand: LOUD, They have been normalized by hand: LOUD,
NOISY, NOISE all mapped to LOUDNOISY, NOISE all mapped to LOUD
AN EXAMPLE: STRAWBERRYAN EXAMPLE: STRAWBERRY
Speaker-generated features:
Matching Features Collected Using Our Text Patterns – (with
frequency)
red (20) red (5), colour (5), color (1)
fruit (18) fruit (5)
sweet (13) sweetness (8)
has seeds (12) seeds (6), seed (2)
grows (10) growth (1), ripening (10)
small (6) size (19)
taste (6) taste (6), flavor (6), flavour (2)
food (5) nutrition (1)
from garden (5) cultivation (7), harvest (6), harvester (2)
juice (5) juice (10), juices (3)
dessert (3) sweetness (8)
eat (3) nutrition (1)
Problem: differences between Problem: differences between feature normsfeature norms
motorcyclemotorcycle– Vinson & Vigliocco:Vinson & Vigliocco:
wheel, motor, loud, vehicle, wheel, fast, handle, ride, wheel, motor, loud, vehicle, wheel, fast, handle, ride, transport, bike, human, danger, noise, seat, brake, drive, transport, bike, human, danger, noise, seat, brake, drive, fun, gas, machine, object, open, small, travel, windfun, gas, machine, object, open, small, travel, wind
– Garrard et al:Garrard et al: vehicle, wheel, fast, handlebar, light, seat, make a noise, vehicle, wheel, fast, handlebar, light, seat, make a noise,
tank, metal, unstable, tyre, coloured, sidecar, indicator, tank, metal, unstable, tyre, coloured, sidecar, indicator, pannier, pedal, speedometer, manoeuvrable, race, brakes, pannier, pedal, speedometer, manoeuvrable, race, brakes, stop, move, engine, petrol, economical, gearsstop, move, engine, petrol, economical, gears
– McRae et al:McRae et al: wheels, 2_wheels, dangerous, engine, fast, helmets, wheels, 2_wheels, dangerous, engine, fast, helmets,
Harley_Davidson, loud, 1_or_2_people, vehicle, leather, Harley_Davidson, loud, 1_or_2_people, vehicle, leather, transportation, 2_people, fun, Hell's_Angels, gasolinetransportation, 2_people, fun, Hell's_Angels, gasoline
– Mutual correlation of ranks ranges from 0.4 to 0.7Mutual correlation of ranks ranges from 0.4 to 0.7
DISCUSSIONDISCUSSION
– Preliminary conclusions: need to collect new Preliminary conclusions: need to collect new feature norms for CLfeature norms for CL
E.g., use similar techniques to collect attributes for E.g., use similar techniques to collect attributes for WordNetWordNet
See Kremer & Baroni 2008See Kremer & Baroni 2008
– For more work on using feature norms for For more work on using feature norms for conceptual acquisition, see conceptual acquisition, see
Schulte im Walde 2008Schulte im Walde 2008 Baroni et al to appearBaroni et al to appear
– For the correlation between feature norms and For the correlation between feature norms and information in WordNet (meronymy, isa, plus information in WordNet (meronymy, isa, plus info from glosses): Barbu & Poesio GWC 2008info from glosses): Barbu & Poesio GWC 2008
OUTLINEOUTLINE
Vector space representations Vector space representations An example of `Semantic-based’ An example of `Semantic-based’
vector space modelvector space model How to evaluate such models How to evaluate such models Attribute extraction and Feature Attribute extraction and Feature
normsnormsCategory distinctions and brain Category distinctions and brain
datadata
USING BRAIN DATA TO IDENTIFY USING BRAIN DATA TO IDENTIFY CATEGORY DISTINCTIONSCATEGORY DISTINCTIONS
Studies of brain-damaged patients have Studies of brain-damaged patients have been shown to provide useful insights in been shown to provide useful insights in the organization of conceptual knowledge the organization of conceptual knowledge in the brainin the brain– Warrington and Shallice 1984, Caramazza & Warrington and Shallice 1984, Caramazza &
Shilton 1998Shilton 1998 fMRI has been used to identify these fMRI has been used to identify these
distinctions in healthy patients as welldistinctions in healthy patients as well– E.g., Martin & ChaoE.g., Martin & Chao
See, e.g., Capitani et al 2003 for a surveySee, e.g., Capitani et al 2003 for a survey
CATEGORY DISTINCTIONS IN CATEGORY DISTINCTIONS IN THE BRAINTHE BRAIN
ANIMALSTOOLS
CORPUS DATA AND BRAIN CORPUS DATA AND BRAIN DATADATA
Can brain data (from healthy Can brain data (from healthy patients) be used to get an patients) be used to get an ‘objective’ picture of categorical ‘objective’ picture of categorical distinctions in the brain?distinctions in the brain?
Can our findings be useful to Can our findings be useful to understand better the neurological understand better the neurological results?results?
Ongoing project: using EEG and fMRI Ongoing project: using EEG and fMRI to identify such distinctionsto identify such distinctions
EEG Spectral Analysis of EEG Spectral Analysis of ConceptsConcepts
Participants presented Participants presented with aural or visual with aural or visual concept stimuli concept stimuli
EEG apparatus EEG apparatus records electrical records electrical activity on the scalpactivity on the scalp
Waveforms can be Waveforms can be reduced to frequency reduced to frequency componentscomponents
EEG vs. fMRIEEG vs. fMRI
EEG pros and consEEG pros and cons
Pros:Pros:– Lighter Lighter – CheaperCheaper– Better temporal resolution (ms)Better temporal resolution (ms)
Cons:Cons:– Coarser spatial resolution (cm)Coarser spatial resolution (cm)– Noisy (e.g., very sensitive to skull depth)Noisy (e.g., very sensitive to skull depth)
A CATEGORY DISTINCTION A CATEGORY DISTINCTION EXPERIMENT WITH EEGEXPERIMENT WITH EEG
Murphy Poesio Bovolo DalPonte & Murphy Poesio Bovolo DalPonte & Bruzzone, Cogsci 2008Bruzzone, Cogsci 2008
Seven Italian native speakersSeven Italian native speakers Image stimuli only:Image stimuli only:
– 30 tools30 tools– 30 animals30 animals
Each stimulus presented six timesEach stimulus presented six times Optimal time / frequency window identified Optimal time / frequency window identified
automatically automatically – 100-370ms100-370ms– 3-17 Hz3-17 Hz
Stimuli: Images from WebStimuli: Images from Web
Data analysisData analysis
PreprocessingPreprocessing– Artefact removalArtefact removal
Feature extraction Feature extraction – CSSD: a form of supervised component CSSD: a form of supervised component
analysisanalysis ClassificationClassification
– Using SVMsUsing SVMs
EEG SIGNALS: EEG SIGNALS: TIME-FREQUENCY (PER CHANNEL)TIME-FREQUENCY (PER CHANNEL)
Extraction of features (EEGOXELS) Extraction of features (EEGOXELS) from EEG datafrom EEG data
Data analysisData analysis
Classification System Schematic
Filte
r by T
ime
, F
req
an
d E
ele
ctr.
CS
SD
D
eco
mp
ositio
n
Ve
ctor T
ran
sform
Su
pV
ec M
ach
ine
var(“to
ol”),
var(“a
nim
al”)
64 channels preprocessed data
X channels filtered data* “Tool” component
“Animal” component
Feature vector
Answer
?
RESULTSRESULTS
Representation of categories in Representation of categories in CSSD spacesCSSD spaces
component analysis identifies 2-component analysis identifies 2-dimensional spacesdimensional spaces
Analysis of these spaces may provide Analysis of these spaces may provide useful data against which to compare useful data against which to compare our corpus modelsour corpus models
CSSD-derived conceptual spacesCSSD-derived conceptual spaces
BRAIN DATA AND CORPUS BRAIN DATA AND CORPUS DATADATA
What is the relation between the What is the relation between the conceptual spaces induced from conceptual spaces induced from corpora with the conceptual spaces corpora with the conceptual spaces elicited using EEG? elicited using EEG?
PREDICTING BRAIN (FMRI) ACTIVATION PREDICTING BRAIN (FMRI) ACTIVATION USING CONCEPT DESCRIPTIONSUSING CONCEPT DESCRIPTIONS
T. Mitchell, S. Shinkareva, A. Carlson, K. Chang, V. Malave, R. Mason and M. Just. 2008. Predicting human brain activity associated with the meanings of nouns. Science 320, 1191–1195
MITCHELL ET AL 2008: MITCHELL ET AL 2008: METHODSMETHODS
Record fMRI activation for 60 nominal conceptsRecord fMRI activation for 60 nominal concepts– And extract 200 ‘best’ features, or VOXELsAnd extract 200 ‘best’ features, or VOXELs
Build conceptual descriptions for these concepts Build conceptual descriptions for these concepts from corpora (the Web)from corpora (the Web)– 25 features for each concept 25 features for each concept – 25 verbs expressing typical properties of living things / 25 verbs expressing typical properties of living things /
toolstools– Collect strength of association between these features Collect strength of association between these features
and each conceptand each concept Learn association between each voxel and the 25 Learn association between each voxel and the 25
verbal features using 58 conceptsverbal features using 58 concepts Use learned model to predict activation of 2 held-Use learned model to predict activation of 2 held-
out data (compare using Euclidean distance)out data (compare using Euclidean distance)– Accuracy: 77%Accuracy: 77%
MITCHELL ET AL 2008MITCHELL ET AL 2008
MITCHELL ET AL 2008: MITCHELL ET AL 2008: VERB FEATURESVERB FEATURES
MITCHELL ET AL: MITCHELL ET AL: LEARNING ASSOCIATIONSLEARNING ASSOCIATIONS
OUR EXPERIMENTS OUR EXPERIMENTS Replicate the Mitchell et al study using Replicate the Mitchell et al study using
EEG data instead of fMRIEEG data instead of fMRI– Different feature selection mechanismsDifferent feature selection mechanisms
Compare different methods for building Compare different methods for building concept descriptionsconcept descriptions– In addition to hand-picked, also a variety of In addition to hand-picked, also a variety of
standard corpus modelsstandard corpus models For ItalianFor Italian B. Murphy, M. Baroni, M. Poesio, B. Murphy, M. Baroni, M. Poesio, EEG
responds to conceptual stimuli and corpus semantics, EMNLP 2009
RESULTS USING THE HAND-RESULTS USING THE HAND-PICKED FEATURESPICKED FEATURES
RESULTS USING AUTOMATICALLY RESULTS USING AUTOMATICALLY SELECTED FEATURESSELECTED FEATURES
MITCHELL ET AL
AA-MP
RECAPRECAP
We need to relate the evidence from We need to relate the evidence from corpora with evidence about concepts corpora with evidence about concepts coming from empirical work in the coming from empirical work in the neuroscience and psychologyneuroscience and psychology
Feature norms databases could be used Feature norms databases could be used to evaluate attribute extractionto evaluate attribute extraction– But: need to find better ways of collecting But: need to find better ways of collecting
themthem Brain data may give us information about Brain data may give us information about
the ‘real’ conceptual categoriesthe ‘real’ conceptual categories– Results still preliminary Results still preliminary
COLLABORATORSCOLLABORATORS
ABDULRAHMAN ALMUHAREB(Essex PhD 2006 Now at KACST, Saudi Arabia)
HEBA LAKANY(formerly Essex, now Strathclyde)
BRIAN MURPHY(Trento)
EDUARD BARBU(Trento PhD forthc)
MARCO BARONI(Trento)
THANKSTHANKS
To the audience …..
And to Galja, Ruslan & the other organizers for yet another splendid RANLP!