Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0...
Transcript of Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0...
![Page 1: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/1.jpg)
0
![Page 2: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/2.jpg)
WelcometotheOPERA
1
![Page 3: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/3.jpg)
AIDAin2019…achallenge
Nomoretrainingdata,onlyexamplesthatillustratetheevaluationIncreasinglydata-intensiveneurallearnersWhatdowedo???
2
![Page 4: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/4.jpg)
Arangeofresponses…
• Justmakemachinelearningwork!
• Learning,augmentedwithexternaldata
• Half-half
• Include(some)learningbutonlyifit’seasy
• Forgetmachinelearning!3
5
4
3
2
1
![Page 5: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/5.jpg)
Overview
1. Systemoverview2. TA1Englishentityandrelationprocessing3. TA1Rus/Ukrentityandeventprocessing4. TA1/2KBconstructionandvalidation5. TA3Hypotheses
4
![Page 6: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/6.jpg)
SYSTEMOVERVIEWZaidSheikh,AnkitDangi,EduardHovy
5
![Page 7: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/7.jpg)
OPERAarchitecture
TA1extractionengines
TA2corefengine
TA3hypothesisformation
Ontology CSR
PowerLoomdatabase
textspeechimagesvideo
6
![Page 8: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/8.jpg)
OPERAframework
Coref
Mini-KBcreation/AIFvalidation
TA1Mini-KB
Input
Mini-KBcreation/AIFvalidation
Hypothesisformation
BeliefGraphconstruction
TA2Mini-KB
Queries
Englishentities
Mini-KBcreation/AIFvalidation
ImagesSpeech
Rus/Ukrentities
Text
Englishevents
Rus/Ukrevents
7
![Page 9: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/9.jpg)
TA1frameworkInput
Mini-KBcreation/AIFvalidation
Domainfilter&languagedetection
Ru/Ukpipeline
MT:Ru/Uk–>
Eng
Ru/UkEntityandEventdetection
EventframeassemblyII
CSRCombination
Imagepipeline
Entitydetection
PersonandGeoID
Englishentity
detection
Englishevent
detection
Englishentitylinking
Engentityrelations
EventframeassemblyI
Englishpipeline
Englishargumentdetection
Englishcoref
Speechpipeline
8
![Page 10: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/10.jpg)
OPERATA2+TA3framework
Coref
Mini-KBcreation/AIFvalidation
TA1Mini-KB
Mini-KBcreation/AIFvalidation
Hypothesisformation
BeliefGraphconstruction
TA2Mini-KB
Queryinput
9
![Page 11: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/11.jpg)
KBsandnotations
• AllresultswritteninOPERA-internalframenotation(json)andstoredinCSR(BlazeGraph)
• Input/outputconvertersfrom/toAIDAAIF
• TwoseparateKBcreationandvalidationprocedures,fortwoparallelKBs(givesinsurance,coverage,andbackup):– Chalupsky:usesPowerLoomandChameleonreasoner
– Chaudhary:usesspecializedrules10
5
4
![Page 12: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/12.jpg)
Internaldryruns
• Internaldryrunmini-evalsusingthepracticeannotationsreleasedbyLDC
• Evaluatedresultsmanually
• Resultslookpromising,BUT…hardtocalculateP/R/F1forvariouspartsoftheTA1pipelinebecauseLDCdoesnotlabelallmentionsofevents,relationsandentities,justthe“salient”or“informative”ones(sowehavetojudgethemourselves…laboriousandnotguaranteed)
11
![Page 13: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/13.jpg)
TA1TEXT:ENGLISHENTITIESANDRELATIONS
XiangKong,XianyangChen,EduardHovy
12
![Page 14: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/14.jpg)
OPERATA1framework
Input
Mini-KBcreation/AIFvalidation
Domainfilter&languagedetection
Ru/Ukpipeline
MT:Ru/Uk–>
Eng
Ru/UkEntityandEventdetection
EventframeassemblyII
CSRCombination
Imagepipeline
Entitydetection
PersonandGeoID
Englishentity
detection
Englishevent
detection
Englishentitylinking
Engentityrelations
EventframeassemblyI
Englishpipeline
Englishargumentdetection
Englishcoref
Speechpipeline
13
![Page 15: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/15.jpg)
1.Entitydetection:Type-basedNERdata
• Multi-levellearning:– Trainseparatedetectorsfortype,subtype,andsubsubtype-leveltypeclassification
– Addressesdataimbalance– Mayintroducelayer-inconsistenttypes!
• Type-levelfromLDContology:– Trainingdata:KBPNERdataandasmallamountofself-annotateddata
• Sub(sub)type-level:– Trainingdata:YAGOknowledgebase(350k+entitytypes)obtainedfromHengJi—thanks!
14
2
3
![Page 16: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/16.jpg)
2.Entitylinking
• Task:GivenNERoutputmentions,linkthemtothereferenceKB
• Challenges:Over-largeKB,noisyGeonames– PreprocessKB:Removeduplicatedandunimportantentries(i.e.,notlocatedinRussiaorUkraine,ornoWikipediapage)
• Approach,givenanentity:– UseLucenetofindallcandidatesinKB– Filterspuriousmatches– Buildconnectednessgraph,withPageRanklinkstrengthscores
– Prune(densify)graphtodisambiguateentity15
2
![Page 17: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/17.jpg)
3.Entityrelationextraction
• Task:Extractentitypropertiesandeventparticipants• Four-stepapproach:
1. BERTwordembeddingsforfeatures2. Convolution:extractandmergealllocalfeaturesforasentence3. Piecewisemaxpooling:splitinputintothreesegments(byposition)
andreturnmaxvalueineachsegment,for2entities+1relation4. Softmaxclassifiertocomputeconfidenceofeachrelation
16
1
![Page 18: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/18.jpg)
Englishentity/relationdiscussion
• Challengesandproblems– Subsubtypeissuperfine-grained;ourNERengineisstillnotrobustenough
– Wereturnbothtypeandsubsubtypelabels,butintheevalNISTwilljudgeonlyoneofthem
• Mostlylearned,butsomemanualassistance
17
321 2
![Page 19: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/19.jpg)
TA1RUSSIANANDUKRAINIANMariiaRyskina,Yu-HsuanWang,AnatoleGershman
18
![Page 20: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/20.jpg)
OPERATA1framework
Input
Mini-KBcreation/AIFvalidation
Domainfilter&languagedetection
Ru/Ukpipeline
MT:Ru/Uk–>
Eng
Ru/UkEntityandEventdetection
EventframeassemblyII
CSRCombination
Imagepipeline
Entitydetection
PersonandGeoID
Englishentity
detection
Englishevent
detection
Englishentitylinking
Engentityrelations
EventframeassemblyI
Englishpipeline
Englishargumentdetection
Englishcoref
Speechpipeline
19
![Page 21: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/21.jpg)
Goalsandchallenges
• Goal:ExtractentityandeventmentionsfromRussianandUkrainiantext,andbuildframes
• Challenges:– Lackofpretrainedoff-the-shelfextractors– Lackofannotateddatatotrainsystems– Highlyspecificontology
• Twopipelines:1. RusandUkrsourcetext2. MTintoEnglish
20
![Page 22: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/22.jpg)
ExampleinputandoutputInput:Про-российскиесепаратистыатаковалиКраматорскийаэропорт.Translation:Pro-RussianseparatistsattackedKramatorskairport.Output:mn0:eventConflict.Attack, text:атаковали
Attacker:mn1,Target:mn3mn5:relationGeneralAffiliation.MemberOriginReligionEthnicity
Person:mn1,EntityOrFiller:mn2, text:Про-российскиесепаратистыmn6:relationPhysical.LocatedNear, text:Краматорскийаэропорт
EntityOrFiller:mn3,Place:mn4
mn1:entityORG, text:Про-российскиесепаратистыmn2:entityGPE.Country.Country, text:Про-российскиеmn3:entityFAC.Installation.Airport, text:Краматорскийаэропортmn4:entityGPE.UrbanArea.City, text:Краматорский
21
![Page 23: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/23.jpg)
Approach1:ProcessinginRus/Ukr
UniversalDependencyParsing
ConceptualMentionExtraction(COMEX)
StanfordNLP UDPipe
Ontology Lexicon
• OurontologyisasupersetoftheNIST/LDContology• Lexiconsare(semi-)manuallycreatedfromthetrainingdata• Conceptualextractionusing(manual)rule-basedinference• Focusisonhighprecision
22
5
![Page 24: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/24.jpg)
Parsing/tagging/chunkingpipeline
• Syntaxpipeline:– UDPipe1.2(Straka&Strakova2017)– Extractheadnounsanddependents– Notallentitiesandeventsneeded
• Eventframeconstruction:COMEX– OurontologyisasupersetoftheAIDAontology– Triggertermsmanuallymappedtoontology:
• Directmatching—manuallycuratedlistoftriggerwords• Englishtriggers—translationorWordNet/dictionarylookup
– Analysisguidedbyannotation:• LDCannotationsfromseedlingcorpus• Ownmanualannotationaswell
23
5
5
![Page 25: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/25.jpg)
COMEXontology
*fighter-planeLDC_ent_146
*airplaneLDC_ent_142
*vehicleLDC_ent_140
*weaponLDC_ent_160
*physical-entity
*entity
*mil-vehicleLDC_ent_145
*MiG-29
• Multipleinheritance• Greatercoverage
24
5
![Page 26: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/26.jpg)
COMEXlexicons
• Connectwordstoontologyconceptsviawordsenses• Providerulesforconnectingconceptsintoamentiongraph• Semanticrequirementsforslotfillersarespecifiedintheontology
W,атаковать,WS:attack-physical,WS:attack-verbalS,WS:attack-physical,*attack-physical,VERBA,WS:attack-physical, Attacker=Pull:active-subj;Pull:passive-subjA,WS:attack-physical, Target=Pull:active-dir-obj;Pull:passive-dir-objA,WS:attack-physical, Instr=Pull:active-subjA,WS:attack-physical, Place=Pull:obl-in#R,Pull:active-subj, nsubj, Trigger->Voice=ActR,Pull:passive-subj, obl, Trigger->Voice=Pass,Target->Case=Ins
Whilethelexiconscontainhundredsofwords,thenumberofrulesissmall25
![Page 27: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/27.jpg)
Lexiconconstruction• InitialvocabularyandthecorrespondingconceptsfromtheavailableLDCannotations
• VocabularyenrichmentbyextractingallnamedandnominalentitiesfromtheseedlingcorpusfilesthatcontainatleastoneLDCannotation
• EventtriggerenrichmentusingWordNet• Cross-languagevocabularyenrichmentusingMTandalignment• Manualcurationoftheresultingvocabulary• Manualadditionofattributerules• Iterativeimprovementprocess:
1. Extractmentionsfromanewfile2. Scoreresults3. Addvocabulary,fixrulesanddocross-languagetransfer
26
5
![Page 28: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/28.jpg)
SampleCOMEXperformance
English Russian UkrainianPrecision 0.91–1.0 0.93–1.0 1.0Recall 0.22–0.56 0.11–0.70 0.07–0.42F1 0.35–0.70 0.20–0.62 0.13–0.59Vocabulary 178 1483 1430*Rules 33 30 13
COMEXisthemost‘manual’ofOPERA’sTA1extractionmodules
27
(Thisworkcontinues;thenumberschangeeveryday)
![Page 29: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/29.jpg)
Approach2:Rus/Ukr—>English• Pipeline:
– MTRus/Ukr–>EnglishusingMSAzure– RunOPERATA1extractors– AlignsourcetexttoextractedmentionsinEng
• Back-translatefromEng,includingXML-likeentity/eventtags
• Outputisgenerallygood(espwhennoXMLtags)
• Problemsinback-translation:– SometimesmessesuptheXMLtags– Mayswitcheventarguments– Maymessuppropernames(e.g.Slavyansk–>Slavska,Slavovsk,Slavic
– ThingsliketyposoruncommonwordsgettranslatedincorrectlyintoEng,butmaybeeasytofixinthesourceusingfuzzymatching 28
2
MT
![Page 30: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/30.jpg)
Approachescomplementary
• Rus/Ukr:moreprecise– Lessnoise,betterentitytyping
• MT:moregeneral– Betteratnames,time/numbers,eventtyping
• Overlapsanddifferences:– Entityoverlap:84%ofRus/Ukr=44%ofMToutput– Eventoverlap:58%ofRus/Ukr=49%ofMToutput– Typeagreement:87%ofoverlap– Remainingmentions:65–70%correctoneachside– Differencesinspans,eventvs.entitychoices
29
![Page 31: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/31.jpg)
Rus/Ukrentity/relationdiscussion
• Challengesandproblems– Slowmanualrulebuilding,limitedcoverage(buthighprecision)
– COMEX<—>AIDAontologyalignment– Noiseintranslation
• Mostlymanual
30
52 5 5
![Page 32: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/32.jpg)
TA1/2KBCONSTRUCTIONANDVALIDATION
HansChalupsky
31
![Page 33: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/33.jpg)
OPERATA1framework
Input
Mini-KBcreation/AIFvalidation
Domainfilter&languagedetection
Ru/Ukpipeline
MT:Ru/Uk–>
Eng
Ru/UkEntityandEventdetection
EventframeassemblyII
CSRCombination
Imagepipeline
Entitydetection
PersonandGeoID
Englishentity
detection
Englishevent
detection
Englishentitylinking
Engentityrelations
EventframeassemblyI
Englishpipeline
Englishargumentdetection
Englishcoref
Speechpipeline
32
![Page 34: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/34.jpg)
CSR:PowerLoom-basedCommonsemanticrepository
• ContainsallKEs– Containsdiscretetermpropositions,[structured]distributionalvectors/tensors,continuousembeddings
– Eachwithvectorofscores(e.g.,TA1extractionconfidence,sourcetrustworthiness,reasoningimplicationconfidence,cross-KEcompatibility,hypothesis-basedlikelihoods,etc.)
• RepresentedinPowerLoom(Chalupskyetal.2010)– Predicate-logic-basedrepresentationbasedonKIFthatisasupportedsyntaxofCommonLogic
– Dynamic,scalable,multi-contextualsystemtostore,manageandreasonwithinformation
– Blazegraphdatabasetechforscalabilityandintegration– Representhypothesesandprobabilitiesviareification
• IntheCSReverythingisahypothesis33
5
4
![Page 35: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/35.jpg)
M9Approach:3-stepdecouplingforKBconstructionandvalidation
Multi-mediaAnnotations
NEREntityCorefEDLDBPediaEventsEventCorefRelationsAudioImage,Video…..
Extractors×N Annotation Ontology
Domain Ontology
AIDASeedling
34
KnowledgeAggregation
HypothesisIntegration,Evaluation,Inference
Domain Ontology Domain
Ontology Export
Ontologies
AIDAFull
…..
BackgroundKBBlazegraphTriplestore
Lotsoftypeheterogeneity
ReuseTACEVICdomainmodel
Exporttodifferenttargets
![Page 36: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/36.jpg)
M18Approach:SingleaugmentedontologyforKBconstructionandvalidation
Multi-mediaAnnotations
NEREntityCorefEDLDBPediaEventsEventCorefRelationsAudioImage,Video…..
Extractors×N Domain Ontology
35
KnowledgeAggregation
HypothesisIntegration,Evaluation,Inference
AIDAFull
BackgroundKBBlazegraphTriplestore
Supportsometypeheterogeneity
Domainadditions,APItocode
Annotation Augmentations
Domain Augmentations
5
2
![Page 37: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/37.jpg)
Incrementalcycleofhypothesisrepresentation,evaluation,refinement
Createhypothesis
representation
NERDoc.CorefEDLEventsEventCorefRelationsAudioImage,Video…..
Extractors×NDomain
Ontology
Select Import
Infer,EvalFix
Rules, Constraints
KB
• Cycle:– Usecorefsandotheridentitytoconnectannotations(mentionoverlap,
namelinks,EDL,within-doccoref,eventcoref)– Applyinferences,evaluateconstraints,detectconflicts,doattribution– Fixconflicts“ViktorYanukovych”?=“ViktorViktorovychYanukovych”
—no:irreflexive(parent)36
52
![Page 38: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/38.jpg)
TA1/2KBintegrationchallenges• Challenges:Ontological
– Multipletypesystems:NERtypes,relationtypes,eventtypes,KBschemas,targetschemas…
– Missingtypes,conflictingtypesoncethingsarelinked– Types,eveniffine-grained,primarilyusefulasconstraints,notas
equalitysignal–“Humvee17generally-not-equal-toHumvee42”– Inferencerequirements:“Donechyna”and“Ukraine”arecompatible
locationsofaneventbutnotwithrespecttohaving“Donetsk”astheircapital
– Ontological“fluidity”—thingschangeuntillateinthegame
• Challenges:Datasparsityandnoise– Multi-lingualnamesandcross-lingualmatching– Language-specificnamingschemes(e.g.,patronyms)– Cross-lingualuseofcontextvectors– Nofine-graineddocument,textormediacontextallowedacross
documents– Linkingdecisionsaggregatesupportandontologicalconflictwhich
propagates 37
5
4
5
![Page 39: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/39.jpg)
TA1scores
TA1Classqueries TA1Graphqueries
BestMAP
WorstMAP
TRECMAP
0.4843 0.4737 0.4773
0.4527 0.3697 0.4020
0.4379 0.2816 0.3278
0.4243 0.1470 0.1957
0.2290 0.0892 0.1244
Prec Recall F10.4715 0.2163 0.29660.4944 0.1328 0.20940.3605 0.0533 0.09290.0398 0.0312 0.03500.0138 0.0040 0.0062
Run:TA1a_OPERA_TA1a_aditi_V2
OPERA
![Page 40: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/40.jpg)
TA3HYPOTHESISCONSTRUCTIONAditiChaudhary,AnatoleGershman,JaimeCarbonell
39
![Page 41: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/41.jpg)
OPERATA2+TA3framework
Coref
Mini-KBcreation/AIFvalidation
TA1Mini-KB
Mini-KBcreation/AIFvalidation
Hypothesisconstruction
BeliefGraphconstruction
TA2Mini-KB
Queryinput
40
![Page 42: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/42.jpg)
Candidatehypothesisgeneration
1. (HavecompletedBeliefGraphandbeliefscorepropagationthroughout)
2. RetrieveKEscorrespondingtoentrypoints
3. RetrieveeventsEandrelationsRthatmatchconstraints.If:– Zero-hop:theretrievedeventisonehypothesis– One-hop:obtainaneventforeveryrole.Prioritizeevents/relationswithmaximumoverlapwiththeroles—thismaygivemanypermutations
4. GeneratehypothesiscandidatesetH=h1,h2...hnfromtheretrievedEandR
5
![Page 43: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/43.jpg)
Approach
Mention
matcher
BG
EvidenceKEs
Information Need
statement
EntrypointsarematchedtoEvidenceNodes
HypothesisGraph
candidates
HypothesisGraphselector
Role
inference
Augmented
HypothesisGraphs
Hypothesis
rankingand
filtering Hypotheses
42
![Page 44: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/44.jpg)
Candidatehypothesisgeneration
![Page 45: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/45.jpg)
Hypothesisranking
• GiveninformationneedIandcandidatesetH=h1,h2...hn,weneedtorankHbasedonrelevanceanddiversity
• Maximalmarginalrelevance:MMR=
– Sim(hi,I)=similarityscorebetweenhypothesishiandtheinformationneedI—givesrelevance
– Sim(hi,hj)=similarityscorebetweenhypotheseshiandhj—givesdiversity
λargmaxhi,E,HSim(hi,I)-(1-λ)argmaxhj,E,HSim(hi,hj)
![Page 46: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/46.jpg)
Relevanceanddiversity
• Sim(hi,I)=Measuringrelevance… sumof:– PercentageofframescoveredinI– Percentageofeventssatisfyingtheeventframes– Percentageofrelationssatisfyingtherelationframes– Numberofrole-entityexactmatchconstraints
• Sim(hi,hj)=Measuringdiversity(=inversesimilarity)betweentwohypotheses…sumof:– Numberofoverlappingevents– Numberofoverlappingrelations– Numberofoverlappingentities– Numberofoverlappingargumentsfortheaskedframes
![Page 47: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/47.jpg)
TA3M18evaluation
• Thiswasanextremelycomplextask• Wereceivedalotofnumbers• We’restillanalyzingthem
• Mostofthenumbersarenothelpfulforus• Manythingsconfuseus• Wewishformoredetailaboutcertainaspects
46
![Page 48: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/48.jpg)
Task3a(usingown/anyTA2KB)
Hypossubmit-ted
Theoriesmatched
Correct-ness
Edgecohe-rence
KEcohe-rence
Relstrict
Rellenient Coverage
24 6 0.4393 0.4834 0.6655 0.2832 0.6554 0.0320
42 4 0.2607 0.2894 0.423 0.1343 0.4192 0.0127
7 1 1.0000 1.0000 1.0000 1.0000 1.0000 0.0035
2 1 0.4167 0.4167 1.0000 0 1.0000 0.0032
42 1 0.3864 0.4475 0.5851 0.3295 0.4836 0.0032
OPERA
![Page 49: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/49.jpg)
Task3a(usingotherTA2teams’KBs)
Hypossubmit-ted
Theoriesmatched
Correct-ness
Edgecohe-rence
KEcohe-rence
Relstrict
Rellenient
Coverage
34 2 0.1042 0.2178 0.3711 0.1002 0.3512 0.0079
20 2 0.5107 0.6072 0.8145 0.3823 0.7973 0.0061
7 1 1.0000 1.0000 1.0000 1.0000 1.0000 0.0035
2 1 0.4167 0.4167 1.0000 0 1.0000 0.0032
42 1 0.3864 0.4475 0.5851 0.3295 0.4836 0.0032
![Page 50: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/50.jpg)
Task3b(usingLDCKB)
Hypossubmit-ted
Theoriesmatched
Correct-ness
Edgecohe-rence
KEcohe-rence
Relstrict
Rellenient
Coverage
42 2 0.5961 0.6249 0.839 0.3829 0.839 0.0238
45 6 0.8065 0.8069 0.9589 0.6263 0.9589 0.0153
42 4 0.8266 0.8612 0.8979 0.8312 0.9034 0.0100
24 2 0.8207 0.8406 1.0000 0.5905 1.0000 0.0060
29 1 0.8925 0.9063 1.0000 0.7421 1.0000 0.0051
![Page 51: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/51.jpg)
Hypothesisassessmentprocedure
• Assessmentprocedure:– Firstassesseachedgeascorrect/incorrect– Foronlycorrectones,matchagainstthegoldprevailingtheory
• Howassess/match?Decisions:– TypeandinformativementionofEdge– TypeandinformativementionofLeftside– TypeandinformativementionofRightside
50
Definedinontology.Smalldifferences.
Undefined.Manydifferencesofopinion
![Page 52: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/52.jpg)
Confusion#1:Initialedgefiltering
• Differenceinassessedhypothesisscoresonsamegold-standardLDCKBinput:
• Whythediscrepancies?OurSIN-drivenhypothesiscreationwasdifferent.ButwhydoesLDC’sownPrecisionnotgetupto.80? 51
Edgescorrect EdgessubmittedOPERA 59.6% 2545BBN 80.6% 908GAIA 82.1% 571UTexas 82.6% 1079PNNL 89.3% 394LDConTA2 0.59Precision
![Page 53: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/53.jpg)
Confusion#2:• WhydidGAIAdoalotbetteronGAIA’sownKBsthanonLDC’sKBs?
52
0.0320 _version2_QueryTypeBthroughE_GAIA_1.GAIA_2.GAIA_2_v2 GAIA2_v2runningonGAIAKBs
0.0248 _version4_QueryTypeBthroughE_GAIA_1.GAIA_2p.GAIA_2 GAIA2runningonGAIAKBs
0.0238 _version2_QueryTypeBthroughE_LDC_2.LDC_2.OPERA_TA3b_2 OPERArunningonLDCKBs
0.0153 _version4_QueryTypeBthroughE_LDC_2.LDC_2.BBN_TA3_v2a BBNrunningonLDCKBs
0.0127 _version2_QueryTypeBthroughE_OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.OPERA_TA3a_2 OPERArunningonOPERAKBs
0.0100 _version3_QueryTypeBthroughE_LDC_2.LDC_2.UTexas_3 UTexasrunningonLDCKBs
0.0079 _version3_QueryTypeBthroughE_GAIA_1.GAIA_2.OPERA_TA3a_1 OPERArunningonGAIAKBs
0.0061 _version2_QueryTypeBthroughE_BBN_1.BBN_TA2_v2.GAIA_2 GAIArunningonBBNKBs
0.0060 _version2_QueryTypeBthroughE_LDC_2.LDC_2.GAIA_2 GAIArunningonLDCKBs
0.0051 _version3_QueryTypeBthroughE_LDC_2.LDC_2.PNNL_sheafbox_10 PNNLrunningonLDCKBs
coverage
![Page 54: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/54.jpg)
Confusion#3:Informativementions
53
evt/rel:data:relation-instance-HYP-E102-3-r201907150216-23type:ldcOnt:Physical.LocatedNearhandle:womanofOdessaedgeprov:HC000Q7MI:(5700-0)-(5714-0)womanofOdessa-------------------------edge:ldcOnt:Physical.LocatedNear_EntityOrFillerarg:data:entity-instance-HYP-E102-3-r201907150216-0handle:thestrangledwomanofOdessa,whoforpro-Russianshas
becomeasymboloftheWestspartialityintheUkrainiancrisisassessed:CORRECT-------------------------edge:ldcOnt:Physical.LocatedNear_Placearg:data:entity-instance-HYP-E102-3-r201907150216-24handle:Odessaassessed:WRONG
![Page 55: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/55.jpg)
Arg1:thewoman
54
edge:ldcOnt:Physical.LocatedNear_EntityOrFillerarg:data:entity-instance-HYP-E102-3-r201907150216-0handle:thestrangledwomanofOdessa,whoforpro-Russianshasbecomea
symboloftheWestspartialityintheUkrainiancrisisconf:1.000000assessed:CORRECTbestPT:E102Theory4argprov:HC000Q7MI:(5686-0)-(5804-0)thestrangledwomanofOdessa,whoforpro-Russianshasbecomeasymbol
oftheWestspartialityintheUkrainiancrisiscontext:xactlywhathappened.Thisscarcityofinformationexplainswhysomany
rumourshaveemergedaround>>thestrangledwomanofOdessa,whoforpro-RussianshasbecomeasymboloftheWestspartialityintheUkrainiancrisis<<.Thisphotowasoriginallypostedhere.
![Page 56: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/56.jpg)
Arg2:Odessa
• Whyisitwrong?Sometheories:– OdessaisnotLocatedNearOdessa,itISOdessa– ThewomanwasborninOdessabutnowlivesinKiev– Thewomanwasonlyrumoredtohavebeenstrangled– …andmore… 55
edge:ldcOnt:Physical.LocatedNear_Placearg:data:entity-instance-HYP-E102-3-r201907150216-24handle:Odessaconf:1.000000assessed:WRONGargprov:HC000Q7MI:(5709-0)-(5714-0)Odessacontext:hisscarcityofinformationexplainswhysomanyrumourshaveemerged
aroundthestrangledwomanof>>Odessa<<,whoforpro-RussianshasbecomeasymboloftheWestspartialityintheUkrainiancrisis.Thisp
![Page 57: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/57.jpg)
Challenges
• Hypothesesgraphsaretooextensive,aseventsareconnectedbyatleastonecommonargument—needtoaddrestrictions
• Backgroundknowledgesometimesrequiredtolinkentities(e.g.,SU25==militaryjet)—perhapspre-populateKBwithbackgroundknowledge?
56
![Page 58: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/58.jpg)
FINALE
57
![Page 59: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/59.jpg)
Finale
• OPERAisanend-to-endsystem– Successfulcombinationofmachinelearningandmanualcomponentsandapproaches
– Task3b(usingLDC’sKB)submissionhadthehighestcoverage
– Managedwithlimiteddataandchangingontology• Absorbed&processedGAIATA1&TA2outputs
– GottopTA2graphqueryF1(1a)scoreusingGAIAKBs• Currentfocus:
– (Ofcourse)improvementseverywhere– Newdomainandontology– Seriousintegrationofcomponent-levelscores 58
![Page 60: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/60.jpg)
Somediscussionpoints
1. Multipleinheritanceintheontology
2. Mainroleofannotateddataisasexamples:continuousteam–LDCinteractiontogtsystemfeedback?
3. Itishardtoreconstructwhatexactlyassessorssaw.Ifthespecifictextualcontextthatanassessorlookedatforadecisionisrecorded,thenwecan– seethetexttheybasedtheirjudgmenton– maybealsogetsomefinerclassificationoftheerrortype
4. TA1bbias:Component-basedre-processingofsameresultswhengivennewhypotheses
59
![Page 61: Welcome to the...Sample COMEX performance English Russian Ukrainian Precision 0.91–1.0 0.93–1.0 1.0 Recall 0.22–0.56 0.11–0.70 0.07–0.42 F1 0.35–0.70 0.20–0.62 0.13–0.59](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fdf66b23b3e027cf04c8216/html5/thumbnails/61.jpg)
60
60
543
2
1