Overview of TAC-KBP2014 Entity Discovery and Linking Tasks

Heng Ji1, Joel Nothman2, Ben Hachey2

1 Computer Science Department, Rensselaer Polytechnic Institutejih@rpi.edu

2 School of Information Technologies, University of Sydney{joel.nothman,ben.hachey}@gmail.com

Abstract

In this paper we give an overviewof the Entity Discovery and Linkingtasks at the Knowledge Base Populationtrack at TAC 2014. In this year weintroduced a new end-to-end Englishentity discovery and linking task whichrequires a system to take raw textsas input, automatically extract entitymentions, link them to a knowledge base,and cluster NIL mentions. In this paper weprovide an overview of the task definition,annotation issues, successful methods andresearch challenges associated with thisnew task. This new task has attracteda lot of participants and has intriguedmany interesting research problems andpotential approaches. We believe it’sa promising task to be extended to atri-lingual setting in KBP2015.

1 Introduction

From 2009 to 2013 the Entity Linking (EL) trackat NIST Text Analysis Conference’s KnowledgeBase Population (TAC-KBP) track aimed tolink a given named entity mention from asource document to an existing Knowledge Base(KB). An EL system is also required to clustermentions for those NIL entities that don’t havecorresponding KB entries.

Most earlier EL work in the KBP community isusually formulated as a ranking problem, either by(i) Non-collective approaches, which resolve onemention at each time relying on prior popularity,context similarity, and other local features with

supervised models (Mihalcea and Csomai, 2007;Milne and Witten, 2008; Han and Sun, 2011;Guo et al., 2013); or (ii) Collective approaches,which disambiguate a set of relevant mentionssimultaneously by leveraging the global topicalcoherence between entities through graph-basedapproaches (Cucerzan, 2007; Milne and Witten,2008; Han and Zhao, 2009; Kulkarni et al., 2009;Pennacchiotti and Pantel, 2009; Ferragina andScaiella, 2010; Fernandez et al., 2010; Radfordet al., 2010; Cucerzan, 2011; Guo et al., 2011;Han et al., 2011; Ratinov et al., 2011; Kozarevaet al., 2011; Hoffart et al., 2011; Cassidy etal., 2012; Shen et al., 2013; Liu et al., 2013;Huang et al., 2014b). On the other hand, alot of research has been done in parallel in theWikification community (Bunescu, 2006) whichaims to extract prominent ngrams as conceptmentions, and link each concept mention to theKB. One important research direction of theKBP program is “Cold-start”, namely we aim todevelop an automatic system to construct a KBfrom scratch. To meed these new needs andchallenges, this year we designed a new task calledEntity Discovery and Linking (EDL). An EDLsystem is required to take a source collection ofraw documents as input, identify and classify allentity mentions, link each entity mention to theKB, and cluster all NIL mentions. In 2014 wehave explored it only for English, but in next yearwe aim to extend it to the tri-lingual setting withsource corpora from three languages: English,Chinese and Spanish.

Compared to the KBP entity linking evaluationsin previous years, the main changes andimprovement in KBP2014 include:

• Extend English task to Entity Discoveryand Linking (full Entity Extraction + EntityLinking + NIL Clustering);

• Add discussion forums to Cross-lingualtracks (some entity morphs naturally exist inChinese discussion forums);

• Provide basic NLP, full IE, Entity Linkingand semantic annotations for some sourcedocuments to participants;

• Share some source collections and querieswith regular and cold-start slot filling tracks,to investigate the role of EDL in the entirecold-start KBP pipeline.

The rest of this paper is structured as follows.Section 2 describes the definition of each EntityDiscovery and Linking task in KBP2014. Section3 briefly summarizes the participants. Section 4highlights some annotation efforts. Section 5and 6 summarize the general architecture of eachtask’s systems and evaluation results, and providesome detailed analysis and discussion. From eachparticipant, we only select the best submissionwithout Web access for comparison. Section 7discusses the experimental results and sketchesour future work.

2 Task Definition and Evaluation Metrics

This section will summarize the Entity Discoveryand Linking tasks conducted at KBP 2014. Moredetails regarding data format and scoring softwarecan be found in the KBP 2014 website1.

2.1 Mono-lingual Entity Discovery andLinking Task

2.1.1 Task DefinitionBased on the above motivations, this year weadded a new task of Entity Discovery andLinking (EDL) in the mono-lingual Englishtrack. The goal is to conduct end-to-end entityextraction, linking and clustering. Given adocument collection, an EDL system is requiredto automatically extract (identify and classify)entity mentions (“queries”), link them to the KB,and cluster NIL mentions (those that don’t havecorresponding KB entries). Compared to EntityLinking from previous years, an EDL systemneeds to extract queries automatically. In contrast

1http://nlp.cs.rpi.edu/kbp/2014/

to Wikification (Bunescu, 2006; Mihalcea andCsomai, 2007; Ratinov and Roth, 2012), EDLonly focuses on three types of entities (Person(PER), Organization (ORG) and Geo-politicalEntity (GPE, a location with a government)) andrequires NIL clustering. In order to evaluate theimpact of entity name mention extraction on thisnew EDL task, we also organized a diagnosticevaluation on English entity linking as defined inKBP2013, with perfect entity name mentions asinput.

The input to EDL is a set of raw documents.We selected a subset of the TAC 2014 documentcollection from multiple genres includingnewswire, web data, and discussion forum posts,which include high values in terms of bothambiguity and variety and substantial amount ofNIL entity mentions. An EDL system is requiredto automatically generate the following two files.

(1). Mention Query File: An EDL system isrequired to identify and classify name mentionsinto person (PER), organization (ORG) orgeo-political entity (GPE); and then represent eachname mention as a query that consists of a namestring, a document ID, and a pair of UTF-8character offsets indicating the beginning and endlocations of the name string in the document. Thedetailed definition of an entity name mention (aquery) is presented in the LDC query developmentguideline 2. For example:

〈query id=‘‘EDL14 ENG 0001"〉〈name〉cairo〈/name〉〈docid〉bolt-eng-DF-200-192451-5799099〈/docid〉〈beg〉2450〈/beg〉〈end〉2454〈/end〉〈/query〉

(2). Link ID File: Then for each entity mentionquery, an EDL system should attempt to link it tothe given knowledge base (KB). The EDL systemis also required to cluster queries referring to thesame non-KB (NIL) entities and provide a uniqueID for each cluster, in the form of NILxxxx (e.g.,“NIL0021”). It should generate a link ID file thatconsists of the entity type of the query, the IDof the KB entry to which the name refers, or a“NILxxxx” ID if there is no such KB entry, anda confidence value.

2http://nlp.cs.rpi.edu/kbp/2014/elquery.pdf

Short name Name in scoring software Filter Key Evaluates TaskMention evaluationNER strong mention match NA span NER EDLNERC strong typed mention match NA span,type NER and classification EDLLinking evaluationNERL strong all match NA span,kbid (NER and) linking EL,EDLNEL strong link match is linked span,kbid Link recognition DiagnosticNEN strong nil match is nil span NIL recognition DiagnosticTagging evaluationKBIDs entity match is linked docid,kbid Document tagging DiagnosticClustering evaluationCEAFm mention ceaf NA span NER and clustering EDLB-Cubed b cubed NA span Clustering ELB-Cubed+ b cubed plus NA span,kbid Linking and clustering EL

Table 1: Evaluation measures for entity discovery and linking, each reported as P , R, and F1. Span isshorthand for (document identifier, begin offset, end offset). Type is PER, ORG or GPE. Kbid is the KBidentifier or NIL.

2.1.2 Scoring MetricsTable 1 lists the official evaluation measures forTAC 2014 entity discovery and linking (EDL). Italso lists measures for the diagnostic entity linking(EL) task, which are identical to the TAC 2011-13evaluations. Since systems use gold standardmentions for EL, it isolates linking and clusteringperformance. The scorer is available at https://github.com/wikilinks/neleval.

Set-based metrics Recognizing and linkingentity mentions can be seen as a tagging task. Hereevaluation treats an annotation as a set of distincttuples, and calculates precision and recall betweengold (G) and system (S) annotations:

P =|G ∩ S||S|

R =|G ∩ S||G|

For all measures P and R are combined as theirbalanced harmonic mean, F1 =

2PRP+R .

By selecting only a subset of annotated fieldsto include in a tuple, and by including only thosetuples that match some criteria, this metric can bevaried to evaluate different aspects of systems (cf.Hachey et al. (2014) which also relates such metricvariants to the entity disambiguation literature).As shown in Table 1, NER and NERC metricsevaluate mention detection and classification,while NERL measures linking performance butdisregards entity type and NIL clustering. Inthe EL task where mentions are given, NERL isequivalent to the linking accuracy score reportedin previous KBP evaluations.

Results below also refer to other diagnosticmeasures, including NEL which reports linking(and mention detection) performance, discarding

NIL annotations; NEN reports the performanceof NIL annotations alone. KBIDs considersthe set of KB entities extracted per document,disregarding mention spans and discarding NILs.This measure, elsewhere called bag-of-titlesevaluation, does not penalize boundary errors inmention detection, while also being a meaningfultask metric for document indexing applications ofnamed entity disambiguation.

Clustering metrics Alternatively, entity linkingis understood as a cross-document coreferencetask, in which the set of tuples is partitioned bythe assigned entity ID (for KB and NIL entities),and a coreference evaluation metric is applied.

Where previous KBP evaluations employedB-Cubed (Bagga and Baldwin, 1998), 2014 is thefirst year to apply CEAF (Luo, 2005). B-Cubedassesses the proportion of each tuple’s cluster thatis shared between gold and system clusterings,while CEAF calculates the optimal one-to-onealignment between system and gold clusters basedon a provided similarity metric, and returns thesum of aligned scores relative to aligning eachcluster with itself. In the Mention CEAF (CEAFm)variant used here, cluster similarity is simplymeasured as the number of overlapping tuples.

Again, variants may be introduced by selectinga subset of fields or filtering tuples. For the presentwork, only the mention’s span is considered,except in B-Cubed+ which treats mention asspan,kbid tuples, only awarding mentions that areclustered together if their KB link is correct.

We now define the clustering metrics formally.If we let Gi ∈ G describe the gold partitioning,and Si ∈ S for the system, we calculate the

maximum score bijection m:

m = argmaxm

|G|∑i=1

∣∣Gi ∩ Sm(i)

∣∣s.t. m(i) = m(j) ⇐⇒ i = j

Then we can define CEAFm, and provideB-Cubed for comparison:

PCEAFm =

∑|G|i=1

∣∣Gi ∩ Sm(i)

∣∣∑|S|i=1 |Si|

RCEAFm =

∑|G|i=1

∣∣Gi ∩ Sm(i)

∣∣∑|G|i=1 |Gi|

PB-Cubed =

∑|G|i=1

∑|S|j=1

|Gi∩Sj |2|Sj |∑

i |Si|

RB-Cubed =

∑|G|i=1

∑|S|j=1

|Gi∩Sj |2|Gi|∑

i |Gi|

Like the set-based metrics, B-Cubed andCEAFm report the number of mentions that arecorrect by some definition. Hence these metricsare likely to be correlated, and B-Cubed+ is bydefinition bounded from above by both B-Cubedand NERL. CEAF prefers systems that return thecorrect quantity of clusters, and splitting an entityinto multiple clusters mean some of those clusterswill be awarded no score. Thus a system thatincorrectly splits Abu Mazen and Mahmoud Abbasinto different entities will be assigned no scorefor the smaller of those clusters (both precisionand recall error). In B-Cubed, the same systemis awarded for correctly predicting that multiplementions were coreferent with each other, despitethe entity being split.

Confidence intervals We calculate c%confidence intervals for set-based metrics bybootstrap resampling documents from the corpus,calculating these pseudo-systems’ scores, anddetermining their values at the 100−c

2 th and100+c

2 th percentiles of 2500 bootstrap resamples.This procedure assumes that the system annotatesdocuments independently, and intervals are notreliable where systems use global clusteringinformation in their set-based output (i.e. beyondNIL cluster assignment). For similar reasons, wedo not calculate confidence intervals for clusteringmetrics.

2.2 Cross-lingual Chinese/Spanish to EnglishEntity Linking

The cross-lingual entity linking tasks follow themonolingual entity linking in previous years (Ji etal., 2011) in which the entity mention queries aregiven; the steps are: (1) link non-NIL queries toEnglish KB entries; and (2) cluster NIL queries.

3 Participants Overview

Table 2 summarizes the participants for EnglishEDL task. In total 20 teams submitted 75 runsfor the full task and 17 teams submitted 55 runsfor the diagnostic task (Entity Linking with perfectmentions as input).

4 Data Annotation and Resources

The details of the data annotation for KBP2014are presented in a separate paper by the LinguisticData Consortium (Ellis et al., 2014). Thenew EDL task has also introduced many newchallenges to the annotation guideline, especiallyon defining entity mentions in the new setting ofcold-start KBP. We released a new KBP entitymention definition guideline 3. There still remainsome annotation errors on both entity mentionextraction and linking. We will continue refiningthe annotation guidelines and conduct a systematiccorrection on the annotation errors.

In addition, we devoted a lot of time atcollecting related publications and tutorials 4,resources and softwares 5 to lower down the entrycost for EDL.

5 Mono-lingual Entity Linking

5.1 Approach Overview5.1.1 General ArchitectureA typical KBP2014 mono-lingual EDL systemarchitecture is summarized in Figure 1. Itincludes six steps: (1) entity mention extraction- identify and classify entity mentions (“queries”)from the source documents; (2) query expansion- expand the query into a richer set of formsusing Wikipedia structure mining or coreferenceresolution in the background document; (3)candidate generation - finding all possible KBentries that a query might link to; (4) candidateranking - rank the probabilities of all candidates

3http://nlp.cs.rpi.edu/kbp/2014/elquery.pdf4http://nlp.cs.rpi.edu/kbp/2014/elreading.html5http://nlp.cs.rpi.edu/kbp/2014/tools.html

Table 2: The Number of Runs Submitted by KBP2014 English Entity Discovery and Linking Participants

using non-collective or collective approaches; thelinking decision (knowledge from the KB) canbe used as feedback to refine the entity mentionextraction results from step (1); (4) NIL detectionand clustering - detect the NILs which got lowconfidence at matching the top KB entries fromstep (4), and group the NIL queries into clusters.

Most the ranking algorithms are inherited fromprevious years, except the novel Progammingwith Personalized PageRank algorithm developedby the CohenCMU team (Mazaitis et al., 2014).A nice summary of the state-of-the-art rankingfeatures can be found in Tohoku NL team’ssystem description paper (Zhou et al., 2014).In the following subsections we will highlightthe new and effective techniques used in entitylinking.

5.2 Evaluation Results5.2.1 Overall PerformanceThe linking and clustering results of mono-lingualEDL are summarized in Figures 2 and 3respectively.6

6In most figures, we only show the performance of the topsystem per team on the primary task score (CEAFm for EDL

Although CEAFm does not explicitly takelink target into account, its results closelymirror NERL, with some variation: for example,BUPTTeam1 ranks much better under CEAFmthan NERL. The confidence intervals (CIs) shownin Figure 2 highlight two main cohorts ofhigh-achieving systems: those with performancearound or above 60% NERL F1 may besignificantly different from those below. Onlyother lcc systems and RPI BLENDER5 fall in theNERL 90% CI for the top system (lcc20142).

5.2.2 Impact of Mention ExtractionA major change from previous KBP evaluations isconsidering the end-to-end task in which systemsare required to identify, link and cluster mentions.Mention extraction is very important to goodNERL and CEAFm performance, as shown by theNER performance in Figure 4.

The KBIDs measure results show that a fewsystems (particularly UBC5, IBM1) are ableto predict the set of correct KB targets for adocument but are let down on the span-sensitivemetrics by low NER performance. This measure

and B-Cubed+ for EL), and ordered according to that score.

Figure 1: General Mono-lingual Entity LinkingSystem Architecture

is also highly correlated with NERL/NER, theratio of linking to mention extraction performance,which should isolate the difficulty of linkinggiven noisy system mentions. By this ratio,state-of-the-art linking performance lies between80 and 90%, with some systems such as IBM1achieving a very high score by this ratio, whilelet down by a low NER performance. Thisagrees with the NERL results of the diagnosticEL evaluation – which provides systems withgold-standard mentions – shown in Figure 5.

lcc20142

RPI_BLE

NDER5

MSIIPL_

THU1

NYU1

Tohok

u_NL2

ICTC

AS_OKN1

HITS2

UBC1

vera

1

BUPTTe

am1

CSFG4

UI_CCG2

IBM1

OSU20.3

0.4

0.5

0.6

0.7

0.8

0.9

NER

L sc

ore

P

R

F1

Figure 2: Linking performance (NERL) of top 14monolingual EDL systems (CEAFm F1 > 50%)with 90% confidence intervals

lcc20142

RPI_BLE

NDER5

MSIIPL_

THU1

NYU1

Tohok

u_NL2

ICTC

AS_OKN1

HITS2

UBC1

vera

1

BUPTTe

am1

CSFG4

UI_CCG2

IBM1

OSU20.3

0.4

0.5

0.6

0.7

0.8

0.9

CEA

Fm s

core

P

R

F1

Figure 3: Clustering performance (CEAFm) oftop 14 monolingual EDL systems (CEAFm F1 >50%)

lcc20142

RPI_BLE

NDER5

MSIIPL_

THU1

NYU1

Tohok

u_NL2

ICTC

AS_OKN1

HITS2

UBC1

vera

1

BUPTTe

am1

CSFG4

UI_CCG2

IBM1

OSU20.3

0.4

0.5

0.6

0.7

0.8

0.9

F1

NER

NERC

NERL

NERCL

KBIDs

NERL/NER

Figure 4: EDL F1 measures related to mentionextraction

MS_MLI1

ICTC

AS_OKN1

bobwils

on1

NYU2

lcc20143

RPI_BLE

NDER5UBC2

MSIIPL_

THU1

UI_CCG1

HITS2

vera

1

NTUA_IL

SP3

CSFG2

TALP

_UPC

20.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

F1

B-Cubed+

B-Cubed

CEAFm

NERL

KBIDs

Figure 5: Performance of top 15 Diagnostic EDL(perfect mention as input) systems (B-Cubed+F1 > 50%) under various measures 8

lcc20142

RPI_BLE

NDER5

MSIIPL_

THU1

NYU1

Tohok

u_NL2

ICTC

AS_OKN1

HITS2

UBC1

vera

1

BUPTTe

am1

CSFG4

UI_CCG2

IBM1

OSU20.3

0.4

0.5

0.6

0.7

0.8

0.9

F1

NERL

NEL

NEN

KBIDs

Figure 6: EDL F1 measures comparing NIL andnon-NIL performance

Note also that KBIDs corresponds closely withNERL in the EL diagnostic task, which suggeststhat the disparity between KBIDs and NERL inEDL pertains to NER performance rather thanin-document coreference.

Comparing NEL (linked only) and NEN (NILonly) in Figure 6 shows that the systems thatperform particularly well on KBIDs also handleKB mentions better than NILs. Conversely,the MSIIPL THU systems rank well by CEAFmwith a system that handles NILs better than KBmentions.

As well as identifying mentions, systems wererequired to assign each entity a type, which isuseful for downstream KBP tasks such as slotfilling. Identifying mentions of entities other thanthe sought types also leads to precision errors in allother measures. Thus teams with good NER alsohave good entity types (NERC), HITS being thenotable exception. Performance reduces as little as3% F1 due to type detection. Note that top NERCperformance is only 74.9%, which is much lowerthan state-of-the-art named tagging performance(around 89% F-score) on standard datasets suchas ACE or CONLL (e.g., Ratinov and Roth(2009)). Other tasks incorporate a different setof entity types, such as the catch-all “MISC”type and non-GPE locations; their loss may makethe task more difficult. However, systems didnot tend to train their recognisers directly onthe in-domain corpus, and the lower performanceappears consistent with out-of-domain results forNERC (Nothman et al., 2013).

5.2.3 Entity Types and Textual Genres

In Figures 7 and 8 we plot, respectively, theEDL and EL NERL performance when subsetsof the queries are considered, breaking down thecollection by entity type and textual genre. For

lcc20142

RPI_BLE

NDER5

MSIIPL_

THU1

NYU1

Tohok

u_NL2

ICTC

AS_OKN1

HITS2UBC1

vera

1

BUPTTe

am1

CSFG4

UI_CCG2

IBM1

OSU2

All

PER

ORG

GPE

NW

WB

DF

PER & NW

PER & WB

PER & DF

ORG & NW

ORG & WB

ORG & DF

GPE & NW

GPE & WB

GPE & DF

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Figure 7: EDL NERL F1 when selecting a subsetof annotations by entity type and text genre. PER= person, ORG = organization, GPE = geopoliticalentity; NW = newswire, WB = web log, DF =discussion forum.

MS_MLI1

ICTC

AS_OKN1

bobwils

on1NYU

2

lcc20143

RPI_BLE

NDER5UBC2

MSIIPL_

THU1

UI_CCG1

HITS2

vera

1

NTUA_IL

SP3

CSFG2

TALP

_UPC

2

All

PER

ORG

GPE

NW

WB

DF

PER & NW

PER & WB

PER & DF

ORG & NW

ORG & WB

ORG & DF

GPE & NW

GPE & WB

GPE & DF

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Figure 8: EL NERL F1 when selecting a subset ofannotations by entity type and text genre. See keyin Figure 7.

lcc20142

RPI_BLE

NDER5

MSIIPL_

THU1

NYU1

Tohok

u_NL2

ICTC

AS_OKN1

HITS2

UBC1

vera

1

BUPTTe

am1

CSFG4

UI_CCG2

IBM1

OSU20.3

0.4

0.5

0.6

0.7

0.8

0.9

F1

NERL

CEAFm

Typed CEAFm

Typed CEAFm+

B-Cubed

B-Cubed+

Figure 9: EDL F1 measures related to clustering

EDL, the gold standard annotations are filteredby gold standard entity type and the systemannotations by predicted entity type. CEAFmresults show similar trends.

Performance is generally much higher onperson entities than the other types. GPE clearlyplaces second in EDL, perhaps reflecting thedifficulty of NER for diverse organization names,particularly out of the news genre. But the rankingof GPE and ORG is more equivocal in EL; UBCand HITS perform markedly better on GPE andORG than PER.

In both task variants, discussion forum textproved easiest to process, at least for PERand GPE extraction, followed by newswire andweb logs. The variation in mention detectionperformance on PERs in forums derives fromthe need to identify poster names, not merelynames in running text. Systems with lowperformance here – ITCAS OKN1, HITS2 andIBM1 – appear not to have specifically targetedthese mentions. Recognition of person names inweb logs appears particularly difficult relative toother genres, while GPEs are difficult to link (evenwith gold mentions) in web logs.

5.2.4 Clustering Performance

The newly adopted CEAFm measure provides aranking that is very similar to that of B-Cubedor B-Cubed+, as shown in Figure 9. Thisis particularly true across the top six teams,with more variation in the lower rankings. Thedifferences are more pronounced when comparingto B-Cubed+, since it is a combined measure oflinking and clustering performance.

Notably, the EL task reports higher B-Cubedthan CEAFm scores in all cases (Figure 5), whileEDL shows the reverse trend (Figure 9). Thishighlights the sensitivity of B-Cubed to mentionextraction errors, as discussed in Cai and Strube

(2010).

5.3 What’s New and What Works5.3.1 EDL MilestonesOverall Entity Linking and Wikification researchis a rapidly growing and thriving area acrossvarious research communities including NaturalLanguage Processing, Data Mining and SemanticWeb, demonstrated by the 130 representativepapers published during 2006-2014 9. We listedsome milestones in the following.

• 2006: The first definition of Wikification taskis proposed by (Bunescu, 2006).

• 2009: TAC-KBP Entity Linking waslaunched (McNamee and Dang, 2009).

• 2008-2012: Supervised learning-to-rank withdiverse levels of features such as entityprofiling, various popularity and similaritymeasures were developed (Chen and Ji, 2011;Ratinov et al., 2011; Zheng et al., 2010;Dredze et al., 2010; Anastacio et al., 2011).

• 2008-2013: Collective Inference, Coherencemeasures were developed (Milne and Witten,2008; Kulkarni et al., 2009; Ratinov et al.,2011; Chen and Ji, 2011; Ceccarelli et al.,2013; Cheng and Roth, 2013).

• 2012: Various applications(e.g., KnowledgeAcquisition (via grounding), Coreferenceresolution (Ratinov and Roth, 2012) andDocument classification (Vitale et al., 2012;Song and Roth, 2014; Gao et al., 2014).

• 2014: TAC-KBP Entity Discovery andLinking (end-to-end name tagging,cross-document entity clustering, entitylinking).

• 2012-2014: Many different versions ofinternational evaluations were inspired fromTAC-KBP.

5.3.2 Joint Extraction and LinkingSome recent work (Sil and Yates, 2013; Meij etal., 2012; Guo et al., 2013; Huang et al., 2014b)proved that mention extraction and mentionlinking can mutually enhance each other. Inspiredby the these successes, many teams includingIBM (Sil and Florian, 2014), MSIIPL THU (Zhao

9http://nlp.cs.rpi.edu/kbp/2014/elreading.html

et al., 2014), SemLinker (Meurs et al., 2014),UBC (Barrena et al., 2014) and RPI (Honget al., 2014) used the properties in externalKBs such as DBPedia as feedback to refine theidentification and classification of name mentions.Using this approach, RPI system successfullycorrected 11.26% wrong mentions. The HITSteam (Judea et al., 2014) proposed a joint approachthat simultaneously solves extraction, linking andclustering using Markov Logic Networks. Forexample, in the following sentence “Bosch willprovide the rear axle.”, linking “Bosch” to “RobertBosch Tool Corporation” based on context “rearaxle” can help us type it as an organization.Similarly, it’s relatively easy to link “San Antoni”in the following sentence: “Parker was 15 for21 from the field, putting up a season high whilescoring nine of San Antonio’s final 10 points inregulation.” to “San Antonio Spurs” and type it asan organization instead of GPE.

5.3.3 Task-specific and Genre-specificMention Extraction

The new definition of entity mentions in the KBPsetting and new genres require us to do someeffective adaptation of traditional name taggingapproaches. For example, 4% entity mentionsincluded nested mentions. Posters in discussionforum should be extracted. Due to time limit,several teams including HITS (Judea et al., 2014),LCC (Monahan et al., 2014), MSIIPL THU (Zhaoet al., 2014), NYU (Nguyen et al., 2014) andRPI (Hong et al., 2014) developed heuristic rulesto improve name tagging. In the future, we expectthat more effective genre adaptation approach canbe developed to further improve the performance.

5.4 Better Meaning RepresentationAddressing many linking challenges requiresacquiring and better representing the deeper,richer and more discriminative semanticknowledge of each entity mention and itscontext, beyond the semantic attributes fromtypical information extraction, dependencyparsing and semantic role labeling techniques.For example, in

“Local OWS activists were part of thisprotest”,

the candidate entities for “OWS” include “Orderof World Scouts”, “Occupy Wall Street”, “OilyWater Separator”, “Overhead Weapon Station”,

“Open Window School” and “Open GeospatialConsortium”. The knowledge that “OWS” isassociated with a “protest” event is neededto correctly link it to “Occupy Wall Street”.Similarly, in the following discussion forum posts:

“It was a pool report typo. Here is exactRhodes quote: ”this is not gonna be acouple of weeks. It will be a period ofdays.” At a WH briefing here in Santiago,NSA spox Rhodes came with a litany ofpushback on idea WH didn’t consult withCongress. Rhodes singled out a Senateresolution that passed on March 1st whichdenounced Khaddafy’s atrocities. WH saysUN rez incorporates it.”,

in order to link “Rhodes” to the speech writer“Ben Rhodes” in the KB, we rely on knowledgethat a “speech writer” usually initiates: “quote”,“report”, “briefing” and “singled out”, and theorganizations a “speech writer” usually interactswith: “WH”, “NSA”, “Congress”, “Senate” and“UN”.

RPI system (Zheng et al., 2014) exploitedthe Abstract Meaning Representation(AMR) (Banarescu et al., 2013) and an AMRparser (Flanigan et al., 2014) to discover andrepresent rich knowledge from the sourcetexts. They proved that using AMR a simpleunsupervised collective inference method withoutusing any labeled EL data can outperform otherrepresentations such as semantic role labeling andall state-of-the-art unsupervised linking.

5.5 Select Collaborators from Rich ContextMany teams including lkd (Dojchinovski et al.,2014), NYU (Nguyen et al., 2014) and RPI (Honget al., 2014) exploited the rich properties andstructures in DBPedia for collective inference.The basic intuition is that the candidate entityand its collaborators decided by the mention’scollaborators in the source text should be stronglyconnected in the KB. Collective inference isparticularly effective to disambiguate entities withcommon names in discussion forum posts. Forexample, many countries can have a “SupremeCourt” or “LDP”; “Newcastle University” can belocated in UK or Australia; and many personentities share the same common names such as“Albert”; etc..

However, there might be many entitymentions in the context of a target entity

mention that could potentially be leveraged fordisambiguation. Various coherence measureswere introduced in recent research to choosethe “collaborators” (related mentions), suchas collaborative learning (Chen and Ji, 2011),ensemble ranking (Pennacchiotti and Pantel,2009; Kozareva et al., 2011), co-occurred conceptmentions (McNamee et al., 2011; Ratinov et al.,2011), topic modeling (Cassidy et al., 2012),relation extraction (Cheng and Roth, 2013),coreference (Huang et al., 2014a), semanticrelatedness based meta-paths (Huang et al.,2014a) and social networks (Cassidy et al., 2012;Huang et al., 2014a). RPI system (Zheng et al.,2014) exploited the semantic relation links amongconcepts from AMR to select collaborators.

Table 3 presents the various types of entitycontexts that may help disambiguate entities.In addition, some global context such asdocument creation time will be helpful for entitydisambiguation.

5.6 Graph-based NIL Entity Clustering

The CUNY-BLENDER KBP2012 Entity Linkingsystem (Tamang et al., 2012) explored morethan 40 clustering algorithms and found thatadvanced graph-based clustering algorithms didnot significantly our-perform single baseline“All-in-one” clustering algorithm on the overallqueries (except the most difficult ones). However,this year it’s very encouraging that LCC (Monahanet al., 2014) proved that graph partition basedalgorithm achieved significant gains.

5.7 Remaining Challenges

Regardless of the progress that we have achievedwith EDL, many challenges still remain. Wehighlight some outstanding ones as follows.

5.7.1 Mention Extraction is not a SolvedProblem

The best name mention extraction F-score inthe EDL task is only about 75%, which ismuch lower than the 89% F-score reported inthe literature (Ratinov and Roth, 2009; Li etal., 2012). Table refnamemilestone lists someprevious milestones for name tagging.

Compared to the first name tagging paper in1966, we made good progress on developingmachine learning algorithms and incorporating afew more features. There hasn’t been a lot ofactive work in the field after ACE because we we

tend to believe it’s a solved problem. Dojchinovskiet al. (2014) provided some detailed analysis onmention extraction challenges in EDL, especiallyon the confusion between ORG and GEP (e.g.,between sports teams and host cities). In addition,we summarized the new challenges as follows.

• Our name taggers are getting old. Almost allof them were trained from 2003 news but nowtested on 2012 news.

• Need effective genre adaption techniques.Table 5 shows the performance of astate-of-the-art name tagger (Li et al.,2012) trained and tested from various genrecombinations. We can see that a taggertrained from newswire can obtain 89.3%F-score on newswire but only 56.2% onbroadcast news.

• Need to address the new definitions ofname mention in the cold-start KBP settingby following the “extraction for linking”philosophy.

• Need to re-visit some old unsolved problems.For example, it remains difficult to identifylong nested organization names such as“Asian Pulp and Paper Joint Stock Company, Lt. of Singapore”. We should also continueto explore external knowledge sources suchas Word clustering, Lexical KnowledgeDiscovery (Ratinov and Roth, 2009; Jiand Lin, 2009) and more feedback fromentity linking, coreference, relation and eventextraction to improve the quality of nametagging. For example, without knowingthat “FAW” referring to “First AutomotiveWorks” in the following sentence “FAW hasalso utilized the capital market to directlyfinance,. . . ”, it’s very challenging to classify“FAW” as an organization name.

5.7.2 Linking Challenge: NormalizationCurrently there is no normalization schema toeasily align the property types across multiple KBswith different naming functions (e.g., “extinction”vs. “date of dissolved” for an organization)and granularities (e.g., “birthYear” vs. “date ofbirth” for an organization). It’s also challengingto automatically map the relation types amongentities in source documents and the propertytypes in KBs. For example, when a person

Source Knowledge BaseSocial RelationNo matter what, he never should have given [MichaelJackson]c that propofol. He seems to think a “proper” courtwould have let [Murray]m go free.

relation[Conrad Murray]e was charged with involuntarymanslaughter for causing [Michael Jackson]c’s death onJune 25, 2009, from a massive overdose of the generalanesthetic propofol.

Family[Mubarak]m, the wife of deposed Egyptian President [HosniMubarak]c, . . .

relation[Suzanne Mubarak]e (born 28 February 1941) is the wifeof former Egyptian President [Hosni Mubarak]c and was theFirst Lady of Egypt during her husband’s presidential tenurefrom 14 October 1981 to 11 February 2011.

EmploymentHundreds of protesters from various groups converged onthe state capitol in Topeka, [Kansas]c today. . . Second,I have a really hard time believing that there were anyACTUAL “explosives” since the news story they link to talksabout one guy getting arrested for THREATENING Governor[Brownback]m.

relation[Sam Bronwnback]e was elected Governor of [Kansas]c in2010 and took office in January 2011.

AffiliationDuring the 19th century of the [United States]c, the[Republican Party]m stood against the extension of slaveryto the country’s new territories and, ultimately, for slavery’scomplete abolition.

relationThe [Republican Party]e, also commonly called the GOP(abbreviation for Grand Old Party), is one of the two majorcontemporary political parties in the [United States]c.

Part-wholeAT&T coverage in [GA]c is good along the interstates and inthe major cities like Atlanta, Athens, [Rome]m, Roswell, andAlbany.

relationAt the 2010 census, [Rome]e had a total population of 36,303,and is the largest city in Northwest [Georgia]c and the 19thlargest city in the state.

Start-Position EventGoing into the big [Super Tuesday]c, [Romney]m had wonthe most votes, states and delegates, [Santorum]c had wonsome contests and was second, [Gingrich]c had won only onecontest.

relationThe [Super Tuesday]c primaries took place on March 6. [MittRomney]m carried six states, [Rick Santorum]c carried three,and [Newt Gingrich]c won only in his home state of Georgia.

Table 3: Various Types of Entity Context

Table 5: Cross-genre Name Tagging Performance

A and an organization B are represented as“ARG1 (recipient)” and “ARG0 (agent)” of a“nominate-01” event and B is identified as a“political-party”, this relation can be aligned tothe link by A’s Wikipedia infobox “political party”(B). The Semantic Web community has spent alot of manual efforts. Further improvement islikely to be obtained if such normalization can beautomatically done using relation discovery andclustering techniques (Hasegawa et al., 2004) andtechniques.

5.7.3 Linking Challenge: Morph

Another unique challenge in social media isresolving entity morphs which were created toavoid censorship, express strong sentiment orhumor (Huang et al., 2013; Zhang et al., 2014).For example, “Christie the Hutt” is used to referto “Chris Christie”; “Dubya” and “Shrub” areused to refer to “George W. Bush”. Morphmentions usually share little surface features withtheir KB titles, and thus simple string matchingmethods fail to retrieve the correct candidates. Onthe other hand it’s computationally intractable tosearch among all KB entries. Therefore, some

Table 4: Name Tagging Milestones

global temporal distribution or social distancebased methods that beyond text processing arerequired to address this problem.

5.7.4 Linking: Commonsense KnowledgeSome other remaining cases require commonsense knowledge. For example, many approachesmistakenly linked “William J. Burns” in thefollowing sentence ‘During talks in Genevaattended by William J. Burns Iran refused torespond to Solana’s offers” to “William J. Burns(1861-1932)” in the KB. If we knew this sourcedocument was created in July 26, 2008 and so theentity was alive at that time since he was involvedin the events such as attending talks, we would nothave linked it to the dead person described in theKB. Such commonsense knowledge is very hardto acquire and represent. Recent work by Chenet al. (2014) attempted automatic discovery ofcommonsense knowledge for relation extraction.It may be also worth exploiting existing manuallycreated commonsense knowledge from ConceptNet (Liu and Singh, 2004) or FrameNet (Bakerand Sato, 2003).

6 Cross-lingual Entity Linking

Only two teams HITS (Judea et al., 2014) andIBM (Sil and Florian, 2014) submitted runs tothe Spanish-to-English cross-lingual entity linkingtask. Table 6 presents the B-cubed+ scores forthese two systems. Both systems followed theirEnglish Entity Linking approaches. It’s veryencouraging to see that IBM system achievedsimilar performance with the top English EDL

Table 6: Cross-lingual Entity LinkingPerformance

system, although the difficulty level of queries arenot comparable.

This year many teams expressed interests inparticipating the Chinese-to-English cross-lingualentity linking task, but they all ended up withfocusing on English EDL task. In KBP2015we will propose a new tri-lingual EDL task byextending the source collection from English onlyto three languages: English, Chinese and Spanish.

7 Conclusions and Future Work

The new EDL task has attracted many interestsfrom the KBP community and produced someinteresting research problems and new directions.In KBP2015 we will focus on the followingextension and improvement:

• Improve the annotation guideline andannotation quality of the training andevaluation data sets;

• Develop more open sources, data andresources for Spanish and Chinese EDL;

• Encourage researchers to re-visit the entitymention extraction problem in the newcold-start KBP setting;

• Propose a new tri-lingual EDL task ona source collection from three languages:English, Chinese and Spanish;

• Investigate the impact of EDL on theend-to-end cold-start KBP framework.

Acknowledgments

Special thanks to Hoa Trang Dang forcoordinating and overseeing this track, andthanks to Xiaoqiang Luo and Ralph Grishman foradvice on the EDL scoring metrics.

ReferencesI. Anastacio, B. Martins, and P. Calado. 2011.

Supervised learning for linking named entities toknowledge base entries. In Proc. Text AnalysisConference (TAC2011).

Amit Bagga and Breck Baldwin. 1998. Algorithmsfor scoring coreference chains. In Proc. theLREC Workshop on Linguistic Coreference, pages563–566.

Collin F. Baker and Hiroaki Sato. 2003. The framenetdata and software. In Proc. ACL2003.

L. Banarescu, C. Bonial, S. Cai, M. Georgescu,K. Griffitt, U. Hermjakob, K. Knight, P. Koehn,M. Palmer, and N. Schneider. 2013. Abstractmeaning representation for sembanking. In Proc.Linguistic Annotation Workshop.

A. Barrena, E. Agirre, and A. Soroa. 2014. Ubc entitydiscovery and linking and diagnostic entity linkingat tac-kbp 2014. In Proc. Text Analysis Conference(TAC2014).

Razvan Bunescu. 2006. Using encyclopedicknowledge for named entity disambiguation. InEACL, pages 9–16.

Jie Cai and Michael Strube. 2010. Evaluation metricsfor end-to-end coreference resolution systems. InProc. SIGDIAL, pages 28–36.

T. Cassidy, H. Ji, L. Ratinov, A. Zubiaga, andH. Huang. 2012. Analysis and enhancement ofwikification for microblogs with context expansion.In Proceedings of COLING 2012.

D. Ceccarelli, C. Lucchese, S. Orlando, R. Perego,and S. Trani. 2013. Learning relatedness measuresfor entity linking. In Proceedings of the 22NdACM International Conference on Conference onInformation & Knowledge Management, CIKM’13.

Z. Chen and H. Ji. 2011. Collaborative ranking: Acase study on entity linking. In Proc. EMNLP2011.

L. Chen, Y. Feng, S. Huang, Y. Qin, and D. Zhao.2014. Encoding relation requirements for relationextraction via joint inference. In Proc. the52nd Annual Meeting of the Association forComputational Linguistics (ACL2014).

X. Cheng and D. Roth. 2013. Relationalinference for wikification. In Proceedings of the2013 Conference on Empirical Methods in NaturalLanguage Processing.

Silviu Cucerzan. 2007. Large-scale named entitydisambiguation based on wikipedia data. InEMNLP-CoNLL 2007.

S. Cucerzan. 2011. Tac entity linking by performingfull-document entity extraction and disambiguation.In Proc. TAC 2011 Workshop.

M. Dojchinovski, I. Lasek, O. Zamazal, and T. Kliegr.2014. Entityclassifier.eu and semitags: Entitydiscovery, linking and classification with wikipediaand dbpedia. In Proc. Text Analysis Conference(TAC2014).

M. Dredze, P. McNamee, D. Rao, A. Gerber, andT. Finin. 2010. Entity disambiguation forknowledge base population. In Proceedings of the23rd International Conference on ComputationalLinguistics.

J. Ellis, J. Getman, and S. M. Strassel. 2014.Overview of linguistic resource for the tac kbp 2014evaluations: Planning, execution, and results. InProc. Text Analysis Conference (TAC2014).

N. Fernandez, J. A. Fisteus, L. Sanchez, and E. Martin.2010. Webtlab: A cooccurence-based approach tokbp 2010 entity-linking task. In Proc. TAC 2010Workshop.

P. Ferragina and U. Scaiella. 2010. Tagme:on-the-fly annotation of short text fragments (bywikipedia entities). In Proceedings of the 19thACM international conference on Information andknowledge management, CIKM ’10.

J. Flanigan, S. Thomson, J. Carbonell, C. Dyer, andN. A. Smith. 2014. A discriminative graph-basedparser for the abstract meaning representation. InProc. the 52nd Annual Meeting of the Associationfor Computational Linguistics (ACL2014).

T. Gao, J. R. Hullman, E. Adar, B. Hecht, andN. Diakopoulos. 2014. Newsviews: an automatedpipeline for creating custom geovisualizations fornews. In Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems (CHI2014).

Y. Guo, W. Che, T. Liu, and S. Li. 2011. Agraph-based method for entity linking. In Proc.IJCNLP2011.

S. Guo, M. Chang, and E. Kiciman. 2013. To linkor not to link? a study on end-to-end tweet entitylinking. In Proceedings of the 2013 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies.

Ben Hachey, Joel Nothman, and Will Radford. 2014.Cheap and easy entity evaluation. In Proc. ofthe 52nd Annual Meeting of the Association forComputational Linguistics (Vol. 2), pages 464–469.

X. Han and L. Sun. 2011. A generative entity-mentionmodel for linking entities with knowledge base. InProc. ACL2011.

X. Han and J. Zhao. 2009. Named entitydisambiguation by leveraging wikipedia semanticknowledge. In Proceedings of the 18th ACMconference on Information and knowledgemanagement, CIKM 2009.

X. Han, L. Sun, and J. Zhao. 2011. Collective entitylinking in web text: A graph-based method. In Proc.SIGIR2011.

T. Hasegawa, S. Satoshi Sekine, and R. Grishman.2004. Discovering relations among named entitiesfrom large corpora. In Proc. the Annual Meeting ofAssociation of Computational Linguistics (ACL 04).

J. Hoffart, M. Yosef, I. Bordino, H. Furstenau,M. Pinkal, M. Spaniol, B. Taneva, S. Thater,and G. Weikum. 2011. Robust disambiguationof named entities in text. In Proceedings ofthe Conference on Empirical Methods in NaturalLanguage Processing, EMNLP ’11.

Y. Hong, X. Wang, Y. Chen, J. Wang, T. Zhang,J. Zheng, D. Yu, Q. Li, B. Zhang, H. Wang, X. Pan,and H. Ji. 2014. Rpi-soochow kbp2014 systemdescription. In Proc. Text Analysis Conference(TAC2014).

H. Huang, Z. Wen, D. Yu, H. Ji, Y. Sun, J. Han, andH. Li. 2013. Resolving entity morphs in censoreddata. In Proceedings of the 51st Annual Meetingof the Association for Computational Linguistics(Volume 1: Long Papers).

H. Huang, Y. Cao, X. Huang, H. Ji, and C. Lin.2014a. Collective tweet wikification based onsemi-supervised graph regularization. In Proc.the 52nd Annual Meeting of the Association forComputational Linguistics (ACL2014).

H. Huang, Y. Cao, X. Huang, H. Ji, and C.Y. Lin.2014b. Collective tweet wikification based onsemi-supervised graph regularization. Proceedingsof the 52nd Annual Meeting of the Associationfor Computational Linguistics (Volume 1: LongPapers).

H. Ji and D. Lin. 2009. Gender and animacyknowledge discovery from web-scale n-grams forunsupervised person mention detection. In Proc.PACLIC 2009.

H. Ji, R. Grishman, and H. T. Dang. 2011. Anoverview of the tac2011 knowledge base populationtrack. In Proc. Text Analysis Conf. (TAC’11),Gaithersburg, Maryland, Nov.

A. Judea, B. Heinzerling, A. Fahrni, and M. Strube.2014. Hits’ monolingual and cross-lingual entitylinking system at tac 2014. In Proc. Text AnalysisConference (TAC2014).

Z. Kozareva, K. Voevodski, and S. Teng. 2011. Classlabel enhancement via related instances. In Proc.EMNLP2011.

S. Kulkarni, A. Singh, G. Ramakrishnan, andS. Chakrabarti. 2009. Collective annotation ofwikipedia entities in web text. In KDD.

Q. Li, H. Li, H. Ji, W. Wang, J. Zheng, andF. Huang. 2012. Joint bilingual nametagging for parallel corpora. In Proc. 21stACM International Conference on Information andKnowledge Management (CIKM’12), Maui, Hawaii,Oct.

H. Liu and P. Singh. 2004. Conceptnet - a praticalcommonsense reasoning toolkit. 22:211–226.

X. Liu, Y. Li, H. Wu, M. Zhou, F. Wei, and Y. Lu.2013. Entity linking for tweets. In Proceedingsof the 51st Annual Meeting of the Associationfor Computational Linguistics (Volume 1: LongPapers).

Xiaoqiang Luo. 2005. On coreference resolutionperformance metrics. In Proc. HLT/EMNLP, pages25–32.

K. Mazaitis, R. C. Wang, B. Dalvi, and W. W.Cohen. 2014. A tale of two entity linkingand discovery systems. In Prof. Text AnalysisConference (TAC2014).

P. McNamee and H.T. Dang. 2009. Overview of thetac 2009 knowledge base population track. In TextAnalysis Conf. (TAC) 2009.

P. McNamee, J. Mayfield, D. Lawrie, D. W. Oard, andD. Doermann. 2011. Cross-language entity linking.In Proc. IJCNLP2011.

E. Meij, W. Weerkamp, and M. de Rijke. 2012.Adding semantics to microblog posts. InProceedings of the fifth ACM internationalconference on Web search and data mining, WSDM’12.

M. Meurs, H. Almeida, L. Jean-Louis, and E. Charton.2014. Semlinker entity discovery and linkingsystem at tac kbp2014. In Proc. Text AnalysisConference (TAC2014).

R. Mihalcea and A. Csomai. 2007. Wikify!:linking documents to encyclopedic knowledge.In Proceedings of the sixteenth ACM conferenceon Conference on information and knowledgemanagement, CIKM ’07.

D. Milne and I.H. Witten. 2008. Learning tolink with wikipedia. In Proceeding of the 17thACM conference on Information and knowledgemanagement, pages 509–518. ACM.

S. Monahan, D. Carpenter, M. Gorelkin, K. Crosby,and M. Brunson. 2014. Populating a knowledgebase with entities and events. In Proc. Text AnalysisConference (TAC2014).

T. H. Nguyen, Y. He, M. Pershina, X. Li, andR. Grishman. 2014. New york university 2014knowledge base population systems. In Proc. TextAnalysis Conference (TAC2014).

Joel Nothman, Nicky Ringland, Will Radford, TaraMurphy, and James R. Curran. 2013. Learningmultilingual named entity recognition fromWikipedia. Artificial Intelligence, 194:151–175,January.

M. Pennacchiotti and P. Pantel. 2009. Entity extractionvia ensemble semantics. In Proc. EMNLP2009.

W. Radford, B. Hachey, J. Nothman, M. Honnibal,and J. R. Curran. 2010. Cmcrc at tac10:Document-level entity linking with graph-basedre-ranking. In Proc. TAC 2010 Workshop.

Lev Ratinov and Dan Roth. 2009. Design challengesand misconceptions in named entity recognition. InProc. CONLL, pages 147–155.

L. Ratinov and D. Roth. 2012. Learning-basedmulti-sieve co-reference resolution with knowledge.In EMNLP.

L. Ratinov, D. Roth, D. Downey, and M. Anderson.2011. Local and global algorithms fordisambiguation to wikipedia. In Proc. of the AnnualMeeting of the Association of ComputationalLinguistics (ACL).

W. Shen, J. Wang, P. Luo, and M. Wang. 2013.Linking named entities in tweets with knowledgebase via user interest modeling. In Proceedings ofthe 19th ACM SIGKDD International Conference onKnowledge Discovery and Data Mining, KDD ’13.

A. Sil and R. Florian. 2014. The ibm systems forenglish entity discovery and linking and spanishentity linking at tac 2014. In Proc. Text AnalysisConference (TAC2014).

A. Sil and A. Yates. 2013. Re-ranking for jointnamed-entity recognition and linking. In Proc.CIKM2013.

Y. Song and D. Roth. 2014. On dataless hierarchicaltext classification. In Proc. AAAI 2014.

S. Tamang, Z. Chen, and H. Ji. 2012. Cuny blendertac-kbp2012 entity linking system and slot fillingvalidation system. In Proc. Text Analytics Conf.(TAC’12)), Gaithersburg, Maryland, Nov.

D. Vitale, P. Ferragina, and U. Scaiella. 2012.Classification of short texts by deploying topicalannotations. In ECIR, pages 376–387.

B. Zhang, H. Huang, X. Pan, H. Ji, K. Knight, Z. Wen,Y. Sun, J. Han, and B. Yener. 2014. Be appropriateand funny: Automatic entity morph encoding. InProc. the 52nd Annual Meeting of the Associationfor Computational Linguistics (ACL2014).

G. Zhao, P. Lv, R. Xu, Q. Zhi, and J. Wu. 2014. Msiiplthu at tac 2014 entity discovery and linking track. InProc. Text Analysis Conference (TAC2014).

Z. Zheng, F. Li, M. Huang, and X. Zhu. 2010.Learning to link entities with knowledge base. InProceedings of NAACL-HLT2010.

J. G. Zheng, D. Howsmon, B. Zhang, J. Hahn,D. McGuinness, J. Hendler, and H. Ji. 2014.Entity linking for biomedical literature. In Proc.CIKM2014 Workshop on Data and Text Mining inBiomedical Informatics.

S. Zhou, C. Kruengkrai, and K. Inui. 2014. Tohoku’sentity discovery and linking system at tac 2014. InProf. Text Analysis Conference (TAC2014).