Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2860–2871Florence, Italy, July 28 - August 2, 2019. c©2019 Association for Computational Linguistics

2860

Open Domain Event Extraction Using Neural Latent Variable Models

Xiao Liu1,2 and Heyan Huang1,2 and Yue Zhang3,4∗

1School of Computer Science and Technology, Beijing Institute of Technology2Zhejiang Lab, China

{xiaoliu,hhy63}@bit.edu.cn3School of Engineering, Westlake University

4Institute of Advanced Technology, Westlake Institute for Advanced Studyyue.zhang@wias.org.cn

Abstract

We consider open domain event extraction, thetask of extracting unconstraint types of eventsfrom news clusters. A novel latent variableneural model is constructed, which is scalableto very large corpus. A dataset is collected andmanually annotated, with task-specific evalua-tion metrics being designed. Results show thatthe proposed unsupervised model gives betterperformance compared to the state-of-the-artmethod for event schema induction.

1 Introduction

Extracting events from news text has receivedmuch research attention. The task typically con-sists of two subtasks, namely schema induction,which is to extract event templates that specifyargument slots for given event types (Chambers,2013; Cheung et al., 2013; Nguyen et al., 2015;Sha et al., 2016; Huang et al., 2016; Ahn, 2017;Yuan et al., 2018), and event extraction, which isto identify events with filled slots from a pieceof news (Nguyen et al., 2016b; Sha et al., 2018;Liu et al., 2018a; Chen et al., 2018, 2015; Fenget al., 2016; Nguyen and Grishman, 2016; Liuet al., 2018b). Previous work focuses on extract-ing events from single news documents accordingto a set of pre-specified event types, such as arson,attack or earthquakes.

While useful for tracking highly specific typesof events from news, the above setting can be rel-atively less useful for decision making in secu-rity and financial markets, which can require com-prehensive knowledge on broad-coverage, fine-grained and dynamically-evolving event cate-gories. In addition, given the fact that differentnews agencies can report the same events, redun-dancy can be leveraged for better event extrac-tion. In this paper, we investigate open domain

∗Corresponding author.

MUC 4 ODEE

Document

News

ReportNews

ReportNews

Report

News Cluster

Trigger raise

Agent UnitedHealth,

UnitedHealth shares

Patient2018 forecast,

better-than-expected profits,

the insurance business

Time the third quarter

Variation 28%

Unconstrained Types of Open Doamin

Events with Their Own Schemas

Trigger report

Agent UnitedHealth Group,

the largest U.S. health insurer

Patient better-than-expected

third-quarter earnings

Time Tuesday

Trigger predict

Agent UnitedHealth Group

Patient Medicare growth

Type

Perpetrator

Instrument

Target

Victim

Four Types of Events with Fixed Slots:

Arson, Attack, Bombing and Kidnapping

Figure 1: Comparison between MUC 4 and ODEE.

event extraction (ODEE), which is to extract un-constraint types of events and induce universalevent schemas from clusters of news reports.

As shown in Figure 1, compared with tradi-tional event extraction task exemplified by MUC4 (Sundheim, 1992), the task of ODEE poses ad-ditional challenges to modeling, which have notbeen considered in traditional methods. First,more than one event can be extracted from a newscluster, where events can be flexible in havingvarying numbers of slots in the open domain, andslots can be flexible without identical distributionsregardless of the event type, which has been as-sumed by previous work on schema induction.Second, mentions of the same entities from differ-ent reports in a news cluster should be taken intoaccount for improved performance.

We build an unsupervised generative model toaddress these challenges. While previous workon generative schema induction (Chambers, 2013;Cheung et al., 2013; Nguyen et al., 2015) relieson hand-crafted indicator features, we introducelatent variables produced by neural networks forbetter representation power. A novel graph model

2861

is designed, with a latent event type vector for eachnews cluster from a global parameterized normaldistribution, and textual redundancy features forentities. Our model takes advantage of contextu-alized pre-trained language model (ELMo, Peterset al. (2018)) and scalable neural variational infer-ence (Srivastava and Sutton, 2017).

To evaluate model performance, we collect andannotate a large-scale dataset from Google Busi-ness News1 with diverse event types and explain-able event schemas. In addition to the standardmetrics for schema matching, we adapt slot coher-ence based on NPMI (Lau et al., 2014) for quan-titatively measuring the intrinsic qualities of slotsand schemas, which are inherently clusters.

Results show that our neural latent variablemodel outperforms state-of-the-art event schemainduction methods. In addition, redundancy ishighly useful for improving open domain eventextraction. Visualizations of learned parametersshow that our model can give reasonable latentevent types. To our knowledge, we are the first touse neural latent variable model for inducing eventschemas and extracting events. We release ourcode and dataset at https://github.com/lx865712528/ACL2019-ODEE.

2 Related Work

The most popular schema induction and event ex-traction task setting is MUC 4, in which four eventtypes - Arson, Attack, Bombing and Kidnapping- and four slots - Perpetrator, Instrument, Targetand Victim - are defined. We compare the tasksettings of MUC 4 and ODEE in Figure 1. ForMUC 4, the inputs are single news documents,and the output belongs to four types of eventswith schemas consisting of fixed slots. For ODEE,in contrast, the inputs are news clusters ratherthan the individual news, and the output is uncon-strained types of open domain events and uniqueschemas with various slot combinations.

Event Schema Induction seminal work stud-ies patterns (Shinyama and Sekine, 2006; Fila-tova et al., 2006; Qiu et al., 2008) and eventchains (Chambers and Jurafsky, 2011) for tem-plate induction. For MUC 4, the current dominantmethods include probabilistic generative methods(Chambers, 2013; Cheung et al., 2013; Nguyenet al., 2015) that jointly model predicate and ar-

1https://news.google.com/?hl=en-US&gl=US&ceid=US:en, crawled from Oct. 2018 to Jan. 2019.

gument assignment, and ad-hoc clustering algo-rithms for inducing slots (Sha et al., 2016; Huanget al., 2016; Ahn, 2017; Yuan et al., 2018). Thesemethods all rely on hand-crafted discrete featureswithout fully model the textual redundancy. Thereare also works on modeling event schemas andscripts using neural language models (Modi andTitov, 2014; Rudinger et al., 2015; Pichotta andMooney, 2016), but they do not explore neural la-tent variables and redundancy.

Event Extraction work typically assumes thatevent schemas are given, recognizing event trig-gers and their corresponding arguments. This canbe regarded as a subtask of ODEE. Existing workexploits sentence-level (McClosky et al., 2011; Liet al., 2013; Liu et al., 2016; Yang and Mitchell,2016) and document-level statistics (Liao and Gr-ishman, 2010b; Ji and Grishman, 2008; Honget al., 2011; Reichart and Barzilay, 2012). Therehas also been work using RNNs (Nguyen et al.,2016b; Sha et al., 2018; Liu et al., 2018a; Chenet al., 2018), CNNs (Chen et al., 2015; Feng et al.,2016; Nguyen and Grishman, 2016) and GCNs(Liu et al., 2018b) to represent sentences of events.Event extraction has been treated as a supervisedor semi-supervised (Liao and Grishman, 2010a;Huang and Riloff, 2012) task. In contrast, ODEEis a fully unsupervised setting.

Event Discovery in Tweet Streams extractsnews-worthy clusters of words, segments andframes. Both supervised and unsupervised meth-ods have been used. The former (Sakaki et al.,2010; Benson et al., 2011) are typically designedto monitor certain event types, while the lat-ter cluster features according to their burstiness(Becker et al., 2011; Cui et al., 2012; Li et al.,2012; Ritter et al., 2012; Qin et al., 2013; Ifrimet al., 2014; McMinn and Jose, 2015; Qin et al.,2017). This line of work is similar to our workin using information redundancy, but different be-cause we focus on formal news texts and inducestructural event schemas.

First Story Detection (FSD) systems aim toidentify news articles that discuss events not re-ported before. Most work on FSD detects firststories by finding the nearest neighbors of newdocuments (Kumaran and Allan, 2005; Moranet al., 2016; Panagiotou et al., 2016; Vuurens andde Vries, 2016). This line of work exploits textualredundancy in massive streams predicting whetheror not a document contains a new event as a clas-

2862

sification task. In contrast, we study the eventschemas and extract detailed events.

3 Task and Data

Task Definition. In ODEE, the input consists ofnews clusters, each containing reports about thesame event. The output is a bag of open-domainevents, each consisting of an event trigger and alist of event arguments in its own schema. In mostcases, one event is semantically sufficient to rep-resent the output.

Formally, given an open-domain news corpusN containing a set of news clusters {c ∈ N},suppose that there are Mc news reports {di ∈c|i = 1, · · · ,Mc} in the news cluster c focus-ing on the same event Ec. The output is a pair(Ec, TE ), where Ec is the aforementioned set ofopen-domain events and TE is a set of schemas thatdefine the semantic slots for this set of events.Data Collection. We crawl news reports fromGoogle Business News, which offers news clustersabout the same events from different sources. Ineach news cluster, there are no more than five newsreports. For each news report, we obtain the title,publish timestamp, download timestamp, sourceURL and full text. In total, we obtain 55,618 busi-ness news reports with 13,047 news clusters in 288batches from Oct. 17, 2018, to Jan. 22, 2019.The crawler is executed about three times per day.The full text corpus is released as GNBusiness-Full-Text. For this paper, we trim the news reportsin each news cluster by keeping the title and firstparagraph, releasing as GNBusiness-All.

Inspired by the general slots in FrameNet(Baker et al., 1998), we design reference eventschemas for open domain event types, which in-clude eight possible slots: Agent, Patient, Time,Place, Aim, Old Value, New Value and Variation.Agent and Patient are the semantic agent and pa-tient of the trigger, respectively; Aim is the tar-get or reason for the event. If the event involvesvalue changes, Old Value serves the old value,New Value serves the new value and Variation isthe variation between New Value and Old Value.Note that the roles that we define are more the-matic and less specific to detailed events as someof the existing event extraction datasets do (Sund-heim, 1992; Nguyen et al., 2016a), because wewant to make our dataset general and useful for awide range of open domain conditions. We leavefiner-grained role typing to future work.

Split #C #R #S #WTest 574 2,433 5,830 96,745Dev 106 414 991 16,839Unlabelled 12,305 52,464 127,416 2,101,558All 12,985 55,311 134,237 2,215,142Full-Text 12,985 55,311 1,450,336 31,103,698

Table 1: Data split statistics. (C news clusters; R newsreports; S sentences; W words.)

Dataset #D #L #T #SMUC 4 1700 400 4 4ACE 2005 599 599 33 36ERE 562 562 38 27ASTRE 1038 100 12 18GNBusiness 12,985 680 – 8

Table 2: Comparison with existing datasets. (D doc-uments or news clusters; L labeled documents or newsclusters; T event types; S slots.)

We randomly select 18 batches of news clusters,with 680 clusters in total, dividing them into a de-velopment set and a test set by a ratio of 1 : 5.The development set, test set and the rest unla-beled clusters are released as GNBusiness-Dev,GNBusiness-Test and GNBusiness-Unlabeled, re-spectively. One coauthor and an external anno-tator manually label the events in the news clus-ters as gold standards. For each news cluster, theyassign each entity which participants in the eventor its head word a beforehand slot. The inter-annotator agreement (IAA) for each slot realiza-tion in the development set has a Cohen’s kappa(Cohen, 1960) κ = 0.7.

The statistics of each data split is shown in Ta-ble 1, and a comparison with existing event ex-traction and event schema induction datasets, in-cluding ASTRE (Nguyen et al., 2016a), MUC 4,ACE 20052 and ERE3, is shown in Table 2. Com-pared with the other datasets, GNBusiness has amuch larger number of documents (i.e., news clus-ters in GNBusiness), and a comparable number oflabeled documents.

4 Method

We investigate three incrementally more complexneural latent variable models for ODEE.

4.1 Model 1

Our first model is shown in Figure 2(a). It canbe regarded as a neural extension of Nguyen et al.

2https://catalog.ldc.upenn.edu/LDC2006T06

3https://catalog.ldc.upenn.edu/LDC2013E64

2863

s S

h

fβ

λ

E

(a) ODEE-F

s

h

f

t α

θ

β

λ

E

C

(b) ODEE-FE

s

h

f

t α

θβ

rγ

λ

E

C

(c) ODEE-FER

Figure 2: Plate notations for models. (S – # of slots; E – # of entities; C – # of news clusters; V – head wordvocabulary size; the grey circles are observed variables and the white circles are hidden variables.)

Algorithm 1 ODEE-F1: for each entity e ∈ E do2: Sample a slot s ∼ Uniform(1, S)3: Sample a head h ∼ Multinomial(1, λs)4: Sample a feature vector f ∼ Normal(β)5: end for

(2015). Given a corpus N , we sample a slot sfor each entity e from a uniform distribution ofS slots, and then a head word h from a multino-mial distribution, as well as a continuous featurevector f ∈ Rn produced by a contextual encoder.For simplicity, we assume that f follows a multi-variable normal distribution whose covariance ma-trix is a diagonal matrix. We mark all the parame-ters (mean vectors and diagonal vectors of covari-ance matrixes) for the S different normal distri-butions for f as β ∈ RS×2n, where n representsthe dimension of f , treating the probability matrixλ ∈ RS×V in the slot-head distribution as parame-ters under the row-wise simplex constraint, whereV is the head word vocabulary size. We call thismodel ODEE-F.

Pre-trained contextualized embeddings such asELMo (Peters et al., 2018), GPTs (Radford et al.,2018, 2019) and BERT (Devlin et al., 2018) giveimprovements on a range of natural language pro-cessing tasks by offering rich language model in-formation. We choose ELMo4 as our contex-tual feature encoder, which manipulates unknownwords by using character representations.

The generative story is shown in Algorithm 1.The joint probability of an entity e is

pλ,β(e) = p(s)× pλ(h|s)× pβ(f |s) (1)

4In practice, we use the “small” ELMo model with 2 ×128-d output in https://allennlp.org/elmo as ini-tial parameters and fine-tune it on GNBusiness-Full-Text.

Algorithm 2 ODEE-FE1: for each news cluster c ∈ N do2: Sample a latent event type vector t ∼ Normal(α)3: for each entity e ∈ Ec do4: Sample a slot s ∼ Multinomial(MLP(t; θ))5: Sample a head h ∼ Multinomial(1, λs)6: Sample a feature vector f ∼ Normal(βs)7: end for8: end for

4.2 Model 2

A limitation of ODEE-F is that sampling slot as-signment s from a global uniform distribution doesnot sufficiently model the fact that different eventsmay have different slot distributions. Thus, in Fig-ure 2(b), we further sample a latent event type vec-tor t ∈ Rn for each news cluster from a globalnormal distribution parameterized by α. We thenuse t and a multi-layer perceptron (MLP) with pa-rameters θ to encode the corresponding slot distri-bution logits, sampling a discrete slot assignments ∼ Multinomial(MLP(t; θ)). The output of theMLP is passed through a softmax layer before be-ing used. We name this model as ODEE-FE.

The generative story is shown in Algorithm 2.The joint probability of a news cluster c is

pα,β,θ,λ(c) = pα(t)×∏e∈Ec

pθ(s|t)

× pλ(h|s)× pβ(f |s) (2)

4.3 Model 3

Intuitively, the more frequently a coreferential en-tity shows up in a news cluster, the more likelyit is with an important slot. Beyond that, differ-ent news agencies focus on different aspects ofevent arguments, which can offer complementaryinformation through textual redundancy. One intu-

2864

Algorithm 3 ODEE-FER1: for each news cluster c ∈ N do2: Sample a latent event type vector t ∼ Normal(α)3: for each entity e ∈ Ec do4: Sample a slot s ∼ Multinomial(MLP(t; θ))5: Sample a head h ∼ Multinomial(1, λs)6: Sample a feature vector f ∼ Normal(βs)7: Sample a redundancy ratio r ∼ Normal(γs)8: end for9: end for

ition is that occurrence frequency is a straightfor-ward measure for word-level redundancy. Thus,in Figure 2(c), we additionally bring in the nor-malized occurrence frequency of a coreferentialslot realization as an observed latent variable r ∼Normal(γs). We call this model ODEE-FER.

Formally, a news cluster c receives a latent eventtype vector t where each entity e ∈ Ec receives aslot type s. The generative story is shown in Al-gorithm 3. The joint distribution of a news clus-ter with head words, redundant contextual featuresand latent event type is

pα,β,γ,θ,λ(c) = pα(t)×∏e∈Ec

pθ(s|t)

× pλ(h|s)× pβ(f |s)× pγ(r|s) (3)

4.4 InferenceWe now consider two tasks for ODEE-FER: (1)learning the parameters and (2) performing infer-ence to obtain the posterior distribution of the la-tent variables s and t, given a news cluster c. Weadapt the amortized variational inference methodof Srivastava and Sutton (2017), using neural in-ference network to learn the variational parame-ters. For simplicity, we concatenate f with r as anew observed feature vector f ′ in ODEE-FER andmerge their parameters as β′ ∈ RS×(2n+2).

Following Srivastava and Sutton (2017), we col-lapse the discrete latent variable s to obtain anEvidence Lower BOund (ELBO) (Kingma andWelling, 2014) of the log marginal likelihood:

log pα,β′,θ,λ(c)

= log∫t[∏e∈Ec

pλ,θ(h|t) pβ′,θ(f ′|t)] pα(t) dt

≥ ELBOc(α, β′, θ, λ, ω)

= Eqω(t)log pβ′,θ,λ(c|t)−DKL[qω(t)‖pα(t)](4)

where DKL[qω‖pα] is the KL divergence betweenthe variational posterior qω and the prior pα. Due

BatchNorm & ExpBatchNorm

Fully Connected Average Pooling

Dropout Dropout

Concatenate

Head Word Histograms: ℎ Contextual Features: "’

Mean Vector: $ Variance Vector: %&

SoftPlusSoftPlus

Fully Connected

Fully Connected Fully Connected

Figure 3: The framework of our inference network.

to the difficulty in computing the KL divergencebetween different categories of distributions andthe existence of simple and effective reparameter-ization tricks for normal distributions, we chooseqω(t) to be a normal distribution parameterized byω, which is learned by a neural inference network.As shown in Figure 3, our inference network takesthe head word histograms h (the times of eachhead word appears in a news cluster) and contex-tual features f ′ as inputs, and computes the meanvector µ and the variance vector σ2 of qω(t).

Equation 4 can be solved by obtaining a MonteCarlo sample and applying reparameterizationtricks for the first term, and using the closed-form for the KL divergence term. We then usethe ADAM optimizer (Kingma and Ba, 2014) tomaximumize the ELBO. In addition, to allevi-ate the component collapsing problem (Dinh andDumoulin, 2016), we follow Srivastava and Sut-ton (2017) and use high moment weight (> 0.8)and learning rate (in [0.001, 0.1]) in the ADAMoptimizer, performing batch normalization (Ioffeand Szegedy, 2015) and dropout (Srivastava et al.,2014). After learning the model, we make slot as-signment for each entity mention by MLE, choos-ing the slot s that maximizes the likelihood

pβ′,θ,λ(s|e, t) ∝ pβ′,θ,λ(s, h, f ′, t)= pθ(s|t)× pλ(h|s)× pβ′(f ′|s)

(5)

2865

Name ValueSlots number S 30

Feature Dimension n 256Fully connected layer size 100

MLP layer number 1Activation function softplus

Learning rate 0.002Momentum 0.99Dropout rate 0.2Batch size 200

Table 3: Hyper-parameters setting.

4.5 Assembling Events for Output

To assemble the events in a news cluster c for fi-nal output, we need to find the predicate for eachentity, which now has a slot value. We use POS-tags and parse trees produced by the Stanford de-pendency parser (Klein and Manning, 2003) to ex-tract the predicate for the head word of each entitymention. The following rules are applied: (1) ifthe governor of a head word is VB, or (2) if thegovernor of a head word is NN and belongs to thenoun.ACT or noun.EVENT category of WordNet,then it is regarded as a predicate.

We merge the predicates of entity mentions inthe same coreference chain as a predicate set. Foreach predicate v in these sets, we find the entitieswhose predicate set contains v, treating the entitiesas arguments of the event triggered by v. Finally,by ranking the numbers of arguments, we obtaintop-N open-domain events as the output Ec.

5 Experiments

We verify the effectiveness of neural latent vari-able modeling and redundancy information forODEE, and conduct case analysis. All our exper-iments are conducted on the GNBusiness dataset.Note that we do not compare our models and ex-isting work on MUC 4 or ACE 2005 due to the factthat these datasets do not consist of news clusters.

Settings. The hyper-parameters in our models andinference network are shown in Table 3. Mostof the hyper-parameters directly follow Srivastavaand Sutton (2017), while the slot number S is cho-sen according to development experiments.

5.1 Evaluation Metrics

Schemas Matching. We follow previous workand use precision, recall and F1-score as the met-rics for schema matching (Chambers and Juraf-sky, 2011; Chambers, 2013; Cheung et al., 2013;Nguyen et al., 2015; Sha et al., 2016; Ahn, 2017).

The matching between model answers and refer-ences is based on the head word. Following pre-vious work, we regard as the head word the right-most word of an entity phrase or the right-mostword before the first “of”, “that”, “which” and“by” if any.

In addition, we also perform slot mapping, be-tween slots that our model learns and slots in theannotation. Following previous work on MUC 4(Chambers, 2013; Cheung et al., 2013; Nguyenet al., 2015; Sha et al., 2016; Ahn, 2017), weimplement automatic greedy slot mapping. Eachreference slot is mapped to a learned slot thatranks the best according to the F1-score metric onGNBusiness-Dev.

Slot Coherence. Several metrics of qualitativetopic coherence evaluation have been proposed.Lau et al. (2014) showed that normalized point-wise mutual information (NPMI) between all thepairs of words in a set of topics the most closelymatches human judgment among all the compet-ing metrics. We thus adopt it as slot coherence5.

Formally, the slot coherence CNPMI(s) of a slots is calculated by using its top-N head words as

CNPMI(s) =2

N2 −N

N∑i=2

i−1∑j=1

NPMI(wi, wj)

(6)

NPMI(wi, wj) =log p(wi,wj)+ε

p(wi)·p(wj)

−log(p(wi, wj) + ε)(7)

where p(wj) and p(wi, wj) are estimated basedon word co-occurrence counts derived within asliding window over external reference documentsand ε is added to avoid zero logarithm.

Previous work on topic coherence usesWikipedia and Gigaword as the reference corpusto calculate word frequencies (Newman et al.,2010; Lau et al., 2014). We use GNBusiness-Full-Text, in which there are 1.45M sentencesand 31M words, which is sufficient for estimatingthe probabilities. To reduce sparsity, for eachnews report, we count word co-occurrences in thewhole document instead of a sliding window. Inaddition, for each slot, we keep the top-5, top-10,top-20, and top-100 head words, averaging the4× S coherence results over a test set.

5We use the implementation in https://github.com/jhlau/topic_interpretability.

2866

10 15 20 25 30 35 40 45 50number of slots S

10

20

30

40

50

Performance (%)

Nguyen et al. (2015) F1Nguyen et al. (2015) CNPMIClustering F1ODEE-F F1ODEE-F CNPMIODEE-FE F1ODEE-FE CNPMIODEE-FER F1ODEE-FER CNPMI

Figure 4: F1 scores of schemas matching and averagedslot coherences CNPMI of the five models with differentnumbers of slots S.

Method Schema Matching (%)P R F1

Nguyen et al. (2015) 41.5 53.4 46.7Clustering 41.2 50.6 45.4ODEE-F 41.7 53.2 46.8ODEE-FE 42.4 56.1 48.3ODEE-FER 43.4 58.3 49.8

Table 4: Overall performance of schema matching.

5.2 Development Experiments

We learn the models on GNBusiness-All and useGNBusiness-Dev to determine the slot number Sby grid search in [10, 50] with the step equals to 5.Figure 4 shows the F1 scores of schemas matchingand averaged slot coherences of the five modelswe introduce in the next subsection with differentnumbers of slots S ranging from 10 to 50. Wecan see that for the best F1 score of ODEE-FER,the optimal number of slots is 30, while for thebest slot coherence, the optimal number of slots is25. A value of S larger than 30 or smaller than25 gives lower results on both F1 score and slotcoherence. Considering the balance between F1

score and slot coherence, we chose S = 30 as ourfinal S value for the remaining experiments.

5.3 Final Results

Table 4 and Table 5 show the final results. Thep values based on the appropriate t-test are pro-

Method Ave Slot CoherenceNguyen et al. (2015) 0.10ODEE-F 0.10ODEE-FE 0.16ODEE-FER 0.18

Table 5: Averaged slot coherence results.

vided below in cases where the compared valuesare close. We compare our work with Nguyenet al. (2015), the state-of-the-art model on MUC4 representing each entity as a triple containing ahead word, a list of attribute relation features anda list of predicate relation features. Features in themodel are discrete and extracted from dependencyparse trees. The model structure is identical to ourODEE-F except for the features.

To test the strengths of our external featuresin isolation, we build another baseline modelby taking the continuous features of each entityin ODEE-F and runing spectral clustering (vonLuxburg, 2007). We call it Clustering.

Schemas Matching. Table 4 shows the overallperformance of schema matching on GNBusiness-Test. From the table, we can see that ODEE-FERachieves the best F1 scores among all the methods.By comparing Nguyen et al. (2015) and ODEE-F (p = 0.01), we can see that using continuouscontextual features gives better performance thandiscrete features. This demonstrates the advan-tages of continuous contextual features for allevi-ating the sparsity of discrete features in texts. Wecan also see from the result of Clustering that us-ing only the contextual features is not sufficientfor ODEE, while combining with our neural latentvariable model in ODEE-F can achieve strong re-sults (p = 6×10−6). This shows that the neural la-tent variable model can better explain the observeddata.

These results demonstrate the effectivenesses ofour method in incorporating with contextual fea-tures, latent event types and redundancy infor-mation. Among ODEE models, ODEE-FE givesa 2% gain in F1 score against ODEE-F, whichshows that the latent event type modeling is ben-eficial and the slot distribution relies on the latentevent type. Additionally, there is a 1% gain in F1

score by comparing ODEE-FER and ODEE-FE(p = 2 × 10−6), which confirms that leveragingredundancy is also beneficial in exploring whichslot an entity should be assigned.

Slot Coherence. Table 5 shows the comparison ofaveraged slot coherence results over all the slotsin the schemas. Note that we do not report theslot coherence for the Clustering model becauseit does not output the top-N head words in eachslot. The averaged slot coherence of ODEE-FERis the highest, which is consistent with the conclu-sion from Table 4. The averaged slot coherence

2867

Boston Dynamics' reveals its robodog Spot dancing

Arby's will debut sous vide duck sandwich

Prime Deli Corporation Recalls Salads

Massive recall issued for frozen beef, chicken taquitos

Wendy's Offering $1 Any Size Fry For A Limited Time

Netflix shares surge

IBM drops 4.3% aftermarket

Walmart lowered its profit targets

UnitedHealth shares rise

Intel shares gain

Figure 5: T-SNE visualization results of the latent eventtype vectors in the test set with colored labels producedby spectral clustering.

of ODEE-F is comparable to that of Nguyen et al.(2015) (p = 0.3415), which again demonstratesthat the contextual features are a strong alterna-tive to discrete features. The scores of ODEE-FE(p = 0.06) and ODEE-FER (p = 10−5) are bothhigher than that of ODEE-F, which proves that thelatent event type is critical in ODEE.

5.4 Latent Event Type Analysis

We are interested in learning how well the latentevent type vectors can be modeled. To this end,for each news cluster in GNBusiness-Dev, we useour inference network in Figure 3 to calculate themean µ for the latent event type vector t. T-SNEtransformation (Maaten and Hinton, 2008) of themean vectors are shown in Figure 5. Spectralclustering is further applied, and the number ofclusters is chosen by the Calinski-Harabasz Score(Calinski and Harabasz, 1974) in grid search.

In Figure 5, there are four main clusters markedin different colors. Representative titles of newsreports are shown as examples. We find that thevectors show salient themes for each main clus-ter. For example, the red cluster contains news re-ports about rise and drop of stocks such as Netflixshares surge, IBM drops, Intel shares gain, etc; thenews reports in the purple cluster are mostly aboutproduct related activities, such as Boston Dynam-ics’ reveals its robodog Spot dancing, Arby’s willdebut sous vide duck sandwich, Wendy’s Offering$1 Any Size Fry, etc. The green cluster and the

DOC 12018-10-16 07:00:03UnitedHealth shares rise after posting a 28% rise in third-quarter profit, raises 2018 forecast

UnitedHealth, the largest U.S. health insurer, reported better-than-expected third-quarter earnings and revenue on Tuesday.

DOC 22018-10-16 00:00:00UnitedHealth's 2018 so far: Three quarters, three boosts to outlook

DOC 32018-10-17 00:32:09UnitedHealth Group predicts Medicare growth

The comments came as the insurer beat profit expectations for Q3.

DOC 42018-10-16 10:53:06UnitedHealth beats all around in 3Q, raises outlook again

MINNEAPOLIS (AP) —UnitedHealth reported better-than-expected profits and revenue for the third quarter and the company raised its outlook yet again on strong trends in the insurance business.

Trigger raise

Agent UnitedHealth,UnitedHealth shares

Patient2018 forecast,

better-than-expected profits,the insurance business

Time the third quarter

Variation 28%

Event 1

Trigger report

Agent UnitedHealth Group,the largest U.S. health insurer

Patient better-than-expectedthird-quarter earnings

Time Tuesday

Trigger predict

Agent UnitedHealth Group

Patient Medicare growth

Event 2

Event 3

Figure 6: Extracted open domain events for United-Health shares rise.

orange cluster are also interpretable. The formeris about organization reporting changes, while thelatter is about service related activities.

5.5 Case Study

We further use the news cluster UnitedHealthshares rise in Figure 5 for case study. Figure6 shows the top-3 open-domain events extractedfrom the news cluster, where four input news re-ports are shown on the left and three system-generated events are shown on the right withmapped slots.

By comparing the plain news reports and the ex-tracted events, we can see that the output eventsgive a reasonable summary for the news clusterwith three events triggered by “raise”, “report” and“predict”, respectively. Most of the slots are mean-ingful and closely related to the trigger, while cov-ering most key aspects. However, this examplealso contains several incorrect slots. In the event1, the slot “Variation” and its realization “28%”are only related to the entity “better-than-expectedprofits”, but there are three slot realizations inthe event, which causes confusion. In addition,the slot “Aim” does not appear in the first event,whose realization should be “third-quarter profit”in document 1. The reason may be that we as-semble an event only using entities with the samepredicate, which introduces noise. Besides, due to

2868

the preprocessing errors in resolving coreferencechains, some entity mentions are missing from theoutput.

There are also cases where one slot realizationis semantically related to one trigger but eventu-ally appears in a different event. One example isthe entity “better-than-expected profits”, which isrelated to the predicate word “report” but finallyappears in the “raise” event. The cause can beerrors propagated from parsing dependency trees,which confuse the syntactic predicate of the headword of an entity.

6 Conclusion

We presented the task of open domain event ex-traction, extracting unconstraint types of eventsfrom news clusters. A novel latent variable neu-ral model was investigated, which explores la-tent event type vectors and entity mention redun-dancy. In addition, GNBusiness dataset, a large-scale dataset annotated with diverse event typesand explainable event schemas, is released alongwith this paper. To our knowledge, we are the firstto use neural latent variable model for inducingevent schemas and extracting events.

Acknowledgments

We thank the anonymous reviewers for their valu-able comments and suggestions. We thank Kiem-Hieu Nguyen from Hanoi University of Scienceand Technology for providing source code andsolving confusions for their work. We thankKatherine Keith from University of Massachusettsat Amherst for sharing valuable experiences onprobabilistic models. This work is supported byNational Natural Science Foundation of China No.61751201, National Key Research and Develop-ment Plan No. 2016QY03D0602, Research Foun-dation of Beijing Municipal Science and Technol-ogy Commission No. Z181100008918002, thefunding from Rxhui Inc6 and China ScholarshipCouncil No. 201806030142. The work is donewhen Xiao Liu is visiting Yue Zhang.

ReferencesNatalie Ahn. 2017. Inducing event types and roles in

reverse: Using function to discover theme. In Pro-ceedings of the Events and Stories in the News Work-shop@ACL 2017, pages 66–76.

6https://rxhui.com

Collin F. Baker, Charles J. Fillmore, and John B. Lowe.1998. The berkeley framenet project. In Proceed-ings of the 36th Annual Meeting of the Associa-tion for Computational Linguistics and 17th Inter-national Conference on Computational Linguistics,pages 86–90.

Hila Becker, Mor Naaman, and Luis Gravano. 2011.Beyond trending topics: Real-world event identifi-cation on twitter. In Proceedings of the 5th Inter-national Conference on Weblogs and Social Media,pages 438–441.

Edward Benson, Aria Haghighi, and Regina Barzilay.2011. Event discovery in social media feeds. InProceedings of the 49th Annual Meeting of the As-sociation for Computational Linguistics, pages 389–398.

Tadeusz Calinski and Jerzy Harabasz. 1974. A den-drite method for cluster analysis. Communicationsin Statistics-theory and Methods, 3(1):1–27.

Nathanael Chambers. 2013. Event schema inductionwith a probabilistic entity-driven model. In Pro-ceedings of the 2013 Conference on Empirical Meth-ods in Natural Language Processing, pages 1797–1807.

Nathanael Chambers and Dan Jurafsky. 2011.Template-based information extraction withoutthe templates. In Proceedings of the 49th AnnualMeeting of the Association for ComputationalLinguistics, pages 976–986.

Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng,and Jun Zhao. 2015. Event extraction via dy-namic multi-pooling convolutional neural networks.In Proceedings of the 53rd Annual Meeting of theAssociation for Computational Linguistics and the7th International Joint Conference on Natural Lan-guage Processing of the Asian Federation of NaturalLanguage Processing, pages 167–176.

Yubo Chen, Hang Yang, Kang Liu, Jun Zhao, and Yan-tao Jia. 2018. Collective event detection via a hier-archical and bias tagging networks with gated multi-level attention mechanisms. In Proceedings of the2018 Conference on Empirical Methods in NaturalLanguage Processing, pages 1267–1276.

Jackie Chi Kit Cheung, Hoifung Poon, and Lucy Van-derwende. 2013. Probabilistic frame induction. InProceedings of the 2013 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 837–846.

Jacob Cohen. 1960. A coefficient of agreement fornominal scales. Educational and psychologicalmeasurement, 20(1):37–46.

Anqi Cui, Min Zhang, Yiqun Liu, Shaoping Ma, andKuo Zhang. 2012. Discover breaking events withpopular hashtags in twitter. In Proceedings of the21st ACM International Conference on Informationand Knowledge Management, pages 1794–1798.

2869

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018. BERT: pre-training ofdeep bidirectional transformers for language under-standing. CoRR, abs/1810.04805.

Laurent Dinh and Vincent Dumoulin. 2016. Trainingneural bayesian nets. Technical report.

Xiaocheng Feng, Lifu Huang, Duyu Tang, Heng Ji,Bing Qin, and Ting Liu. 2016. A language-independent neural network for event detection. InProceedings of the 54th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 66–71.

Elena Filatova, Vasileios Hatzivassiloglou, and Kath-leen R. McKeown. 2006. Automatic creation of do-main templates. In Proceedings of the 44th AnnualMeeting of the Association for Computational Lin-guistics and the 21st International Conference onComputational Linguistics, pages 207–214.

Yu Hong, Jianfeng Zhang, Bin Ma, Jian-Min Yao,Guodong Zhou, and Qiaoming Zhu. 2011. Usingcross-entity inference to improve event extraction.In roceedings of the 49th Annual Meeting of theAssociation for Computational Linguistics, pages1127–1136.

Lifu Huang, Taylor Cassidy, Xiaocheng Feng, Heng Ji,Clare R. Voss, Jiawei Han, and Avirup Sil. 2016.Liberal event extraction and event schema induction.In Proceedings of the 54th Annual Meeting of the As-sociation for Computational Linguistics, pages 258–268.

Ruihong Huang and Ellen Riloff. 2012. Bootstrappedtraining of event extraction classifiers. In Proceed-ings of the 13th Conference of the European Chap-ter of the Association for Computational Linguistics,pages 286–295.

Georgiana Ifrim, Bichen Shi, and Igor Brigadir. 2014.Event detection in twitter using aggressive filter-ing and hierarchical tweet clustering. In Proceed-ings of the SNOW 2014 Data Challenge co-locatedwith 23rd International World Wide Web Confer-ence, pages 33–40.

Sergey Ioffe and Christian Szegedy. 2015. Batch nor-malization: Accelerating deep network training byreducing internal covariate shift. In Proceedingsof the 32nd International Conference on MachineLearning, pages 448–456.

Heng Ji and Ralph Grishman. 2008. Refining event ex-traction through cross-document inference. In Pro-ceedings of the 46th Annual Meeting of the Associa-tion for Computational Linguistics, pages 254–262.

Diederik P. Kingma and Jimmy Ba. 2014. Adam:A method for stochastic optimization. CoRR,abs/1412.6980.

Diederik P Kingma and Max Welling. 2014. Auto-encoding variational bayes. In Proceedings of the2014 International Conference on Learning Repre-sentations.

Dan Klein and Christopher D. Manning. 2003. Ac-curate unlexicalized parsing. In Proceedings of the41st Annual Meeting of the Association for Compu-tational Linguistics, pages 423–430.

Giridhar Kumaran and James Allan. 2005. Usingnames and topics for new event detection. In Pro-ceedings of the 2005 Conference on Empirical Meth-ods in Natural Language Processing, pages 121–128.

Jey Han Lau, David Newman, and Timothy Baldwin.2014. Machine reading tea leaves: Automaticallyevaluating topic coherence and topic model quality.In Proceedings of the 14th Conference of the Euro-pean Chapter of the Association for ComputationalLinguistics, pages 530–539.

Chenliang Li, Aixin Sun, and Anwitaman Datta.2012. Twevent: segment-based event detection fromtweets. In Proceedings of the 21st ACM Interna-tional Conference on Information and KnowledgeManagement, pages 155–164.

Qi Li, Heng Ji, and Liang Huang. 2013. Joint eventextraction via structured prediction with global fea-tures. In Proceedings of the 51st Annual Meetingof the Association for Computational Linguistics,pages 73–82.

Shasha Liao and Ralph Grishman. 2010a. Filteredranking for bootstrapping in event extraction. InProceedings of the 23rd International Conference onComputational Linguistics, pages 680–688.

Shasha Liao and Ralph Grishman. 2010b. Using doc-ument level cross-event inference to improve eventextraction. In Proceedings of the 48th Annual Meet-ing of the Association for Computational Linguis-tics, pages 789–797.

Jian Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2018a.Event detection via gated multilingual attentionmechanism. In Proceedings of the 32nd AAAI Con-ference on Artificial Intelligence, pages 4865–4872.

Shulin Liu, Yubo Chen, Shizhu He, Kang Liu, andJun Zhao. 2016. Leveraging framenet to improveautomatic event detection. In Proceedings of the54th Annual Meeting of the Association for Compu-tational Linguistics, pages 2134–2143.

Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018b.Jointly multiple events extraction via attention-based graph information aggregation. In Proceed-ings of the 2018 Conference on Empirical Methodsin Natural Language Processing, pages 1247–1256.

Ulrike von Luxburg. 2007. A tutorial on spectral clus-tering. Statistics and Computing, 17(4):395–416.

Laurens van der Maaten and Geoffrey Hinton. 2008.Visualizing data using t-sne. Journal of machinelearning research, 9(Nov):2579–2605.

2870

David McClosky, Mihai Surdeanu, and Christopher D.Manning. 2011. Event extraction as dependencyparsing. In Proceedings of the 49th Annual Meet-ing of the Association for Computational Linguis-tics, pages 1626–1635.

Andrew James McMinn and Joemon M. Jose. 2015.Real-time entity-based event detection for twitter. InProceedings of the Experimental IR Meets Multilin-guality, Multimodality, and Interaction - 6th Inter-national Conference of the CLEF Association, pages65–77.

Ashutosh Modi and Ivan Titov. 2014. Inducing neu-ral models of script knowledge. In Proceedings ofthe Eighteenth Conference on Computational Natu-ral Language Learning, pages 49–57.

Sean Moran, Richard McCreadie, Craig Macdonald,and Iadh Ounis. 2016. Enhancing first story detec-tion using word embeddings. In Proceedings of the39th International ACM SIGIR conference on Re-search and Development in Information Retrieval,pages 821–824.

David Newman, Jey Han Lau, Karl Grieser, and Tim-othy Baldwin. 2010. Automatic evaluation of topiccoherence. In Proceedings of the 2010 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, pages 100–108.

Kiem-Hieu Nguyen, Xavier Tannier, Olivier Ferret,and Romaric Besancon. 2015. Generative eventschema induction with entity disambiguation. InProceedings of the 53rd Annual Meeting of theAssociation for Computational Linguistics and the7th International Joint Conference on Natural Lan-guage Processing of the Asian Federation of NaturalLanguage Processing, pages 188–197.

Kiem-Hieu Nguyen, Xavier Tannier, Olivier Ferret,and Romaric Besancon. 2016a. A dataset for openevent extraction in english. In Proceedings of the10th International Conference on Language Re-sources and Evaluation, pages 1939–1943.

Thien Huu Nguyen, Kyunghyun Cho, and Ralph Gr-ishman. 2016b. Joint event extraction via recurrentneural networks. In Proceedings of the 2016 Con-ference of the North American Chapter of the Asso-ciation for Computational Linguistics: Human Lan-guage Technologies, pages 300–309.

Thien Huu Nguyen and Ralph Grishman. 2016. Mod-eling skip-grams for event detection with convolu-tional neural networks. In Proceedings of the 2016Conference on Empirical Methods in Natural Lan-guage Processing, pages 886–891.

Nikolaos Panagiotou, Cem Akkaya, Kostas Tsiout-siouliklis, Vana Kalogeraki, and Dimitrios Gunop-ulos. 2016. First story detection using entitiesand relations. In Proceedings of the 26th Inter-national Conference on Computational Linguistics,pages 3237–3244.

Matthew E. Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word repre-sentations. In Proceedings of the 2018 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, pages 2227–2237.

Karl Pichotta and Raymond J. Mooney. 2016. Usingsentence-level LSTM language models for script in-ference. In Proceedings of the 54th Annual Meetingof the Association for Computational Linguistics.

Yanxia Qin, Yue Zhang, Min Zhang, and DequanZheng. 2013. Feature-rich segment-based newsevent detection on twitter. In Proceedings of the6th International Joint Conference on Natural Lan-guage Processing, pages 302–310.

Yanxia Qin, Yue Zhang, Min Zhang, and DequanZheng. 2017. Semantic-frame representation forevent detection on twitter. In Proceedings of the2017 International Conference on Asian LanguageProcessing, pages 264–267.

Long Qiu, Min-Yen Kan, and Tat-Seng Chua. 2008.Modeling context in scenario template creation. InProceedings of the 3rd International Joint Confer-ence on Natural Language Processing, pages 157–164.

Alec Radford, Karthik Narasimhan, Tim Salimans, andIlya Sutskever. 2018. Improving language under-standing by generative pre-training.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan,Dario Amodei, and Ilya Sutskever. 2019. Languagemodels are unsupervised multitask learners.

Roi Reichart and Regina Barzilay. 2012. Multi-eventextraction guided by global constraints. In Proceed-ings of the 2012 Conference of the North AmericanChapter of the Association for Computational Lin-guistics: Human Language Technologies, pages 70–79.

Alan Ritter, Mausam, Oren Etzioni, and Sam Clark.2012. Open domain event extraction from twitter.In Proceedings of the 18th ACM SIGKDD Inter-national Conference on Knowledge Discovery andData Mining, pages 1104–1112.

Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro,and Benjamin Van Durme. 2015. Script inductionas language modeling. In Proceedings of the 2015Conference on Empirical Methods in Natural Lan-guage Processing, pages 1681–1686.

Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo.2010. Earthquake shakes twitter users: real-timeevent detection by social sensors. In Proceedingsof the 19th International Conference on World WideWeb, pages 851–860.

2871

Lei Sha, Sujian Li, Baobao Chang, and Zhifang Sui.2016. Joint learning templates and slots for eventschema induction. In Proceedings of the 2016 Con-ference of the North American Chapter of the Asso-ciation for Computational Linguistics: Human Lan-guage Technologies, pages 428–434.

Lei Sha, Feng Qian, Baobao Chang, and Zhifang Sui.2018. Jointly extracting event triggers and argu-ments by dependency-bridge RNN and tensor-basedargument interaction. In Proceedings of the 32ndAAAI Conference on Artificial Intelligence, pages5916–5923.

Yusuke Shinyama and Satoshi Sekine. 2006. Preemp-tive information extraction using unrestricted rela-tion discovery. In Proceedings of the 2006 Confer-ence of the North American Chapter of the Associ-ation for Computational Linguistics: Human Lan-guage Technologies, pages 304–311.

Akash Srivastava and Charles Sutton. 2017. Autoen-coding variational inference for topic models. InProceedings of the 2017 International Conferenceon Learning Representations.

Nitish Srivastava, Geoffrey E. Hinton, AlexKrizhevsky, Ilya Sutskever, and Ruslan Salakhutdi-nov. 2014. Dropout: a simple way to prevent neuralnetworks from overfitting. Journal of MachineLearning Research, 15(1):1929–1958.

Beth Sundheim. 1992. Overview of the fourth mes-sage understanding evaluation and conference. InProceedings of the 4th Conference on Message Un-derstanding, pages 3–21.

Jeroen B. P. Vuurens and Arjen P. de Vries. 2016. Firststory detection using multiple nearest neighbors. InProceedings of the 39th International ACM SIGIRconference on Research and Development in Infor-mation Retrieval, pages 845–848.

Bishan Yang and Tom M. Mitchell. 2016. Joint ex-traction of events and entities within a documentcontext. In Proceedings of the 2016 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, pages 289–299.

Quan Yuan, Xiang Ren, Wenqi He, Chao Zhang, XinheGeng, Lifu Huang, Heng Ji, Chin-Yew Lin, and Ji-awei Han. 2018. Open-schema event profiling formassive news corpora. In Proceedings of the 27thACM International Conference on Information andKnowledge Management, pages 587–596.