Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3622–3631,November 16–20, 2020. c©2020 Association for Computational Linguistics

3622

Neural Extractive Summarization with Hierarchical AttentiveHeterogeneous Graph Network

Ruipeng Jia1,2, Yanan Cao1∗, Hengzhu Tang1, Fang Fang1∗, Cong Cao1 and Shi Wang3

1Institute of Information Engineering, Chinese Academy of Sciences2School of Cyber Security, University of Chinese Academy of Sciences

3Institute of Computing Technology, Chinese Academy of Sciences1,2{jiaruipeng, caoyanan, tanghengzhu, fangfang0703, caocong}@iie.ac.cn

3wangshi@ict.ac.cn

Abstract

Sentence-level extractive text summarizationis substantially a node classification taskof network mining, adhering to the infor-mative components and concise representa-tions. There are lots of redundant phrasesbetween extracted sentences, but it is diffi-cult to model them exactly by the generalsupervised methods. Previous sentence en-coders, especially BERT, specialize in mod-eling the relationship between source sen-tences. While, they have no ability to con-sider the overlaps of the target selected sum-mary, and there are inherent dependenciesamong target labels of sentences. In this pa-per, we propose HAHSum (as shorthand forHierarchical Attentive Heterogeneous Graphfor Text Summarization), which well mod-els different levels of information, includ-ing words and sentences, and spotlights re-dundancy dependencies between sentences.Our approach iteratively refines the sen-tence representations with redundancy-awaregraph and delivers the label dependenciesby message passing. Experiments on largescale benchmark corpus (CNN/DM, NYT, andNEWSROOM) demonstrate that HAHSumyields ground-breaking performance and out-performs previous extractive summarizers.

1 Introduction

Single document extractive summarization aims toselect subset sentences and assemble them as infor-mative and concise summaries. Recent advances(Nallapati et al., 2017; Zhou et al., 2018; Liu andLapata, 2019; Zhong et al., 2020) focus on balanc-ing the salience and redundancy of sentences, i.e.selecting the sentences with high semantic similar-ity to the gold summary and resolving the redun-dancy between selected sentences. Taking Table

∗Corresponding authors: Yanan Cao and Fang Fang

Salience Label Sentence

sent1: 0.7 0 Deanna Holleran is charged in murder.

sent2: 0.1 0 Jackson County Prosecutor Jean Peters Baker announced today.

sent3: 0.7 1 Deanna Holleran faces a charge of traffic accident.

sent4: 0.7 1 The fatal traffic accident is a murder.

sent5: 0.2 0 It took the life of Marianna Hernandez near 9th Hardesty.

Summary: Woman faces a charge of murder for a fatal traffic accident.

Table 1: Simplified News from Jackson County Prose-cutor. Salience score is an approximate estimation de-rived from semantic and Label is converted from goldsummary to ensure the concision and accuracy of theextracted summaries.

1 for example, there are five sentences in a docu-ment, and each of them is assigned one saliencescore and one label indicating whether this sentenceshould be contained in the extracted summary. Al-though sent1, sent3, and sent4 are assigned highsalience score, just sent3 and sent4 are selected asthe summary sentences (with label 1) because thereare too much redundancy information between un-selected sent1 and selected sent3. That is to say,whether one sentence could be selected depends onits salience and the redundancy with other selectedsentences. However, it is still difficult to model thedependency exactly.

Most of the previous approaches utilize autore-gressive architecture (Narayan et al., 2018; Mendeset al., 2019; Liu and Lapata, 2019; Xu et al., 2020),which just models the unidirectional dependencybetween sentences, i.e., the state of the current sen-tence is based on previously sentence labels. Thesemodels are trained to predict the current sentencelabel given the ground truth labels of the previoussentences, while feeding the predicted labels ofthe previous sentences as input in inference phase.As we all know, the autoregressive paradigm faceserror propagation and exposure bias problems (Ran-zato et al., 2015). Besides, reinforcement learningis also introduced to consider the semantics of ex-tracted summary (Narayan et al., 2018; Bae et al.,

3623

2019), which combines the maximum-likelihoodcross-entropy loss with the rewards from policygradient to directly optimize the evaluation metricfor the summarization task. Recently, the popularsolution is to build a summarization system withtwo-stage decoder. These models extract salientsentences and then rewrite (Chen and Bansal, 2018;Bae et al., 2019), compress (Lebanoff et al., 2019;Xu and Durrett, 2019; Mendes et al., 2019), ormatch (Zhong et al., 2020) these sentences.

Previous models generally use top-k strategy asan optimal strategy: for different documents, thenumber of selected sentences is constant whichconflicts with the real world. For example, almostall previous approaches extract three sentencesfrom the source articles (top-3 strategy (Zhou et al.,2018; Liu and Lapata, 2019; Zhang et al., 2019b;Xu et al., 2020)), although 40% documents inCNN/DM contain more or less than 3-sentences or-acle summary. That’s because these approaches aredifficult to measure the salience and redundancysimultaneously with error propagation. Notably,Mendes et al. (2019) introduces the length variableinto the decoder and Zhong et al. (2020) can chooseany number of sentences by match candidate sum-mary in semantic space.

In order to address above issues, we constructthe source article as a hierarchical heterogeneousgraph (HHG) and propose a Graph Attention Net(Velickovic et al., 2018) based model (HAHSum)to extract sentences by simultaneously balancingsalience and redundancy. In HHG, both words andsentences are constructed as nodes, the relationsbetween them are constructed as different typesof edges. This hierarchical graph can be viewedas a two-level graph: word-level and sentence-level. For word-level graph (word-word), we de-sign an Abstract Layer to learn the semantic rep-resentation of each word. Then, we transduce theword-level graph into the sentence-level one, byaggregating each word to its corresponding sen-tence node. For sentence-level graph (sentence-sentence), we design a Redundancy Layer, whichfirstly pre-labels each sentence and iteratively up-dates the label dependencies by propagating redun-dancy information. The redundancy layer restrictsthe scale of receptive field for redundancy informa-tion, and the information passing is guided by theground-truth labels of sentences. After obtainingthe redundancy-aware sentence representations, weuse a classifier to label these sentence-level nodes

with a threshold. In this way, the whole frameworkextracts summary sentences simultaneously insteadof autoregressive paradigm, taking away the top-kstrategy.

The contributions of this paper are as below:1) We propose a hierarchical attentive heteroge-neous graph based model(HAHSum) to guide theredundancy information propagating between sen-tences and learn redundancy-aware sentence rep-resentation; 2) Our architecture is able to extractflexible quantity of sentences with a threshold, in-stead of top-k strategy; 3) We evaluate HAHSumon three popular benchmarks (CNN/DM, NYT,NEWSROOM) and experimental results show thatHAHSum outperforms the existing state-of-the-artapproaches. Our source code will be available onGithub 1.

2 Related Work

2.1 Extractive SummarizationNeural networks have achieved great success in thetask of text summarization. There are two mainlines of research: abstractive and extractive. Theabstractive paradigm (Rush et al., 2015; See et al.,2017; Celikyilmaz et al., 2018; Sharma et al., 2019)focuses on generating a summary word-by-wordafter encoding the full document. The extractiveapproach (Cheng and Lapata, 2016; Zhou et al.,2018; Narayan et al., 2018) directly selects sen-tences from the document to assemble into a sum-mary.

Recent research work on extractive summariza-tion spans a large range of approaches. Thesework usually instantiate their encoder-decoder ar-chitecture by choosing RNN (Nallapati et al., 2017;Zhou et al., 2018), Transformer (Wang et al.,2019; Zhong et al., 2019b; Liu and Lapata, 2019;Zhang et al., 2019b) or Hierarchical GNN (Wanget al., 2020) as encoder, autoregressive (Jadhavand Rajan, 2018; Liu and Lapata, 2019) or non-autoregressive (Narayan et al., 2018; Arumae andLiu, 2018) decoders. The application of RL pro-vides a means of summary-level scoring and bringsimprovement (Narayan et al., 2018; Bae et al.,2019).

2.2 Graph Neural Network for NLPRecently, there is considerable amount of interestin applying GNN to NLP tasks and great successhas been achieved. Fernandes et al. (2019) applied

1http://github.com/coder352/HAHSum

3624

sequence GNN to model the sentences with namedentity information. Yao et al. (2019) used two-layer GCN for text classification and introduced awell-designed adjacency matrix. GCN also playedan important role in Chinese named entity (Dinget al., 2019). Liu et al. (2019) proposed a new con-textualized neural network for sequence learningby leveraging various types of non-local contex-tual information in the form of information passingover GNN. These studies are related to our workin the sense that we explore extractive text summa-rization by message passing through hierarchicalheterogeneous architecture.

3 Methodology

3.1 Problem DefinitionLet S = {s1, s2, ..., sN} denotes the source docu-ment sequence which contains N sentences, wheresi is the i-th sentence of document. Let Tdenotes the hand-crafted summary. Extractivesummarization aims to produce summary S∗ ={s∗1, s∗2, ..., s∗M} by selecting M sentences from S,where M ≤ N . Labels Y = {y1, y2, ..., yN} arederived from T , where yi ∈ {0, 1} denotes whethersentence si should be included in the extractedsummary. Oracle summary is a subset of S, whichachieves the highest ROUGE score calculated withT .

3.2 Graph ConstructionIn order to model the redundancy relation betweensentences, we use a heterogeneous graph whichcontains multi-granularity levels of information torepresent a document, as shown in Figure 1. In thisgraph, there are three types of nodes: named en-tity, word, and sentence. To reduce semantic spar-sity, we replace text spans of Named Entity byanonymized tokens (e.g. [Person A], [Person B],[Date A]). Word node is the original textual item,representing word-level information. Differentfrom DivGraphPointer (Sun et al., 2019), whichaggregates identical words into one node, we keepeach word occurrence as one node to avoid theconfusion of different contexts. Each Sentencenode corresponds to one sentence and representsthe global information of one sentence.

We also define four types of edges to representvarious structural information in HAHSum:

1. We connect sequential named entities andwords in one sentence using directed Nextedges.

2. We connect one named entity node or wordnode to one sentence node with directed Inedge if the named entity or word occurs in thissentence.

3. We connect two named entity nodes with undi-rected Same edge if they are the same namedentity.

4. We connect two sentence nodes with undi-rected Similar edge if they have trigramoverlapping.

The topological structure of graph can be rep-resented by adjacency matrix A, where the bool-type element is indicating whether there is anedge between nodes. Because HAHsum containsmulti-granularity levels of information, it can bedivided into three subgraphs: the word-level, word-sentence, and sentence-level subgraph. So, wedefine three adjacency matrices: Aword is usedfor the word-level graph, constructed by Entitynode, Word node, Next edge and Same edge.Aword−sent is used for the word-sentence graph,constructed by three types of nodes and In edge.Asent is used for sentence-level graph, constructedby Sentence node and Similar edge. Bypropagating the information from word-level tosentence-level graph, we can obtain the sentencerepresentation and model the redundancy betweensentences.

Generally, the message passing over graphs canbe achieved in two steps: aggregation and combi-nation, and this process can be conducted multipletimes (referred as layers or hops in GNN literature)(Tu et al., 2019). Therefore, we iteratively updatethe sentence nodes representation with redundancymessage passing which will be described in thefollowing sections.

3.3 Graph Attention NetworkTo represent graph structure A and node contentX in a unified framework, we develop a variant ofGraph Attention Network (GAT) (Velickovic et al.,2018). GAT is used to learn hidden representationsof each node by aggregating the information fromits neighbors, with the attention coefficients:

eij = LeakReLU(a(Wxi||Wxj)) (1)

where W ∈ Rd×d is a shared linear transformationweight matrix for this layer, || is the concatena-tion operation, and a ∈ R2d is a shared attentionalweight vector.

3625

Multi-Head GATs GAT (Words => Sentence )Classifier

Redundancy Message Passing

Abstract Layer Redundancy Layer Output Layer

Threshold OutputWord Token Node Named Entitiy Node Sentence Node

Next Edge Same Edge Similar Edge

ALBERT Encoder

Self AttentionToken Hidden State

Sentence Word

In Edge Update

[Person_A]

have

signed

[Country_A]

international

[Person_B]

on

a

[Date_A]

deal

Sentence

New Zealand

Francis Salili

two-year

(b) Detail: In Edge(a) Whole Architecture

Munster

Figure 1: Overview of Hierarchical Attentive Heterogeneous Graph

To make the attention coefficients easily compa-rable across different nodes, we normalize them asfollows:

αij = softmax(eij) =exp(eij)∑

k∈Niexp(eik)

(2)

whereNi denotes the neighbors of node i accordingto adjacency matrix A.

Then, the normalized attention coefficients areused to compute a linear combination of features.

x′i = σ(∑j∈Ni

αijWxj) +W ′xi (3)

where W ′ is used to distinguish the informationbetween xi and its neighbors.

3.4 Message PassingShown in Figure 1, HAHSum consists of ALBERTEncoder, Abstract Layer, Redundancy Layer, andOutput Layer. We next introduce how the informa-tion propagates over these layers.

3.4.1 ALBERT EncoderIn order to learn the contextual representation ofwords, we use a pre-trained ALBERT (Lan et al.,2019) for summarization and the architecture issimilar to BERTSUMEXT (Liu and Lapata, 2019).The output of ALBERT encoder contains word hid-den states hword and sentence hidden states hsent.Specifically, ALBERT takes subword units as in-put, which means that one word may correspondto multiple hidden states. In order to accuratelyuse these hidden states to represent each word, weapply an average pooling function to the outputs ofALBERT.

3.4.2 Abstract LayerThe abstract layer contains three GAT sublayerswhich are described in Section 3.3: two for word-level graph and one for word-sentence transduction.The first two GAT sublayers are used to learn thehidden state of each word based on its two-orderneighbors inspired by Kipf and Welling[2017],

W = GAT(GAT(hword, Aword), Aword) (4)

where Aword denotes the adjacency matrix of theword-level subgraph, and W denotes the hiddenstate of the word nodes.

The third GAT sublayer is to learn the initialrepresentation of each sentence node, derived fromthe word hidden states:

[W,Sabs] = GAT([W, hsent], Aword−sent) (5)

where Aword−sent denotes the adjacency matrix ofthe word-sentence subgraphs, and Sabs (abs is forabstract) is the initial representation of sentencenodes.

3.4.3 Redundancy LayerThe BERT encoder and abstract layer specialize inmodeling salience with overall context representa-tion of sentences, while it is powerless for redun-dancy information with dependencies among targetlabels. So, redundancy layer aims to model theredundancy, by iteratively updating the sentencerepresentation with redundancy message passing,and this process is supervised by ground-truth la-bels.

3626

This layer only deals with sentence-level infor-mation S = {h1, h2, ..., hN} and iteratively up-dates it L times with classification scores:

S lre = GAT(GAT(S lre, Asent), Asent)

P (yi = 1|S lre) = σ(FFN(LN(hli +MHAtt(hli))))

(6)

where S0re = Sabs (re is for redundancy) and weget SLre at the end, Wc, Wr are weight parameters,FFN, LN, MHAtt are feed-foreard network, layernormalization and multi-head attention layer.

We update hli by reducing the redundancy in-formation gli, which is the weighted summation ofneighbors information:

gli =1

|Ni|∑j∈Ni

P (yj = 1|S lre) ∗ hlj

hl+1i =W l

c ∗ hli − hlTi W lr tanh(g

li)

S l+1re = (hl+1

1 , hl+12 , ..., hl+1

|S| )

(7)

where Ni is redundancy receptive field for node i,according to Asent.

Specifically, we employ a gating mechanism(Gilmer et al., 2017) for the information update,so that: 1) to avoid GNN smoothing problem; 2)the original overall information from ALBERT isaccessible for the ultimate classifier.

hl′i =W l

c ∗ hli − hlTi W lr tanh(g

li)

plg = σ(f lg([hli; h

l′i ]))

hl+1i = hli � plg + hl

′i � (1− plg)

(8)

where � denotes element-wise multiplication.

3.5 Objective Function

Previous approaches for modeling the salience andredundancy is autoregressive, where observationsfrom previous time-steps are used to predict thevalue at current time-step:

P (Y |S) =|S|∏t=1

P (yt|S, y1, y2, ..., yt−1) (9)

The autoregressive models have some disadvan-tages: 1) the error in inference will propagate sub-sequently, 2) label yt is generated just depend onprevious sentences y<t rather than considering bidi-rectional dependency, and 3) it is difficult to decidehow many sentences to extract.

Datasetsavg.doc length avg.summary length

words sentences words sentences

CNN 760.50 33.98 45.70 3.59DailyMail 653.33 29.33 54.65 3.86NYT 800.04 35.55 45.54 2.44Newsroom (Ext) 605.44 28.78 40.95 1.90

Table 2: Data Statistics: CNN/Daily Mail, NYT, News-room

Our HAHSum predicts these labels simultane-ously:

P (Y |S) =|S|∏t=1

P (yt|S,Sabs,Sre) (10)

where we extract flexible quantity of sentences witha threshold instead of top-k. For L classifiers in ourmodel, we train them simultaneously with differentproportions. For each training pair (X,Y ) andthe predicted Y , the loss function is formalized asfollows:

L =−L∑l=0

L+ l

2L

|S|∑i=1

{yi logP (yi|S lre)+

(1− yi) log(1− P (yi|S lre))}

(11)

4 Experiments Setting

4.1 Benchmark DatasetsAs shown in Table 2, we employ three datasetswidely-used with multiple sentences summary(CNN/DM (Hermann et al., 2015), NYT (Sand-haus, 2008), and NEWSROOM (Grusky et al.,2018)). These summaries vary with respect tothe type of rewriting operations, e.g., CNN/DMand NYT prefer to the abstractive approaches andNewsroom(Ext) is genuinely extractive. We em-ploy the greedy method to obtain ground-truth sen-tence labels (Nallapati et al., 2017).

CNN/DailyMail: We use the standard splits fortraining, validation, and test (90,266/1,220/1,093for CNN and 196,96/12,148/10,397 for DailyMail)(Liu and Lapata, 2019). Input documents are trun-cated to 768 BPE tokens, with anonymized entitiesand processed by Stanford CoreNLP.

NYT: Following previous work (Zhang et al.,2019b; Xu and Durrett, 2019), we use 137,778,17,222 and 17,223 samples for training, validation,and test, respectively. Input documents were trun-cated to 768 BPE tokens too. Note that there are

3627

different division for NYT (Durrett et al., 2016; Liuand Lapata, 2019) and several models are not eval-uated on NYT officially (See et al., 2017; Mendeset al., 2019), so we re-train and evaluate them onNYT with the source code from Github.

Newsroom(Ext): We employ the extractivepart of Newsroom with same divisioin method(Mendes et al., 2019) for training/validation/test(331,778/36,332/36,122). Input documents aretruncted to 768 BPE tokens.

4.2 Evaluation Metric & Parameter Settings

Metric: ROUGE (Lin, 2004) is the standard met-ric for evaluating the quality of summaries. Wereport the ROUGE-1, ROUGE-2, and ROUGE-Lof HAHSum by ROUGE-1.5.5.pl, which calculatesthe overlap lexical units of extracted sentences andground-truth.

Graph Structure: For abstract layer, we ex-tract the named entities ([Person], [Date],[Country], [Buildings], [Monetary])using CoreNLP, and replace them by anonymizedtokens. Similar to Fernandes et al. (2019), we havetried to add dependency parse edges and they didn’tshow significant benefits, owing to the facts that 1)the dependency tree is substantially a permutationsequential structure, with little advancements fororiginal information; 2) the performance is influ-enced by the accuracy of the upstream annotators.We have tried the iteration steps of [1, 2, 3, 5] forupdating redundancy layer, and L = 3 is the bestvalue in experiment result.

Parameters: We employ pre-trained ‘albert-xxlarge-v2’2 and reuse the implementation ofPreSumm3. We train our model (with about400M parameters) one day for 100,000 steps on2 GPUs(Nvidia Tesla V100, 32G) with gradientaccumulation every two steps. We select the top-3checkpoints according to the evaluation loss on val-idation set and report the averaged results on thetest set. Adam with β1 = 0.9, β2 = 0.999 is usedas optimizer and learning rate schedule follows thestrategies with warming-up on first 10,000 steps(Vaswani et al., 2017). The final threshold in ex-traction is 0.65 for CNN/DM, 0.58 for NYT and0.64 for Newsroom, with the highest ROUGE-1score individually. A higher threshold will be with

2https://github.com/huggingface/transformers3https://github.com/nlpyang/PreSumm

more concise summary and the lower threshold willreturn more information.

4.3 BaselinesExtractive Methods: Oracle is the extractedsummary according to the ground-truth labels.Lead is a base method for extractive text sum-marization that chooses first several sentencesas a summary. SummaRuNNer takes content,salience, novelty, and position of each sentenceinto consideration when deciding if a sentenceshould be included in the extractive summary. PN-BERT tries to employ the unsupervised transfer-able knowledge. BERTSUMEXT applies pre-trained BERT in text summarization and proposes ageneral framework for both extractive and abstrac-tive models. MATCHSUM is a two-stage methodfor extract-then-match, and the first-stage is BERT-SUMEXT.

Abstractive Methods: ABS is the normal ar-chitecture with RNN-based encoder and decoder.PGC augments the standard Seq2Seq attentionalmodel with pointer and coverage mechanisms.TransformerABS employs Transformer in textsummarization. MASS proposes masked Seq2Seqpre-training for encoder-decoder. UniLM presentsunified pre-trained language model, that can be fine-tuned for summarization. BART, and Prophet-Net are pre-trained on large unlabeled data andperform excellent performance with Transformerarchitecture. PEGASUS proposes Transformer-based models with extracted gap-sentences for ab-stractive summarization.

Specifically, these Transformer-based ap-proaches are divided into Base and Largeversions, according to the layers of Transformer.

5 Analysis

5.1 Rouge ScoresThe experiment results on three benchmark datasetsare shown in Table 3. There are ignored positionsfor Newsroom(Ext), which is designed for extrac-tive approaches, eliminating the demanding of ab-stractive ones. It is obvious that HAHSum almostoutperforms all the baselines across most of theevaluation metrics. For CNN/DM, there is littlegap between the performance of extractive and ab-stractive architectures, particularly demonstratingthe popularity and generality of this dataset. WhileNYT prefers to abstractive methods, and NEWS-ROOM(Ext) is constructed by extracting sentences

3628

ModelsCNN/DM NYT Newsroom (Ext)

R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L

Abstractive

ABS (2015) 35.46 13.30 32.65 42.78 25.61 35.26 6.1 0.2 5.4PGC (2017) 39.53 17.28 36.38 43.93 26.85 38.67 39.1 27.9 36.2TransformerABS (2017) 40.21 17.76 37.09 45.36 27.34 39.53 40.3 28.7 36.5MASSLarge (2019) 43.05 20.02 40.08 - - - - - -UniLMLarge (2019) 43.33 20.21 40.51 - - - - - -BARTLarge (2019) 44.16 21.28 40.90 48.73 29.25 44.48 - - -PEGASUSLarge (2019a) 44.17 21.47 41.11 - - - - - -ProphetNetLarge (2020) 44.20 21.17 41.30 - - - - - -

Extractive

Oracle 55.61 32.84 51.88 64.22 44.57 57.27 - - -Lead 40.42 17.62 36.67 41.80 22.60 35.00 53.1 49.0 52.4SummaRuNNer (2017) 39.60 16.20 35.30 42.37 23.89 38.74 48.96 44.33 49.57Exconsumm (2019) 41.7 18.6 37.8 43.18 24.43 38.92 68.4 62.9 67.3PNBERTBase (2019a) 42.69 19.60 38.85 - - - - - -BERTSUMEXTLarge (2019) 43.85 20.34 39.90 48.51 30.27 44.65 70.85 67.03 69.61MATCHSUMBase (2020) 44.41 20.86 40.55 - - - - - -HAHSumLarge(Ours) 44.68 21.30 40.75 49.36 31.41 44.97 71.31 68.75 70.83

Table 3: Automatic Evaluation or ROUGE

directly. For extractive approaches, HAHSum,MATCHSUM, and BERTSUMEXT are outstand-ing with the power of pre-trained BERT-like mod-els. For abstractive methods, these variants ofTransformer perform extremely with deep archi-tectures and large-scale unlabeled corpus.

HAHSum outperforms all other extractive ap-proaches for that: 1) HAHSum achieves improve-ments to mitigate the redundancy bias by measur-ing salience and redundancy simultaneously, whilethis would not be possible with any framework inthe autoregressive literature because salience andredundancy are treated as two different processesdue to the dependency among target labels. 2)The promising results of heterogeneous sequence-graph models outperform pure sequence models.Sequence encoders with a graph component canreason about long-distance relationships in weaklystructured data such as text, which requires non-trivial understanding of the input, while attentivesequential architectures prefer to calculate the rele-vance merely.

5.2 Ablation StudiesWe propose several strategies to improve the per-formance by relieving the semantic sparsity andredundancy bias, including abstract layer(AL), theiterative redundancy layer(RL), and pre-trained AL-BERT. To investigate the influence of these factors,

Models R-1 R-2 R-L

HAHSum 44.68 21.30 40.75w/o AL 44.35 20.98 40.49w/o RL 44.49 21.11 40.58w/o ALBERT 44.57 21.14 40.53

Table 4: Ablation Study on CNN/DM Test Set

we conduct the experiments and list the resultsin Table 4. Significantly, AL is more importantthan RL, for the reason that there are lots of mean-ingless named entities. Besides, RL mechanismenlarges the advantage of extraction without top-kstrategy, for there are more than 40% documents inCNN/DM contains more or less than 3-sentencesoracle summary. As shown in Table 6, HAHSumpredicts sequence exactly with two sentences, sameas the oracle summary. While BERTSUMEXT ex-tracts top-3 sentences strictly, in spite of the inac-curateness and redundancy.

5.3 Human Evaluation for Summarization

It is not enough only relying on the ROUGE eval-uation for a summarization system, although theROUGE correlates well with human judgments(Owczarzak et al., 2012). To evaluate the perfor-mance of HAHSum more accurately, we design

3629

Models 1st 2nd 3rd 4th 5th MeanR

SummaRuNNer 0.14 0.27 0.24 0.22 0.13 2.93BERTSUMEXT 0.20 0.28 0.31 0.16 0.05 2.58MATCHSUM 0.24 0.36 0.16 0.15 0.09 2.49HAHSum 0.45 0.34 0.18 0.03 0.00 2.24Ground-Truth 0.70 0.21 0.05 0.04 0.00 1.43

Table 5: Human evaluation on Daily Mail.

an Amazon Mechanical Turk experiment basedon ranking method. Following Cheng and Lapata(2016); Narayan et al. (2018); Zhang et al. (2019b),firstly, we randomly select 40 samples from DailyMail test set. Then the human participants arepresented with a original document and a list ofcorresponding summaries produced by differentmodel systems. Participants are requested to rankthese summaries (ties allowed) by taking informa-tiveness (Can the summary capture the importantinformation from the document) and fluency (Is thesummary grammatical) into account. Each docu-ment is annotated by three different participantsseparately.

Following the previous work, the input arti-cle and ground truth summaries are also shownto the human participants in addition to thefour model summaries (SummaRuNNer, BERT-SUMEXT, MATCHSUM and HAHSum). Fromthe results shown in Table 5, we can see that HAH-Sum is better in relevance compared with others.

5.4 Visualization

We visualize the learned embedding of word andsentence nodes in a two-dimensional space by ap-plying the t-SNE algorithm. We randomly select500 continuous word nodes (approximately 30 sen-tences in a document) and 1000 sentence nodesfrom BERTSUMEXT and HAHSum separately.As shown in Figure 2, for word nodes, the darknessdetermines it’s position in one document; whilefor sentence nodes, red points are the sentenceswith label 1, and green points are with label 0. Theresult shows: 1) It is amazing that sentence-levelsummarization constrains word representations tobe shared across whole sentence, and there are ob-viously word clusters in BERTSUMEXT; 2) Theword clusters are more distinct and meaningful inHAHSum equipped with abstract layer and GAT;3) Intuitively, the redundancy layer has particularlystrong representation power and generalizability,for that oracle sentence nodes in HAHSum are easyto identify, without autoregressive formalism used

for capturing sentence-level redundancy.

40 20 0 20 40X

60

40

20

0

20

40

60

80

Y

(a) Word Nodes/BERTSUM

40 20 0 20 40X

30

20

10

0

10

20

30

40

50

Y

(b) Word Nodes/HAHSum

30 20 10 0 10 20 30 40X

60

40

20

0

20

40

60

Y

(c) Sent Nodes/BERTSUM

40 20 0 20 40X

40

20

0

20

40

Y

(d) Sent Nodes/HAHSum

Figure 2: T-SNE Visualization on CNN/DM Test Set

Source Document (truncated): built at a cost of # 1 billion, new broadcast-ing house is the jewel in the crown of the bbc and the setting for its self-mocking satire w1a. (...) new broadcasting house is home to three 24-hournews channels, nine radio networks and 6,000 staff.

Oracle Summary: new broadcasting house was opened by the queen in

2013–four years behind schedule and at least #55 million over budget.

the bbc has admitted it ‘occasionally’ runs out of meeting rooms in its #

1billion new broadcasting house

HAHSum: new broadcasting house was opened by the queen in

2013–four years behind schedule and at least #55 million over budget.

the bbc has admitted it ‘occasionally’ runs out of meeting rooms in its #

1billion new broadcasting house

BERTSUMEXT: new broadcasting house was opened by the queen in

2013–four years behind schedule and at least #55 million over budget.

another said: ‘it’s bonkers to hold meetings across the street.’

a bbc spokesman said: ‘it is occasionally necessary to book nearby venues,

especially for larger meetings.

Table 6: Case Study on CNN/DM Test Set

6 Conclusion

In this paper, we propose hierarchical attentiveheterogeneous graph, aiming to advance text sum-marization by measuring salience and redundancysimultaneously. Our approach model redundancyinformation by iteratively update the sentence infor-mation with message passing in redundancy-awaregraph. As a result, HAHSum produces more fo-cused summaries with fewer superfluous and theperformance improvements are more pronouncedon more extractive datasets.

3630

Acknowledgments

This research is supported by the National KeyResearch and Development Program of China(NO.2018YFB1004703). We thank all authors fortheir contributions and all anonymous reviewersfor their constructive comments.

ReferencesKristjan Arumae and Fei Liu. 2018. Reinforced extrac-

tive summarization with question-focused rewards.In ACL.

Sanghwan Bae, Taeuk Kim, Jihoon Kim, and Sanggoo Lee. 2019. Summary level training of sentencerewriting for abstractive summarization. In Confer-ence on Empirical Methods in Natural LanguageProcessing, Workshop on New Frontiers in Summa-rization.

Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, andYejin Choi. 2018. Deep communicating agents forabstractive summarization. In NAACL-HLT.

Yen-Chun Chen and Mohit Bansal. 2018. Fast abstrac-tive summarization with reinforce-selected sentencerewriting. In ACL.

Jianpeng Cheng and Mirella Lapata. 2016. Neural sum-marization by extracting sentences and words. InACL.

Ruixue Ding, Pengjun Xie, Xiaoyan Zhang, Wei Lu,Linlin Li, and Luo Si. 2019. A neural multi-digraphmodel for chinese ner with gazetteers. In ACL.

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xi-aodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou,and Hsiao-Wuen Hon. 2019. Unified languagemodel pre-training for natural language understand-ing and generation. In NIPS.

Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein.2016. Learning-based single-document summariza-tion with compression and anaphoricity constraints.In ACL.

Patrick Fernandes, Miltiadis Allamanis, and MarcBrockschmidt. 2019. Structured neural summariza-tion. In ICLR.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley,Oriol Vinyals, and George E Dahl. 2017. Neuralmessage passing for quantum chemistry. In Interna-tional Conference on Machine Learning.

Max Grusky, Mor Naaman, and Yoav Artzi. 2018.Newsroom: A dataset of 1.3 million summaries withdiverse extractive strategies. In NAACL-HLT.

Karl Moritz Hermann, Tomas Kocisky, Edward Grefen-stette, Lasse Espeholt, Will Kay, Mustafa Suleyman,and Phil Blunsom. 2015. Teaching machines to readand comprehend. In NIPS.

Aishwarya Jadhav and Vaibhav Rajan. 2018. Extrac-tive summarization with swap-net: Sentences andwords from alternating pointer networks. In ACL.

Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutionalnetworks. In ICLR.

Zhenzhong Lan, Mingda Chen, Sebastian Goodman,Kevin Gimpel, Piyush Sharma, and Radu Soricut.2019. Albert: A lite bert for self-supervised learn-ing of language representations. In ICLR.

Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt,Doo Soon Kim, Seokhwan Kim, Walter Chang, andFei Liu. 2019. Scoring sentence singletons and pairsfor abstractive summarization. ACL.

Mike Lewis, Yinhan Liu, Naman Goyal, Mar-jan Ghazvininejad, Abdelrahman Mohamed, OmerLevy, Ves Stoyanov, and Luke Zettlemoyer.2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, trans-lation, and comprehension. In arXiv preprintarXiv:1910.13461.

Chin-Yew Lin. 2004. Rouge: A package for automaticevaluation of summaries. In Text summarizationbranches out.

Pengfei Liu, Shuaichen Chang, Xuanjing Huang, JianTang, and Jackie Chi Kit Cheung. 2019. Contextu-alized non-local neural networks for sequence learn-ing. In AAAI.

Yang Liu and Mirella Lapata. 2019. Text summariza-tion with pretrained encoders. In EMNLP.

Afonso Mendes, Shashi Narayan, Sebastiao Miranda,Zita Marinho, Andre FT Martins, and Shay B Co-hen. 2019. Jointly extracting and compressing doc-uments with summary state representations. InNAACL-HLT.

Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017.Summarunner: A recurrent neural network based se-quence model for extractive summarization of docu-ments. In AAAI.

Shashi Narayan, Shay B Cohen, and Mirella Lapata.2018. Ranking sentences for extractive summariza-tion with reinforcement learning. In NAACL-HLT.

Karolina Owczarzak, John M Conroy, Hoa Trang Dang,and Ani Nenkova. 2012. An assessment of the accu-racy of automatic evaluation in summarization. InProceedings of Workshop on Evaluation Metrics andSystem Comparison for Automatic Summarization.

Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli,and Wojciech Zaremba. 2015. Sequence level train-ing with recurrent neural networks. In ICLR.

Alexander M. Rush, Sumit Chopra, and Jason Weston.2015. A neural attention model for abstractive sen-tence summarization. In EMNLP.

3631

Evan Sandhaus. 2008. The new york times annotatedcorpus. In Linguistic Data Consortium, Philadel-phia.

Abigail See, Peter J. Liu, and Christopher D. Manning.2017. Get to the point: Summarization with pointer-generator networks. In ACL.

Eva Sharma, Luyang Huang, Zhe Hu, and Lu Wang.2019. An entity-driven framework for abstractivesummarization. In EMNLP.

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequencepre-training for language generation. In Interna-tional Conference on Machine Learning.

Zhiqing Sun, Jian Tang, Pan Du, Zhi-Hong Deng, andJian-Yun Nie. 2019. Divgraphpointer: A graphpointer network for extracting diverse keyphrases.In International ACM SIGIR Conference on Re-search and Development in Information Retrieval.

Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xi-aodong He, and Bowen Zhou. 2019. Multi-hop read-ing comprehension across multiple documents byreasoning over heterogeneous graphs. In ACL.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In NIPS.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova,Adriana Romero, Pietro Lio, and Yoshua Bengio.2018. Graph attention networks. In ICLR.

Danqing Wang, Pengfei Liu, Yining Zheng, XipengQiu, and Xuanjing Huang. 2020. Heterogeneousgraph neural networks for extractive document sum-marization. In ACL.

Danqing Wang, Pengfei Liu, Ming Zhong, Jie Fu,Xipeng Qiu, and Xuanjing Huang. 2019. Explor-ing domain shift in extractive text summarization. InarXiv preprint arXiv:1908.11664.

Jiacheng Xu and Greg Durrett. 2019. Neural extrac-tive text summarization with syntactic compression.EMNLP.

Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu.2020. Discourse-aware neural extractive text sum-marization. In ACL.

Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu,Nan Duan, Jiusheng Chen, Ruofei Zhang, and MingZhou. 2020. Prophetnet: Predicting future n-gramfor sequence-to-sequence pre-training. In arXivpreprint arXiv:2001.04063.

Liang Yao, Chengsheng Mao, and Yuan Luo. 2019.Graph convolutional networks for text classification.In AAAI.

Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe-ter J Liu. 2019a. Pegasus: Pre-training with ex-tracted gap-sentences for abstractive summarization.In arXiv preprint arXiv:1912.08777.

Xingxing Zhang, Furu Wei, and Ming Zhou. 2019b.Hibert: Document level pre-training of hierarchicalbidirectional transformers for document summariza-tion. In ACL.

Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang,Xipeng Qiu, and Xuanjing Huang. 2020. Extractivesummarization as text matching. In ACL.

Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu,and Xuanjing Huang. 2019a. Searching for effectiveneural extractive summarization: What works andwhats next. In ACL.

Ming Zhong, Danqing Wang, Pengfei Liu, Xipeng Qiu,and Xuanjing Huang. 2019b. A closer look at databias in neural extractive summarization models. InEMNLP.

Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang,Ming Zhou, and Tiejun Zhao. 2018. Neural docu-ment summarization by jointly learning to score andselect sentences. In ACL.