Generating Question-Answer Hierarchies

Kalpesh Krishna & Mohit IyyerCollege of Information and Computer Sciences

University of Massachusetts Amherst{kalpesh,miyyer}@cs.umass.edu

http://squash.cs.umass.edu/

Abstract

The process of knowledge acquisition can beviewed as a question-answer game between astudent and a teacher in which the student typi-cally starts by asking broad, open-ended ques-tions before drilling down into specifics (Hin-tikka, 1981; Hakkarainen and Sintonen, 2002).This pedagogical perspective motivates a newway of representing documents. In this paper,we present SQUASH (Specificity-controlledQuestion-Answer Hierarchies), a novel andchallenging text generation task that con-verts an input document into a hierarchy ofquestion-answer pairs. Users can click onhigh-level questions (e.g., “Why did Frodoleave the Fellowship?”) to reveal related butmore specific questions (e.g., “Who did Frodoleave with?”). Using a question taxonomyloosely based on Lehnert (1978), we classifyquestions in existing reading comprehensiondatasets as either GENERAL or SPECIFIC. Wethen use these labels as input to a pipelinedsystem centered around a conditional neu-ral language model. We extensively evaluatethe quality of the generated QA hierarchiesthrough crowdsourced experiments and reportstrong empirical results.

1 Introduction

Q: What is this paper about?A: We present a novel text generation taskwhich converts an input document into a model-generated hierarchy of question-answer (QA)pairs arranged in a top-down tree structure (Fig-ure 1). Questions at higher levels of the treeare broad and open-ended while questions atlower levels ask about more specific factoids. Anentire document has multiple root nodes (“keyideas”) that unfold into a forest of question trees.While readers are initially shown only the rootnodes of the question trees, they can “browse” thedocument by clicking on root nodes of interest

Massive Attack (band)

On 21 January 2016, the iPhoneapplication "Fantom" was released.The application was developed bya team including Massive Attack'sRobert Del Naja and let users hear

parts of four new songs byremixing them in real time, usingthe phone's location, movement,clock, heartbeat, and camera. On28 January 2016, Massive Attackreleased a new EP, Ritual Spirit,which includes the four songs

released on Fantom.

Q. What was the iPhone application Fantom? A. The app... let users hear parts of ... real time,

Q. Who created it? A. ... team including ... Robert Del Naja

On 26 July 2016, Massive Attackpreviewed three new songs: "ComeNear Me", "The Spoils", and "Dear

Friend" on Fantom, an iPhoneapplication on which they

previously previewed the foursongs from the Ritual Spirit EP ...

The video for "The Spoils",featuring Cate Blanchett, and

directed by Australian director JohnHillcoat, ...

Q. What is Ritual Spirit? A. On ... Attack released a new EP Ritual Spirit

Q. What did they do in 2016?A. On ... 2016 ... three new songs: "Come NearMe", "The Spoils", and "Dear Friend" on Fantom,

Q. Who was in the video?A. The video ... featuring Cate Blanchett,Q. Who was the Australian director?A. John Hillcoat,

Q. When did they release a song with"Fantom''? A. On 28 January 2016,

...

...

Figure 1: A subset of the QA hierarchy generated byour SQUASH system that consists of GENERAL andSPECIFIC questions with extractive answers.

to reveal more fine-grained related information.We call our task SQUASH (Specificity-controlledQuestion Answer Hierarchies).

Q: Why represent a document with QA pairs?1

A: Questions and answers (QA) play a criticalrole in scientific inquiry, information-seeking dia-logue and knowledge acquisition (Hintikka, 1981,1988; Stede and Schlangen, 2004). For example,web users often use QA pairs to manage and shareknowledge (Wagner, 2004; Wagner and Bolloju,2005; Gruber, 2008). Additionally, unstructuredlists of “frequently asked questions” (FAQs) areregularly deployed at scale to present information.Industry studies have demonstrated their effective-ness at cutting costs associated with answeringcustomer calls or hiring technical experts (Daven-port et al., 1998). Automating the generation ofQA pairs can thus be of immense value to compa-nies and web communities.

1Our introduction is itself an example of the QA format.Other academic papers such as Henderson et al. (2018) havealso used this format to effectively present information.

arX

iv:1

906.

0262

2v2

[cs

.CL

] 2

1 Ju

l 201

9

Q: Why add hierarchical structure to QA pairs?A: While unstructured FAQs are useful, ped-agogical applications benefit from additionalhierarchical organization. Hakkarainen andSintonen (2002) show that students learn conceptseffectively by first asking general, explanation-seeking questions before drilling down into morespecific questions. More generally, hierarchiesbreak up content into smaller, more digestablechunks. User studies demonstrate a strongpreference for hierarchies in document summa-rization (Buyukkokten et al., 2001; Christensenet al., 2014) since they help readers easily identifyand explore key topics (Zhang et al., 2017).

Q: How do we build systems for SQUASH?A: We leverage the abundance of reading compre-hension QA datasets to train a pipelined systemfor SQUASH. One major challenge is the lackof labeled hierarchical structure within existingQA datasets; we tackle this issue in Section 2 byusing the question taxonomy of Lehnert (1978)to classify questions in these datasets as eitherGENERAL or SPECIFIC. We then condition aneural question generation system on these twoclasses, which enables us to generate both typesof questions from a paragraph. We filter andstructure these outputs using the techniquesdescribed in Section 3.

Q: How do we evaluate our SQUASH pipeline?A: Our crowdsourced evaluation (Section 4)focuses on fundamental properties of the gener-ated output such as QA quality, relevance, andhierarchical correctness. Our work is a first steptowards integrating QA generation into documentunderstanding; as such, we do not directly evalu-ate how useful SQUASH output is for downstreampedagogical applications. Instead, a detailed qual-itative analysis (Section 5) identifies challengesthat need to be addressed before SQUASH can bedeployed to real users.

Q: What are our main contributions?A1: A method to classify questions according totheir specificity based on Lehnert (1978).A2: A model controlling specificity of generatedquestions, unlike prior work on QA generation.A3: A novel text generation task (SQUASH),which converts documents into specificity-basedhierarchies of QA pairs.A4: A pipelined system to tackle SQUASH alongwith crowdsourced methods to evaluate it.

Q: How can the community build on this work?A: We have released our codebase, dataset anda live demonstration of our system at http://squash.cs.umass.edu/. Additionally, weoutline guidelines for future work in Section 7.

2 Obtaining training data for SQUASH

The proliferation of reading comprehensiondatasets like SQuAD (Rajpurkar et al., 2016,2018) has enabled state-of-the-art neural questiongeneration systems (Du et al., 2017; Kim et al.,2018). However, these systems are trained forindividual question generation, while the goal ofSQUASH is to produce a general-to-specific hier-archy of QA pairs. Recently-released conversa-tional QA datasets like QuAC (Choi et al., 2018)and CoQA (Reddy et al., 2018) contain a sequen-tial arrangement of QA pairs, but question speci-ficity is not explicitly marked.2 Motivated by thelack of hierarchical QA datasets, we automaticallyclassify questions in SQuAD, QuAC and CoQAaccording to their specificity using a combinationof rule-based and automatic approaches.

2.1 Rules for specificity classification

What makes one question more specific thananother? Our scheme for classifying questionspecificity maps each of the 13 conceptual ques-tion categories defined by Lehnert (1978) to threecoarser labels: GENERAL, SPECIFIC, or YES-NO.3

As a result of this mapping, SPECIFIC questionsusually ask for low-level information (e.g., en-tities or numerics), while GENERAL questionsask for broader overviews (e.g., “what happenedin 1999?”) or causal information (e.g, “whydid...”). Many question categories can be reliablyidentified using simple templates and rules; Acomplete list is provided in Table 1.4

Classifying questions not covered by templates:If a question does not satisfy any template or rule,how do we assign it a label? We manage to clas-sify roughly half of all questions with our tem-

2“Teachers” in the QuAC set-up can encourage “students”to ask a follow-up question, but we cannot use these annota-tions to infer a hierarchy because students are not required toactually follow their teachers’ directions.

3We add a third category for YES-NO questions as theyare difficult to classify as either GENERAL or SPECIFIC.

4Questions in Lehnert (1978) were classified using a con-ceptual dependency parser (Schank, 1972). We could not finda modern implementation of this parser and thus decided touse a rule-based approach that relies on spaCy 2.0 (Honnibaland Montani, 2017) for all preprocessing.

Conceptual class Specificity Question asks for... Sample templates

Causal Antecedent, Goal Oriented,Enablement, Causal Consequent,Expectational

GENERAL the reason for occurrenceof an event and the conse-quences of it

Why ..., What happened after / be-fore ..., What was the cause / rea-son / purpose ..., What led to ...

Instrumental GENERAL a procedure / mechanism How question with VERB parentfor How in dependency tree

Judgemental GENERAL a listener’s opinion Words like you, your present

Concept Completion, Feature Speci-fication

GENERAL orSPECIFIC

fill-in-the-blank informa-tion

Where / When / Who ...(“SPECIFIC” templates)

Quantification SPECIFIC an amount How many / long ...

Verification, Disjunctive YES-NO Yes-No answers first word is VERB

Request N/A an act to be performed (absent in datasets)

Table 1: The 13 conceptual categories of Lehnert (1978) and some templates to identify them and their specificity.

plates and rules (Table A1); for the remaininghalf, we resort to a data-driven approach. First,we manually label 1000 questions in QuAC5 us-ing our specificity labels. This annotated datais then fed to a single-layer CNN binary classi-fier (Kim, 2014) using ELMo contextualized em-beddings (Peters et al., 2018).6 On a 85%-15%train-validation split, we achieve a high classifica-tion accuracy of 91%. The classifier also trans-fers to other datasets: on 100 manually labeledCoQA questions, we achieve a classification accu-racy of 80%. To obtain our final dataset (Table 2),we run our rule-based approach on all questionsin SQuAD 2.0, QuAC, and CoQA and apply ourclassifier to label questions that were not coveredby the rules. We further evaluate the specificity ofthe questions generated by our final system usinga crowdsourced study in Section 4.3.

Dataset Size GENERAL SPECIFIC YES-NO

SQuAD 86.8k 28.2% 69.7% 2.1%QuAC 65.2k 34.9% 33.5% 31.6%CoQA 105.6k 23.6% 54.9% 21.5%All 257.6k 28.0% 54.5% 17.5%

Table 2: Distribution of classes in the final datasets. Weadd some analysis on this distribution in Appendix A.

3 A pipeline for SQUASHing documents

To SQUASH documents, we build a pipelined sys-tem (Figure 2) that takes a single paragraph as in-put and produces a hierarchy of QA pairs as out-put; for multi-paragraph documents, we SQUASHeach paragraph independently of the rest. At

5We use QuAC because its design encourages a higherpercentage of GENERAL questions than other datasets, as thequestion-asker was unable to read the document to formulatemore specific questions.

6Implemented in AllenNLP (Gardner et al., 2018).

a high level, the pipeline consists of five steps:(1) answer span selection, (2) question genera-tion conditioned on answer spans and specificitylabels, (3) extractively answering generated ques-tions, (4) filtering out bad QA pairs, and (5) struc-turing the remaining pairs into a GENERAL-to-SPECIFIC hierarchy. The remainder of this sec-tion describes each step in more detail and after-wards explains how we leverage pretrained lan-guage models to improve individual componentsof the pipeline.

3.1 Answer span selectionOur pipeline begins by selecting an answer spanfrom which to generate a question. To train thesystem, we can use ground-truth answer spansfrom our labeled datasets, but at test time how dowe select answer spans? Our solution is to con-sider all individual sentences in the input para-graph as potential answer spans (to generate GEN-ERAL and SPECIFIC questions), along with all en-tities and numerics (for just SPECIFIC questions).We did not use data-driven sequence tagging ap-proaches like previous work (Du and Cardie, 2017,2018), since our preliminary experiments withsuch approaches yielded poor results on QuAC.7

More details are provided in Appendix C.

3.2 Conditional question generationGiven a paragraph, answer span, and desiredspecificity label, we train a neural encoder-decoder model on all three reading comprehensiondatasets (SQuAD, QuAC and CoQA) to generatean appropriate question.

7We hypothesize that answer span identification on QuACis difficult because the task design encouraged “teachers” toprovide more information than just the minimal answer span.

RCDatasets

Does Q matchtemplate?

SpecificityClassifier

Question Generation

No

Span SelectionDocument

Span

ClassQuestion

Answering

x13questions QA

Filtering

Building QA Hierarchy

Yes

(SimilarPipeline)

SQUASH Output

Training Data

x13QA

GENERAL SPECIFIC

Figure 2: An overview of the process by which we generate a pair of GENERAL-SPECIFIC questions , which

consists of feeding input data (“RC” is Reading Comprehension) through various modules, including a

question classifier and a multi-stage pipeline for question generation, answering, and filtering.

Data preprocessing: At training time, we usethe ground-truth answer spans from these datasetsas input to the question generator. To improvethe quality of SPECIFIC questions generated fromsentence spans, we use the extractive evidencespans for CoQA instances (Reddy et al., 2018)instead of the shorter, partially abstractive answerspans (Yatskar, 2019). In all datasets, we removeunanswerable questions and questions whoseanswers span multiple paragraphs. A few verygeneric questions (e.g. “what happened in thisarticle?”) were manually identified removed fromthe training dataset. Some other questions (e.g.,“where was he born?”) are duplicated many timesin the dataset; we downsample such questions toa maximum limit of 10. Finally, we preprocessboth paragraphs and questions using byte-pairencoding (Sennrich et al., 2016).

Architecture details: We use a two-layer biL-STM encoder and a single-layer LSTM (Hochre-iter and Schmidhuber, 1997) decoder with softattention (Bahdanau et al., 2015) to generatequestions, similar to Du et al. (2017). Ourarchitecture is augmented with a copy mecha-nism (See et al., 2017) over the encoded paragraphrepresentations. Answer spans are marked with<SOA> and <EOA> tokens in the paragraph, andrepresentations for tokens within the answer spanare attended to by a separate attention head. Wecondition the decoder on the specificity class(GENERAL, SPECIFIC and YES-NO)8 by concate-nating an embedding for the ground-truth class tothe input of each time step. We implement modelsin PyTorch v0.4 (Paszke et al., 2017), and thebest-performing model achieves a perplexity of11.1 on the validation set. Other hyperparametersdetails are provided in Appendix B.

8While we do not use YES-NO questions at test time, wekeep this class to avoid losing a significant proportion oftraining data.

Test time usage: At test time, the question gen-eration module is supplied with answer spans andclass labels as described in Section 3.1. To pro-mote diversity, we over-generate prospective can-didates (Heilman and Smith, 2010) for every an-swer span and later prune them. Specifically, weuse beam search with a beam size of 3 to gener-ate three highly-probable question candidates. Asthese candidates are often generic, we additionallyuse top-k random sampling (Fan et al., 2018) withk = 10, a recently-proposed diversity-promotingdecoding algorithm, to generate ten more ques-tion candidates per answer span. Hence, for everyanswer span we generate 13 question candidates.We discuss issues with using just standard beamsearch for question generation in Section 5.1.

3.3 Answering generated questions

While we condition our question generation modelon pre-selected answer spans, the generated ques-tions may not always correspond to these inputspans. Sometimes, the generated questions are ei-ther unanswerable or answered by a different spanin the paragraph. By running a pretrained QAmodel over the generated questions, we can detectquestions whose answers do not match their orig-inal input spans and filter them out. The predictedanswer for many questions has partial overlap withthe original answer span; in these cases, we dis-play the predicted answer span during evaluation,as a qualitative inspection shows that the predictedanswer is more often closer to the correct answer.For all of our experiments, we use the AllenNLPimplementation of the BiDAF++ question answer-ing model of Choi et al. (2018) trained on QuACwith no dialog context.

3.4 Question filtering

After over-generating candidate questions froma single answer span, we use simple heuristics

to filter out low-quality QA pairs. We removegeneric and duplicate question candidates9 andpass the remaining QA pairs through the multi-stage question filtering process described below.

Irrelevant or repeated entities: Top-k randomsampling often generates irrelevant questions;we reduce their incidence by removing anycandidates that contain nouns or entities unspec-ified in the passage. As with other neural textgeneration systems (Holtzman et al., 2018), wecommonly observe repetition in the generatedquestions and deal with this phenomenon by re-moving candidates with repeated nouns or entities.

Unanswerable or low answer overlap: Weremove all candidates marked as “unanswerable”by the question answering model, which prunes39.3% of non-duplicate question candidates.These candidates are generally grammaticallycorrect but considered irrelevant to the originalparagraph by the question answering model. Next,we compute the overlap between original andpredicted answer span by computing word-levelprecision and recall (Rajpurkar et al., 2016). ForGENERAL questions generated from sentencespans, we attempt to maximize recall by setting aminimum recall threshold of 0.3.10 Similarly, wemaximize recall for SPECIFIC questions generatedfrom named entities with a minimum recallconstraint of 0.8. Finally, for SPECIFIC questionsgenerated from sentence spans, we set a minimumprecision threshold of 1.0, which filters out ques-tions whose answers are not completely present inthe ground-truth sentence.

Low generation probability: If multiple candi-dates remain after applying the above filtering cri-teria, we select the most probable candidate foreach answer span. SPECIFIC questions generatedfrom sentences are an exception to this rule: forthese questions, we select the ten most probablecandidates, as there might be multiple question-worthy bits of information in a single sentence. Ifno candidates remain, in some cases11 we use afallback mechanism that sequentially ignores fil-ters to retain more candidates.

9Running Top-k random sampling multiple times can pro-duce duplicate candidates, including those already in the topbeams.

10Minimum thresholds were qualitatively chosen based onthe specificity type.

11For example, if no valid GENERAL questions for the en-tire paragraph are generated.

Subsequently, Yoda battles Palpatine in alightsaber duel that wrecks the SenateRotunda. In the end, neither is able to

overcome the other and Yoda is forced toretreat. He goes into exile on Dagobah so

that he may hide from the Empire andwait for another opportunity to destroy the

Sith. At the end of the film, it wasrevealed that Yoda has been in contactwith Qui-Gon's spirit, learning the secretof immortality from him and passing it on

to Obi-Wan.

GQ. What happened in thebattle with Palpatine?

SQ. Where wasthe battle?

SQ. Where did hego on exile?

GQ. What is revealed atthe end of the film?

SQ. Who does hewant to destroy?

Figure 3: Procedure used to form a QA hierarchy. Thepredicted answers for GQs ( GENERAL questions), areunderlined in blue. The predicted answers for SQs( SPECIFIC questions) are highlighted in red .

3.5 Forming a QA hierarchy

The output of the filtering module is an unstruc-tured list of GENERAL and SPECIFIC QA pairsgenerated from a single paragraph. Figure 3shows how we group these questions into a mean-ingful hierarchy. First, we choose a parent foreach SPECIFIC question by maximizing the over-lap (word-level precision) of its predicted an-swer with the predicted answer for every GEN-ERAL question. If a SPECIFIC question’s answerdoes not overlap with any GENERAL question’sanswer (e.g., “Dagobah” and “destroy the Sith”)we map it to the closest GENERAL question whoseanswer occurs before the SPECIFIC question’s an-swer (“What happened in the battle ...?”).12

3.6 Leveraging pretrained language models

Recently, pretrained language models based on theTransformer architecture (Vaswani et al., 2017)have significantly boosted question answering per-formance (Devlin et al., 2019) as well as the qual-ity of conditional text generation (Wolf et al.,2019). Motivated by these results, we modifycomponents of the pipeline to incorporate lan-guage model pretraining for our demo. Specif-ically, our demo’s question answering module isthe BERT-based model in Devlin et al. (2019),and the question generation module is trainedby fine-tuning the publicly-available GPT2-smallmodel (Radford et al., 2019). Please refer to Ap-pendix D for more details. These modificationsproduce better results qualitatively and speed upthe SQUASH pipeline since question overgenera-tion is no longer needed.

Note that the figures and results in Section 4 areusing the original components described above.

12This heuristic is justified because users read GEN-ERAL questions before SPECIFIC ones in our interface.

Experiment Generated GoldScore Fleiss κ Score Fleiss κ

Is this question well-formed? 85.8% 0.65 93.3% 0.54

Is this question relevant? 78.7% 0.36 83.3% 0.41(among well-formed) 81.1% 0.39 83.3% 0.40

Does the span partially contain the answer? 85.3% 0.45 81.1% 0.43(among well-formed) 87.6% 0.48 82.1% 0.42(among well-formed and relevant) 94.9% 0.41 92.9% 0.44

Does the span completely contain the answer? 74.1% 0.36 70.0% 0.37(among well-formed) 76.9% 0.36 70.2% 0.39(among well-formed and relevant) 85.4% 0.30 80.0% 0.42

Table 3: Human evaluations demonstrate the high individual QA quality of our pipeline’s outputs. All interanno-tator agreement scores (Fleiss κ) show “fair” to “substantial” agreement (Landis and Koch, 1977).

4 Evaluation

We evaluate our SQUASH pipeline on documentsfrom the QuAC development set using a vari-ety of crowdsourced13 experiments. Concretely,we evaluate the quality and relevance of individ-ual questions, the relationship between generatedquestions and predicted answers, and the struc-tural properties of the QA hierarchy. We empha-size that our experiments examine only the qualityof a SQUASHed document, not its actual useful-ness to downstream users. Evaluating usefulness(e.g., measuring if SQUASH is more helpful thanthe input document) requires systematic and tar-geted human studies (Buyukkokten et al., 2001)that are beyond the scope of this work.

4.1 Individual question quality and relevance

Our first evaluation measures whether questionsgenerated by our system are well-formed (i.e.,grammatical and pragmatic). We ask crowd work-ers whether or not a given question is both gram-matical and meaningful.14 For this evaluation, weacquire judgments for 200 generated QA pairs and100 gold QA pairs15 from the QuAC validation set

13All our crowdsourced experiments were conducted onthe Figure Eight platform with three annotators per example(scores calculated by counting examples with two or morecorrect judgments). We hired annotators from predominantlyEnglish-speaking countries with a rating of at least Level 2,and we paid them between 3 and 4 cents per judgment.

14As “meaningful” is potentially a confusing term forcrowd workers, we ran another experiment asking only forgrammatical correctness and achieved very similar results.

15Results on this experiment were computed after remov-ing 3 duplicate generated questions and 10 duplicate goldquestions.

(with an equal split between GENERAL and SPE-CIFIC questions). The first row of Table 3 showsthat 85.8% of generated questions satisfy this cri-terion with a high agreement across workers.Question relevance: How many generated ques-tions are actually relevant to the input paragraph?While the percentage of unanswerable questionsthat were generated offers some insight into thisquestion, we removed all of them during the filter-ing pipeline (Section 3.4). Hence, we display aninput paragraph and generated question to crowdworkers (using the same data as the previous well-formedness evaluation) and ask whether or notthe paragraph contains the answer to the question.The second row of Table 3 shows that 78.7% ofour questions are relevant to the paragraph, com-pared to 83.3% of gold questions.

4.2 Individual answer validity

Is the predicted answer actually a valid answerto the generated question? In our filtering pro-cess, we automatically measured answer overlapbetween the input answer span and the predictedanswer span and used the results to remove low-overlap QA pairs. To evaluate answer recall af-ter filtering, we perform a crowdsourced evalua-tion on the same 300 QA pairs as above by askingcrowdworkers whether or not a predicted answerspan contains the answer to the question. We alsoexperiment with a more relaxed variant (partiallycontains instead of completely contains) and re-port results for both task designs in the third andfourth rows of Table 3. Over 85% of predictedspans partially contain the answer to the gener-

Cowell formed a new companySyco, which is divided into threeunits - Syco Music, Syco TV and

Syco Film. Cowell returned tomusic with his latest brainchild

signed to Syco ...

What is Syco?

How many unitsdoes Syco have?

Returning home to Brantfordafter six months abroad, Bell

continued experiments with his"harmonic telegraph". The basic

concept behind his device was that messages could ...

What was Bell'stelegraph?

Where did he takehis experiments?

   After five years, however,Limon would return to Broadwayto star as a featured dancer inKeep Off the Grass under the

choreographer GeorgeBalanchine.

Why did he return toBroadway?

Who did he workwith?

Tan Dun earned widespreadattention after composing thescore for Ang Lee's Crouching

Tiger, Hidden Dragon (2000), forwhich he won an Academy

Award, a Grammy Award ....

How was Tan Dunreceived?

What award did hewin?

From 1969 to 1971, Cashstarred in his own television

show, The Johnny Cash Show,on the ABC network. The showwas performed at the Ryman

Auditorium in Nashville. ...

What did he do in 1969?

What network washe in?

Figure 4: SQUASH question hierarchies generated by our system with reference snippets . Questions in the hier-archy are of the correct specificity class (i.e., GENERAL , SPECIFIC ).

ated question, and this number increases if we con-sider only questions that were previously labeledas well-formed and relevant. The lower gold per-formance is due to the contextual nature of thegold QA pairs in QuAC, which causes some ques-tions to be meaningless in isolation (e.g.“What didshe do next?” has unresolvable coreferences).

Experiment Score Fleiss κ

Which question type asks formore information?

89.5% 0.57

Which SPECIFIC question iscloser to GENERAL QA?

different paragraph 77.0% 0.47same paragraph 64.0% 0.30

Table 4: Human evaluation of the structural correctnessof our system. The labels “different / same paragraph”refer to the location of the intruder question. The re-sults show the accuracy of specificity and hierarchies.

4.3 Structural correctness

To examine the hierachical structure of SQUASHed documents, we conduct three experiments.

How faithful are output questions to inputspecificity? First, we investigate whether ourmodel is actually generating questions with thecorrect specificity label. We run our specificityclassifier (Section 2) over 400 randomly sampledquestions (50% GENERAL, 50% SPECIFIC) andobtain a high classification accuracy of 91%.16

This automatic evaluation suggests the model iscapable of generating different types of questions.

Are GENERAL questions more representativeof a paragraph than SPECIFIC questions?To see if GENERAL questions really do providemore high-level information, we sample 200GENERAL-SPECIFIC question pairs17 grouped

16Accuracy computed after removing 19 duplicates.17We avoid gold-standard control experiments for struc-

tural correctness tests since questions in the QuAC datasetwere not generated with a hierarchical structure in mind. Pi-lot studies using our question grouping module on gold data

together as described in Section 3.5. For eachpair of questions (without showing answers), weask crowd workers to choose the question which,if answered, would give them more informationabout the paragraph. As shown in Table 4,in 89.5% instances the GENERAL question ispreferred over the SPECIFIC one, which confirmsthe strength of our specificity-controlled questiongeneration system.18

How related are SPECIFIC questions to theirparent GENERAL question? Finally, we inves-tigate the effectiveness of our question groupingstrategy, which bins multiple SPECIFIC QA pairsunder a single GENERAL QA pair. We show crowdworkers a reference GENERAL QA pair and askthem to choose the most related SPECIFIC ques-tion given two choices, one of which is the sys-tem’s output and the other an intruder question.We randomly select intruder SPECIFIC questionsfrom either a different paragraph within the samedocument or a different group within the sameparagraph. As shown in Table 4, crowd workersprefer the system’s generated SPECIFIC questionwith higher than random chance (50%) regardlessof where the intruder comes from. As expected,the preference and agreement is higher when in-truder questions come from different paragraphs,since groups within the same paragraph often con-tain related information (Section 5.2).

5 Qualitative Analysis

In this section we analyze outputs (Figure 4, Fig-ure 5) of our pipeline and identify its strengths andweaknesses. We additionally provide more exam-ples in the appendix (Figure A1).

led to sparse hierarchical structures which were not favoredby our crowd workers.

18We also ran a pilot study asking workers “Which ques-tion has a longer answer?” and observed a higher preferenceof 98.6% for GENERAL questions.

Weston was born Paul Wetstein in Springfield,Massachusetts, to Paul Wetstein, a teacher, and

Anna "Annie" Grady. The family moved toPittsfield when Weston was two, and he spent

his formative years in the town. His parents wereboth interested in music, and when Paul Sr

taught at a private girls' school, he was allowedto bring the school's gramophone ...

Q. What are his parents like? A. Paul Wetstein, a teacher, and Anna"Annie" Grady.

Q. Who was born in Springfield? A. Weston was born Paul Wetsteinin Springfield, Massachusetts, toPaul Wetstein, a teacher, andAnna "Annie" Grady.

Q. Where was Weston born? A. Springfield, Massachusetts, Q. Who were his parents? A. Paul Wetstein, a teacher, andAnna "Annie" Grady.

Q. Where did he move to? A. The family moved to PittsfieldQ. How old was Weston when hewas born? A. two

Q. How did he get into music?A. His parents were both interested inmusic, and when Paul Sr taught at a privategirls' school,

Q. Where did he go to school? A. Paul Sr taught at a private girls'school,

Paul Weston

...The treaty granted the United States control ofPuerto Rico, Guam, Cuba, the Philippines, and

parts of the West Indies. Many of Bryan'ssupporters were opposed to what they perceivedas Republican aspirations of turning the country

into an imperial power ... However, when theBacon Resolution (a proposed supplement to theTreaty of Paris which would allow the Filipinos a"stable and independent government") failed to

pass, Bryan began publicly speaking out againstthe Republicans' imperial aspirations.

William Bryan

Q. What was the treaty?A. The treaty granted the United Statescontrol of Puerto Rico, Guam, Cuba, thePhilippines, and parts of the West Indies.

Q. Where did the Treaty of Pariscome from?A. The treaty granted the UnitedStates control of Puerto Rico,Guam, Cuba, the Philippines, andparts of the West Indies.

Q. Why was this bad?A. Many of Bryan's supporters wereopposed to what they perceived asRepublican aspirations of turning thecountry into an imperial power

Q. What was a result of the resolution?A. failed to pass, Bryan began publiclyspeaking out against the Republicans'imperial aspirations.

Figure 5: Two SQUASH outputs generated by oursystem. The William Bryan example has interest-

ing GENERAL questions. The Paul Weston exampleshowcases several mistakes our model makes.

“In 1942, Dodds enlisted in the US army and served asan anti aircraft gunner during World War II.”

BIn what year did the US army take place?In what year did the US army take over?In what year did the US army take place in the US?

TWhat year was he enlisted?When did he go to war?When did he play as anti aircraft?

Table 5: Beam Search (B) vs Top-k sampling (T)for SPECIFIC question generation. Top-k candidatestend to be more diverse.

5.1 What is our pipeline good at?

Meaningful hierarchies: Our method of group-ing the generated questions (Section 3.5) produceshierarchies that clearly distinguish between GEN-ERAL and SPECIFIC questions; Figure 4 containssome hierarchies that support the positive resultsof our crowdsourced evaluation.

Top-k sampling: Similar to prior work (Fanet al., 2018; Holtzman et al., 2019), we noticethat beam search often produces generic or repet-itive beams (Table 5). Even though the top-kscheme always produces lower-probable questionsthan beam search, our filtering system prefers atop-k question 49.5% of the time.

5.2 What kind of mistakes does it make?

We describe the various types of errors our modelmakes in this section, using the Paul WestonSQUASH output in Figure 5 as a running example.Additionally, we list some modeling approacheswe tried that did not work in Appendix C.

Reliance on a flawed answering system: Ourpipeline’s output is tied to the quality of the pre-trained answering module, which both filters outquestions and produces final answers. QuAC haslong answer spans (Choi et al., 2018) that causelow-precision predictions with extra information(e.g., “Who was born in Springfield?”). Addi-tionally, the answering module occasionally swapstwo named entities present in the paragraph.19

Redundant information and lack of discourse:In our system, each QA pair is generated indepen-dently of all the others. Hence, our outputs lackan inter-question discourse structure. Our systemoften produces a pair of redundant SPECIFIC ques-tions where the text of one question answers theother (e.g., “Who was born in Springfield?” vs.“Where was Weston born?”). These errors canlikely be corrected by conditioning the generationmodule on previously-produced questions (or ad-ditional filtering); we leave this to future work.

Lack of world knowledge: Our models lackcommonsense knowledge (“How old was Westonwhen he was born?”) and can misinterpret polyse-mous words. Integrating pretrained contextualizedembeddings (Peters et al., 2018) into our pipelineis one potential solution.

Multiple GENERAL QA per paragraph: Oursystem often produces more than one tree perparagraph, which is undesirable for short, focusedparagraphs with a single topic sentence. To im-prove the user experience, it might be ideal to re-strict the number of GENERAL questions we showper paragraph. While we found it difficult togenerate GENERAL questions representative of en-tire paragraphs (Appendix C), a potential solutioncould involve identifying and generating questionsfrom topic sentences.

Coreferences in GENERAL questions: Manygenerated GENERAL questions contain corefer-ences due to contextual nature of the QuAC

19For instance in the sentence “The Carpenter siblingswere born in New Haven, to Harold B. and Agnes R.” themodel incorrectly answers the question “Who was born inNew Haven?” as “Harold B. and Agnes R.”

and CoQA training data (“How did he get intomusic?”). Potential solutions could involve ei-ther constrained decoding to avoid beams withanaphoric expressions or using the CorefNQGmodel of Du and Cardie (2018).

5.3 Which models did not work?

We present modelling approaches which did notwork in Appendix C. This includes, i) end-to-end modelling to generate sequences of questionsusing QuAC, ii) span selection NER system, iii)generation of GENERAL questions representativeof entire paragraphs, iv) answering system trainedon the combination of QuAC, CoQA and SQuAD.

6 Related Work

Our work on SQUASH is related to research inthree broad areas: question generation, informa-tion retrieval and summarization.

Question Generation: Our work builds uponneural question generation systems (Du et al.,2017; Du and Cardie, 2018). Our work conditionsgeneration on specificity, similar to difficulty-conditioned question generation (Gao et al., 2018).QA pair generation has previously been used fordataset creation (Serban et al., 2016; Du andCardie, 2018). Joint modeling of question genera-tion and answering has improved the performanceof individual components (Tang et al., 2017; Wanget al., 2017; Sachan and Xing, 2018) and enabledvisual dialog generation (Jain et al., 2018).

Information Retrieval: Our hierarchies are re-lated to interactive retrieval setting (Hardtke et al.,2009; Brandt et al., 2011) where similar webpagesare grouped together. SQUASH is also relatedto exploratory (Marchionini, 2006) and facetedsearch (Yee et al., 2003).

Summarization: Our work is related to query-focused summarization (Dang, 2005; Baumelet al., 2018) which conditions an output summaryon an input query. Hierarchies have also been ap-plied to summarization (Christensen et al., 2014;Zhang et al., 2017; Tauchmann et al., 2018).

7 Future Work

While Section 5.2 focused on shortcomings in ourmodeling process and steps to fix them, this sec-tion focuses on broader guidelines for future workinvolving the SQUASH format and its associatedtext generation task.

Evaluation of the SQUASH format: As dis-cussed in Section 1, previous research shows sup-port for the usefulness of hierarchies and QA inpedagogical applications. We did not directlyevaluate this claim in the context of SQUASH, fo-cusing instead on evaluating the quality of QApairs and their hierarchies. Moving forward, care-ful user studies are needed to evaluate the efficacyof the SQUASH format in pedagogical applica-tions, which might be heavily domain-dependent;for example, a QA hierarchy for a research paperis likely to be more useful to an end user than aQA hierarchy for an online blog. An importantcaveat is the imperfection of modern text gener-ation systems, which might cause users to pre-fer the original human-written document over agenerated SQUASH output. One possible solu-tion is a three-way comparison between the origi-nal document, a human-written SQUASHed doc-ument, and a system-generated output. For faircomparison, care should be taken to prevent exper-imenter bias while crowdsourcing QA hierarchies(e.g., by maintaining similar text complexity in thetwo human-written formats).

Collection of a SQUASH dataset: Besides mea-suring the usefulness of the QA hierarchies, alarge dedicated dataset can help to facilitate end-to-end modeling. While asking human annotatorsto write full SQUASHed documents will be ex-pensive, a more practical option is to ask themto pair GENERAL and SPECIFIC questions in ourdataset to form meaningful hierarchies and writeextra questions whenever no such pair exists.

QA budget and deeper specificity hierarchies:In our work, we generate questions for every sen-tence and filter bad questions with fixed thresh-olds. An alternative formulation is an adaptivemodel dependent on a user-specified QA budget,akin to “target length” in summarization systems,which would allow end users to balance cover-age and brevity themselves. A related modifi-cation is increasing the depth of the hierarchies.While two-level QA trees are likely sufficient fordocuments structured into short and focused para-graphs, deeper hierarchies can be useful for longunstructured chunks of text. Users can control thisproperty via a “maximum children per QA node”hyperparameter, which along with the QA budgetwill determine the final depth of the hierarchy.

8 Conclusion

We propose SQUASH, a novel text generation taskwhich converts a document into a hierarchy of QApairs. We present and evaluate a system whichleverages existing reading comprehension datasetsto attempt solving this task. We believe SQUASHis a challenging text generation task and we hopethe community finds it useful to benchmark sys-tems built for document understanding, questiongeneration and question answering. Additionally,we hope that our specificity-labeled reading com-prehension dataset is useful in other applicationssuch as 1) finer control over question generationsystems used in education applications, curiosity-driven chatbots and healthcare (Du et al., 2017).

Acknowledgements

We thank the anonymous reviewers for their in-sightful comments. In addition, we thank NaderAkoury, Ari Kobren, Tu Vu and the other membersof the UMass NLP group for helpful comments onearlier drafts of the paper and suggestions on thepaper’s presentation. This work was supported inpart by research awards from the Allen Institutefor Artificial Intelligence and Adobe Research.

ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2015. Neural machine translation by jointlylearning to align and translate. In Proc. Inter-national Conference on Learning Representations(ICLR).

Tal Baumel, Matan Eyal, and Michael Elhadad. 2018.Query focused abstractive summarization: Incorpo-rating query relevance, multi-document coverage,and summary length constraints into seq2seq mod-els. arXiv preprint arXiv:1801.07704.

Christina Brandt, Thorsten Joachims, Yisong Yue, andJacob Bank. 2011. Dynamic ranked retrieval. InProceedings of the fourth ACM international con-ference on Web search and data mining.

Orkut Buyukkokten, Hector Garcia-Molina, and An-dreas Paepcke. 2001. Seeing the whole in parts: textsummarization for web browsing on handheld de-vices. In Proceedings of the 10th international con-ference on World Wide Web.

Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettle-moyer. 2018. Quac: Question answering in context.In Proc. Conference on Empirical Methods in Natu-ral Language Processing (EMNLP).

Janara Christensen, Stephen Soderland, Gagan Bansal,et al. 2014. Hierarchical summarization: Scaling upmulti-document summarization. In Proc. Associa-tion for Computational Linguistics (ACL).

Hoa Trang Dang. 2005. Overview of duc 2005. InDocument Understanding Conferences.

Thomas H Davenport, David W De Long, andMichael C Beers. 1998. Successful knowledgemanagement projects. Sloan management review,39(2):43–57.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. Bert: Pre-training of deepbidirectional transformers for language understand-ing. In Proc. Conference of the North AmericanChapter of the Association for Computational Lin-guistics – Human Language Technologies (NAACLHLT).

Xinya Du and Claire Cardie. 2017. Identifying whereto focus in reading comprehension for neural ques-tion generation. In Proc. Conference on Em-pirical Methods in Natural Language Processing(EMNLP).

Xinya Du and Claire Cardie. 2018. Harvest-ing paragraph-level question-answer pairs fromwikipedia. In Proc. Association for ComputationalLinguistics (ACL).

Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn-ing to ask: Neural question generation for readingcomprehension. In Proc. Association for Computa-tional Linguistics (ACL).

Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hi-erarchical neural story generation. In Proc. Associ-ation for Computational Linguistics (ACL).

Yifan Gao, Jianan Wang, Lidong Bing, Irwin King, andMichael R Lyu. 2018. Difficulty controllable ques-tion generation for reading comprehension. arXivpreprint arXiv:1807.03586.

Matt Gardner, Joel Grus, Mark Neumann, OyvindTafjord, Pradeep Dasigi, Nelson Liu, Matthew Pe-ters, Michael Schmitz, and Luke Zettlemoyer. 2018.Allennlp: A deep semantic natural language pro-cessing platform. arXiv preprint arXiv:1803.07640.

Tom Gruber. 2008. Collective knowledge systems:Where the social web meets the semantic web.Web semantics: science, services and agents on theWorld Wide Web, 6(1).

Kai Hakkarainen and Matti Sintonen. 2002. The inter-rogative model of inquiry and computer-supportedcollaborative learning. Science & Education,11(1):25–43.

David Hardtke, Mike Wertheim, and Mark Cramer.2009. Demonstration of improved search result rel-evancy using real-time implicit relevance feedback.Understanding the User-Logging and Interpreting

User Interactions in Information Search and Re-trieval (UIIR-2009).

Michael Heilman and Noah A Smith. 2010. Goodquestion! statistical ranking for question generation.In Proc. Conference of the North American Chapterof the Association for Computational Linguistics –Human Language Technologies (NAACL HLT).

Peter Henderson, Riashat Islam, Philip Bachman,Joelle Pineau, Doina Precup, and David Meger.2018. Deep reinforcement learning that matters. InProc. Association for the Advancement of ArtificialIntelligence (AAAI).

Jaakko Hintikka. 1981. The logic of information-seeking dialogues: A model. Werner Becker andWilhelm K. Essler Konzepte der Dialektik, pages212–231.

Jaakko Hintikka. 1988. What is the logic of experi-mental inquiry? Synthese, 74(2):173–190.

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Longshort-term memory. Neural computation.

Ari Holtzman, Jan Buys, Maxwell Forbes, AntoineBosselut, David Golub, and Yejin Choi. 2018.Learning to write with cooperative discriminators.In Proc. Association for Computational Linguistics(ACL).

Ari Holtzman, Jan Buys, Maxwell Forbes, and YejinChoi. 2019. The curious case of neural text degen-eration. arXiv preprint arXiv:1904.09751.

Matthew Honnibal and Ines Montani. 2017. spacy 2:Natural language understanding with bloom embed-dings, convolutional neural networks and incremen-tal parsing. To appear.

Unnat Jain, Svetlana Lazebnik, and Alexander GSchwing. 2018. Two can play this game: visualdialog with discriminative question generation andanswering. In Proc. IEEE Computer Society Conf.Computer Vision and Pattern Recognition (CVPR).

Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Ky-omin Jung. 2018. Improving neural question gen-eration using answer separation. arXiv preprintarXiv:1809.02393.

Yoon Kim. 2014. Convolutional neural networks forsentence classification. In Proc. Conference on Em-pirical Methods in Natural Language Processing(EMNLP).

Diederik P. Kingma and Jimmy Ba. 2014. Adam: Amethod for stochastic optimization. In Proc. Inter-national Conference on Learning Representations(ICLR).

J Richard Landis and Gary G Koch. 1977. The mea-surement of observer agreement for categorical data.biometrics, pages 159–174.

Wendy G Lehnert. 1978. The process of question an-swering: A computer simulation of cognition, vol-ume 978. Lawrence Erlbaum Hillsdale, NJ.

Thang Luong, Hieu Pham, and Christopher D Man-ning. 2015. Effective approaches to attention-basedneural machine translation. In Proc. Conference onEmpirical Methods in Natural Language Processing(EMNLP).

Gary Marchionini. 2006. Exploratory search: fromfinding to understanding. Communications of theACM, 49(4):41–46.

Adam Paszke, Sam Gross, Soumith Chintala, Gre-gory Chanan, Edward Yang, Zachary DeVito, Zem-ing Lin, Alban Desmaison, Luca Antiga, and AdamLerer. 2017. Automatic differentiation in pytorch.In NIPS 2017 Autodiff Workshop.

Matthew Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word rep-resentations. In Proc. Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics – Human Language Technologies(NAACL HLT).

Alec Radford, Jeff Wu, Rewon Child, David Luan,Dario Amodei, and Ilya Sutskever. 2019. Languagemodels are unsupervised multitask learners.

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.Know what you don’t know: Unanswerable ques-tions for squad. In Proc. Association for Computa-tional Linguistics (ACL).

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, andPercy Liang. 2016. Squad: 100,000+ questions formachine comprehension of text. In Empirical Meth-ods in Natural Language Processing, pages 2383–2392.

Siva Reddy, Danqi Chen, and Christopher D. Manning.2018. Coqa: A conversational question answeringchallenge. arXiv preprint arXiv:1808.07042.

Mrinmaya Sachan and Eric Xing. 2018. Self-trainingfor jointly learning to ask and answer questions. InProc. Conference of the North American Chapter ofthe Association for Computational Linguistics – Hu-man Language Technologies (NAACL HLT).

Roger C Schank. 1972. Conceptual dependency: Atheory of natural language understanding. Cognitivepsychology, 3(4):552–631.

Abigail See, Peter J Liu, and Christopher D Manning.2017. Get to the point: Summarization with pointer-generator networks. In Proc. Association for Com-putational Linguistics (ACL).

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016. Neural machine translation of rare words withsubword units. In Proc. Association for Computa-tional Linguistics (ACL).

Iulian Vlad Serban, Alberto García-Durán, CaglarGulcehre, Sungjin Ahn, Sarath Chandar, AaronCourville, and Yoshua Bengio. 2016. Generat-ing factoid questions with recurrent neural net-works: The 30m factoid question-answer corpus.In Proc. Association for Computational Linguistics(ACL).

Manfred Stede and David Schlangen. 2004.Information-seeking chat: Dialogues driven bytopic-structure. In Proceedings of Catalog (the8th workshop on the semantics and pragmatics ofdialogue; SemDial04).

Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, andMing Zhou. 2017. Question answering and ques-tion generation as dual tasks. arXiv preprintarXiv:1706.02027.

Christopher Tauchmann, Thomas Arnold, AndreasHanselowski, Christian M Meyer, and MargotMieskes. 2018. Beyond generic summarization: Amulti-faceted hierarchical summarization corpus oflarge heterogeneous data. In Proceedings of theEleventh International Conference on Language Re-sources and Evaluation (LREC).

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Proc. Neural Information ProcessingSystems (NIPS).

Christian Wagner. 2004. Wiki: A technology for con-versational knowledge management and group col-laboration. Communications of the association forinformation systems, 13(1):19.

Christian Wagner and Narasimha Bolloju. 2005. Sup-porting knowledge management in organizationswith conversational technologies: Discussion fo-rums, weblogs, and wikis. Journal of DatabaseManagement, 16(2).

Tong Wang, Xingdi Yuan, and Adam Trischler. 2017.A joint model for question answering and questiongeneration. arXiv preprint arXiv:1706.01450.

Thomas Wolf, Victor Sanh, Julien Chaumond, andClement Delangue. 2019. Transfertransfo: A trans-fer learning approach for neural network based con-versational agents. CoRR, abs/1901.08149.

Mark Yatskar. 2019. A qualitative comparison of coqa,squad 2.0 and quac. Proc. Human Language Tech-nology/Conference of the North American Chap-ter of the Association for Computational Linguistics(HLT/NAACL).

Ka-Ping Yee, Kirsten Swearingen, Kevin Li, and MartiHearst. 2003. Faceted metadata for image searchand browsing. In Proceedings of the SIGCHI con-ference on Human factors in computing systems.

Amy X. Zhang, Lea Verou, and David Karger. 2017.Wikum: Bridging discussion forums and wikis usingrecursive summarization. In Conference on Com-puter Supported Cooperative Work and Social Com-puting (CSCW).

Appendix

A Question Classification Details

Confirming our intuition, Table 2 shows usthat QuAC has the highest percentage of GEN-ERAL questions. On the other hand CoQA andSQuAD, which allowed the question-asker to lookat the passage, are dominated by SPECIFIC ques-tions. These findings are consistent with acomparison across the three datasets in Yatskar(2019). Interestingly, the average answer lengthfor SPECIFIC questions in QuAC is 12 tokens,compared to 17 tokens for GENERAL questions.We provide the exact distribution of rule-labeled,hand-labeled and classifier-labeled questions inTable A1.

B Hyperparameters for QuestionGeneration

Our question generation system consists of a twolayer bidirectional LSTM encoder and a unidirec-tional LSTM decoder respectively. The LSTMhidden unit size in each direction and token em-bedding size is each set to 512. The class speci-ficity embeddings size is 16. Embeddings areshared between the paragraph encoder and ques-tion decoder. All attention computations use abilinear product (Luong et al., 2015). A dropoutof 0.5 is used between LSTM layers. Models aretrained using Adam (Kingma and Ba, 2014) witha learning rate of 10−3, with a gradient clipping of5.0 and minibatch size 32. Early stopping on val-idation perplexity is used to choose the best ques-tion generation model.

C What did not work?

End-to-End Sequential Generation. We ex-perimented with an end-to-end neural modelwhich generated a sequence of questions given asequence of answer spans. As training data, weleveraged the sequence IDs and follow-up infor-mation in the QuAC dataset, without specificity la-bels. We noticed that during decoding the modelrarely attended over the history and often pro-duced questions irrelevant to the context. A poten-tial future direction would involve using the speci-ficity labels for an end-to-end model.

Span Selection NER system. As discussed inSection 3.1 and Du and Cardie (2017), we could

frame answer span selection as a sequence la-belling problem. We experimented with the NERsystem in AllenNLP (with ELMo embeddings) onthe QuAC dataset, after the ground truth answerspans marked with BIO tags, after overlapping an-swers were merged together. We recorded lowF1 scores of 33.3 and 15.6 on sentence-level andparagraph-level input respectively.

Paragraph-level question generation. Ourquestion generation model rarely generatedGENERAL questions representative of the entireparagraph, even when we fed the entire paragraphas the answer span. We noticed that most GEN-ERAL questions in our dataset were answered byone or two sentences in the paragraph.

Answering system trained on all datasets. Re-cently, Yatskar (2019) reported small improve-ments on the QuAC validation set by pre-trainingthe BiDAF++ model on SQuAD 2.0 or CoQA.We tried combining the training data in all threedatasets but achieved a validation F1 score of just29.3 (compared to 50.2 after using just QuACtraining data).

Dataset Size Rule Hand CNN

SQuAD 86.8k 30.5% 0.0% 69.5%QuAC 65.2k 59.3% 1.5% 39.2%CoQA 105.6k 57.1% 0.1% 42.8%All 257.6k 48.7% 0.4% 50.9%

Table A1: Distribution of scheme adopted to classifyquestions in different datasets. “CNN” refers to thedata-driven classifier. Roughly half the questions wereclassified using the rules described in Table 1.

Before the final of the 100-meter butterfly, US bornSerbian swimmer Milorad Cavic caused a minor stir

when he said it would be "good" if Phelps lost. "It'd begood for him if he loses. It would be nice if historianstalk about Michael Phelps winning seven gold medalsand losing the eighth to 'some guy.' I'd like to be that

guy", Cavic said. Phelps responded, "When people saythings like that, it fires me up more than anything." OnAugust 16, Phelps won his seventh gold medal of the

Games in the men's 100-meter butterfly, setting anOlympic record for the event with a time of 50.58

seconds and edging out his nearest competitor Cavic,by one hundredth (0.01) of a second. 

Q. Why was he lost?A. "It'd be good for him if he loses

Q. What did Phelps do on August 16?A. On August 16, Phelps won his seventhgold medal of the Games in the men's 100-meter butterfly,

Q.Who did he win against?A. 100-meter butterfly, Q. Who is the Serbian swimmer?A. US born Serbian swimmerMilorad CavicQ. Who did he lose?A. Milorad Cavic

Q. When did he win a medal? A. On August 16

Q. How many gold medals didhe win?A. Phelps won his seventh goldmedal of the Games in the men's100-meter butterfly,

Q. Who did he beat? A. edging out his nearestcompetitor Cavic, by onehundredth (0.01) of a second.

On February 25, 2003 Converge released their firstofficial DVD, The Long Road Home. The DVD is

modeled after band home videos such as Metallica'sCliff Em' All release. Deathwish Inc describes the DVD

as a "two disc collection that is as energetic andexciting as the moments the release captures". The

DVD also comes with a bonus disk that included threefull live sets from the band.

Q. What did they do in 2003? A. On February 25, 2003 Convergereleased their first official DVD, The LongRoad Home.

Q. What was their first DVD?A. On February 25, 2003Converge released their firstofficial DVD, The Long RoadHome.Q. When did they release this?A. On February 25, 2003

Q. Where were the release?A. The Long Road Home.

Q.What was the DVD about?A. The DVD is modeled after band homevideos such as Metallica's Cliff Em' Allrelease.

Q. What other videos did theyhave?A. Metallica's Cliff Em' All release.

Q. How many sets were from theband?A. three full live sets from the bad

Q. What is Deathwise Inc?A. Deathwish Inc describes the DVD as a"two disc collection that is as energetic andexciting as the moments the releasecaptures".

Converge (band)Michael PhelpsOrson WellesBreaking with the Federal Theatre Project in 1937,Welles and Houseman founded their own repertory

company, which they called the Mercury Theatre. Thename was inspired by the title of the iconoclasticmagazine, The American Mercury. Welles was

executive producer, and the original company includedsuch actors as Joseph Cotten, George Coulouris,

Geraldine Fitzgerald, Arlene Francis, Martin Gabel,John Hoyt, Norman Lloyd, Vincent Price, Stefan

Schnabel and Hiram Sherman.

Q. What is Mercury Theatre?A. Breaking with the Federal TheatreProject in 1937, Welles and Housemanfounded their own repertory company,which they called the Mercury Theatre.

Q. What company did theyform?A. Federal Theatre Project

Q. When was the FederalTheatre Project founded?A. 1937,Q. Who started it?A. Welles and Houseman foundedtheir own repertory company,which they called the MercuryTheatre.

Q. Why was it called the FederalTheatre?A. The name was inspired by the title of theiconoclastic magazine, The AmericanMercury.

Q. What was the name of theiconoclastic magazine?A. The American Mercury

Q. Who was the producer?A. Welles was executive producer,and the original company includedsuch actors as Joseph Cotten,George Coulouris, GeraldineFitzgerald, Arlene Francis, MartinGabel,

Figure A1: Three SQUASH outputs generated by our system, showcasing the strengths and weaknesses describedin Section 5.

D Technical Note on Modified System

This technical note describes the modificationsmade to the originally published system to makea faster and more accurate system. We have in-corporated language modelling pre-training in ourmodules using GPT-2 small (Radford et al., 2019)for question generation and BERT (Devlin et al.,2019) for question answering. The official codeand demo uses the modified version of the system.

D.1 Dataset

The primary modification in the dataset tackles theproblem of coreferences in GENERAL questions,as described in Section 5.2. This is a commonproblem in QuAC and CoQA due to their con-textual setup. We pass every question through thespaCy pipeline extension neuralcoref20 usingthe paragraph context to resolve co-references. Wehave also black-listed a few more question tem-plates (such as “What happened in <year>?”) dueto their unusually high prevalence in the dataset.

D.2 Question Generation

Our question generation system is now fine-tunedfrom a pretrained GPT-2 small model (Radfordet al., 2019). Our modified system is basedon Wolf et al. (2019) and uses their codebase21 asa starting point.

We train our question generation model usingthe paragraph and answer as language modellingcontext. For GENERAL questions, our inputsequence looks like “<bos> ..paragraphtext.. <answer-general> ..answertext.. <question-general>..question text.. <eos>” and equiv-alently for SPECIFIC questions. In addition, weleverage GPT-2’s segment embeddings to denotethe specificity of the answer and question. Eachtoken in the input is assigned one out of fivesegment embeddings (paragraph, GENERAL an-swer, SPECIFIC answer, GENERAL question andSPECIFIC question). Finally, answer segmentembeddings were used in place of paragraphsegment embeddings at the location of the answerin the paragraph to denote the position of theanswer in the paragraph. For an illustration, referto Figure A2.

20https://github.com/huggingface/neuralcoref/

21https://github.com/huggingface/transfer-learning-conv-ai

GPT-2, Causal TransformerLanguage Model

Word Embeddings

PositionalEmbeddings

Segment Embeddings SA P P SA SQ SQPPP

P

SA

SQ

paragraph segment

SPECIFIC answer segment

SPECIFIC question segment

Paragraph Ans QuestionInput

Figure A2: An illustration of the model used for gener-ating a SPECIFIC question. A paragraph (context), an-swer and question and concatenated and the model isoptimized to generate the question. Separate segmentembeddings are used for paragraphs, GENERAL an-swers, GENERAL questions, SPECIFIC answers andSPECIFIC questions. Note that the answer segment em-bedding is also used within the paragraph segment todenote the location of the answer.

The question generation model now uses top-pnucleus sampling with p = 0.9 (Holtzman et al.,2019) instead of beam search and top-k sampling.Due to improved question generation quality, weno longer need to over-generate questions.

D.3 Question AnsweringWe have switched to a BERT-based question an-swering module (Devlin et al., 2019) which istrained on SQuAD 2.0 (Rajpurkar et al., 2018).We have used an open source PyTorch implemen-tation to train this model22.

D.4 Question FilteringWe have simplified the question filtering processto incorporate a simple QA budget (described inSection 7). Users are allowed to specify a cus-tom “GENERAL fraction” and “SPECIFIC fraction”which denotes the fraction of GENERAL and SPE-CIFIC questions retained in the final output.

22https://github.com/huggingface/pytorch-pretrained-BERT