Proceedings of the BioNLP 2018 workshop, pages 35–46Melbourne, Australia, July 19, 2018. c©2018 Association for Computational Linguistics

35

Identifying Key Sentences for Precision Oncology Using Semi-SupervisedLearning

Jurica Seva, Martin Wackerbauer and Ulf leserKnowledge Managment in Bioinformatics

Humboldt Universitat zu BerlinBerlin, Germany

{seva,wackerbm,leser}@informatik.hu-berlin.de

Abstract

We present a machine learning pipelinethat identifies key sentences in abstractsof oncological articles to aid evidence-based medicine. This problem is charac-terized by the lack of gold standard data-sets, data imbalance and thematic differ-ences between available silver standardcorpora. Additionally, available trainingand target data differs with regard to theirdomain (professional summaries vs. sen-tences in abstracts). This makes super-vised machine learning inapplicable. Wepropose the use of two semi-supervisedmachine learning approaches: To mit-igate difficulties arising from heterogen-eous data sources, overcome data imbal-ance and create reliable training data wepropose using transductive learning frompositive and unlabelled data (PU Learn-ing). For obtaining a realistic classifica-tion model, we propose the use of abstractssummarised in relevant sentences as un-labelled examples through Self-Training.The best model achieves 84% accuracyand 0.84 F1 score on our dataset.

1 Introduction

The ever-growing amount of biomedical literat-ure accessible online is a valuable source of in-formation for clinical decisions. The PubMeddatabase (National Library of Medicine, 1946-2018), for instance, lists approximately 30 millionarticles’ abstracts. As a consequence, machinelearning (ML) based text mining (TM) is increas-ingly employed to support evidence-based medi-cine by finding, condensing and analysing relev-ant information (Kim et al., 2011). Practitionersin this field search for clinically relevant articles

and findings, and are typically not interested in thebulk of search results which are devoted to basicresearch. However, defining clinical relevance in agiven abstract is not a trivial task. On top, althoughabstracts provide a very brief summary of theircorresponding articles’ content, practitioners de-termine abstracts’ clinical relevance based on onlya few key sentences (McKnight and Srinivasan,2003). To optimally support such users, it is thusnecessary to first retrieve only clinically relevantarticles and next to identify the sentences in thosearticles which express their clinical relevance.

Any survival benefit of dMMR was lost inN2 tumors. Mutations in BRAF(V600E)(HR, 1.37; 95% CI, 1.08 to 1.70; P = .009)or KRAS (HR, 1.44; 95% CI, 1.21 to 1.70;P ¡ .001) were independently associatedwith worse DFS. The observed MMR by tu-mor site interaction was validated in an in-dependent cohort of stage III colon cancers(P(interaction) = .037).

Example 1: Snippet of highlighted clin-ically relevant (or key; yellow backgroundcolor) and irrelevant (no background color)sentences in a precision oncology setting.Source document with PMID 24019539.

In this work, we present an ML pipeline toidentify key (clinically relevant) sentences, in aprecision oncology setting, in abstracts of oncolo-gical articles to aid evidence-based medicine. Thissetting is implied throughout the text when refer-ring to clinical relevance or key (clinically relev-ant) sentences. An example of relevant and ir-relevant sentences is shown in Example 1. Forsolving this problem no gold standard corporais available. Additionally, clinical relevance has

36

only a vague definition and is a subjective meas-ure. As manually labelling text is expensive, semi-supervised learning offers the possibility to util-ize related annotated corpora. We focus on Self-Training (Wang et al., 2008), which mostly relieson supervised classifiers trained on labelled dataand use of unlabelled examples to improve the de-cision boundary. Several corpora can be used tomitigate the issues arising from the lack of goldstandard data set and data imbalance. These cor-pora implicitly define characteristics of key sen-tences, but cannot be considered as gold standards.In the following, we call them “silver standard”corpora - collections of sentences close to the in-tended semantic but with large amounts of noise.Specifically, we employ Clinical Interpretations ofVariants in Cancer (CIViC) (Griffith et al., 2017)for implicit notion of clinical relevance and posit-ive data points, i.e. sentences or abstracts whichhave clinical relevance. Unfortunately, negativedata points, i.e. sentences or abstracts which donot have clinical relevance, are not present in thisdata set. PubMed abstracts, referenced by CIViC,are used as unlabelled data. Since we considerall sentences in CIViC to be positive examplesand the corresponding abstracts are initially un-labelled, additional data for negative examples isrequired. We utilize the Hallmarks of CancerCorpus (HoC) (Baker et al., 2016) as an auxili-ary source of noisy labelled data. To expand onour set of labelled data points we propose trans-ductive learning from positive and unlabelled data(PU Learning) to identify noise within HoC, withCIViC as a guide set for determining the relevanceof sentences from HoC. This gives us additional,both positive and negative data points, used as aninitialization for Self-Training. The pipeline isavailable at https://github.com/nachne/semisuper.

2 Related Work

Sentence classification is a special case of text cat-egorisation. It has been used in a wide range offields, like sentiment analysis (Yu and Hatzivassi-loglou, 2003; Go et al., 2009; Vosoughi et al.,2015), rhetorical annotation, and automated sum-marisation (Kupiec et al., 1995; Teufel and Moens,2002). Between the two, feature engineering hasbeen reported as the major difference. For in-stance, common stop words like “but”, “was”, and“has” are often among the top features for sen-tence classification, and verb tense is useful to

determine a sentence’s precise meaning (Agarwaland Yu, 2009; Khoo et al., 2006). Additional fea-tures beyond the pure language level have alsobeen proposed. For sentiment analysis, Yu andHatzivassiloglou (2003) use a dictionary of se-mantically meaningful seed words to estimate thelikely positive or negative polarity of co-occurringwords, from which in turn a sentences’ polarity isdetermined. Teufel and Moens (2002) focus onidentifying rhetorical roles of sentences for auto-matic summarisation of scientific articles. Theyuse sentence length and location, the presence ofcitations and of words included in headlines, la-bels of preceding sentences, and predefined cuewords and formulaic expressions accompanyingBag of Words (BOW).

Text represented as high dimensional BOW vec-tors has been reported to be often linearly separ-able, making Support Vector Machines (Joachims,1998) (SVM) a popular choice for classifiers.Conditional Random Fields (CRF) have been usedto predict sequences of labels rather than labellingsentences one by one (Kim et al., 2011). In recentyears, Neural Networks (NN) and Deep Learn-ing (DL) has increasingly been used, e.g. us-ing Convolutional Neural Networks (CNN) (Kim,2014; Rios and Kavuluru, 2015; Zhang et al.,2016; Conneau et al., 2017). Other authors employvarious versions of Recurrent Neural Networks(RNN): LSTM (Hassan and Mahmood, 2017), bi-directional LSTM (Dernoncourt et al., 2017; Zhouet al., 2016) or convolutional LSTM (Zhou et al.,2015). The use of DL has also popularised the useof pre-trained word embedding vectors. Habibiet al. (2017) show that the use of word embed-dings, in general, increases the quality of biomed-ical named entity recognition pipelines.

Specific to the biomedical domain, sentenceclassification has been used to determine the rhet-orical role sentences play in an article or abstract.Ruch et al. (2007) propose using the “Conclu-sion” section of abstracts as examples for key sen-tences. McKnight and Srinivasan (2003) haveclassified sentences in abstracts of randomisedcontrol trials as belonging to the categories “Intro-duction”, “Methods”, “Results”, and “Discussion”(IMRaD), using section headlines as soft labels fortraining data in addition to a smaller hand annot-ated corpus. They also report that adding sentencelocation as a feature improved performance on the“Introduction” and “Discussion” categories. Kim

37

et al. (2011) used a CRF for sequential classifica-tion, trained on the hand-annotated Population, In-tervention, Background, Outcome, Study Designof evidence-based medicine, or Other (PIBOSO)corpus, with sentences annotated with one of theaforementioned categories. Unfortunately, sincesentences in our primary source of positive dataare from a different context than the abstracts to beclassified, section headings, preceding sentences,and location are not available for our task.

2.1 Semi-Supervised Learning

Semi-supervised learning has the potential tomatch the performance of supervised learningwhile requiring considerably less labelled data(Wang et al., 2008; Thomas et al., 2012; Liuet al., 2013). Soft labelling (e.g. aforementionedheuristics for using section headlines as labels)is sometimes subsumed under semi-supervisedlearning as Distant Supervision (Go et al., 2009;Vosoughi et al., 2015; Wallace et al., 2016). La-bel Propagation (Zhu and Ghahramani, 2002) andLabel Spreading (Zhou et al., 2003) can be seen aslargely unsupervised classification, using labelleddata to initialise and control clustering. Likewise,Nigam et al. (2011) propose Naive Bayes (NB)based variants of the unsupervised Expectation-Maximisation (EM) algorithm for utilising unla-belled data in semi-supervised text classification.

2.1.1 PU LearningPU Learning is a special case of semi-supervisedlearning where examples in the unlabelled set Uare to be classified as positive (label 1) or neg-ative (label 0), with only positive labelled dataP initially available. Therefore, the PU Learn-ing problem can be approximated by learningto discriminate P from U (Mordelet and Vert,2014). For that, learning should favour false pos-itive errors over false negatives, e.g. by usingclass-specific weights for error penalisation. Ap-proaches include one-class SVMs, which approx-imate the support of the positive class and treatnegative examples as outliers; ranking methods,which rank unlabelled examples by their decreas-ing similarity to the mean positive example; andtwo-step heuristics, which try to identify reliablenegative examples in the unlabelled data to ini-tialise semi-supervised learning. We consider theaforementioned heuristics useful for outlier detec-tion to reduce noise in our auxiliary data, anduse variations of PU Learning algorithms in semi-

supervised learning, as our problem of findingsummary-like sentences without explicitly definednegative sentences is closely related to PU Learn-ing. An overview is available in (Liu et al., 2003).Additional information can be found in (Elkan andNoto, 2008; Plessis et al., 2014, 2015). An ex-ample of the use of PU Learning, for spotting on-line fake reviews, is available in (Li et al., 2014).

Without known negative examples, measuringclassification performance using accuracy or F1-score in PU Learning is not possible. Lee andLiu (2003) suggest an alternative score, calledPU-score, defined as Pr[f(X) = 1|Y =1]2/Pr[f(X) = 1], for comparing PU Learningclassifiers that can be derived from positive andunlabelled data alone. The authors show theoretic-ally that maximising the PU-score is equivalent tomaximising the F1-score and can be used to com-pare different models classifying the same data.Nonetheless, it should be noted that this metric isnot bounded, making it viable only for comparingclassifiers trained and tested on the same data; it isnot an indicator for an individual classifier’s per-formance.

2.1.2 Self-TrainingSelf-Training, used in this work, starts from an ini-tial classifier trained on the labelled data. Previ-ously unlabelled examples that were labelled withhigh confidence are added to the training data.This procedure repeats iteratively, retraining theclassifier until a terminating condition is met. NBis a popular classifier for Self-Training becausethe probabilities it produces provide a confidenceranking, but any other algorithm may be used aslong as confidence scores can be derived (Wanget al., 2008).

3 Methods

We present the data sources we use, the prepro-cessing pipeline and describe in detail the experi-ments performed with both PU Learning and Self-Training.

3.1 Used CorporaCIViC is a database of clinical evidence summar-ies. Entries consist of evidence statements aboutgene variants, such as their association with dis-eases and the outcome of drug response trials. Ad-ditional information includes the names of the re-spective genes, variants, drugs, diseases, and avariant summary. Each entry contains the PubMed

38

ID of the respective publication the information istaken from. Evidence statements are prototypesof high-quality information in condensed form.However, they are not themselves contained in theabstracts they are based on, as they try to summar-ize the entire article. At time of writing, CIViCcontains about 2,300 evidence statements consist-ing of 6,600 sentences (5,300 without duplicates).They make up our initial corpus of positive sen-tences (P ).

PubMed abstracts referenced in CIViC. We ex-tract about 12,700 sentences from 1,300 abstractsreferenced in CIViC, and use them as the unla-belled corpus (U ). We use CIViC summaries andthe PubMed abstract corpus to estimate the ac-ceptable range for the ratio of key sentences inan abstract. We use the ratio of overall sentencecounts in the two corpora (CIViC summaries andPubMed abstracts) as an upper bound of ≈ 0.4(5,300/12,700). As a rough estimate for the lowerbound, based on an informed guess that half of thesentences could be redundant, since one abstractmay correspond to multiple CIViC entries for dif-ferent drug/variant combinations. This results in alower bound ≈ 0.2. Although this is a simplifyingassumption and disregards e.g. any differences ininformation density in our data sources’ sentences,it provides a rough guideline for the ratio of keysentences in U a classifier should find.

Hallmarks of Cancer (HoC) describe commontraits of all forms of cancer. We use it as a silverstandard corpus consisting of about 13,000 sen-tences from 1,580 PubMed abstracts. Sentencesnot relevant to any of the hallmarks are left unla-belled. We assume unlabelled sentences are lesslikely to be clinically relevant than sentences withone or more labels, aggregating them in the likelynegative set HoCn (about 8,900 sentences) andthe likely positive set HoCp (about 4,300 sen-tences). In order to improve generalisation, as wellas to be able to validate our classifier, which re-quires positive as well as negative labelled data,we use HoC as auxiliary data. To utiliseHoCp andHoCn as sources of realistic positive and negat-ive sentences for training and test data, but avoid-ing propagation of misclassification errors result-ing from our simplifying assumption, they must befiltered for noise (Section 3.3).

3.2 Text Preprocessing and Feature Selection

As features, we use word n-grams, character n-grams, and sentence length, concatenating them toform a mixed feature space. All tokens are con-verted to lower-case. Biomedical scientific textexhibits some particularities that have to be takeninto consideration during text preprocessing. Tonormalise all text, before sentence splitting withthe PunktSentenceTokenizer of the Python NaturalLanguage Toolkit (NLTK) (Bird et al., 2009), weuse regular expressions: we substitute spaces forfull stops in common abbreviations followed by aspace and lower-case letter or digit (e.g. “ca. 5”→ “ca 5”). As the pattern “patient no. V[123]”is quite frequent in CIViC, we introduce a specialrule for not splitting it despite the upper-case. Allwhitespace characters are replaced by spaces toavoid splitting on newlines. Afterwards, to avoidcharacter encoding-related problems and to reducealphabet size, we normalize all text to ASCII be-fore tokenization.

For word-level tokenization, we use NLTK’sTreebankWordTokenizer and split the resultingtokens at characters in {“-”, “/”, “.”, “,”, “—”, “¡”,“¿”}. Sentences below a minimum character countof 8 are denoted by a special “ empty sentence ”token. To prepare word n-grams, we replacetokens representing numbers or ordinals and theirspelled out versions by a special “ num ” tokenand do the equivalent for e.g. ranges, inequal-ities, percentages, measurement units, and years.Tokens with suffixes common for drugs but notfound in common speech, such as “-inib”, are re-placed by “ chemical ”, and sequences that startwith a letter, but contain digits, are replaced by“ abbrev ” in the hope of catching identifiers ofbiomedical entities. We evaluated the use of wordn-grams with n bounded from (1,1) (the bag-of-words case) up to (1,4), thereby retaining inform-ation about the order of words in a sentence.

We evaluated the use of character n-gram withn in ranges from (2,3) to (2,6). To reduce alphabetsize and avoid overfitting, all sequences of non-word characters except those in {“-”, “%”, “=”},which may carry semantic information, are re-placed by single spaces, and all digits are replacedby 1.

In feature vectors, word and character n-gramsare weighted by their tf-idf score (Aizawa, 2003)and sentence length is represented as inverse char-acter count. Character n-grams proved to be more

39

expressive than word n-grams, yielding better ac-curacy scores when we tested each of them inisolation. However, a combination of characterand word level n-grams and text length performedbest.

3.3 Noise reduction with PU Learning

Using PU Learning, we filterHoCn andHoCp forsentences that are likely to be useful in our classi-fication task. We explored several approaches toPU Learning and subsequent noise reduction.

3.3.1 PU LearningFirst, we explored several Two-Step techniques,which (1) identify a set of reliable negative ex-amples (RN ) from U and (2) train a classifieron P and RN and retrain the classifier using thepredicted labels until a stopping criterion is met.We present them next. i-EM is a variation of theExpectation-Maximisation algorithm that relies ona NB classifier, with predicted probabilities of la-bels in range [0, 1]. s-EM is an extension to i-EM.Initially, a subset S ⊂ P is added to U as spy doc-uments. After training an initial classifier on P \Svs. U ∪ S, the predicted probabilistic labels forthe known hidden positives in S are used to de-termine a threshold t; all u ∈ U with probabilisticlabels p(yu) < t are moved to RN . In Step 2,starting from a classifier trained to discriminate Pfrom RN , the EM algorithm is iterated as in i-EMuntil it converges or the estimated classification er-ror is deteriorating. Roc-SVM uses the Rocchioalgorithm for Step 1: Firstly, prototype vectors pfor P and u for U are computed as a weighted dif-ferences between the two sets’ respective averageexamples. Using these vectors, RN is defined as{u ∈ U : cos(u, u) < cos(u, p)}, i.e. all unla-belled sentences that are more similar to the pro-totype of the unlabelled set than the positive sets.Step 2 uses SVMs to expand RN . Initially, anSVM is trained to discriminate P from RN . Af-terwards, all u ∈ U \ RN with predicted label0 are added to RN for iteratively retraining theclassifier as long as RN changes. This iterationmay go wrong and result in poor recall on the pos-itive class; as a fallback strategy, if the classifierat convergence misclassifies too large a portion ofP , the initial classifier is returned instead. CR-SVM is a minor extension to Roc-SVM. P andU are each ranked by decreasing cosine similarityto the mean positive example; a probably negat-ive set PN is built from the u ∈ U with a lower

score than a given ratio of least typical examplesin P . The negative prototype vector is then com-puted using PN rather than U . Step 2 is the sameas in Roc-SVM. Additionally, we explored BiasedSVM, a soft-margin SVM that uses class-specificweights for positive and negative errors. Weightparameters are selected in a grid search mannerto find a combination that optimises the PU-score;this effectively assumes U to contain only negli-gible amounts of hidden positive examples.

3.3.2 Noise reductionWe experiment with two heuristics for noise re-duction in HoC. For both of them, let clf(P,U)(x)be the label for x predicted by classifier clf trainedon P and U . Appendix B (Figure 1) summarisescorpora used for this task.

Strict mode: Remove CIViC-like sentences,i.e. likely hidden positives, from HoCn for thereliable negative set HoC ′

n. Keep only CIViC-like sentences in HoCp for a reliable positive setHoC ′

p. This implies rather pessimistic assump-tions about HoCp’s relevance, considering onlyoutliers as key sentences.

HoC′n := HoCn \ {x ∈ HoCn : clf(CIV iC,HoCn)(x) = 1}

HoC′p := {x ∈ HoCp : clf(CIV iC,HoCp)(x) = 1}

Tolerant mode: Remove CIViC-like sentencesfrom HoCn as before. But rather than requiringsentences from HoCp to be CIViC-like, removethose sentences from HoCp that are similar to re-liable negative sentences, i.e. the purified HoC ′

n.In doing so, HoCp is assumed to be largely relev-ant, contaminated with non-key sentences.

HoC′n := HoCn \ {x ∈ HoCn : clf(CIV iC,HoCn)(x) = 1}

HoC′p := HoCp \ {x ∈ HoCp : clf(HoC′

n, HoCp)(x) = 1}

3.4 Semi-supervised learning withSelf-Training

In the following, let the labelled set be L :=P ∪N , with positive labelled set P := CIV iC ∪HoC ′

p and negative labelled set N := HoC ′n. The

unlabelled set of original abstracts is denoted byU . The purified sets HoC ′

p and HoC ′n are ob-

tained using either of the above heuristics. Ap-pendix B (Figure 2) summarises corpora usedfor this task. We use: Standard Self-Training(ST) with a confidence threshold. Having exper-imented with different values, we use a threshold

40

Reliable negatives: Strict mode: Tolerant mode:P := CIV iC, P := CIV iC, P := HoC ′

n

U := HoCn U := HoCp U := HoCp

PU- pos. ratio PU- pos. ratio PU- pos. ratioMethod score in Utest score in Utest score in Utest

i-EM 2.06 0.11 1.61 0.06 0.94 0.36s-EM 2.06 0.11 1.61 0.06 0.94 0.40Roc-SVM 2.19 0.07 1.67 0.06 1.07 0.31CR-SVM 2.19 0.08 1.67 0.06 1.04 0.57Biased-SVM 2.28 0.03 1.70 0.05 1.13 0.31

Table 1: Removing noise fromHoCn andHoCp. Results for different PU Learning techniques, averagedover 10 runs, on 20% reserved test sets Ptest ⊂ P and Utest ⊂ U . To generate HoC ′

n as required fortolerant mode, Roc-SVM (highlighted in bold) was used in the previous step.

Algorithm 1 Self-Training1: procedure SELF-TRAINING(training data L

with labels, unlabelled data U )2: while U is not empty do3: train classifier clf on L4: predict labels for U with clf5: move examples with most confidently

predicted labels from U to L6: end while7: return clf8: end procedure

of 0.75 for classifiers producing class probabilit-ies, and 0.5 for the absolute values of SVM’s de-cision function; “Negative” Self-Training (NST):Rather than using a confidence criterion, all unla-belled examples classified as negative are added tothe training data for the next iteration. This is ana-logous to the iterative SVM step of Roc-SVM, ex-cept for the predefined rather than heuristically es-timated initial negative set, and has shown to helpavoid an unrestricted propagation of positive la-bels; A variant of the Expectation-Maximisation(EM) algorithm as used in i-EM. Starting withP and N as initial fixed-label examples, iter-ate a NB classifier until convergence, using theclass probabilities predicted for U as labels forthe next training iteration; Label Propagation andLabel Spreading: These algorithms propagate la-bels through high-density regions using a graphrepresentation of the data. Both are implemen-ted in Scikit-learn with Radial Basis Function(RBF) and k-Nearest-Neighbour (kNN) kernelsavailable. We were unable to obtain competitiveresults with these techniques. In the Self-Training

algorithm (shown in Algorithm 1), we use Scikit-learn’s implementations of SVM, NB, and Lo-gistic Regression (LR) as underlying classifiers.

4 Results

Section 4.1 describes the effects of noise reductionheuristics using PU Learning. The performancesof different semi-supervised approaches for train-ing a classifier, with both strict and tolerant noisereduction scenarios, are shown in Section 4.2.

4.1 PU Learning for Noisy Data

Table 1 summarises the PU-scores and ratio of ex-amples in U classified as positive for different al-gorithms for reducing noise in HoCn and HoCp

using the strict vs. tolerant heuristics.Cleaning up HoCn removes some 2 to 7% of

examples, depending on the classifier. Additionalmanual inspection of a subset of the sentences re-moved confirms them as true negatives with re-spect to key sentences.

Regarding HoCp, the strict heuristics keepsonly 5.5%, some 250 sentences, of positive ex-amples. We suspect this is due to the differentthematic foci of HoC and the articles summar-ised in CIV iC, as well as the summaries’ differ-ent writing style. This leaves us with N := 8,300sentences, P := 5,600 sentences and U := 12,700sentences. As our experiments show, choosing thisvery selective approach drastically improves thenominal accuracy of subsequent steps; however,it leaves a lack of real-world data in the positivetraining set and harbours the risk of overfitting.

On the other hand, using the tolerant strategy,roughly 25% of HoCp are removed due to be-ing very similar to HoC ′

n. This results in a 50%

41

Ptest : Ntest : U :Method parameters acc p r F1 p r F1 pos. ratioNST(SVM) C = 0.3 0.94 0.95 0.90 0.92 0.93 0.97 0.95 0.33NST(LR) C = 6.0 0.89 0.99 0.75 0.84 0.86 0.99 0.92 0.13NST(NB) α = 0.1 0.90 0.97 0.78 0.86 0.87 0.98 0.92 0.31ST(SVM) C = 0.4 0.96 0.97 0.93 0.95 0.95 0.98 0.97 0.62ST(LR) C = 4.0 0.96 0.96 0.94 0.95 0.96 0.98 0.97 0.62ST(NB) α = 0.1 0.92 0.93 0.88 0.90 0.92 0.96 0.94 0.60EM α = 0.1 0.91 0.92 0.85 0.88 0.91 0.95 0.93 0.62Label Propagation RBF kernel 0.83 0.91 0.63 0.74 0.79 0.96 0.87 0.35Label Propagation kNN kernel 0.69 0.96 0.22 0.36 0.66 0.99 0.79 0.03Label Spreading RBF kernel 0.85 0.93 0.68 0.79 0.82 0.97 0.89 0.50Label Spreading kNN kernel 0.79 0.92 0.54 0.68 0.76 0.96 0.85 0.32

Table 2: Performance of different semi-supervised approaches trained on P , N , and U after strict noisefiltering. ST = Self-Training. NST = “Negative” Self-Training. Results averaged over 10 runs withrandomised 20% validation sets from P and N ; min-df threshold = 0.002, 25% of most relevant featuresselected with χ2.

Ptest : Ntest : U :Method parameters acc p r F1 p r F1 pos. ratioNST(SVM) C = 0.3 0.84 0.84 0.84 0.84 0.84 0.83 0.83 0.32NST(LR) C = 6.0 0.81 0.88 0.72 0.79 0.76 0.90 0.82 0.17NST(NB) α = 0.1 0.76 0.85 0.64 0.73 0.70 0.88 0.78 0.30ST(SVM) C = 0.4 0.85 0.90 0.81 0.85 0.83 0.89 0.86 0.62ST(LR) C = 6.0 0.86 0.87 0.85 0.85 0.84 0.86 0.85 0.66ST(NB) α = 0.1 0.76 0.88 0.62 0.72 0.69 0.91 0.79 0.70EM α = 0.1 0.74 0.88 0.58 0.70 0.68 0.91 0.78 0.70Label Propagation RBF kernel 0.72 0.88 0.50 0.64 0.64 0.92 0.76 0.36Label Propagation kNN kernel 0.58 0.90 0.20 0.32 0.54 0.98 0.70 0.02Label Spreading RBF kernel 0.74 0.88 0.56 0.68 0.67 0.92 0.77 0.56Label Spreading kNN kernel 0.68 0.91 0.43 0.58 0.62 0.96 0.77 0.34

Table 3: Performance of different semi-supervised approaches trained on P , N , and U after tolerantnoise filtering. Results averaged over 10 runs with randomised 20% validation sets from P and N ;min-df threshold = 0.002, 25% of most relevant features selected with χ2. The model we consider mostsuitable for identifying key sentences is highlighted in bold.

larger and less homogenous positive labelled setcompared to strict noise filtering, which we expectto provide greater generality and robustness to ourclassifier. This leaves us with N := 8,300 sen-tences, P := 8,600 sentences and U := 12,700sentences. This is enough to assume a noticeablereduction of noise and easier distinction betweenHoC ′

n and HoC ′p, but it still contributes a con-

siderable amount of data to the positive set andis not suspect to overfitting. Typical topics ofsentences removed as irrelevant include biochem-ical research hypotheses and non-human studysubjects; however, as this heuristic is indirectly

defined, its decisions are not quite as clearly cor-rect as those directly linked to CIViC.

Our results confirm Biased-SVM nominallyperforms best among the PU Learning techniquesdescribed above; this is simply because the PU-score is maximised by minimising the amount ofpositive examples found in U , which Biased-SVMdoes by regarding U as negative and performingsupervised classification. However, we do not findthis to be useful for our purpose of noise detection,or for finding hidden positive data in unlabelleddata in general. The EM-based techniques tend togo the opposite direction and consider comparably

42

large ratios of U as positive, were more sensitiveto distributions, and misclassified positive labelleddata. Roc-SVM, on the other hand, had stable per-formance in our tests and scores close to those ofBiased-SVM, which is why we use this approachto filter HoC for the subsequent steps. Our res-ults also suggest the iterative second step is morecrucial than the exact heuristics for choosing a re-liable negative set from the unknown data.

4.2 Semi-Supervised classification of keysentences with Self-Training

We report accuracy (acc), precision (p), recall(r), F1-score (F1) and the ratio of key sen-tences found in U (pos.ratio) of different semi-supervised learning methods for strict (Table 2)and tolerant (Table 3) noise filtering scenarios.We consider classification to have gone wrong ifthe ratio of positive sentences in U significantlydeviates from the acceptable range [0.2, 0.4] (asdefined in Section 3.1). Additionally, results of su-pervised ML pipeline on data sets generated afternoise filtering are available in Appendix A.

Our experiments show that strict noise filter-ing leads to greatly improved classification accur-acy; however, it may be fallacious to judge thisapproach only by the scores it produces. GivenCIViC’s deviations from typical language in sci-entific articles, the different thematic foci of CIViCand HoC, and the negligible amount of realisticpositive sentences added in this scenario (Table1), we suspect classifiers may overfit to superfi-cial and incidental differences rather than learningto generalise to correctly identify key sentences inunseen abstracts. In order to avoid this, we discardstrict noise filtering.

On the other hand, tolerant filtering of HoCn

and HoCp still allows for reasonable classifica-tion accuracy considering the data’s heterogen-eity. We expect additional positive sentences toprovide improvements to generalisation that out-weigh the lower nominal performance scores andpossible errors propagated due to remaining noise.Although HoC’s notion of relevant sentences isnot identical to that implied by CIViC, our exper-iments show that removing only the least suitablesentences is enough to use HoC ′

p as meaningfultraining data.

Standard Self-Training yields performance res-ults very similar to supervised classification, ana-logous to what can be observed in strict mode, but

a larger ratio of positive predictions forU . The lin-ear classifiers SVM and Logistic Regression per-form much better than NB, the latter modelling aninaccurate probability distribution. In both strictand tolerant mode, methods with an emphasis onunsupervised clustering (EM, Label Propagation,and Label Spreading) underperform, with a strongbias towards the negative class. Label Propaga-tion with k-Nearest-Neighbours kernel performsparticularly poorly, failing to find any positive ex-amples in the unlabelled set. In contrast, NSTwith base classifiers leads to positive ratios in Uclose to our preliminary estimate, as well as ac-ceptable classification accuracy. SVM performsbetter than Logistic Regression and has balancedprecision and recall for both classes, appearing themore robust choice.

5 Conclusion

We have developed a pipeline for identifying themost informative key sentences in oncological ab-stracts, judging sentences’ clinical relevance im-plicitly by their similarity to clinical evidencesummaries in the CIViC database. To accountfor deviations from typical content between pro-fessional summaries and sentences appearing inabstracts, we use the abstracts corresponding tothese summaries as unlabelled data in an semi-supervised learning setting. An auxiliary silverstandard corpus is used for more realistic trainingand validation data. To mitigate introducing er-rors due to miscategorised examples in partitionsof the auxiliary data, we propose using PU Learn-ing techniques in a noise detection preprocessingstep.

We evaluate different heuristics for semi-supervised learning and measure their perform-ance with heterogenous data. While methods withan emphasis on unsupervised clustering performpoorly, (which we attribute to the data violatingsmoothness assumptions) Self-Training with lin-ear classifiers proved robust to unfavourably dis-tributed data, reaching performance scores similarto those of supervised classifiers trained withoutthe unlabelled data. By adapting Self-Trainingwith SVMs to iteratively expand only the negat-ive training set as in PU Learning, we were ableto restrict the amount of hidden positive examplesfound in unlabelled data while maintaining goodaccuracy scores. Our best model using this methodreaches 84% accuracy and 0.84 F1-score.

43

As a byproduct of the proposed pipeline, we ob-tain a silver standard corpus consisting of approx-imately 12,700 sentences from our unlabelled set,annotated with sentences’ estimated clinical relev-ance, which may be useful for future classificationtasks. Our final pipeline can be used to help clini-cians quickly assess which articles are relevant totheir work, e.g. by incorporating it into workflowsfor the retrieval of cancer-related literature. Assuch, it has been integrated in to Variant Inform-ation Search Tool1 (VIST), a query-based docu-ment retrieval system which ranks scientific ab-stracts according to the clinical relevance of theircontent given a (set of) variations and/or genes.

We encountered various difficulties resultingfrom using a gold standard with atypical andsolely positive examples and the heterogeneity ofdifferent training corpora. Although our problemof finding key sentences is a standard PU Learn-ing task, the methods described in the PU Learn-ing literature cannot be used in a verifiable way onreal-world data without negative validation data.Even for semi-supervised learning with positiveas well as negative labelled data, standard met-rics alone are not enough to judge a classifier’sadequacy, since the amount of noise in automat-ically gathered training data is never completelycertain and the way unlabelled data is handled by aclassifier is not represented in performance scores.By using heuristics for noise filtering and adapt-ing self-training to incorporate unlabelled data ina way suitable to our goal, we alleviate these diffi-culties.

ReferencesShashank Agarwal and Hong Yu. 2009. Automatically

classifying sentences in full-text biomedical articlesinto introduction, methods, results and discussion.Bioinformatics, 25(23):3174–3180.

Akiko Aizawa. 2003. An information-theoretic per-spective of tf–idf measures. Information Processing& Management, 39(1):45–65.

Simon Baker, Ilona Silins, Yufan Guo, Imran Ali, Jo-han Hogberg, Ulla Stenius, and Anna Korhonen.2016. Automatic semantic classification of sci-entific literature according to the hallmarks of can-cer. Bioinformatics, 32(3):432–440.

Steven Bird, Ewan Klein, and Edward Loper. 2009.Natural language processing with Python: analyz-ing text with the natural language toolkit. O’Reilly.

1https://triage.informatik.hu-berlin.de:8080/

Alexis Conneau, Holger Schwenk, Loıc Barrault, andYann Lecun. 2017. Very deep convolutional net-works for text classification. In Proceedings of the15th Conference of the European Chapter of the As-sociation for Computational Linguistics: Volume 1,Long Papers, volume 1, pages 1107–1116.

Franck Dernoncourt, Ji Young Lee, and Peter Szo-lovits. 2017. Neural networks for joint sentenceclassification in medical paper abstracts. In Pro-ceedings of the 15th Conference of the EuropeanChapter of the Association for Computational Lin-guistics: Volume 2, Short Papers, pages 694–700.Association for Computational Linguistics.

Charles Elkan and Keith Noto. 2008. Learning classi-fiers from only positive and unlabeled data. In Pro-ceedings of the 14th ACM SIGKDD internationalconference on Knowledge discovery and data min-ing, pages 213–220. ACM.

Alec Go, Richa Bhayani, and Lei Huang. 2009. Twit-ter sentiment classification using distant supervision.CS224N Project Report, Stanford, 1(12).

Malachi Griffith, Nicholas C Spies, Kilannin Krysiak,Joshua F McMichael, Adam C Coffman, Arpad MDanos, Benjamin J Ainscough, Cody A Ramirez,Damian T Rieke, Lynzey Kujan, et al. 2017. Civicis a community knowledgebase for expert crowd-sourcing the clinical interpretation of variants in can-cer. Nature genetics, 49(2):170.

Maryam Habibi, Leon Weber, Mariana Neves,David Luis Wiegandt, and Ulf Leser. 2017. Deeplearning with word embeddings improves biomed-ical named entity recognition. Bioinformatics,33(14):i37–i48.

A. Hassan and A. Mahmood. 2017. Deep learning forsentence classification. In 2017 IEEE Long IslandSystems, Applications and Technology Conference(LISAT), pages 1–5.

Thorsten Joachims. 1998. Text categorization withsupport vector machines: Learning with many rel-evant features. In Machine Learning: ECML-98,Chemnitz, Germany, volume 1398 of Lecture Notesin Computer Science, pages 137–142. Springer.

Anthony Khoo, Yuval Marom, and David Albrecht.2006. Experiments with sentence classification. InProceedings of the Australasian Language Techno-logy Workshop 2006, pages 18–25.

Su Kim, David Martınez, Lawrence Cavedon, and LarsYencken. 2011. Automatic classification of sen-tences to support evidence based medicine. BMCBioinformatics, 12(S-2):S5.

Yoon Kim. 2014. Convolutional neural networks forsentence classification. In Proceedings of the 2014Conference on Empirical Methods in Natural Lan-guage Processing (EMNLP), pages 1746–1751. As-sociation for Computational Linguistics.

44

Julian Kupiec, Jan O. Pedersen, and Francine Chen.1995. A trainable document summarizer. InSIGIR’95, Seattle, Washington, USA, pages 68–73.ACM Press.

Wee Sun Lee and Bing Liu. 2003. Learning withpositive and unlabeled examples using weightedlogistic regression. In Machine Learning, Pro-ceedings of the Twentieth International Conference(ICML 2003), August 21-24, 2003, Washington, DC,USA, pages 448–455. AAAI Press.

Huayi Li, Zhiyuan Chen, Bing Liu, Xiaokai Wei, andJidong Shao. 2014. Spotting fake reviews via col-lective positive-unlabeled learning. In Proceedingsof the 2014 IEEE International Conference on DataMining, ICDM ’14, pages 899–904, Washington,DC, USA. IEEE Computer Society.

Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, andPhilip S. Yu. 2003. Building text classifiers usingpositive and unlabeled examples. In Proceedingsof the 3rd IEEE International Conference on DataMining (ICDM 2003), Melbourne, Florida, USA,pages 179–188. IEEE Computer Society.

Zhiguang Liu, Xishuang Dong, Yi Guan, and Jin-feng Yang. 2013. Reserved self-training: Asemi-supervised sentiment classification methodfor chinese microblogs. In Sixth InternationalJoint Conference on Natural Language Processing,Nagoya, Japan, pages 455–462. Asian Federation ofNatural Language Processing / ACL.

Larry McKnight and Padmini Srinivasan. 2003. Cat-egorization of sentence types in medical abstracts.In American Medical Informatics Association An-nual Symposium, Washington, DC, USA. AMIA.

Fantine Mordelet and Jean-Philippe Vert. 2014. A bag-ging SVM to learn from positive and unlabeled ex-amples. Pattern Recognition Letters, 37:201–209.

National Library of Medicine. 1946-2018. Pubmed.https://www.ncbi.nlm.nih.gov/pubmed. Accessed: 2018-02-01.

Bhawna Nigam, Poorvi Ahirwal, Sonal Salve, andSwati Vamney. 2011. Document classification us-ing expectation maximization with semi supervisedlearning. CoRR, abs/1112.2028.

Marthinus Du Plessis, Gang Niu, and Masashi Sug-iyama. 2014. Analysis of learning from positive andunlabeled data. In Proceedings of the 27th Interna-tional Conference on Neural Information ProcessingSystems - Volume 1, NIPS’14, pages 703–711, Cam-bridge, MA, USA. MIT Press.

Marthinus Du Plessis, Gang Niu, and Masashi Sug-iyama. 2015. Convex formulation for learning frompositive and unlabeled data. In Proceedings of the32nd International Conference on Machine Learn-ing, volume 37 of Proceedings of Machine LearningResearch, pages 1386–1394, Lille, France. PMLR.

Anthony Rios and Ramakanth Kavuluru. 2015. Con-volutional neural networks for biomedical text clas-sification: Application in indexing biomedical art-icles. In Proceedings of the 6th ACM Conference onBioinformatics, Computational Biology and HealthInformatics, BCB ’15, pages 258–267, New York,NY, USA. ACM.

Patrick Ruch, Celia Boyer, Christine Chichester,Imad Tbahriti, Antoine Geissbuhler, Paul Fabry,Julien Gobeill, Violaine Pillet, Dietrich Rebholz-Schuhmann, Christian Lovis, and Anne-LiseVeuthey. 2007. Using argumentation to extract keysentences from biomedical abstracts. I. J. MedicalInformatics, 76(2-3):195–200.

Simone Teufel and Marc Moens. 2002. Summariz-ing scientific articles: Experiments with relevanceand rhetorical status. Computational Linguistics,28(4):409–445.

Philippe Thomas, Tamara Bobic, Ulf Leser, Mar-tin Hofmann-Apitius, and Roman Klinger. 2012.Weakly labeled corpora as silver standard for drug-drug and protein-protein interaction. In Proceed-ings of the Workshop on Building and EvaluatingResources for Biomedical Text Mining (BioTxtM)on Language Resources and Evaluation Conference(LREC).

Soroush Vosoughi, Helen Zhou, and Deb Roy. 2015.Enhanced Twitter Sentiment Classification UsingContextual Information. In Proceedings of the6th Workshop on Computational Approaches toSubjectivity, Sentiment and Social Media Analysis,pages 16–24, Stroudsburg, PA, USA. Associationfor Computational Linguistics.

Byron C. Wallace, Joel Kuiper, Aakash Sharma,Mingxi (Brian) Zhu, and Iain James Marshall. 2016.Extracting PICO sentences from clinical trial reportsusing supervised distant supervision. Journal ofMachine Learning Research, 17:132:1–132:25.

Bin Wang, Bruce Spencer, Charles X. Ling, and HarryZhang. 2008. Semi-supervised self-training for sen-tence subjectivity classification. In Advances in Ar-tificial Intelligence , 21st Conference of the Cana-dian Society for Computational Studies of Intelli-gence Windsor, Canada, pages 344–355. Springer.

Hong Yu and Vasileios Hatzivassiloglou. 2003. To-wards answering opinion questions: Separating factsfrom opinions and identifying the polarity of opin-ion sentences. In Proceedings of the 2003 Confer-ence on Empirical Methods in Natural LanguageProcessing, EMNLP ’03, pages 129–136, Strouds-burg, PA, USA. Association for Computational Lin-guistics.

Ye Zhang, Stephen Roller, and Byron C. Wallace. 2016.MGNC-CNN: A simple approach to exploiting mul-tiple word embeddings for sentence classification.In NAACL HLT 2016, San Diego California, USA,pages 1522–1527. The Association for Computa-tional Linguistics.

45

Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Fran-cis C. M. Lau. 2015. A c-lstm neural network fortext classification. CoRR, abs/1511.08630.

Dengyong Zhou, Olivier Bousquet, Thomas NavinLal, Jason Weston, and Bernhard Scholkopf. 2003.Learning with local and global consistency. In Ad-vances in Neural Information Processing Systems16 [Neural Information Processing Systems, NIPS2003, December 8-13, 2003, Vancouver and Whist-ler, British Columbia, Canada], pages 321–328.MIT Press.

Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu,Hongyun Bao, and Bo Xu. 2016. Text classificationimproved by integrating bidirectional lstm with two-dimensional max pooling. pages 3485–3495.

Xiaojin Zhu and Zoubin Ghahramani. 2002. Learningfrom labeled and unlabeled data with label propaga-tion.

A Semi-supervised ML on filtereddatasets

Table 4 shows results of supervised baseline clas-sifiers trained on P and N after strict filtering.Performance is very good for all classifiers tested,which is not surprising as CIViC and HoCn areeasy to separate even without filtering. The ra-tio of the unlabelled set U classified as positive,however, is outside of the acceptable range [0.2,0.4] for selecting key sentences, probably due tothe more similar contents of CIViC and the corres-ponding abstracts compared to HoC.

Table 5 shows the results of supervised classifi-ers trained on only P andN after tolerant filtering.Accuracies and F1-scores are about 10 percentpoints lower compared to results in the strict fil-tering scenario, which can be explained by HoCp

and HoCn being comparably difficult to separate.However, performance is better compared to dis-tinguishing CIV iC ∪ HoCp vs. HoCn withoutany noise filtering.

B PU Learning and Self-Training: usedcorpora

46

Ptest : Ntest : U :Method parameters acc p r F1 p r F1 pos. ratioSVM C = 3.0 0.96 0.97 0.94 0.95 0.96 0.98 0.97 0.63LR C = 6.0 0.96 0.96 0.94 0.95 0.96 0.97 0.97 0.64NB α = 0.1 0.94 0.95 0.91 0.93 0.94 0.97 0.95 0.61

Table 4: Supervised classifiers trained on P and N after strict noise filtering. Results averaged over 10runs with randomised 20% reserved test sets; min-df threshold = 0.002, 25% of most relevant featuresselected with χ2.

Ptest : Ntest : U :Method parameters acc p r F1 p r F1 pos. ratioSVM C = 3.0 0.86 0.88 0.84 0.86 0.84 0.88 0.86 0.63LR C = 6.0 0.86 0.88 0.85 0.86 0.85 0.87 0.86 0.63NB α = 0.1 0.79 0.89 0.67 0.77 0.72 0.91 0.81 0.70

Table 5: Supervised classifiers trained on P and N after tolerant noise filtering. Results averaged over10 runs with randomised 20% validation sets; min-df threshold = 0.002, 25% of most relevant featuresselected with χ2.

Figure 1: PU Learning for noise reduction - used corpora

Figure 2: Semi-supervised training with Self-Training - used corpora