Geometry matters: Exploring language examples at the decision boundary

Debajyoti Datta (dd3ar@virginia.edu)1 Shashwat Kumar1 Laura Barnes1 Tom Fletcher2

1Systems Engineering, University of Virginia, Charlottesville, Virginia, USA2Computer Science Dept., University of Virginia, Charlottesville, Virginia, USA

Abstract

A growing body of evidence has suggested thatmetrics like accuracy overestimate the classifier’sgeneralization ability. Several state of the art NLPclassifiers like BERT and LSTM rely on super-ficial cue words(e.g., if a movie review has theword “romantic”, the review tends to be positive),or unnecessary words (e.g., learning a proper nounto classify a movie as positive or negative). Oneapproach to address this is how teachers discovergaps in a student’s understanding: by finding prob-lems where small variations confuse the student,it’s possible to perturb and study NLP classifiersin a similar way. While several perturbation strate-gies like contrast sets or random word substitutionshave been proposed, they are typically based onheuristics and/or require expensive human involve-ment. In this work, using tools from informationgeometry, we propose a principled way to quantifythe fragility of an example for an NLP classifier.By discovering such difficult examples for severalstate of the art NLP models like BERT, LSTM,and CNN, we demonstrate their susceptibility tomeaningless perturbations like noun/synonym sub-stitution, causing their accuracy to drop down to20 percent in some cases. Our approach is sim-ple, architecture agnostic and can be used to studythe fragilities of text classification models. All thecode used will be made publicly available, includ-ing a tool to explore the difficult examples for otherdatasets.

1 INTRODUCTION

NLP classifiers have achieved state of the art performancein several tasks like sentiment analysis Maas et al. [2011],semantic entailment Bowman et al. [2015] and question

answering Rajpurkar et al. [2016]. Despite their successes,several studies have pointed out issues in features learntby such classifiers. Gururangan et al. [2018] discoveredthat several high performing NLP models were using trivialfeatures like vagueness and negation to perform classifica-tion. Geirhos et al. [2020] performed several experimentsto discover that “romantic” movies tend to be classified aspositive movie reviews due to the presence of unnecessarywords like proper nouns, a phenomena they called shortcutlearning. Even models like BERT (Devlin et al. [2018]) re-lied on superficial cue words like “not” to infer the line ofargumentation. Strategies like this Niven and Kao [2020] en-able models to perform prediction without inherently usingthe semantic meaning of the sentence. Simple attributes likeword lengths are also exploited by models for predictionPoliak et al. [2018].

In order to address these issues, several perturbation basedapproaches have been proposed. Contrast Sets Gardner et al.[2020] and Counterfactual examples Kaushik et al. [2019]get human annotators to perform minimal token substitu-tions to change the classifier prediction. While useful inaddressing biases, manual curation of datasets are oftentime consuming and require extensive efforts. Using un-supervised training to mitigate issues related to shortcutlearning also has not been successful in practice Niven andKao [2020]. The problem has also been approached from anadversarial perspective, with several works focused on in-troducing at the character level Gao et al. [2018], Ebrahimiet al. [2018] and/or through word substitutions Li et al.[2016], Liang et al. [2018]. Due to the lack of coherenceof the generated sentences in the adversarial examples fu-ture work in this area aimed at generated grammaticallycorrect sentences such that they are indistinguishable fromoriginal dataset. In Alzantot et al. [2018], Garg and Ramakr-ishnan [2020], the authors use black-box based approach togenerate semantically equivalent adversarial examples.

While interesting, most traditional formulations treat thisinput embedding space as flat, thus reasoning that the gradi-ent of the likelihood in the input space gives us the direction

arX

iv:2

010.

0721

2v3

[cs

.CL

] 2

8 O

ct 2

021

g : X −→ Z

A B

Figure 1: A: A neural network can be considered as a mapping g between the sentence manifold X on the left (representedas a Euclidean space for simplification) and the statistical manifold of probability distribution over outputs Z. The fishermetric defines a particular Riemannian metric over this manifold. Let us consider an ε ball around two setences x1 (bluecircle) and x2 (red circle). As we can see from the distortion of the blue circle, for the x1, any location perturbation canonly result in a small change over the models output probability distribution. The eigenvalue of the fisher matrix quantifiesthis local distortion. These examples are not useful from a fragility based augmentation point of view. Perturbing the redexamples however can result in drastic changes in the models output probability distribution. By measuring this distortionusing the fisher matrix locally, we can study which examples affect the model output more. B: We train a neural network tolearn a decision boundary (black) to separate two gaussians and color each data point by the local distortion. As we cansee from the colors, perturbing a point close to the decision boundary results in a larger change over the model’s outputprobability distribution than a point away from the boundary (blue circle).

which causes the most significant change in terms of likeli-hood. If we were however, to consider the discrete likelihoodof the class probabilities as the model output and the inputas a pullback of this output, the space of output probabilitydistributions is certainly non-linear, and the euclidean dis-tance metric no longer suffices. We show this schematicallyin figure 1A, where the transformation g maps elements ofthe sentence manifold X to a non-linear statistical manifoldZ. A natural distance metric to consider on this manifoldis the Fisher Information Metric which along with beinga Hessian of the KL divergence, it also a useful distancemeasure between probability distributions. Furthermore, theFisher Information Metric is invariant to transformationslike changing the model architecture, provided the likeli-hood remains the same. We thus use the eigenvalues of theFisher matrix to discover high fragility regions in the statisti-cal manifold. In regions with high λmax small perturbationscan cause large changes in the output probability distribu-tion (Fig 1A, red circle). Linguistically, this corresponds tothe classifier being susceptible to meaningless perturbationslike noun, synonym substitutions. Similarly, in regions withlow λmax no local perturbation can affect the classifier, re-sulting in the classifier to being resilient upto 20% percentof word substituions.

Our main contribution can thus be summarized as the fol-lowing. 1. We propose a second order statistic, the log ofthe largest eigen value of Fisher matrix in order to capturelinguistic fragilities. 2. We extensively establish the empir-ical relationship between λmax and success probability of

random word substitutions.

To the best of our knowledge, this is the first work analyz-ing properties of the fisher metric to understand classifierfragility in NLP. The rest of the paper is organized as fol-lows: In Section 2, we summarize related work. In Section3, we discuss our approach of computing the FIM and thegradient-based perturbation strategy. In Section 4, we dis-cuss the results of the eigenvalues of FIM in synthetic dataand sentiment analysis datasets with BERT and CNN. Wealso do extensive quantitative evaluations using 4 other textdatasets. Finally, in Section 5, we discuss the implications ofstudying the eigenvalues of FIM for evaluating NLP models.

2 RELATED WORK

In NLP, machine learning models for classification rely onspurious statistical patterns of the text and use shortcut forlearning to classify. These can range from annotation arti-facts, as was shown by Goyal et al. [2017], Kaushik andLipton [2018], Gururangan et al. [2018], spelling mistakesas in McCoy et al. [2019], or new test conditions that requireworld knowledge Glockner et al. [2018]. Simple decisionrules that the model relies on are hard to quantify. Trivial pat-terns like relying on the answer “2” for answering questionsof the format “how many” for the visual question answeringdataset Antol et al. [2015], would correctly answer 39%of the questions. Jia and Liang [2017] showed that adver-sarially inserted sentences that did not change the correct

Table 1: CNN, Counterfactual Examples: In difficult examples (larger eigenvalue), individual synonym/antonym substitutionsare effective in changing the classifier output. In easy examples (smaller eigenvalue) multiple antonym substitutionssimultaneously have no effect on the classifier output.

Perturbed sentiment Word substitutionsPositive → Negative diffi-cult example λmax =4.38

This move was on TV last night. I guess as a time filler, because it was incredible! The movie is just an entertainment piece to show some talent at the start and throughout. (Not bad talent at all). Butthe story is too brilliant for words. The "wolf", if that is what you can call it, is hardly shown fully save his teeth. When it is fully in view, you can clearly see they had some interns working on the CGI,because the wolf runs like he’s running in a treadmill, and the CGI fur looks like it’s been waxed, all shiny :)<br /><br />The movie is full of gore and blood, and you can hardly spot who is goingto get killed/slashed/eaten next. Even if you like these kind of splatter movies you will be surprised, they did do a good job at it.<br /><br />Don’t even get me started on the actors... Very amazinglines and the girls hardly scream at anything. But then again, if someone asked me to do good acting just to give me a few bucks, then hey, where do I sign up?<br /><br />Overall exciting → boring,extraordinary, uninteresting, exceptional and frightening horror.

Negative → Negative easyexample λmax =0.013

I couldn’t stand this movie. It is a definite waste of a movie. It fills you with boredom. This movie is not worth the rental or worth buying. It should be in everyones trash. Worst → Excellent movie Ihave seen in a long time. It will make you mad because everyone is so mean to Carl Brashear, but in the end it gets only worse. It is a story of cheesy romance, bad → good drama, action, and plentyof unfunny → funny lines to keep you rolling your eyes. I hated a lot of the quotes. I use them all the time in mocking the film. They did not help keep me on task of what I want to do. It shows thatanyone can achieve their dreams, all they have to do is whine about it until they get their way. It is a long movie, but every time I watch it, I dradr that it is as long as it is. I get so bored in it, that it goesso slow. I hated this movie. I never want to watch it again.

Table 2: CNN, IMDb dataset: Unlike the difficult examples (larger eigenvalue), word substitutions are ineffective in changingthe classifier output for the easier examples (smaller eigenvalue). In difficult examples synonym or change of name, changesclassifier label. In easy examples, despite multiple simultaneous antonym substitutions, the classifier sentiment does notchange.

Perturbed sentiment Word substitutionsPositive → Nega-tive difficult example(λmax =5.25)

Going into this movie, I had heard good things about it. Coming out of it, I wasn’t really amazed nor disappointed. Simon Pegg plays a rather childish character much like his other movies. There werea couple of laughs here and there– nothing too funny. Probably my favorite → preferred parts of the movie is when he dances in the club scene. I totally gotta try that out next time I find myself ina club. A couple of stars here and there including: Megan Fox, Kirsten Dunst, that chick from X-Files, and Jeff Bridges. I found it quite amusing to see a cameo appearance of Thandie Newton in ascene. She of course being in a previous movie with Simon Pegg, Run Fatboy Run. I see it as a toss up, you’ll either enjoy it to an extent or find it a little dull. I might add, Kirsten Dunst → NicoleKidman, Emma Stone, Megan Fox, Tom Cruise, Johnny Depp, Robert Downey Jr. is adorable in this movie. :3

Negative → Negative easyexample (λmax =0.0008)

I missed this movie in the cinema but had some idea in the back of my head that it was worth a look, so when I saw it on the shelves in DVD I thought "time to watch it". Big mistake!<br /><br />Along list of stars cannot save this turkey, surely one of the worst → best movies ever. An incomprehensible → comprehensible plot is poorly → exceptionally delivered and poorly → brilliantlypresented. Perhaps it would have made more sense if I’d read Robbins’ novel but unless the film is completely different to the novel, and with Robbins assisting in the screenplay I doubt it, the novelwould have to be an excruciating → exciting read as well.<br /><br />I hope the actors were well paid as they looked embarrassed to be in this waste of celluloid and more lately DVD blanks, take forexample Pat Morita. Even Thurman has the grace to look uncomfortable at times.<br /><br />Save yourself around 98 minutes of your life for something more worthwhile, like trimming your toenailsor sorting out your sock drawer. Even when you see it in the "under $5" throw-away bin at your local store, resist the urge!

answer, would cause state of the art models to regress inperformance in the SQuAD Rajpurkar et al. [2016] ques-tion answering dataset. Glockner et al. [2018] showed thattemplate-based modifications by swapping just one wordfrom the training set to create a test set highlighted mod-els’ failure to capture many simple inferences. Dixon et al.[2018] evaluated text classifiers using a synthetic test set tounderstand unintended biases and statistical patterns. Usinga standard set of demographic identity terms, the authorsreduce the unintended bias without hurting the model perfor-mance. Shen et al. showed that word substitution strategiesinclude stylistic variations that change the sentiment analy-sis algorithms for similar word pairs. Evaluations of thesemodels through perturbations of the input sentence are cru-cial to evaluating the robustness of models.

Another issue of language recently has been that staticbenchmarks like GLUE by Wang et al. [2018] tend to sat-urate quickly because of the availability of ever-increasingcompute and harder benchmarks are needed like SuperGlueby Wang et al. [2019]. A more sustainable approach to thisis the development of moving benchmarks, and one notableinitiative in this area is the Adversarial NLI by Nie et al.[2019], but most of the research community hardly validatetheir approach against this sort of moving benchmark. In theAdversarial NLI dataset, the authors propose an iterative,adversarial human-and-model-in-the-loop solution for Natu-ral Language Understanding dataset collection, where thegoal post continuously shifts about useful benchmarks andmakes models robust by training the model iteratively on

difficult examples. Approaches like never-ending learningby Mitchell et al. [2018] where models improve, and testsets get difficult over time is critical. A moving benchmarkis necessary since we know that improving performance ona constant test set may not generalize to newly collecteddatasets under the same condition Recht et al. [2019], Beeryet al. [2018]. Therefore, it is essential to find difficult ex-amples in a more disciplined way. Approaches based ongeometry have recently started gaining traction in computervision literature. Zhao et al. [2019] et al used a similar ap-proach for understanding adversarial examples in images, aformulation we extend to language in our work

3 METHODS

Consider the manifold of all possible sentence embeddingsX. We show this schematically in Figure 1A, left. Although,We represent this as a euclidean space for simplicity, inthe general setting this manifold could be non linear. LetxεX be a sentence on this manifold, and y be the labelvector corresponding to this sentence. The neural networkp(y|x) maps each sentence to a probability distributionover y. The set of all such probability distributions formsa statistical manifold Z (Figure 1A, right), Depending onthe properties of the neural network, an η change in x (redcircle/blue circle) can result in a large change in p(y|x). Weseek to quantify this change by measuring the KL divergencebetween the two original and η perturbed sentence.

Table 3: BERT: Difficult Examples change sentiment with a single word substituted. Easy examples, however, retain positivesentiment despite multiple substitutions of positive words with negative words.

Perturbed sentiment Word substitutionsPositive → Nega-tive difficult example(λmax =0.78)

OK, I kinda like the idea of this movie. I’m in the age demographic, and I kinda identify with some of the stories. Even the sometimes tacky and meaningful dialogue seems realistic, and in a differentmovie would have been forgivable.<br /><br />I’m trying as hard as possible not to trash this movie like the others did, but it’s easy when the filmmakers were trying very hard.<br /><br />The editingin this movie is terrific! Possibly the best → worst editing I’ve ever seen in a movie! There are things that you don’t have to go to film school to learn, leaning good editing is not one of them, butidentifying a bad one is.<br /><br />Also, the shot... Oh my God the shots, just fantastic! I can’t even go into the details, but we sometimes just see random things popping up, and that, in conjunctionwith the editing will give you the most exhilirating film viewing experience.<br /><br />This movie being made on low or no budget with 4 cast and crew is an excuse also. I’ve seen short films onyoutube with a lot less artistic integrity! ...

Positive → Positive easyexample (λmax =0.55)

This is the best and most original show seen in years. The more I watch it the more I fall in love with → hate it. The cast is excellent → terrible , the writing is great → bad. I personally loved→ hated every character. However, there is a character for everyone as there is a good mix of personalities and backgrounds just like in real life. I believe ABC has done a great service to the writers,actors and to the potential audience of this show, to cancel so quickly and not advertise it enough nor give it a real chance to gain a following. There are so few shows I watch anymore as most TV isawful . This show in my opinion was right down there with my favorites Greys Anatomy and Brothers and Sisters. In fact I think the same audience for Brothers and Sisters would hate this show if theyeven knew about it.

KL(p(y|x)‖p(y|x + η))

= −Ep(y|x)logp(y|x) + Ep(y|x)logp(y|x + η)

We now perform a Taylor expansion of the first term on theright hand side

= −Ep(y|x)(logp(y|x) + η∇xlogp(y|x)

+ηT∇2xlogp(y|x)η + ...) + Ep(y|x)logp(y|x)

∼ −Ep(y|x)ηT∇2

xlogp(y|x)ηSince the expectation of score is zero and the first and lastterms cancel out, we are left with.

= ηTGη

By studying the eigenvalues of the fisher matrix, we canquantify the local distortion. As we see in Figure 1, the redsentence has a larger λmax, resulting in a large change inthe prob distribution. The blue sentence has a much smallerλmax, meaning that perturbations around this sentence areless likely to affect p(y|x) and thus the model accuracy.

After getting the eigenvalues of the FIM, we can use thelargest eigenvalue λmax to quantify how fragile an exampleis to linguistic perturbation. These examples, thus, are alsomore confusing and more difficult for the model to classify.We propose the following procedure to calculate λmax

Algorithm 1 Algorithm for estimating fragility of an exam-ple

Input: x: Sentence representation, f: Neural NetworkOutput: λmax

Calculate probability vector :1: p = f (x)

Calculate Jacobian of log probability w.r.t x2: J =∇xlogp

Duplicate probability vector along rows to match J’sshape

3: pc = duplicate(p, J.dim[0])Compute the FIM

4: G = pc J JT

Perform eigendecomposition to get the eigenvalues5: λs,vs = eigendecomposition(G)6: return max(λs)

3.1 MODEL DESCRIPTIONS

We use multiple model architectures to understand the im-plications of the FIM.

• Convolutional Neural Network (CNN): We use thesame paradigm Kim [2014] for text classification usingGloVe Pennington et al. [2014] embeddings.

• Long Short Term Memory Network (LSTM):LSTM’s have been used extensively in Natural Lan-guage Processing for variety of tasks like text classifi-cation and language modeling Schmidhuber [2015]

• Bag of Tricks for Efficient Text Classifica-tion(Fasttext) We used the Fasttext based modelJoulin et al. [2017] for classifying text and examiningthe quantitative aspects of Fisher Information Metricsince the model is incredibly computational efficiencyand strong performance across multiple datasets.

• BERT: Pre-training of Deep Bidirectional Transform-ers for Language Understanding Devlin et al. [2018]Large scale pretrained language models have becomeextremely popular for data efficiency when trained ondownstream tasks. However it has been shown recently,that on large datasets ? performance increase from fine-tuning is often within 1% of BERT. Thus we limitedour analysis of using BERT for qualitative analysis forverification of it’s relationship with FIM since accuracydifference is minimal and it is computation intensiveto calculate the eigenvalue of the FIM of BERT basedfine tuned models. We instead used BERT embeddingsin downstream tasks for qualitative evaluation of FIM.For the Fasttext, CNN and LSTM models we used earlystopping with a patience of 5.

3.2 DATASETS

• IMDB: Large Movie Review Dataset Maas et al.[2011]: IMDb dataset is the most common sentimentanalysis dataset (Positive/Negative) for text classifi-cation collected from movie reviews on the IMDbplatform. We do extensive quantitative and qualita-tive evaluation on the IMDb dataset across multiplearchitectures.

Figure 2: On YelpReviewPolarity we observe a correlation coefficient of -0.4 between minimum perturbation strength andthe difficulty score. Similarly, we observe a correlation of 0.35 between the difficulty score and empirical success of randomword substitutions. This suggests that fim eigenvalue captures perturbation sensitivity in both embedding space and wordsubstitutions.

• AG_NEWS Zhang et al. [2015]: News classifica-tion dataset into 4 topics: World, Sports, Business,Sci/Tech.

• Sogou News Zhang et al. [2015]: A text classificationdataset based on News Articles from SogouCA andSogouCS news corpora to 5 categories: sports, finance,entertainment, automobile, technology.

• Yelp Review Polarity: Large Yelp Review DatasetZhang et al. [2015]: A large sentiment classificationdataset with labels (Positive/Negative), extract fromthe Yelp Dataset Challenge 2015.

4 DISCUSSION AND RESULTS

We now explore both quantitatively and qualitatively howFisher Information metric relates to metrics like accuracyand investigate it’s use as finding examples that are suscep-tible to perturbations.

4.1 FIM CAPTURES RESILIENCE TOLINGUSTIC PERTURBATIONS

In this section we explore relationship of FIM with linguis-tic perturbations (synonym/antonym substitutions, noun re-placements), token level substitutions (random word choice,nearest neighbors in GloVE embeddings) and embeddingperturbations.

4.1.1 λmax is correlated with fragility to wordsubstitutions

In order to investigate the relationship between λmax andlinguistic fragility, we perform the following experiment:We randomly sample 1000 examples from Yelp ReviewPo-larity dataset. For each sampled example, we try a batchof word substitutions based on nearest neighbours in gloveembedding space. In each substitution attempt, we try toflip between 10 percentage of words at a time, and calculatethe percentage of successful flips which change the classi-fier prediction p f lip. We also calculate the λmax for theseexamples using the procedure outlined in Algorithm 1.

As we see from Figure 2 and Table 1, On YelpReviewPo-larity, we observe r values of 0.3, 0.38 and 0.38 for CNN,FastText and LSTM respectively. Despite FIM being definedin a continuous space, we observe a strong linear relation-ship with success probability of token substitutions, whichare discrete. This could be indicative of the fact that flip-ping a small of tokens in a large sentence is akin to an εperturbation in embedding space, which gets captured byλmax More examples of token substitution and relationshipof FIM with accuracy are in the Appendix.

Figure 3: Top row: Low FIM examples (easy examples) suffer from minimal accuracy loss when token substitutions areperformed whereas accuracy on high FIM examples are siginificantly low. Bottom row: Easy examples require minimalperturbation to change classifier prediction. The x-axis represents the norm of perturbation. This suggests FIM eigenvaluecaptures perturbation sensitivity in both embedding space and token substitutions.

4.1.2 λmax captures strength of the minimal sufficientperturbation

We were also interested in investigating the relationship be-tween λmax and norm of the smallest vector (oriented alongthe largest eigenvector emax) which can cause the classifierto flip it’s prediction. We perform the following experiment:We randomly sample 500 examples YelpReview. For eachof these examples, we calculate λmax and emax and perturbwith a strength of η, where

x f lip = xorgin + ηemax

‖emax‖

By performing binary search on η, we can determine theminimum perturbation strength sufficient to flip the classifierprediction.

As evident from Figure 2 and Table 4, we obtain r values of-0.41, -0.36 and -0.39 for CNN, FastText and LSTM. Thesecorrelations suggest that low λmax examples are much moreresilient, requiring η around X in order to flip them. Highλmax examples on the other hand, get perturbed even byapplying small perturbations upto norm of 0.1 strength ofthe eigenvector.

4.1.3 Qualitative exploration of fragilities

We also qualitatively explore the nature of linguistic fragili-ties for high λmax examples. We sample high and low λmaxexamples from the two tails of the λmax distributions. Wethen perform several token level substitutions manually: syn-onym, antonym substitutions, noun, and article substitutions.The tokens selected for substitutions were not random, butinstead an Integrated Gradient was used to select the to-kens Sundararajan et al. [2017]. Integrated gradients assignimportance score to words by the network and thus pro-vide a more methodical approach to word substitutions thanrandom word substitutions for qualitative evaluations.

As seen in Table 2, either replacing favorite with preferredor “Kirsten Dunst” with any of the listed actors/actresses suf-fices to change the classifier’s prediction. Note that, “MeganFox’s” name appears in the same review in the previous sen-tence. Similar in Table 5, for counterfactual examples, it’ssufficient to replace “exciting” with either an antonym (“bor-ing” or “uninteresting”) or a synonym (“extraordinary” or“exceptional”). For easy examples however, despite trying toreplace four or more high attribution words simultaneouslywith antonyms, the predicted sentiment did not change. Sub-stitutions include “good” to “bad”, “unfunny” to “funny”,“factually correct” to “factually incorrect”. Even though thepassage included words like “boredom”, a word that is gen-erally associated with a negative movie review, the model

did not assign it a high attribution score. Consequently, wedid not try to substitute these words for testing robustnessor fragility of word substitutions.

For our models, difficult examples have a mix of positiveand negative words in a movie review. The models also strug-gled with examples of movies that selectively praise someattributes like acting (e.g., "Exceptional performance of theactors got me hooked to the movie from the beginning")while simultaneously use negative phrases (e.g., "howeverthe editing was horrible"). Difficult examples also have hightoken attributions associated with irrelevant words like "nu-clear," "get," and "an." Thus substituting one or two wordsin difficult examples change the predicted label of the classi-fier. Similarly, easier examples have clearly positive reviews(e.g., "Excellent direction, clever plot and gripping story").Combining integrated gradients with high FIM examplescan thus yield insights into the fragility of NLP models.

Several interesting observations emerge: CNN, Fasttext andLSTMs as a consequence of being trained on less amountof data is fragile to substitutions like Noun substitutions.BERT models on the other hand, because of their pretrainingare robust to these sort of substitutions. High FIM BERTexamples are more robust to meaningless changes, but singleword substitutions still cause classifier prediction to changecompared to low FIM examples.

4.2 HIGH λMAX EXAMPLES CAUSESUBSTANTIAL DROP IN ACCURACY

In order to study the effect of λmax on classification ac-curacy, we obtain examples with low FIM and high FIMλmax examples and plot classifier accuracy as a function ofnumber n. As we see from Figure 3, low λmax examplesretain accuracies upto 90 percent even after the test set hasbeen replaced with perturbed examples. In the high λmaxcases however, the accuracy drops belows 20 percent as weincrease the number of examples.

We also investigated the effect of perturbation strengthalong largest eigenvector emax on classifier accuracy. Aswe see from Figure 3, part B, low λmax examples are signifi-cantly more robust to perturbations that high λmax examples,where even minimal perturbations of the weight of 0.1 cancause the classifier prediction to change.

5 DISCUSSION

As evident from our experiments above, λmax is correlatedwith susceptibility to linguistic perturbations. Furthermore,λmax is directly correlated to the minimum perturbationneeded to flip the classifier prediction. Finally, we showa stark difference in the response of low and high λmaxexamples, with the high λmax examples, perturbed exam-ples being susceptible to meaningless perturbations (eg

Table 4: Statistics of correlation between success probabilityand λmax.

Dataset Architecture p-value r-valueYelpReviewPolarity CNN 1.32e-11 0.30YelpReviewPolarity FastText 2.66e-18 0.38YelpReviewPolarity LSTM 1.24e-18 0.38

Table 5: Statistics of correlation between success probabilityand λmax.

Dataset Architecture p-value r-valueYelpReviewPolarity CNN 5.76e-42 -0.411YelpReviewPolarity FastText 7.66e-32 -0.359YelpReviewPolarity LSTM 7.75e-39 -0.396

noun/synonym substitutions).

These experiments have several interesting ramifications:Firstly, it’s risky to over rely on accuracy while studying thegeneralizability of NLP classifiers. Most of the classifiers weused in our experiments attain really high accuracy on thetest set despite failing simple sanity checks (eg invarianceto synonym substitutions). It’s thus important to identify theexamples which are most susceptible to perturbations andtest their resilience. The fisher information provides us witha theoretically motivated framework to address this issue. Byextracting the high λmax examples and perturbing them byusing glove based synonym substitutions, we can constructa rolling test set for our NLP classifiers. Furthermore, bymining only the high λmax examples, FIM provides NLPpractitioners with an interactive way to perturb and explorethe flaws of their classifiers, something which would beextremely tedious to do on the entire dataset otherwise.While the focus of our work was not to generate adversarialexamples, an alternate framing of this problem is to considerhigh λmax sentences to be more susceptible to adversarialattacks whereas low λmax sentences are generally robust tosignificant adversarial perturbations.

6 CONCLUSION

In this paper, we introduced a method to discover the highlyfragile examples for an NLP classifier. our method discov-ered in several state of the art classifiers like BERT, CNNand LSTM. Furthermore, our experiments shed some lighton the links between geometry of NLP classifiers and theirlinguistic and embedding space perturbability. Our methodprovides NLP practitioners with both an automated an in-teractive human in the loop framework to better understandtheir models.

There are several interesting extensions. Firstly, it would beinteresting to decode the optimal perturbation vector emaxassociated with λmax as a sentence. We are developing sev-eral approaches based on finding token substitutions which

maximize this dot product in order to address this. It willalso be interesting to study the link between purely geo-metrical properties like distance to the decision boundaryand linguistic resilience. Finally, we are also interested inmeasuring the norm of token substitutions in embeddingspace to understand the relationship between the two.

References

Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Gener-ating natural language adversarial examples. In Proceed-ings of the 2018 Conference on Empirical Methods inNatural Language Processing, pages 2890–2896, 2018.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, MargaretMitchell, Dhruv Batra, C Lawrence Zitnick, and DeviParikh. Vqa: Visual question answering. In Proceedingsof the IEEE international conference on computer vision,pages 2425–2433, 2015.

Sara Beery, Grant Van Horn, and Pietro Perona. Recog-nition in Terra Incognita. pages 456–473, 2018. URLhttp://openaccess.thecvf.com/content_ECCV_2018/html/Beery_Recognition_in_Terra_ECCV_2018_paper.html.

Samuel R Bowman, Gabor Angeli, Christopher Potts, andChristopher D Manning. A large annotated corpus forlearning natural language inference. arXiv preprintarXiv:1508.05326, 2015.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and KristinaToutanova. Bert: Pre-training of deep bidirectional trans-formers for language understanding. arXiv preprintarXiv:1810.04805, 2018.

Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, andLucy Vasserman. Measuring and mitigating unintendedbias in text classification. In Proceedings of the 2018AAAI/ACM Conference on AI, Ethics, and Society, pages67–73, 2018.

Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou.Hotflip: White-box adversarial examples for text classi-fication. In Proceedings of the 56th Annual Meeting ofthe Association for Computational Linguistics (Volume 2:Short Papers), pages 31–36, 2018.

Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi.Black-box generation of adversarial text sequences toevade deep learning classifiers. In 2018 IEEE Securityand Privacy Workshops (SPW), pages 50–56. IEEE, 2018.

Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Be-rant, Ben Bogin, Sihao Chen, Pradeep Dasigi, DheeruDua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta,Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi,

Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mul-caire, Qiang Ning, Sameer Singh, Noah A. Smith, SanjaySubramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang,and Ben Zhou. Evaluating NLP Models via ContrastSets. arXiv:2004.02709 [cs], April 2020. URL http://arxiv.org/abs/2004.02709. arXiv: 2004.02709.

Siddhant Garg and Goutham Ramakrishnan. Bae: Bert-based adversarial examples for text classification. InProceedings of the 2020 Conference on Empirical Meth-ods in Natural Language Processing (EMNLP), pages6174–6181, 2020.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis,Richard Zemel, Wieland Brendel, Matthias Bethge,and Felix A. Wichmann. Shortcut Learning in DeepNeural Networks. arXiv:2004.07780 [cs, q-bio],April 2020. URL http://arxiv.org/abs/2004.07780. arXiv: 2004.07780.

Max Glockner, Vered Shwartz, and Yoav Goldberg. Break-ing nli systems with sentences that require simple lexicalinferences. arXiv preprint arXiv:1805.02266, 2018.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, DhruvBatra, and Devi Parikh. Making the v in vqa matter: Ele-vating the role of image understanding in visual questionanswering. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 6904–6913, 2017.

Suchin Gururangan, Swabha Swayamdipta, Omer Levy, RoySchwartz, Samuel R. Bowman, and Noah A. Smith. An-notation Artifacts in Natural Language Inference Data.arXiv:1803.02324 [cs], April 2018. URL http://arxiv.org/abs/1803.02324. arXiv: 1803.02324.

Robin Jia and Percy Liang. Adversarial examples for eval-uating reading comprehension systems. arXiv preprintarXiv:1707.07328, 2017.

Armand Joulin, Édouard Grave, Piotr Bojanowski, andTomáš Mikolov. Bag of tricks for efficient text classi-fication. In Proceedings of the 15th Conference of the Eu-ropean Chapter of the Association for Computational Lin-guistics: Volume 2, Short Papers, pages 427–431, 2017.

Divyansh Kaushik and Zachary C Lipton. How muchreading does reading comprehension require? a criticalinvestigation of popular benchmarks. arXiv preprintarXiv:1808.04926, 2018.

Divyansh Kaushik, Eduard Hovy, and Zachary C Lip-ton. Learning the difference that makes a differencewith counterfactually-augmented data. arXiv preprintarXiv:1909.12434, 2019.

Yoon Kim. Convolutional neural networks for sentenceclassification. arXiv preprint arXiv:1408.5882, 2014.

Jiwei Li, Will Monroe, and Dan Jurafsky. Understandingneural networks through representation erasure. arXivpreprint arXiv:1612.08220, 2016.

Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, XirongLi, and Wenchang Shi. Deep text classification can befooled. In Proceedings of the 27th International JointConference on Artificial Intelligence, pages 4208–4215,2018.

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang,Andrew Y Ng, and Christopher Potts. Learning wordvectors for sentiment analysis. In Proceedings of the49th annual meeting of the association for computationallinguistics: Human language technologies, pages 142–150, 2011.

R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Rightfor the wrong reasons: Diagnosing syntactic heuris-tics in natural language inference. arXiv preprintarXiv:1902.01007, 2019.

Tom Mitchell, William Cohen, Estevam Hruschka, ParthaTalukdar, Bishan Yang, Justin Betteridge, Andrew Carl-son, Bhanava Dalvi, Matt Gardner, Bryan Kisiel, et al.Never-ending learning. Communications of the ACM, 61(5):103–115, 2018.

Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal,Jason Weston, and Douwe Kiela. Adversarial nli: A newbenchmark for natural language understanding. arXivpreprint arXiv:1910.14599, 2019.

Timothy Niven and Hung Yu Kao. Probing neural networkcomprehension of natural language arguments. In 57thAnnual Meeting of the Association for ComputationalLinguistics, ACL 2019, pages 4658–4664. Association forComputational Linguistics (ACL), 2020.

Jeffrey Pennington, Richard Socher, and Christopher D Man-ning. Glove: Global vectors for word representation. InProceedings of the 2014 conference on empirical methodsin natural language processing (EMNLP), pages 1532–1543, 2014.

Adam Poliak, Jason Naradowsky, Aparajita Haldar, RachelRudinger, and Benjamin Van Durme. Hypothesis OnlyBaselines in Natural Language Inference. In Proceed-ings of the Seventh Joint Conference on Lexical andComputational Semantics, pages 180–191, New Orleans,Louisiana, June 2018. Association for ComputationalLinguistics. doi: 10.18653/v1/S18-2023. URL https://www.aclweb.org/anthology/S18-2023.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, andPercy Liang. Squad: 100,000+ questions for machinecomprehension of text. arXiv preprint arXiv:1606.05250,2016.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, andVaishaal Shankar. Do imagenet classifiers generalize toimagenet? arXiv preprint arXiv:1902.10811, 2019.

Jürgen Schmidhuber. Deep learning in neural networks: Anoverview. Neural networks, 61:85–117, 2015.

Judy Hanwen Shen, Lauren Fratamico, Iyad Rahwan, andAlexander M Rush. Darling or babygirl? investigatingstylistic bias in sentiment analysis.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Ax-iomatic attribution for deep networks. In InternationalConference on Machine Learning, pages 3319–3328.PMLR, 2017.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill,Omer Levy, and Samuel R Bowman. Glue: A multi-taskbenchmark and analysis platform for natural languageunderstanding. arXiv preprint arXiv:1804.07461, 2018.

Alex Wang, Yada Pruksachatkun, Nikita Nangia, AmanpreetSingh, Julian Michael, Felix Hill, Omer Levy, and SamuelBowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advancesin Neural Information Processing Systems, pages 3261–3275, 2019.

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. InNIPS, 2015.

Chenxiao Zhao, P Thomas Fletcher, Mixue Yu, Yaxin Peng,Guixu Zhang, and Chaomin Shen. The adversarial attackand detection under the fisher information metric. InProceedings of the AAAI Conference on Artificial Intelli-gence, volume 33, pages 5869–5876, 2019.

A APPENDIX

Figure 4: Left figure: The difficult examples on the FIM test set are challenging for the classifiers with accuracy between1-5%. For the easy examples, (low FIM λmax) the accuracy ranges between 60-100%. Right figure: We plot the classifieraccuracy with respect to the l2 norm of the perturbation in embedding space. For difficult examples, the classifier accuracydrops below 5% at small perturbation strengths (l2 norm < 0.3)

Figure 5: a) Distribution of difference in largest eigenvalue of FIM of the original and the perturbed sentence in contrast setand counterfactual examples for BERT and CNN. With a mean near 0, these perturbations are not difficult for the model.Adhoc perturbations are thus not useful for evaluating model robustness.