New Strategies for Finding failures and its domains

New strategies for finding failures and their domainsMian Asbaat Ahmad24-01-2013

Development of new automated strategies with the following objectives:

To find the maximum number of faults and their domainsTo use a minimum number of test callsTo take the least possible timeTo require minimal resourcesTo give comprehensive resultsTo create an extensive test suite

ObjectivesDirt Spot Sweeping Random (DSSR) Strategy

Automated Discovery of Failure Domain (ADFD) Strategy

ADFD with Daikon (ADFD-D) Strategy

Thesis contributionsWhy random testing?Because:Testing of all values is typically impossibleTime and resources are limitedEasy to implementFree from human biasQuick and effective in finding faults

What is Random Testing?Black-box testing techniqueDynamic testing processRandom SelectionTest executionOutput evaluationResult generation

Automated Random TestingAutomation of the process for:Analysis of system under testSelection of test valuesExecution of testsEvaluation of resultsDelivery of results Generation of test suite

Automated Random Testing Tools

York Extensible Testing InfrastructureRandom testing toolDeveloped in JavaSupports multiple languagesHigh speed (up to 106 calls per minute)Both CLI and GUI interfacesInteractive testingExperimentally verifiedRunnable in cloud8Yeti is a random testing tool, strategy used is random testingCoded in Javaby testing java.lang and iTextDecoupling ?Language agnostic ?High speed of processingYeti can be operated from both CLI and GUITesting can be changed at run time

8YETI Front-end

Why Improve Random Testing?To increase code coverageIf(x == 3333) then where x is 32 bit variableTo increase efficiencyTo decrease over-headTo generate friendly outputTo introduce automationEnhanced versions of Random TestingAdaptive Random Testing (ART)Quasi Random Testing (QRT)Mirror Adaptive Random Testing (MART)Restricted Random Testing (RRT)Feedback-Directed Random Testing (F-DRT)Random Plus Testing (R+)

Types of Fault DomainPoint Fault DomainFaults lies scattered across input domain

Block Fault DomainFaults lies in a block across input domain

Strip Fault Domain Faults lies in a strip across the input domain

Main idea of DSSRWe start by testing with YETI (using random+)

When a new failure is found we add the neighboring values to the list of interesting values

These have higher priority, effectively sweeping around the faults found

Dirt Spot Sweeping Random StrategyBased on three strategiesRandomRandom PlusSpot Sweeping

Block sweepingStrip sweepingExample to illustrate DSSR strategy/** * Calculate square of given number and verify results. * The code contain 3 faults. * @author (Mian and Manuel) */public class Math1{public void calc (int num1) {// Square num1 and store result.int result1 = num1 * num1;int result2 = result1 / num1; // 1 assert Math.sqrt(result1) == num1; // 2 assert result1 >= num1; // 3} }

Performance of DSSR compared to R and R+ St.60 classes from 32 projects were tested by R, R+ and DSSR strategiesIn 43 classes all three strategies found the same number of faultsIn remaining 17 classes DSSR strategy found the highest number of unique failures (1117) followed by R+ (1106) and R (1090) respectivelyOn overall basis DSSR strategy performed better than R and R+ strategies

Test Results of 17/60 classesDSSR strategy better up to 33% than R, up to 17% than R+Limitations of DSSR strategyLack of improvement in point fault domainExcess time involved to find first fault5% over-head than R and 2% than R+ in executionAutomated Discovery of Failure Domain (ADFD)A new automated strategy implemented in YETI Provides GUI interfaceFacilitates control on lower and upper boundFinds faults and their domainsPresents results in graphical formMain idea of ADFDWe start by testing with YETI (using random+)

When a new failure is found ADFD generates programs that will test each direction independently

We then compile and execute these programsThey generate failure domains for each direction

We show the results in the interface

Using ADFDLaunch the ADFD applicationSelect language, ADFD strategy and timeSelect class for TestingSpecify lower and upper boundsClick run button to start testingProgram is dynamically generatedClick count button to find number of files generatedClick button to compile the programClick button to execute the programData is generatedClick button to plot graphFront-end of ADFD

Illustration of ADFD strategy by examples of Point, Block and Strip Fault DomainsPoint Fault Domain one Argument/** * Point Fault Domain example for one argument * @author (Mian and Manuel) */public class PointDomainOneArgument{

public static void pointErrors (int x){ if (x == -66 ) abort(); if (x == -2 ) abort(); if (x == 51 ) abort(); if (x == 23 ) abort();}}

Point Fault Domain two Argument

/** * Point Fault Domain example for two arguments * @author (Mian and Manuel) */public class PointDomainOneArgument{

public static void pointErrors (int x, int y){ int z = x/y;}

}

Block Fault Domain one argument/** * Block Fault Domain example for one argument * @author (Mian and Manuel) */public class BlockDomainOneArgument{

public static void blockErrors (int x){if((x > -2) && (x < 2))abort();if((x > -30) && (x < -25))abort();if((x > 50) && (x < 55))abort();

}}

Block Fault Domain two argument/** * Block Fault Domain example for two arguments * @author (Mian and Manuel) */public class BlockDomainTwoArgument{

public static void pointErrors (int x, int y){

if(((x > 0)&&(x < 20)) || ((y > 0) && (y < 20))){abort();} }

}

Strip Fault Domain one argument/** * Strip Fault Domain example for one argument * @author (Mian and Manuel) */

public class StripDomainOneArgument{

public static void stripErrors (int x){if((x > -5) && (x < 35))abort(); }}

Strip Fault Domain two argument/** * Strip Fault Domain example for two arguments * @author (Mian and Manuel) */public class StripDomainTwoArgument{

public static void pointErrors (int x, int y){

if(((x > 0)&&(x < 50)) || ((y > 0) && (y < 500))){abort();} }

}

Development of ADFD with Daikon (ADFD-D) strategySalient FeaturesTo execute SUT by Daikon to generate invariantsTo add values from invariants to the list of interesting valuesTo Execute DSSR strategyDaikon Invariant DetectorAutomated toolDeveloped by MIT Program Analysis GroupDynamically reports likely program invariantsDetect C, C++, Eiffel, IOA, Java and Perl programs.freely available: http://groups.csail.mit.edu/pag/daikon/download/Daikon with ADFD strategyWe are utilizing Daikon invariants:As oracle for comparing test resultsTo find border (interesting) valuesTo remove the dependence of DSSR on R+ strategyTo increase test performance by restricting the upper and lower bound.

Dirt Spot Sweeping Random (DSSR) StrategyUp to 33% better compared to R strategyUp to 17% better compared to R+ strategy Automated Discovery of Failure Domain (ADFD) StrategyFinds and Plots fault domainExhaustive search of projections on each axis

ADFD with Daikon (ADFD-D) StrategyCombination of ADFD strategy with DaikonResearch still in Progress

ConclusionsThank You