Lecture FOUR Metamorphic Reactions and Protoliths of Metamorphic Rocks
Applications of Metamorphic Testing Chris Murphy University of Pennsylvania [email protected]...
-
date post
15-Jan-2016 -
Category
Documents
-
view
217 -
download
0
Transcript of Applications of Metamorphic Testing Chris Murphy University of Pennsylvania [email protected]...
About Me
Lecturer, University of Pennsylvania
PhD-Computer Science, Columbia Univ, 2010 Advisor: Prof. Gail Kaiser Research: software testing, CS education
Seven years experience in software development industry
Problem:When testing a piece of software, how can we know that we’ve createdenough test cases?
Problem:When testing a piece of software, how can we create more test cases?
Solution:We use properties of the software tocreate new test cases from existing ones(particularly those that have not failed).
Result:This approach, known as metamorphictesting, is more effective at testing certaintypes of software than other approaches.
Today's Talk
What is metamorphic testing?
How is metamorphic testing used to find bugs in software?
How can metamorphic testing be applied to applications that do not have test oracles?
What are the open research questions related to metamorphic testing?
UPenn CIS 573 Software Engineering
Graduate-level software engineering course Just over 100 students Focuses on software maintenance issues:
– Testing
– Formal verification
– Debugging
– Fault Localization
– Refactoring
It's your first day of work as asoftware engineer at BloobleSoft.
Your boss gives you 6,000 lines of code and a specification and says “find the bugs”.
Where would you start?
SpecificationCode
Test CaseGeneration
Strategy
TestCasesTest
CasesTest
CasesTest
Cases
You've created 4,837 test cases.
All of them pass.
How do you know when you'redone creating test cases?
Testing Requirements
TestCases
Desired Adequacy Level
MeasurableAdequacy Criteria
Acceptable?CoverageLevel
Your test cases are achieving100% coverage.
None of them havefound any bugs.
Are those test cases useful?
Maybe those test cases can beused to create new test cases.
The more test cases, the better.Right?
This is the idea behind“metamorphic testing”. [Chen et al., HKUST TR CS-98-01, 1998]
A(really, really, really, really)
simple example
Let's say you're testing acosine function.
(I know, I know...)
How could we use this test caseto create new test cases?
You have a test case {45º, 0.7071}, i.e.cos(45º) = 0.7071
We know that the cosine functionexhibits certain properties.
That is, if we make certain changesto the input, we can predict theeffect on the output.
These are referred to as“metamorphic properties”.
What are the metamorphic propertiesof the cosine function?
cos(x + 360º) = cos(x)
cos(x + 180º) = -1 * cos(x)
cos(x - 360º) = cos(x)
That is, if we add 360 to the input,the output should not change.
Given our original test case {45°, 0.7071}, we can create three follow-on test cases.
Property: cos(x + 360º) = cos(x)Input: 45º + 360º = 405ºOutput: cos(45º) = 0.7071
Property: cos(x - 360º) = cos(x)Input: 45º - 360º = -315ºOutput: cos(45º) = 0.7071
Property: cos(-1 * x) = -1 * cos(x)Input: -1 * 45º = -45ºOutput: -1 * cos(45º) = -0.7071
x f f(x)
Initial test case{x, f(x)}
t(x) f f(t(x))Follow-on test case
{t(x), f(t(x))}
t
A metamorphic property of a function fis a pair of functions (t, g)such that f(t(x)) = g(f(x))for all inputs x
g(f(x))
g
=
but wait, isn’t that the same as…
Program invariants: -1 ≤ cos(x) ≤ 1
Algebraic properties: cos²(x) = 1 – sin²(x)
Describe legal ranges/values of a function,but not how it should react when theinput is changed.
Describe the relationships between multiple functions, but not a single function.
simple categories of properties
a b c d e fInitialtest case
c e b a f dPermute
2a 2b 2c 2d 2e 2fMultiply
ssum
sum
sum
s
sumAdd a+2b+2c+2d+2e+2f+2 s+12
2s
sum s+ga b c d e fInclude g
a b c d eExclude s-fsum
a b c d e fInitialtest case #1 ssum
g h i j k lInitialtest case #2 tsum
Compose s+tsuma b c d e f
g h i j k l
Combinationof properties
2s+2tsum2h 2d 2a 2k 2e 2g
2i 2c 2l 2f 2b 2j
Common Metamorphic Properties
• Additive: Increase (or decrease) numerical values by a constant
• Multiplicative: Multiply numerical values by a constant
• Permutative: Randomly permute the order of elements in a set
• Invertive: Create the “opposite” of a set
• Inclusive: Add a new element to a set
• Exclusive: Remove an element from a set
• Compositional: Compose a set[Murphy et al., SEKE’08]
Other Types of Properties
• Noise-based: include input values that will not affect the output
• Semantically Equivalent: create inputs that are have the same “meaning” as the original
• Heuristic: create inputs that are “close” to the original
• Statistical: create inputs that exhibit the same statistical properties
one more example
Consider a function that takes a setof Points (x-y coordinates) and calculatesthe total distance from the first to the last,via the rest.
What are that function’s metamorphicproperties?
Okay, I think I get it.
But does it really work?!?!
In order to find bugs…
But how can this be?!?!
1. The original test case must pass, even though there is a bug.
2. The follow-on test case must fail.
/* Return the smallest value in the array */int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min;}
/* Return the smallest value in the array */int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min;}
Test case { {2, 1, 4, 3}, 1} PASS!
100% statement coverage!
100% branch coverage!
/* Return the smallest value in the array */int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min;}
Test case { {2, 1, 4, 3}, 1}
Metamorphic property: If we permute theinput, the output remains the same.
Follow-on test case: { {4, 2, 3, 1}, 1} FAIL!
metamorphic testing inthe real world
Bioinformatics [Chen et al., BMC Bioinf., 2009]
Machine Learning [Xie et al., JSS, 2011]
Network Simulation [Chen et al., FTDS, 2009]
Computer Graphics [Guderlei et al., QSIC, 2007]
what types of applicationsis metamorphic testing good for?
Applications that deal primarily withnumerical input and numerical output.
Applications that do not have test oracles.
Applications that use graph-basedalgorithms.
Compilers.[Zhou et al., ISFST’04]
SpecificationProgram
TestInput
ActualOutput
ExpectedOutput
Oracle
what if there is no oracle?
Machine Learning
Length of Stay versus Utilization
0
50
100
150
200
250
300
0 2 4 6 8 10 12
number of beds
unit
s of
tim
e
0
2
4
6
8
10
12
14
16
perc
ent
utiliz
ation
LOS
DoctorUtilizationNurseUtilizationTriageUtilizationClerkUtilization
Discrete Event Simulation
x f f(x)
t(x) f f(t(x))Actual
t
If f(t(x)) = g(f(x)) that does notmean that the output is correct.
g
=?
But if f(t(x)) != g(f(x)) thenone (or both) must be incorrect.
g(f(x))Expected
example: RapidMiner
RapidMiner is a suite of machine learningalgorithms implemented in Java.
That is:c = Classify(M(T), e)
In its NaïveBayes implementation, aconfidence level c is reported wheneverit classifies an example e using a modelM created from a training data set T.
We expect that if we modify T to includean extra instance of e, then the confidencelevel should double, since we are twiceas certain about the classification. That is:Classify(M(T+e), e) = 2 * Classify(M(T), e)
Our testing detected violations of this property, thus revealing a bug.[Murphy et al., ICST’09]
empirical study
Goal:Show that metamorphic testing is moreeffective than other techniques at findingbugs in applications without test oracles.
Approach:Use mutation analysis to insert faultsinto the applications, and see how manyare detected using various techniques.
Application domains investigated:
1. Machine Learning (C4.5, MartiRank, Support Vector Machines, PAYL)
2. Discrete Event Simulation (JSim)
3. Information Retrieval (Lucene)
4. Optimization (gaffitter)
Techniques investigated:
1. Metamorphic Testing
2. Runtime Assertion Checking
3. Partial Oracle
Experimental Results
0 20 40 60 80 100 120
TOTAL
gaffitter
Lucene
JSim
PAYL
SVM
MartiRank
C4.5
% of Mutants Killed
Partial Oracle Runtime Assertion Checking Metamorphic Testing
[Murphy et al., ISSTA’09]
can we do better?
That experiment used application-level metamorphic properties.
What if we test at the function level, too?
This is known asMetamorphic Runtime Checking. [Murphy et al., TR CUCS-042-09, 2009]
And continuously conduct those tests while the software is running?
0 20 40 60 80 100 120
TOTAL
gaffitter
Lucene
JSim
PAYL
SVM
MartiRank
C4.5
Partial Oracle Runtime Assertion Checking Metamorphic Testing MT + MRC
Experimental Results
research directions
When I run my test, I see that themetamorphic property is violated.
Does that mean there's a bug?
How can we know whether themetamorphic properties are sound?
Well, not necessarily....
I've used the guidelines to identify asmany metamorphic properties as I could.
Does that mean that's all of them?
How can we know whether the set ofmetamorphic properties is complete?
Well, not necessarily....
I've used the guidelines to identify asmany metamorphic properties as I could.
Does that mean that's all of them?
Could we detect (likely) metamorphic properties automatically?
Well, not necessarily....
I have a function for which I expect that,if I double the input, the output shouldbe doubled.
Could I verify that property without actually executing the code?
Can metamorphic properties be verified statically?
Well, probably....
summary
Metamorphic testing is a method ofcreating new test cases from existing ones.
It depends heavily on the software’smetamorphic properties, which areoften numerical.
Metamorphic testing is particularlyeffective at finding bugs in applications that do not have test oracles.
thanks!
Applications ofMetamorphic Testing
Chris MurphyUniversity of [email protected]