Constraints in Dynamic Symbolic Execution: Bitvectors or ...
Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution...
Transcript of Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution...
![Page 1: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/1.jpg)
Advanced Test Coverage Criteria: Specificationand Support in Automatic Testing Tools
Nikolai Kosmatov
joint work with Sebastien Bardin, Omar Chebaro, MickaelDelahaye, Michael Marcozzi, Yves Le Traon, Mike Papadakis,
Virgile Prevosto. . .
CEA, LIST, Software Security LabParis-Saclay, France
TAROT 2017, Napoli, June 26, 2017
Nikolai Kosmatov Advanced Test Coverage Criteria 1/ 42
![Page 2: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/2.jpg)
Context : White-Box Testing
Nikolai Kosmatov Advanced Test Coverage Criteria 2/ 42
![Page 3: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/3.jpg)
Coverage criteria in white-box testing
Global goal : bridge the gap between criteria and testing tools
Nikolai Kosmatov Advanced Test Coverage Criteria 3/ 42
![Page 4: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/4.jpg)
Main ingredients of the talk :
Labels : a generic specification mechanism for coverage criteria
◮ based on predicates, can easily encode a large class of criteria◮ w.r.t related work : semantic view, more formal treatment
DSE⋆ : an efficient integration of labels into DSE
◮ no exponential blowup of the search space◮ can be added to DSE in a black-box manner
LTest : Implementation on top of Frama-C and PathCrawler
◮ huge savings compared to existing approaches◮ handles labels with a very low overhead (2x average, up to 7x)
HTOL : Hyperlabel Specification Language, extension of labels
◮ capable to encode almost all common criteria
[Bardin et al., ICST 2014, TAP 2014, ICST 2015][Marcozzi et al., ICST 2017 (research), ICST 2017 (tool)]
Nikolai Kosmatov Advanced Test Coverage Criteria 4/ 42
![Page 5: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/5.jpg)
Outline
1 Dynamic Symbolic Execution (DSE)
2 LabelsNotationExpressiveness
3 Efficient DSE for labelsDirect instrumentationDSE⋆
Tight instrumentationIterative Label Deletion
4 LTest toolset : Implementation and Experiments
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 5/ 42
![Page 6: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/6.jpg)
Dynamic Symbolic Execution
Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,. . . ]
X very powerful approach to white-box test generation
X many tools and many successful case-studies since mid 2000’s
X arguably one of the most wide-spread use of formal methodsin “common software” [SAGE at Microsoft]
Nikolai Kosmatov Advanced Test Coverage Criteria 6/ 42
![Page 7: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/7.jpg)
Dynamic Symbolic Execution
Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,. . . ]
X very powerful approach to white-box test generation
X many tools and many successful case-studies since mid 2000’s
X arguably one of the most wide-spread use of formal methodsin “common software” [SAGE at Microsoft]
Symbolic Execution [King 70’s]
consider a program P on input v, and a given path σ
a path predicate ϕσ for σ is a formula s.t. for any input vv satisfies ϕσ ⇔ P(v) follows σ
old idea, recently renewed interest [requires powerful solvers]
Nikolai Kosmatov Advanced Test Coverage Criteria 6/ 42
![Page 8: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/8.jpg)
Dynamic Symbolic Execution
Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,. . . ]
X very powerful approach to white-box test generation
X many tools and many successful case-studies since mid 2000’s
X arguably one of the most wide-spread use of formal methodsin “common software” [SAGE at Microsoft]
Symbolic Execution [King 70’s]
consider a program P on input v, and a given path σ
a path predicate ϕσ for σ is a formula s.t. for any input vv satisfies ϕσ ⇔ P(v) follows σ
old idea, recently renewed interest [requires powerful solvers]
Dynamic Symbolic Execution [Korel+, Williams+, Godefroid+]
interleaves dynamic and symbolic executions
drives the search towards feasible paths for free
gives hints for relevant under-approximations
Nikolai Kosmatov Advanced Test Coverage Criteria 6/ 42
![Page 9: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/9.jpg)
Dynamic Symbolic Execution (2)
input : a program P
output : a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable ? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 7/ 42
![Page 10: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/10.jpg)
Dynamic Symbolic Execution (2)
input : a program P
output : a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable ? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 7/ 42
![Page 11: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/11.jpg)
Dynamic Symbolic Execution (2)
input : a program P
output : a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable ? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 7/ 42
![Page 12: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/12.jpg)
Dynamic Symbolic Execution (2)
input : a program P
output : a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable ? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 7/ 42
![Page 13: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/13.jpg)
Dynamic Symbolic Execution (2)
input : a program P
output : a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable ? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 7/ 42
![Page 14: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/14.jpg)
Dynamic Symbolic Execution (2)
input : a program P
output : a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable ? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 7/ 42
![Page 15: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/15.jpg)
Dynamic Symbolic Execution (2)
input : a program P
output : a test suite TS covering all feasible paths of Paths≤k(P)
pick an uncovered path σ ∈ Paths≤k(P)is the path predicate ϕσ satisfiable ? [smt solver]
if SAT(s) then add a new pair < s, σ > into TSloop until no more paths to cover
Nikolai Kosmatov Advanced Test Coverage Criteria 7/ 42
![Page 16: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/16.jpg)
The problem
Dynamic Symbolic Execution
X very powerful approach to white-box test generationX arguably one of the most wide-spread use of formal methods
in “common software”
Nikolai Kosmatov Advanced Test Coverage Criteria 8/ 42
![Page 17: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/17.jpg)
The problem
Dynamic Symbolic Execution
X very powerful approach to white-box test generationX arguably one of the most wide-spread use of formal methods
in “common software”× lack of support for many coverage criteria
Nikolai Kosmatov Advanced Test Coverage Criteria 8/ 42
![Page 18: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/18.jpg)
The problem
Dynamic Symbolic Execution
X very powerful approach to white-box test generationX arguably one of the most wide-spread use of formal methods
in “common software”× lack of support for many coverage criteria
Challenge : extend DSE to a large class of coverage criteria
well-known problem
recent efforts in this direction through instrumentation[Active Testing, Mutation DSE, Augmented DSE]
limitations :
◮ exponential explosion of the search space [APex : 272x avg]◮ very implementation-centric mechanisms◮ unclear expressiveness
Nikolai Kosmatov Advanced Test Coverage Criteria 8/ 42
![Page 19: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/19.jpg)
Outline
1 Dynamic Symbolic Execution (DSE)
2 LabelsNotationExpressiveness
3 Efficient DSE for labelsDirect instrumentationDSE⋆
Tight instrumentationIterative Label Deletion
4 LTest toolset : Implementation and Experiments
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 9/ 42
![Page 20: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/20.jpg)
Labels and the notion of simulation
Given a program P , a label l is a pair (loc , ϕ), where :
ϕ is a well-defined predicate in P at location loc
ϕ contains no side-effect expression
Basic definitions
a test datum t covers l if P(t) reaches loc and satisfies ϕ
new criterion LC (label coverage) for annotated programs
a criterion C can be simulated by LC if for any P , after adding“appropriate” labels in P , TS covers C ⇔ TS covers LC.
Goal : show the relative expressiveness of LC
Nikolai Kosmatov Advanced Test Coverage Criteria 10/ 42
![Page 21: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/21.jpg)
Simulation of coverage criteria by labels : DC
statement_1;
if (x==y && a<b)
{...};
statement_3;
−−−−−→
statement_1;
// l1: x==y && a<b
// l2: !(x==y && a<b)
if (x==y && a<b)
{...};
statement_3;
Decision Coverage (DC)
Nikolai Kosmatov Advanced Test Coverage Criteria 11/ 42
![Page 22: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/22.jpg)
Simulation of coverage criteria by labels : CC
statement_1;
if (x==y && a<b)
{...};
statement_3;
−−−−−→
statement_1;
// l1: x==y
// l2: !(x==y)
// l3: a<b
// l4: !(a<b)
if (x==y && a<b)
{...};
statement_3;
Condition Coverage (CC)
Nikolai Kosmatov Advanced Test Coverage Criteria 12/ 42
![Page 23: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/23.jpg)
Simulation of coverage criteria by labels : MCC
statement_1;
if (x==y && a<b)
{...};
statement_3;
−−−−−→
statement_1;
// l1: x==y && a<b
// l2: x==y && a>=b
// l3: x!=y && a<b
// l4: x!=y && a>=b
if (x==y && a<b)
{...};
statement_3;
Multiple-Condition Coverage (MCC)
Nikolai Kosmatov Advanced Test Coverage Criteria 13/ 42
![Page 24: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/24.jpg)
Weak Mutation (WM) testing in a nutshell
mutant M = syntactic modification of program P
weakly covering M = finding t such that P(t) 6= M(t) justafter the mutation
Nikolai Kosmatov Advanced Test Coverage Criteria 14/ 42
![Page 25: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/25.jpg)
Simulation of coverage criteria by labels : WM
One label per mutant
Mutation inside a statement
lhs := e 7→ lhs := e’
◮ add label : e 6= e′
lhs := e 7→ lhs’ := e
◮ add label : &lhs 6= &lhs ′ ∧ (lhs 6= e ∨ lhs ′ 6= e)
Mutation inside a decision
if (cond) 7→ if (cond’)
◮ add label : cond ⊕ cond ′
Beware : no side-effect inside labels
Nikolai Kosmatov Advanced Test Coverage Criteria 15/ 42
![Page 26: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/26.jpg)
Simulation results
Theorem
The following coverage criteria can be simulated by LC : IC, DC,FC, CC, MCC, Input Domain Partition, Run-Time Errors.
Theorem
For any finite set O of side-effect free mutation operators, WMO
can be simulated by LC.
Goals
X GOAL1 : generic specification mechanism for coverage criteria
� GOAL2 : efficient integration into DSE
Nikolai Kosmatov Advanced Test Coverage Criteria 16/ 42
![Page 27: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/27.jpg)
Outline
1 Dynamic Symbolic Execution (DSE)
2 LabelsNotationExpressiveness
3 Efficient DSE for labelsDirect instrumentationDSE⋆
Tight instrumentationIterative Label Deletion
4 LTest toolset : Implementation and Experiments
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 17/ 42
![Page 28: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/28.jpg)
Direct instrumentation P′[APex, Mutation DSE]
Covering label l ⇔ Covering branch True
Nikolai Kosmatov Advanced Test Coverage Criteria 18/ 42
![Page 29: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/29.jpg)
Direct instrumentation P′[APex, Mutation DSE]
Covering label l ⇔ Covering branch True
X sound & complete instrumentation w.r.t. LC
Nikolai Kosmatov Advanced Test Coverage Criteria 18/ 42
![Page 30: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/30.jpg)
Direct instrumentation P′ is not good enough
Nikolai Kosmatov Advanced Test Coverage Criteria 19/ 42
![Page 31: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/31.jpg)
Direct instrumentation P′ is not good enough
Nikolai Kosmatov Advanced Test Coverage Criteria 19/ 42
Non-tightness 1
× P ′ has exponentially more pathsthan P
![Page 32: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/32.jpg)
Direct instrumentation P′ is not good enough
Nikolai Kosmatov Advanced Test Coverage Criteria 19/ 42
Non-tightness 1
× P ′ has exponentially more pathsthan P
Non-tightness 2
× Paths in P ′ too complex◮ at each label, require to cover
p or to cover ¬p◮ π′ covers up to N labels
![Page 33: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/33.jpg)
Direct instrumentation P′ is not good enough
Nikolai Kosmatov Advanced Test Coverage Criteria 19/ 42
X sound & complete instrumentation w.r.t. LC
× dramatic overhead [theory & practice]
![Page 34: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/34.jpg)
Our approach
The DSE⋆ algorithm
Tight instrumentation P⋆ : totally prevents “complexification”
Iterative Label Deletion : discards some redundant paths
Both techniques can be implemented in a black-box manner
Nikolai Kosmatov Advanced Test Coverage Criteria 20/ 42
![Page 35: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/35.jpg)
DSE⋆ : Tight Instrumentation P⋆
Covering label l ⇔ Covering exit(0)
Nikolai Kosmatov Advanced Test Coverage Criteria 21/ 42
![Page 36: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/36.jpg)
DSE⋆ : Tight Instrumentation P⋆
Covering label l ⇔ Covering exit(0)
X sound & complete instrumentation w.r.t. LC
Nikolai Kosmatov Advanced Test Coverage Criteria 21/ 42
![Page 37: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/37.jpg)
DSE⋆ : Direct vs tight instrumentation, P ′ vs P⋆
Nikolai Kosmatov Advanced Test Coverage Criteria 22/ 42
![Page 38: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/38.jpg)
DSE⋆ : Direct vs tight instrumentation, P ′ vs P⋆
Nikolai Kosmatov Advanced Test Coverage Criteria 22/ 42
![Page 39: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/39.jpg)
DSE⋆ : Direct vs tight instrumentation, P ′ vs P⋆
Nikolai Kosmatov Advanced Test Coverage Criteria 22/ 42
![Page 40: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/40.jpg)
DSE⋆ : Direct vs tight instrumentation, P ′ vs P⋆
Nikolai Kosmatov Advanced Test Coverage Criteria 22/ 42
Tightness
X P⋆ has (only) linearly more paths than P
X paths in P⋆ are simple : covers ≤ 1 label
![Page 41: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/41.jpg)
DSE⋆ : Direct vs tight instrumentation, P ′ vs P⋆
Nikolai Kosmatov Advanced Test Coverage Criteria 22/ 42
X sound & complete instrumentation w.r.t. LC
X no complexification of the search space
![Page 42: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/42.jpg)
DSE⋆ : Iterative Label Deletion
Observations
we need to cover each label only once
yet, DSE explores paths of P⋆ ending in already-covered labels
we burden DSE with “useless” paths w.r.t. LC
Nikolai Kosmatov Advanced Test Coverage Criteria 23/ 42
![Page 43: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/43.jpg)
DSE⋆ : Iterative Label Deletion
Observations
we need to cover each label only once
yet, DSE explores paths of P⋆ ending in already-covered labels
we burden DSE with “useless” paths w.r.t. LC
Solution : Iterative Label Deletion
keep a covered/uncovered status for each label
symbolic execution ignores paths ending in a covered label
dynamic execution updates the status [truly requires DSE]
Implementation
symbolic part : a slight modification of P⋆
dynamic part : a slight modification of P ′
Nikolai Kosmatov Advanced Test Coverage Criteria 23/ 42
![Page 44: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/44.jpg)
DSE⋆ : Iterative Label Deletion
Observations
we need to cover each label only once
yet, DSE explores paths of P⋆ ending in already-covered labels
we burden DSE with “useless” paths w.r.t. LC
Solution : Iterative Label Deletion
keep a covered/uncovered status for each label
symbolic execution ignores paths ending in a covered label
dynamic execution updates the status [truly requires DSE]
Implementation
symbolic part : a slight modification of P⋆
dynamic part : a slight modification of P ′
Iterative Label Deletion is relatively complete w.r.t. LC
Nikolai Kosmatov Advanced Test Coverage Criteria 23/ 42
![Page 45: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/45.jpg)
DSE⋆ : Iterative Label Deletion (2)
Nikolai Kosmatov Advanced Test Coverage Criteria 24/ 42
![Page 46: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/46.jpg)
DSE⋆ : Iterative Label Deletion (3)
Nikolai Kosmatov Advanced Test Coverage Criteria 25/ 42
![Page 47: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/47.jpg)
Summary
The DSE⋆ algorithm
Tight instrumentation P⋆ : totally prevents “complexification”
Iterative Label Deletion : discards some redundant paths
Both techniques can be implemented in black-box
Nikolai Kosmatov Advanced Test Coverage Criteria 26/ 42
![Page 48: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/48.jpg)
Summary
The DSE⋆ algorithm
Tight instrumentation P⋆ : totally prevents “complexification”
Iterative Label Deletion : discards some redundant paths
Both techniques can be implemented in black-box
Nikolai Kosmatov Advanced Test Coverage Criteria 26/ 42
Goals
X GOAL1 : generic specification mechanism forcoverage criteria
X GOAL2 : efficient integration into DSE
![Page 49: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/49.jpg)
Outline
1 Dynamic Symbolic Execution (DSE)
2 LabelsNotationExpressiveness
3 Efficient DSE for labelsDirect instrumentationDSE⋆
Tight instrumentationIterative Label Deletion
4 LTest toolset : Implementation and Experiments
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 27/ 42
![Page 50: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/50.jpg)
The LTest toolset for labels [TAP 14]
Nikolai Kosmatov Advanced Test Coverage Criteria 28/ 42
Implementation on top of Frama-C
Frama-C is a toolset for analysis of C programs
◮ an extensible, open-source, plugin-orientedplatform
◮ offers value analysis (VA), weakest precondition(WP), specification language ACSL,...
LTest is open-source except test generation
◮ based on the PathCrawler test generation tool
![Page 51: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/51.jpg)
The LTest toolset for labels [TAP 14]
Nikolai Kosmatov Advanced Test Coverage Criteria 28/ 42
Supported criteria
DC, CC, MCC
FC, IDC, WM
Encoded with labels [ICST 2014]
treated in a unified way
easy to add new criteria
![Page 52: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/52.jpg)
The LTest toolset for labels [TAP 14]
Nikolai Kosmatov Advanced Test Coverage Criteria 28/ 42
DSE⋆ procedure [ICST 2014]
DSE with native support for labels
extension of PathCrawler
![Page 53: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/53.jpg)
The LTest toolset for labels [TAP 14]
Nikolai Kosmatov Advanced Test Coverage Criteria 28/ 42
Uses static analyzers from Frama-C
sound detection of uncoverablelabels
![Page 54: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/54.jpg)
The LTest toolset for labels [TAP 14]
Nikolai Kosmatov Advanced Test Coverage Criteria 28/ 42
Service cooperation
share label statuses
Covered, Infeasible, ?
Uses static analyzers from Frama-C
sound detection of uncoverablelabels
![Page 55: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/55.jpg)
Experiments
Implementation
inside PathCrawler
follows DSE⋆
search heuristics : “label-first DFS”
run in deterministic mode
Goal of experiments
evaluate DSE⋆ versus DSE’
evaluate overhead of handling labels
Benchmark programs
SQLite, OpenSSL
12 programs taken from standard DSE benchmarks (Siemens,Verisec, MediaBench)
3 coverage criteria : CC, MCC, WM
Nikolai Kosmatov Advanced Test Coverage Criteria 29/ 42
![Page 56: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/56.jpg)
Experiments (2)
Results
DSE’ : 4 timeouts (TO), max overhead 122x [excluding TO]
DSE⋆ : no TO, max overhead 7x (average : 2.4x)
on one example, 94s instead of a TO [1h30]
DSE⋆ achieves very high LC-coverage [> 90% on 28/36]
after a static analysis step for detection of uncoverable labels,it becomes even higher [> 99%]
Nikolai Kosmatov Advanced Test Coverage Criteria 30/ 42
![Page 57: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/57.jpg)
Experiments (2)
Results
DSE’ : 4 timeouts (TO), max overhead 122x [excluding TO]
DSE⋆ : no TO, max overhead 7x (average : 2.4x)
on one example, 94s instead of a TO [1h30]
DSE⋆ achieves very high LC-coverage [> 90% on 28/36]
after a static analysis step for detection of uncoverable labels,it becomes even higher [> 99%]
Nikolai Kosmatov Advanced Test Coverage Criteria 30/ 42
![Page 58: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/58.jpg)
Experiments (2)
Results
DSE’ : 4 timeouts (TO), max overhead 122x [excluding TO]
DSE⋆ : no TO, max overhead 7x (average : 2.4x)
on one example, 94s instead of a TO [1h30]
DSE⋆ achieves very high LC-coverage [> 90% on 28/36]
after a static analysis step for detection of uncoverable labels,it becomes even higher [> 99%]
Nikolai Kosmatov Advanced Test Coverage Criteria 30/ 42
![Page 59: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/59.jpg)
Experiments (2)
Results
DSE’ : 4 timeouts (TO), max overhead 122x [excluding TO]
DSE⋆ : no TO, max overhead 7x (average : 2.4x)
on one example, 94s instead of a TO [1h30]
DSE⋆ achieves very high LC-coverage [> 90% on 28/36]
after a static analysis step for detection of uncoverable labels,it becomes even higher [> 99%]
Conclusion
DSE⋆ performs significantly better than DSE’
The overhead of handling labels is kept reasonable
still room for improvement
Nikolai Kosmatov Advanced Test Coverage Criteria 30/ 42
![Page 60: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/60.jpg)
Outline
1 Dynamic Symbolic Execution (DSE)
2 LabelsNotationExpressiveness
3 Efficient DSE for labelsDirect instrumentationDSE⋆
Tight instrumentationIterative Label Deletion
4 LTest toolset : Implementation and Experiments
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 31/ 42
![Page 61: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/61.jpg)
Limitations of labels
Nikolai Kosmatov Advanced Test Coverage Criteria 32/ 42
![Page 62: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/62.jpg)
Hyperlabel Specification Language (HTOL)
Nikolai Kosmatov Advanced Test Coverage Criteria 33/ 42
![Page 63: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/63.jpg)
HTOL : Examples
Nikolai Kosmatov Advanced Test Coverage Criteria 34/ 42
![Page 64: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/64.jpg)
HTOL : Examples
Nikolai Kosmatov Advanced Test Coverage Criteria 35/ 42
![Page 65: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/65.jpg)
HTOL : Examples
Nikolai Kosmatov Advanced Test Coverage Criteria 36/ 42
![Page 66: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/66.jpg)
HTOL : Taxonomy of coverage criteria
Nikolai Kosmatov Advanced Test Coverage Criteria 37/ 42
![Page 67: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/67.jpg)
HTOL : Expressiveness and support
Nikolai Kosmatov Advanced Test Coverage Criteria 38/ 42
![Page 68: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/68.jpg)
Impact of a generic toolset like LTest
Nikolai Kosmatov Advanced Test Coverage Criteria 39/ 42
![Page 69: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/69.jpg)
Outline
1 Dynamic Symbolic Execution (DSE)
2 LabelsNotationExpressiveness
3 Efficient DSE for labelsDirect instrumentationDSE⋆
Tight instrumentationIterative Label Deletion
4 LTest toolset : Implementation and Experiments
5 Hyperlabel Specification Language (HTOL)
6 Conclusion
Nikolai Kosmatov Advanced Test Coverage Criteria 40/ 42
![Page 70: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/70.jpg)
Summary
Goal = express and support a large class of coverage criteria
Results
Labels : a well-defined and expressive specification mechanismfor coverage criteria
DSE⋆ : an efficient integration of labels into DSE◮ no exponential blowup of the search space◮ only a low overhead [huge savings w.r.t. related work]
Hyperlabels : an extension of labels, capable to express almostall existing coverage criteria
Nikolai Kosmatov Advanced Test Coverage Criteria 41/ 42
![Page 71: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/71.jpg)
Summary
Goal = express and support a large class of coverage criteria
Results
Labels : a well-defined and expressive specification mechanismfor coverage criteria
DSE⋆ : an efficient integration of labels into DSE◮ no exponential blowup of the search space◮ only a low overhead [huge savings w.r.t. related work]
Hyperlabels : an extension of labels, capable to express almostall existing coverage criteria
Dynamic Symbolic Execution [dart, cute, exe, sage, pex, klee, . . . ]
X very powerful approach to (white box) test generation
X arguably one of the most wide-spread uses of formal methodsin “common software”
Nikolai Kosmatov Advanced Test Coverage Criteria 41/ 42
![Page 72: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/72.jpg)
Summary
Goal = express and support a large class of coverage criteria
Results
Labels : a well-defined and expressive specification mechanismfor coverage criteria
DSE⋆ : an efficient integration of labels into DSE◮ no exponential blowup of the search space◮ only a low overhead [huge savings w.r.t. related work]
Hyperlabels : an extension of labels, capable to express almostall existing coverage criteria
Dynamic Symbolic Execution [dart, cute, exe, sage, pex, klee, . . . ]
X very powerful approach to (white box) test generation
X arguably one of the most wide-spread uses of formal methodsin “common software”
× support only basic coverage criteriaNikolai Kosmatov Advanced Test Coverage Criteria 41/ 42
![Page 73: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/73.jpg)
Summary
Goal = express and support a large class of coverage criteria
Results
Labels : a well-defined and expressive specification mechanismfor coverage criteria
DSE⋆ : an efficient integration of labels into DSE◮ no exponential blowup of the search space◮ only a low overhead [huge savings w.r.t. related work]
Hyperlabels : an extension of labels, capable to express almostall existing coverage criteria
Dynamic Symbolic Execution [dart, cute, exe, sage, pex, klee, . . . ]
X very powerful approach to (white box) test generation
X arguably one of the most wide-spread uses of formal methodsin “common software”
X can be efficiently extended to a large class of coverage criteriaNikolai Kosmatov Advanced Test Coverage Criteria 41/ 42
![Page 74: Advanced Test Coverage Criteria: Specification and Support ... · Dynamic Symbolic Execution Dynamic Symbolic Execution [dart,cute,pathcrawler,exe,sage,pex,klee,...] X very powerful](https://reader036.fdocuments.net/reader036/viewer/2022070709/5ebef6f112f8d33e101fc731/html5/thumbnails/74.jpg)
Future work
An efficient dedicated support of hyperlabels in testgeneration (DSE)
Further optimizations of LTest (e.g. detection of uncoverablehyperlabels)
Developing the emerging interest for LTool in industry
Nikolai Kosmatov Advanced Test Coverage Criteria 42/ 42