Fitness-Guided Path Exploration in Automated Test Generation
Automated unit test generation presentation - Java2ays 2013
-
Upload
vassil-popovski -
Category
Technology
-
view
736 -
download
4
Transcript of Automated unit test generation presentation - Java2ays 2013
Automated Unit Test Generation
Vassil Popovski, VMware
The Cost of Software Bugs?
$59.9 B per year in USA (NIST, 2002)*
(*) http://www.nist.gov/director/planning/upload/report02-3.pdf
$312 B per year worldwide (Cambridge University, 2012)**
(**) http://www.roguewave.com/DesktopModules/Bring2mind/DMX/Download.aspx?entryid=1606&command=core_download&PortalId=0&TabId=607
$312,000,000,000
The Cost of Software Bugs?
The Cost of Software Bugs?
Total market capitalization for all 3 companies(*):
!
$177.36 B
(*) Google finance - as of Nov 2nd, 2013
The Cost of World Hunger?
$40B per year(*)
(*) According to United Nations
The Cost of Fixing Bugs?
http://www.amazon.com/Assessments-Benchmarks-Addison-Wesley-Information-Technology/dp/0201485427?ie=UTF8&s=books&qid=1209056706&sr=1-1
Software Testing Pyramid
http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid/
Unit Testing: 80-90%
Integration testing: 4-15%
E2E/GUI testing: 1-5%
Techniques/Tools
1. Random data generation 2. Search based 3. Symbolic/Concolic based
The goal: High (close to 100% code coverage*)
Anatomy of a Unit Test
Arrange Act Assert
Anatomy of a Unit Test
Arrange Data Arrange Mocks/Stubs Act Assert
Anatomy of a Unit Test
Evaluation of Tools
Arrange/Generate DataArrange/Generate Mocks/StubsAct/Invoke methodsAssert/Verify the results
Techniques/Tools
x != 100 x == 100
x>=z x<z
Techniques/Tools
x>=z x<z
Random data generation: !
Feed x,y with random data x != 100 x == 100
Demo
Evaluation of Tools
Arrange/Generate DataArrange/Generate Mocks/StubsAct/Invoke methodsAssert/Verify the results
yes - some
no
no
no
junit-quickcheck feed4JUnit
Techniques/Tools
1. Random Data Generation: • junit-quickcheck (Java) https://github.com/pholser/junit-quickcheck/ • Feed4JUnit & Feed4TestNG + Benerator (java) http://databene.org/feed4-tools.html http://databene.org/databene-benerator.html • Randoop (Java + .NET) https://code.google.com/p/randoop/ • T2 (Java) https://code.google.com/p/t2framework/
Techniques/Tools
x != 100 x == 100
x>=z x<z
Search based approach: !
1. Start with random/initial data (x=1, y=1)
Techniques/Toolsx=1, y=1
x != 100 x == 100
x>=z x<z
Search based approach: !
1. Start with random/initial data (x=1, y=1)
Techniques/Toolsx=1, y=1
x != 100 x == 100
x>=z x<z
Search based approach: !
1. Start with random/initial data (x=1, y=1)
Techniques/Tools
Search based approach: !
1. Start with random/initial data (x=1, y=1)
2. Based on coverage information - select new value(s) (x==100)
x=100, y=1
x != 100 x == 100
x>=z x<z
Techniques/Tools
Search based approach: !
1. Start with random/initial data (x=1, y=1)
2. Based on coverage information - select new value(s) (x==100) !
x=100, y=1
x != 100 x == 100
x>=z x<z
Demo
Evaluation of Tools
Arrange/Generate DataArrange/Generate Mocks/StubsAct/Invoke methodsAssert/Verify the results
yes - some
no
no
no
EvoSuite
yes
no
yes
yes
junit-quickcheck feed4JUnit
Techniques/Tools
2. Search Based: • EvoSuite (Java)
http://www.evosuite.org/ • Testful (Java)
https://code.google.com/p/testful/
Techniques/Tools
x != 100 x == 100
x>=z x<z
Symbolic based approach: !1. Evaluate path conditions (PC)
on every branch PC: X!=100, Y PC: X == 100, Y
x=X, y=Y, z=2*Y
PC: X == 100, X>=2*Y
PC: X == 100, X<2*Y
1. X!=100, Y => (x=1, y=1) 2. X=100, X>=2*Y =>(x=100, y=1) 3. X=100, X<2*Y =>(x=100, y=51)
2. For each leaf - calculate X & Y using a constraints solver:
Techniques/Tools
x != 100 x == 100
x>=z x<z
Concolic based approach: !
Optimization of symbolic execution - but use real values for some of the branches
Demo
Evaluation of Tools
Arrange/Generate DataArrange/Generate Mocks/StubsAct/Invoke methodsAssert/Verify the results
yes - some
no
no
no
EvoSuite
yes
no
yes
yes
PEX (+ Moles)
yes
yes - some
yes
yes
junit-quickcheck feed4JUnit
Techniques/Tools
3. Symbolic/Concolic based: • Microsoft Pex (.NET) http://research.microsoft.com/en-us/projects/pex/ • jCUTE (Java) http://osl.cs.uiuc.edu/software/jcute/ • jalangi (Java Script) https://github.com/SRA-SiliconValley/jalangi • AgitarOne (Java, commercial) http://www.agitar.com/solutions/products/agitarone.html • Jtest (Java, commercial) http://www.parasoft.com/jsp/products/jtest.jsp
!
Best Usage of Auto Generated Unit Tests
1.Regression testing of legacy code !
2.To support your unit testing / TDD
Q&A