Nikolai Tillmann, Jonathan de Halleux Tao Xie Microsoft Research Univ. Illinois at Urbana-Champaign.
-
Upload
august-fields -
Category
Documents
-
view
219 -
download
1
Transcript of Nikolai Tillmann, Jonathan de Halleux Tao Xie Microsoft Research Univ. Illinois at Urbana-Champaign.
Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger Nikolai Tillmann, Jonathan de Halleux
Tao Xie Microsoft Research
Univ. Illinois
at Urbana-Champaign
(Automated) Test Generation
Human Expensive, incomplete, …
Brute Force Pairwise, predefined data, etc…
Tool Automation!!
MSR Automated Test Generation Tool: Pex & Relatives
Pex (released on May 2008) 30,388 download# (20 months, Feb 08-Oct 09) Active user community: 1,436 forum posts
during ~3 years (Oct 08- Nov 11) Moles (released on Sept 2009)
Shipped with VS 12 as Fakes “Provide Microsoft Fakes w/ all Visual Studio
editions” got 1,457 community votes Code Digger (released on Oct 2008 for VS
08/10, on Apr 2013 in VS Gallery for VS 12) 22,466 download# (10 months, Apr 13-Jan 14)
http://research.microsoft.com/pex/
Example Comments for Code Digger in VS Gallery
“Great tool to generate unit tests for parameter boundary tests. I like to see it integrated into Visual Studio and the testing features as far as in ReSharper! :)”
“What an awesome tool.. Help us to explore our logic by providing accurate input parameter for each logic branch.. You should try this as one of your ultimate tool :) It really saves a lot of our time to explore every logic branch in our apps..”
Pex4Fun & Code Hunt
1,565,195 clicked 'Ask Pex!'
http://pex4fun.com/
https://www.codehunt.com/
What are Behind Pex
NOT Random: Cheap, Fast “It passed a thousand tests” feeling
…But Dynamic Symbolic Execution:
e.g., Pex, CUTE,EXE White box Constraint Solving
Dynamic Symbolic Execution
Code to generate inputs for:
Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==1234567890
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Observed constraints
a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890
a!=null &&a.Length>0 &&a[0]==1234567890
Data
null
{}
{0}
{123…}a==null
a.Length>0
a[0]==123…T
TF
T
F
F
Execute&MonitorSolve
Choose next path
Done: There is no path left.
Negated condition
MSR Automated Test Generation Tool: Pex & Relatives
Pex (released on May 2008): 30,388 download# (20 months, Feb 08-Oct 09) Active user community: 1,436 forum posts
during ~3 years (Oct 08- Nov 11) Moles (released Sept 2009)
Shipped with VS 12 as Fakes “Provide Microsoft Fakes w/ all Visual Studio
editions” got 1,457 community votes Code Digger (released on Oct 2008 for VS
08/10, on Apr 2013 in VS Gallery for VS 12) 22,466 download# (10 months, Apr 13-Jan 14)
How to make such successful case????
Lesson 1. Evolving “Dreams”
void TestAdd(ArrayList a, object o) { Assume.IsTrue(a!=null); int i = a.Count; a.Add(o); Assert.IsTrue(a[i] == o);}
Parameterized Unit Tests Supported by Pex
Moles/Fakes
Code Digger
Pex4Fun/Code Hunt
Surrounding (Moles/Fakes) Simplifying (Code Digger) Retargeting (Pex4Fun/Code
Hunt)
Lesson 2. Dealing with “Chicken and Egg” Problem
Developer/manager: “Who is using your tool?”
Pex team: “Do you want to be the first?” Developer/manager: “I love your tool but
no.”
Tool Adoption by (Mass) Target Users
Tool Shipping with Visual Studio
Macro Perspective
Micro Perspective
Lesson 2. Dealing with “Chicken and Egg” Problem cont.
Tackle real-world challenges Select/demo applying Pex on real-world cases (e.g.,
ResourceReader) beyond textbook examples Select/demo applying Moles to well address important
scenarios (e.g., unit testing of SharePoint code) Address technical and non-technical barriers
for technology adoption in industry (e.g., tool license)
Incremental shipping (e.g., shipping Code Digger before full-fledge Pex)
Find early adopters Provide quantitative info (reflecting tool’s
importance or benefit extent) Not all downloads are equal! (e.g., those from
Fortune 500)
Lesson 3. Human Factors – Generated Data Consumed by Human
Developer: “Code digger generates a lot of “\0” strings as input. I can’t find a way to create such a string via my own C# code. Could any one show me a C# snippet? I meant zero terminated string.”
Pex team: “In C#, a \0 in a string does not mean zero-termination. It’s just yet another character in the string (a very simple character where all bits are zero), and you can create as Pex shows the value: “\0”.”
Developer: “Your tool generated “\0”” Pex team: “What did you expect?” Developer: “Marc.”
Lesson 3. Human Factors – Generated Name Consumed by Human
Developer: “Your tool generated a test called Foo001. I don’t like it.”
Pex team: “What did you expect?” Developer:“Foo_Should_Fail_When_Bar_Is_Ne
gative.”
Lesson 3. Human Factors – Generated Results Consumed by Human
Object Creation messages suppressed (related to Covana by Xiao et al. [ICSE’11])
Exception Tree View Exploration Tree
View
Exploration Results View
Lesson 4. Best vs. Worst Cases
public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++)
if (y[i] == 15) x++;
if (x == 110) return true;
} return false;}
Key observations: with respect to the coverage target not all paths are equally promising
for branch-node flipping not all branch nodes are equally
promising to flip
• Our solution: – Prefer to flip branch nodes on the most promising paths
– Prefer to flip the most promising branch nodes on paths
– Fitness function to measure “promising” extents
Fitnex by Xie et al. [DSN’09]
To avoid local optimal or biases, the fitness-guided strategy is integrated with Pex’s fairness search strategies
http://research.microsoft.com/apps/pubs/default.aspx?id=81089
Lesson 5. Tool Users’ Stereotypical Mindset or Habits
“Simply one mouse click and then everything would work just perfectly” Often need environment isolation w/ Moles/Fakes or
factory methods, … “One mouse click, a test generation tool would
detect all or most kinds of faults in the code under test” Developer: “Your tool only finds null references.” Pex team: “Did you write any assertions?” Developer: “Assertion???”
“I do not need test generation; I already practice unit testing (and/or TDD). Test generation does not fit into the TDD process”
Lesson 6. Listening to Practitioners
Gathered feedback from target tool users Directly, e.g., via
MSDN Pex forum, tech support, outreach to MS engineers and .NET user groups
Indirectly, e.g., via interactions with MS Visual Studio team (a tool
vendor to its huge user base) Motivations of Moles
Refactoring testability issue faced resistance in practice
Observation at Agile 2008: high attention on mock objects and tool supports
Lesson 7. Collaboration w/ Academia
Win-win collaboration model Win (Ind Lab): longer-term research innovation, man
power, research impacts, … Win (Univ): powerful infrastructure,
relevant/important problems in practice, both research and industry impacts, …
Industry-located Collaborations Faculty visits, e.g., Fitnex, Pex4Fun Student internships, e.g., FloPSy, DyGen,
state cov Academia-located Collaborationshttp://research.microsoft.com/pex/community.aspx#publications
Lesson 7. Collaboration w/ Academia
Academia-located Collaborations Immediate indirect impacts, e.g.,
Reggae [ASE’09s] Rex MSeqGen [FSE’09] DyGen Guided Cov [ICSM’10] state coverage
Long-term indirect impacts, e.g., DySy by Csallner et al. [ICSE’08] Seeker [OOPSLA’11] Covana [ICSE’11]
http://research.microsoft.com/pex/community.aspx#publications
Summary
Pex practice impacts Moles/Fakes, Code Digger, Pex4Fun/Code
Hunt Lessons in transferring tools
Evolving Dreams “Chicken and Egg” Problem Human Factors Best vs. Worst Cases Tool Users’ Stereotypical Mindset or Habits Practitioners’ Voice Collaboration w/ Academia
Thank you
http://research.microsoft.com/pex
Summary
Pex practice impacts Moles/Fakes, Code Digger, Pex4Fun/Code
Hunt Lessons in transferring tools
Evolving Dreams “Chicken and Egg” Problem Human Factors Best vs. Worst Cases Tool Users’ Stereotypical Mindset or Habits Practitioners’ Voice Collaboration w/ Academia