Model-Driven Testing For Agile Teams
-
Upload
kerrykimbrough -
Category
Documents
-
view
216 -
download
3
description
Transcript of Model-Driven Testing For Agile Teams
![Page 1: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/1.jpg)
Model-Driven Testing
for Agile Teams
Kerry Kimbrough Cornutum Project
![Page 2: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/2.jpg)
This Workshop Is About...
• A design technique...
– Input space modeling
• Using a tool...
– Tcases, a model-driven test case generator
• To tackle test design issues...
![Page 3: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/3.jpg)
Test Design Issues
• How do I pick test cases?
– Define an input model. Then generate test cases.
• How many test cases do I need?
– Define a coverage model. Then generate a minimal test set.
• Are my tests good enough?
– Is my input model good enough?
– Is my coverage model good enough?
![Page 4: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/4.jpg)
Why Does It Matter?
• Defects are a drag!
– The friction that slows delivery
• The Agile way
– Test early
– Test continuously
• And yet...
![Page 5: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/5.jpg)
The Curse of the Undone
Umm, sure Sprint’s done!
Stories finished? Except for fixing
the defects!
Dev QA Product
![Page 6: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/6.jpg)
The Busted Sprint
We had a lot of stuff to fix up in the 1st story.
We planned 3 stories, but only
finished 2?
Dev QA Product
![Page 7: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/7.jpg)
The Sprint That Never Was
We had to work on those production problems!
Dev QA Product
We planned 2 stories, but only
finished 1??
![Page 8: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/8.jpg)
The Hardening (?!) Sprint
Almost. Just have to finish the hardening sprint!
Dev QA Product
Stories done! Can we release now?
![Page 9: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/9.jpg)
Why Model-Driven Testing?
• Much more powerful tests
• For little or no extra effort
• At every step of the way
– Before dev
– During dev
– During system test
![Page 10: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/10.jpg)
Tcases: How To Get It
www.cornutum.org
Download it here
See “Installing Tcases”
![Page 11: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/11.jpg)
Example: find <pattern> <file>
Locates one or more instances of a given pattern in a text file.
All lines in the file that contain the pattern are written to standard output. A line containing the pattern is written only once, regardless of the number of times the pattern occurs in it.
The pattern is any sequence of characters whose length does not exceed the maximum length of a line in the file. To include a blank in the pattern, the entire pattern must be enclosed in quotes. To include a quotation mark in the pattern, two quotes in a row ("") must be used.
![Page 12: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/12.jpg)
Modeling The Input Space
• All combinations of values
• For each dimension of variation
![Page 13: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/13.jpg)
Defining System Functions
<System name=“Examples” <!-- All system function definitions go here -->
<Function name=“find” <!-- All input definitions for “find” go here -->
</Function> </System>
![Page 14: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/14.jpg)
Defining Input Variables
<Function name="find“>
<Input type="arg"> <!--arg: Direct input arguments (the default)-->
<Var name="fileName"> ... </Var> ... </Input>
<Input type="env"> <!--env: Environment state variables-->
... </Input> </Function>
![Page 15: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/15.jpg)
Defining Input Values
<Function name=“find” <Input type=“arg”>
<Var name=“fileName”
<!-- The required file name is defined -->
<Value name=“defined”/>
<!-- The required file name is missing: an error -->
<Value name=“missing” failure=“true”/>
</Var> ... </Input> </Function>
![Page 16: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/16.jpg)
To Define Input Values...
• Enumerate all values
• Or identify “equivalence classes”
– A subset of values that are “test-equivalent”
– No need to test all -- any one will do
– Examples
• Integer: Negative, Zero, Positive
• Bounded: Below-Min, In-Range, Above-Max
• List: Empty, One, Many
![Page 17: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/17.jpg)
Modeling Complex Inputs
find <pattern> <file>
All lines in the file that contain the pattern are written to standard output. A line containing the pattern is written only once, regardless of the number of times the pattern occurs in it.
![Page 18: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/18.jpg)
Defining Variable Sets
<VarSet name="file">
<!-- Does the file exist? -->
<Var name="exists"> ... </Var>
<!-- Does the file contain... -->
<VarSet name="contents">
<!-- ... any matching lines? -->
<Var name="patterns"> ... </Var>
<!-- ... multiple matches in a line? -->
<Var name="patternsInLine"> ... </Var>
</VarSet> </VarSet>
![Page 19: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/19.jpg)
Modeling Complex Inputs
find <pattern> <file>
The pattern is any sequence of characters whose length does not exceed the maximum length of a line in the file. To include a blank in the pattern, the entire pattern must be enclosed in quotes. To include a quotation mark in the pattern, two quotes in a row ("") must be used.
![Page 20: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/20.jpg)
Defining Variable Sets
<VarSet name="pattern">
<!-- How many chars in pattern? -->
<Var name="size"> ... </Var>
<!-- Is it quoted? -->
<Var name="quoted"> ... </Var>
<!-- Does it contain blanks? -->
<Var name="blanks"> ... </Var>
<!-- Does it contain embedded quotes? -->
<Var name="embeddedQuotes"> ... </Var>
</VarSet>
![Page 21: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/21.jpg)
System Input Model Basics
<System ...> <!-- A system has... -->
<Function ...> <!-- functions, having... -->
<Input ...> <!-- different types of inputs: -->
<Var ...> <!-- simple variables, having... -->
<Value ...> ...</Value> <!-- (classes of) values -->
</Var>
<VarSet ...> <!-- or complex inputs, composed of...-->
<Var ...> ...</Var> <!-- many variables... -->
<VarSet ...> ...</VarSet> <!-- and many levels -->
</VarSet>
</Input>
</Function>
</System>
![Page 22: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/22.jpg)
Defining Constraints
• Values can have properties
– Assumed by any test case that includes this
value
• Values can have conditions
– Properties that must appear in any test case
that includes this value.
– Properties that must not appear in any test
case that includes this value.
![Page 23: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/23.jpg)
When Variables Are Irrelevant
<Function name="find"> <Input type="arg">
<VarSet name="pattern" when="fileExists">
<Var name="blanks" whenNot="empty"> ... </Var> ... </VarSet>
... </Input> ... </Function>
![Page 24: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/24.jpg)
Input Model vs. Coverage Model
• Input model
– Defines the input space
– Defines the input combinations possible
• Coverage model
– Defines the input combinations tested
![Page 25: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/25.jpg)
What Is Input Space Coverage?
• Measures input combinations tested
• A “black box” coverage metric
– Specification-based
• Complement to “white box” coverage
– Code-based
– Measures lines/branches/conditions tested
• Basic unit: an N-tuple
– Combination of values from N input variables
![Page 26: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/26.jpg)
Default: 1-Tuple Coverage
<TestCase id="2"> <Input type="arg"> <Var name="pattern.size" value="manyChars"/> <Var name="pattern.quoted" value="yes"/> <Var name="pattern.blanks" value="many"/> <Var name="pattern.embeddedQuotes" value="none"/> <Var name="fileName" value="defined"/> </Input> <Input type="env"> <Var name="file.exists" value="yes"/> <Var name="file.contents.linesLongerThanPattern" value="many"/> <Var name="file.contents.patterns" value="one"/> <Var name="file.contents.patternsInLine" value="one"/> </Input> </TestCase>
![Page 27: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/27.jpg)
2-Tuple (Pairwise) Coverage
<TestCase id="2"> <Input type="arg"> <Var name="pattern.size" value="manyChars"/> <Var name="pattern.quoted" value="yes"/> <Var name="pattern.blanks" value="many"/> <Var name="pattern.embeddedQuotes" value="none"/> <Var name="fileName" value="defined"/> </Input> <Input type="env"> <Var name="file.exists" value="yes"/> <Var name="file.contents.linesLongerThanPattern" value="many"/> <Var name="file.contents.patterns" value="one"/> <Var name="file.contents.patternsInLine" value="one"/> </Input> </TestCase>
![Page 28: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/28.jpg)
Failures From Multiple Factors
Source: NIST http://csrc.nist.gov/groups/SNS/acts
![Page 29: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/29.jpg)
Example: Web-Client.Config
<Function name="Config"> <!-- Set up the configuration for a Web client test --> <Input> <Var name="Browser"> <Value name="Firefox"/> <Value name="Chrome"/> </Var> <Var name="OS"> <Value name="Linux"/> <Value name="Windows"/> <Value name="Mac-OS-X"/> </Var> <Var name="Flash"> <Value name="Yes"/> <Value name="No"/> </Var> </Input> </Function>
![Page 30: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/30.jpg)
Example: Web-Client.Config
• Default coverage (1-tuple)
– 7 tuples
– 3 test cases
• 2-tuple coverage
– 16 tuples
– 6 test cases (minimum)
• All permutations
– 12 tuples
– 12 test cases
![Page 31: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/31.jpg)
Tool Tips: Use Tcases To...
• Define multiple coverage levels
• Reuse and extend previous test cases
• Create random combinations
• Simplify tests with once=“true”
• Transform test case output
![Page 32: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/32.jpg)
Model-Driven Testing Workflow
Define Build
Verify
![Page 33: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/33.jpg)
Model-Driven Testing Workflow
Define
Example use cases:
Explore and clarify
with high-level
system input model
![Page 34: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/34.jpg)
Model-Driven Testing Workflow
Build
Unit tests: Generate and
improve with detailed
unit input models
![Page 35: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/35.jpg)
Model-Driven Testing Workflow
Verify
System tests: Generate coverage
for system configuration space
![Page 36: Model-Driven Testing For Agile Teams](https://reader036.fdocuments.net/reader036/viewer/2022081514/5583849ad8b42a8e0c8b46e5/html5/thumbnails/36.jpg)
The End
Done! Done? Hell, yeah!
Dev QA Product