The Design Processpeople.tamu.edu/~rob.light/spring2019/csce315... · TESTING FOCUS •Broadly,...
Transcript of The Design Processpeople.tamu.edu/~rob.light/spring2019/csce315... · TESTING FOCUS •Broadly,...
TESTINGPROGRAMMING STUDIO, SPRING 2019
ROBERT LIGHTFOOT
Reference: Code Complete 2nd Edition Ch. 22, Ch. 8, ..
TESTING – WHAT IT IS AND WHAT IT IS NOT
• Testing helps find that errors exist, does not fix them
• Debugging finds their source and fixes them
• Systematic attempt to break a program that is working
• Unlike all other parts of software development, whose goal is to avoid errors
• Requires a different mindset by the developers
• Can never prove absence of errors
• Can you think of all possible cases?
• Testing alone does not improve quality
• To improve quality, the development has to be made better instead of just running
more test cases
• Testing is less effective than “collaborative construction” (will get to that later)
• Different statistics show that Testing alone can only find between 50-60% errors
present
TESTING FOCUS
• Broadly, there are 2 types of testing:
• Black-box testing
• White-box testing
• Black-box testing is done without knowing the code
• Done by non-developers, customers etc.
• Example: Beta testing of Windows outside Microsoft
• White-box is performed by the developers
• Knowledge of the code is used to form adversarial and tactical
test cases
• This is a.k.a developer testing, which is the main focus here
TYPES OF DEVELOPER TESTING
• Unit testing• Testing of a single class, routine, program• Code comes from a single programmer• Testing in isolation from the system
• Component testing• Testing of a class, package, program• Usually involves code from multiple programmers or teams• Still in isolation from the whole system
• Integration testing• Combined test of two or more classes, packages,
components, or subsystems• Absolutely necessary before finish/release, but can start
very early
TYPES OF DEVELOPER TESTING(CONTINUED)
• Regression testing
• Repetition of previously tested cases to find new errors introduced
• The set of test cases keeps growing
• System testing
• Executing software in final configuration, including integration with all other systems and hardware
• Security, performance, resource loss, timing issues
OTHER TYPES OF TESTING
• Usually by specialized test personnel
• User tests
• Performance tests
• Configuration tests
• Usability tests
• Etc.
• We’re interested now in developer tests (and later
in some usability tests)
WRITING TEST CASES FIRST• Helps identify errors more quickly
• Doesn’t take any more effort than writing tests later• Just requires resequencing work
• Requires thinking about requirements and design before writing code• Shows problems with requirements sooner (can’t write
code without good requirements)
• Foundation of Test Driven Development (we will get to this…)
TESTING AS YOU WRITE CODE• Motivation: Waiting until later means you have to
relearn code• Fixes will be less thorough and more fragile• Fixes further from the source are more costly
• Types of Testing in this category:• Boundary Testing• Pre- and Post-conditions• Assertions• Defensive Programming• Error Returns
Table taken from “Code Complete”
BOUNDARY TESTING
• Most bugs occur at boundaries
• If it works at and near boundaries, it likely works elsewhere
• Check loop and conditional bounds when written
• Check that extreme cases are handled
• e.g. Full array, Empty array, One element array
• Usually should check value and +/- 1
• Mental test better than none at all
• Very effective in many cases
if (a < MAX_VAL){….
}Test cases:
a = {MAX_VAL-1, MAX_VAL, MAX_VAL+1}
for(i=0; i<num; i++){….
}Test cases:
num = {-1, 0, 1}
vector<int> a, b(1);sort (a);sort (b);
PRECONDITIONS AND POSTCONDITIONS
• Verify that routine starts with correct preconditions and produces correct post-conditions
• Check to make sure preconditions met• Handle failures cleanly without surprise• Example: Check the denominator before dividing,
check if the file was actually opened before writing etc.
• Verify that post-conditions are met• No inconsistencies created• The array does not get deleted after sorting
• Need to define pre-/post-conditions clearly• “Provable” software relies on this approach
ASSERTIONS• Available in most languages
• assert.h (C/C++),
• Assert, AssertionError (Java)
• Way of checking for pre-/post-conditions
• Helps identify where problem occurs
• Before the assertion
• e.g. usually in calling routine, not callee
• Problem: causes abort
• So, useful for cleansing errors in the beginning
…int a = 1;assert (a >= 1); // will run w/o any problemassert (a < 1); // will abort the program after ..
// ..reporting assertion violation
DEFENSIVE PROGRAMMING
• Add code to handle the “can’t happen” cases
• Program “protects” itself from bad data
• Idea is similar to “defensive driving” training
• Skeptical about the rest of the world (i.e., gracefully handle bad
input w/o crashing)
• Example: Negative ISBN for the books in the IA
ERROR RETURNS
• Good API and routine design includes error codes
• Need to be checked
• Similar in functionality is Exception handling (via try-
catch) supported in many languages now
• Need to be clear about what exceptions are thrown, though
• Should be maintained when code evolves
ERROR RETURN EXAMPLE
int main (){
int writestatus = writetofile (somedata);if (writestatus == 0){
cout << "Write operation successful" << endl;return 0;
}else if (writestatus == ERR_TOO_MUCH_DATA){
cout << "Cannot write: Data too long" <<endl;
return 1;}else if (writestatus == ERR_CANNOT_OPEN){
cout << "Cannot write: File cannot be opened"<< endl;
return 1;}else{
cout << "Unrecognized response from writetofile() function" << endl;
return 1;}
}
#define MAX_DATA_LEN 1024#define ERR_TOO_MUCH_DATA 1#define ERR_CANNOT_OPEN 2
int writetofile (string filename, string data){
if (data.size () > MAX_DATA_LEN){return ERR_TOO_MUCH_DATA;
}ofstream myfile (filename);if (!myfile.is_open()) {
return ERR_CANNOT_OPEN;}myfile << data;myfile.close();return 0;
}
TRY-CATCH EXAMPLE
int main (){
try{writetofile (somedata);
}catch (int writestatus){
if (writestatus == ERR_TOO_MUCH_DATA){cout << "Cannot write: Data too long" << endl;return 1;
}else if (writestatus == ERR_CANNOT_OPEN){
cout << "Cannot write: Cannot open file" << endl;return 1;
}else{
cout << "Unrecognized exception caught from writetofile() function" << endl;
return 1;}
}cout << "Write operation successful" << endl;return 0;
}
#define MAX_DATA_LEN 1024#define ERR_TOO_MUCH_DATA 1#define ERR_CANNOT_OPEN 2
void writetofile (string filename,string data){
if (data.size () >MAX_DATA_LEN){
throw ERR_TOO_MUCH_DATA;}
ofstream myfile (filename);if (!myfile.is_open()) {
throw ERR_CANNOT_OPEN;}myfile << data;myfile.close();
}
SYSTEMATIC TESTING SUGGESTIONS
• Test incrementally
• Test simple parts first
• Know what output to expect
• Verify conservation properties
• Compare independent implementations
• Measure test coverage
TEST INCREMENTALLY
• Don’t wait until everything is finished before test
• Test components, not just system
• Test components individually before connecting
them
TEST SIMPLE PARTS FIRST
• Test most basic, simplest features
• Learn to doubt your simplest code – it is not easy
• Finds the “easy” bugs (and usually most important)
first
class Record{int num;int price;
};
cout<<sizeof(Record);
class Record{int num;int price;void f();
};
cout<<sizeof(Record);
class Record{int num;int price;virtual void f();
};
cout<< sizeof(Record);
KNOW WHAT OUTPUT TO EXPECT
• Design test cases that you will know the answer to!
• Seeing some answer does not mean your program is right
• Make hand-checks convenient
• Pick test cases that you can hand-calculate
• Example: sort an array {1,3,2}
• Not always easy to do
• e.g. compilers, numerical programs, graphics
VERIFY CONSERVATION PROPERTIES
• Specific results may not be easily verifiable
• Have to write the program to compute the answer to
compare to
• But, often we have known output properties related
to input
• e.g. #Start + #Insert - #Delete = #Final
• Can verify these properties even without verifying
larger result
COMPARE INDEPENDENT IMPLEMENTATIONS
• Multiple implementations to compute same data should agree
• Useful for testing tricky code, e.g. to increase performance• Write a slow, brute-force routine
• Compare the results to the new, “elegant” routine
• If two routines communicate (or are inverses), different people writing them helps find errors• Only errors will be from consistent misinterpretation of
description
MEASURE TEST COVERAGE
• What portion of code base is actually tested?
• Important, because actual coverage can be less than what developers usually think
• Code-structure-dependent and analytically computable
• Specific rules to calculate them
• Tend to work well on only small/moderate code pieces
• For large software, tools help judge coverage
STRUCTURED BASIS TESTING
• Testing every line in a program
• Ensure that every statement gets tested
• Need to test each part of a logical statement
• Easily catch basic and typo-like errors
• Fewer cases than other tests
• But, also not as thorough
• Goal is to minimize total number of test cases
• One test case can test several statements
STRUCTURED BASIS TESTING (CONTINUED)
• Start with base case where all Boolean conditions are true
• Design test case for that situation
• Each branch, loop, case statement increases minimum
number of test cases by 1
• One more test case per variation, to test the code for that
variation
if (a)statement_1
elsestatement_2
statement_3if (b)
statement_4else
statement_5
How many test cases?
Case #1: a=T, b=T
Case #2: a=F, b=F
STRUCTURED BASIS TESTING (CONTINUED)
• Start with base case where all Boolean conditions are true
• Design test case for that situation
• Each branch, loop, case statement increases minimum number
of test cases by 1
• One more test case per variation, to test the code for that
variation
How many test cases now?
Case #1: a=T, b=X, c=T
Case #2: a=F, b=T, c=F
Case #3: a=F, b=F, c=X
“X” means does not matter
if (a)statement_1
else if (b)statement_2
else statement_6
statement_3if (c)
statement_4else
statement_5
LOGIC COVERAGE
• A.k.a. Code Coverage
• Testing every path through the code
• Can grow (nearly) exponentially with number of
choices/branches
• Only suitable for small to medium size codes
• Why? - Dependency between statements
if (a)statement_1
elsestatement_2
statement_3if (b)
statement_4else
statement_5
How many possible paths in this code?
Path #1: Statements 1, 3, 4 (a=T, b=T)
Path #2: Statements 1, 3, 5 (a=T, b=F)
Path #3: Statements 2, 3, 4 (a=F, b=T)
Path #4: Statements 2, 3, 5 (a=F, b=F)
How about code with more if conditions??
# of test cases = # of paths (here, 4)
COVERAGE PROBLEM
• Structured basis testing is:
• More practical since test cases grow linearly with number of branching
• However, does not cover dependency between statements
• Logic coverage on the other hand:
• Covers all possible dependency issues
• But very quickly becomes overwhelming
• Due of exponential growth with branching statements
• We need something in-between:
• Answer is Data Flow Testing covered next
DATA FLOW TESTING
• Examines data rather than control
• Data in one of three states:
• Defined – Initialized but not used (int a =1)
• Used – In computation or as argument (b = a+1)
• Killed – Undefined in some way, e.g.,
• memory deleted (delete x)
• File closed (fclose (fp))
• Variables related to routines
• Entered – Routine starts just before variable is acted upon
• Exited – Routine ends immediately after variable is acted upon
DATA FLOW TESTING (CONTINUED)
• First, check for any anomalous data sequences• Defined-defined
• Defined-exited
• Defined-killed
• Entered-killed
• Entered-used
• Killed-killed
• Killed-used
• Used-defined
• Often can indicate a serious problem in code design
• After that check, write test cases
int a = 1;
a = 5;
…
DATA FLOW TESTING (CONTINUED)
• First, check for any anomalous data sequences• Defined-defined
• Defined-exited
• Defined-killed
• Entered-killed
• Entered-used
• Killed-killed
• Killed-used
• Used-defined
• Often can indicate a serious problem in code design
• After that check, write test cases
void func ()
{
...
...
int a = 1;
return;
}
DATA FLOW TESTING (CONTINUED)
• First, check for any anomalous data sequences• Defined-defined
• Defined-exited
• Defined-killed
• Entered-killed
• Entered-used
• Killed-killed
• Killed-used
• Used-defined
• Often can indicate a serious problem in code design
• After that check, write test cases
int* x = new int [10];
delete x;
DATA FLOW TESTING (CONTINUED)
• First, check for any anomalous data sequences• Defined-defined
• Defined-exited
• Defined-killed
• Entered-killed
• Entered-used
• Killed-killed
• Killed-used
• Used-defined
• Often can indicate a serious problem in code design
• After that check, write test cases
void func ()
{
delete x;
fclose (fp);
...
...
}
DATA FLOW TESTING (CONTINUED)
• First, check for any anomalous data sequences• Defined-defined
• Defined-exited
• Defined-killed
• Entered-killed
• Entered-used
• Killed-killed
• Killed-used
• Used-defined
• Often can indicate a serious problem in code design
• After that check, write test cases
void func ()
{
int a = x;
...
...
}
DATA FLOW TESTING (CONTINUED)
• First, check for any anomalous data sequences• Defined-defined
• Defined-exited
• Defined-killed
• Entered-killed
• Entered-used
• Killed-killed
• Killed-used
• Used-defined
• Often can indicate a serious problem in code design
• After that check, write test cases
delete x;
...
delete x;
DATA FLOW TESTING (CONTINUED)
• First, check for any anomalous data sequences• Defined-defined
• Defined-exited
• Defined-killed
• Entered-killed
• Entered-used
• Killed-killed
• Killed-used
• Used-defined
• Often can indicate a serious problem in code design
• After that check, write test cases
delete x;
...
int a = *x + 1
DATA FLOW TESTING (CONTINUED)
• First, check for any anomalous data sequences• Defined-defined
• Defined-exited
• Defined-killed
• Entered-killed
• Entered-used
• Killed-killed
• Killed-used
• Used-defined
• Often can indicate a serious problem in code design
• After that check, write test cases
int x;
...
int a = x + 1;
...
x = 100;
DATA FLOW TESTING (CONTINUED)
• Write test cases to examine all defined-used
paths
• Usually requires:
• More cases than structured basis testing
• Fewer cases than logic coverage
EXAMPLE – LOGIC/CODE COVERAGE
1. Conditions: T T T
2. Conditions: T T F
3. Conditions: T F T
4. Conditions: T F F
5. Conditions: F T T
6. Conditions: F T F
7. Conditions: F F T
8. Conditions: F F F
Tests all possible paths
if (cond1) {
x = a;
} else {
x = b;
}
if (cond2) {
y = x+1;
} else {
y = x+2;
}
if (cond3) {
z = c;
} else {
z = d;
}
EXAMPLE - STRUCTURED BASIS TESTING
1. Conditions: T T T
2. Conditions: F F F
Tests all lines of code
if (cond1) {
x = a;
} else {
x = b;
}
if (cond2) {
y = x+1;
} else {
y = x+2;
}
if (cond3) {
z = c;
} else {
z = d;
}
EXAMPLE - DATA FLOW TESTING
1. Conditions: T T T
2. Conditions: T F F
3. Conditions: F T X
4. Conditions: F F X
Tests all defined-used paths
Note: cond3 is independent of
first two
if (cond1) {
x = a;
} else {
x = b;
}
if (cond2) {
y = x+1;
} else {
y = x+2;
}
if (cond3) {
z = c;
} else {
z = d;
}
EXAMPLE - DATA FLOW TESTING
1. Conditions: T T T
2. Conditions: T F F
3. Conditions: F T X
4. Conditions: F F X
Tests all defined-used paths
Note: cond3 is independent of
first two
if (cond1) {
x = a;
} else {
x = b;
}
if (cond2) {
y = x+1;
} else {
y = x+2;
}
if (cond3) {
z = c;
} else {
z = d;
}
EXAMPLE - DATA FLOW TESTING
1. Conditions: T T T
2. Conditions: T F F
3. Conditions: F T X
4. Conditions: F F X
Tests all defined-used paths
Note: cond3 is independent of
first two
if (cond1) {
x = a;
} else {
x = b;
}
if (cond2) {
y = x+1;
} else {
y = x+2;
}
if (cond3) {
z = c;
} else {
z = d;
}
EXAMPLE - DATA FLOW TESTING
1. Conditions: T T T
2. Conditions: T F F
3. Conditions: F T X
4. Conditions: F F X
Tests all defined-used paths
Note: cond3 is independent of
first two
if (cond1) {
x = a;
} else {
x = b;
}
if (cond2) {
y = x+1;
} else {
y = x+2;
}
if (cond3) {
z = c;
} else {
z = d;
}
TEST CASE DESIGN(IF YOU DON’T KNOW THE CODE)
• Boundary analysis still applies
• Equivalence partitioning• Don’t create multiple tests to do the same thing
• Bad data• Too much/little
• Wrong kind/size
• Uninitialized
• Good data• Minimum/maximum normal configuration
• “Middle of the Road” data
• Compatibility with old data
TEST AUTOMATION
• Should do lots of tests, and by-hand is not usually appropriate
• Scripts can automatically run test cases, report on errors in output• But, we need to be able to analyze output
automatically…
• Can’t always simulate good input (e.g. interactive programs)
• People cannot be expected to remain sharp over many tests
• Automation reduces workload on programmer, remains available in the future
REGRESSION TESTING
• Goal: Find anything that got broken by “fixing”
something else
• Save test cases, and correct results
• With any modifications, run new code against all
old test cases
• Add new test cases as appropriate
• Part of Test-Driven Development (TDD) (coming
up)
TEST SUPPORT TOOLS
• Test Scaffold
• Framework to provide just enough support and interface
to test
• Stub Routines and Test Harness
• Test Data Generators
• System Perturber
STUB ROUTINES
• This is the thing (routine/class) under test
• Doesn’t provide full functionality, but pretends to
do something when called
• Return control immediately
• Burn cycles to simulate time spent
• Print diagnostic messages
• Return standard answer
• Get input interactively rather than computed
• Could be “working” but slow or less accurate
TEST HARNESS
• Calls the routine being tested
• Fixed set of inputs
• Interactive inputs to test
• Command line arguments
• File-based input
• Predefined input set
• Can run multiple iterations
TEST DATA GENERATORS
• Can generate far more data than by hand
• Can test far wider range of inputs
• Can detect major errors/crashes easily
• Need to know answer to test correctness
• Useful for “inverse” processes – e.g.
encrypt/decrypt
• Should weight toward realistic cases
• The feasible scope is always limited
SYSTEM PERTURBERS
• Modify system so as to avoid problems that
are difficult to test otherwise
• Reinitialize memory to something other than 0
• Find problems not caught because memory is “usually”
null
• Rearrange memory locations
• Find problems where out-of-range queries go to a
consistent place in other tests
• Memory bounds checking
• Memory/system failure simulation
OTHER TESTING TOOLS
• Diff tools• Compare output files for differences
• Coverage monitors• Determine which parts of code tested
• Data recorder/loggers• Log events to files, save state information
• Error databases• Keep track of what’s been found, and rates of errors
• Symbolic debuggers• Will discuss debugging later, but useful for tests
TEST-DRIVEN DEVELOPMENT
• Generally falls under Agile heading
• We will come back to what is meant by this
• A style of software development, not just a matter of
testing your code
• Enforces testing as part of the development process
TEST DRIVEN DEVELOPMENT OVERVIEW
• Repeat this process:
1. Write a new test
2. Run existing code against all tests; it should generally
fail on the new test
3. Change code as needed
4. Run new code against tests; it should pass all tests
5. Refactor the code
TEST WRITING FIRST
• Idea is to write tests, where each test adds some degree
of functionality
• Passing the tests should indicate working code (to a point)
• The tests will ensure that future changes don’t cause
problems
RUNNING TESTS
• Use a test harness/testing framework of some sort
to run the tests
• A variety of ways to do this, including many existing
frameworks that support unit tests
• JUnit is the most well-known, but there is similar
functionality across a wide range of languages
TEST FRAMEWORK
• Specify a test fixture
• Basically builds a state that can be tested
• Set up before tests, removed afterward
• Test suite run against each fixture
• Set of tests (order should not matter) to verify various aspects of
functionality
• Described as series of assertions
• Runs all tests automatically
• Either passes all, or reports failures
• Better frameworks give values that caused failure
REFACTORING
• As code is built, added on to, it becomes messier
• Need to go back and rewrite/reorganize sections
of the code to make it cleaner
• Do this on a regular basis, or when things seem like
they could use it
• Only refactor after all tests are passing
• Test suite guarantees refactoring doesn’t hurt.
REFACTORINGCOMMON OPERATIONS
• Extract Class
• Extract Interface
• Extract Method
• Replace types with subclasses
• Replace conditional with polymorphic objects
• Form template
• Introduce “explaining” variable
• Replace magic number with symbolic constant
RESOURCES
• Test-Driven Development By Example
• Kent Beck; Addison Wesley, 2003
• Test-Driven Development A Practical Guide
• David Astels; Prentice Hall, 2003
• Software Testing A Craftsman’s Approach (3rd edition)
• Paul Jorgensen; Auerback, 2008
• Many other books on testing, TDD, also