Software Testing & Quality Assurance · Software Testing & Quality Assurance Overview The...
Transcript of Software Testing & Quality Assurance · Software Testing & Quality Assurance Overview The...
Software Testing & Quality Assurance
Overview
� The Psychology of Testing
� Testing terminology and techniques
– i.e., black-box vs. white-box testing, static vs. dynamic
� Integration Testing
� Software Inspections
� Control-Flow Analysis
TESTING 1
So you think you know testing?...
Consider the following example:
Write a suite of tests for a program that reads three integers from the
command line. Those three values will be interpreted as the lengths of
the legs of a triangle. The program will determine if this triangle is
equilateral, isoceles, or scalene and output its result.
� Write down a set of inputs which you think would be good tests for this
program.
TESTING 2
The dark side of testing
� The majority of start-up companies (and some established ones) today have a
rapid development mentality.
� They tend to employ quick and dirty schemes of developing software.
� Unfortunately, today’s quick and dirty rapid development schemes have
major holes in them.
� “Yes, we ran it and it worked”
� “Yes, I sat down and typed lots of inputs at it and it didn’t fall over.”
� “No, we don’t have time. It compiled and linked OK.”
TESTING 3
Problems with these naïve views
� “Yes, we ran it and it worked”
� So what requirements did you test?
� “Yes, I sat down and typed lots of inputs at it and it didn’t fall over.”
� So what part of the system passed or failed?
� “No, we don’t have time. It compiled and linked OK.”
� So when can testing start?
We need systematic approaches to finding out if our software runs properly.
i.e., according to what we intended when we set out to build our software.
TESTING 4
The Psychology of Testing
“Testing is the process of executing a program with the intent of finding
errors.” Glenford Myers, The Art of Software Testing
� Because we are human, if we are supposed to prove the correctness of a
program, we will undeliberately run tests that are less likely to detect faults.
� Software developers have this mentality.
� “If the goal is to show the absence of errors, we will find very few of
them.”
� However, if the goal of a test is to find faults, then we will automatically
orient our efforts in this direction.
� Software testers have this mentality.
� “If the goal is to show the presence of errors, we will discover a large
number of them.”
TESTING 5
Testing vs. Debugging
These are two distinct processes related to the psychology of developers and
testers.
� Verification and validation (i.e., testing ) is concerned with establishing the
existence of defects in a program.
� This is along the psyche of the software testers.
� Debugging is concerned with locating and repairing these defects.
� It also involves formulating a hypothesis about program behaviour and
testing this hypothesis to find the error.
� This is along the psyche of the software developers.
TESTING 6
Verification and Validation
Validation
� Is the right thing built?
� The software should do what the customer really requires.
i.e., was the specification what the customer intended?
Verification
� Is the thing built right?
� The software must conform to its specification.
We will concentrate most on verification techniques, as validation deals more
with requirements analysis.
TESTING 7
Static vs. Dynamic Verification
These are two methods used to verify programs.
Software testing (dynamic)
� This method is concerned with executing the software product using test data
and observing its behaviour.
Software inspections (static)
� This method is concerned with the analysis of the static system
representations to discover faults.
� It may be supplemented by tool-based analysis of documents and program
code.
TESTING 8
Testing Terminology
Let’s revisit the different error definitions.
� A fault is the design flaw, the mistake someone made that eventually caused
the system to fail (or not).
� This is a static phenomenon.
� A failure is the event that made you take notice that something is wrong.
� It’s the crash, the bad output.
� It may occur far from the fault.
� This is a dynamic phenomenon.
TESTING 9
Testing Terminology (ctd.)
Some more jargon that you should be familiar with:
� A test case is the collection of inputs, expected results, environment and
procedural requirement for a single test.
� A test suite is a collection of test cases necessary to “adequately” test a
product.
� A test plan is a document describing the scope, approach, resources, and
scheduled testing activities.
� A test harness (a.k.a test driver) is some code driver that is used to
(interactively) test the routine(s)/classes required.
TESTING 10
Classification of Types of Testing
There are different types of testing used at various stages of the testing process.
� Structured testing (i.e., white-box testing) are test cases generated based on
the internal structure of the software.
– i.e., Structural testing � white-box techniques.
� Functional testing (i.e., black-box testing) are test cases generated based on
the software specification.
– i.e., Functional testing � black-box techniques.
� Grey-box testing is based on test cases for class or module interactions.
– Slightly more recent idea for OO and structurally rich designs.
– Not dependent on code details but more than just call-and-return
functional testing.
– The tests must be established after detailed software design.
TESTING 11
Testing stages vs. Testing types
There are five stages of testing usually found in the development of a system:
1. Unit
2. Integration
3. System
4. Installation
5. Acceptance
Unit Integration System Installation Acceptance
Structural Y ? N N N
Functional N ? Y Y Y
Grey-box N Y N N N
TESTING 12
Structural (White-box) Testing
� Here, we get to look in to the source code as much as possible.
– i.e., we examine code-level issues such as statements, blocks, logical
paths
– The test cases are volatile as they may not survive maintenance.
� There are two forms of white-box testing which we will examine:
1. Logic Coverage Testing
2. Loop Testing
TESTING 13
What is Logic Coverage Testing?
� Logic Coverage is one of the oldest and most widely used white-box testing
strategies today.
� Test cases are derived from the program’s logical structure.
� It attempts to expose software defects by exercising a unique combination of
the program’s statements known as a path.
– Logic coverage is sometimes referred to as path testing.
TESTING 14
Logic Coverage Testing Strategies
� There are different strategies within logic coverage testing.
� They vary in their coverage criteria.
– This is the domain of paths through the program that acheive a
“complete” test.
1. Exhaustive Path Coverage Criterion
2. Statement Coverage Criterion
3. Decision (or Branch) Coverage Criterion
4. Condition Coverage Criterion
5. Decision/Condition Coverage Criterion
6. Multiple Condition Coverage Criterion
TESTING 15
Exhaustive Path Testing
� Theoretically the most complete of the logic coverage criteria.
� It requires EVERY possible path to be exercised.
� Unfortunately, even programs of modest complexity can have an enormous
number of paths due to the combinatorial explosion of loops and branches.
– In reality, this is almost impossible to achieve.
TESTING 16
Statement Coverage Testing
� This is the simplest coverage criterion with the fewest number of test cases.
� It requires every statement in the program to be executed at least oncea.
� Consider the following piece of code:
if (A && B) {
C;
}
D;
� A single test case (A is true and B is true) will satisfy the criterion.
� We haven’t verified that statement C will NOT be executed when A is false
and/or B is false.
� Statement coverage is a necessary but not sufficient criterion for effectivestructural testing.
aUntil we switched to Java for first and second year courses, there was a tool calledanalyze coverage which ran these types of coverage tests for Modula-3 programs.
TESTING 17
Decision (or Branch) Coverage Testing
� Decision coverage requires that each branch within the decision be evaluated.
� When considering the code from the previous slide, only two test cases
would be required to satisfy this criterion.
– A is true and B is truea
– A is false and B is true
� Two other cases derived from the condition are left unevaluated.
– A is true and B is true
– A is false and B is false
� Decision coverage subsumes statement coverage.
� However, decision coverage does not detect all faults in compound
conditions.
aAlso satisfies the statement coverage criterion
TESTING 18
Condition Coverage Testing
� This is similar to decision coverage but it is more sensitive to the control flow.
� Condition coverage requires that each condition in a decision to take on all
possible values at least once.
� Consider a slightly different piece of code:
if (A || B) {
C;
}
D;
� So, only two test cases would be required to satisfy this criterion.
– A is true and B is falsea
– A is false and B is truea
� However, these test cases neglect to bypass statement C.
� Thus, the decision coverage criterion is no longer satisfied.aAlso satisfies the statement coverage criterion
TESTING 19
Decision/Condition Coverage Testing
� This combines both the decision and condition coverage criteria.
� It requires that each condition take on all possible outcomes at least once and
each decision take on all possible outcomes at least once.
� Again from the previous slide,
if (A || B) {
C;
}
D;
� But, still we can find just two test cases required to satisfy this criterion.
– A is true and B is truea
– A is false and B is false
aAlso satisfies the statement coverage criterion
TESTING 20
Multiple-Condition Coverage Testing
� Multiple-condition coverage subsumes statement coverage, decision
coverage, condition coverage and decision/condition coverage criteria and
does not mask conditions.
� i.e., In the previous example, (A is false and B is true) was masked
because it was subsumed in the first test case.
� It requires all possible combinations of condition outcomes in each decision.
� Thus, there are
� n combinations where n is the number of conditions.
� Unfortunately, it is extremely tedious to determine the minimum set of test
cases required, especially for complex Boolean expressions.
TESTING 21
Loop Testing
� There are four types of loops that need to be examined in structural testing:
1. Deterministic Loops (usually for)
� The number of times the loop will execute is known in advance.
� Before the first statement is executed, a variable is defined that will
not change throughout the running of the loop.
2. Non-deterministic Loops (usually while)
� The number of times the loop will execute is not known in advance.
� There are three instances where non-deterministic loops may arise:
(a) The max. value is unknown.
(b) Processing within the loop changes the control variable, if any.
(c) A condition contained in the loop causes it to exit hastily.
3. Nested Loops
4. Unstructured Loops
� Occurs when a process jumps out of or into the middle of a loop.
TESTING 22
Loop Testing: Critical Test Values
� Critical test values are a combination of values allowing loops to proceed
through normal and faulty operation.
Bypass: any value that will cause the loop to exited immediately.
Once: values causing the loop to be executed once.
Twice: values causing the loop to be executed exactly twice.
Typical: a typical number of iterations.
Max: the maximum number of allowed iterations.
Max + 1: one more than the max allowed.
Max - 1: one less than the max allowed.
Min: the minimum required.
Min - 1: one less than the minimum required.
Null: no input.
Negative: negative input.
TESTING 23
Loop Testing Example: Course Handout System
� We are testing a software system that handles course handouts with a
minimum of 1 and a maximum of 12 students in the class.
� Using critical test value criteria, we design the following inputs:
Bypass: No students.
Once: One student.
Twice: Two students.
Typical: 10 students.
Max: 12 students.
Max + 1: 13 students.
Max - 1: 11 students.
Min: Redundant.
Min - 1: Redundant
Null: Redundant
Negative: Negative # of students?
� After defining our critical test values, we would test the looping specification
with the inputs and examine the theoretical output.
� Remember: even though some cases above are a stretch, testers are paidto cause software behaviour to act out of the ordinary.
TESTING 24
Loop Testing: Conclusions
� Loop testing is a common technique based on the experience programmers
have with errors in starting and finishing loops.
� i.e., the majority of loops aren’t simple for-loops.
� The most common types of loops implemented are the deterministic,
non-deterministic and nested loops.
� By using the method of defining critical test values, an analyst uses systems
specifications to design tests capable of breaking certain loops prematurely.
� This technique is also effective for testing not only control-flow graphs,
but also all flow models with the possible exception of finite state
machines (too many loops).
TESTING 25
Integration Testing
� This method of testing is needed whenever modules or subsystems are
combined to create a larger system.
� The goal is to identify errors due to interface faults or to invalid interface
assumptions.
� This technique is particularly important in OO system developement.
TESTING 26
Integration Testing: Types of Interfaces
� Shared Memory Interfaces (Concurrent packages)
� A block of memory is shared between components.
� Procedural Interfaces (e.g., libraries, components)
� Set of procedures/classes encapsulated in a package or a subsystem.
� Message-Passing Interfaces (client-server)
� Subsystems require services from each other.
� Pass the messages from the client to the server.
TESTING 27
Integration Testing: Interface Errors
� Interface misuse
� Wrong parameter order.
� Wrong number of parameters.
� Wrong parameter types.
� Interface misunderstanding
� The call to the component make incorrect assumptions about the
component being called.
� Timing errors (threads and concurrency)
� Race conditions.
� Data synchronization errors.
TESTING 28
Integration Testing Guidelines
� Design your tests so actual parameters passed are at extreme ends of formal
parameter ranges.
� i.e., Percentile ranges should be tested at 0 and 100.
� Test pointer variables with null values.
� Frequently, the problem with objects being passed is that they are not
initialized.
� Design tests that cause components to fail.
� Use stress testing in message passing systems.
� i.e., make sure that the server can handle the load from the number of
clients.
� In shared memory systems, vary the order in which components are activated.
� This can frequently lead to a combinatorial explosion of possible
combinations and should be first examined more closely.
TESTING 29
System Testing Guidelines
� Testing of critical systems must often rely on simulators for sensor and
activator data.
� You don’t want to endanger people and/or profit.
� Testing for normal operations should be done using a safely obtained
operational profile.
� Tests for exceptional conditions will need to involve simulators.
TESTING 30
Software Inspections and Walk-throughs
� A software inspection is an example of qualified individuals formally
examining a source code representation to discover anomalies and defects.
� An inspection does not require the system to execute, so it may occur
before implementation.
� It may be applied to any type of system documentation.
– i.e., document, model, test data, code, etc.
� Walk-throughs are informal inspections.
� Inspections are very effective for discovering defects.
� It is possible to discover several defects in a single inspection.
� The inspections use the domain and programming knowledge of the
“experts” reviewing the code.
– i.e., reviewers help the authors avoid make common errors.
TESTING 31
Inspections and Testing
� Inspections and testing are complementary processes.
� Inspections are used to check conformance to specifications.
� Testing checks compliance with non-functional system characteristics
such as performance, usability, etc...
� Like testing, inspections focus on defect detection, not defect correction.
� Defects uncovered may be logic errors, coding errors, or non-compliance
with development standards.
� Inspections and walk-throughs are considered two of the first methods of
testing without the use of a computer.
� These methods have also been called human testing.
TESTING 32
Before you start your inspection...
� A precise specification must be available.
� The reviewing team members must be familiar with organization standards.
� All representations must be syntactically correct.
� An error checklist must be prepared in advance.
� Management must buy into the fact that inspections will increase the early
development costs.
� Inspections cannot be used to evaluate staff performance.
TESTING 33
Doing the inspection...
1. Code and associated documents are distributed to the team in advance.
2. The system overview is presented to the inspection team and the inspection is
driven via a checklist of common errors.
� The error checklist should be language dependent.
� The weaker the type-checking of the language, the larger the checklist is
likely to become.
3. Errors discovered during the inspection are recorded.
4. Product modifications are recorded to repair defects.
� Re-inspection may or may not be required.
TESTING 34
The Inspection Team
There are at least four members on the inspection team:
� The product author
� The inspector
– Looks for errors, omissions, and inconsistencies.
� The reader
– Reads the code to the team.
� The moderator
– Chairs meeting and records errors uncovered.
TESTING 35
Inspection Fault Classes
� There are several types of fault classes to observe when performing an
inspection.
– Data faults (e.g., array bounds, variables not initialized)
– Control faults (e.g., loop termination, unreachable code)
– Input/output faults (e.g., all the data read, duplicate variables output)
– Interface faults (e.g., parameter assignment & type mismatches)
– Storage management faults (e.g., memory leaks, pointer arithmetic)
– Exception management faults (e.g., all error conditions trapped)
TESTING 36
Spending Time on Inspections
� There are three places where the system is reviewed.
1. During the overview
– approx. 500 statements per hour.
2. During individual preparation
– approx. 125 statements per hour.
3. During the team team inspection
– approx. 90-125 statements per hour.
� Including preparation time, each 100 lines of code costs approximately
one-person day, if a 4 person team is used.
TESTING 37
Automated Static Analysis Tools vs. Inspections
� Performed by software tools that process source code listings.
– i.e., integrated tools into commercial IDEs, program understanding tools,
reverse engineering tools, etc.
� They can be used to flag potentially erroneous conditions for the inspection
team to examine.
� These tools should supplement before hand the reviews done by the
inspectors.
TESTING 38
Control Flow Testing
� The control flow of a program may be modelled by a directed graph.
� Nodes can be considered as statements functions, procedures, objects, or
even subsystems.
– Nodes that are fundamental indivisible operations or statements are
called basic blocks.
� Edges are directed relations between the nodes.
– i.e., interface connections (procedural, message-passing ...)
� These flows can model representations of sequences, selections and iterations
within a set of specifications.
� Control flow graphs are used in static analysis.
� This technique is not restricted to structural (white-box) or functional
(black-box) testing.
– However, we will focus on its capabilities for black-box testing of
specifications.
TESTING 39
Control Flow Testing Definitions
� An independent path is any path that
introduces at least one new set of
nodes.
– i.e., it must traverse an edge not
previously covered in another
path.
What are the independent paths in
this example?6
1
2
3
5
4
� The cyclomatic complexity is the upper bound on the number of independent
paths.
cyclomatic complexity = edges - nodes + 1
TESTING 40
Control Flow Testing Definitions (ctd.)
� An predicate node is a node that
has two or more out-edges, such
that each edge is weighted by a
corresponding predicate value
(likely a logical boolean result).
� A selector node is a node with
more than two out-edges, where
each edge is weighted by a
selector value
� A junction node is a node with
two or more in-edges.
11
11.1
11.4
11.3
11.2
9
8’8TRUE
FALSE
Married
Divorced
Widow
Single
TESTING 41
Modeling Compound Predicates
� Analyzing and testing compound predicates can be daunting due to the level
of complexity hidden within them.
� The higher the complexity � the higher the likelihood that a softwareengineer has introduced an error during coding.
� So, to model a compound predicate, you would build a predicate tree with as
many branches as there are possibilities.
� The rule is for n predicates, there are
� n branches.
� Building a predicate tree:
1. Break the compound predicate into separate nodes.
2. The branches are then attached to the model and flow of the predicate is
then determined.
� To validate a compound predicate, you must model
� n cases independent ofone another
� Alternative: Build a truth table.
TESTING 42
Design and Execution Steps in Control Flow Testing
1. Examine the requirements and analyze them for operationally satisfactory
completeness.
2. Rewrite the specification as a sequence of short sentences.
3. Number the sentences uniquely (i.e., 1, 2, 3, ...)
4. Build the model.
5. Verify the model because your work is as “buggy” as the programmer’s code.
6. Select test paths.
7. Sensitize the paths you picked.
8. Predict and record the expected outcome for each test.
9. Define the validation criteria for each test.
10. Run the tests.
11. Confirm the outcomes.
12. Confirm the path.
TESTING 43
Control Flow Testing Example - A Vending Machine
� Examine the Insert Coin function for a vending machine program.
– Requirements:
1. The coin is inserted.
2. Check if the coin is legal.
3. If the coin is illegal, then throw an exception.
4. If the coin is legal, then add it to the current value inserted.
5. Display the total coins inserted on screen.
� Even though we don’t have the code for this function, we can still apply
control-flow testing as a black-box approach.
� Let’s follow the 12 design and execution steps to model this function.
TESTING 44
Control Flow Testing Example - A Vending Machine
1. Label each line of the insertmethod independently which will
be used as predicate labels.
2. Once we have the code labeled,we build the control-flow model.
3. Since we only have one predicate
node, node 2, in this model, thenext step of selecting our test
paths is rather simple.
4. Next we must sensitize the twotest paths by selecting input values
that would cause the method to do
the equivalent traversal in theabsence of errors.
5. After defining the inputs, we
researched similar test cases and
predicted the outcome of our tests.
6. Now we use the inputs to run the
model and record the outputs.
1
2
3 4-5
Exception Exit
TESTING 45
Control Flow Testing Conclusions
� Control-flow testing is a fundamental model of functional testing.
� To build a control-flow model, you must refer to documents such as the
Software Requirements Specification to ensure cohesiveness between the test
and the specification.
� During path design, you must ensure that all of the links are covered by as
many test paths as needed.
� In the end, the validity of a test case can only be ensured if the program and
the test model are bug free.
TESTING 46
Software Testing Conclusions
� There are many ways to approach software testing.
� However, the best frame of mind to be in is to search for errors in a
program, and not to check and see if the program works properly.
� This methodology can be done statically via code inspections, or
dynamically by verifying the code’s behaviour using test cases.
� There are two fundamental methods used to approach dynamic testing:
1. Structural (white-box) testing, which bases test-cases on the internal
structure of the software (e.g., coverage & loop testing).
2. Functional (black-box) testing, which bases test-cases on the software’s
specifications (e.g., control-flow).
� There are different stages of testing for different views of the system.
– i.e., unit, integration, system, acceptance, etc.
TESTING 47