1

COSC 4406COSC 4406

Software Software EngineeringEngineering

Haibin Zhu, Ph.D.Haibin Zhu, Ph.D.Dept. of Computer Science and mathematics, Nipissing University, Dept. of Computer Science and mathematics, Nipissing University,

100 College Dr., North Bay, ON P1B 8L7, Canada, 100 College Dr., North Bay, ON P1B 8L7, Canada, haibinz@nipissingu.ca, http://www.nipissingu.ca/faculty/haibinz, http://www.nipissingu.ca/faculty/haibinz

2

Lecture 10Lecture 10 Software Testing Strategies Software Testing Strategies

3

Software Software TestingTesting

Testing is the process of exercising aTesting is the process of exercising aprogram with the specific intent of findingprogram with the specific intent of findingerrors prior to delivery to the end user.errors prior to delivery to the end user.

Verification: refers to the set of activities that ensure that software correctly implements a specific function.

Validation: refers to a different set of activities that ensure that software that has been built is traceable to customer requirements.

4

What Testing ShowsWhat Testing Shows

errorserrors

requirements conformancerequirements conformance

performanceperformance

an indicationan indicationof qualityof quality

5

Who Tests the Who Tests the Software?Software?

developerdeveloper independent testerindependent tester

Understands the system Understands the system

but, will test "gently"but, will test "gently"

and, is driven by "delivery"and, is driven by "delivery"

Must learn about the system,Must learn about the system,but, will attempt to break itbut, will attempt to break itand, is driven by qualityand, is driven by quality

6

Testing StrategyTesting Strategy

unit testunit test integrationintegrationtesttest

validationvalidationtesttest

systemsystemtesttest

7

Testing and SLDCTesting and SLDC

8

Testing StrategyTesting Strategy

We begin by ‘We begin by ‘testing-in-the-small’testing-in-the-small’ and move and move toward ‘toward ‘testing-in-the-large’testing-in-the-large’

For conventional softwareFor conventional software The module (component) is our initial focusThe module (component) is our initial focus Integration of modules followsIntegration of modules follows

For OO softwareFor OO software our focus when “testing in the small” changes from an our focus when “testing in the small” changes from an

individual module (the conventional view) to an OO individual module (the conventional view) to an OO class that encompasses attributes and operations and class that encompasses attributes and operations and implies communication and collaborationimplies communication and collaboration

9

Strategic IssuesStrategic Issues

Specify product requirements in a quantifiable manner long Specify product requirements in a quantifiable manner long before testing commences.before testing commences.

State testing objectives explicitly. State testing objectives explicitly. Understand the users of the software and develop a profile Understand the users of the software and develop a profile

for each user category.for each user category. Develop a testing plan that emphasizes “rapid cycle testing.”Develop a testing plan that emphasizes “rapid cycle testing.” Build “robust” software that is designed to test itselfBuild “robust” software that is designed to test itself Use effective formal technical reviews as a filter prior to Use effective formal technical reviews as a filter prior to

testingtesting Conduct formal technical reviews to assess the test strategy Conduct formal technical reviews to assess the test strategy

and test cases themselves. and test cases themselves. Develop a continuous improvement approach for the testing Develop a continuous improvement approach for the testing

process. process.

10

Test StepsTest Steps

11

Unit TestingUnit Testing

modulemoduleto beto betestedtested

test casestest cases

resultsresults

softwaresoftwareengineerengineer

12

Unit TestingUnit Testing

interface interface local data structureslocal data structures

boundary conditionsboundary conditionsindependent pathsindependent pathserror handling pathserror handling paths

modulemoduleto beto betestedtested

test casestest cases

13

Selective TestingSelective Testing Common errors:Common errors:

Misunderstood or incorrect arithmetic precedenceMisunderstood or incorrect arithmetic precedence Mixed mode operationsMixed mode operations Incorrect initializationsIncorrect initializations Precision inaccuracyPrecision inaccuracy Incorrect symbolic representation of an expressionIncorrect symbolic representation of an expression

The errors that should be uncoveredThe errors that should be uncovered Comparison of different data typesComparison of different data types Incorrect operators or precedenceIncorrect operators or precedence Expectation of equality when precision error makes equality Expectation of equality when precision error makes equality

unlikelyunlikely Incorrect comparison of variablesIncorrect comparison of variables Proper or nonexistent loop terminationProper or nonexistent loop termination Failure to exit when divergent iteration is encounteredFailure to exit when divergent iteration is encountered Improperly modified loop variables Improperly modified loop variables

14

Unit Test Unit Test EnvironmentEnvironment

ModuleModule

stubstub stubstub

driverdriver

RESULTSRESULTS

interface interface

local data structureslocal data structures

boundary conditionsboundary conditions

independent pathsindependent paths

error handling pathserror handling paths

test casestest cases

15

Integration Testing StrategiesIntegration Testing StrategiesIntegration testingIntegration testing is a systematic technique for is a systematic technique for constructing the software architecture while at the same constructing the software architecture while at the same time conducting tests to uncover errors associated with time conducting tests to uncover errors associated with interfacing.interfacing.

Options:Options:•• the “big bang” approachthe “big bang” approach•• an incremental construction strategyan incremental construction strategy

16

Top Down IntegrationTop Down Integration

top module is tested with top module is tested with stubsstubs

stubs are replaced one at stubs are replaced one at a time, "depth first"a time, "depth first"

as new modules are integrated, as new modules are integrated, some subset of tests is re-runsome subset of tests is re-run

AA

BB

CC

DD EE

FF GG

17

Bottom-Up IntegrationBottom-Up Integration

drivers are replaced one at a drivers are replaced one at a time, "depth first"time, "depth first"

worker modules are grouped into worker modules are grouped into builds and integratedbuilds and integrated

AA

BB

CC

DD EE

FF GG

clustercluster

18

Sandwich TestingSandwich Testing

Top modules areTop modules aretested with stubstested with stubs

Worker modules are grouped into Worker modules are grouped into builds and integratedbuilds and integrated

AA

BB

CC

DD EE

FF GG

clustercluster

19

Regression TestingRegression Testing

It is the re-execution of some subset of tests that It is the re-execution of some subset of tests that have already been conducted to ensure that have already been conducted to ensure that changes have not propagated unintended side changes have not propagated unintended side effects.effects.

It is the activity that helps to ensure that It is the activity that helps to ensure that changes do not introduce unintended behaviors changes do not introduce unintended behaviors or additional errors.or additional errors.

20

Smoke TestingSmoke Testing A common approach for creating “daily builds” A common approach for creating “daily builds”

for product softwarefor product software Smoke testing steps:Smoke testing steps:

Software components that have been translated into Software components that have been translated into code are integrated into a “build.” code are integrated into a “build.”

A build includes all data files, libraries, reusable A build includes all data files, libraries, reusable modules, and engineered components that are required modules, and engineered components that are required to implement one or more product functions.to implement one or more product functions.

A series of tests is designed to expose errors that will A series of tests is designed to expose errors that will keep the build from properly performing its function. keep the build from properly performing its function.

The intent should be to uncover “show stopper” errors The intent should be to uncover “show stopper” errors that have the highest likelihood of throwing the software that have the highest likelihood of throwing the software project behind schedule.project behind schedule.

The build is integrated with other builds and the entire The build is integrated with other builds and the entire product (in its current form) is smoke tested daily. product (in its current form) is smoke tested daily.

The integration approach may be top down or bottom up.The integration approach may be top down or bottom up.

21

Object-Oriented Object-Oriented TestingTesting

begins by evaluating the correctness and begins by evaluating the correctness and consistency of the OOA and OOD modelsconsistency of the OOA and OOD models

testing strategy changestesting strategy changes the concept of the ‘unit’ broadens due to encapsulationthe concept of the ‘unit’ broadens due to encapsulation integration focuses on classes and their execution integration focuses on classes and their execution

across a ‘thread’ or in the context of a usage scenarioacross a ‘thread’ or in the context of a usage scenario validation uses conventional black box methodsvalidation uses conventional black box methods

test case design draws on conventional methods, test case design draws on conventional methods, but also encompasses special featuresbut also encompasses special features

22

Broadening the View of Broadening the View of “Testing”“Testing”

It can be argued that the review of OO analysis and It can be argued that the review of OO analysis and design models is especially useful because the design models is especially useful because the same semantic constructs (e.g., classes, attributes, same semantic constructs (e.g., classes, attributes, operations, messages) appear at the analysis, operations, messages) appear at the analysis, design, and code level. Therefore, a problem in the design, and code level. Therefore, a problem in the definition of class attributes that is uncovered definition of class attributes that is uncovered during analysis will circumvent side effects that during analysis will circumvent side effects that might occur if the problem were not discovered might occur if the problem were not discovered until design or code (or even the next iteration of until design or code (or even the next iteration of analysis). analysis).

23

OOT OOT StrategyStrategy

class testing is the equivalent of unit testingclass testing is the equivalent of unit testing operations within the class are testedoperations within the class are tested the state behavior of the class is examinedthe state behavior of the class is examined

integration applied three different strategiesintegration applied three different strategies thread-based testing—integrates the set of classes thread-based testing—integrates the set of classes

required to respond to one input or eventrequired to respond to one input or event use-based testing—integrates the set of classes use-based testing—integrates the set of classes

required to respond to one use caserequired to respond to one use case cluster testing—integrates the set of classes cluster testing—integrates the set of classes

required to demonstrate one collaborationrequired to demonstrate one collaboration

24

Validation testingValidation testing Focus is on software requirementsFocus is on software requirements Test if the product is the right oneTest if the product is the right one

The function or performance characteristics conforms to The function or performance characteristics conforms to specification.specification.

A deviation from specification is uncovered and a A deviation from specification is uncovered and a deficiency list is created.deficiency list is created.

Configuration reviewConfiguration review To ensure that all elements of the software configuration To ensure that all elements of the software configuration

have been properly developed have been properly developed Alpha/Beta testingAlpha/Beta testing

Focus is on customer usageFocus is on customer usage Alpha test is conducted at the developer’s site.Alpha test is conducted at the developer’s site. Beta test is conducted at the end-user’s site.Beta test is conducted at the end-user’s site.

25

System testingSystem testing System testing System testing focuses on system integrationfocuses on system integration

It is a series of different test whose primary purpose is to fully It is a series of different test whose primary purpose is to fully exercise the computer-based system. exercise the computer-based system.

Recovery testingRecovery testing forces the software to fail in a variety of ways and verifies that forces the software to fail in a variety of ways and verifies that

recovery is properly performed, MTTR is evaluated.recovery is properly performed, MTTR is evaluated. Security testingSecurity testing

verifies that protection mechanisms built into a system will, in verifies that protection mechanisms built into a system will, in fact, protect it from improper penetrationfact, protect it from improper penetration

Stress testingStress testing executes a system in a manner that demands resources in executes a system in a manner that demands resources in

abnormal quantity, frequency, or volumeabnormal quantity, frequency, or volume Performance TestingPerformance Testing

test the run-time performance of software within the context of test the run-time performance of software within the context of an integrated systeman integrated system

26

Debugging: Debugging: A Diagnostic ProcessA Diagnostic Process

27

Debugging processesDebugging processes

28

Debugging EffortDebugging Effort

time requiredtime requiredto diagnose theto diagnose thesymptom andsymptom anddetermine thedetermine thecausecause

time requiredtime requiredto correct the errorto correct the errorand conductand conductregression testsregression tests

29

Symptoms & CausesSymptoms & Causes

symptomsymptomcausecause

symptom and cause may be symptom and cause may be geographically separated geographically separated

symptom may disappear when symptom may disappear when another problem is fixedanother problem is fixed

cause may be due to a cause may be due to a combination of non-errors combination of non-errors

cause may be due to a system cause may be due to a system or compiler erroror compiler error

cause may be due to cause may be due to assumptions that everyone assumptions that everyone believesbelieves

symptom may be intermittentsymptom may be intermittent

30

Consequences of BugsConsequences of Bugs

damagedamage

mildmild annoyingannoying

disturbingdisturbingseriousserious

extremeextremecatastrophiccatastrophic

infectiousinfectious

Bug TypeBug Type

Bug Categories:Bug Categories: function-related bugs, function-related bugs, system-related bugs, data bugs, coding bugs, system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards design bugs, documentation bugs, standards violations, etc.violations, etc.

31

Debugging TechniquesDebugging Techniques

brute force / testingbrute force / testing

backtrackingbacktracking

inductioninduction

deductiondeduction

32

Debugging: Final Debugging: Final ThoughtsThoughts

Don't run off half-cocked, Don't run off half-cocked, thinkthink about the about the symptom you're seeing.symptom you're seeing.

Use toolsUse tools (e.g., dynamic debugger) to gain (e.g., dynamic debugger) to gain more insight.more insight.

If at an impasse, If at an impasse, get helpget help from someone else.from someone else.

Be absolutely sure to Be absolutely sure to conduct regression testsconduct regression tests when you do "fix" the bug.when you do "fix" the bug.

1.1.

2.2.

3.3.

4.4.

33

SummarySummary

Software TestSoftware Test To uncover errors that were made inadvertently.To uncover errors that were made inadvertently.

Strategic IssuesStrategic Issues Test Strategies for conventional softwareTest Strategies for conventional software Test Strategies for OO softwareTest Strategies for OO software Validation testingValidation testing System testingSystem testing Debugging Debugging