Sunita Bodigutla

04/10/2304/10/23 11

Testing: A RoadmapTesting: A Roadmap

Mary Jean HarroldMary Jean Harrold

Sunita BodigutlaSunita Bodigutla

CS 589 Fall 2003CS 589 Fall 2003

04/10/2304/10/23 22

DefinitionDefinition

A verification A verification method that applies method that applies a controlled set of a controlled set of conditions and conditions and stimuli for the stimuli for the purpose of finding purpose of finding errors.errors.

The paper says:The paper says: It’s a process that is It’s a process that is

performed to support performed to support quality assurancequality assurance

04/10/2304/10/23 33

Quality AssuranceQuality Assurance

ISO 9000:2000 ISO 9000:2000 "Part of quality management focused on "Part of quality management focused on

providing confidence that quality requirements providing confidence that quality requirements will be fulfilled“will be fulfilled“

It is a planned program consisting of: all actions It is a planned program consisting of: all actions necessary to provide adequate confidence that necessary to provide adequate confidence that an item or product an item or product conformsconforms to established to established technical requirements; and, activities designed technical requirements; and, activities designed to evaluate the to evaluate the processprocess by which products are by which products are developed or manufactured.developed or manufactured.

A product does what it is expected to.A product does what it is expected to.

04/10/2304/10/23 44

TestingTesting

Designing test casesDesigning test cases executing the software with those executing the software with those

test casestest cases examining the results produced by examining the results produced by

those executionsthose executions

04/10/2304/10/23 55

ExpensiveExpensive

Testing accounts Testing accounts for for 50% 50% of the cost of the cost of software of software developmentdevelopment

Higher for critical Higher for critical systemssystems

Efficient ways to Efficient ways to perform testing to perform testing to control the costcontrol the cost

04/10/2304/10/23 66

What this paper is aboutWhat this paper is about

A report written in 2000A report written in 2000 assesses the state of the art in assesses the state of the art in

software testingsoftware testing outlines future directions in software outlines future directions in software

testingtesting gives pointers to software testing gives pointers to software testing

resourcesresources

04/10/2304/10/23 77

Testing Componentbased Systems

Testing based on Precode artifacts

Demonstrating effectiveness

of testing techniques

Testing Evolving Software

creating effective testing processes

Other testing approaches

Using testing artifacts

Methods and Tools

Empirical Studies

Practical Testing

Methods, Tools, and Processes

forDevelopment of

High-QualitySoftware

Fundamental Research

Status of Testing

Methods,Tools,and

Processes

fig 1. Software Testing Roadmap

04/10/2304/10/23 88









Methods and Tools

Empirical Studies

Practical Testing


forDevelopment of



Status of Testing

Methods,Tools,and

Processes

fig 2

04/10/2304/10/23 99

Component Based SystemsComponent Based Systems

Increased size and Increased size and complexity of complexity of software systems software systems has led to has led to component based component based systemssystems

Component Component Developer vs. Developer vs. Component userComponent user

04/10/2304/10/23 1010

Component UserComponent User

Component

ReusableService

Semantics

Repository

System

System Under Assemble

Select

Composible

Executable

Understand

Assembly

Compile

fig 3

04/10/2304/10/23 1111

Component Based Systems Component Based Systems (contd.)(contd.)

COTS COTS : Commercial off the shelf software: Commercial off the shelf software e.g. Microsoft Active X, MFC, Java Applets e.g. Microsoft Active X, MFC, Java Applets

etcetc Why COTS-Why COTS-

saves moneysaves money saves timesaves time component reusecomponent reuse standardizationstandardization simplified designsimplified design

04/10/2304/10/23 1212

Problems with Testing COTSProblems with Testing COTS

Unavailability of source codeUnavailability of source code Technical HeterogeneityTechnical Heterogeneity Enterprise HeterogeneityEnterprise Heterogeneity Dynamic evolutionDynamic evolution Component Trust ProblemComponent Trust Problem

04/10/2304/10/23 1313

Component Based SoftwareComponent Based Software

Techniques for testing by Component Techniques for testing by Component provider- a number of themprovider- a number of them

Techniques available to component Techniques available to component user-very fewuser-very few

Theory of Theory of test adequacytest adequacy by by RosenblumRosenblum

04/10/2304/10/23 1414

Test AdequacyTest Adequacy

A test adequacy criterion is a systematic criterion that is used to determine whether a test suite provides an adequate amount of testing for a component under test

C-adequate-for-P for adequate unit testing of a component

C-adequate-on-M for adequate integration testing of a component-based system

fig 4

04/10/2304/10/23 1515

Future DirectionsFuture Directions

Efficient techniques and tools forEfficient techniques and tools for testing aspects like security, dependability and safetytesting aspects like security, dependability and safety representing and computing the types of testing representing and computing the types of testing

information (meta-data) that the component-user needsinformation (meta-data) that the component-user needs Java Beans: BeanInfoJava Beans: BeanInfo

storing the informationstoring the information using the informationusing the information A Framework for component deployment [1] called A Framework for component deployment [1] called

‘component deployment testing (CDT) framework’‘component deployment testing (CDT) framework’ component runtime introspectioncomponent runtime introspection reflection API of Javareflection API of Java

04/10/2304/10/23 1616









Methods and Tools

Empirical Studies

Practical Testing


forDevelopment of



Status of Testing

Methods,Tools,and

Processes

fig 5

04/10/2304/10/23 1717

Integration test strategies and architectural Integration test strategies and architectural stylesstyles

fig 6

04/10/2304/10/23 1818

Fault type and Integration Test StrategyFault type and Integration Test Strategy

fig 7

04/10/2304/10/23 1919

Testing based on Architecture Testing based on Architecture specificationsspecifications

Formal notation for architecture specification Formal notation for architecture specification ‘‘CHAMCHAM’ : Chemical Abstract Machine.’ : Chemical Abstract Machine. Architecture: a set of components (molecules)Architecture: a set of components (molecules) states and interactions governed by rules states and interactions governed by rules

(reactions)(reactions) CHAM is specified by defining molecules m1,m2,…, CHAM is specified by defining molecules m1,m2,…,

solutions s1, s2,….. and transformation rules T, T’,solutions s1, s2,….. and transformation rules T, T’,…..…..

Solutions are multisets of molecules interpreted as Solutions are multisets of molecules interpreted as defining the states of CHAMdefining the states of CHAM

Transformation rules describe how the system Transformation rules describe how the system dynamically evolves through reaction stepsdynamically evolves through reaction steps

04/10/2304/10/23 2020

Testing based on Architecture specifications Testing based on Architecture specifications (contd.)(contd.)

[4] Using the formal description of SA, a transition [4] Using the formal description of SA, a transition graph depicting all possible behaviors of the graph depicting all possible behaviors of the system is arrived at.system is arrived at.

A suitable set of reduced graphs are identified, A suitable set of reduced graphs are identified, each subgraph specific architectural properties of each subgraph specific architectural properties of the system.the system.

For each subgraph, test cases are derived to For each subgraph, test cases are derived to cover all the transitions in it.cover all the transitions in it.

04/10/2304/10/23 2121


Techniques for Techniques for automatic generation of test cases from automatic generation of test cases from

formal specification of software formal specification of software architecturearchitecture

evaluating software architectures for evaluating software architectures for testabilitytestability

04/10/2304/10/23 2222









Methods and Tools

Empirical Studies

Practical Testing


forDevelopment of



Status of Testing

Methods,Tools,and

Processes

fig 8

04/10/2304/10/23 2323

Testing Evolving SoftwareTesting Evolving Software

Regression Testing: Regression Testing: ensuring that no ensuring that no new errors are introduced into new errors are introduced into previously tested codepreviously tested code

ExpensiveExpensive Selective Regression Testing: which Selective Regression Testing: which

subset of a test suite must be rerun subset of a test suite must be rerun to test a new version of the systemto test a new version of the system

04/10/2304/10/23 2424


Techniques toTechniques to assist in managing the test suitesassist in managing the test suites reduce the size of test suitesreduce the size of test suites prioritize test cases prioritize test cases

04/10/2304/10/23 2525









Methods and Tools

Empirical Studies

Practical Testing


forDevelopment of



Status of Testing

Methods,Tools,and

Processes

fig 9

04/10/2304/10/23 2626

Test CriteriaTest Criteria

Effective for certain classes of faultsEffective for certain classes of faults data flowdata flow complex boolean expressions: Modified complex boolean expressions: Modified

Condition/decision coverageCondition/decision coverage

04/10/2304/10/23 2727

Modified Condition/Decision CoverageModified Condition/Decision Coverage

Definition:Definition: Every point of entry and exit in the Every point of entry and exit in the

program has been invoked atleast onceprogram has been invoked atleast once every condition in a decision in the every condition in a decision in the

program has taken on all possible program has taken on all possible outcomes atleast onceoutcomes atleast once

each condition has been shown to each condition has been shown to independently affect the decisions independently affect the decisions outcomeoutcome

04/10/2304/10/23 2828

Modified Condition/Decision Modified Condition/Decision Coverage (contd.)Coverage (contd.)

Simple exampleSimple example AA and and BB Extension of truth Extension of truth

table to pairs tabletable to pairs table The test set { (T T),The test set { (T T),

(T F), (F T)} is (T F), (F T)} is adequate for MC/DC adequate for MC/DC coveragecoverage

NumbNumberer ABAB ResulResul

ttAA BB

11 T TT T TT 33 22

22 T FT F FF 11

33 F TF T FF 11

44 F FF F FF

fig 10

04/10/2304/10/23 2929

Future directionsFuture directions

Find ways Find ways to determine interaction between the to determine interaction between the

various test- selection criteria and find various test- selection criteria and find ways to combine them to perform more ways to combine them to perform more effective testingeffective testing

04/10/2304/10/23 3030









Methods and Tools

Empirical Studies

Practical Testing


forDevelopment of



Status of Testing

Methods,Tools,and

Processes

fig 11

04/10/2304/10/23 3131

Effective Processes for Effective Processes for TestingTesting

Process used for planning and implementing Process used for planning and implementing itit

Test plan Test plan Perpetual testingPerpetual testing Microsoft uses a different modelMicrosoft uses a different model

synchronize and stabilizesynchronize and stabilize Future directionsFuture directions

validate the existing modelsvalidate the existing models integration of various quality techniques and toolsintegration of various quality techniques and tools

04/10/2304/10/23 3232

Using Testing ArtifactsUsing Testing Artifacts

Execution TracesExecution Traces Information about whichInformation about which

statements were executedstatements were executed paths were executedpaths were executed values particular variables acquiredvalues particular variables acquired

Artifacts can be stored for use in retesting the Artifacts can be stored for use in retesting the software after it is modifiedsoftware after it is modified dynamic program slicesdynamic program slices heuristic approachesheuristic approaches List of invariantsList of invariants Regression TestingRegression Testing Visualization techniqueVisualization technique

04/10/2304/10/23 3333

Other topicsOther topics

Methods and ToolsMethods and Tools ScalabilityScalability Computational tradeoffsComputational tradeoffs Automatic generationAutomatic generation

Empirical StudiesEmpirical Studies Testing ResourcesTesting Resources

04/10/2304/10/23 3434

ConclusionConclusion

OverviewOverview List of techniquesList of techniques No comparison No comparison Future directionsFuture directions Testing is a wide areaTesting is a wide area Dynamic fieldDynamic field

04/10/2304/10/23 3535

[1] Antonia Bertolino, Andrea Polini, “A Framework for Component Deployment [1] Antonia Bertolino, Andrea Polini, “A Framework for Component Deployment Testing”Testing”

[2] A.Orso, M.J.Harrold and D.Rosenblum, “Component Metadata for Software [2] A.Orso, M.J.Harrold and D.Rosenblum, “Component Metadata for Software Engineering Tasks”, in W.Emmerich and S.Tai(Eds) EDO 2000, LNCS 1999, pp. 129-Engineering Tasks”, in W.Emmerich and S.Tai(Eds) EDO 2000, LNCS 1999, pp. 129-144144

[3] J.A.Stafford and A.L.Wolf, “Annotating components to support component based [3] J.A.Stafford and A.L.Wolf, “Annotating components to support component based static analysis of software systems “, static analysis of software systems “, Proc. the Grace Hopper Celeb. of women in Proc. the Grace Hopper Celeb. of women in computing 2001computing 2001

[4] Antonia Bertolino, Paola Inverardi, Henry Muccini, Andrea Rosetti, “An Approach [4] Antonia Bertolino, Paola Inverardi, Henry Muccini, Andrea Rosetti, “An Approach to Integration Testing based on Architectural Descriptions”to Integration Testing based on Architectural Descriptions”

References

04/10/2304/10/23 3636

fig 12. Sequential Multiphase Compiler Architecture

04/10/2304/10/23 3737

fig 13.Transition Graph for the Sequential Compile

04/10/2304/10/23 3838

fig 14. Concurrent Reduced Graph

04/10/2304/10/23 3939

Sunita Bodigutla

Documents

Transcript of Sunita Bodigutla