The Erik Jonsson School of Engineering and Computer Science Software Testing Process Control A Novel...

The Erik Jonsson School of Engineering and Computer Science

Software Testing Process Control

A Novel ApproachJoão W. Cangussu

University of Texas at DallasDepartment of Computer Science

[email protected]/~cangussu

Association for Software Engineering Excellence

September 13, 2005


Motivation

➋ Can we achieve a level o control for the STP that is close to what is observed in other engineering disciplines?

❶ Current techniques for the Software Testing Process (STP) are mostly based on guidelines and semi-formal methods.


Goal

Increase the controllability and prediction capabilities of the Software Testing Process by

developing a control mechanism to correct for deviations in the

Testing Process


Testing Process Control Status

✔ Several recommended procedures are available. None is based on formal models. There is a lack of control algorithms.

✔ COCOMO: For cost and effort estimation; no feedback loop.

✔ Reliability models: For the estimation of software reliability; once again, no feedback loop.

So what do Level 5 companies/groups do?

➊ Often chaotic.


The Problem

Software TestProcess

inputparameters

observedquality (Qo)

expectedquality (Qe)

TestManager

error = Qe - Qo

ControlMechanism

set ofsolutions


System Testing

t- timecp1 cp2 cp3 cp4 cp5 cp6 cp7 cp8 cp9

where cpi = check point ir0

approximation

deadlinet0

num

ber

of r

emai

ning

def

ects

rf

schedule set bythe test manager

observed

Estimated


Control Approach

Actual STPwf+wf

+

wf+wf

+ STP State Model

robserved(t)

rexpected(t)

Initial Settings(wf,)

rerror(t)Controller

’

w’f

Test Manager

+

+

wf


Why a State Model?

➊ State Variable approach has been used to successfully model different types of system

✔ Predator/Prey Model (Voltera)

✔ Economic Model (Samuelson)

✔ Warfare Model (Lanchester)

➋ Enables the use of Control Theory techniques

✔ Feedback Control - Friedland, Luenberger

✔ Optimization - Chong, Zak

✔ System Identification - Ljung


Modeling

Deming states:

All models are wrong, some models are

useful.


Assumption I

The rate at which the velocity of the number of remaining errors changes is directly proportional to the net applied effort (en) and inversely proportional to the complexity of the program under test.

rseser cnc

n


Assumption II

rwse fcf )(

for an appropriate bc

c ss

)(

The effective test effort (ef) is proportional to the product of the applied work force and the number of remaining errors.


Assumption III

rer

1

for an appropriate constant .

The error reduction resistance (er) is proportional to the error reduction velocity ( r ) and is inversely proportional to the overall quality of the test phase (). er opposes r.


State Model

d

ccb

c

fF

sr

r

ss

wr

r

1

010

)1(


Analogy

Block

Dashpot

Rigid surface

External force

Xcurrent

Xequilibrium

X: Position

Number of remainingerrors

Spring Force

Effective Test Effort

Software

Mass of the blockSoftware

complexity

Quality of thetest process

Viscosity

Spring

To err isHuman.


Parameter Calibration

➋Good historic data help avoid initial over/under estimation

➊Models are approximations and need to be calibrated

➌Calibration algorithm considerably decreases the effect of initial over/under estimation


Model Parameters

➊ Parameters to Estimate

✔ sc (software complexity)✔ γ (quality of the testing process)

✔ R0

✔ ζ✔ ξ

➋ Automatic Parameter Calibration


Estimating sc

sc

1 2

3n

M1: KLOC

M2: CyclomaticComplexity

M3: Halstead Metric

Mn: InformationFlow Metric

n

ii

n

iiic Ms

1

1

1


Estimating γ

1 2

3n

Q1Q2

Q3Qn

n

ii

n

iii Q

1

1

1

Where Qi is a quality factor


Estimating the α weights

Quality Factor Qi Q1 Q2 … Qn

Q1: test plan adequacy

1 2 … 1/4

Q2: test team experience

1/2 1 … 3

… … … 1 …

Qn: testing tool adequacy/automation

4 1/3 … 1

The eigenvector associated with the largest eigenvalue of this matrix give us the appropriated weights


Estimating R0

λ1 λ2

λ1 λ2

Approximator

R0

EstimatorData from ongoing testing

R0

CorrectionR0

R0c

Turn key solution:

does not depend on historical or “unavailable” data from previous projects

does not depend on initial values ofInternal parameters


Estimating ξ and ζ

)ln(1

)ln(1

Theorem

Mapping

Spectral

:M of seigenvalue

:A of seigenvalue

M2

12

M2

M1

11

M1

2M1

21

12

11

Te

Te

AT

AT

M

AA

A

A


Parametric Control

),(

)()(

maxmax

max

qualityeffort

eTrtTr t

r -

num

ber

ofre

mai

ning

err

ors

t- time

R0

Rf

r(T)

r(T+t)

tt0


Computing Δwf and Δγ

teTrtTr max)()(

bc

f

c

cb

c

f

s

w

s

ss

wAI

12

1

ˆ

ˆ

ˆ

ˆ1

detdet

fwff wwandwhere ˆˆ


Results – Control and Time Prediction


Results – TEX78


Results – Transformer Project


Results – Feedback Application


Results – More Projects


Results – More Projects

21.2 weeks21 weeks

13.6 weeks

13.4 weeks76463518

21.4 weeks

21 weeks13.6

weeks13.4

weeks75859016

21.6 weeks21 weeks

13.6 weeks

13.4 weeks75853514

18 weeks21 weeks10 weeks13.4 weeks55745812

16 weeks21 weeks10 weeks13.4 weeks

5572819

EstimatedActualEstimate

d Actual

90% Defect Reduction

70% Defect ReductionEstimated

R0

Observ-ed

Defect #Week


Convergence of Completion Time Estimates


Convergence of Completion Time Estimates

68.5% required at most 11% of total time to converge

90% required at most 21% of total time to converge


Estimates for R0


Comparing R0 Estimates


Convergence of R0 Estimates


Sensitivity Analysis

➊ Changes in the process’s parameters are more effective at early stages of the process

➋ Under certain conditions, late changes can make the process slow down instead of speed up (Brooks’s law).

➌ Improvements in the quality of the process is better than increases the size of the test team.


Conclusions

❶ The state model is reasonably accurate in modeling the behavior of the STP

➋ The parameter identification technique provides accurate results for model calibration and defect estimation

➌ Static and dynamic analysis has shown consistency of the model with the real world


Future Work

❶ Develop a modular hierarchy of models for the other phases of the SDP and interconnect them

➌ Develop a secure web-based tool to host the techniques

➋ Investigate alternative control techniques

✔ Stochastic control✔ Fuzzy control

The Erik Jonsson School of Engineering and Computer Science Software Testing Process Control A Novel...

Documents

Transcript of The Erik Jonsson School of Engineering and Computer Science Software Testing Process Control A Novel...