Test Processes for Information Systems

TPRO

1 Levels of Test Technology

Test Process according to IEEE-829

Testing as a Separate Subproject

Requirement Solicitation

Test Phases of the V-Model

Test Objects of the V-Model

Life Cycle Test Process by W. Perry

Dual Track Process Model

Test as a parallel Project by H. Sneed

A Standard Test Process

ANECON System Test Process

Sneed System Test Process

System Testing Cycle

Defining Test Levels

Establishing a Testing Profile

Planning the System Test

Defining Test Requirements

Allocating Test Requirements

Developing Test Scenarios

Implementing the Test

TPI Test Process Evaluation Model

Key Areas of a System Testing Process

Test Process Maturity Levels

Test Process Maturity Categories

Sample Test Process Maturity Matrix

Conclusions for testing Information Systems

2

3

4

5

6

7

8

9

10

14

15

16

17

18

19

20

21

22

23

11

12 25

24

26

TEMA

13

Levels of Testing TechnologyTPRO-1

TestProcess

Test Strategy

Test Methods

Test Procedures

Test Tools

Fromabstract

IncreasingLevel

OfDetail

ToConcrete

TestScripts

TestSelectionMethod

Choice of aTesting Strategy

Railroad Process Model

Data Driven System Test

Selective Test DataGeneration

Assertion Language forTest Data Specification

Assertion-basedTest DataGenerator

TEMA

Test Process according to ANSI/IEEE-829

TEMA TPRO-2

Test-Plan

Test-Design

Test CaseSpecificat.

Test ProcCoding

Test ObjectDelivery

TestLog

TestIncident

Reporting Test-Result

Reporting

T e s t Execution

Testing as a Separate SubprojectTPRO-3

The Main Project

RequirementsSubproject

DevelopmentSubproject Test

Subproject

TEMA

Requirement Solicitation is the Basis of Testing

TPRO-4

System RequirementsFunctional / Non-

Functional

System RequirementsFunctional / Non-

Functional

RequirementCollection

Basin

SystemRequirementDocuments

Hardware Software

TEMA

Test Phases in the V-Modell

TEMA TPRO-5

Components

UnitTest

Coding

DesignIntegration

test

Systemtest

Analysis

AcceptanceRequirements

Test Objects in the V-Modell

TEMA TPRO-6

Code

Req. Specs

SystemDesign

ModuleDesign

Systemtest

Procedures

Integration test

Procedures

Unit testProcedures

System test cases

Integration test cases

Unit test cases

Life Cycle Test Process by W. Perry

SYST TPRO-7

Development

Test

Projektstart

Requirements Design Programming Test Integration Maintainenance

SDLC-Phase nach ISO-12207

Req

uire

men

ts

Requirem

ents

Sys

tem

test

System

Mai

nten

ance

Regression test

Dual Track Process Model

TPRO-8

Requirement Specification

Dev

elop

men

t T

rack

Tes

ting

Tra

ck

Delivered Software System

TEMA

Ties

Introduced by Gelperin & HetzelFounders of the ISQI

Test as a Parallel Project

TEMA TPRO-9

Design Coding Unit Test

SystemTest

TestEvaluation

Test Data &Procedures

Test Casespecification

ProjectPlanning

Analysis

TestDesign

Testplanning

Acceptance

Require-ments

Correct?

N

Y

Development Project

Test Project

A Standard Test Process

TEMA TPRO-10

TestPlanning

TestConception

TestSpec.

TestExecution

TestEvaluation

Test Plan

TestConcept

TestCases

TestResults

TestDocum.

Closure

Init

ANECON System Testing Process

SYST TPRO-11

ReqSpecs

RequirementAnalysis

TestMetrics

TestPlanning

Test Plan TestDesign

Test CaseSpecification

PreliminaryTest Cases

FinalTest Cases

Test DataGenerationSchemas

TestData

TestExecution

TestResults

ExpectedResults

TestEvaluation

TestDocumentation

TEXT

TPLN

TSPC

TDAT

TEXE

TCON

TVAL

Sneed System Test Process (1)TPRO-12.1

RequirementSpecs.

German/EnglishTexts

RequirementAnalysis

Req. BasedTest Cases

Test Plan

InformationSystem

Architecture

UMLDiagrams

DesignAnalysis

Design BasedTest Cases

DESIGN

Base Line Document

Test CaseDatabase

CODING

CodeAnalysis

Code BasedTest Cases

CodeComponent

s

JavaClasses

Test GoalsTest StrategyTest MethodTest ProceduresTest Effort Estimation.................

Test Case IdTest Case PurposeTest Case TriggerRequirement TestedObjects TestedUse Case TestedPre ConditionsPost Conditions.................

TEMA

Sneed System Test Process (2) TPRO-12.2

Test CaseSpecification

DataInterfaces

InformationSystemDesign

Test DataGeneration

Tester

TestData

ValidationReport

TestResult

Evaluation

InformationSystem Test

Scenarios

TestDesign

Test CaseDatabase

Test CaseAnalysis

Test CaseDeficiencies

InformationSystem Test

Requirements

TestEnvironmentConstruction

TestResults

Test

Environment

SystemUnderTest

TestMonitor

Test Coverage

TestEvaluation

PostStates

Pre States

TEMA

System Testing Cycle

TPRO-13

AnalyzeRequirement

PlanTest

ExtractTest Cases

SpecifyTest Cases

DesignTest Scenarios

GenerateTest Data

ConstructTest

Environment

ExecuteTest

EvaluateTest

Soll

Ist

TEMA

DOD Testing Levels

TPRO-14

Software Development Plan

Software Test Plan

Secondary Test Plans, Procedures, Scenarios, and Data

Level 1

ModuleTest

Level 2

UnitTest

Level 3 Level 4 Level 5 Level 6

Bui

ldIn

tegr

atio

n

Sub

syst

em f

unct

iona

l(a

lloca

ted

by

build

)

Sys

tem

Inte

grat

ion

SystemTesting

TEMA

Establishing a Testing ProfileTPRO-15

Level 6

Level

6

Level

5

Level

4

Level

3

Level

2

Level

1

1. Configuration management

2. Organizational strategies

3. External simulator4. Regression testing5. Hierarchal testing6. Random testing7. Software

instrumentation

1. Configuration management

2. Data reviews3. Test tool

definition

Level 4 Level 3

1. Configuration management

2. Regression testing3. System simulation4. Organizational

autonomy5. Incremental

qualification6. Software

instrumentation

Level 5

Level 2 Level 1

TEMA

TPRO-16Planning the System Test

SystemTest-plan

SoftwareDevelopment

plan

SoftwareTestplan

Phase 1. Phase 2.

SystemProcedures

SystemTest

Scenarios

SystemTestData

Test Profile

BuildTest

Plan 1

General UnitTest Plan

FunctionalProcedures

BuildTest

Folder

Test tools,Techniques andmethodologies

ScenarioData

Procedures,Scenarios,

data

Secondaryplanning

documentation

TEMA

TPRO-17

Defining the Test RequirementsSystemTest-plan

SoftwareDevelopment

planProgramming

Practices,Standards and

conventions

Quality guidelines

Configurationmanagement

SoftwareTestplan

Testroadmap

Testspecs.

BuildTest

Plan 1

FunctionalProcedures

SoftwareDesign

SpecificationBuild Test

Folders

BuildProcedures,Scenariosand Data

TestScenariosand Data

TEMA

Construction Rules

Planned Metrics

Allocating the Test ResourcesTPRO-18

SystemTest-planSoftware

Developmentplan

SoftwareTest-plan

Schedules

FunctionalSpecification

Software architecture

Database architecture

_________________________

BuildTest-plan

BuildTest

Folder

FunctionalProceduresAllocatedTo builds

TEMA

Developing Test ScenariosTPRO-19

SoftwareTest-plan

SystemTest-plan

Level 3

TestProcedure

Level 5

TestProcedure

IntegrationTest requirement Test step Test scenario

Test stepTest scenario

Test scenario

Test scenario

Test step

Test step

TEMA

Implementing the TestTPRO-20

Test implementation and scenario

Test analysis

Unit developmentfolder

Test environmentcreation

Test execution

Test planning and development

Test plans

Buildtest

folder

Systemtest

folder

Testprocedure

Testscenario

Testscenario

Testscenario

Testdata

Testdata

Testdata

Test analysis

Test toolingTest execution

Dev

elop

men

t te

stin

g

Test implementation and execution

TEMA

The TPI Test Process Evaluation Model (Koomen & Pol)

TPRO-21TEMA

Key Areas

Maturity levels

Test maturity matrix

Acquired pointsImpovementSuggestions

Key Areas of a System Test ProcessTPRO-22.1TEMA

Key Area Level A Level B Level C Level D

Test Strategy Strategy for single high-level

test

Combined strategy for high-

level tests

Combined strategy for

high-level tests plus low-

lvel tests or evaluation

Combined strategy for

all test and evaluation

levels

Life-cycle model Planning, Specification,

Execution

Planning, Preparation,

Specification, Execution, and

Completion

Moment of

involvement

Completion of test basis Start of test basis Start of requirements

definition

Project initiation

Estimating and

planning

Substantiated estimating and

planning

Statistically substantiated

estimating and planning

Test specification

techniques

Informal techniques Formal techniques

Static test techniques Inspection of test basis Checklists

Metrics Project metrics (product) Project metrics (process) System metrics Organization metrics

(>1 system)

Test tools Planning and control tools Execution and analysis tools Extensive automation of

the test process

Test environment Managed and controlled test

environment

Testing in the most suitable

environment

‘Environment-on-call’

Office environment Adequate and timely office

environment

Key Areas of a System Test ProcessTPRO-22.2TEMA

Key Area Level A Level B Level C Level D

Commitment and motivation

Assignment of budget and time

Testing integrated in project organization

Test-engineering

Test functions and training

Test manager and testers (Formal) Methodical, Technical and Functional Support, Management

Formal internal Quality Assurance

Scope of methodology Project specific Organization generic Organization optimizing, R & D activities

Communication Internal communication Project communication (defects, change control)

Communication within organization about the quality of the test processes

Reporting Defects Progress (status of tests and products), activities (costs and time, milestones), defects with priorities

Risks and recommendations, substantiated with metrics

Recommendations have a Software Process Improvement character

Defect management Internal defect management

Extensive defect management with flexible reporting facilities

Project defect management

Testware management Internal testware management

External management of test basis and test object

Reusable testware Traceability system requirements to test cases

Test process management

Planning and execution Planning, execution, monitoring, and adjusting

Monitoring and adjusting within organization

Evaluation Evaluation techniques Evaluation strategy

Low-level testing Low-level test life cycle (planning, specification and execution)

White-box techniques Low-level test strategy

Test Process Maturity LevelsTPRO-23TEMA

AMinimum Fulfillment, low maturity level (1)

BMedium Fulfillment, average maturity level (2)

CGeneral Fulfillment, high maturity level (3)

DFull Fulfillment, maximum maturity level (4)

Test Process Maturity CategoriesTPRO-24TEMA

1-6 Controlled Test Process is planned and controlled

Test cases are specified, Test results are documented

Errors are recorded. Test progress is reported

7-10 Efficient Test automation, Data generation,

Result validation, Test documentation

11-13 Optimizing Control improvement

Tuning, Calibrating

First structure, then automate (Pol)Automating chaos leads to faster chaos (Graham)

Sample Test Process Maturity MatrixTPRO-25TEMA

Key Area 0 1 2 3 4 5 6 7 8 9 10 11 12 13

Controlled Efficient Optimizing

Test strategy A B C D

Life-cycle model A B

Moment of involvement A B C D

Estimating and planning A B

Test specification teschniques A B

Static test techniques A B

Metrics A B C D

Test Tools A B C

Test environment A B C

Office environment A

Commitment and motivation A B C

Test functions and training A B C

Scope of methodology A B C

Communication A B C

Reporting A B C D

Defect management A B C

Testware management A B C D

Test process management A B C

Evaluation A B

Low-level testing A B C

Scale

TPRO-26

Conclusions for Testing Information Systems

The development project should be accompanied by a

separate test project.

Both development and test should be based on the

requirements specification.

The budget for the test project should be equivalent to that

for the development project.

The test project should proceed methodologically from the

test design to the test execution and evaluation.

The test project results should be subject to a similar quality

assurance as the development results.

1

2

3

4

5

TEMA