ISO 29119: The New International Software

Testing StandardStuart Reid,

Testing Solutions Group,UK

Europe’s Premier Software Testing EventWorld Forum Convention Centre, The Hague, Netherlands

WWW.QUALTECHCONFERENCES.COM

“The Future of Software Testing”

ISO/IEC 29119:

The New International Software Testing Standard

Stuart Reid

sreid@testing-solutions.com

Testing Solutions Groupwww.testing-solutions.com

EuroSTAR 2008 Workshop

Workshop Goals

• Introduce ISO 29119

• Delegate groups to:

– Review the overall scope & structure

– Review and debug the test processes

– Brainstorm the test techniques (part 4)

• What techniques do we want in it?

– Define the ongoing review process

Who Needs Standards?

What are standards?

• “Guideline documentation that reflects agreements on products, practices, or operations by nationally or internationally recognized industrial, professional, trade associations or governmental bodies”

- ISO

– Guidelines documents as they are not compulsory unless mandated by an individual or an organization

– Agreements because they often reflect a certain level of consensus

Motivation for ISO 29119

• Lacking in current standards:

– Organizational Testing not covered

• e.g. Test Policy and Organizational Test Strategy

– no Project Test Management

– BS 7925 only covers unit testing

– common functional techniques missing

– poor coverage of non-functional testing

• Conflicts in definitions, processes & procedures

– plethora of standards to be replaced by one

• e.g. IEEE 829, IEEE 1008, BS 7925-1/-2, IEEE 1028

– practitioners do not know which standard to follow

Documentation

Part 3

TestingTechniques

Part 4

Processes

Part 2

Concepts & Vocabulary

Part 1

BS 7925-2 IEEE 829IEEE 1008

BS 7925-1

BS 7925-2

ISO 29119 – Scope & Structure

May 08

May 09

May 10

May 11

May 12

…

ISO 29119 - Timeline

Working Draft (WD)Committee Draft (CD)Final Committee Draft (FCD)Final Draft International Standard (FDIS)Final International Standard (FIS)

Parts 2 & 3

Parts 1 & 4

WD

CD

FCD

FDIS

FIS

WD

CD

FCD

FDIS

FIS

Organizational Test Strategy Process

Project Test Management Process

Test Level Process

Organizational Test Policy Process

Test Execution Cycle

Organizational Test Policy

Organizational Test Strategy

Strategise, Plan and Manage Test Project

Strategise, Plan and Manage Test Level

Design & Execute Tests

Manage Test Environment

Report Defects

Organizational Test Processes

Organizational Test Strategy Process

Create/Maintain Test Strategy

Test Policy

IT ProjectManagement Policy

Test Strategy

Organizational Test Policy Process

Create/Maintain Test PolicyIT Policy

Management PolicyTest Policy

Test Policy Test Policy Updates

Project Test Management Process

OrganizationalTest Strategy

Comments on Organizational Test Strategy usage

Manage Conformance

with Test Policy(TP4)

Organizational Test Policy

[Test Policy not working effectively]

[Test Policy working effectively]

Create/Update Test Policy

(TP1)

Organizational Test PolicyProcess

Organizational Test Policy Issue Log

Organizational Test Policy Update Log

Gain Consensus

on Test Policy(TP2)

Approved Organizational

Test Policy

Publish Test Policy

(TP3)

Review Test

Policy(TP5)

[major revision required]

•Views of primary stakeholders;•Knowledge of the current test practice within the organization;and, where available:• Organization’s

Mission Statement;

• IT Policy;• IT Project

Management Policy;

• Quality Policy;• Current Test

Policy;• Test Policy

Update Log. [Suitable period has elapsed:• since Test Policy was published; or• since Test Policy was last reviewed; or• major organizational change]

Comments onOrganizational

Test Policy usage

Draft OrganizationalTest Policy

[Test Policy changes not required]

Project Test Management Process

Comments onOrganizationalTest Strategy usage

ProjectTestCompletionReport

Create/Maintain Project Test

Plan(TM1)

Monitor and Control(TM2)

ReportTest Project Completion

(TM3)

ProjectTest Plan

ProjectTestStatusReports

Project Test Plan Updates

Project Test Management Process

TestStrategy

Organizational Test Strategy Process

Project Management Plan,Product Documentation,Software Development Plan, Regulatory Standards

Level Test Plan,Level Test Status Reports

Project Test Plan

Level Test CompletionReport

Control Directives

Test Level Process

• Organizational Test Policy• Organizational Test Strategy• Regulatory Standards

• Project Management Plan• Applicable product documentation

(e.g. System Requirements, Test Item Specifications)• Software Development Plan• Project and Product Risks• Project Test Plan Updates

Create/Maintain Project Test Plan

Plan Creation of

Project Test Plan(TM1.2) Identify &

Analyze Risks

(TM1.3)

Design Project Test

Strategy(TM1.5)Determine

Staffing and

Scheduling(TM1.6)Document

Project Test Plan(TM1.7)

Schedule, Staffing Profile

Project Test Strategy

Analyzed Risks

Project scope

Identify Risk

Mitigation Approaches

(TM1.4)

Gain Consensus on Project Test Plan(TM1.8)

Approved Project Test Plan

Draft Project Test Plan

Project Test Plan Publish

Project Test Plan(TM1.9)

Understand Project Context(TM1.1)

Mitigation Approaches

Fundamental Test Process

ProjectTestPlan

Test Level

Status Report

Level Test CompletionReports

Control Directives

Project Test Management Process

ReportAnomalies

(TL6)

ReportTest Level Completion

(TL8)

Level Test Plan

TestSpecification

TestResults

[Issuenoticed]

[No issues noticed]

Anomaly Report

Test Level Process

Set-up & Maintain TestEnvironment

(TL4)

Plan Testing (TL1)

RunTests(TL5)

Test Environment Readiness Report

Specific TestEnv’t Requirements

Level Test CompletionReport

Design & BuildTests(TL3)

Test Execution Cycle

Close Down Testing(TL7)

Control Directives

Monitor and Control(TL2)

Test Process Info

Asset Info

Measures

Level Test Plan Updates

Project Management Plan,Product Documentation,Software Development Plan, Regulatory Standards

Test Activity Descriptions Example – Design & build tests

• Each testing activity is described using a standard template (following ISO 24774), for example:

Purpose

The purpose of Design and Build Tests is to design and document test cases that will be executed during testing.

Entry Criteria

•the Test Plan has been agreed

•the Test Item Specification(s) is available

Inputs

•Test Plan

•Test Item Specification(s)

•System Requirements

Roles

•Test Designer

Tasks

1.Analyse the test item specification(s) to identify specification items;

2.Determine the test conditions by identifying which specification items to test using the selection criteria from the Test Plan;

3.Derive test coverage items to be exercised by applying the test case design technique(s) to the test conditions;

4.Derive test cases by selecting input values to exercise the test coverage items and corresponding expected results;

5.Create a test set by rationalising the test cases (i.e. removing redundancy, duplication, etc.);

6.Derive a test script by ordering the test cases within the test set using their pre- and post-conditions;

7.Identify and document any specific test environment requirements that are not already included in the Test Plan.

Outputs

•Test Specification

•Specific Test Environment Requirements

Outcomes

•the Test Specification has been documented.

Exit Criteria

•the Test Specification has been agreed.

Part 4: Techniques

• Test case design techniques

– Static testing techniques

• Reviews - inspections & walkthroughs, etc.

– Dynamic testing techniques

• Black-box, white-box, etc.

– Non-functional testing techniques

• Security, performance, usability, etc.

• Test measurement techniques (e.g. coverage)

• Appendices

– Examples of each technique, at each level of testing

– Test technique effectiveness

– Mappings to existing standards

Your turn....

• Session 1– Choose the 1st topic you wish to ‘review’

– Create a group around the topic

– Review the topic

– Report back to the other groups

• Break

• Session 2– Choose your 2nd topic

• it can be the same as your 1st topic

– Create a group around the topic

– Review the topic

– Report back

Topics

1. Review the overall scope & structure

2. Four-tier process model

3. Organizational testing (policy & strategy)

4. Project Test Management

5. Creating (and Maintaining) a Test Plan

6. Fundamental Test Process

7. Test techniques (part 4)

8. The ongoing review process

Conclusions

• We will consider the workshop feedback

• Do you want to be involved?– Join the working group

• through your national standards body

• 6 day meetings, every 6 months

• contribute between meetings

– Contribute materials

– Review drafts

– May 2008 meeting - Berlin• 18 participants representing 12 countries

– November 2008 (Seoul) – last week...

– May 2009 (India)

For more information

• sreid@testing-solutions.com – if interested in trialling the standard on a project, reviewing drafts or writing examples

• http://www.jtc1-sc7.org/ – access to official documents released by WG 26