Tor Stålhane

Requirements Specification and TestingRequirements testability

Testability definitionAccording to ISO 9126, testability is defined

as:

Testability: The capability of the software product to enable modified software to be validated.

 

NOTE - Values of this sub-characteristic may be altered by the modifications under consideration.

Testability concerns

Testability touches upon two areas of concern:• How easy is it to test the implementation?• How test-friendly is the requirement?

These two concerns are not independent and need to be considered together. We will first look at it from the requirements side.

Testability Three basic ways to check that we have achieved our

goals:

• Executing a test. Give input, observe and check output. A test can be a

– Black box test

– White box test

– Grey box test

• Run experiments

• Inspect the code and other artifacts

Usually, we will include all of these activities in the term testing

When to use what

The diagram on the next slide is a high level overview of when to use

• T – Tests. Input / output. Involves the computer system and peripherals.

• E – Experiments. Input / output but involves also the users.

• I – Inspections. Evaluation based on documents.

TDT 4242

Concrete requirements from high level goals

E T

EE

I I

E

ET

T

Testability

In order to be testable, a requirement needs to be stated in a precise way. For some requirements this is in place right from the start:When the ACC system is turned on, the

“Active” light on the dashboard shall be turned on.

In other cases we need to change a requirement to get a testable version. The system shall be easy to use.

Testability challenges - 1

Some requirements are more difficult to test than others. Problems might rise due to:

• Volume of tests needed, e.g. response time or storage capacity.

• Type of event to be tested, e.g. error handling or safety mechanisms.

• The required state of the system before testing, e.g. a rare failure state or a certain transaction history.

Testability challenges – 2 We can test the requirements at any level. The formulation of the test will depend on the level

Making a requirement testable – 1

One way to make requirements testable is the “Design by Objective” method introduced by Tom Gilb.

The method is simple in principle but is in some cases difficult to use. There are two problems

• The resulting tests can in some be rather extensive and thus quite costly.

• The method requires access to the system’s end users.

Making a requirement testable – 2

1. What do you mean by <requirement>? This will give us either (a) a testable requirement or (b) a set of testable and non-testable sub-requirements.

2. In case (a) we are finished. In case (b) we will repeat question 1 for each non-testable sub-requirement

Making a requirement testable – 3

Requirement: Reverse thrust may only be used, when the airplane is landed.

The important questions are

• “How do you define landed?”

• Who should you ask – e.g. pilots, airplane construction engineers, or airplane designers?

Requirements for testability – 1

First and foremost:

The customer needs to know what he wants and why he wants it. In some cases it is easier to test if the user actually has achieved his goal than to test that the system implements the requirement.

Unfortunately, the “why”-part is usually not stated as part of a requirement.

Requirements for testability – 2

Each requirement needs to be• Correct, i.e. without errors• Complete, i.e. has all possible situations

been covered? • Consistent, i.e. not in disagreement with

other requirements.• Clear, i.e. stated in a way that is easy to

read and understand – e.g. using a commonly known notation.

Requirements for testability – 3 Each requirement needs to be• Relevant, i.e. pertinent to the system’s

purpose and at the right level of restrictiveness.

• Feasible, i.e. possible to realize. If it is difficult to implement, is might also be difficult to test.

• Traceable, i.e. it must be possible to relate it to one or more– Software components– Process steps

Completeness

All possible situations must be covered.“If X then….”, “If Y then….” Must also consider

what will happen “If neither X nor Y…”

Automatic door opener – what is missing?If the door is closed and a person is detected

then send signal Open_Door. If no person is detected after 10 sec., send signal Close_Door.

Consistency

Consistency is a challenge since we, at least in the general case, need a complete overview of all requirements.

In most cases, we can make do with checking all requirements that are related to the same event, function or parameter.

Clear – 1 This is mainly a question of representation

such as choice of

• Diagram notation

• Description language

• Level of details

Who shall understand the requirement?

• Customers

• Developers, including hired-in consultants

• Testers

Clear – 2

Simple example:

Print the “accounts ledger” for all accounts

This requirement is perfectly clear for developers who are working in the banking business.

Other developers might experience some problems.

Relevant Two questions are important:

• Do we really need this requirement?

• Is it at the right level of strictness – i.e. not too strict and not too lax.

Only the second question is important for the tester.

• Too strict means more work for the developers

• Too lax means more work for the tester.

Feasible This question is really related to the contract but

we should also consider it here – can we really do this?

Testers can contribute to the feasibility question by asking how it should be tested. This will help / force everybody involved to make the requirement more clear and thus improve on the requirement.

Requirements that are difficult to tests are also usually difficult to implement – mainly because they are badly defined.

Some sound advice

The following set of advices on requirements and testability are quoted from Ludwig Consulting Services, LLC.

They are not a definition and not “the final words” on requirements testability. Instead, they should be used as a checklist.

That one of the following rules are not obeyed does not mean that the requirement is wrong. It should, however, be reviewed for potential problems.

Modifying PhrasesWords and phrases that include: • as appropriate• if practical• as required• to the extent necessary / practical. Their meaning • is subject to interpretation • make the requirement optional Phrases like "at a minimum" only ensure the minimum,

while "shall be considered" only requires the contractor to think about it.

Vague WordsVague words inject confusion. Examples of frequently

used vague verbs are: • manage • track• handle • flag

Information systems receive, store, calculate, report, and transmit data. Use words that express what the system must do.

Requirement: The system shall process ABC data to the extent necessary to store it in an appropriate form for future access.

Correction: The system shall edit ABC data.

Pronouns With No Reference

Example: It shall be displayed.

When this occurs, the writer is usually relying on a nearby requirement in the requirements document for the meaning of "it."

As requirements are assigned for implementation, they are often reordered and regrouped, and the defining requirement is no longer nearby.

Passive Voice

Requirements should be written in active voice, which clearly shows X does or provides Y.

Passive voice: Z shall be calculated.

Active voice: the system shall calculate Z.

Negative RequirementsEverything outside the system is what the system

does not do. Testing would have to continue forever to prove

that the system does not do something. State what the system does. Substitute an active

verb that expresses what the system must do. • Change "the system shall not allow X," to "the

system shall prevent Y." • Use the prefix "un," such as: The system shall

reject unauthorized users.

Assumptions and Comparisons – 1

The requirement "the system shall increase throughput by 15%" sounds testable, but isn't.

The assumption is "over current system throughput." By comparing to another system, the meaning of the requirement changes when the other system changes

Assumptions and Comparisons – 2 An example, sometimes found in requests for

proposals, is: "The system shall address the future needs of

users." The writer is probably thinking ahead to after the

contract is awarded. The requirement is meaningless because whenever it is read, it will point to the future.

A requirement on change management included in the project management processes, would make more sense than making it a requirement for the system.

Indefinite Pronouns

• All• Another• Any• Anybody• Anything• Each• Either• Every

• Everybody• Everyone• Everything• Few• Many • Most• Much

Indefinite pronouns are “stand in” for unnamed people or things, which makes their meaning subject to interpretation. Some of these may find their way into requirements:

The implementation

We will shortly look at four concerns related to implementation and testability:

• Autonomy of the system under test• Observability of the testing progress• Re-test efficiency • Test restartability

Autonomy – 1

How many other systems are needed to test this requirement?

It is best if it can be tested using only the SUT and the autonomy and testability is successively reduced as the number of other, necessary systems increase.

If the other systems needed are difficult to include at the present we will have to write more or less complex stubs.

Example – 1 Requirement: “If the door is closed and a

person is detected then send signal Open_Door”

• Sensors and actuators can be tested in the lab.

• The system with a simulated actuator: simulate a “person detected” signal on the sensor and check if a Open_Door signal is sent to the actuator.

Example – 2

We can build a complete system – door, sensor, door motor and software system and test by

• Letting persons approach the sensor• Check if the door opens

– Early enough– Fast enough

Observability How easy is it to observe the • Progress of the test execution?

This is important for tests that do not produce output – e.g. the requirement is only concerned with an internal state change or update of a database.

• Results of the test? Important for tests where the output is dependent on an internal state or database content.

Re-test effiencey

Retest efficiency is concerned with how easy it is to perform the “test – check – change – re-test” cycle.

This includes • Observability – observe the test result• Traceability – identify all tests related to the

change

Test restartability

This is mostly a question of check points in the code. How easy is it to

• Stop the test temporarily• Study current state and output• Start the test again from

– The point where it was stopped– Start

Final comments

That a requirement is testable does not necessarily mean that it is easy to test.

In order to have testable requirements it is important that

• The testers are involved right from the start of the project. It is difficult to add testability later.

• The tests are an integrated part of the requirement