Top Challenges in Testing Requirements

T13

Requirements

5/8/2014 1:30:00 PM

Top Challenges in Testing

Requirements

Presented by:

Lloyd Roden

Lloyd Roden Consultancy

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Lloyd Roden

Lloyd Roden Consultancy

With more than twenty-eight years in the software industry, Lloyd Roden has worked as a developer, test analyst, and test manager for many different organizations. Lloyd was a consultant/partner with Grove Consultants for twelve years. In 2011 he created Lloyd Roden Consultancy, an independent UK-based training and consultancy company specializing in software testing. Lloyd’s passion is to enthuse, excite, and inspire people in the area of software testing. He has spoken at conferences worldwide including STAREAST, STARWEST, Better Software, EuroSTAR, AsiaSTAR, and Special Interest Groups in software testing in several countries. In 2004, he won the European Testing Excellence award.

Top challenges in testing requirements Keynote Presentation

Written by Lloyd Roden www.lloydrodenconsultancy.com Version 1_0 © Lloyd Roden

© Lloyd Roden Testing Requirements 131014

testing requirements-1

Top Challenges in Testing Requirements

1

Contents

Introduction Top challenges for testing requirements

Rising to the challenge

2



What is a requirement?

Definition: !  something that is required/needed !  something essential to the existence of something else

why do we have requirements?

to make money

requirement for a new product

to improve

requirement for an existing system

to remain relevant

requirement for changing technology

3

What do requirements look like?

specifications

use cases

user stories

story boards

emails verbal

some forms are better than others 4



Root Cause Analysis shows…

56%

27%

10% 7%

Defects Originate in Project Phase

Requirements Design Code Other

similar statistics from other projects •  58% requirement bugs for large mobile phone company •  73% requirement bugs for large financial company

Source: Binder & Associates

5

Problems that we have with requirements

"   non-existent requirements "   expectations are not fully understood

"   evolutionary requirements "   constantly changing the scope of the project

"   wrong or bad requirements "   wasted time in development and testing

"   vague or ambiguous requirements "   leading to misunderstandings

"   too much detail in the requirements "   lack of creativity

"   too complex requirements "   unable to test effectively

"   too many requirements "   overbearing and frustrating 6



Contents



7

Challenge #1: No requirements

"   can we test with no requirements? "   we do it all the time

"   performance, usability…

"   should we test with no requirements? "   challenge is knowing whether something is right

“I know it when I see it”

“I want something, but I don’t know

what”

“and by the way, could you give me

an estimate”

8



requirement

Challenge #2: Evolutionary requirements

requirement idea

design

code

test

9

The good and bad of evolutionary requirements

"   excellent when customers don’t really know what they want

"   often produces systems that meet customer’s expectations

"   high collaboration

"   documentation can be sparse/non-existent

"   tester’s nightmare "   automation usually not

possible until final iteration

10



Challenge #3: Bad or wrong requirements

some bad products…

•  all started out as “bad” ideas/requirements •  time “wasted” testing them

we need to recognise and challenge any bad/wrong requirements 11

Challenge #4: Vague/ambiguous requirements

which do you consider to be “good” requirements "   the help messages must be meaningful and informative "   the fields provided must all have context sensitive help "   the system must be user-friendly "   the system must be reliable "   navigation must be consistent across all applications "   the system updates must be fast "   the website must sustain 5000 simultaneous users "   the mobile phone application must be secure "   developers must be able to fix a bug within 2 days

why are some requirements better than others?

12



What is wrong with these requirements?

"   requirement 1

"   requirement 2

online orders can be made at anytime, but will be processed on the next working day. Payment must be made using PayPal or standard credit/debit cards. Orders will be dispatched within 2 days. Customers will be notified via email if a partial order has been sent and can return their order within 28 days and obtain a full refund.

working day standard within

partial within full

A computer program plays chess with one user at a time. It displays the board and pieces on the screen. Moves are made by dragging the pieces

13

Challenge #5: Too much detail in the requirement

"   is there such a thing of having too much detail in requirements?

"   why are too much detail in requirements a challenge? "   it often prevents creativity in development and test "   mistakes are made due to a lack of questioning "   too much detail could lead to conflicting

requirements

14



Challenge #6: Complex requirements

and we call this progress…

1990 2013

1990 2013

1990 2013

“yo, I am outside

your door. lol

15

Challenge complexity at every opportunity

"   simplicity seen as weak and uninteresting "   who wants a “basic mobile phone?”

"   complex is seen as good "   I don’t understand this,

so it must be really good (everyone else understands)

"   $1m pen

suggestion: challenge requirements and design documents at every opportunity to see whether

complexity is needed 16



Features and functions used

Jim Johnson XP2002 Standish Study Group

Features and Functions Used16%

13%

7%

19%

45%Sometimes Used

Often Used

Always Used

Rarely Used

Never Used

Features and Functions Used

20%

64%

16%

Often and Alw aysUsed

Rarely or Never Used

Sometimes

this means we have driven up complexity by putting in things

that are not required

17

”.. If there is too little demand on them, people are bored. If there is too much for them to handle, they get anxious. Flow occurs in that delicate zone between boredom and anxiety.”

Perf

orm

ance

Arousal level

Flow

Strained concentration

Boredom

Distracted

Csikszentmihalyi

Challenge #7: Too many requirements

low medium high

low

high

m

ediu

m

Stress

Laid back

Anxious

Panic/Anger/Fear

Optimum Performance

18



MoSCoW

"   Must "   describes a requirement that must be satisfied in the final

solution for the solution to be considered a success

"   Should "   represents a high-priority requirement that should be in the

solution if possible. It is often a critical requirement but one which can be satisfied in other ways if necessary

"   Could "   describes a requirement which is considered desirable but

not necessary. This will be included if time and resources permit

"   Wont "   represents a requirement that stakeholders have agreed will

not be implemented this time but may be considered for the future 19

Contents



20



Choose the most appropriate lifecycle

Integration Testing

Acceptance Testing

System Testing

Component Testing Code

Design Specification

System Specification

Business Requirements

sequential iterative agile

•  detailed requirements •  complex requirements •  vague requirements

•  no requirements •  evolutionary requirements •  vague requirements

•  too many requirements •  complex requirements •  vague requirements

there is no lifecycle applicable for bad/wrong requirements 21

"   an agile development approach to component testing

Use Test-Driven Development

pick a req.

yes no

need more tests for requirement?

write enough code

to pass the test

write component test

automate test using a tool

22



Use test design techniques as a review technique

"   example requirement:

by reading this requirement we may have a few questions, or we might think we understand what is required. However…

A thermostat displays the temperature of a greenhouse from 15oC to 35oC. If the temperature falls below 20oC then “too cold”

appears on the display and a blue light flashes. It the temperature exceeds 30oC then “too hot” appears and a red light

flashes.

23

Review using test case design techniques

"   sharpens our understanding "   generates tests "   finds more defects with requirements Tests for our example: Temperature Expected result 17oC, 15oC, Too Cold, blue light flash 20oC, 25oC, 30oC No display, no light? 32oC, 35oC Too hot: red light flash 10oC, 40oC ???

2 lights or 1 light?

whole degrees?

up to and including…

where is the temperature taken…isolated or whole room? 24



Produce some requirement guidelines

do not use pronouns do not use unqualified adjectives and adverbs do not use the word “etc” is the requirement is testable?

do we have the necessary expertise? are the requirements prioritised?

does the requirement need breaking down?

is the requirement clear and understood?

25

Use walkthroughs to clarify and understand

"   walkthroughs are a powerful static testing technique "   to educate, aid understanding and find bugs in documents

"   the power and full potential of the walkthrough is often not recognized or utilized "   watch for what is said and how it is said "   do not underestimate body language "   resist peer pressure and challenge when

you don’t understand

26



Use the Quality Attribute Template

Concept Test Scale Plan Must Now Best completeness compare

menu and help items

percentage of menu

items in the help

100% 90% 40% 100%

ease of access

sample 25 features

average number of keystrokes

to find a feature

3 7 12 2

accuracy

collect data each

month

number of reported defects in the help

<5 20 50? 0

Usa

bilit

y O

n-lin

e he

lp

high level

attribute

sub attribute

descriptio

n

of attri

bute how

is it to be

measured what

is being

measured measure

to be

achieved worse

measure

acceptable curre

nt

measure ultim

ate

measure

particularly useful for vague non-functional requirements 27

Perform Exploratory Testing

"   it is not obvious what the next test should be

"   we want to go beyond the obvious tests

"   we want to assess the product quickly

"   we want to explore areas of the system that are unclear

"   creative skills in testing are being encouraged

"   we don’t know all the detail of the requirements

ET is useful when…

28



Summary

"   various forms of requirements exist

"   there are many challenges that we are often faced with when testing requirements

"   testing must rise to these challenges by using a variety of techniques, methods and tools in order to be as effective as we can in testing the requirements

29

Top Challenges in Testing Requirements

Technology

Transcript of Top Challenges in Testing Requirements