Ambiguity Reviews: Building Quality Requirements

T19

Requirements

5/8/2014 3:00:00 PM

Ambiguity Reviews: Building

Quality Requirements

Presented by:

Susan Schanta

Cognizant Technology Solutions

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Susan Schanta

Cognizant Technology Solutions

Susan Schanta has spent twenty years managing large-scale and quality assurance programs in both new ventures and global Fortune 500 companies in the financial, healthcare, and retail domains. Susan has led corporate initiatives in business optimization, onshore-to-offshore transitions, and QA enterprises including SDLC management, automation, and performance. Her expertise in industry-best practices and project execution has helped companies achieve their goals. Susan’s experience includes implementing lifecycle disciplines through change management, sharply reducing budget variances related to estimation techniques, improving performance of tiered applications, introducing new disciplines for UAT execution to achieve improved quality and business workflow processes, and devising calibration methods to measure these achievements.

1

| © 2014, Cognizant 0© 2014, Cognizant | All rights reserved. The information contained herein is subject to change without notice.

Susan Schanta

Process, Quality & Consulting

April, 2014

Ambiguity Reviews Building Quality Requirements

| © 2014, Cognizant 1

Agenda

The Cost of Quality 2

What is an Ambiguity Review? 3

Ambiguity Review Place in the SDLC 11

Ambiguity Review Classification 14

Ambiguity Review Types 20

Ambiguity Review Template 23

Ambiguity Review – Keyword Searches 27

Ambiguity Review – Content Review 30

Ambiguity Review Metrics 35

Ambiguity Review – Benefit to QA 40

Ambiguity Review – Be Nice About It! 42

Appendix 44

2


The Cost of Quality

2

• IRS Tax Systems modernization project spent $3.3b before

canceling (Federal Computer Week March, 2002)

• Time Warner Communications spent $1b on a failed

information system to break into the Residential Telephone

Business (Computerworld May, 1997)

• Sainsbury, a British food retailer wrote off $526m invested in an Automated Supply-Chain Management

System.

• Ford Motor Company spent $400m on a Purchasing

System before abandoning it in 2004

50% of defects are due to requirements problems (Schwaber, 2006)


What is an Ambiguity Review?

3


What is an Ambiguity?

Ambiguity can be defined as/1. Use of words that allow alternative interpretations 2. An unclear, indefinite, or equivocal word, expression,

meaning, etc.

3. The possibility of interpreting an expression in two or more

distinct ways

4. Doubtfulness or uncertainty of meaning or intention: to speak

with ambiguity; an ambiguity of manner. (dictionary.com)


What is an Ambiguity Review?

An Ambiguity Review is a formal review process that focuses

on the identification of ambiguities in the language, structureand logic of a requirement.

− Provides a measure of whether requirements are

quantitative, clear, correct and complete.

− Eliminates requirements defects from being identified in

later phases by building quality into the product. (Bender)

4


What Are the Benefits of Ambiguity Reviews?

Ambiguity Benefits realized�

• Scope Creep is controlled

• Reduced cost of maintenance• Reduced number of change requests• Increased traceability from requirements to test cases

−Defects per requirement

Tangible Results/

Healthcare Insurer

• Reduced Defect Leakage from 35% to 8%

• Introduced BA Style Guide to drive down ambiguities

Fortune 50 Insurance Company #1

• Reduced ambiguities from an avg. 7 per Req. to 2 per Req.

• Introduced BA Style Guide to drive down ambiguities

Fortune 50 Insurance Company #2

• 63% of all defects were traced to Requirement Defects


Business Users Focus on Happy Path

Impact to business users

• Who focus on Happy Path

Results identifyI

• Open ended questions

What are your pain points?

• Requirements lack input/output, alternative flows, error conditions and constraints

• Development makes assumptions to fills in gaps in the

requirements

• Potential Defect leakage to production

Why do an Ambiguity Review?

5


Development Drives Requirements

Impact to Development

• Who drive application behavior based on assumptions

where requirement gaps exist

Results identifyI

• Higher maintenance is a higher percentage of IT budget

− Limits dollars spent on revenue producing software

− Increases redo work

• Change requests trend higher

• Quality Assurance is expected to test quality in



QA Builds In Quality, Right?

Impact to QA

• Who are expected to test quality into the productQuality cannot be tested into the product/

Results identifyI

• Test Cases may not address..

−Gaps in user scenarios

−Gaps in alternative flows (error conditions)

−Gaps in business rules

• Confusion and delays with requirements often results in a

compressed schedule for testing

− Increases redo work

• Defect leakage into production


6


Ambiguity Review Benefits

• Limit Scope Creep

• Increase Requirements Traceability to Test Cases

• Reduce Defect Leakage to Production

• Improves Estimation Accuracy

• Reduce Cost of Maintenance

− How much is your organization spending to maintain systems today?

• Reduces redo work

• Increases velocity of Test Case creation

• Increase BA productivity and work product

− Elicitation Checklists

− BA Style Guide

How do I measure success?


Ambiguity Review Place in the SDLC

7


Ambiguity Reviews in Requirements Phase

Requirements Phase

• Elicitation Preparation

• Elicitation Session

• Document Elicitation Results

• Confirm Elicitation Results

−BA Peer Review

−Ambiguity Review

−Final Review & Signoff by Key Stakeholders

Note: Ambiguity Reviews are SDLC agnostic. The process

can be applied to any lifecycle and any document format.

Building Quality In/


Ambiguity Review in the Requirements PhaseWhere do Ambiguity Reviews fit into the Lifecycle?

8


Ambiguity Review Classification


Ambiguity of Reference & Ambiguous Statements

A condition when a requirement uses words such as pronouns,

adjectives, adverbs and verbs that can be interpreted differently

based on the reader’s view.

• The report shall run frequently.

Ambiguity: What is the name of the report?

Ambiguity: What are the data elements in the report?

Ambiguity: Frequently is not measurable

Ambiguity: What is the report generation schedule?

Ambiguity: What happens if there is no data for the report?

Ambiguity: What happens if the report fails?

(Wiegers, Bender)

9


Boundary Ambiguity

A condition when the author uses terms - among or up to. The

scope of the requirement is ambiguous because the stated

requirement can be interpreted in multiple ways.

Example:

1. If an employee makes less than $20,000 per year, the

employer pays 100% of the healthcare premium.

2. If an employee makes more than $20,000 per year, the

employer pays 50% of the healthcare premium for the

employee.

Ambiguity: What if an employee makes exactly $20,000?

(Wiegers, Bender)


Built-In Assumptions

A condition when the author assumes that all consumers of the

document will have the same level of domain knowledge

• Industry domain knowledge

• Subject domain knowledge

• Functional knowledge

• Environmental knowledge

Example: The system must apply the same limitations to searches for existing groups as currently exists in Google Search.

Ambiguity: The requirement assumes the reader knows how the

functionality exists today.

(Wiegers, Bender)

10


Dangling Else

A condition when a requirement states expected results (what

normally happens) but does not state exceptions and error conditions.

Dangling Else

Can

Shall

Could

Should

Is one of

Will

Must

Would

Example: The employee address type shall be either house, apartment or condominium.

Ambiguity: The requirement does not consider exception conditions

such as PO Box.

(Wiegers, Bender)


Etc.

Etcetera is not a quantifiable measurement that can be confirmed so

it is considered totally ambiguous. (Phrases or sentences ending with

etcI)

Example: Subscribers shall identify themselves with unique

information (policy number, social security, etc.) when they call

Customer Care for information about their policy.

(Wiegers, Bender)

11


Ambiguity Review Types


Categorizing Ambiguities to Support Metrics

Ambiguity Categorizations will help withI

• Tracking patterns of ambiguities

− For a Business Analyst

− For the Program

• Building Elicitation Checklists

− Lessons learned turned into questionnaires

− Mentoring sessions for the BA Team

• Team Performance Measurements

− Scope Management

− Defect Leakage to Production

− Requirement Defects

Purpose of Ambiguity Types

12


Ambiguity Type Description

Ambiguous TermTerms (Phrase or Word) used in requirements which can be interpreted

by the reader in multiple ways e.g. frequently, occasionally, efficiently

Conflicting Requirement

Requirements which contradict each other – either in the same

document or across multiple documents.

e.g. The field name is Effective Date but the data type is defined as an

integer

GlossaryWord or acronym used in requirements that is new or not commonly

used but has not been defined in the Glossary/Definitions section.

Grammar, Spelling

& Wording

Grammar, spelling corrections and proposed wording improvements to

increase clarity of the requirement

Incomplete Requirement

Incomplete requirement or statement describing conditions when

information is not fully detailed preventing design or test validation

e.g. The system shall handle 15-25% increase in the second year

Missing Requirement

Missing requirements that were not documented or may not have been

elicited from the business user.

e.g. Missing requirements – alternative flows, business rules,

exceptions and error conditions (Questions – What, When, Where)

Unclear Requirement

Requirements or statements requiring further clarification to allow the

reader user to fully understand the requirement (Questions – How,

Why, What do you mean byI)

(Wiegers, Bender)


Ambiguity Review Template

13


Ambiguity Review Template

• Transfer requirements document to Excel template

− Revision History

− Introductory information (free text)

− Business Requirements

− Data presented in tables such as Glossary, Field Elements

and Financial Information segregated to its own worksheet

• Ambiguity Columns to the Right

− Ambiguity Type

− Ambiguity Description

Transform the Requirements Document to Excel


Requirements Worksheet ExampleNot all columns need to be transferred to the Ambiguity Template/

14


Glossary Worksheet ExampleText transferred to Excel template/


Ambiguity Review – Keyword Searches

15


Ambiguity Review Keywords

An initial review for keywords helps to identify Incomplete

Requirements

All files are transmitted daily.

All 820 Payments Files shall be transmitted to the Federal and

Maryland Jurisdiction after the nightly batch jobs complete

• The Federal 820 Payment File shall be sent daily during the

transmission window between 11 PM and 4 AM EST

• The Maryland 820 Payment File shall be sent daily during the

transmission window between Midnight and 4 AM EST

• If no payments are made for a jurisdiction, an empty file shall

be sent to the jurisdiction

Clues to Finding Ambiguities/


Ambiguity of Reference

Ambiguous Adjectives

Ambiguous Adverbs Ambiguous VariablesAmbiguous

Verbs

above ordinary infrequently the database derive

below rare intuitively the field determine

it same just about the file edit

such seamless more often than not the frame enable

the previous several more or less the information improve

them similar mostly the message indicate

these some nearly the module manipulate

they standard normally the page match

this the complete not quite the rule maximize

those the entire often the screen may

Ambiguous Adjectives transparent on the odd occasion the status might

all typical ordinarily the system minimize

any usual rarely the table modify

appropriate valid roughly the value optimize

custom Ambiguous Adverbs seamlessly the window perform

efficient accordingly seldom Ambiguous Verbs process

every almost similarly adjust produce

few approximately sometime alter provide

frequent by and large somewhat amend support

improved commonly transparently calculate update

infrequent customarily typically change validate

intuitive efficiently usually compare verify

invalid frequently Ambiguous Variables compute

many generally the application convert

most hardly ever the component create

normal in general the data customize

(Bender/Wiegers)

16


Ambiguity Review – Content Review


Ambiguity Review for Content

• Ambiguities in content focus on the following

− Conflicting Requirements

− Incomplete Requirements

− Missing Requirements

− Unclear Requirements

• Questions of Who, What, When, Where & Why

• Questions of How tend toward design questions

− Only use to clarify expected behavior or outcome

− Do not use to ask how the system will process the function

behind the scenes

Clues to Finding Ambiguities/

17


Sample Ambiguity Questions

• What is the name of the file?

• Who is the user?

• What permissions does the user need to review the file?

• When is the file sent?

• What are the contents of the file?

• Where is the file sent?

• What are the business rules to validate the file?

• What are the error messages displayed when a business rule

validation fails?

• What happens if the file is corrupted and can’t be read?

• What happens if the file fails to be generated?

• Is an alert sent if a file is late, corrupt or fails?

• Who is the alert sent to?

Who, What, When, Where, Why/


Ambiguity Review Example

If the policy is a subscriber + spouse/domestic partner +

dependents policy, subscriber wants to cancel from the policy,

Exchange will send us 834 file to enroll the spouse/ domestic

partner as the subscriber under a new SID and the dependents

will remain as dependents under the new subscriber.

Note: Subscriber + Family

Content Questions Raised for this Requirement

18


Ambiguity Questions Posed<


Ambiguity Review Metrics

19


Ambiguity Review MetricsPatterns within the Requirements Document

36

Top 10 Ambiguities by Type Total

Ambiguous Term 55

Grammar Spelling & Wording 79

Incomplete Requirements 59

Unclear Requirements 57

Total Ambiguities 250


Program Level Ambiguity Review MetricsPatterns across the Requirements Document

20


Program Level Ambiguity Review Metrics

Patterns across the Requirements Document


Program Level Ambiguity Review Metrics

Patterns across the Requirements Document

Ambiguity Review Summary Totals

Total Requirement Considered* 3,737

Total Ambiguous Observations 26,478

Top 10 Ambiguities 7,463

Percent Ambiguities in Top Ten 28%

Average Ambiguities Per Requirement 7

*Requirements include Non-Requirement Sections such a s Introduction, Purpose, etc.

21


Ambiguity Review – Benefit to QA


QA Benefits from Ambiguity Review

• Geographically dispersed teams benefit from the additional

level of detail documented in the requirements

• Test Case Creation is simplified by clear requirements

− Normal Flow & Alternative Flows detailed in Use Cases

− Error conditions with error messages

− Business rules and constraints

− Data requirements

− Expected Outcome

• Traceability to requirements provides QA Engineer with path

to use cases and functional/non functional requirements

− Diagnosis of defect types clear

• Requirements vs. Code vs. Test Case

Quantifiable Impact

22


Ambiguity Review – Be Nice About It!


Business Analysts Have Feelings

Ambiguity Reviews are meant to be objective and focus on the

requirements content. HoweverI You are critiquing a

Business Analyst’s work.

• Recognize the Business Analyst’s work effort

• Recognize the Business Analyst’s challenges with eliciting

requirements from business users.

• Be courteous when providing feedback.

Write Ambiguity Questions that are quantitative, clear, correctand complete!

Be respectful of your feedback

23


Appendix


Requirements Review Phase

24


References

Writing High Quality Requirements

By Karl E. Wiegers, 2006

The Ambiguity Review Process

By Richard Bender

Assessing the Impact of Poor Requirements on Companies

By Keith Ellis

Executive Guide to Evaluating Business Requirements Quality

By Keith Ellis

Getting Consensus on Business Requirements – Tips and Traps

By Keith Ellis

The Quest for Good Requirements

By Dr. Martin Schedlbauer, 2011

How to Prevent the Negative Impacts of Poor Requirements

By Sergey Korban, April 30, 2013

The Business Value of Better Requirements

By Karl E. Wiegers, 2006

Ambiguity Reviews: Building Quality Requirements

Technology

Transcript of Ambiguity Reviews: Building Quality Requirements