Control model testing

Matthew Sul l ivan Scot t Barber Sof tware Test Profess ionals Conference Fal l 2011

MANAGING RISK FOR SOFTWARE PRODUCTS

Copyright © 2011 PerfTestPlus, Inc. All rights reserved.

“STATE OF THE S/W TESTING PRACTICE”

• Find bugs (identify risks) OR • Check for compliance (V&V) “Role” of

QA/Testing

• Appears undervalued, BUT •Doesn’t provide nearly the value it

could “Value” of

QA/Testing

• Business goals & value propositions • Business risks & risk controls • Executive information needs

QA/Testing is “out of sync” with


“THE UNDER-INFORMED DIRECTING THE UNDER-TRAINED TO DO THE UNIMPORTANT”

Artifacts (the Unimportant)

•Bugs no one wants to fix •Metrics no one

understands •Documents no one

reads

Testers (the Untrained)

•Don’t know what the executives need, SO

•They do what they are asked to

Executives (the Uninformed):

•Don’t know how to ask for what they need, SO

•They ask for what they know


IMPROVING THE SITUATION (PART 1)

Focus on: •Delivering business value •Reducing business risk

At every business layer, identify & balance: •Responsibility •Accountability

Get your superiors to read Ch 16:Rightsizing the Cost of Testing: Tips for Executives of How to Reduce the Cost of Software Testing; CRC Press 2011


http://www.perftestplus.com/Rightsizing_the_Cost_of_Testing.pdf


http://www.amazon.com/gp/product/1439861552/ref=as_li_tf_il?ie=UTF8&tag=perftestplus-20&linkCode=as2&camp=217145&creative=399373&creativeASIN=1439861552

IMPROVING THE SITUATION (PART 2)


Businesses reduce allocation of resources to testing because of a perception of diminished value.

FEELING UNDER SIEGE?


WHAT DIMINISHES VALUE FOR TESTING?

1. Lack of insight into future

2. Redundancy 3. Specification

blocks 4. Lack of

independence 5. Scope

constraint Copyright © 2011 PerfTestPlus, Inc. All rights

reserved.

LACK OF INSIGHT INTO THE FUTURE

Why didn’t this come up in

testing!


REDUNDANCY

Sign here, and then sign the next box attesting to

the authenticity of the previous signature.


SPECIFICATION BLOCK

Honestly I’d love to start testing today, but first I need detailed requirements. VERY

detailed requirements


LACK OF INDEPENDENCE

Its not fun being the captain’s “no-

man”.


SCOPE CONSTRAINT

Someone else was supposed to be watching

for icebergs.


REQUIREMENT-DRIVEN APPROACH


The purpose of testing is to reduce uncertainty about the future impact of technology.

THE MEANING OF LIFE (FOR TESTERS)


ALTERNATIVE APPROACH


RISK AS A COMMON LANGUAGE

Risk

Security

Functional

Performance Usability

Compliance

Whether explicitly or implicitly, all forms of testing revolve around the reduction and management of risk.


To effectively manage risk, you must effectively manage knowledge.

THE SECRET TO MANAGING RISK


Control Model Testing is a business-aligned approach to software testing that derives “test cases” from knowledge models of the system based on a risk-based taxonomy .

WHAT IS CONTROL MODEL TESTING?


WHAT IS OUR TAXONOMY BASED UPON?

COSO Enterprise Risk

Management Integrated Framework

The Open Group Technical Standard on

Risk Taxonomy

PerfTest Plus Taxonomy Extensions for

Control Model Testing


WHAT ARE THE BASIC ENTITIES?


THE OPEN GROUP’S RISK ASSESSMENT FRAMEWORK


Business •Financial •Legal •Brand or Reputation

Product •Security •Performance •Usability •Other Qualities

Project •Budget •Schedule •Communication

RISK LAYERS


UNADDRESSED RISK


Controls prevent or mitigate risk which may impact business objectives. Control Model Testing helps identify and assess these controls.

HOW CAN TESTS ADDRESS THREATS AND LEVEL OF RISK?


Systems • Firewalls • Encryption • Load Balancing

Preferences • Settings • Security and Access Model

Policies • Code Standards • Monitor and Response • HR

TYPES OF CONTROLS


Development • Development and Test Tools • Code standards • Software components

Implementation • Checklists • Installation scripts

Maintenance • Alerts and Triggers • SOPs • Configuration Management

CONTROLS CONTEXT


“SAMSARIC” TEST LIFECYCLE

Knowledge

Effort

Analyze

Assess

Evaluate

Report


Examine • System • Users • Environment

Identify • Objectives • Processes • Threats • Controls

Output • Initial Control Model

ANALYSIS


INITIAL CONTROL MODEL


Activities •Identify authorities •Solicit opinions •Evaluate exposure •Determine impact

Outcomes •Risk assessment •Assessed Control Model •Test plan

ASSESSMENT


ASSESSED CONTROL MODEL


Activities •Execute planned and

derivative tests • Identify discrepancies •Determine capability

Outcomes •Tested Control Model •Test results • Issues /

recommendations

EVALUATION


EXECUTED CONTROL MODEL


Activities •Communicate •Recommend •Respond

Outcomes •Implementation plan •Knowledgebase update •Confirmation of or

revisions to test plan

REPORTING


Leader

Manager

Coordinator

Tester

THE FOUR ROLES IN CONTROL MODEL TESTING


Responsibilities: •Representation •Roadmaps

Interests • Information • Certainty

Talents • Communication • Vision

Typical Business Titles •Director of Testing or Quality Assurance • Chief Audit Officer (or Assistant to..) • Principle Consultant

LEADER


Responsibilities: •Organizing •Developing

Interests •Capability •Consistency

Talents •Understanding •Motivating

Typical Business Titles •Test Manager

MANAGER


Responsibilities • Planning •Oversight

Interests • Successful outcome • Thoroughness

Talents • Teamwork • Attention

Typical Business Titles • Test or QA Lead or Senior • Analyst or Engineer Level 2 or 3 •Manager 1

COORDINATOR


Responsibilities •Execution •Analysis

Interests •Discovery •Experimentation

Talents •Curiosity •Skepticism

Typical Business Titles •Test or QA Analyst or Engineer •Analyst or Engineer Level 1 or 2

TESTER


Business

Test Leader

Product

Test Manager

Project

Test Coordinator Tester

RISK LAYERS AND ROLES


Testing should be an indispensible advisor for leadership Testing should not be a convenience or scapegoat for

development All types of testing revolve around risk management The key to managing risk is managing knowledge Testing needs to be a learning discipline in the context of risk

taxonomy The test process should be a continuous cycle reducing effort

through increased knowledge Testing roles should correlate to management or risk, not

resources


SUMMARY

[email protected] [email protected]


QUESTIONS?

mailto:[email protected]

mailto:[email protected]

The Open Group (http://www3.opengroup.org/): Risk Taxonomy Technical Standard - https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?publicationid=12156

The Committee of Sponsoring Organizations of the Treadway Commission, or COSO (http://www.coso.org/)

Enterprise Risk Management-Integrated Framework - http://www.coso.org/ERM-IntegratedFramework.htm

PerfTestPlus, Inc. (http://www.perftestplus.com/) Control-Model Testing – (http://www.perftestplus.com/control-model-testing) Rightsizing the Cost of Testing: Tips for Executives of How to Reduce the Cost of Software Testing; CRC Press 2011


RECOURCES

http://www3.opengroup.org/

https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?publicationid=12156

https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?publicationid=12156

http://www.coso.org/

http://www.coso.org/ERM-IntegratedFramework.htm

http://www.perftestplus.com/

http://www.perftestplus.com/control-model-testing

http://www.perftestplus.com/control-model-testing




Matthew Sullivan Quality Control Engineer CCH TeamMate Wolters Kluwer

Test and Support Engineer for PricewaterhouseCoopers for 10 years

Extensive experience in audit and risk management industry

Specialist in testing Microsoft .NET, MS SQL Server, and Lotus Notes applications

MS in Sof tware Engineering from Regis University

Scott Barber CTO, PerfTestPlus, Inc Widely regarded exper tise in

per formance. Contributor to:

Performance Testing Guidance for Web Applications– Microsoft Press

Beautiful Testing - O’Reilly Press How to Reduce the Cost of Testing -

Taylor and Francis

Executive Director of the Association for Sof tware Testing

Co-Founder of the Workshop of Per formance and Reliabil ity

ABOUT US


Control model testing

Technology

Transcript of Control model testing