Overview of Product QA

Product QA: Why it is tough

• Product is more than software– Software application– Platform and variations– Documentation: online help, manuals,

tech briefs, examples, training– Add-ins and third party product

integrations– Installation experience– Handling upgrades and updates

Product QA: Why it is tough

• Quality is more than lack of bugs– Does it solve the problem when it works?– Is it practical to use? Is it designed for

efficient use?– Does it keep working or works only

under ideal conditions?– Is it economical to build, maintain and

respond rapidly to customer/user needs?

• QA is everything you do to minimize risk of failure and promote excellence

• Help producing products that satisfy target customer base

Role of QA in Product Company

Role of QA in Product Company

• Broad functions of QA– Risk management: release planning, release

approval– Bring customer involvement: early release programs– Act as skillful developers: mimic tough customer!– Process definition and improvement– Inspection and testing– Experience-based improvement: contribute to

product definition, many finer points

• Should play an important role throughout The Life of a Product

5 4

3

21

6

Planning Design

Coding

TestingRelease

Post-Release

Process Definition

Process Improvement

QA Planning

Risk Assessment

QA Certification

Release Inspection

Test Development

Test Suite Design

Product DefinitionInputs

Early ReleaseProgram

Test Execution

Code Review

Design Review

Structure and Function of QA

Non-testing Functions

• What QA is required to do• QA procedures, expectations from

other teams• Effort estimation and team

planning• Resource planning, tools and

software procurement

QA Planning

• Design the tests in parallel to software design

• Tests need organization into independent test suites

• Build the tests before you build the software

• If you find a bug, write a test that proves the bug exists, fix the bug and then rerun the test to prove it is fixed

Test Suite Design

• Goal: Clarification and accept/reject design decisions

• Starts with a presentation of the design for the component/module to a selected panel of developers

• Peer review is commonly used, with most relevant engineers from other parts of the product included

• Discussion and coordination of comments incorporation

Test Suite Design Review

• Check-in review done with peers of the dev team– Including new features and bug fixes

• Code walkthrough by dev across streams, including QA– review meeting with prepared participants– Sample source files are selected for review

• Focus is on finding defects, before they slip through to testing phases– Lower costs by catching them early

Check-in Review, Walkthrough

• Need to ensure smooth porting of current customers

• Test ISV applications available internally

• Public program to increase the test base– Alpha and Beta testing

Early Release Programs

• QA evaluates quality, independent of developers– Developers should not determine how

much quality is sufficient

• QA acts as internal customers and verifies release readiness

Release Readiness

Product Testing

Based on Class Description

How a test is written

Black box Test does not use knowledge of product code and design

White box Test uses product internals, even calls unexposed functions

When a test is being run

Sanity testing Basic set of tests run after every nightly build, to ensure that nothing is broken

Alpha testing Testing done on code release to QA

Beta testing Testing done with the help of external partners, early release customers, etc.

Final testing Final set of testing done by QA to approve GA release

What purpose the test is serving

Functional Verify that functional requirements of the product are as per specifications

Compliance Test suite for compliance to industry standards

Load Verify performance and scalability of the product; also produce data on installation requirements

Reliability Negative testing to ensure that the product functions gracefully under unexpected conditions

Usability Testing the product for practicality, efficiency of use and ease of learning of the product interface

Our Test Classification

• Key aspects to cover (they are endless!)– Correctness– Interoperability– Compatibility– Compliance– Security

• Think of permutations– Functions vs. input data vs. states– Product vs. platforms– System vs. external factors– Testing vs. versions of product

Functional Testing

• Emphasis on when product is expected to be declared “compliant” with Industry Standard (such as J2EE™)

• Compliance test suite (TCK for J2EE) might be made available

• Compliance to standards does not mean the product is functionally tested for use

Compliance Testing

• Performance testing– Performance that can be achieved on a configuration setup– Deterioration with load– Variations across platforms– Writing a sizing guide with suggested deployment

configurations– Generating a near real-world load

• Scalability testing– How does the system scale with increase in platform capacity– Fail over and load balancing under clustered configurations

• Benchmarking– Comparative performance testing and analysis with competitor

products– Public benchmarking, such as SPECjAppserver™ or ECperf™

Load Testing

Performance Reports

• Maturity: how the product expects erroneous user handling

• Fault-tolerance: Ability to perform under faulty hardware/platform conditions with degraded performance but near full functionality

• Integrity: How the system cannot be broken by malicious users

Reliability Testing Aspects

Reliability Testing Aspects...

• Recoverability: ability of the product to recover to a consistent state when power outage, network fault or other disaster happens

• Safety: protection of user data, backup, logging, etc.

• Understandability: clarity of interface, intuitiveness

• Learnability: ease of learning, matches user’s current understanding of other products

• Consistency: across versions, all across the product

Usability Testing Aspects

Usability Testing Aspects...

• Its not only about GUI, its also about file formats, XML, alternatives such as command line tools

• We normally work with external consultants who are specialized in usability engineering

• Efficiency– Storage– Processing

• Maintainability– Code organization, branching– Modularity for ease of changes, maintenance,

re-factoring

• Portability– Platform support– Reusability of code

Other Aspects of Testing

Some of these aspects may not be important to end-customers, but very important for engineering

• Document the design

• Use internal error checking

• Test units before integrating them

• Tell the tester what’s new or flaky

• Test the build yourself, first

• Evolve product in functional layers

Making Testing Easier

Tools and Infrastructure

• Test automation helps repeatability of testing

• Useful test automation is a major software project

• Engineers are able to detect more problems, more accurately

Testing Automation

Test Case Management System

J2EE™ Compliance Testing

• J2EE CTS Kit– From Sun

• Standards Verification• Automated Harness

• Augment CTS– For Complex scenarios

• Built on Studio Framework– Homegrown

• Ease of writing new tests• Automated Discovery of new tests• Extensive tests metadata

– Compose new plans/suites

• Integrated into the Build System– For automated post-build testing

Test Management Framework

Test Management Framework

Reliability Testing

Distributed Stress Testing System

Thank you.