ISTQB Ravinder Document

Ravinder Puranam

[email protected]

of 27

Simplified Document to prepare for ISTQB, a Certified Tester (Foundation Level)

– August 2011

******************************************************************************

K1: remember; K2: understand; K3: apply; K4: analyze

Chapter and Number of Questions Expected from each chapter

Chapter 1. The principles of testing -- 7

Chapter 2. Testing throughout the life-cycle -- 6

Chapter 3. Static testing -- 3

Chapter 4. Test design techniques -- 12

Chapter 5. Test management -- 8

Chapter 6. Tool support for testing 4

******************************************************************************

Chapter 1. The principles / fundamentals of testing -- 7

******************************************************************************

1.1. Why is Testing necessary?? ---

Terms: Bug, defect, error, failure, fault, mistake, quality, risk

Generally accepted set of testing definitions/terminologies used worldwide are from BS 7925-1

standards document.

Sub-Topics:

Software Systems Context ---

Softwares have been an integral part of our life...Software that does not work correctly can lead

to many problems, including loss of money, time or business reputation, and could lead to many

serious outputs...

Cause of Software Defects ---

Human being can make an error (mistake), which produces a defect (fault, bug) in the program

code, or in a document. If a defect in code is executed, the system may fail (output is not

expected result) causing a failure.

Defects occur because of human errors due to many reasons like time/complex code/technology.

Failure can be caused by environmental conditions also.

Role of Testing in S/W Development, Maintenance and Operations ---

Testing contributes to a quality software system by correcting the defects, hence reducing the

risks before the system is released to production / operation.

Testing and Quality ---

Testing can give confidence in the quality of the software if it finds few or no defects. A properly

designed test that passes reduces the overall level of risk in a system.

When testing does find defects, the quality of the software system increases when those defects

are fixed.

Ravinder Puranam

[email protected]

of 27

It is possible to measure the quality of the software in terms of defects found, for both functional

and non-functional software requirements and characteristics like

reliability/usability/efficiency/maintainability/portability.

Process can be improved when the lessons learnt from earlier projects is taken care for root cause

analysis, this improving the quality of future systems.

How much testing is enough?

How much testing is enough considers in to account the level of risks, including technical,

safety, and business risks (RISK of missing important faults, incurring failure costs, releasing

untested or under-tested software, losing credibility and market share, missing a market window,

over-testing, ineffective testing), and project constraints such as time and budget.

*** Prioritize tests so that, whenever you stop testing, you have done the best testing in the time

available.

How to prioritize?? Possible ranking criteria (all risk based): test where a failure would be most

severe, test where failures would be most visible, test where failures are most likely, ask the

customer to prioritize the requirements, what is most critical to the customer‟s business, areas

changed most often, areas with most problems in the past, most complex areas, or technically

critical.

********************************************

1.2. What is Testing?? ---

Terms: Debugging, requirement, review, test case, testing, test objective

Testing is not only executing the test cases / test scenarios, but many other testing activities...

Testing has objectives like: Finding defects, Gaining confidence about the level of quality,

providing information for decision-making and preventing defects.

Debugging (Development activity) and testing are different.

********************************************

1.3. Seven Testing Principles ---

Terms: Exhaustive Testing

Many testing principles are suggested and offer common guidelines for all testing.

Principle 1: Testing shows presence of defects, but cannot prove that there are no defects. It only

reduces the probability of undiscovered defects remaining in the software.

Principle 2: Exhaustive testing (All combinations) is impossible except for trivial cases, instead

risk analysis and priorities should be used to focus testing efforts.

Principle 3: To find defects early, start EARLY TESTING in SDLC and focus on defined

objectives.

Principle 4: Defect Clustering is needed. Testing effort shall be focused proportionally to the

expected and later observed defect density of modules.

Ravinder Puranam

[email protected]

of 27

Principle 5: Pesticide paradox, test cases need to be regularly reviewed and revised, and new and

different tests need to be written to exercise different parts of the software / system

Principle 6: Testing is context dependent, i.e. for each domain / application there is different way

of testing approach

Principle 7: Absence-of-errors fallacy, i.e. Finding/Fixing defects does not help if system built is

unusable and not as per user needs / expectations.

********************************************

1.4. Fundamental Test Process ---

Terms: Confirmation testing, re-testing, exit criteria, incident, regression testing, test basis, test

condition, test coverage, test data, test execution, test log, test plan, test procedure, test policy,

test suite, test summary report, test ware.

The Fundamental Test Process consists of

1.4.1. Test Planning and Control --- Test Planning is activity of defining the objectives /

specifications of testing activities; whereas Test control is the ongoing activity of comparing

actual progress against the plan which takes different information to come to conclusion.

1.4.2. Test analysis and design --- Is the activity during which general testing objectives are

transformed into test conditions and test cases, Test analysis and design activity has many major

tasks like writing / reviewing the test artifacts, prioritizing test conditions, traceability, and test

cases, etc.

1.4.3. Test implementation and reporting --- Is the activity where designed test artifacts are taken

with added I/P's or any other information with environment set. Execution, defect logging and

reporting are also considered in this stage.

1.4.4. Evaluating exit criteria and reporting --- Test execution is assessed against the defined test

objectives...This has major tasks like checking test logs against exit criteria, assessing if more

tests are needed or if exit criteria specified to be changed, and publishing test summary report.

1.4.5. Test closure activities --- Test closure activities collect data from completed test activities

to consolidate information of the major tasks done in Test process which gives project milestone

for release / test project complete / maintenance release, etc.

********************************************

1.5. The Psychology of Testing ---

Terms: Error guessing, independence

The mindset to be used while testing and reviewing is different from that used while developing

software. Independent testing may be carried out at any level of testing or stage of SDLC.

Due to special testing independence effective testing may be carried out by clearly stating the

objectives of testing.

Testing is often seen as a destructive activity, even though it is very constructive in the

management of product risks.

Communication problems may occur, particularly if testers are seen only as messengers of

unwanted news about defects.

Ravinder Puranam

[email protected]

of 27

Effective communication and constructive environment always removed the differences between

testing and other teams.

********************************************

1.6. Code of Ethics ---

Certified testers always act in the best interests of their clients and employer, consistent with

public interest.

Professional standards, Integrity, independent in judgment, reputation, cooperation, supportive

and continuous learning are all the ethics / code of conduct as per ISTQB

Ravinder Puranam

[email protected]

of 27

******************************************************************************

2. Testing throughout the Software life-cycle -- 6

******************************************************************************

2.1. Software Development models

Terms: Commercial Off-The-Shelf (COTS), iterative-incremental development model,

validation, verification, V-model

Different testing approach / activities are related based on different SDLC used.

2.1.1. Relationship between testing and development life cycles.

Although variants of the V-model exist, common types of V-model uses four test levels:

-- Component (unit) testing

-- Integration testing

-- System testing

-- Acceptance testing

With various development and test levels in V-model or any SDLC models, the basis of testing

are the product documents (like use cases, requirement specs, code, design docs, etc)

2.1.2. SDLC based on project and product characteristics.

Considering Iterative-incremental Development Model where the SDLC/system has short

development cycles

E.g.: Prototyping, Rapid Application Development (RAD), Rational Unified Process (RUP) and

agile development models.

A system that is produced using these models may be tested at several test levels during each

iteration.

Regression testing is increasingly important on all iterations after the first one. Verification and

Validation can be carried out on each increment.

2.1.3. Recall characteristics of good testing that are applicable to any life cycle.

Testing within any Life Cycle Model has several characteristics of good testing like:

--- For each development activity there is a corresponding testing activity

--- Each test level has test objectives specified to that level

--- Analysis and design for a given test level should begin during the corresponding development

activity

--- Testers should be involved in reviewing documents as soon as drafts are available in the

Development life cycle.

Ravinder Puranam

[email protected]

of 27

Test levels can be combined or reorganized depending on the nature of the project or the system

architecture.

E.g.: For integration of a Commercial Off-The-Shelf (COTS) software product into a system, the

purchaser may perform integration testing at the system level (Integration to infrastructure and

other systems, or system deployment) and acceptance testing.

********************************************

2.2. Test Levels

Terms: Alpha testing, beta testing, component testing, driver, field testing, functional

requirement, integration, integration testing, non-functional requirement, robustness testing, stub,

system testing, test environment, test level, test driven development, user acceptance testing.

Testing a system's configuration data shall be considered during test planning.

2.2.1. Compair different levels / stages of testing

Some of the levels of testing are:

--- Component Testing (Unit, module or program testing)

Test basis: Component requirements, Detailed design, and Code

Typical test objectives: Components, Programs, Data conversion / migration programs, Database

modules

Stubs, drivers and simulators may be used in Component Testing. It may include functional,

specific non-functional (such as resource-behavior like memory leaks or robustness testing), as

well as structural testing (e.g. decision coverage)

Generally Component Testing is done with the involvement of the developer who wrote code

and defects are fixed without formally managing these defects.

One approach to component testing is to prepare and automate test cases before coding since it is

highly iterative.

--- Integration Testing

Test basis: Software and system design, Architecture, Workflows, use cases

Typical test objects: Subsystems, Database implementation, Infrastructure, Interfaces, System

configuration and configuration data

Integration testing tests interfaces between components, interactions with different parts of a

system, such as operating system, file system and hardware, and interfaces between systems.

Systematic integration strategies may be based on the system architecture (Top-down and

bottom-up). Integration should normally be incremental rather than "big bang" (not all modules

integration at a time).

Ravinder Puranam

[email protected]

of 27

Non-functional testing (like performance) may also be included in integration testing with

functional testing.

Function increment and Bing Bang both are integration testing techniques but the difference is in

function increment, we add function one by one and then test but in big bang we add all

functions at once then test. Driver: also know as Test harness and scaffolding.

--- System Testing

Test basis: System and software requirement specification, Use cases, Functional specification,

Risk analysis reports.

Typical test objects: System, user and operation manuals; System configuration and

configuration data.

System testing is concerned with the behavior of a whole system/product.

The test environment should correspond to the final target or production environment as much as

possible to minimize the risk of environment-specific failures not being found in testing.

This is performed to investigate functional and non-functional requirements of the system, and

data quality characteristics.

An independent test team often carries out system testing.

--- Acceptance Testing

Test basis: User requirements, System requirements, Use cases, Business processes, Risk

analysis reports.

Typical test objectives: Business processes on fully integrated system, Operational and

maintenance processes, user procedures, Forms, Reports, Configuration data.

Acceptance testing is often the responsibility of the customers or the users of a system; other

stakeholders may be involved as well.

The goal is to establish confidence in the system, parts of the system or specific non-functional

characteristics of the system.

To assess the systems readiness for readiness for deployment and use.

Operational (acceptance) testing: (Done by system administrators) Testing of backup/restore,

disaster recovery, user management, maintenance tasks, data load and migration tasks, periodic

check of security vulnerabilities.

Contract and regulation acceptance testing: Is performed against a contract's acceptance criteria

for producing custom-developed software (OEM) by adhering to legal or safety regulations.

Alpha and Beta (or field) testing:

Alpha testing is performed at the developing organization's site but not by the developing team.

Beta testing, or field-testing, is performed by customers or potential customers at their own

locations.

Ravinder Puranam

[email protected]

of 27

Organizations may use terms such as factory acceptance testing and site acceptance testing

before/after moving to a customer's site.

********************************************

2.3. Test Types

Terms: Black-box testing, code coverage, functional testing, interoperability testing, load testing,

maintainability testing, performance testing, portability testing, reliability testing, security

testing, stress testing, structural testing, usability testing, white-box testing

Objectives of test type always focus on functional, non-functional (reliability or usability),

system, change related.

2.3.1. Compair Functional / Non-Functional / Structural / change related software test

types

2.3.2. Recognize functional and structural tests occur at any level

2.3.3. Identify and describe non-functional test types based on non-functional

requirements

2.3.4. Identify and describe test types based on analysis of software architecture

2.3.5. Describe purpose of confirmation testing and regression testing

Functional tests are based on functions and features and their interoperability with specific

systems, and may be performed at all test levels.

Non-functional tests are not limited to, performance testing, load testing, stress testing, usability

testing, maintainability testing, reliability testing and portability testing. It is testing of HOW the

system works.

Structural testing (white-box) may be performed at all test levels, but especially in component

testing and component integration testing.

Structural testing may be based on the architecture of the system, such as a calling hierarchy.

Structural testing approaches can also be applied at system, system integration or acceptance

testing levels.

Defect when detected and fixed, needs to be re-tested so as to confirm that the original defect has

been successfully removed. This is called confirmation testing.

Debugging is a development activity, not a testing activity.

Regression testing is the repeated testing of an already tested program, after modification, to

discover any defects introduced or uncovered as a result of the changes in one/more of the

module of the software.

Regression testing can be performed at all test levels, and includes functional, non-functional and

structural testing.

If the software is stable and since regression test suits are run many times due to small changes,

so regression testing leads to automation testing.

********************************************

Ravinder Puranam

[email protected]

of 27

2.4. Maintenance Testing

2.4.1. Compare maintenance testing with normal new software testing

2.4.2. Recognize indicators for maintenance testing

2.4.3. Describe role of regression testing and impact analysis in maintenance

Once deployed, a software system is often in service for years or decades. During this time the

system, its configuration data, or its environment are changed or extended.

Planning of any releases in advance is crucial for successful maintenance testing.

Maintenance testing is done on an existing Operating System, and is triggered by modifications,

migration, or retirement of the software or system.

Modifications include planned enhancement changes, corrective and emergency changes, and

changes in environment, such as planned operating system or database upgrades, planned

upgrades of Commercial-Off-The-Shelf software, or patches to correct newly exposed or

discovered vulnerabilities of the operating system.

Determining how the existing system may be affected by changes is called impact analysis, and

is used to help decide how much regression testing to do. the impact analysis may be used to

determine the regression test suite.

Maintenance testing can be difficult if specifications are out of date or missing, or testers with

domain knowledge are not available.

Ravinder Puranam

[email protected]

of 27

******************************************************************************

Chapter 3: Static testing -- 3

******************************************************************************

3.1 Static Techniques and the test process

Terms: Dynamic Testing, Static Testing

3.1.1. Recognize software work products that can be examined by the different static techniques

3.1.2. Importance and value of considering static techniques for the assessment of software work

products

3.1.3. Difference between static and dynamic techniques, considering objectives, types of defects

to be identified, and role of these techniques within the SDLC.

Static testing relies on manual methods. Dynamic testing requires execution of software.

Both have same objective, to find defects. Additionally, static testing is doing bug RCA.

Static testing can be used by the developer who wrote the code, in isolation. Code reviews,

inspections and walkthroughs are also used.

Dynamic testing is done by the testing team....Some of the methodologies include unit testing,

integration testing, system testing and acceptance testing.

Static is generally not detailed testing, but checks mainly for the sanity of the code, algorithm, or

document. It is primarily syntax checking of the code or and manually reading of the code or

document to find errors.

Dynamic analysis refers to the examination of the physical response from the system to variables

that are not constant and change with time

Static testing is the verification portion of Verification and Validation

Dynamic testing is the validation portion of Verification and Validation.

********************************************

3.2. Review Process

3.2.1. Recall activities, roles and responsibilities of a typical formal review.

3.2.2. Differences between informal review, technical review, walkthrough and inspection.

3.2.3. Explain factors for successful performance of reviews.

Terms: Entry criteria, formal review, informal review, inspection, metric, moderator, peer

review, reviewer, scribe, technical review, walkthrough

Review can be done at any stage and depends on the need for review, maturity of the

development process, agreed objectives,

Ravinder Puranam

[email protected]

of 27

Activities of a Formal Review:

Planning: Define review criteria, selecting the personnel, allocating roles, defining entry & exit

criteria for more formal review types (e.g. inspections), selecting which parts of document to

review, checking entry criteria

Kick-off: distributing documents and explaining the objectives, process and documents to the

participants.

Individual preparation: Preparing for the review meetings by reviewing the documents and

noting potential defects, questions and comments.

Examination/evaluation/recording of results of review meeting: discussing or logging, with

documented results, noting defects, making recommendations/decisions regarding handling the

defects and examining/evaluating and recording issues during physical meetings

Rework: Fixing defects found and recording updated status of defects.

Follow-up: Checking defects fixed, gathering metrics and checking on exit criteria.

Roles and responsibilities:

Manager: Decides on execution of reviews, allocates time in project schedules and determines if

review objectives have been met.

Moderator: Person who leads the review of documents, planning review, running review meeting

and following-up after meeting.

Author: Person responsible for document to be reviewed.

Reviewers: Persons with specific technical / business background who reviews the product and

participates in all review meetings.

Scribe: Documents issues, problems and open points that were identified during meetings.

Types of Reviews:

Apart from formal review there are many other review types as below, of which any one can be

used depending on the use by their organization.

--- Informal Review: No formal process, any technical lead can review, results may be

documented, varies in usefulness depending on the reviewers, Main purpose: inexpensive way to

get some benefit

--- Walkthrough: Meeting led by author, may take the form of scenarios, dry runs, peer group

participation, open-ended (optional pre/post reviews) sessions, optional scribe, may become a bit

formal, Main purpose: learning, gaining understanding, finding defects.

--- Technical Review: similar to formal review activities, ideally led by trained moderator,

optional use of checklists, Main purpose: discussing, making decisions, evaluating alternatives,

finding defects, solving technical problems, and checking conformation to specifications, plans,

regulations and standards.

--- Inspection: Led by trained moderator, same as Formal process based on rules and checklist,

Main purpose: finding defects.

Ravinder Puranam

[email protected]

of 27

Success factors for reviews:

Predefined objectives, right people for review objectives are involved, testers are valued

reviewers who review and prepare tests earlier, defects found are welcome and expressed

objectively, used to increase effectiveness of defect identification, review is conducted in an

atmosphere of trust,

Training is giving especially more formal techniques like inspection; Management supports a

good review process, emphasis on learning and process improvement.

********************************************

3.3. Static Analysis by Tools

3.3.1. Defects and errors identified by static analysis and compare them with dynamic

testing

3.3.2. Typical benefits of static analysis

3.3.3. Code and design defects that may be identified by static analysis tools.

Terms: Compiler, complexity, control flow, data flow, static analysis.

Objective of static analysis is to find defects in software source code and software models,

dynamic testing does execute the software.

Static analysis can locate defects that are hard to find in dynamic testing.

Static analysis tools are typically used by developers before and during component and

integration testing or when checking-in code to configuration management tools, and by

designers during software modeling.

Many warning messages are produced by the tools which need to be well-managed to allow the

most effective use of the tool.

Compilers may offer some support for static analysis, including calculation of metrics.

Value of Static Testing: Early detection of defects before test execution which are not easily

found in dynamic testing, early warnings about code or design, detecting dependencies and

inconsistencies in software models such as links, improved maintainability and prevention of

defects.

Typical defects discovered by static analysis tools: referencing a variable with an undefined

value, Inconsistent interfaces, unused variables, unreachable code, missing and erroneous logic,

overly complicated constructs, programming standards violation, security vulnerabilities, syntax

violations of code and software models.

Ravinder Puranam

[email protected]

of 27

******************************************************************************

Chapter 4: Test design techniques -- 12

******************************************************************************

4.1. The Test Development Process

Terms: Test case specification, test design, test execution schedule, test procedure specification,

test script, traceability

The Test development process can be done in different ways, from very informal with little or no

document, to very formal.

The level of formality depends on the context of the testing, including the maturity of testing and

development processes, time constraints, safety or regulatory requirements, and the people

involved.

4.1.1. Differentiate between a test design specification, test case specification and test

procedure specification

4.1.2. Compare the terms test condition, test case and test procedure

4.1.3. Evaluate the quality of test cases in terms of clear traceability to the requirements

and expected results

4.1.4. Translate test cases into a well-structured test procedure specification at a level of

detail relevant to knowledge of the testers

Test analysis, the test basic documentation is analyzed in order to determine what to test i.e. test

conditions.

A Test condition (test scenario) is defined as an item or event that could be verified by one or

more test cases.

Test cases must and should consist of set of input values, execution preconditions, expected

results, obtained outputs and execution post conditions, defined to cover a certain test objectives

/ test conditions.

Test procedure specifies the sequence of actions for the execution of a test either

manual/automation are formed into a test execution schedule that defines the order of execution.

During test implementation the test cases are developed, implemented, prioritized and organized

in test procedures.

Relating traceability from test conditions back to the specifications and requirements enables

both effective impact analysis when requirements change and determining requirements

coverage for a set of tests.

During test analysis the test approach is implemented to select the test design techniques based

on risks and other criteria

During test design the test cases and test data are created and specified.

IEEE STD 829-1998 document describes content of test design specifications and test case

specifications.

********************************************

Ravinder Puranam

[email protected]

of 27

4.2. Categories of Test Design Techniques

Terms: Black-box test design technique, experience-based test design technique, test design

technique, white-box test design technique

4.2.1. Recall reasons that both specification-based (black-box) and structure-based

(white-box) test design techniques are useful and list common techniques for each

4.2.2. Explain the characteristics, commonalities, and differences between specification-

based testing, structure-based testing and experience-based testing

The purpose of Test Design Techniques is to identify test conditions, test cases, and test data

Test design techniques are distincted generally as black-box or white-box.

Black-box testing (specification-based test design) does not use any information related to the

internal structure of the system to be tested, test conditions/test cases/test data are derived from

test basis documentation which includes both functional and non-functional testing.

--- Models, either formal or informal, are used for the specification of the problem to be solved,

the software or its components.

--- Test cases can be derived systematically from these Models.

White-box testing (structure-based or structural test design) is based on analysis of the structure

of the component or system.

--- Information about how the software is constructed is used to derive the test cases

--- Extent of software covered is measured and further test cases are derived systematically to

increase coverage.

Black-box and White-box test designs can be combined with experience-based techniques to

leverage the experience of developers, testers and users to determine what should be tested.

Some techniques fall into either of the two test design techniques (Black-box/White-box), others

have more than one category.

Experience-based test design techniques include:

--- Knowledge and experience of people are used to derive test cases.

--- Knowledge of testers, developers, users and other stakeholders about the

software/usage/environment is one source of information

--- Knowledge about likely defects and their distribution is another source of information.

Black box White box

Equivalence partitioning Statement testing

Boundary value analysis Branch / Decision testing

State transition testing Data flow testing

Cause-effect graphing Branch condition testing

Decision table testing Branch condition combination testing

Use case testing Modified condition decision testing

LCSAJ testing

Ravinder Puranam

[email protected]

of 27

4.3. Specification-based or Black-box Techniques

Terms: Boundary value analysis, decision table testing, equivalence partitioning, state transition

testing, use case testing

4.3.1. Write test cases from given software model using equivalence partitioning,

boundary value analysis, decision tables and state transition diagrams/tables

4.3.2. Explain main purpose of each of the four testing techniques, what level and type of

testing could use the techniques, and how coverage may be measured.

4.3.3. Explain the concepts of use case testing and its benefits

Many Black-box techniques are present, some of them are:

First method: Test one right and one wrong value one right are the type of positive data like

correct username and password, proper navigation, etc

The one wrong are the type of negative data like wrong username and password, blank data, etc

In this pick alternately one right and one wrong values...this method is called Test one right and

one wrong value method

Second method: Equivalence partitioning --- Large input data is partitioned into groups that are

expected to exhibit similar behavior so as to cover maximum inputs

E.g. a) Say I/P data is 10 to 100, then Equivalence values will be less than 10, between 10 to 100

and above 100 i.e. 3 values.

b) Say I/p data is 10; 11-50; 51-75; 76-100…..then the Equivalence values will be below

10,between 11-50, between 51-75, between 76-100 and above 100 i.e. 5 values.

Third method: Logical combination testing --- Combination testing with discrete values

Fourth method: State-transition testing --- State-transition testing tests only the state machine

logic in itself

Fifth method: Boundary value and domain analysis --- The boundaries are the values on and

around the beginning and end of a partition

Boundary value analysis can very easily be combined with equivalence partitioning.

E.g. Say I/P range is between X and Y, then the values will be below X, between X and Y (may

include X/Y also sometimes) and above Y i.e. 3 values; If any complicated ranged then will be

according to the same logic.

Sixth method: Decision Table Testing --- All combinations of triggering conditions are used as

inputs and all logical decisions are covered which is the strength of this test design technique.

Seventh method: Use case Testing --- Test cases are derived from use cases, which tell the

interactions between the system with Pre-conditions and post-conditions.

This is very useful to do effective integration / system testing where use-cases are available. This

can be combined with other specification-based test techniques.

Ravinder Puranam

[email protected]

of 27

Out of many techniques, some of the most useful or often discussed once are:

- Random and statistical testing --- Tested by generating random inputs, and checking the output

- Exhaustive testing --- Every possible combination of input values is tested. (Almost impossible)

- Error guessing --- Concentrating on typical problems like low memory, too much inputs, etc

(Corner test cases)

- White-box testing (Explained in below section 4.4.) --- Typically used for unit or component

testing where there is access to the program code.

- The test oracle --- It is defined as a mechanism that decides if a result is right or wrong using

spreadsheet calculations, or any other tools...the only thing you do is check that the program does

not crash.

********************************************

4.4. Structure based or white-box Techniques

Terms: Code coverage, decision coverage, statement coverage, structure-based testing

4.4.1. Describe the concept and value of code coverage

4.4.2. Explain the concepts of statement and decision coverage, and give reasons why

these concepts can also be used at test levels other than component testing (E.g.. On

business procedures at system level)

4.4.3. Write test cases from given control flows using statement and decision test design

techniques

4.4.4. Assess statement and decision coverage for completeness with respect to defined

exit criteria

Structure-based or white-box testing is based on an identified structure of the software or the

system which needs Component level (Statements, decisions, branches or even distinct paths),

Integration level and System level

Mainly has 3 code-related structural test design techniques like:

--- Statement Testing and Coverage, Is determined by no. of executable statements covered by

test cases divided by the number of all executable statements.

--- Decision Testing and Coverage: For this a control flow diagram may be used to visualize the

alternatives for each decision (E.g... True / False).

It is determined by the number of all decision outcomes covered by test cases divided by the

number of all possible decision outcomes in the code under test.

Decision coverage is stronger than statement coverage.

100% LCSAJ coverage will imply 100% Branch/Decision coverage

100% branch coverage guarantees 100% decision coverage.

100% decision/branch coverage guarantees 100% statement coverage.

*LCSAJ = Linear Code Sequence and Jump.

--- Other Structure-based techniques...There are stronger levels of structural coverage beyond

decision coverage, for e.g. condition and multiple condition coverage.

********************************************

Ravinder Puranam

[email protected]

of 27

4.5. Experience-based Techniques

Terms: Exploratory testing, (fault) attack

4.5.1. Recall reasons for writing test cases based on intuition, experience and knowledge

about common defects.

4.5.2. Compare experience-based techniques with specification-based techniques

Tests are derived from tester's skill and intuition and their experience with similar applications

and technologies. Special tests are identified which are not easily captured by formal approaches.

A common is error guessing, which is also called as fault attack.

These defects and failure lists can be built based on experience, available defect and failure data,

and from common knowledge about why software fails.

More formally, Exploratory testing is concurrent test design, test execution, test logging and

learning, based on a test charter containing test objectives, and carried out within time-boxes. It

can serve as a check on test process, to help ensure that the most serious defects are found.

********************************************

4.6. Choosing Test Techniques

4.6.1. Classify test design techniques according to their fitness to a given context, for the

test basis, respective models and software characteristics.

Choise of which test techniques to use depends on a number of factors, including the type of

system, regulatory standards, customer or contractual requirements, level of risks, type of risks,

test objectives, documents, knowledge level of testers, time and budget, SDLC, use case models

and previous experience with types of defects found.

Some techniques are more applicable to certain situations and test levels, others are applicable to

all test levels.

When creating test cases, testers generally use a combination of test techniques including

process, rule and data-driven techniques to ensure adequate coverage of object under test.

Exploratory testing is normal testing + some tests probably based on several factors, including

past experience on similar projects, specific skills and detailed knowledge on Domain/Project,

and perhaps also aspects of personality, such as learning style etc without domain knowledge if

the testing is done by the tester and explores all the scenarios then it is called exploratory testing

Ravinder Puranam

[email protected]

of 27

******************************************************************************

Chapter 5. Test management -- 8

******************************************************************************

Learning objectives for Test management

The Objectives identify what you will be able to do following the completion of each module.

5.1. Test Organization

Terms: Tester, test leader, test manager

5.1.1. Recognize the importance of independent testing

5.1.2. Explain the benefits and drawbacks of independent testing within an organization - K2

5.1.3. Recognize the different team members to be considered for the creation of a test team

-- Options like forming groups from development, outsourced testers, independent testers,

etc...But for large/critical projects as multiple level of testing is involved, it is suggested to have

an independent testing team but drawbacks like developers focus on quality and isolation from

development team may happen.

5.1.4. Recall the tasks of a typical test leader and tester

Sometimes, Test leader is called as a test manager or test coordinator. The role of the test leader

may be performed by a project manager, a development manager, a QA manager or the manager

of a test group.

In large projects test leader and test manager both exists. Tipically test leader plans, monitors and

controls the testing activities and tasks as defined in 1.4 section

-- Coordinate the test strategy, plan with Project Manager and others, Write/Review test strategy,

test policy of organization, plan tests, plan project activities, resources, define test levels, cycles,

adapt planning based on test results and test/project progress, metrics to measure the team, to go

for automation or not, select tools for testing, implementation of test environment, test summary

reports based on information gathered during testing

Tester performs all the tasks assigned to him by the Test Leader / Manager

-- Tester reviews/contributes to test plan, SRS assessing, any other specifications/Documents

review, test data, test cases/scripts, execution, automation tests, bug/defect tracking, measure

performance of components and systems, review tests developed by others, etc.

At some stages testers are developers for component and integration level; business experts for

User Acceptance level and operators for operational acceptance testing.

********************************

Ravinder Puranam

[email protected]

of 27

5.2. Test Planning and Estimation - K3

Terms: Test approach, test strategy

IEEE 829 - Standard for Software Test Documentation

5.2.1. Recognize the different levels and objectives of test planning

5.2.2. Summarize the purpose and content of the test plan, test design specification and test

procedure documents according to the 'Standard for Software Test Documentation' - IEEE 829-

1998 - K2

5.2.3. Difference b/w conceptually different test approaches, such as analytical, model-based,

methodical, process/standard compliant, dynamic/heuristic, consultative and regression-averse -

K2

5.2.4. Differentiate between the subject of test planning for a system and schedule test execution

- K2

5.2.5. Write a test execution schedule for a given set of test cases, considering prioritization, and

technical and logical dependencies - K3

5.2.6. List test preparation and execution activities that should be considered during test planning

5.2.7. Recall typical factors that influence the effort related to testing

5.2.8. Differentiate b/w two conceptually different estimation approaches: the metrics-based and

expert-based approaches - K2

5.2.9. Recognize/justify adequate entry and exit criteria for specific test levels and groups of test

cases i.e. integration/acceptance/usability/any other

Test planning is a continuous activity and is performed in all life cycle processes and activities.

Feedback from test activities is used to recognize changing risks so that planning can be

adjusted.

Entry and Exit criteria..

Test estimate: Metrics-based approach (former or similar projects) and Expert-based approach.

Test approach is the implementation of the test strategy for a specific project, which is defined in

the test plans and test designs.

Test approach includes decisions made based on project's goal and risk assessment. It is

base/start point for selecting test design techniques and test types, and for Entry and Exit criteria.

Commercial Off-The-Shelf (COTS) software product vs Custom built products.

Typical approaches include: Analytical approaches (such as risk-based testing), Model-based

approaches (stochastic testing using statistical information about failure rates or usage),

Methodical approaches (such as failure-based), experience-based, checklist-based, failure-based,

characteristic-based, process- or standard-complaint approaches or various agile methodologies,

Dynamic and heuristic approaches (exploratory testing), consultative approaches (driven by

domain experts), Regression-averse approaches (reuse of existing material and standard test

suits)

Different Test approaches may be combined, e.g. risk-based dynamic approach.

********************************

Ravinder Puranam

[email protected]

of 27

5.3. Test Progress monitoring and Control - K2

Terms: Defect density, failure rate, test control, test monitoring, test summary report

5.3.1. Recall common metrics used for monitoring test preparation and execution

5.3.2. Explain and compare test metrics for test reporting and test control (defect found/fixed &

test pass/fail) related to purpose and use - K2

5.3.3. Summarize the purpose and content of the test summary report document according to the

'Standard for Software Test Documentation' - IEEE 829-1998 - K2

The purpose of test monitoring is to provide feedback and visibility about test activities.

Metrics may be used to assess progress against the planned schedule and budget.

Common Test Metrics include: Percentage of work done in test case preparation / work done in

test environment preparation / test case execution / defect information / test coverage / dates of

test milestones, testing costs.

Test reporting is concerned with summarizing information about the testing endeavor, including

what happened during a period of testing, such as dates when exit criteria were met; analyzed

information and metrics to support recommendations and decisions about future actions, such as

assessment of defects remaining, outstanding risks, economic benefit of continued testing, and

level of confidence in tested software.

Outline of test summary report is given in 'Standard for Software Test Documentation - IEEE

829-1998'

Metrics should be collected during and end of test level to assess: adequacy of test objectives for

that test level, adequacy of test approaches taken, effectiveness of testing with respect to

objectives.

Test control describes any guiding or corrective actions taken as a result of information and

metrics gathered and reported. E.g. Making decisions based on information from test monitoring;

Re-prioritizing tests when identified risk occurs; Changing the test schedule due to availability or

unavailability; setting entry criterion to have been re-tested (confirmation testing) by a developer

before accepting them into a build.

********************************

5.4. Confirmation Management - K2

Terms: Configuration management, version control

5.4.1. Summarize how configuration management supports testing - K2

The purpose of configuration management is to establish and maintain the integrity of the

products (components, data and documentation) of the software or system through the project

and product life cycle.

Ravinder Puranam

[email protected]

of 27

Configuration management may involve ensuring:

-- All items of test ware, version controlled, tracking for changes, related to each other and

related to development items so that traceability can be maintained throughout the test process

-- All identified documents and software items are referenced unambiguously in test

documentation.

For testers, configuration management helps to uniquely identify the tested items, test

documents, tests and test harness.

During test planning the configuration management procedures and infrastructure should be

chosen, documented and implemented.

********************************

5.5. Risks and Testing - K2

Terms: Product risk, project risk, risk, risk-based testing

Risks can be defined as the chance of an event, hazard, threat or situation occurring and resulting

in undesirable consequences or a potential problem. The level of risk depends on the adverse

event happening and the impact.

5.5.1. Describe a risk which can be a threat to one/more project objectives - K2

5.5.2. Remember that the level of risks is determined by likelihood and impact

5.5.3. Distinguish between the project and project risks - K2

5.5.4. Recognize typical product and project risks

5.5.5. Describe, using examples, how risk analysis and risk management may be used for test

planning - K2

Project Risks: Project risks are the risks that surround the project's capability to deliver its

objectives, such as:

Organization factors: Skill, training, staff shortages, personnel issues, political issues

(communication issues), improper attitude

Technical issues: problems in defining right requirements, extent to which requirements

cannot be met, test environment not ready, late data conversion, migration planning and

development and conversion/migration tools.

Supplier issues: Failure of a third party, contractual issues.

Product Risks: Potential failure areas in the software or system are known as product risks, as

they are a risk to the quality of the product

Failure-prone software delivered: potential Software/Hardware could cause hard to an individual

or company, poor software characteristics, poor data integrity and quality, software that does not

perform its intended functions.

Risks are used to decide where to start testing and where to test more; testing is used to reduce

the risk of an adverse effect occurring, or to reduce the impact of an adverse effect.

Product risks are a special type of risk to the success of a project, since effectiveness of critical

defect removal and of contingency plans.

Ravinder Puranam

[email protected]

of 27

A risk-based approach to testing provides proactive opportunities to reduce the level of product

risk, starting in the initial stages of a project.

Risk-based testing draws on the collective knowledge and insight of the project stakeholders to

determine the risks and the level of testing required to address those risks.

To ensure the chance of product failure is minimized, risk management activities provide an

approach to: Assess what can go wrong, determine what risks are important to deal with,

implement actions to deal with those risks.

In addition, testing may support the identification of new risks, may help to determine what risks

should be reduced, may lower uncertainty about risks.

********************************

5.6. Incident Management - K3

Terms: Incident logging, incident management, incident report.

5.6.1. Recognize the content of an incident report according to the 'Standard for Software Test

Documentation' - IEEE 829-1998

5.6.2. Write an incident report covering the observation of a failure during testing - K3

Since one of the objectives of testing is to find defects, the discrepancies between actual and

expected outcomes need to be logged as incidents.

An incident must be investigated and may turn out to be a defect.

Incidents and defects should be tracked from discovery and classification to correction and

confirmation of the solution.

In order to manage all incidents to completion, an organization should establish an incident

management process and rules for classification.

Incident reports have following objectives: provide everyone with feedback about problem,

isolation and correction as necessary, provide test leaders a means of tracking the quality of

system under test and progress of testing, provide ideas for test process improvement.

Details of incident report may include:

Data of issue, issuing organization, author; expected and actual results; identification of test item

and environment; Software/System life cycle in which the incident was observed; description of

incident to enable reproduction and resolution, including logs, DB dumps / screenshots; scope /

degree of impact on stakeholders interests; severity; Urgency/priority to fix; Status of incident;

conclusions, recommendations and approvals; Global issues; Change history, such as sequence

of actions taken by project team members with respect to incident to isolate, repair, and confirm

it as fixed; References, including the identity of the test case specification that revealed the

problem.

Structure of an incident report is also covered in the 'Standard for Software Test documentation

(IEEE std 829-1998)

Ravinder Puranam

[email protected]

of 27

******************************************************************************

Chapter 6. Tool support for testing -- 4

******************************************************************************

6.1. Types of Test Tools - K2

Terms: Configuration management tool, coverage tool, debugging tool, dynamic analysis tool,

incident management tool, load testing tool, modeling tool, monitoring tool, performance testing

tool, probe effect, requirements management tool, review tool, security tool, static analysis tool,

stress testing tool, test comparator, test data preparation tool, test design tool, test harness, test

execution tool, test management tool, unit test framework tool.

6.1.1. Classify different types of test tools according to their purpose and to the activities of the

fundamental test process and the software life cycle - K2

6.1.2. Explain the term test tool and the purpose of tool support for testing - K2

A tool is something which makes process and testing easy in each stage of an SDLC.

There are different test tools which help in testing of a system faster as the process is automated.

Requirements Testing tools:

Automated support for verification and validation of requirements models

-- Consistency checking

-- Animation

Static Analysis tools:

Provide information about the quality of software…….Code is examined, not executed

Objective measures

-- Cyclomatic complexity: L-N+2P where L: No. of Edges; N: No. of Nodes & P: No. of

disconnected parts

-- Others: nesting levels, size

Test Design Tools:

Generate test inputs

-- From a formal specification or CASE repository

-- From code (e.g. code not covered yet)

Test data preparation tools:

Data manipulation

-- selected from existing databases or files

-- created according to some rules

-- edited from other sources

Test running tools 1:

-- Interface to the software being tested and Run tests as though run by a human tester

-- Test scripts in a programmable language

-- Data, test inputs and expected results held in test repositories

-- Most often used to automate regression testing

Ravinder Puranam

[email protected]

of 27

Test running tools 2:

Character-based

-- simulates user interaction from dumb terminals

-- capture keystrokes and screen responses

GUI (Graphical User Interface)

-- simulates user interaction for WIMP applications (Windows, Icons, Mouse, Pointer)

-- capture mouse movement, button clicks, and keyboard inputs

-- Capture screens, bitmaps, characters, object states

Comparison tools:

Detect differences between actual test results and expected results

-- Screens, characters, bitmaps

-- Masking and filtering

Test running tools normally include comparison capability

Stand-alone comparison tools for files or databases

Test harness and drivers:

-- Used to exercise software which does not have a user interface (yet)

-- Used to run groups of automated tests or comparisons

-- Often custom-build

-- Simulators (where testing in real environment would be too costly or dangerous)

Performance testing tools:

Load generation

-- drive application via user interface or test harness

-- simulates realistic load on the system & logs the number of transactions

Transaction measurement

-- Response times for selected transactions via user interface

Reports based on logs, graphs of load versus response times

Dynamic analysis tools:

Provide run-time information on software (while tests are run)

-- Allocation, use and de-allocation of resources, e.g. memory leaks

-- flag unassigned pointers or pointer arithmetic faults

Debugging tools:

Used by programmers when investigating, fixing and testing faults

Used to reproduce faults and examine program execution in detail

-- Single-stepping

-- Breakpoints or watch points at any statement

-- examine contents of variables and other data

Coverage measurement tools:

-- Objective measure of what parts of the software structure was executed by tests

-- Code is instrumented in a static analysis pass

-- Tests are run through the instrumented code

Ravinder Puranam

[email protected]

of 27

-- Tool reports what has and has not been covered by those tests, line by line and summary

statistics

-- Different types of coverage: statement, branch, condition, LCSAJ, et al

********************************

6.2. Effective use of tools: Potential benefits and Risks - K2

6.2.1. Summarize the potential benefits and risks of test automation and tool support for testing -

K2

6.2.2. Remember special considerations for test execution tools, static analysis, and test

management tools - K1

Benefits: Repetitive work is reduced, greater consistency and repeatability, objective assessment,

ease of access to information about tests or testing

Risks: Unrealistic expectations for the tool; underestimating the time, cost and effort; over-

reliance on the tool; Neglecting version control of test assets within the tool; Neglecting

relationships and interoperability issues between critical tools; Risk of tool vendor going out of

business; poor support for the tool; unforeseen (inability to support a new platform).

Static analysis tools: Refer Chapter: 3.3

Test management tools:

-- Management of test ware: test plans, specifications, results

-- Project management of the test process, e.g. estimation, schedule tests, log results

-- Incident management tools (may include workflow facilities to track allocation, correction and

retesting)

-- Traceability (of tests to requirements, designs)

Test Execution Tools: Test execution tools execute test objects using automated test scripts. This

type of tool often requires significant effort in order to achieve significant benefits.

Data-driven, keyword-driven are some of the testing approaches in execution tool

Advantages of recording manual tests:

Documents what the tester actually did

-- Useful for capturing ad hoc tests (e.g. end users)

-- May enable software failures to be reproduced

Produces a detailed “script”

-- Records actual inputs

-- Can be used by a technical person to implement a more maintainable automated test

Ideal for one-off tasks

-- Such as long or complicated data entry

Capture test scripts:

Will not be very understandable

-- It is a programming language after all!

-- During maintenance will need to know more than can ever be „automatically commented‟

Ravinder Puranam

[email protected]

of 27

Will not be resilient to many software changes

-- A simple interface change can impact many scripts do not include verification

-- May be easy to add a few simple screen based comparisons

Automation Verification:

There are many choices to be made

-- Dynamic / post execution, compare lots / compare little, resilience to change / bug finding

effective scripts can soon become very complex

-- More susceptible to change, harder to maintain there is a lot of work involved

-- Speed and accuracy of tool use is very important usually there is more verification that can

(and perhaps should) be done

-- Automation can lead to better testing (not guaranteed!)

Effort to automate:

The effort required to automate any one test varies greatly

-- Typically between 2 and 10 times the manual test effort

And depends on:

-- Tool, skills, environment and software under test

-- Existing manual test process which may be:

-- Unscripted manual testing

-- scripted (vague) manual testing

-- scripted (detailed) manual testing

*** Don‟t automate too much long term

********************************

6.3. Introducing a tool into an organization

6.3.1. State the main principles of introducing a tool into an organization

6.3.2. State the goals of a proof-of-concept for tool evaluation and a piloting phase for tool

implementation

6.3.3. Recognize that factors other than simply acquiring a tool are required for good tool support

Main considerations in selecting a tool for an organization include:

-- Assessment of organizational maturity, evaluation against clear requirements and objectives,

proof-of-concept, evaluation of the vendor, identification of internal resources / training needs,

estimation of cost-benefit ratio based on concrete business case.

-- Introducing the selected tool into an organization starts with a pilot project with evaluation,

deciding and assessment.

-- Success factors for deployment of the tool within an organization includes: rolling out the tool,

adapting and improving, providing training, defining usage guidelines, monitoring tool use and

benefits, providing support for test team for a given tool, gathering lessons learned from all

teams.

Ravinder Puranam

[email protected]

of 27

******************************************************************************

Glossary important extracts: Standards

******************************************************************************

- BS 7925-2:1998. Software Component Testing / Cause-Effect graphing

- DO-178B:1992. Software Considerations in Airborne Systems and Equipment

Certification, Requirements and Technical Concepts for Aviation (RTCA SC167)

- IEEE 610.12:1990. Standard Glossary of Software Engineering Terminology

- IEEE 829:1998. Standard for Software Test Documentation

- IEEE 1008:1993. Standard for Software Unit Testing

- IEEE 1012:2004 Standard for Verification and Validation Plans

- IEEE 1028:1997. Standard for Software Reviews and Audits

- IEEE 1044:1993. Standard Classification for Software Anomalies

- IEEE 1219:1998. Software Maintenance

- ISO/IEC 2382-1:1993. Data processing - Vocabulary

- ISO 9000:2005. Quality Management Systems – Fundamentals and Vocabulary

- ISO/IEC 9126-1:2001. Software Product Quality

- ISO/IEC 12207:1995. Software Life Cycle Processes.

- ISO/IEC 14598-1:1999. Software Product Evaluation

Trademarks:

- CMM and CMMI are registered trademarks of Carnegie Mellon University

- TMap, TPA and TPI are registered trademarks of Sogeti Nederland BV

- TMM is a registered service mark of Illinois Institute of Technology

- TMMi is a registed trademark of the TMMi Foundation

ISTQB Ravinder Document

Documents

Transcript of ISTQB Ravinder Document