Reviews Test Analyst

8/6/2019 Reviews Test Analyst

1/54

1

6 Reviews

6.1 Introduction

6.2 The Principles of Reviews

6.3 Types of Reviews

6.4 Introducing Reviews6.5 Success Factors for Reviews


2/54

2

6 Reviews6.1 Introduction

A successful review process requires planning,participation and follow-up.

Test managers have responsibilities for the planningand follow-up activities.

Testers / Test Analysts must be active participantsin the review process, providing their unique views.They should have formal review training to better

understand their respective roles in any technicalreview process.


3/54

3

6 Reviews

6.1 Introduction

6.2 The Principles of Reviews




4/54

4

Reviews Learning Objectives

The Learning Objective for the video session is to: (K2) Compare review types with each other and show

their relative strengths, weaknesses and fields of use.

The concept of checklists will also be introduced but a

practical session on their use is deferred to the Liveclassroom event

(K3) Use a review checklist to verify code andarchitecture from a testers perspective

(K3) Use a review checklist to verify requirements anduse cases from a testers perspective


5/54

5

Quiz 6.2a6.2 The Principles of Reviews

a) To find defects

b) Not only to find defects but find defects earlier inthe life cycle

c) Not only to find defects but find defects earlier inthe life cycle and remove the causes from the

process

Which of the following best describes the benefits ofReviews?


6/54

6

Quiz 6.2a (suggested answer)6.2 The Principles of Reviews

a) To find defects

b) Not only to find defects but find defects earlier in the life cycle

c) Not only to find defects but find defects earlier in the life cycleand remove the causes from the process

The answer is (c).

a) Not just find defects

b) Not just find defects early but !

Which of the following best describes the benefits ofReviews?


7/54

7

Review6.2 The Principles of Reviews

Review: a type of static testing

an evaluation of software elements or project statusto ascertain discrepancies from planned results andto recommend improvement

(IEEE 1028 Standard for Software Reviews)

Objective:

not just to find defects, but also, to find defectsearlier in the life cycle and to remove the causesfrom the process


8/54

8

Why reviews?6.2 The Principles of Reviews

Objective: to find defects, but also, to find defects earlier and to

remove the causes from the process

Early defects are often the most important

defects have the characteristic to multiply themselvestop-down

cost of rework rises exponentially

60% of the defects have already been made before

coding/implementation has started


9/54

9

Cost Escalation Model6.2 The Principles of Reviews

Cost of fixing errors escalates as we move project towards field use


10/54

10

More Benefits6.2 The Principles of Reviews

Other defects (not failures!!) are found compared todynamic testing

e.g. deviations from std.s, missing requirements,

insufficient maintainability

Development productivity improvements Reduced lead-times

Reduced dynamic testing costs & time(less test cycles and re-testing)

Improved communication


11/54

11

Code Reviews & Dynamic Testing6.2 The Principles of Reviews

Review ofsource code (faults)

Dynamic testing(failures)


12/54

12

Code Reviews & Dynamic Testing6.2 The Principles of Reviews

Code reviews are more cost effective than dynamictesting

dynamic testing done well finds fewer defects thanstatic testing done well

But code reviews cannot find all the faults, e.g.:

Complier introduced errors

Processor specific errors

Environment specific problems What about Static Analysis?


13/54

13

IEEE Std 1028-19976.2 The Principles of Reviews

The ISTQB syllabus is broadly based on thisstandard

The syllabus identifies key principles and provides acomparative description of commonly used formalreview types including:

Management review

Audit

Walkthrough Technical review

Inspection


14/54

14

Basic Review Process6.2 The Principles of Reviews

PlanningKick-off

Individualpreparation

Reviewmeeting

Rework/follow-up

Metrics/sign-off


15/54

15

Source Documents6.2 The Principles of Reviews

Requirements & Specifications

Standards Rules /checklists

Reviews are more effectivewhen source documents areavailable

Without source documents areview is only likely to find errorsof omission and internalinconsistency in the productdocument


16/54

16

Checklists6.2 The Principles of Reviews

A list of questions

A negative answer identifiesan error

Interpret the rules Identify frequently made

mistakes

Must be maintained


17/54

17

Checklist questions6.2 The Principles of Reviews

Does this document conform to your standards andhouse-style?

Is this document clear and unambiguous to theintended readership?

Are all statements consistent with other statementsin the same or related documents?


18/54


19/54

19

Possible Review Outcomes6.2 The Principles of Reviews

Is it fit for purpose?

Possible outcomes:

Product can be used as is;

Issues need to be dealt with,but a further review is notrequired;

Issues need to be dealt withand a further review is required


20/54

20

Review Roles6.2 The Principles of Reviews

Leader (for the completereview process)

Author

Scribe

Reviewers

Presenter

Chair


21/54

21

Quiz 6.2b6.2 The Principles of Reviews

What are the benefits of reviews compared todynamic testing and to other static testingtechniques?

Five minutes to note down your ideas!


22/54

22

Quiz 6.2b (suggested answer)6.2 The Principles of Reviews

Benefits of reviews compared to dynamic testingand to other static testing techniques include:

Help to improve communications via the reviewmeeting participation

Help to increase our understanding of key aspects ofthe application via walkthroughs

Provide an opportunity to check the effectiveness ofproject management via management review

Provide the evidence to check process via audit

Can be very cost effective compared with dynamictesting and static analysis earlier in SDLC

Can validate requirements conformance whereasstatic analysis checks intrinsic code quality


23/54

23

Key Learning Points6.2 The Principles of Reviews

You should now be able to

Explain the benefits of reviews compared todynamic testing and other static testing techniques


24/54

24

6 Reviews

6.1 Introduction6.2 The Principles of Reviews




25/54

25

Review Types and Objectives6.3 Types of Review

Review types (from Foundation): Informal

Walkthrough

Technical review

Inspection

Other review types include Management Reviewand Audit

Choice of review depends upon: Objectives of the review

Culture and maturity of the organization performingthe review


26/54

26

Learning Objectives6.3 Types of Review

Compare review types with each other and showtheir relative strengths, weaknesses and fields ofuse. (K2)


27/54

27

Types of Formal Review6.3 Types of Review

Formal Review types: Management review

Audit

Previously covered in Foundation

Walkthrough

Technical review

Inspection


28/54

28

Types of Formal Reviews6.3 Types of Review

Management review (IEEE 1028-1997):A systematic evaluation of a software acquisition,

supply, development, operation, or maintenanceprocess performed by or on behalf of management

that monitors progress, determines the status of plansand schedules, confirms requirements and their systemallocation, or evaluates the effectiveness ofmanagement approaches used to achieve fitness for

purpose.Strength: forum for monitoring and control

Weakness: may be biased by management over-control

Field of use: progress reporting, problem solving


29/54

29

Key Characteristics Management Review6.3 Types of Review

To monitor progress, assess status, and makedecisions Carried out by or for managers having direct responsibility for the

project or system Carried out by or for a stakeholder or decision maker, e.g. a

higher level manager or director Checks consistency with and deviations from plans, or adequacy

of management procedures Includes assessment of project risks Outcome includes action items and issues to be fixed Participants expected to prepare, decisions are documented

Note that test managers should participate in and mayinstigate management reviews of testing progress.


30/54

30

Types of formal reviews : Audits6.3 Types of Review

Audit: An independent examination of a software product,software process, or set of software processes to assesscompliance with specifications, standards, contractualagreements, or other criteria (IEEE 1028:1997)

Strengths: Audits can be extremely formal, and are usually

performed to demonstrate conformance to some set of expectations,most likely an applicable standard or a contractual obligation, e.g.

ISO 9000:2000 QA Audit

CMMI /TMMi

Weaknesses: Audits are the least effective at revealing defects inthe product

Field of use: Evaluate/find defects in/conformance to the process!


31/54

31

Key Characteristics Audit6.3 Types of Review

Main purpose: provide independent evaluation ofcompliance to processes, regulations, standards etc.

A lead auditor is responsible for the audit and acts asthe moderator

Auditors collect evidence of compliance throughinterviews, witnessing and examining documents

Outcome includes observations, recommendations,corrective actions and a pass/fail assessment

R i t W lkth h


32/54

32

Review type : Walkthrough6.3 Types of Review

Walkthrough (IEEE 1028-1997)A static analysis technique in which a designer orprogrammer leads members of the development team andother interested parties through a software product, and the

participants ask questions and make comments aboutpossible errors, violation of development standards, and otherproblems.

Strength: easy to do, little prep, also educational

Weakness: less evidence (review records)

Field of use: Design trade-offs and proof of concepts

R i t T h i l R i


33/54

33

Review type : Technical Review6.3 Types of Review

Technical review (IEEE 1028-1997)A systematic evaluation of a software product by a team of

qualified personnel that examines the suitability of thesoftware product for its intended use and identifies

discrepancies from specifications and standards. Technicalreviews may also provide recommendations of alternativesand examination of various alternatives.

Strength: Technical rigour, evidence record, discuss alternatives

Weakness: analysis paralysis! Time consuming!

Field of use: Technical solution, integrity!

Review type : Inspection


34/54

34

Review type : Inspection6.3 Types of Review

Inspection (IEEE 1028-1997)A visual examination of a software product to detect andidentify software anomalies, including errors and deviationsfrom standards and specifications. Inspections are peerexaminations led by impartial facilitators who are trained

in inspection techniques. Determination of remedial orinvestigative action for an anomaly is a mandatory element ofa software inspection, although the solution should not bedetermined at the inspection meeting.

Strength: proven track record, review records!Weakness: needs formal training, checklists & time

Field of use: critical systems, process/product improvement

R i Q i 6 3


35/54

1. Agreeing control activities to manage reduced time scales

2. Identifying defects and process improvements in a requirements document

3. Checking whether a team has conformed to a stated process flow

4. Assessing the effectiveness of metrics and reporting on a project

5. Getting feedback when time is short but you need quick ideas

6. Agreeing solutions to problems in a system architecture

7. Introducing a new business process to a group of business users

8. Checking whether a design process is a suitable choice for a team to use

35

Reviews Quiz 6.36.3 Types of Review

Suggest a suitable review type for each situation:

R i Q i 6 3 S t d A


36/54

36

Reviews Quiz 6.3 Suggested Answer6.3 Types of Review

1. Agreeing control activities to manage reduced time scales Managementreview

2. Identifying defects and process improvements in a requirements documentInspection

3. Checking whether a team has conformed to a stated process flow Audit4. Assessing the effectiveness of metrics and reporting on a project

Management review

5. Getting feedback when time is short but you need quick ideas Informalreview or a walkthrough

6. Agreeing solutions to problems in a system architecture Technical review7. Introducing a new business process to a group of business users

Walkthrough

8. Checking whether a design process is a suitable choice for a team to useAudit for example for compliance to a specific CMMi level

Suggest a suitable review type for each situation:

R i Q i 6 2


37/54

a) Planning, preparation, kick-off, meeting & follow-up

b) Kick-off, planning, preparation, meeting, rework &follow-up

c) Preparation, planning, kick-off, meeting, rework &follow-up

d) Planning, kick-off, preparation, meeting, rework &follow-up

37

Reviews Quiz 6.2c6.2 The Principals of Reviews - revision

Can you remember: in what order would thephases of a formal review usually occur?

R i Q i 6 2 S t d A


38/54

38

Reviews Quiz 6.2c Suggested Answer6.2 The Principals of Reviews - revision

a) Planning, preparation, kick-off, meeting & follow-up

b) Kick-off, planning, preparation, meeting, rework &follow-up

c) Preparation, planning, kick-off, meeting, rework &follow-up

d) Planning, kick-off, preparation, meeting, rework& follow-up

Can you remember: in what order would thephases of a formal review usually occur?

K L i P i t


39/54

39

Key Learning Points6.3 Types of Review

You should now be able to: Compare review types with each other and show

their relative strengths, weaknesses and fields ofuse. (K2)

A i t 6 4 I t d i R i


40/54

40

Assignment 6.4 Introducing Reviews

1. What is your current experience with reviews?2. What review types have you participated in?

3. What worked well?

4. Do you use any check lists?

if so, document examples

5. What is your typical reading rate?

6. What suggestion for improvements do you have?

7. If you have not used reviews where could you applythem?

Document your response ready for the virtual classroomsession.


41/54

41

6 Reviews




Success Factors


42/54

42

Success Factors6.5 Success Factors for Review

Technical Factors Organizational Factors

People Factors

Success Factors Technical Factors (1/2)


43/54

43

Success Factors - Technical Factors (1/2)6.5 Success Factors for Review

Ensure the defined process for the review type isfollowed correctly, particularly for more formal types ofreview such as inspection

Record the costs of reviews (including time spent) andbenefits achieved

Review early drafts or partial documents to identifypatterns of defects before they are built into the wholedocument

Ensure that the documents or partial documents arereview-ready before starting a review process (i.e. applyentry criteria)

Use organization-specific checklists of common defects

Success Factors Technical Factors (2/2)


44/54

44

Success Factors - Technical Factors (2/2)6.5 Success Factors for Review

Use more than one type of review, depending onobjectives: document cleanup, technical improvement,transfer of information, or progress management

Review or inspect documents on which importantdecisions will be made, for example, inspect a proposal,contract or high level requirement before a managementreview authorizing major expenditure

Sample a limited subset of a document for assessment

Encourage finding the most important defects: focus oncontent not format

Continuously improve the review process!

Success Factors Organizational Factors (1/2)


45/54

45

Success Factors - Organizational Factors (1/2)6.5 Success Factors for Review

Ensure managers allow adequate time to be spent onreview activities, even under deadline pressure

Remember, time and budget spent are not in proportionto the defects found

Allow adequate time for rework of defects identified byreviews

Never use the metrics from reviews for individualperformance appraisal

Ensure that the right people are involved in the differenttypes of review

Provide training in reviews, particularly the more formalreview types

Success Factors Organizational Factors (2/2)


46/54

46

Success Factors - Organizational Factors (2/2)6.5 Success Factors for Review

Support a test analyst forum to share experience,checklists and ideas for improvement

Ensure everyone participates in reviews and everyonehas their own documents reviewed

Apply the strongest review techniques to the mostimportant documents

Ensure a well-balanced review team of people withdifferent skills and backgrounds

Support process improvement actions to addresssystemic problems

Recognize improvements gained through the reviewprocess

Success Factors People Factors


47/54

47

Success Factors - People Factors6.5 Success Factors for Review

Educate stakeholders to expect that defects will be foundand to allow time for rework and re-review

Ensure the review is a positive experience for the author

Welcome the identification of defects in a blame-free

atmosphere Ensure comments are constructive, helpful and

objective, not subjective

Do not review if the author does not agree or is not

willing Encourage everyone to think deeply about the most

important aspects of the documents being reviewed

Quiz 6 5a


48/54

48

Quiz 6.5a6.5 Success Factors for Reviews

A group of test analysts who has been on a testingcourse return enthusiastic about implementing reviewson their project.

What factors do they need to explain to their managerto support reviews being introduced successfully?

Quiz 6 5a (suggested answer)


49/54

49

Quiz 6.5a (suggested answer)6.5 Success Factors for Reviews

A group of test analysts who has been on a testing course returnenthusiastic about implementing reviews on their project.

What factors do they need to explain to their manager to supportreviews being introduced successfully?

Technical Factors: for example, make sure everyone agrees and

understand the review process, make sure the documents arereviewed at the right time, use the right types of review

Organizational Factors: for example, management support in timeand budget, use of metrics to measure efficiency and effectivenessof reviews (Return on Investment) but NOT measurement of people

People Factors: for example, helping the motivation of the authorand reviews by use of objective reporting of defects, helpstakeholders to understand why the reviews are helping

Quiz 6 5b - Checklists


50/54

50

Quiz 6.5b - Checklists6.5 Success Factors for Reviews

A requirements document has the paragraph:For all new investment products, whenever a newaccount is opened the customer record must beupdated with the new account number and investment

product code. Please note that for joint accounts bothclient records must be updated with the investmentcode.

What observations do you have and what type ofchecklist points might help you identify defects?

Quiz 6 5b Checklists (Suggested Answer)


51/54

51

Quiz 6.5b Checklists (Suggested Answer)6.5 Success Factors for Reviews

A requirements document has the paragraph:(ref?)Forallnew investment products, whenever a new account isopened thecustomerrecord must be updated with the new accountnumber andinvestment product code. Please note that for jointaccounts bothclientrecords must be updated with theinvestment code.

The words in red are potential defects. For example, is the investmentproduct code the same data item as the investment code? Is thecustomer record the same as the client record? Checklists would helpus identify items for consistency, completeness, accuracy, ambiguity,adherence to standards and traceability. Examples are

Consistency

is each data item uniquely and consistently named?

Traceability do the requirements have unique references so they can betracked when they change? Is there sufficient traceability to allow us totrack when requirements have been tested?

Accuracy and ambiguity the words ALL and NEVER are warning signsdo we mean that?

A i 6 5


52/54

52

Assignment 6.5

As a Test Analyst you should be using checklists to : Verify code and architecture from a testers

perspective

Verify requirements and use cases from a testers

perspective Thinking about the quiz you have just done,

investigate the topic of checklists for the aboveareas and produce what you believe to be

appropriate checklists Have your response ready for the Live Virtual

Classroom session

Conclusion


53/54

53

Conclusion6 Reviews

Reviews are an effective and efficient means toimprove the quality of the software product

requirements, specification, code, test specifications

There is more than inspection

Objective determines the review type

Lots of proof and knowledge available

JUST DO IT!

End of Chapter 6 Summary


54/54

End of Chapter 6 Summary6 Reviews


6.3 Types of Reviews (1 LO)

6.4 Introducing Reviews

6.5 Success Factors for Reviews

The other 2 LOs (K3) for use of checklists are

covered further in your assignments and in theVirtual Classroom

A table of review types is in your Reader.

ReaderChapter 6

Reviews Test Analyst

Documents

Transcript of Reviews Test Analyst