Pitfalls and Countermeasures in Software Quality Measurements … · Pitfalls and Countermeasures...

Pitfalls and Countermeasures in Software Quality Measurements and Evaluations

Hironori Washizaki [email protected]

Prof., Global Software Engineering Laboratory,

Waseda University

Visiting Prof., National Institute of Informatics

Director, SYSTEM INFORMATION CO., LTD.

Vice-Chair, IEEE CS Japan Chapter

Chair, SEMAT Japan Chapter

Convenor, ISO/IEC/JTC1/SC7/WG20

PC Co-Chair, APSEC 2018 Nagoya!

5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ)

Nanjing, Dec 4, 2017

H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017

mailto:[email protected]

Metric and Measurement• Mapping attributes to values/names

on scales– Quality control by single metric– Estimating metric values by other metrics

• You cannot control what you cannot measure!

Req. Def.

Design Impl. Testing

Func. Spec.

Function point

Module design

Coupling

Source code

ComplexityLOC

Defect report

Defectdensity

Testcase

N. tests passed

Effort, time

Cyclomaticcomplexity

0

0.2

0.4

0.6

0.8

1

0 0.05 0.1 0.15 0.2 0.25 0.3

Re

use

rate

#Header files included

2

Pitfalls and Countermeasures

3

Pitfall CountermeasureNegativeHawthorne effects

Goal-orientationMultidimensional measurements

Organizationmisalignment

Visualization of relationships amongorganizational goals, strategies, andmeasurementsExhaustive identification of rationales

Uncertain futurePrediction incorporating uncertaintyMeasurement program improvement bymachine learning

Self-certified qualityStandard-based evaluationPattern-based evaluation



4








Hawthorne Effect

5nicolasdsampson.com, Observe And Learn: The Magic Of Paying Attentionhttp://nicolasdsampson.com/wp-content/uploads/2012/10/2010_12_06_observe-learn-magic-paying-attention.jpg

http://nicolasdsampson.com/wp-content/uploads/2012/10/2010_12_06_observe-learn-magic-paying-attention.jpg

Goal-Question-Metric (GQM) Paradigm• Goal-oriented framework for identifying goals and necessary

corresponding metrics• Goal: measurement goals• Question: questions for evaluating goal achievement• Metric: objective or subjective metrics for obtaining

necessary quantitative data to answer questions

Identifyingmetrics

Interpreting measurementresults

Metric

Goal

Question

Collected data

Goal achievement

evaluation

Answer

Measured

value

R. van Solingen, E. Berghout, “The Goal/Question/Metric Method” McGraw-Hill Education, 1999

7

GQM-based Multidimensional Measurements

H. Washizaki, R. Namiki, T. Fukuoka, Y. Harada, H. Watanabe, "A Framework for Measuring and Evaluating Program

Source Code Quality", 8th Int’l Conference on Product Focused Software Development and Process Improvement (Profes2007)

Reliability

Maintain

-ability

Portability

Efficiency

Reusability

Analysability

Changeability

Testability

Goal Metric

Depth of

call graph T

Cyclomatic

Number min

Estimated static

path length min

Measured by

some tools

Measured by

some tools

Are functions

not

complicated?

Question Sub-

Question

Is call-

nesting

not too deep?

Is logic

not too

complex?

IndependentIndependent SpecificSpecific

Are entities

named

properly?

(Scale type: T threshold, min smaller is better, max larger is better)

SEMAT-based Multidimensional measurements

8

Software System

Opportunity Stakeholders

Requirements

Work Team

Way of Working

Customer

Solution

Endeavour

Ivar Jacobson, Pan-Wei Ng, Paul E. McMahon, Ian Spence, Svante Lidman, “The Essence of Software Engineering:

The SEMAT Kernel,” Communications of the ACM, vol.55, no.12, pp.42-49, December 2012.

alpha as Project Measurement

9

State 1

XXXXXXXXXXXXXXXXXX

XXXXXXXXXXX

XXXXXXXXXXXX

State 2 XXXXXXXXXXXXXXXXXX

XXXXXXXXXXX

XXXXXXXXXXXX

State 3

XXXXXXXXXXXXXXXXXX

XXXXXXXXXXX

XXXXXXXXXXXX

…….. …….

Checklist

Adopted/modified from I. Jacobson, et al.: Tutorial: Essence - Kernel and Language for Software

Engineering Practices, ICSE'13

alphaIndicator

1 2 3 ４５

Example from ITA WG on project failuers

10

• An employee in charge of Bank office inquires "registered customer information cannot be browsed from the terminal“.

• It was because a batch processing for the previous day has ended abnormally due to wrong data input.

• It took about two hours to recover the data, and employees at each office had to handle customer inquiry manually.

• For that reason, we had received plenty of complaints from customers who had been waiting for a long time!

H. Washizaki, “Analyzing and refining project failure cases from wider viewpoints by using SEMAT Essence,” Essence

Conference in Seoul, 2017.

11

Delay in service deploymentDelay in service deployment

Huge data inputs manuallyHuge data inputs manually

Wrong data inputWrong data input

Defects in operation manualDefects in operation manual

Misunderstanding of abnormal state as normal state

Misunderstanding of abnormal state as normal state

Problem

Cause

Root causeRequirements: NG “Coherent”

Software System: NG “Demonstrable”

Automate data inputAutomate data input

Countermeasure

Frequent rule-violationsFrequent rule-violationsWork:

NG “Under Control”

Way of Working:NG “In Use”

Review and modify rulesReview and modify rules

Monitor and enforce compliance with rules

Monitor and enforce compliance with rules

Opportunity:NG “Benefit Accrued”

Team:NG “Collaborating”

Stakeholders:NG “Satisfied in Use”


12








Organization misalignment

13

We increase customer satisfaction by quality and

usability improvement!

Top managementWe reduce customer-reported defects by

improving testing process and product maintainability.

Developmentteam We track defect data and

code quality metrics.

Quality assuranceteam

GQM+Strategies

M1: Customer

satisfaction survey

M1: Customer

satisfaction survey

Measurement DataGoals and Strategies

Busin

ess L

evel

Soft

ware

Level

Goal: Increase customer

satisfaction by 10%

Goal: Increase customer

satisfaction by 10%

Strategy: Improve product quality

Strategy: Improve product quality

Strategy: Improve usability of

product

Strategy: Improve usability of

product

Goal: Reduce customer-

reported defects by 20%

Goal: Reduce customer-

reported defects by 20%

No sub-level

goal defined

Strategy: Improve efficiency of

system testing

Strategy: Improve efficiency of

system testing

Strategy: Improve maintainability of

software

Strategy: Improve maintainability of

software

Isolated

strategy

M2: Field defect

data

M2: Field defect

data

M3: Code quality

metrics (McCabe,

coupling, cohesion)

M3: Code quality

metrics (McCabe,

coupling, cohesion)

Isolated

data 14

• Alignment and tracing among goal, strategy and data

Context & Assumption

Context & Assumption

Context-Assumption-Matrix [IEICE’16]

15Takanobu Kobori, Hironori Washizaki, et al., “Exhaustive and efficient identification of rationales using GQM+Strategies with

stakeholder relationship analysis,” IEICE Transactions on Information and Systems, Vol.E99-D, No.9, pp.2219-2228, 2016.

TargetView

HospitalInsurancecompany

Healthsensormaker

・・・

Hospital

Insurancecompany

Health sensormaker

・・・

Insurance company can

offer personalized products

by having medical info…

Context / Assumption

Insurancecompany

Medical info

provider

Personal medical

info.

Customized product

Resource

Goal

depends

16

Insurance

company

G

S

A

S

Health

sensor maker

Medical

info.

provider

Expectation from the view of insurance

company

Expectation from the view of health

sensor maker

Receive medical

info.

Personalized

products developed

Strategy

Insurance product and

fee can be personalized

based on personal

medical info.

Goal


Sales of new products

Personal fitness

Confirm personal info.

Management system of

receiver of info.

Secure management

of medical info.

Strategy

Must ensure

personal info.

guideline

Goal


Insurancecompany Medical

info. provider

Confirming management

systems

Patient

Getting consent

Conflict, redundancy, complement

Conflict, redundancy, complement

17

Towards clear dependency and alignment

G

S

A

S

Insurancecompany

Medical info.

provider

Personal medical info.

Realizing personalized

insurance

Objective ofmedical info. use

Managingpersonal info.

Clarify objective

of use of medical

info.

Strategy

Establish personal

info. management

system

Strategy

Medical

info.

provider

Secure management

of medical info.

Strategy

Must ensure

personal info.

guideline

Goal


Health

sensor maker

Insurance

company

Receive medical

info.

Personalized

products developed

Strategy

Goal


Sales of new products

Personal fitness

Insurance product and

fee can be personalized

based on personal

medical info.

S1 S2

S3 S5

S6

S4 S7

S1

S2

S3

S4

S6

S5

S7

Interpretive Structural Modeling (ISM)

18

Impact

S1 S2 S3 S4 S5 S6 S7

S1 1 0 0 0 0 0 0

S2 0 1 0 0 0 0 0

S3 1 1* 1 0 1 0 0

S4 1 1* 0 1 0 0 1

S5 1* 1 1 0 1 0 0

S6 1* 1 0 1 0 1 1*

S7 1* 1 0 1 0 0 1

Pow

er.

Hierarchical structuring

可到達行列S1 S2 S3 S4 S5 S6 S7

S1 1 0 0 0 0 0 0

S2 1 1 0 0 0 0 0

S3 1 0 1 0 1 0 0

S4 1 0 0 1 0 0 1

S5 0 1 1 0 1 0 0

S6 0 1 0 1 0 1 0

S7 0 1 0 1 0 0 1

Relation matrix Reachability matrix

ISM-based Alignment [HICSS’16]

S1

S2

S3

S4・・

S1 S2 S3 S4・・

1

1(1) Decompose into elements

(2)Restructuring

(3)Analysis&alignment

S5

S1 S2

S3

S4

S6 S7

Hierarchical structure

S5

S1 S2 S3

S4 S6 S7

ISM

1

Elements (especially strategies)

11

1

1

S5

S1 S2

S3 S4 S6 S7

GQM+Strategies

G

GG G

Reachability matrix

Conflict specified

• Alignment for single GQM+Strategies model• Future: alignment over areas and stakeholders

Yohei Aoki, Takanobu Kobori, Hironori Washizaki, et al., “Identifying Misalignment of Goals and Strategies across Organizational

Units by Interpretive Structural Modeling,” 49th Hawaii International Conference on System Sciences (HICSS), 2016


20








Uncertain Future

21

Identifying parts that are hard to

maintain…

How large?

ELOC N. functions

Measurement System Improvement by ML

N. Tsuda, et al. Iterative Process to Improve GQM Models with Metrics Thresholds to Detect High-risk Files, SANER 2015

Review

Quality measurement

Machinelearning

Goal

Question

Measurement

22

10 25

300

150

N. functions

ELOC

OK

NG

71

N. functionsELOC

OK

NG

Improvement

X

]]

#Issues

Actual

Predicted

DayKiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and

Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.

Logistic

Gompertz

Non-homogeneous Poisson

process(NHPP)

Software reliability model (SRM)as actionable metric

23

Prediction with uncertainty

0

10

20

30

40

50

60

70

80

90

0 2 4 6 8 10 12 14

#Is

su

es

Time

ActualData

Our model

24Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and


Prediction with uncertainty

0

10

20

30

40

50

60

70

80

90

0 2 4 6 8 10 12 14

#Is

su

es

Time

ActualData

Our model

-

+

25

Lower (worst case)

Upper (best case)

T1 T2

Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and


Uncertainty patterns and prediction

26

0 0.5 1

ConstantIncrease Decrease

Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and



27








Self-Certified Quality

28Image: How to Spot a Fake: Avoiding Degree Mill Scams https://wenr.wes.org/2015/06/spot-fake-avoiding-degree-mill-scams

ISO/IEC 25000 SQuaRE-based Quality Measurement and Benchmark

29

Q1. Any path going through internal servers only?Q2. Any path going through outside servers?Q3. Any P2P communications?…

G. The events or actions cannot be repudiated later through communication channels (paths).

M. Signed communication path ratio= #Signed_paths / #Total_paths

E.g. Non-repudiationWaseda U. Team Vendor

1 Concretize SQuaREmeasurements by GQM

2 Prepare measurement methods: data forms, static analysis, questionnaire, user-testing

3 Conduct code static analysis, user-testing

Fill data forms, questionnaire

4 Measure and evaluate quality

Scores by using percentile

E.g., Top 30% = 0.7

HighLow

#Products

Measured valueH. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on

ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)

21 Japanese Products Measurement

30

Functionalsuitability

Performanceefficiency

Compatibility Usability

Reliability Security Maintainability Portability

Effectiveness Efficiency Satisfaction Freedom fromrisk

Contextcoverage • Low security and compatibility in some products

• Necessary to address these in IoT eraH. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on


Relationships among characteristics

31

Internal/External Quality Quality in Use

p-value < 0.1p-value < 0.1

性能効率性互換性使用性信頼性セキュリティ保守性移植性有効性効率性満足性リスク回避性利用状況網羅性

機能適合性 0. 31 0. 19 -0. 72 0. 37 -0. 05 0. 50 0. 31 -0. 14 0. 52 1. 00 1. 00 1. 00性能効率性 0. 44 0. 24 0. 36 -0. 17 0. 37 0. 32 0. 32 -0. 10 -0. 50 -0. 50 -0. 50 互換性 0. 04 0. 17 -0. 06 0. 36 -0. 04 -0. 14 0. 05 -0. 50 -0. 50 -0. 50 使用性 0. 17 -0. 21 0. 11 0. 44 -0. 09 -0. 20 -1. 00 -1. 00 -1. 00 信頼性 0. 30 0. 41 0. 45 -0. 08 0. 11 1. 00 1. 00 1. 00セキュリティ -0. 06 0. 19 0. 64 -0. 34 0. 50 0. 50 0. 50保守性 0. 26 -0. 29 0. 01 1. 00 1. 00 1. 00移植性 -0. 21 0. 67 0. 50 0. 50 0. 50

有効性 0. 03 -1. 00 -1. 00 -1. 00 効率性 1. 00 1. 00 1. 00満足性 1. 00 1. 00リスク回避性 1. 00

Func.

Perf.

Comp.

Usa.

Relia.

Sec.

Main.

Port.

Effe.

Effic.

Sati.

Free.

Perf. Comp. Usa. Relia. Sec. Main. Port. Effe. Effic. Sati. Free. Cont.

• Negative correlation between usability and functionality.

• Need to adopt user-centered development

H. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on


Appropriatedesign

Inappropriate design

Security Patterns and Testing

32

Role-based access control (RBAC) pattern

TESEM: Test Driven Secure Modeling Tool[ARES’13][ARES’13][IJSSE’14][ICST’15][Information’16]

33

Security Design Pattern

Problem

Solution

Context

Test design as requirement

! create Actor! create UI :! create Subject..

Test Script Test case

testing

[ARES’13] Validating Security Design Pattern Applications Using Model Testing, Int’l Conf. Availability, Reliability and Security

[ARES’14] Verification of Implementing Security Design Patterns Using a Test Template, Conf. Availability, Reliability and Security

[IJSSE’14] Validating Security Design Pattern Applications by Testing Design Models, Int’l J. Secure Software Engineering 5(4)

[ICST’15] TESEM: A Tool for Verifying Security Design Pattern Applications by Model Testing, IEEE ICST’15 Tools Track

[Information’16] Implementation Support of Security Design Patterns Using Test Templates, Information 7(2)

testing

34

window.onload = setEventHandler;

function setEventHandler() {

$(“reg_type”).onchange = calcPrice;

・・・$(“reg_addcart”).onclick = addCart;

};

function calcPrice() { ・・・ };

function addCart() {

if(isValidInput()) {

reqRunTrans();

} else {

alert(“Invalid user inputs”);

}

};

function reqRunTrans() {

new Ajax.Request(“runTrans.php”, {

method: “GET”, parameters: getParams(),

onSuccess: succeeded });

};

function succeeded() { disableAll();

jumpToConfirm();

};





};




reqRunTrans();

} else {


}

};




onSuccess: succeeded });

};


jumpToConfirm();

};

35

Finite State Machine Extraction……………………………………………

……………………………………………


$(“addcart”).onclick=addCart; ……………………………………………

……………………………………………

……………………………………………

……………………………………………


$(“addcart”).onclick=addCart; ……………………………………………

……………………………………………

onclick

setEventHandler

addCart

E.g. User Action [Mahamoff’06]

“Prevent multiple calls ofspecific user event handler”

Y. Maezawa, K. Nishiura, H. Washizaki, S. Honiden, Validating Ajax Applications Using a Delay-Based Mutation Technique”,

29th IEEE/ACM International Conference on Automated Software Engineering (ASE 2014)

Y. Maezawa, H. Washizaki, Y. Tanabe and S. Honiden , “Automated Verification of Pattern-based Interaction Invariants in Ajax

Applications, 28th IEEE/ACM International Conference on Automated Software Engineering (ASE2013)

Duplicate order

Duplicate order

[Mahamoff’06] M. Mahamoff, “Ajax Design Patterns”, O’Reilly Media Inc., 2006.

36





};




$(“addCart”).disabled = true;

reqRunTrans();

} else {


$(“addCart”).disabled = false;

}

};




onSuccess: succeeded }); };


jumpToConfirm(); };





};




$(“addCart”).disabled = true;

reqRunTrans();

} else {


$(“addCart”).disabled = false;

}

};




onSuccess: succeeded }); };


jumpToConfirm(); };


37







SamurAI CodingIPSJ 6th International AI Programing Contest

World FinalMarch 14 2018 Tokyohttp://samuraicoding.info

APSEC 2018 25th Asia-Pacific Software Engineering Conference

NaraDec 4-7 (due: June) PC Chair: H. Washizaki

Int. Journal of Agile and Extreme Software Development(IJAESD)

Editor-in-Chief: H. Washizaki

COMPSAC 201842nd IEEE Computer Society Int’l Conf. Computers, Software & Applications

TokyoJuly 23-27 (due: Jan 15)

http://samuraicoding.info/

Pitfalls and Countermeasures in Software Quality Measurements … · Pitfalls and Countermeasures...

Documents

Transcript of Pitfalls and Countermeasures in Software Quality Measurements … · Pitfalls and Countermeasures...