Post on 21-Jan-2018
Pitfalls and Countermeasures in Software Quality Measurements and Evaluations
Hironori Washizaki washizaki@waseda.jp
Prof., Global Software Engineering Laboratory,
Waseda University
Visiting Prof., National Institute of Informatics
Director, SYSTEM INFORMATION CO., LTD.
Vice-Chair, IEEE CS Japan Chapter
Chair, SEMAT Japan Chapter
Convenor, ISO/IEC/JTC1/SC7/WG20
PC Co-Chair, APSEC 2018 Nagoya!
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ)
Nanjing, Dec 4, 2017
H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
Metric and Measurement• Mapping attributes to values/names
on scales– Quality control by single metric– Estimating metric values by other metrics
• You cannot control what you cannot measure!
Req. Def.
Design Impl. Testing
Func. Spec.
Function point
Module design
Coupling
Source code
ComplexityLOC
Defect report
Defectdensity
Testcase
N. tests passed
Effort, time
Cyclomaticcomplexity
0
0.2
0.4
0.6
0.8
1
0 0.05 0.1 0.15 0.2 0.25 0.3
Re
use
rate
#Header files included
2
Pitfalls and Countermeasures
3
Pitfall CountermeasureNegativeHawthorne effects
Goal-orientationMultidimensional measurements
Organizationmisalignment
Visualization of relationships amongorganizational goals, strategies, andmeasurementsExhaustive identification of rationales
Uncertain futurePrediction incorporating uncertaintyMeasurement program improvement bymachine learning
Self-certified qualityStandard-based evaluationPattern-based evaluation
H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
Pitfalls and Countermeasures
4
Pitfall CountermeasureNegativeHawthorne effects
Goal-orientationMultidimensional measurements
Organizationmisalignment
Visualization of relationships amongorganizational goals, strategies, andmeasurementsExhaustive identification of rationales
Uncertain futurePrediction incorporating uncertaintyMeasurement program improvement bymachine learning
Self-certified qualityStandard-based evaluationPattern-based evaluation
H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
Hawthorne Effect
5nicolasdsampson.com, Observe And Learn: The Magic Of Paying Attentionhttp://nicolasdsampson.com/wp-content/uploads/2012/10/2010_12_06_observe-learn-magic-paying-attention.jpg
Goal-Question-Metric (GQM) Paradigm• Goal-oriented framework for identifying goals and necessary
corresponding metrics• Goal: measurement goals• Question: questions for evaluating goal achievement• Metric: objective or subjective metrics for obtaining
necessary quantitative data to answer questions
Identifyingmetrics
Interpreting measurementresults
Metric
Goal
Question
Collected data
Goal achievement
evaluation
Answer
Measured
value
R. van Solingen, E. Berghout, “The Goal/Question/Metric Method” McGraw-Hill Education, 1999
7
GQM-based Multidimensional Measurements
H. Washizaki, R. Namiki, T. Fukuoka, Y. Harada, H. Watanabe, "A Framework for Measuring and Evaluating Program
Source Code Quality", 8th Int’l Conference on Product Focused Software Development and Process Improvement (Profes2007)
Reliability
Maintain
-ability
Portability
Efficiency
Reusability
Analysability
Changeability
Testability
Goal Metric
Depth of
call graph T
Cyclomatic
Number min
Estimated static
path length min
Measured by
some tools
Measured by
some tools
Are functions
not
complicated?
Question Sub-
Question
Is call-
nesting
not too deep?
Is logic
not too
complex?
IndependentIndependent SpecificSpecific
Are entities
named
properly?
(Scale type: T threshold, min smaller is better, max larger is better)
SEMAT-based Multidimensional measurements
8
Software System
Opportunity Stakeholders
Requirements
Work Team
Way of Working
Customer
Solution
Endeavour
Ivar Jacobson, Pan-Wei Ng, Paul E. McMahon, Ian Spence, Svante Lidman, “The Essence of Software Engineering:
The SEMAT Kernel,” Communications of the ACM, vol.55, no.12, pp.42-49, December 2012.
alpha as Project Measurement
9
State 1
XXXXXXXXXXXXXXXXXX
XXXXXXXXXXX
XXXXXXXXXXXX
State 2 XXXXXXXXXXXXXXXXXX
XXXXXXXXXXX
XXXXXXXXXXXX
State 3
XXXXXXXXXXXXXXXXXX
XXXXXXXXXXX
XXXXXXXXXXXX
…….. …….
Checklist
Adopted/modified from I. Jacobson, et al.: Tutorial: Essence - Kernel and Language for Software
Engineering Practices, ICSE'13
alphaIndicator
1 2 3 4 5
Example from ITA WG on project failuers
10
• An employee in charge of Bank office inquires "registered customer information cannot be browsed from the terminal“.
• It was because a batch processing for the previous day has ended abnormally due to wrong data input.
• It took about two hours to recover the data, and employees at each office had to handle customer inquiry manually.
• For that reason, we had received plenty of complaints from customers who had been waiting for a long time!
H. Washizaki, “Analyzing and refining project failure cases from wider viewpoints by using SEMAT Essence,” Essence
Conference in Seoul, 2017.
11
Delay in service deploymentDelay in service deployment
Huge data inputs manuallyHuge data inputs manually
Wrong data inputWrong data input
Defects in operation manualDefects in operation manual
Misunderstanding of abnormal state as normal state
Misunderstanding of abnormal state as normal state
Problem
Cause
Root causeRequirements: NG “Coherent”
Software System: NG “Demonstrable”
Automate data inputAutomate data input
Countermeasure
Frequent rule-violationsFrequent rule-violationsWork:
NG “Under Control”
Way of Working:NG “In Use”
Review and modify rulesReview and modify rules
Monitor and enforce compliance with rules
Monitor and enforce compliance with rules
Opportunity:NG “Benefit Accrued”
Team:NG “Collaborating”
Stakeholders:NG “Satisfied in Use”
Pitfalls and Countermeasures
12
Pitfall CountermeasureNegativeHawthorne effects
Goal-orientationMultidimensional measurements
Organizationmisalignment
Visualization of relationships amongorganizational goals, strategies, andmeasurementsExhaustive identification of rationales
Uncertain futurePrediction incorporating uncertaintyMeasurement program improvement bymachine learning
Self-certified qualityStandard-based evaluationPattern-based evaluation
H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
Organization misalignment
13
We increase customer satisfaction by quality and
usability improvement!
Top managementWe reduce customer-reported defects by
improving testing process and product maintainability.
Developmentteam We track defect data and
code quality metrics.
Quality assuranceteam
GQM+Strategies
M1: Customer
satisfaction survey
M1: Customer
satisfaction survey
Measurement DataGoals and Strategies
Busin
ess L
evel
Soft
ware
Level
Goal: Increase customer
satisfaction by 10%
Goal: Increase customer
satisfaction by 10%
Strategy: Improve product quality
Strategy: Improve product quality
Strategy: Improve usability of
product
Strategy: Improve usability of
product
Goal: Reduce customer-
reported defects by 20%
Goal: Reduce customer-
reported defects by 20%
No sub-level
goal defined
Strategy: Improve efficiency of
system testing
Strategy: Improve efficiency of
system testing
Strategy: Improve maintainability of
software
Strategy: Improve maintainability of
software
Isolated
strategy
M2: Field defect
data
M2: Field defect
data
M3: Code quality
metrics (McCabe,
coupling, cohesion)
M3: Code quality
metrics (McCabe,
coupling, cohesion)
Isolated
data 14
• Alignment and tracing among goal, strategy and data
Context & Assumption
Context & Assumption
Context-Assumption-Matrix [IEICE’16]
15Takanobu Kobori, Hironori Washizaki, et al., “Exhaustive and efficient identification of rationales using GQM+Strategies with
stakeholder relationship analysis,” IEICE Transactions on Information and Systems, Vol.E99-D, No.9, pp.2219-2228, 2016.
TargetView
HospitalInsurancecompany
Healthsensormaker
・・・
Hospital
Insurancecompany
Health sensormaker
・・・
Insurance company can
offer personalized products
by having medical info…
Context / Assumption
Insurancecompany
Medical info
provider
Personal medical
info.
Customized product
Resource
Goal
depends
16
Insurance
company
G
S
A
S
Health
sensor maker
Medical
info.
provider
Expectation from the view of insurance
company
Expectation from the view of health
sensor maker
Receive medical
info.
Personalized
products developed
Strategy
Insurance product and
fee can be personalized
based on personal
medical info.
Goal
Context / Assumption
Sales of new products
Personal fitness
Confirm personal info.
Management system of
receiver of info.
Secure management
of medical info.
Strategy
Must ensure
personal info.
guideline
Goal
Context / Assumption
Insurancecompany Medical
info. provider
Confirming management
systems
Patient
Getting consent
Conflict, redundancy, complement
Conflict, redundancy, complement
17
Towards clear dependency and alignment
G
S
A
S
Insurancecompany
Medical info.
provider
Personal medical info.
Realizing personalized
insurance
Objective ofmedical info. use
Managingpersonal info.
Clarify objective
of use of medical
info.
Strategy
Establish personal
info. management
system
Strategy
Medical
info.
provider
Secure management
of medical info.
Strategy
Must ensure
personal info.
guideline
Goal
Context / Assumption
Health
sensor maker
Insurance
company
Receive medical
info.
Personalized
products developed
Strategy
Goal
Context / Assumption
Sales of new products
Personal fitness
Insurance product and
fee can be personalized
based on personal
medical info.
S1 S2
S3 S5
S6
S4 S7
S1
S2
S3
S4
S6
S5
S7
Interpretive Structural Modeling (ISM)
18
Impact
S1 S2 S3 S4 S5 S6 S7
S1 1 0 0 0 0 0 0
S2 0 1 0 0 0 0 0
S3 1 1* 1 0 1 0 0
S4 1 1* 0 1 0 0 1
S5 1* 1 1 0 1 0 0
S6 1* 1 0 1 0 1 1*
S7 1* 1 0 1 0 0 1
Pow
er.
Hierarchical structuring
可到達行列S1 S2 S3 S4 S5 S6 S7
S1 1 0 0 0 0 0 0
S2 1 1 0 0 0 0 0
S3 1 0 1 0 1 0 0
S4 1 0 0 1 0 0 1
S5 0 1 1 0 1 0 0
S6 0 1 0 1 0 1 0
S7 0 1 0 1 0 0 1
Relation matrix Reachability matrix
ISM-based Alignment [HICSS’16]
S1
S2
S3
S4・・
S1 S2 S3 S4・・
1
1(1) Decompose into elements
(2)Restructuring
(3)Analysis&alignment
S5
S1 S2
S3
S4
S6 S7
Hierarchical structure
S5
S1 S2 S3
S4 S6 S7
ISM
1
Elements (especially strategies)
11
1
1
S5
S1 S2
S3 S4 S6 S7
GQM+Strategies
G
GG G
Reachability matrix
Conflict specified
• Alignment for single GQM+Strategies model• Future: alignment over areas and stakeholders
Yohei Aoki, Takanobu Kobori, Hironori Washizaki, et al., “Identifying Misalignment of Goals and Strategies across Organizational
Units by Interpretive Structural Modeling,” 49th Hawaii International Conference on System Sciences (HICSS), 2016
Pitfalls and Countermeasures
20
Pitfall CountermeasureNegativeHawthorne effects
Goal-orientationMultidimensional measurements
Organizationmisalignment
Visualization of relationships amongorganizational goals, strategies, andmeasurementsExhaustive identification of rationales
Uncertain futurePrediction incorporating uncertaintyMeasurement program improvement bymachine learning
Self-certified qualityStandard-based evaluationPattern-based evaluation
H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
Uncertain Future
21
Identifying parts that are hard to
maintain…
How large?
ELOC N. functions
Measurement System Improvement by ML
N. Tsuda, et al. Iterative Process to Improve GQM Models with Metrics Thresholds to Detect High-risk Files, SANER 2015
Review
Quality measurement
Machinelearning
Goal
Question
Measurement
22
10 25
300
150
N. functions
ELOC
OK
NG
71
N. functionsELOC
OK
NG
Improvement
X
]]
#Issues
Actual
Predicted
DayKiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and
Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.
Logistic
Gompertz
Non-homogeneous Poisson
process(NHPP)
Software reliability model (SRM)as actionable metric
23
Prediction with uncertainty
0
10
20
30
40
50
60
70
80
90
0 2 4 6 8 10 12 14
#Is
su
es
Time
ActualData
Our model
24Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and
Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.
Prediction with uncertainty
0
10
20
30
40
50
60
70
80
90
0 2 4 6 8 10 12 14
#Is
su
es
Time
ActualData
Our model
-
+
25
Lower (worst case)
Upper (best case)
T1 T2
Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and
Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.
Uncertainty patterns and prediction
26
0 0.5 1
ConstantIncrease Decrease
Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and
Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.
Pitfalls and Countermeasures
27
Pitfall CountermeasureNegativeHawthorne effects
Goal-orientationMultidimensional measurements
Organizationmisalignment
Visualization of relationships amongorganizational goals, strategies, andmeasurementsExhaustive identification of rationales
Uncertain futurePrediction incorporating uncertaintyMeasurement program improvement bymachine learning
Self-certified qualityStandard-based evaluationPattern-based evaluation
H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
Self-Certified Quality
28Image: How to Spot a Fake: Avoiding Degree Mill Scams https://wenr.wes.org/2015/06/spot-fake-avoiding-degree-mill-scams
ISO/IEC 25000 SQuaRE-based Quality Measurement and Benchmark
29
Q1. Any path going through internal servers only?Q2. Any path going through outside servers?Q3. Any P2P communications?…
G. The events or actions cannot be repudiated later through communication channels (paths).
M. Signed communication path ratio= #Signed_paths / #Total_paths
E.g. Non-repudiationWaseda U. Team Vendor
1 Concretize SQuaREmeasurements by GQM
2 Prepare measurement methods: data forms, static analysis, questionnaire, user-testing
3 Conduct code static analysis, user-testing
Fill data forms, questionnaire
4 Measure and evaluate quality
Scores by using percentile
E.g., Top 30% = 0.7
HighLow
#Products
Measured valueH. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on
ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)
21 Japanese Products Measurement
30
Functionalsuitability
Performanceefficiency
Compatibility Usability
Reliability Security Maintainability Portability
Effectiveness Efficiency Satisfaction Freedom fromrisk
Contextcoverage • Low security and compatibility in some products
• Necessary to address these in IoT eraH. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on
ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)
Relationships among characteristics
31
Internal/External Quality Quality in Use
p-value < 0.1p-value < 0.1
性能効率性 互換性 使用性 信頼性 セキュリティ 保守性 移植性 有効性 効率性 満足性 リスク回避性 利用状況網羅性
機能適合性 0. 31 0. 19 -0. 72 0. 37 -0. 05 0. 50 0. 31 -0. 14 0. 52 1. 00 1. 00 1. 00性能効率性 0. 44 0. 24 0. 36 -0. 17 0. 37 0. 32 0. 32 -0. 10 -0. 50 -0. 50 -0. 50 互換性 0. 04 0. 17 -0. 06 0. 36 -0. 04 -0. 14 0. 05 -0. 50 -0. 50 -0. 50 使用性 0. 17 -0. 21 0. 11 0. 44 -0. 09 -0. 20 -1. 00 -1. 00 -1. 00 信頼性 0. 30 0. 41 0. 45 -0. 08 0. 11 1. 00 1. 00 1. 00セキュリティ -0. 06 0. 19 0. 64 -0. 34 0. 50 0. 50 0. 50保守性 0. 26 -0. 29 0. 01 1. 00 1. 00 1. 00移植性 -0. 21 0. 67 0. 50 0. 50 0. 50
有効性 0. 03 -1. 00 -1. 00 -1. 00 効率性 1. 00 1. 00 1. 00満足性 1. 00 1. 00リスク回避性 1. 00
Func.
Perf.
Comp.
Usa.
Relia.
Sec.
Main.
Port.
Effe.
Effic.
Sati.
Free.
Perf. Comp. Usa. Relia. Sec. Main. Port. Effe. Effic. Sati. Free. Cont.
• Negative correlation between usability and functionality.
• Need to adopt user-centered development
H. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on
ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)
Appropriatedesign
Inappropriate design
Security Patterns and Testing
32
Role-based access control (RBAC) pattern
TESEM: Test Driven Secure Modeling Tool[ARES’13][ARES’13][IJSSE’14][ICST’15][Information’16]
33
Security Design Pattern
Problem
Solution
Context
Test design as requirement
! create Actor! create UI :! create Subject..
Test Script Test case
testing
[ARES’13] Validating Security Design Pattern Applications Using Model Testing, Int’l Conf. Availability, Reliability and Security
[ARES’14] Verification of Implementing Security Design Patterns Using a Test Template, Conf. Availability, Reliability and Security
[IJSSE’14] Validating Security Design Pattern Applications by Testing Design Models, Int’l J. Secure Software Engineering 5(4)
[ICST’15] TESEM: A Tool for Verifying Security Design Pattern Applications by Model Testing, IEEE ICST’15 Tools Track
[Information’16] Implementation Support of Security Design Patterns Using Test Templates, Information 7(2)
testing
34
window.onload = setEventHandler;
function setEventHandler() {
$(“reg_type”).onchange = calcPrice;
・・・$(“reg_addcart”).onclick = addCart;
};
function calcPrice() { ・・・ };
function addCart() {
if(isValidInput()) {
reqRunTrans();
} else {
alert(“Invalid user inputs”);
}
};
function reqRunTrans() {
new Ajax.Request(“runTrans.php”, {
method: “GET”, parameters: getParams(),
onSuccess: succeeded });
};
function succeeded() { disableAll();
jumpToConfirm();
};
window.onload = setEventHandler;
function setEventHandler() {
$(“reg_type”).onchange = calcPrice;
・・・$(“reg_addcart”).onclick = addCart;
};
function calcPrice() { ・・・ };
function addCart() {
if(isValidInput()) {
reqRunTrans();
} else {
alert(“Invalid user inputs”);
}
};
function reqRunTrans() {
new Ajax.Request(“runTrans.php”, {
method: “GET”, parameters: getParams(),
onSuccess: succeeded });
};
function succeeded() { disableAll();
jumpToConfirm();
};
35
Finite State Machine Extraction……………………………………………
……………………………………………
function setEventHandler() {
$(“addcart”).onclick=addCart; ……………………………………………
……………………………………………
……………………………………………
……………………………………………
function setEventHandler() {
$(“addcart”).onclick=addCart; ……………………………………………
……………………………………………
onclick
setEventHandler
addCart
E.g. User Action [Mahamoff’06]
“Prevent multiple calls ofspecific user event handler”
Y. Maezawa, K. Nishiura, H. Washizaki, S. Honiden, Validating Ajax Applications Using a Delay-Based Mutation Technique”,
29th IEEE/ACM International Conference on Automated Software Engineering (ASE 2014)
Y. Maezawa, H. Washizaki, Y. Tanabe and S. Honiden , “Automated Verification of Pattern-based Interaction Invariants in Ajax
Applications, 28th IEEE/ACM International Conference on Automated Software Engineering (ASE2013)
Duplicate order
Duplicate order
[Mahamoff’06] M. Mahamoff, “Ajax Design Patterns”, O’Reilly Media Inc., 2006.
36
window.onload = setEventHandler;
function setEventHandler() {
$(“reg_type”).onchange = calcPrice;
・・・$(“reg_addcart”).onclick = addCart;
};
function calcPrice() { ・・・ };
function addCart() {
if(isValidInput()) {
$(“addCart”).disabled = true;
reqRunTrans();
} else {
alert(“Invalid user inputs”);
$(“addCart”).disabled = false;
}
};
function reqRunTrans() {
new Ajax.Request(“runTrans.php”, {
method: “GET”, parameters: getParams(),
onSuccess: succeeded }); };
function succeeded() { disableAll();
jumpToConfirm(); };
window.onload = setEventHandler;
function setEventHandler() {
$(“reg_type”).onchange = calcPrice;
・・・$(“reg_addcart”).onclick = addCart;
};
function calcPrice() { ・・・ };
function addCart() {
if(isValidInput()) {
$(“addCart”).disabled = true;
reqRunTrans();
} else {
alert(“Invalid user inputs”);
$(“addCart”).disabled = false;
}
};
function reqRunTrans() {
new Ajax.Request(“runTrans.php”, {
method: “GET”, parameters: getParams(),
onSuccess: succeeded }); };
function succeeded() { disableAll();
jumpToConfirm(); };
Pitfalls and Countermeasures
37
Pitfall CountermeasureNegativeHawthorne effects
Goal-orientationMultidimensional measurements
Organizationmisalignment
Visualization of relationships amongorganizational goals, strategies, andmeasurementsExhaustive identification of rationales
Uncertain futurePrediction incorporating uncertaintyMeasurement program improvement bymachine learning
Self-certified qualityStandard-based evaluationPattern-based evaluation
SamurAI CodingIPSJ 6th International AI Programing Contest
World FinalMarch 14 2018 Tokyohttp://samuraicoding.info
APSEC 2018 25th Asia-Pacific Software Engineering Conference
NaraDec 4-7 (due: June) PC Chair: H. Washizaki
Int. Journal of Agile and Extreme Software Development(IJAESD)
Editor-in-Chief: H. Washizaki
COMPSAC 201842nd IEEE Computer Society Int’l Conf. Computers, Software & Applications
TokyoJuly 23-27 (due: Jan 15)