Quality assurance and testing of quality attributes
-
Upload
softwarecentral -
Category
Documents
-
view
2.262 -
download
0
description
Transcript of Quality assurance and testing of quality attributes
![Page 1: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/1.jpg)
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Quality assurance of non-functional requirements
Mika MäntyläTKK/SoberIT/SPRG
T-76.5650 Software engineering seminar
![Page 2: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/2.jpg)
Mika Mäntylä 2
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Outline
IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by
code review, pair-programming exploratory testing
Practical approach
![Page 3: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/3.jpg)
Mika Mäntylä 3
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
QA of …
ReliabilityInteroperability
Performance
Usability
Scalability
Security
Availability
Maintainability
Modifiability
Testability
Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
![Page 4: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/4.jpg)
Mika Mäntylä 4
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
… with
TDDModule Testing
Load testing
Exploratory testing
Bounce checker
Test case based testing
Code review
Load tester
Pair-programming
GUI robot
Unit testing
Functional testing
Acceptance testing
External beta-testers
Usability testing
Static analysis
Iteration Demo
Structural testing
![Page 5: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/5.jpg)
Mika Mäntylä 5
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Qualities Methods
![Page 6: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/6.jpg)
Mika Mäntylä 6
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
![Page 7: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/7.jpg)
Mika Mäntylä 7
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
![Page 8: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/8.jpg)
Mika Mäntylä 8
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Qualities and methods have many to many relationship
![Page 9: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/9.jpg)
Mika Mäntylä 9
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Relationships between qualities and methods are poorly understood
![Page 10: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/10.jpg)
Mika Mäntylä 10
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Method may not be the most important factor affecting a quality
![Page 11: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/11.jpg)
Mika Mäntylä 11
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Specific methods for specific qualities
![Page 12: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/12.jpg)
Mika Mäntylä 12
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Summary
Qualities and methods have many to many relationship
Often relationships between qualities and methods are poorly understood
Method may not be the key factor when assuring particular quality Code review’s effect on security
Largely dependent on the reviewer’s instructions
Specific methods exist for some qualities Usability testing Load testing
![Page 13: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/13.jpg)
Mika Mäntylä 13
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Outline
IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by
code review, pair-programming exploratory testing
Practical approach
![Page 14: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/14.jpg)
Mika Mäntylä 14
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
QA of performance Requirements analysis
Estimate data volumes and user volume & behavior Peak usage vs. Regular usage Realistic data volumes
Acceptable response time Hardware restrictions
With clustered server you can always buy more servers with mobile phone you cannot
Design & Code Testability of the application with load testing tools Design solutions
Bottlenecks, key algorithms, clustering Testing
Data gathered in RA is crucial Automate and parameterizes scripts Measure Use profiler to identify bottlenecks
![Page 15: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/15.jpg)
Mika Mäntylä 15
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Nixon B. A., 2000
![Page 16: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/16.jpg)
Mika Mäntylä 16
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
QA of Usability
Usability – Quality in Use The capability of the software to be understood learned used
and liked by the user under specified conditions (ISO 9126-1)
Understand context and users Design for Usability
Field observation, interviews, prototypes
Usability reviews, e.g. Heuristic evaluation Cognitive walkthrough with task scenarios
Usability testing, e.g. Thinking aloud protocol in usability lab Co-discovery (Pair usability testing) Surveys and questionnaires
![Page 17: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/17.jpg)
Mika Mäntylä 17
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Outline
IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by
code review, pair-programming exploratory testing
Practical approach
![Page 18: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/18.jpg)
Mika Mäntylä 18
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Quality assurance of evolvability
DesignCompany Standards / Coding conventionsCode inspectionsAssessing software evolvability with tools Improving software evolvabilityKnowledge of the system Environment support
![Page 19: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/19.jpg)
Mika Mäntylä 19
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 1 for evolvability: Design
Design for maintainable software Principles
Low and loose coupling System level, E.g.
No database specific functionality Subsystem level, E.g.
Clear separation of responsibilities Interaction through Interfaces
High cohesion Information hiding
Most probable change scenarios should be evaluated Different software component have, e.g. GUI vs. Encryption
different lifetimes different modification rates
Collaborative design Design review
![Page 20: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/20.jpg)
Mika Mäntylä 20
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 2 for evolvability: Coding conventions Program layout has great impact to program
comprehension Documentation
Naming of code elements Should reflect their intent Naming standard
Commenting Studies have shown that too much comments are also harmful ! Use when naming does not clearly state the intent
e.g. “Check needed because of bug in Java virtual machine”
Structure – Rules for acceptable structure Explicit metrics
“Do not write methods over 100 lines” Or heuristics
“Make sure all methods perform single explicit task”
![Page 21: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/21.jpg)
Mika Mäntylä 21
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 3 for evolvability: Code review
Spreads knowledge of the system Increases evolvability inderectly
Distribution of CR issue (performance under functionality)
0.00 %
10.00 %
20.00 %
30.00 %
40.00 %
50.00 %
60.00 %
70.00 %
80.00 %
90.00 %
Evolvability Funtionality FP
Industry case Q
Industry case S
Students
![Page 22: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/22.jpg)
Mika Mäntylä 22
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 4 for evolvability: Tools Code metrics
E.g. Lines of code, number of coupling, execution paths
Quantify the vague concept of evolvability
Predict future development effort
Also industrial adoption Reverse-engineering
i.e. identify the system's components and their interrelationships, e.g. code -> UML
Analyze whether the design matches the actual source
Metrics Pro’s
Provides objective numbers Does not require in depth
understanding of the system Con’s
Does not say anything about the correctness (quality)
e.g. Does the class have the correct associations
Reverse-engineering Pro’s
Provides higher level view of the source code
Can be compared with the intended design
Con’s Requires in depth
understanding of the system
![Page 23: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/23.jpg)
Mika Mäntylä 23
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 5 for evolvability: Improve
Terms Refactoring - Controlled way to improve the software’s
structure without changing its observable behavior Re-engineering - Examination and alteration of a subject
system to reconstitute it in a new form and the subsequent implementation of the new form
Re-writing – Recreate the product starting with clean slate All of the above can come handy during software evolution
Refactoring - continuous activity performed by all software developers
Re-engineering - when current system structure is no longer usable
Rewriting When re-engineering is not enough When moving to different new technology/language
![Page 24: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/24.jpg)
Mika Mäntylä 24
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 6 for evolvability: Knowledge distribution
Further development is harder, if expertise is lost Expertise = solutions and locations to domain problems,
domain knowledge, software architecture, design decisions
Good HRM policy can help maintain knowledge Regardless there is risk of losing the people with
knowledge Retirement, moving to area, dying, changing career etc.
Strategies to keep vital knowledge of the system Documenting Spreading knowledge to several people
![Page 25: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/25.jpg)
Mika Mäntylä 25
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 7 for evolvability: Environment support
Good unit test harness increases evolvability Increases courage to
improve software structure add new features
Reduces developers test efforts
Other environment support can have similar effect “We don't need unit test as we have excessive number
of testers”
![Page 26: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/26.jpg)
Mika Mäntylä 26
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Quality assurance of evolvability
DesignCompany Standards / Coding conventionsCode inspectionsAssessing software evolvability with tools Improving software evolvabilityKnowledge of the system Environment support
![Page 27: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/27.jpg)
Mika Mäntylä 27
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Outline
IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by
code review, pair-programming exploratory testing
Practical approach
![Page 28: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/28.jpg)
Mika Mäntylä 28
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Code review’s effect to qualities
Spreads knowledge of the system Increases evolvability inderectly
Distribution of CR issue (performance under functionality)
0.00 %
10.00 %
20.00 %
30.00 %
40.00 %
50.00 %
60.00 %
70.00 %
80.00 %
90.00 %
Evolvability Funtionality FP
Industry case T
Industry case S
Students
![Page 29: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/29.jpg)
Mika Mäntylä 29
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Pair programming’s effect to qualities
Increases understandabilityMakes design better (evolvability)
![Page 30: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/30.jpg)
Mika Mäntylä 30
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Exploratory testing’s effect to qualaties
Single study: Exploratory testing (ET) vs Test Case Testing (TCT)
ET found more usability and GUI issues TCT found more performance issues
Type ET TCTET/TCT
(%) Total
Documentation 8 4 200 % 12
GUI 70 49 143 % 119
Inconsistency 5 3 167 % 8
Missing function 98 96 102 % 194
Performance 39 41 95 % 80
Technical defect 54 66 82 % 120
Usability 19 5 380 % 24
Wrong function 263 239 110 % 502
Total 556 503 111 % 1059
![Page 31: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/31.jpg)
Mika Mäntylä 31
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Outline
IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by
code review, pair-programming exploratory testing
Practical approach
![Page 32: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/32.jpg)
Mika Mäntylä 32
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Teams with different goals
Rank effort
Rank statem.
Rank memory
Rank clarity
Rank output
Team1 (effort)
1 4 4 5 3
Team2 (statem.)
2-3 1 2 3 5
Team3 (memory)
5 2 1 4 4
Team4 (clarity)
4 3 3 2 2
Team5 (output)
2-3 5 5 1 1
![Page 33: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/33.jpg)
Mika Mäntylä 33
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Practical approach
Functionality Reliability Usability Efficiency Maintainability Portability
Code Review 1 2 3 2 1 1
Test sessions 1 2 1 2 3 3
Load Tests 2 1 3 1 3 3
Multi-platform build 3 2 3 3 2 1
Company specific list of practices
Company specific list of quality attributes, e.g.:• Only developing Windows application get rid of Portability• Developing crypto library remove Usability take in Security
![Page 34: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/34.jpg)
Mika Mäntylä 34
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Summary
Many quality attributes and many quality assurance methods exist Lecture series instead of single lecture?
Plenty of research on assuring certain qualities Usability, Security, Evolvability (=Maintainability)
Lack of research on qa methods effects on qualities Effects of pair-programming, exploratory testing to
qualities
Exists specific methods for specific qualities Usability testing
![Page 35: Quality assurance and testing of quality attributes](https://reader033.fdocuments.net/reader033/viewer/2022061113/5459120db1af9fcf338b5689/html5/thumbnails/35.jpg)
Mika Mäntylä 35
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Topics
Consulting plan for a companyHow to test Unit tests effect to quality X