CS4723 Software Validation and Quality Assurance Lecture 01 Introduction.
-
Upload
gervais-walsh -
Category
Documents
-
view
241 -
download
0
Transcript of CS4723 Software Validation and Quality Assurance Lecture 01 Introduction.
CS4723Software
Validation and Quality Assurance
Lecture 01Introduction
2
Course Instructor
Name:Dr. Xiaoyin Wang (Sean)
Office: NPB 3.208
Email: [email protected]
Experiences Got my PhD from Peking University, China Did my postdoc in UC Berkeley Worked for Microsoft (.net project), and Ensighta (a startup comp
any at Berkeley with 7-8 people)
3
Introduce yourselves!
4
Course Meetings, Web Pages, etc.
Course Meetings: MWF 11:00pm – 11:45pm
MH 3.04.20
Office Hours:
MW 1:00pm - 2:30pm
Course Web Page: http://xywang.100871.net/CS4723.htm
5
Course Textbooks
Choose from the following reference books Software Testing and Quality Assurance: Theory
and Practice, by Kshirasagar Naik and Priyadarshi Tripathy
Software Testing: Principles and Practice, by Srinivasan Desikan and Gopalaswamy Ramesh
Introduction to Software Testing, by Paul Ammann and Jeff Offutt
6
Course Topics
Software Testing Software Debugging Software Verification Anti-pattern Checking and Refactoring Bug Management Research Progress:
Automatic Oracles Automatic Program Repair
7
Grading Scheme
Final Exam: 40% Assignments: 30% Projects: 20% Course participation and presentations: 10%
8
Final Exam
Time: May 7th: 9:45am to 11:45am
Location: MS 3.04.20
Content Selected important topics in the course
We will have a review during the two classes in the last week
9
Assignments
General Post and due date are in the schedule
Both paper and electronic submissions are acceptable is possible, electronic is preferred
Detailed evaluation plan will be introduced at the post time
Assignments Write unit test cases
To be posted
10
Project: Android UI Testing
2-3 people form a small team
Each team is going to do the UI script testing for an Android App
Steps Explore the Android App
Write Testing Reports
Write Scripts for Regression Testing
Deliverable A testing report and the testing scripts
11
Now, let’s go to the real lecture …
12
Why software quality matters?
Prevalent usage Billions of PCs and mobile phones
Chatting, Surfing, Gaming, …
About 1 million lines of code on a new car, MIT Tech Review 2013
Some software on almost all major or small appliances
ATMs
Counters at shopping centers, restaurants, …
Garage Entrances
…
13
Why software quality matters?
We are so dependent on software for critical tasks Traffic Control
Trains, air traffic, …
Transportation Control Ship / plane / car / elevator /…
Financial transactions Bank, stock, online shopping
Power supplies Electricity, Power stations, gas, …
Medical equipments
14
What can caused by software bugs
From inconvenience to disasters Ugly presentation
Not showing some results or output
Unable to do operation
Lost of a hour’s / day’s / week’s work…
Privacy leaks, lost accounts
Money loss
Injuries, Life-threatening dangers
Disasters
15
Some well-known disasters caused by software bugs
Mars Climate Orbiter ($165M, 1998) Sent to Mars to relay signal from the Mars Lander
Smashed to the planet: failing to convert
between English measure to metric values
Shooting down of Airbus A300 (290 death, 1988) US CG-49 shoot down a Airbus A300
Misleading output of the tracking software
THERAC-25 Radiation Therapy (1985) 2 cancer patients at East Texas Cancer
center received fatal overdoses
Miss-handling of race condition of the
software in the equipment
16
Facts about software bugs
On average, 1-5 bugs per KLOC (thousand lines of code) In mature software
More than 10 bugs in prototypes
Windows2000 35MLOC
63K known bugs at the time of release
2 bugs per KLOC
$59.5B loss due to bugs in US 2002 (estimation by NIST)
It is not feasible to remove all bugs But try to reduce critical bugs
17
Why so many bugs in software?
Errors can happen in any engineering discipline
Software is one of the most error-prone products of all engineering areas Requirements are often vague
Software can be really complex, undecidable problems are everywhere
Result Most product are put into the market with no or very
few errors Almost all software in the market has some number of
bugs (we will see that later)
UTSA CS510318
Aspects of software quality
Dependability availability, reliability, safety
Efficiency processing time, memory utilization, responsiveness
Usability appropriate user interface and adequate
documentation
Maintainability ease of change
UTSA CS510319
Dependability (I)
Availability For how large proportion of time the software
system is working
Recover from a crash?
Reliability For how large proportion of inputs the software is
giving correct outputs
Fault tolerance?
UTSA CS510320
Dependability (II)
Safety For critical software, e.g. insulin control
To what extent the software is threatening people?
Runtime fault detection?
Security To what extent the software is able to protect itself
from malicious inputs
Input sanitization?
UTSA CS510321
Efficiency
Processing time for time-consuming tasks Algorithm Complexity
Optimization
Maximal Memory utilization Avoid memory leaks
Release memory at proper time
Responsiveness Interleave UI threads and back-end threads
UTSA CS510322
Usability
User Interface Looks attractive
Easy to understand and convenient to use
Help Documentation Easy to search and read
Cover all the features of the software
UTSA CS510323
Maintainability
Ease to read Code quality
Comments
Ease to extend Design quality
Suitable design patterns
Ease to transplant Avoid hard coded elements
UTSA CS510324
Absolute Software Quality
Impossible Vague Requirements
Free of bugs can never be verified even for some simple programs
Practice on critical software The NASA example
UTSA CS510325
Each execution handles $4Billion equipments, human lives, dreams No prototypes
420k lines of code, 17 errors in 11 versions (is it a large software?)
Commercial equivalent would have at least 1000 bugs
NASA Example
UTSA CS510326
A third of the effort before coding starts
Long specifications, written down, fully discussed 40k pages of specification (longer than the code)
Change to add GPS support (2500 pages more specification)
Specification is almost pseudo code
NASA Example
UTSA CS510327
When fixing bugs, fix what allowed the mistake Unclear API
Insufficient tests
Improper use of tools
Validation and review at all levels 85% of bugs revealed before testing
NASA Example
UTSA CS510328
Cost 260 people
$32 million
A year
TOO EXPENSIVE!!!
Overkill for normal software
NASA Example
29
Quality Economics
Tradeoff of cost and requirements
30
Approaches to Enhance Software Quality
Bug Prevention Prevent bugs during requirement collection,
designing, coding
Bug Removal Remove bugs after coding, but before
release
Bug Tolerance Runtime mechanisms to avoid known or
unknown bugs in the software
31
Approach to prevent bugs
Requirement Engineering Remove ambiguity in requirements
Formal Specification
Design Reduce dependency between modules
Concentrate cross-cut concerns
Coding Proper coding styles
Well written comments
Avoid misleading interfaces
32
Approach to remove bugs
Testing Feed input to software and run it to see whether its
behavior is as expected
Limitations Impossible to cover all cases
Test oracles (what is expected)
Static checking Identify specific problems (e.g., memory leak) in the
software by scanning the code or all possible paths
Limitations Limited problem types
False positives
33
Approach to remove bugs
Formal Proof Formally prove that the program implements the
specification
Limitations Difficult to have a formal specification The proof cost a lot of human efforts
Inspection Manually review the code to detect faults
Limitations: Hard to evaluate Sometime hard to get progress
34
Approach to tolerate bugs
Exception handling Specify what the program should do when
unexpected results are generated
Spend time on writing exception handlers
Consider exceptions in the exception handler
Redundant code Have redundant modules to avoid runtime bugs by
checking consistency of results
Voting for a correct results when bug happens
35
In our course
Bug prevention: something you should learn in software
engineering course, briefly introduce it at the end of our course
Bug removal: The focus of this course
Bug tolerance We will introduce guidance for exception
handling at the end of our course, redundant code will not be introduced because it is rarely used
36
A short review
Software bugs affect every aspect of our life
Software Bugs may cause severe consequences
Software Bugs are prevalent
Aspects of software quality Dependability, Efficiency, Usability, Maintainability
(Bad smells -> bugs)
Approaches to enhance software quality Bug prevention, Bug removal, Bug avoiding