12 Reliability QA Standards
-
Upload
api-3775463 -
Category
Documents
-
view
693 -
download
5
Transcript of 12 Reliability QA Standards
![Page 1: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/1.jpg)
Software Reliability
![Page 2: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/2.jpg)
Organization of this Lecture:
• Introduction. • Reliability metrics• Reliability growth modelling• Statistical testing• Summary
![Page 3: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/3.jpg)
Software Reliability – Alternate Definitions
• Informally denotes a product’s trustworthiness or dependability.
• Probability of the product working “correctly” over a given period of time.
![Page 4: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/4.jpg)
Software Reliability
• a software product having a large number of defects is unreliable.
• reliability of a system improves if the number of defects is reduced.
![Page 5: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/5.jpg)
Reliability
• User’s concern– An important attribute determining
the quality of the product.
• Demand for quantitative estimation of reliability before making buying decision.
![Page 6: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/6.jpg)
Software Reliability Estimation
• Tricky problem– Several factors contribute to making
measurement of software reliability difficult.
![Page 7: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/7.jpg)
Problems in Reliability Measurements
• Errors do not cause failures at the same frequency and severity.– measuring latent errors alone not
enough
• The failure rate is observer-dependent
![Page 8: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/8.jpg)
Difficulties in Software Reliability Measurement (1)
• No simple relationship observed between system reliability and the number of latent software defects.
• Removing errors from parts of software which are rarely used makes little difference to the perceived reliability.
![Page 9: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/9.jpg)
The 90-10 Rule
• Experiments from analysis of behavior of a large number of programs:– 90% of the total execution time is
spent in executing only 10% of the instructions in the program.
– This 10% part is the core of the program.
![Page 10: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/10.jpg)
Effect of 90-10 Rule on Reliability
• removing 60% defects from least used parts would lead to only about 3% improvement to product reliability.
• Reliability improvements from correction of a single error depends on whether the error belongs to the core or the non-core part of the program.
![Page 11: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/11.jpg)
Difficulty in Software Reliability Measurement (2)
• The perceived reliability depends to a large extent upon how the product is used
• In technical terms on its operation profile.
![Page 12: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/12.jpg)
Effect of Operational Profile on Software Reliability Measurement• If we select input data so that only
“correctly” implemented functions are executed none of the errors will be exposed perceived reliability of the product will be high.
• On the other hand, if we select the input data such that only functions containing errors are invoked perceived reliability of the system will be low.
![Page 13: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/13.jpg)
Software Reliability
• Different users use a software product in different ways. – defects which show up for one user
may not show up for another.
• Reliability of a software product– clearly observer-dependent – cannot be determined absolutely
![Page 14: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/14.jpg)
Difficulty in Software Reliability Measurement (3)
• Software reliability keeps changing through out the life of the product each time an error is detected and corrected
![Page 15: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/15.jpg)
Hardware vs. Software Reliability
• Hardware failures inherently different from software failures.
• Most hardware failures are due to component wear and tear:– some component no longer functions
as specified.
![Page 16: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/16.jpg)
Hardware vs. Software Reliability
• A logic gate can be stuck at 1 or 0,– or a resistor might short circuit.
• To fix hardware faults:– replace or repair the failed part.
![Page 17: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/17.jpg)
Hardware vs. Software Reliability
• Software faults are latent:– system will continue to fail unless
changes are made to the software design and code.
![Page 18: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/18.jpg)
Hardware vs. Software Reliability
• Because of the difference in effect of faults, metrics appropriate for hardware reliability measurements are not good software reliability metrics
![Page 19: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/19.jpg)
Hardware vs. Software Reliability
• When a hardware is repaired:– its reliability is maintained
• When software is repaired:– its reliability may increase or
decrease.
![Page 20: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/20.jpg)
Hardware vs. Software Reliability
• Goal of hardware reliability study :– stability (i.e. interfailure times
remains constant)
• Goal of software reliability study – reliability growth (i.e. interfailure
times increases)
![Page 21: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/21.jpg)
Reliability Metrics
• Different categories of software products have different reliability requirements: – level of reliability required for a
software product should be specified in the SRS document.
![Page 22: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/22.jpg)
Reliability Metrics
• A good reliability measure should be observer independent, – so that different people can agree on
the reliability.
![Page 23: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/23.jpg)
Rate Of Occurrence of Failure
• ROCOF measures: – frequency of occurrence failures.– observe the behavior of a software
product in operation over a specified time interval and calculate the total number of failures during the interval.
![Page 24: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/24.jpg)
Mean Time To Failure
• MTTF is average time between two successive failures: – Determine by observing over a large
number of failures.
![Page 25: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/25.jpg)
Mean Time To Failure
• MTTF is not as appropriate for software as for hardware:– Hardware fails due to a component’s
wear and tear thus indicates how frequently the component fails
– When a software error is detected and repaired the same error never appears.
![Page 26: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/26.jpg)
Mean Time To Failure
• We can record failure data for n failures:– let these be t1, t2, …, tn– calculate (ti+1-ti)– the average value is MTTF
(ti+1-ti)/(n-1)
![Page 27: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/27.jpg)
Mean Time to Repair
• Once failure occurs additional time is lost to fix faults
• MTTR measures average time it takes to fix faults.
![Page 28: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/28.jpg)
Mean Time Between Failures
• We can combine MTTF and MTTR:– to get an availability metric:
MTBF = MTTF + MTTR
• MTBF of 100 hours would indicate– Once a failure occurs, the next failure
is expected after 100 hours of clock time (not running time).
![Page 29: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/29.jpg)
Availability
• Measures how likely the system shall be available for use over a period of time: – considers the number of failures
occurring during a time interval, – also takes into account the repair
time (down time) of a system.
![Page 30: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/30.jpg)
Availability
• Important for Systems having real time requirement:– telecommunication systems, – operating systems, etc. which are
supposed to be never down– where repair and restart time are
significant and loss of service during that time is important.
![Page 31: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/31.jpg)
Failure Classes/ Category
• Transient: – Transient failures occur only for certain
inputs.
• Permanent: – Permanent failures occur for all input
values.
• Recoverable: – When recoverable failures occur the system
recovers with or without operator intervention.
![Page 32: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/32.jpg)
Failure Classes
• Unrecoverable: – the system may have to be restarted.
• Cosmetic: – May cause minor irritations. Do not
lead to incorrect results. • Eg. mouse button has to be clicked twice
instead of once to invoke a GUI function.
![Page 33: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/33.jpg)
Reliability Growth Modelling
• A reliability growth model:– a model of how software reliability
grows as errors are detected and repaired.
• A reliability growth model can be used to predict when (or if at all) a particular level of reliability is likely to be attained.– i.e. how long to test the system?
![Page 34: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/34.jpg)
Reliability Growth Modelling
• There are two main types of uncertainty which render any reliability measurement inaccurate:
• Type 1 uncertainty:– our lack of knowledge about how the
system will be used, i.e. • its operational profile
![Page 35: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/35.jpg)
Reliability Growth Modelling
• Type 2 uncertainty:– reflects our lack of knowledge about
the effect of fault removal.• When we fix a fault we are not sure if the
corrections are complete and successful and no other faults are introduced
• Even if the faults are fixed properly we do not know how much will be the improvement to interfailure time.
![Page 36: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/36.jpg)
Step Function Model
• The simplest reliability growth model:– a step function model
• The basic assumption:– reliability increases by a constant
amount each time an error is detected and repaired.
![Page 37: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/37.jpg)
Step Function Model
ROCOF
Time
![Page 38: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/38.jpg)
Step Function Model
• Assumes:– all errors contribute equally to
reliability growth– highly unrealistic:
• we already know that different errors contribute differently to reliability growth.
![Page 39: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/39.jpg)
Jelinski and Moranda Model
• Realizes each time an error is repaired reliability does not increase by a constant amount.
• Reliability improvement due to fixing of an error is assumed to be proportional to the number of errors present in the system at that time.
![Page 40: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/40.jpg)
Jelinski and Moranda Model• Realistic for many applications,• Shortcomings
– Most probable failures (failure types which occur frequently) discovered early during the testing process.
– Repairing these faults contribute maximum to the reliability growth initially.
– This implies rate of reliability growth should be large initially slow down later on
![Page 41: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/41.jpg)
Littlewood and Verall’s Model
• Assumes different fault have different sizes, thereby contributing unequally to failures.
• Allows for negative reliability growth
![Page 42: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/42.jpg)
Littlewood and Verall’s Model
• Large sized faults tends to be detected and fixed earlier
• As number of errors is driven down with the progress in test, so is the average error size, causing a law of diminishing return in debugging
![Page 43: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/43.jpg)
Other Reliability growth models
• Variations exists– LNHPP (Littlewood non homogeneous
Poisson process) model
• Goel – Okumoto (G-O) Imperfect debugging model– GONHPP
• Musa – Okumoto (M-O) Logarithmic Poisson Execution Time model
![Page 44: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/44.jpg)
Applicability of Reliability Growth Models
• There is no universally applicable reliability growth model.
• Reliability growth is not independent of application.
![Page 45: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/45.jpg)
Applicability of Reliability Growth Models
• Fit observed data to several growth models.– Take the one that best fits the data.
![Page 46: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/46.jpg)
Statistical Testing
• A testing process:– the objective is to determine
reliability rather than discover errors.– uses data different from defect
testing.
![Page 47: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/47.jpg)
Statistical Testing
• Different users have different operational profile:– i.e. they use the system in different
ways– formally, operational profile:
• probability distribution of input
![Page 48: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/48.jpg)
Operational profile: Example
• An expert user might give advanced commands:– use command language interface,
compose commands
• A novice user might issue simple commands:– using iconic or menu-based interface.
![Page 49: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/49.jpg)
How to define operational profile?
• Divide the input data into a number of input classes:– e.g. create, edit, print, file operations,
etc.
• Assign a probability value to each input class:– a probability for an input value from
that class to be selected.
![Page 50: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/50.jpg)
Steps involved in Statistical testing (Step-I)
• Determine the operational profile of the software:– This can be determined by analyzing
the usage pattern.
![Page 51: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/51.jpg)
Step 2 in Statistical testing
• Manually select or automatically generate a set of test data: – corresponding to the operational
profile.
![Page 52: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/52.jpg)
Step 3 in Statistical testing
• Apply test cases to the program:– record execution time between each
failure– it may not be appropriate to use raw
execution time
![Page 53: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/53.jpg)
Step 4 in Statistical testing
• After a statistically significant number of failures have been observed:– reliability can be computed.
![Page 54: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/54.jpg)
Statistical Testing
• Relies on using large test data set. • Assumes that only a small
percentage of test inputs:– likely to cause system failure.
![Page 55: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/55.jpg)
Statistical Testing
• It is straight forward to generate tests corresponding to the most common inputs:– but a statistically significant
percentage of unlikely inputs should also be included.
• Creating these may be difficult:– especially if test generators are used.
![Page 56: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/56.jpg)
Advantages of Statistical Testing
• Concentrate on testing parts of the system most likely to be used:– results in a system that the users find
more reliable (than actually it is!).
![Page 57: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/57.jpg)
Advantages of Statistical Testing
• Reliability predictions based on test results:– gives an accurate estimation of
reliability (as perceived by the average user) compared to other types of measurements.
![Page 58: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/58.jpg)
Disadvantages of Statistical Testing
• It is not easy to do statistical testing properly:– there is no simple or repeatable way
to accurately define operational profiles.
• Statistical uncertainty.
![Page 59: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/59.jpg)
Software Quality
![Page 60: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/60.jpg)
a measure of usefulness and
practicality
Quality
a degree or level of innate excellence
extent of departure from the standard,
degree of conformity
a measure of the skill of the creator and value to the person for whom it was created
comparative-dictionary
fitness-for-purpose-Juran
quantitative -manufacturing
subjective -artistic...
![Page 61: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/61.jpg)
Introduction
• Consider a software product:– functionally correct– but unusable user interface.
• Functional correctness alone does not determine quality
![Page 62: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/62.jpg)
Modern view of quality• Associates several quality factors with a
software product :– Correctness– Reliability– Efficiency (includes efficiency of resource
utilization)– Portability– Usability– Reusability– Maintainability
![Page 63: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/63.jpg)
Correctness
• A software product is correct, – if different requirements as specified
in the SRS document have been correctly implemented.
– Accuracy of results.
![Page 64: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/64.jpg)
Portability
• A software product is said to be portable, – Work on different operating systems – Or on different machines– Or with other software products
![Page 65: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/65.jpg)
Reusability
• Different modules of the product can easily be reused to develop new products.
![Page 66: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/66.jpg)
Usability
• A software product has good usability, if different categories of users (i.e. both expert and novice users) can easily invoke the functions of the product.
• HCI aspect
![Page 67: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/67.jpg)
Maintainability
• A software product is maintainable, – If faults can be corrected– Functionalities can be added or
modified (customized)
![Page 68: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/68.jpg)
Software Quality
• the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs .
– ISO 8402
![Page 69: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/69.jpg)
Difficulties in achieving software quality
• its ethereal nature• its discreteness (no tolerances)• informal/semiformal discipline• volatility of requirements• inherent complexity of large
software systems
![Page 70: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/70.jpg)
Cont…
• intellectual (read, semi-disciplined) input
• complex intra-team communication• deadline pressures• rapid advances in technology• high (often unreasonable)
expectations of clients
![Page 71: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/71.jpg)
McCall’s Model of Software Quality
• Operation
• Revision• Transition
![Page 72: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/72.jpg)
Operation Attributes
usability
correctnessintegrityreliability
efficiency
![Page 73: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/73.jpg)
Revision Attributes
maintainability
testability
flexibility
![Page 74: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/74.jpg)
Transition Attributes
portability reusability
interoperability
![Page 75: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/75.jpg)
Crosby’s Maturity Level
Level 1: Uncertainty
Level 2: Awakening
Level 5: Certainty
Level 3: Enlightenment
Level 4: Wisdom
based on management
attitudes
![Page 76: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/76.jpg)
SQA Environment
• Quality Policy• Quality Management• Quality Assurance• Quality Control• Quality Management System• Total Quality Management
![Page 77: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/77.jpg)
Evolution of Quality Systems
• Initial product inspection method :– gave way to quality control (QC).
• Quality control:– not only detect the defective products
and eliminate them – but also determine the causes behind
the defects.
![Page 78: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/78.jpg)
Quality control (QC)
• Quality control aims at correcting the causes of errors: – not just rejecting defective products.
• Statistical quality control– quality of the output of the process is
inferred using statistical methods – in stead of inspection or testing of all
products
![Page 79: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/79.jpg)
Quality control (QC)
• The next breakthrough, – development of quality assurance
principles
![Page 80: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/80.jpg)
Quality assurance
• Basic premise of modern quality assurance: – if an organization's processes are
good and are followed rigorously, • the products are bound to be of good
quality.
![Page 81: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/81.jpg)
Quality assurance
• All modern quality paradigms include:– guidance for recognizing, defining,
analyzing, and improving the production process.
![Page 82: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/82.jpg)
Total quality management (TQM)
• Advocates: – continuous process improvements
through process measurements.
![Page 83: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/83.jpg)
Quality Policy
the overall quality intentions and direction of an organization as regards quality, as formally expressed by top management.
- ISO 8402
![Page 84: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/84.jpg)
Quality Management
that aspect of the overall management function that determines and implements the quality policy.
- ISO 8402
![Page 85: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/85.jpg)
Quality Assurance
a planned and systematic pattern of all actions necessary to provide adequate confidence that the item or product conforms to established technical requirement.
- IEEE 729
![Page 86: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/86.jpg)
Quality Control
the operational techniques and activities that are used to satisfy the quality requirements.
- ISO 8402
![Page 87: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/87.jpg)
Quality Management System
the organizational structure, responsibilities, procedures, processes and resources for implementing quality management.
- ISO 8402
![Page 88: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/88.jpg)
Total Quality Management
Quality is to satisfy customer's requirements continuallyTotal quality is to achieve quality at low costTotal quality management is to obtain total quality by involving everyone's daily commitment
- Kanji
![Page 89: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/89.jpg)
SQA Techniques - Error detection & removal
• inspections• walkthroughs• reviews• testing• demonstrations
![Page 90: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/90.jpg)
SQA Techniques - Error avoidance• systematic and error-proof
determination of user needs• systematic and error-proof translation of
user needs into software products• methods and tools• rigorous specifications• configuration management• adherence to standards• root cause analysis
![Page 91: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/91.jpg)
Role of SQA
• evolve and approve a Software Quality Assurance Plan (SQAP) for every project
• Audit the development process for proper execution in conformance with SQAP and Software Development Plan (SDP)
• Participate in the assessment of internal deliverables and of overall software quality
![Page 92: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/92.jpg)
Role of SQA
• Perform, witness or audit tests; and attend selected life cycle reviews to assure appropriate Verification & Validation
• Participate in configuration management and QIP
• Provide appropriate visibility in situation of non-conformance
![Page 93: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/93.jpg)
Quality Guidelines
• start with management commitment to ensure proper attention
• specify quality goals to promote understanding
• establish strategy to achieve quality to ensure proper resources for quality
• apply predictive measures to evaluate quality and increase insight to identify deficiencies and problem areas
![Page 94: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/94.jpg)
Quality Guidelines
• assess quality achieved in final product to determine how well project satisfied goals
• analyze results to identify areas to improve
• provide feedback to exploit lessons learnt
![Page 95: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/95.jpg)
Quality System Activities:• Auditing of projects• Development of:
– standards, procedures, and guidelines, etc.
• Production of reports for the top management – summarizing the effectiveness of the
quality system in the organization.• Review of the quality system itself.
![Page 96: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/96.jpg)
Business Process reengineering
• A term related to TQM. • Process reengineering goes a step
further than quality assurance:– aims at continuous process
improvement.
![Page 97: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/97.jpg)
Business Process reengineering
• BPR aims at reengineering the way business is carried out in any organization – not just software development
organizations.
![Page 98: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/98.jpg)
Total quality management (TQM)
• TQM goes beyond documenting processes – optimizes them through redesign.
• Over the years the quality paradigm has shifted:– from product assurance to process
assurance.
![Page 99: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/99.jpg)
ISO 9000
• ISO (international Standards Organization): – Consortium established to formulate
and foster standardization.
• ISO published its 9000 series of standards in 1987.
![Page 100: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/100.jpg)
What is ISO 9000 Certification?
• ISO 9000 certification: – serves as a reference for contract
between independent parties.
• The ISO 9000 standard: – specifies guidelines for maintaining a
quality system.
![Page 101: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/101.jpg)
What is ISO 9000 Certification?
• ISO 9000 specifies: – guidelines for repeatable and high
quality product development.– Also addresses organizational aspects
• responsibilities, reporting, procedures, processes, and resources for implementing quality management.
![Page 102: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/102.jpg)
ISO 9000
• A set of guidelines for the production process.– not directly concerned about the
product it self.– a series of three standards:
• ISO 9001, ISO 9002, and ISO 9003.
![Page 103: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/103.jpg)
ISO 9000
• Based on the premise:– if a proper process is followed for
production: • good quality products are bound to
follow.
![Page 104: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/104.jpg)
ISO 9001:
• Applies to:– organizations engaged in design,
development, production, and servicing of goods.
– applicable to most software development organizations.
![Page 105: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/105.jpg)
ISO 9002:• ISO 9002 applies to:
– organizations who do not design products: • but are only involved in production.
• Examples of this category of industries:– steel or car manufacturing industries– buy the product and plant designs from
external sources:• only manufacture products.
– not applicable to software development organizations.
![Page 106: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/106.jpg)
ISO 9003
• Applies to organizations involved only in installation and testing of the products.
![Page 107: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/107.jpg)
ISO 9000 for Software Industry
• ISO 9000 is a generic standard:– applicable to any industry
• Many clauses of ISO 9000 documents:– use generic terminologies– very difficult to interpret them in the
context of software organizations.
![Page 108: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/108.jpg)
Software vs. other industries
• No direct interpretation exist in context of software industry
![Page 109: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/109.jpg)
Software vs. other industries
• Software is intangible. Therefore difficult to control. – In contrast, in a car manufacturing unit:
• we can see a product being developed through stages such as fitting engine, fitting doors, etc.
• one can accurately tell about the status of the product at any time.
– Software project management is an altogether different ball game.
![Page 110: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/110.jpg)
Software vs. other industries• During software development:
– the only raw material consumed is data.• For any other product development:
– Lot of raw materials consumed• e.g. Steel industry consumes large volumes
of iron ore, coal, limestone, etc.
• ISO 9000 standards have many clauses corresponding to raw material control .
• not relevant to software organizations.
![Page 111: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/111.jpg)
Software vs. other industries
• Radical differences exist between software and other product development, – difficult to interpret various clauses of
the original ISO standard in the context of software industry.
![Page 112: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/112.jpg)
ISO 9000 Part-3
• ISO released a separate document called ISO 9000 part-3 in 1991 – to help interpret the ISO standard for
software industry.
• At present official guidance is inadequate
![Page 113: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/113.jpg)
Benefits of ISO 9000 Certification?
• Gain confidence of customers on organization
• Government contracts. This is especially true in the international market.
![Page 114: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/114.jpg)
How to Get ISO 9000 Certification?
• An organization intending to obtain ISO 9000 certification: – applies to a ISO 9000 registrar for
registration.
• ISO 9000 registration process consists of several stages.
![Page 115: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/115.jpg)
How to Get ISO 9000 Certification?
• Application stage:– Applies to a registrar for registration.
• Pre-assessment:– the registrar makes a rough
assessment of the organization.
![Page 116: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/116.jpg)
How to Get ISO 9000 Certification?
• Document review and adequacy audit:– process and quality-related
documents.– the registrar reviews the documents – makes suggestions for improvements.
![Page 117: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/117.jpg)
How to Get ISO 9000 Certification?
• Compliance audit: the registrar checks – whether the suggestions made by it
during review have been complied.
![Page 118: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/118.jpg)
How to Get ISO 9000 Certification?
• Registration:– The registrar awards ISO 9000
certificate after successful completions of all previous phases.
• Continued surveillance:– The registrar continues monitoring
the organization periodically.
![Page 119: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/119.jpg)
ISO 9000 Certification
• An ISO certified organization – can use the certificate for corporate
advertizements– cannot use the certificate to advertize
products.• ISO 9000 certifies organization's process • not any product of the organization.
– An organization using ISO certificate for product advertizements:
• risks withdrawal of the certificate.
![Page 120: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/120.jpg)
Summary of ISO 9001 Requirements
• Management responsibility(4.1):– Management must have an effective
quality policy. – The responsibility and authority of all
those whose work affects quality:• must be defined and documented.
![Page 121: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/121.jpg)
Management responsibility(4.1)
• Responsibility of the quality system. – independent of the development
process, – can work in an unbiased manner.
• The effectiveness of the quality system: – must be periodically by audited.
![Page 122: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/122.jpg)
Quality system (4.2) and contract reviews (4.3):
• A quality system must be maintained and documented.
• Contract reviews (4.3):– Before entering into a contract, an
organization must review the contract • ensure that it is understood,• organization has the capability for
carrying out its obligations.
![Page 123: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/123.jpg)
Design control (4.4):
• The design process must be properly controlled, – this includes controlling coding also.
• A good configuration control system must be in place.
![Page 124: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/124.jpg)
Design control (4.4):
• Design inputs must be verified as adequate.
• Design must be verified.• Design output must be of required
quality.• Design changes must be
controlled.
![Page 125: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/125.jpg)
Document control (4.5):
• Proper procedures for – document approval, issue and
removal.
• Document changes must be controlled. – use of some configuration
management tools is necessary.
![Page 126: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/126.jpg)
Purchasing (4.6):
• Purchased material, including bought-in software:– must be checked for conforming to
requirements.
![Page 127: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/127.jpg)
Purchaser Supplied Products (4.7):
• Material supplied by a purchaser,– for example,
• client-provided software must be properly managed and checked.
![Page 128: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/128.jpg)
Product Identification (4.8):
• The product must be identifiable at all stages of the process. – In software development context this
means configuration management.
![Page 129: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/129.jpg)
Process Control (4.9) :
• The development must be properly managed.
• Quality requirements must be identified in a quality plan.
![Page 130: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/130.jpg)
Inspection and Testing (4.10) :
• In software terms this requires effective testing i.e., – unit testing, integration testing and
system testing.
• Test records must be maintained.
![Page 131: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/131.jpg)
Inspection, measuring and test equipment(4.11):
• If integration, measuring, and test equipments are used, – must be properly maintained and
calibrated.
![Page 132: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/132.jpg)
Control of nonconforming product (4.13) :
• In software terms, – keeping untested or faulty software
out of released product, • or other places whether it might cause
damage.
![Page 133: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/133.jpg)
Corrective Action (4.14) :
• This is both about correcting errors when found, – investigating why they occurred– improving the process to prevent
further occurrences.
• If an error reoccurs despite the quality system, – the system needs improvement.
![Page 134: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/134.jpg)
Handling (4.15) and Quality audits (4.17):
• Handling (4.15) Deals with: – storage, packing, and delivery of the
software product.
• Quality Audits (4.17) :– quality system audit must be carried
out to ensure its effectiveness.
![Page 135: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/135.jpg)
Training (4.18) :
• Training needs must be identified and met.
• Most items of ISO standard– are largely common sense.
![Page 136: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/136.jpg)
Salient features of ISO 9001 requirements:
• All documents concerned with the development of a software product – should be properly managed,
authorized, and controlled.
• Proper plans should be prepared– progress against these plans should
be monitored.
![Page 137: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/137.jpg)
Salient features of ISO 9001 requirements:
• Important documents independently checked and reviewed: – for effectiveness and correctness.
• The product should be tested :– against specification.
• Several organizational aspects: – e.g., management reporting of the
quality team.
![Page 138: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/138.jpg)
Shortcomings of ISO 9001 Certification (1)
• ISO 9000 requires a production process to be adhered to:– but does not guarantee the process to
be of high quality.– Does not give any guideline for
defining an appropriate process.
![Page 139: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/139.jpg)
Shortcomings of ISO 9001 Certification (2)
• ISO 9000 certification process– not fool-proof– no international accredition agency
exists. – likely variations in the norms of
awarding certificates: • among different accredition agencies and
among the registrars.
![Page 140: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/140.jpg)
Shortcomings of ISO 9001 Certification (3)
• Organizations qualifying for ISO 9001 certification:– tend to downplay domain expertise. – tend to believe that since a good
process is in place, • any engineer is as effective as any other
engineer in doing any particular activity relating to software development.
![Page 141: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/141.jpg)
Shortcomings of ISO 9001 Certification (4)
• In manufacturing industry– clear link between process quality and
product quality– once a process is calibrated:
• can be run again and again producing quality goods
• Software development is a creative process:– individual skills and experience is significant
![Page 142: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/142.jpg)
Shortcomings of ISO 9001 Certification (5)
• Many areas of software development are very specialized: – special expertize and experience
(domain expertize) required.
• ISO 9001– does not automatically lead to
continuous process improvement, – does not automatically lead to TQM.
![Page 143: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/143.jpg)
Shortcomings of ISO 9001 Certification (6)
• ISO 9001 addresses mostly management aspects.
• Techniques specific to software development have been ignored– Configuration management– Reviews– Release builds– Problem Notification system– Intranets
![Page 144: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/144.jpg)
SEI Capability Maturity Model
• Developed by Software Engineering Institute (SEI) of the Carnegie Mellon University, USA:– to assist the U.S. Department of
Defense (DoD) in software acquisition.
– The rationale was to include:• likely contractor performance as a factor
in contract awards.
![Page 145: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/145.jpg)
SEI Capability Maturity Model
• Major DoD contractors began CMM-based process improvement initiatives: – as they vied for DoD contracts.
• SEI CMM helped organizations: – Improve quality of software they developed– Realize adoption of SEI CMM model had
significant business benefits.
• Other organizations adopted CMM.
![Page 146: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/146.jpg)
SEI Capability Maturity Model
• In simple words,– CMM is a model for apprising the
software process maturity of a contractor into different levels.
– Can be used to predict the most likely outcome to be expected from the next project that the organization undertakes.
![Page 147: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/147.jpg)
SEI Capability Maturity Model
• Can be used in two ways:– Capability evaluation– Software process assessment.
![Page 148: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/148.jpg)
Capability Evaluation
• Provides a way to assess the software process capability of an organization– Helps in selecting a contractor– Indicates the likely contractor
performance
![Page 149: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/149.jpg)
Software Process Assessment
• Used by an organization to assess its current process:– Suggests ways to improve the
process capability.– This type of assessment is for purely
internal use.
![Page 150: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/150.jpg)
SEI Capability Maturity Model
• The SEI CMM classifies software development industries into: – Five maturity levels.– Stages are ordered so that
improvements at one stage provide foundations for the next
– Based on the pioneering work of Philip Crosby
![Page 151: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/151.jpg)
SEI Capability Maturity Model
Initial (1)
Repeatable (2)
Defined (3)
Managed (4)
Optimizing (5)
![Page 152: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/152.jpg)
Level 1: (Initial)
• Organization operates– without any formalized process or
project plans
• An organization at this level is characterized by – ad hoc and often chaotic activities.
![Page 153: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/153.jpg)
Level 1: (Initial)
• Software production processes are not defined, – different engineers follow their own
process – development efforts become chaotic. – The success of projects depend on
individual efforts and heroics.
![Page 154: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/154.jpg)
Level 2: (Repeatable)
• Basic project management practices – tracking cost, schedule, and functionality
are followed.
• Size and cost estimation techniques – function point analysis, COCOMO, etc. used.
• Production process is ad hoc – not formally defined– also not documented.
![Page 155: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/155.jpg)
Level 2: (Repeatable)
• Process used for different projects might vary between projects: – earlier success on projects with
similar applications can be repeated.– Opportunity to repeat process exist
when a company produces a family of products.
![Page 156: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/156.jpg)
Level 3: (Defined)
• Management and development activities: – defined and documented.– Common organization-wide
understanding of activities, roles, and responsibilities.
![Page 157: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/157.jpg)
Level 3: (Defined)
• The process though defined, – process and product qualities are not
measured.
• ISO 9001 aims at achieving this level.
![Page 158: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/158.jpg)
Level 4: (Managed)
• Quantitative quality goals for products are set.
• Software process and product quality are measured: – The measured values are used to
control the product quality.– Results of measurement used to
evaluate project performance • rather than improve process.
![Page 159: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/159.jpg)
Level 4: (Managed)
• Organization sets quantitative quality goals
• World-wide about 100 organizations assessed at this level.
![Page 160: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/160.jpg)
Level 5: (Optimizing)
• Statistics collected from process and product measurements are analyzed:– continuous process improvement
based on the measurements.• Known types of defects are prevented
from recurring by tuning the process• lessons learned from specific projects
incorporated into the process
![Page 161: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/161.jpg)
Level 5: (Optimizing)
• Identify best software engineering practices and innovations:– tools, methods, or process are
identified– transferred throughout the
organization
• World-wide about 50 organizations have been assessed at this level.
![Page 162: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/162.jpg)
Key Process Areas
• Each level is associated with a key process area (KPA) identifies– where an organization at the previous
level must focus to reach this level
![Page 163: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/163.jpg)
Level 2 KPAs
• Software project planning– Size, cost, schedule.– project monitoring
• Configuration management• Subcontract management
![Page 164: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/164.jpg)
Level 3 KPAs
• Process definition and documentation
• Reviews• Training program
![Page 165: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/165.jpg)
Level 4 KPAs
• Quantitative measurements• Process management
![Page 166: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/166.jpg)
Level 5 KPAs
• Defect prevention• Technology change management• Process change management
![Page 167: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/167.jpg)
Comparison between ISO 9001 and SEI CMM
• ISO 9001 awarded by an international standards body– can be quoted in official documents
and communications
• SEI CMM assessment is purely for internal use.
![Page 168: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/168.jpg)
Comparison between ISO 9001 and SEI CMM
• SEI CMM was developed specifically for software industry:– addresses many issues specific to
software industry.
• SEI goes beyond quality assurance– aims for TQM– ISO 9001 correspond to SEI level 3.
![Page 169: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/169.jpg)
Comparison between ISO 9001 and SEI CMM
• SEI CMM provides a list of key areas– on which to focus to take an organization
from one level to the other
• Provides a way for gradual quality improvements over several stages.– e.g trying to implement a defined process
before a repeatable process:• counterproductive as managers are
overwhelmed by schedule and budget pressure.
![Page 170: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/170.jpg)
Remarks on Quality Model Usage• Highly systematic and measured approach
to software development process suits certain circumstances– negotiated software, safety-critical software, etc
• What about small organizations?• Typically handle applications such as internet, e-comm. • without an established product range,• without revenue base, experience on past projects, etc.• CMM may be incompatible
![Page 171: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/171.jpg)
Small Organizations
• Small organizations tend to believe:– We are all competent people hired to
do a job, we can’t afford training– We all communicate with one another
• Osmosis works because we are so close
– We are all heroes• We do what needs to be done• Therefore rules do not apply to us
![Page 172: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/172.jpg)
Small Organizations
• Often have problems:– Undocumented requirements– Inexperienced managers– Documenting the product– Resource allocation– Training– Peer reviews
![Page 173: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/173.jpg)
Small Organizations
• A two week CMM-based appraisal is probably excessive:
• Small organizations need to operate more efficiently at lower levels of maturity– Must first fluorish if eventually they
are to mature
![Page 174: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/174.jpg)
Personal Software Process (PSP)
• Based on the work of Humphrey• PSP is a scaled down version of
industrial software process– suitable for individual use
• Even CMM assumes that engineers use effective personal practices
![Page 175: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/175.jpg)
Personal Software Process (PSP)
• A process is the set of steps for doing a job
• The quality and productivity of an engineer – largely determined by his process
• PSP is framework that– helps software engineers to measure
and improve the way they work.
![Page 176: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/176.jpg)
Personal Software Process (PSP)
• Helps developing personal skills and methods– Estimating and planning method– Shows how to track performance against
plans– Provides a defined process
• can be fine tuned by individuals• Recognizes that a process for individual use is
different from that necessary for a team project.
![Page 177: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/177.jpg)
Time Management
• Track the way you spend time– Boring activities seem longer then
actual– Interesting activities seem short
• Record time for– Designing – Writing code– Compiling– Testing
![Page 178: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/178.jpg)
Personal Software Process (PSP)
Planning
Design
Code
Compile
TestPostmortem
Logs
Project plan summary
![Page 179: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/179.jpg)
PSP-Planning
• Problem definition• Estimate max, min, and total LOC• Determine minutes/LOC• Calculate max,min, and total
development times• Enter the plan data in project plan
summary form• record the planned time in Log
![Page 180: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/180.jpg)
PSP-Design
• Design the program• Record the design in specified
format• Record the Design time in time
recording log
![Page 181: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/181.jpg)
PSP-Code
• Implement the design• Use a standard format for code
text• Record the coding time in time
recording log
![Page 182: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/182.jpg)
PSP-Compile
• Compile the program• Fix all the defects• Record compile time in time
recording log
![Page 183: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/183.jpg)
PSP-Test/Postmortem
• Test– Test the program– Fix all the defects found– Record testing time in time recording log
• Postmortem– Complete project plan summary form
with actual time and size data– Record postmortem time in time record
![Page 184: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/184.jpg)
Personal Software Process (PSP)
PSP 0
PSP 1
PSP 2
PSP 3
Personal measurement
Basic size measures
Personal planning
Time and schedule
Personal quality management
Design and code reviews
Personal process
evolution
![Page 185: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/185.jpg)
Fitting pieces
![Page 186: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/186.jpg)
Building the CMM Structure
![Page 187: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/187.jpg)
Capability Maturity Models• CMMI SM – CMM IntegrationSM • SW-CMM – Capability Maturity Model® for
Software • P-CMM – People Capability Maturity Model • SA-CMM – Software Acquisition Capability
Maturity Model • SE-CMM – Systems Engineering Capability
Maturity Model • IPD-CMM – Integrated Product
Development Capability Maturity Model
![Page 188: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/188.jpg)
SW - CMM
• Describes principles and practices underlying software process maturity
• Intended to help software organizations improve the maturity of their software processes
![Page 189: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/189.jpg)
SE - CMM
• Describes essential elements of an organization's systems engineering process
• Provides a reference for comparing actual systems engineering practices against these essential elements.
![Page 190: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/190.jpg)
IPD - CMM• Guide organizations in IPD design,
development, appraisal, and improvement.
• Involves a teaming of the functional disciplines to integrate and concurrently apply all necessary processes to produce an effective and efficient product that satisfies the customer's needs
![Page 191: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/191.jpg)
CMMI
• The content and characteristics of SW-CMM [draft version 2(c)], SE-CMM [EIA/IS 731], and IPD-CMM [Version 0.98] models provide the basis for the CMMI product suite
• CMMI models eventually replace SW-CMM Version 1.1
![Page 192: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/192.jpg)
Motivation for Integrations
• Models created within an environment of evolving national and international standards and frameworks
• Maintaining harmonization between them and improvement models become a continuing challenge, particularly across discipline
![Page 193: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/193.jpg)
Frameworks Quagmire
![Page 194: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/194.jpg)
CMM Integration
• Objective is to develop a product with a set of integrated products to support process and product improvement.
• Preserve investment in process improvement and enhance the use of multiple models.
• Output will be integrated model(s), assessment method(s), and training material.
![Page 195: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/195.jpg)
CMMI Objectives• Eliminating inconsistencies• Reducing duplications• Increasing clarity and understanding• Assuring consistency with ISO 15504
Longer-term objective is to lay foundation for later addition (acquisition, security)
![Page 196: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/196.jpg)
CMM Based Appraisals• CMM-Based Appraisal for Internal Process
Improvement (CBA IPI): To determine the state of one's own processes
• Software Capability Evaluation (SCE):To determine the state of someone else's processes.
• To ensure that these methods are both rigorous and consistent, the Process Program developed and maintains the CMM Appraisal Framework (CAF), which defines the standards that an appraisal method must meet.
![Page 197: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/197.jpg)
When to apply CBA IPI
• For investigating your organization, using mostly your own team members, and making principal use of the results yourself, then a CBA IPI assessment is the recommended tool. This method assumes a collaborative environment and a collegial approach to gathering data.
![Page 198: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/198.jpg)
When to apply SCE• If, on the other hand, you are investigating
somebody else's organization, using your own team members (not theirs), and you will use the results for your purposes (even if you plan to share the results with the organization), then an SCE is the recommended tool. This method assumes a potentially audit-like environment and a more intrusive approach to gathering data
![Page 199: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/199.jpg)
SEI’s Role In CMM-Based Appraisals• Authorized Lead Assessors for CBA
IPI are required to return a report to the SEI for each completed assessment. The data is entered into the Process Appraisal Information System. All assessment data that is returned to the SEI is kept confidential.
• The SEI trains and authorizes qualified persons to lead appraisals
![Page 200: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/200.jpg)
Cont…
• The SEI does not validate or certify appraisal results.
• An assessment's findings and maturity level are owned by the assessment sponsor. The sponsor may publicize this information at their discretion.
• The Maturity Profile Report is published by the SEI twice a year
![Page 201: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/201.jpg)
That’s all?
• Not exactly• SQA Activities
– Cleanroom methodology– Six Sigma
• Assessment models– SPR Assessment– Malcolm Baldrige National Quality
Award
![Page 202: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/202.jpg)
Cleanroom Software Engineering
• Metaphor coming from integrated circuit manufacturing– Free from dust, flecks of skin etc
• Whenever defect occurs, they are considered as defect in process than product– Invoke process inspection, review and
improvement cycle
![Page 203: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/203.jpg)
Six Sigma
• Motorola’s baby that won them MBNQA award for quality in 1988
• Six sigma is a quantitative approach to eliminate defects
• Applicable to any industry - manufacturing, product development, service
![Page 204: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/204.jpg)
Six Sigma• To achieve six sigma
– a process must not produce more than 3.4 defects per million opportunities.
– 5 Sigma -> 230 defects per million– 4 Sigma -> 6210 defects per million
• Six sigma methodologies– DMAIC (Define, Measure, Analyze, Improve,
Control)– DMADV: (Define, Measure, Analyze, Design,
Verify)
![Page 205: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/205.jpg)
Six Sigma Methodologies
• The methodologies are implemented by Green belt and Black belt workers– Supervised by Master black belt worker
• Pareto Chart:– Simple bar chart to represent defect
data– Identify the problems that occurs with
greatest frequency• or incur the highest cost
![Page 206: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/206.jpg)
Software Productivity Research Inc.• Methodology developed parallel to SEI
Process maturity model• SPR question covers both Strategic
corporate issues and tactical project issues that affect quality, productivity and user satisfaction
• 400 questionnaire as against SEI’s 85 questions– Lickert Scale for response (Excellent, Good,
Average, Marginal, Poor)
![Page 207: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/207.jpg)
Malcolm Baldrige National Quality Award• Established in 1988 by US Department
of Commerce• Examination criteria against seven
categories– Leadership– Information and analysis– Strategic Quality Planning– Human Resource Utilization– Quality assurance of products and services– Quality Results– Customer satisfaction
![Page 208: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/208.jpg)
References
• CMMI Distilled– Ahern, Clouse, Turner. Addison Wesley
• CMM in Practice– Pankaj jalote. Pearson Education
• Introduction to Personal SW Process, Introduction to Team SW Process– Watts Humphrey. Addison Wesley
![Page 209: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/209.jpg)
Web References• General Info
– http://www.sei.cmu.edu– http://www.sei.cmu.edu/sema/profile.html
• PSP, TSP– http://interactive.sei.cmu.edu
/Features/1999/June/Background/Background.jun99.pdf [Both]
– http://www.sei.cmu.edu/pub/documents/articles/pdf/psp.pdf [PSP]
– http://wuarchive.wustl.edu/languages/ada/sei/documents/00.reports/pdf/00tr023.pdf [TSP]
![Page 210: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/210.jpg)
Alternatives/Additions• ISO Standards
– Generic Specifications in areas of Quality Requirements (9000), Environmental Management (14000)
– Latest on Information Technology (ISO/IEC TR 15504-#:1998)
, where 0<#<10
– Some client demand for ISO Certified institutes
http://www.iso.ch/iso/en/iso9000-14000/tour/magical.html
![Page 211: 12 Reliability QA Standards](https://reader033.fdocuments.net/reader033/viewer/2022051515/5528b5e355034670588b490b/html5/thumbnails/211.jpg)
Other References
• http://www.software.org/quagmire/• Presentation by Rajib Mall