Post on 10-May-2015
description
Rhonda Software Quality Assurance (QA) Services
Configurable software quality assurance process: From requirements acquisition through testing and configuration management
to release and maintenance activities
Quality Assurance – to Serve Your Needs
“Quality is never an accident; it is always the result of intelligent effort.”
John Ruskin
“Software never was perfect and won’t get perfect. But is that a license
to create garbage? The missing ingredient is our reluctance to quantify
quality.” Boris Beizer
“Testing is an infinite process of comparing the invisible to the
ambiguous in order to avoid the unthinkable happening to the
anonymous.” James Bach
2© 2013 Rhonda Software: www.rhondasoftware.com
Quality Assurance – to Serve Your Needs
TECHN
OLOGYflexible & powerful
tools
PROCESSlightweight & mature practices
PEOPLE
talented & skilled
professionals
3© 2013 Rhonda Software: www.rhondasoftware.com
QUALITYBUILDING BLOCKS
Quality Assurance – to Serve Your Needs
There is no secret that it is impossible to develop software product
without any defects in it. And the more complex product you create –
the more errors it will demand.
What?Quality Assurance (QA), as we see, is the process that should be part of each
and every stage of SW product development . It is a well chosen number of best
practices taken from CMM/CMMI and our 18-year experience to ensure high
quality of developed SW product. In the middle of QA is a set of testing activities
integrated into entire production cycle.
Why?Because escaped defects (errors left in released product) may cost too much. Too much money to fix
and too much reputational damage to live with.
When?The sooner the better. Everybody knows that the later a defect is found, the higher its impact and
price will be. However, it is never too late to think about quality – our QA process can be easily
adopted and integrated into any customer process on any project stage.
4© 2013 Rhonda Software: www.rhondasoftware.com
Better Safe Than Sorry
QUALITY
CONTROL
EXECUTE
PLAN
Quality Assurance – to Serve Your Needs
Every year, software errors cause massive amounts of problems all over the world. A 2002 study
commissioned by the National Institute of Standards and Technology found that software bugs cost
the US economy $59.5 billion every year (imagine the global costs). The study estimated that more
than a third of that amount, $22.2 billion, could be eliminated by improved testing.
Here is a couple of relatively fresh examples:
March, 2011. Japanese bank Mizuho Financial Group's clients
experienced a software glitch that collapsed its ATM network and
internet banking systems. The result was $1.5 billion in salary
payment delays and $9 billion in unprocessed payments.
August, 2012. In less than an hour, Knight Capital's computers
executed a series of automatic orders that were supposed to be
spread out over a period of days.
Millions of shares changed hands. The resulting loss, which was
nearly four times the company's 2011 profit, crippled the firm and
brought it to the edge of bankruptcy. Knight only survived after a
group of investors swooped in with a $440 million lifeline.
5© 2013 Rhonda Software: www.rhondasoftware.com
The Cost of Error
Integrated QA Process
6© 2013 Rhonda Software: www.rhondasoftware.com
REQUIREMENTSMANAGEMENT
Identified requirements
Specifications
CODEDEVELOPMENT
Unit testing
Code best practices
Reviews
QUALITYASSURANCEPROJECT
MANAGEMENTProduct/Feature management
Task tracking
Resources
TESTPREPARATION
Requirements traceability
Design
Automation
TESTMANAGEMENT
Scope/Planning
Monitoring
Reporting
TESTEXECUTION
Test passesEscaped defects analysis
Defect tracking
CONFIGURATIONMANAGEMENT
Version control
Continuous integration
Release management
As any other project activity, QA should be carefully
considered in planning stage to define quality goals &
criteria, identify applicable QA/testing procedures and
pre-assign necessary resources (including personnel,
hardware and infrastructure environment).
In the large scale projects, there is an additional practice that
have major impact on resulting product quality.
Product/Feature Management• When there is an entire line of products built on a common code base• When there is incremental product evolution where new functionality is released in
new product versions, especially only in select product tiers• When there are a lot of features: to add something new, to change existing behavior, to
incorporate functionality from other product in the line, to exclude obsolete function,
and so on...
Products and features need to be managed very neatly. Especially since scope and
essence of on-project QA activity is directly connected with selected product feature set.
PROJECTMANAGEMENT
Product/Feature management
Task tracking
Resources
QA Core Components
7© 2013 Rhonda Software: www.rhondasoftware.com
QA Dependencies - Project Management
REQUIREMENTSMANAGEMENT
Identified requirements
Specifications
QA Core Components
Requirements, as customer / user expectations regarding
developed SW product features and capabilities always exist
on a project in some form… even just as a set of ideas in
a head of a project manager.
For the quality sake, it is better to keep product requirements
in a well organized way.
Requirements Specifications• To identify all important aspects of functionality and receive customer’s confirmation• To define restrictions and exceptions avoiding a case when assumed, but
undocumented behavior was never implemented• To establish common understanding across the entire project team• To provide basis for project estimates, code development and validation processes• To be a starting point for effective communication and change management during
product evolution process
Thoroughly identified software requirements, organized as a document may become one
of the major instruments during testing process since Requirements Coverage validation
is the powerful tool to verify that a developed product fulfills all given expectations.
8© 2013 Rhonda Software: www.rhondasoftware.com
QA Dependencies - Requirements Management
Same as Requirements, Configuration Management (CM)
is always present on any software development project,
though sometimes only as a single code version without
any history.
Needless to say that a well-organized tool-based CM is vital
for any project that is more complex then “Hello world!” app.
Version Control & Continuous Integration Environment• If you have a distributed team simultaneously working on the same code base• If you need to keep your code baseline free from breakages introduced by bug fixing• If you want to know code version where new error appeared for the first time, and to
identify what change had introduced it• If you value the possibility to test isolated code deltas knowing exact scope of changes• If you realize all advantages of automated testing on each and every code version
published
One cannot overestimate the impact of mature CM on resulting product quality, especially
when appropriate tools are selected and efficient CM practices are used by the entire
team to store on-project deliverables and track features, tasks and defects found.
CONFIGURATIONMANAGEMENT
Version control
Continuous integration
Release management
QA Core Components
9© 2013 Rhonda Software: www.rhondasoftware.com
QA Dependencies - Configuration Management
There are a lot of aspects in a mature Code Development
process that will influence product quality either directly
or indirectly.
Besides personal skills of talented developers, it is always
helpful to apply unified code production process to support
overall code quality during implementation phase.
Unit Testing, Coding Best Practices & Code Reviews• For unified coding style suitable for further product maintenance and updates without
single team member lock-in• For earliest breakage detection and the most cost effective bug fix• For test-oriented code design, especially helpful in embedded and remote testing• For precise error localization and minimal impact on dependent product components• For knowledge sharing among the development team members and personal skill
improvements
Well-established and regularly applied best code development practices allow to introduce
more effective product testing, since the test team can concentrate on behavioral defects
while architectural weaknesses and logical bugs will be filtered out on early stages.
CODEDEVELOPMENT
Unit testing
Code best practices
Reviews
QA Core Components
10© 2013 Rhonda Software: www.rhondasoftware.com
QA Dependencies – Code Development
Testing is the core process among all
other QA activities.
As a product matures, it proceeds
through test subprocesses and
activities until it is presented to
customers for their evaluation. Each test
activity adds a unique value to find defects as early
as possible in the product development life cycle.
Testing is implemented in separate and/or
overlapped test phases/types:
• Testing on different product levels:
unit, feature, features interaction, system;• Testing during product maintenance:
smoke, sanity, regression;• Testing of special aspects:
bring-up, stress, stability, performance, interoperability, compliance;
TESTPREPARATION
Requirements traceability
Design
Automation
TESTMANAGEMENT
Scope/Planning
Monitoring
Reporting
TESTEXECUTION
Test passesEscaped defects analysis
Defect tracking
QA Core Components
11© 2013 Rhonda Software: www.rhondasoftware.com
Testing Process Components
Testing is a complex set of many different test activities closely
related to other project processes. All of them require to be
planned / monitored (and re-planned during the project life, if
it is required) thoroughly and competently.
The Test Management subprocess is performed from the
very beginning through the end of a project to achieve goals of improved
product quality.
Major Objects of Test Management’s Attention
• Testing Scope / Strategy• Approaches and Constraints• Features / Components / Functional Areas to be or not to be Tested• Test Item Pass / Fail and Test Cycle Suspension / Resumption Criteria• Test Tasks and Deliverables, Environmental Needs• Responsibilities, Staffing and Training Needs, Collaboration with other project groups• Schedule, Risk and Contingencies• Reporting and Quality metrics
QA Core Components
12© 2013 Rhonda Software: www.rhondasoftware.com
Test Management
TESTMANAGEMENT
Scope/Planning
Monitoring
Reporting
Test Preparation subprocess defines tests and
test environment required for each test phase/type in order
to fully verify and validate a product according to test plans.
Basic Test Preparation Activities• Analysis of functional requirements, specifications and
functional areas' principles of work• Development of Requirements Traceability Matrix (RTM)• Design and development of the test cases for all planned phases/types. The following
approaches and techniques are used to ensure effective and optimal test suites:• Various test approaches (Black-box, White-box)• Effective test techniques (Functionality, Equivalence classes, Boundary, Negative/Positive,
Combinatorial, Model-based, Etc.)• Development of additional tools/utilities and Test automation• Development and setup of specific test environment (quite often it requires a very
complex hardware and software solution which in turn requires very high engineering skills from test engineers)
Obviously, test products themselves must have good quality and thus all project test
deliverables undergo a careful review and validation process.
QA Core Components
13© 2013 Rhonda Software: www.rhondasoftware.com
Test Preparation
TESTPREPARATION
Requirements traceability
Design
Automation
The goal of the Test Execution subprocess is straightforward –
apply test definitions elaborated during the Test Preparation
phase to the software products.
Test Execution activities are either scheduled according to
defined test plans or initiated on demand on an event-driven
basis.
Usual Test Execution Activities
• Preparation of developed test environment• Execution of test cases• Deep analysis of execution results• Collecting/Recording of test execution results and failures found • Support development team efforts to reproduce and to analyze failures, as well as
validation of fixes• Analysis of escaped defects• Update test cases/test environment according to results of test execution• Update test cases/test environment according to results of Escaped Defects Analysis
QA Core Components
14© 2013 Rhonda Software: www.rhondasoftware.com
Test Execution
TESTEXECUTION
Test passesEscaped defects analysis
Defect tracking
QA Core Components
Test automation, especially within Continuous
Integration (CI) development cycle, has proven to be
the most cost-effective approach while increasing
the testing speed and quality, eliminating human
errors and reducing testing expenses.
It is a useful instrument in a long-lead product
evolution and maintenance. In some cases
test automation is the only method when test
engineers can keep pace with software
developers and verify each software version with
the same reliability and quality level as previous ones.
However, it is necessary to understand that automated testing
is powerful aid to improving the return on the testing investment when used wisely. Only
competent project management, test preparation, including requirements analysis,
automated test tool selection/development, and test scenarios development can really
increase testing efficiency.
15© 2013 Rhonda Software: www.rhondasoftware.com
QA & AutomationAUTO-TEST MODE
L
AUTO-RUN
TEST RESUL
TS LOG
ANALYSIS
UPDATES & BUG
FIXES
CONTINUOUSINTEGRATION
QA Core Components
As per our experience, Testing is performed in a more
effective manner when suitable tools are used on the
project. They allow to keep all QA-specific information
in a definite common accessible places, perform
planning / monitoring / reporting more effectively and
collect valuable QA metrics.
Different types of QA management assets and tools used are:
• Project / Test Plans, Activities and Tasks• Environment and Hardware Resources• Requirements• Requirements Traceability (test cases vs. requirements)• Test Cases• Product Builds & Code Versions• Test Results• Failures Found (Defect tracking)• Project / Test processes progress metrics
16© 2013 Rhonda Software: www.rhondasoftware.com
QA Assets & Tools
Customizable QA
Here at Rhonda Software we know that each company
has its own SW development process with unique
characteristics and dedicated practices / activities
adapted for specific project needs.
There is no point to change things that are already good
enough and work as a well-oiled machine. Instead let’s
incorporate beneficial QA practices and improvements inline with yours development
framework where they give most noticeable effect.
Our goal is the lightweight and effective QA process and seamless integration of improvements into customers’ existing SW development process
Our customizable QA services are:• Core testing activities – test preparation, execution, management• Test automation• Requirements elicitation and management• CM and Continuous Integration• Introducing effective practices for using Task/defect tracking tools
17© 2013 Rhonda Software: www.rhondasoftware.com
Seamless Integration into Customer Process
Rhonda Software QA Services
Good technical skills helped our SW test
engineers to prepare a multi-network environment
and successfully provide testing services to one
of our customers to test dual modes mobile
products which support cdma2000, 1xEV-DO
and Wi-Fi networks. This very complicated and
labor-consuming task was done in a short period
of time without any external trainings or help
available to our team.
CDMA/Wi-Fi lab was successfully installed by
our engineers based on Agilent 8960 Series 10 Wireless Communications Test Sets and
Linksys Wi-Fi routers.
This test lab has since been used to verify mobile applications that require CDMA and
VoIP voice calls (including inter-networks calls), MO and MT SMS over CDMA and Wi-Fi
networks, 1xEVDO data services and data over Wi-Fi (browser, audio/video streaming,
Java applications), CDMA and Wi-Fi handoffs, etc.
18© 2013 Rhonda Software: www.rhondasoftware.com
Case Study – Multi-network Test Environment
Rhonda Software QA Services
These are some of CV systems developed by Rhonda:• myAudience demographics solutions• Traffic Lights monitoring system• Face Detector IP core for FPGA
In order to assure the quality of system we used different levels of testing:• Local modules unit-tests which can also be run on CI
server (Jenkins)• Functional testing in user scenarios• Web-portal UI acceptance auto testing (using Codeception framework)• End-to-end product auto-testing (scenarios start from device with camera and run till calculated
data is returned via API from web-portal)
In addition to usual testing practices, development of CV products required additional specific testing tasks.• Evaluate system performance metrics: recognition rate, classification rate, detection rate, counting
precision• Support optimal parameters calibration:
• CV algorithms often depend on multiple parameters that can not be easily found• One way to find good parameters is to run the system on different combinations, measure quality and
select the best ones
19© 2013 Rhonda Software: www.rhondasoftware.com
Case Study – Computer Vision (CV) Testing
One of current Rhonda Software clients provides
all-in-one cloud-based solution for disaster recovery,
backup and data storage. And it is a quite challenging
task to sustain best quality, performance and reliability
during continuous evolution and development cycle.
Specific QA activities are applied to ensure continuous defects free production
• Iteration-based Scrum agile process• Test-driven development approach with JUnit, RUnit unit tests• Continuous Integration environment with automated testing – functional, integration
and performance for each component• Additional UI auto-tests for Web-console user interaction component• Two-step deployment process after successful integration / auto-test:
• Mandatory staging proof cycle that includes complementary feature / bug fix & regression
testing in production-like test environment• Post deployment regression testing on production servers for escaped defects control
• Continuous system health remote monitoring
Rhonda Software QA Services
20© 2013 Rhonda Software: www.rhondasoftware.com
Case Study – QA in Cloud-based SaaS
Rhonda Software QA Services
There is no doubt that all aforementioned QA practices and
processes are very helpful and wholesome. However, there is
one more very (maybe the most) important QA process
component – our QA Team.
Practical skills and technical background allowed our QA
engineers to conduct testing on most modern and complex
equipment (especially in the telecom domain) on the customers’
side and to install and setup own test laboratories.
• Strong communication skill set proven by interaction experience with many of our partners around the world
• All engineers have Master’s degrees in Computer Science• Successful on-site testing to support testing process
whenever and wherever needed. Many of our QA engineers visited customers’ facilities in the USA, Great Britain, Japan, Italy, Germany, France, Israel, South Korea and Taiwan
• Deep knowledge of different platforms, programming languages and technologies• Experience in custom test tools and frameworks development, allowing to complete
sophisticated testing tasks
21© 2013 Rhonda Software: www.rhondasoftware.com
QA Team – Highlights and Areas of Expertise EMBEDDED
NETWORKINGMULTIMEDIACOMPUTER VISION
DRIVERS
PROTOCOLSUSER-LEVEL APPS
Rhonda Software QA Services
Our QA engineers have participated in numerous projects for different customers, for
example:
Motorola• Android Kernel/BSP• Multimedia & DRM• Email client• i-mode• CDMA/WiFi DMH• MDS Test process improvement
CSR (former Zoran)• Embedded solutions for digital camera processors• Image processing (still capture, video recording)• Networking
Exalt• Microwave Digital Backhaul radios
Mentor Graphics (Embedded Group)• Testing of Linux BSP ports on new silicon
22© 2013 Rhonda Software: www.rhondasoftware.com
QA Team – Experience & Customers
Rhonda Software QA Services
Rhonda Software as QA service provider deals
with various tools and instruments working on
customer’s projects.
Tools for QA planning and test management• MS Project – Commonly used tool for project and test
activities / tasks planning• JIRA – Tracker tool used to capture and organize issues, assign work, and follow team activity• Rational DOORS – Requirements management solution• Assembla – Set of cloud-based task and issue management tools for software developers• TestRail – Test case management software for QA and development teams• Selenium – Suite of tools to automate web applications for testing purposes across many
platforms• TeamCity – All-in-one pre-integrated solution for Continuous Integration• RSpec – Behavior-driven development (BDD) framework for Ruby programming language• Cucumber – Tool for running automated acceptance tests written in a BDD style• JUnit – Unit testing framework for Java programming language• Mockito – Open source testing framework for Java• EasyMock – Library that provides an easy way to use Mock Objects for given interfaces or classes• TestNG – Testing framework inspired from JUnit and NUnit with new useful features
23© 2013 Rhonda Software: www.rhondasoftware.com
QA Team – Instruments & Tools Skill
Tasks Tools Job Done
Andrey MischenkoCEO, Rhonda Software
Mobile: +7(423)257-1008
E-mail: ceo@rhondasoftware.com
Denis PomogaevVP of Technology Innovations
Mobile: +1(224)715-1154E-mail: dp@rhondasoftware.com
24© 2013 Rhonda Software: www.rhondasoftware.com
Thank You!