What Do Defects Really Cost? Much More Than You Think
-
Upload
techwellpresentations -
Category
Software
-
view
126 -
download
1
Transcript of What Do Defects Really Cost? Much More Than You Think
T13
Test Management
5/7/2015 1:30:00 PM
What Do Defects Really Cost?
Much More Than You Think
Presented by:
Wayne Ariola
Parasoft
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Wayne Ariola
Parasoft
Chief strategy officer Wayne Ariola leads the development and execution of Parasoft’s long-
term strategy. Wayne has contributed to the design of core Parasoft technologies and has been
awarded several patents for his inventions. He is a recognized leader on service virtualization,
API quality, quality policy governance, and application security. Wayne brings more than twenty
years of strategic consulting experience in the technology and software development
industries. Prior to joining Parasoft, he helped co-found a start-up and consulted in the strategic
change practice at PricewaterhouseCoopers.
What Do Defects Really Cost?
Much More Than You Think
What Do Defects Really Cost?
Much More Than You Think
Parasoft Proprietary and Confidential 1
2015-04-30
Much More Than You Think
Wayne Ariola
Chief Strategy Officer - Parasoft
Much More Than You Think
Wayne Ariola
Chief Strategy Officer - Parasoft
Session AbstractSession Abstract
Software quality has traditionally been a “time-boxed”
exercise and we can’t possibly expect that accelerating the
SDLC via DevOps will yield better results from a quality
perspective. Automating the build-test-release-deploy
process will undeniably get the application into the hands of
users faster. However, if you haven’t also re-engineered the
Parasoft Proprietary and Confidential 2Parasoft Proprietary and Confidential 2
users faster. However, if you haven’t also re-engineered the
processes by which the application is built and tested, you’re
quite likely flooding your users with more defects at a more
rapid rate.
Re-Invention…Of TimeRe-Invention…Of Time
Parasoft Proprietary and Confidential 3Parasoft Proprietary and Confidential 3
http://www.scienceworldreport.com/articles/22225/20150210/cool-optical-lattice-
clocks-1000-times-more-precise-cesium-atomic.htm
Software Drives InnovationSoftware Drives Innovation
Parasoft Proprietary and Confidential 4Parasoft Proprietary and Confidential 4
The Cost of Quality
associated with software has shifted dramatically
Software has shifted from process enabler to business
differentiator
SDLC Speed will be the difference between a first mover and a follower
Switching costs
associated with software are dramatically lower
Software Failures = HeadlinesSoftware Failures = Headlines
Financial Airlines/Aero Government Media Internet/Tel
Parasoft Proprietary and Confidential 5
Automotive Technology Retail
Software Failures = Headlines 2014Software Failures = Headlines 2014
The day of the announcement companies lost an average
Software failures make headline news—
eroding customer confidence, shareholder value and brand equity
Parasoft Proprietary and Confidential 6Parasoft Proprietary and Confidential 6
-$2.3 Bn-3.75%
lost an average
of shareholder value
Escalating Cost of Failure: Public Escalating Cost of Failure: Public
-1.50%
-1.00%
-0.50%
0.00%
2012 2013 2014 2015 (est.)
Impact on Market Capitalization
Parasoft Proprietary and Confidential 7Parasoft Proprietary and Confidential 7
-3.12%-3.37%
-3.75%
-4.18%y = -0.0036x - 0.0272-4.50%
-4.00%
-3.50%
-3.00%
-2.50%
-2.00%
Software Failures = HeadlinesSoftware Failures = Headlines
5,500,000
3,490,0004,000,000
5,000,000
6,000,000
Market does not forget— news about failure increase and
average of 167%
Parasoft Proprietary and Confidential 8Parasoft Proprietary and Confidential 8
821,0001,080,000
451,000
1,400,000
896,000
1,550,000
650,000
3,490,000
2,240,000
0
1,000,000
2,000,000
3,000,000
4,000,000
Cloud Services Airline Bank Software Retail
1st Event 2nd Event
The Cost of Software Failure - SonyThe Cost of Software Failure - Sony
Market Value
-22% -33% -30%-19% -11% -12%
Cumulative Loss = $18B
Parasoft Proprietary and Confidential 9Parasoft Proprietary and Confidential 9
http://www.reuters.com/article/2011/04/26/us-sony-stoldendata-idUSTRE73P6WB20110426
http://www.digitaltrends.com/gaming/sony-fined-almost-400000-for-2011-playstation-security-breach/
Market Value
Event 15 Days 30 Days
Impact of Test in the Evolving SDLCImpact of Test in the Evolving SDLC
Plan Develop Build Test Release
Test as a time-boxed event is the prevailing method
Parasoft Proprietary and Confidential 10Parasoft Proprietary and Confidential 10
Fixed Fixed
Testing is a BottleneckTesting is a Bottleneck
Business Views
Fully Automated Provisioning
Data Analytics
Organic SystemsSoftware Quality Processes
Parasoft Proprietary and Confidential 11Parasoft Proprietary and Confidential 11
Basic Tasks
Skilled Tasks
Process Oriented
End-to-End Tasks
Data Analytics
Automation is Incremental
Eff
icie
ncy
Impact of Test in the Evolving SDLCImpact of Test in the Evolving SDLC
Plan Develop
Bu
ild
Release
Bu
ild
Bu
ild
Bu
ild
Bu
ild
Bu
ild
Bu
ild
Bu
ild
Bu
ild
Bu
ild
Bu
ild
Parasoft Proprietary and Confidential 12Parasoft Proprietary and Confidential 12
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
More than automation – “continuous” tests must assess business risk
From Automated to Continuous TestingFrom Automated to Continuous Testing
Parasoft Proprietary and Confidential 13Parasoft Proprietary and Confidential 13
Getting from “Automated” to “Continuous”Getting from “Automated” to “Continuous”
We have to ask a new question
No Longer: “Are we done testing?”
Parasoft Proprietary and Confidential 14Parasoft Proprietary and Confidential 14
“Does the release candidate have an acceptable level of risk?”
Getting from “Automated” to “Continuous”Getting from “Automated” to “Continuous”
There is a big leap moving from “Automated” to “Continuous”
� Business expectations clearly defined
� Business risks identified
� Per application, team, release
� Defects automatically prioritized versus business drivers
� No defect left behind
Parasoft Proprietary and Confidential 15Parasoft Proprietary and Confidential 15
� No defect left behind
� Mitigate business risks
� Clear ownership and workflow for remediation
� Collaborative risk reduction
� Collapse the remediation cycle time
� Feedback loop for defect prevention
� Pattern definition
� Defect prevention practices enhanced
Transforming the Identification of RiskTransforming the Identification of Risk
From To
Business
Objectives
Parasoft Proprietary and Confidential 16Parasoft Proprietary and Confidential 16
From To
Causal Observations Probabilistic
Defect Documentation Most Likely Causes
Only Structured Data Structured and Unstructured
Filters Business Policies
Ad Hoc Automated
Tool Dependent Tool Independent
Next Generation TestingNext Generation Testing
Parasoft Proprietary and Confidential 17Parasoft Proprietary and Confidential 17
Process IntelligenceProcess Intelligence
Process Intelligence aggregates “siloed” information throughout the SDLC
and identifies business risks early and automatically
� Correlate disparate data across:
� Dependent teams
� Dependent components
� Infrastructure systems
Parasoft Proprietary and Confidential 18Parasoft Proprietary and Confidential 18
� Infrastructure systems
� Testing practices and tools
� Apply advanced statistical analysis
� From pre-build patterns
� From constructed
� With Open domain APIs
� Advantages
� Defect remediation
� Exception-based notifications
The Next Generation Software Quality ‘System’The Next Generation Software Quality ‘System’
Continuous Assessment of RiskContinuous Verification of Requirements
Development Operations
Development
(E1)Production
Test
(E2)
Integration
(E3)
Pre-Production
(E4 – En)
Release Candidate
Parasoft Proprietary and Confidential 19Parasoft Proprietary and Confidential 19
Test Environment Management - Provisioning and Elastic Cloud Scale
Test Optimization
Environment Simulation (SV)
Auto Test Construction – Synthetic
APM – Real Time Data
Development Tests
“Shift-Left”“Shift-Left”
Parasoft Proprietary and Confidential 20Parasoft Proprietary and Confidential 20
“Shift-Left”“Shift-Left”
Parasoft Proprietary and Confidential 21Parasoft Proprietary and Confidential 21
“Shift-Left”“Shift-Left”
Parasoft Proprietary and Confidential 22Parasoft Proprietary and Confidential 22
30
Environment Access?Environment Access?
The average number ofdependent applicationassociated with theSystem Under Test (SUT)
Parasoft Proprietary and Confidential 23Parasoft Proprietary and Confidential 23
30BUT, Dev/QA only have “trustworthy”
access to SIXSIXSIXSIXof the applications
302928272625
242322212019
181716151413
121110987
654321
Test Environment ConfigurationTest Environment Configuration
The average percentage of time spent configuring the test environment.
30%
Parasoft Proprietary and Confidential 24Parasoft Proprietary and Confidential 24
50%The average percentage of the test plan ableto complete once configured.
Survey Results…Survey Results…
When testing with a dependentdependentdependentdependent application only need access to a small percentageof the applications
20%
Parasoft Proprietary and Confidential 25Parasoft Proprietary and Confidential 25
of the applications functionality
Service Virtualization delivers a
simulated dev / test environment
allowing an organization to test
Service VirtualizationService Virtualization
Parasoft Proprietary and Confidential 26Parasoft Proprietary and Confidential 26
allowing an organization to test
anytime or anywhere
So, What are you Virtualizing?So, What are you Virtualizing?
AUT
Parasoft Proprietary and Confidential 27Parasoft Proprietary and Confidential 27
• Mainframe• SOA / Web services• Java• .NET languages• XML• WSDL• WADL• UDDI• WSIL• SOAP• PoX (Plain XML)• REST• JSON• BPEL• Web Applications
• Mobile Interfaces• AJAX• JSP• JavaScript• HTML• CSS• WS-*• Standards• MTOM(XOP) / MIME / DIME • OAuth• TCP/IP• webMethods Broker• webMethods IS• HL7• FTP
• HTTP 1.0• HTTP 1.1• HTTPS• JMS• IBM WebSphere MQ• Sonic MQ• RMI• EJB• SMTP• Tibco Rendezvous• .NET WCF• ISO 8583• EDI• SAP • custom
Stubs Vs Service VirtualizationStubs Vs Service Virtualization
� Stubs� Inside-out approach that disassociates a test case
with a dependent systems
� Brittle
� Limited reuse
Service Virtualization provides a complete environment for developing and testing versus complex, dependent systems
Parasoft Proprietary and Confidential 28Parasoft Proprietary and Confidential 28
� Limited reuse
� Static response
� Service Virtualization� Creates an environment in which to run “rich” test scenarios
� Virtualized assets represent real system behavior
� All virtualized-assets are reusable
� Virtualized assets are programmable and extensible
Complexity is a Barrier to InnovationComplexity is a Barrier to Innovation
Scheduled Access
Parasoft Proprietary and Confidential 29Parasoft Proprietary and Confidential 29
3rd Party System
Evolving ComponentMainframe
Access to Dev / Test SystemsAccess to Dev / Test Systems
Configuration
Complex
ERP
Mainframe
ExternalDatabase
CloudApp
Parasoft Proprietary and Confidential 30Parasoft Proprietary and Confidential 30
Test Access
Simple Difficult
Configuration
InternalService
InternalApplication 3rd Party
Service
ExternalApplication
InternalDatabase
App
MessageQueue
ESB
Web Server
Service Virtualization & DependenciesService Virtualization & Dependencies
Service Virtualization
Configuration
� Access� Dependent applications difficult
� Scheduling conflicts
� High access fees
� Geo-political boundaries
� 3rd party or partner applications
� Configuration
Complex
Parasoft Proprietary and Confidential 31Parasoft Proprietary and Confidential 31
Virtual Lab
Configuration
Access
� Configuration� Complex to configure
� No control
� Limited variability
� Consumes test time
DifficultSimple
Easy
The Total Cost of System AccessThe Total Cost of System Access
Service Virtualization dramatically reduces the cost of deliveringa dev / test environment
Hardware Savings
Staged Application Instance
Cost � Only manage the functionality
you need
� Only manage data you need
� Low-Risk “disposable instance” that can be accessed at anytime
� Dramatically reduce test
Parasoft Proprietary and Confidential 32Parasoft Proprietary and Confidential 32
Access EfficiencyStaged w/ Virtual Test Lab
System Behavior
� Dramatically reduce test environment setup and management
Service Virtualization
Service Virtualization ExamplesService Virtualization Examples
Mobile Application development and extension
Agile/Parallel development limited by system dependencies
Parasoft Virtualize is used throughout the SDLC to speed access to and reduce the costs of managing development and test environments
Parasoft Proprietary and Confidential 33Parasoft Proprietary and Confidential 33
Capacity Constrained staged environments
Limited access to mainframes, ERPs, or 3rd party systems
Test data management for complex transactions
The Test Environment ChallengeThe Test Environment Challenge
Test environment access is outside the control of development and test leaving gaps in the process
IT OperationsParallel development
delays… Need simple,
realistic access to
dependent components…
Too much time
Scheduling
DependentApplications
Staged Assets
Parasoft Proprietary and Confidential 34Parasoft Proprietary and Confidential 34
Too much time
waiting for access…Need
reliable test data
Need a realistic test
environment easy to
maintain
Configuration
Access Limits
3rd Party Assets Virtual Environments
Hyper Visor
App App App
The Test Environment ChallengeThe Test Environment Challenge
Test environment access is outside the control of development and test leaving gaps in the process
IT Operations
Scheduling
DependentApplications
Staged Assets
Parasoft Proprietary and Confidential 35Parasoft Proprietary and Confidential 35
Configuration
Access Limits
3rd Party Assets Virtual Environments
Hyper Visor
App App App
Create, Manage, Provision
Value of Service VirtualizationValue of Service Virtualization
� Increase Access to “Trustworthy” Test Environments� Provide a complete test environment
� Provide access 24/7
� Flexibility provides a platform for extending test scenarios, coverage
� Reduce Operational Expenditures
Service Virtualization drives better software qualitywhile reducing CapEx and OpEx
Parasoft Proprietary and Confidential 36Parasoft Proprietary and Confidential 36
� Reduce Operational Expenditures� Significantly reduce test environment configuration time
� Simplify test environment access
� Remove the bottleneck of test environment setup
� Reduce Capital Expenditures� Software assets can be run anywhere
� Significantly reduce bandwidth demand
� Reduce dependency on staged environments
Parasoft Proprietary and Confidential 37Parasoft Proprietary and Confidential 37
Parasoft Service VirtualizationParasoft Service Virtualization
Questions?
Parasoft Proprietary and Confidential 38Parasoft Proprietary and Confidential 38
Wayne AriolaChief Strategy OfficerParasoft [email protected]
Virtual Lab ManagementVirtual Lab Management
Strengths Weaknesses
� “Low” Risk
� Reduce CAPEX
� Reduce Server Sprawl
� Manage Environment
� Must Have Access
� Permissions to Control
� Configuration
Permutations
Parasoft Proprietary and Confidential 39Parasoft Proprietary and Confidential 39
� Manage Environment
Complexity
� Uniformity for Global
Teams
� Operational Efficiency
Permutations
� Test Data
� Performance
� Hardware Capacity
Constraints
The Value/ROI Drivers – HumanThe Value/ROI Drivers – Human
Virtual Lab Service Virtualization
� Reduce OPEX � Eliminate scheduling delays
associated with staged test labs
� Eliminate ‘some’ configuration
delays
� Eliminates configuration time by
delivering more granular ‘service’
versus system
� Eliminates scheduling delays
� Understand
Environment
Complexity
� Gives a good view of inter-
relationship of applications
within geo/political control
� Gives a complete view of all
system dependencies
Parasoft Proprietary and Confidential 40Parasoft Proprietary and Confidential 40
Complexity
� Operational
Efficiency
� Reduces demand on test lab
resources
� Reduces delays of scheduling,
setup and re-configuration
� Eliminates delays associated with
� System configuration
� Data management
� Performance configuration
� Setup and tear down
The Value/ROI Drivers – PhysicalThe Value/ROI Drivers – Physical
Virtual Lab Service Virtualization
� Reduce CAPEX
and server
sprawl
� Versus physical test lab will
significantly reduce CAPEX
� Can reduce incremental CAPEX
due to physical test capacity
constraints
� Can reduce demand on
“virtualized” infrastructure
resources
� Eliminate � Can eliminate physical test lab
BUT be careful what you want to
� Eliminates the need for
incremental increases in physical
Parasoft Proprietary and Confidential 41Parasoft Proprietary and Confidential 41
� Eliminate
physical test
labs
(Careful!)
BUT be careful what you want to
virtualize due to compliance and
data privacy concerns
incremental increases in physical
test labs or virtual lab capacity
The Value/ROI Drivers – Project/ProcessThe Value/ROI Drivers – Project/Process
Virtual Lab Service Virtualization
� Time to Market
(Careful!)
� Increases time to market by
providing access to “production-
like” environments within
geo/political control
� Increases time to market by
providing more complete access
� Provides a net new environment
earlier in the process
� Access anytime
or anywhere
� Good access to completed and
built applications within control
� Access to both complete and
incomplete components
� Manage and � Good view of applications � Much broader view of total
Parasoft Proprietary and Confidential 42Parasoft Proprietary and Confidential 42
� Manage and
understand
Environment
� Good view of applications
managed within resource pool
� Much broader view of total
system interdependencies
� Understanding of 3rd party
impacts
� Defect
Reproduction
� Captures great information for
applications under control
� Extends information by including
components outside of control
� Better Testing � Increased code coverage
� Increase test efficiency
� Increased test coverage
� Increased # and quality of tests
� Increase test efficiency
� Increased test coverage
� Increased # and quality of tests
What’s the View from Your Seat?What’s the View from Your Seat?
Strategy
Risk Profile
Governance
Business
Portfolio
Application
Parasoft Proprietary and Confidential 43Parasoft Proprietary and Confidential 43
Actions
Project
Process
Risk Profile Application
Release
Iteration
Code
From Automated to ContinuousFrom Automated to Continuous
Continuous testing accelerates the SDLC bymanaging quality expectations and actionable tasks
Requirements
Defined
Policy Management
Development
Defect Prevention
Development Testing Integration Testing System Testing
Defect Remediation Tasks
Parasoft Proprietary and Confidential 44Parasoft Proprietary and Confidential 44
Development
Static Analysis
Unit/Component
Peer Review
Automated Tests
API/Service Tests
Smoke Test
Security Tests
Automated Tests
Functional Tests
Scenario Tests
Performance Tests
CI
Build
Go
Release Path
No GoBusiness
Decision
Service Virtualization – Test Environment Access