Post on 24-Jan-2018
1
Lights-Out Testing for Lights-On Business
Ty Henson
September 20, 2017
2
Digital RiskIt’s not just about security, hackers, and data theft
3
Agenda
• Effective automation testing
• Creating on-demand automation
• The right tool for the job
Presented by Ty HensonWorksoft, Senior Solution Architect
4
Late Defect Discovery is Expensive and Disruptive!
0
9599
22781331
$18 ($0)
$180 ($1.7 M)
$360 ($820K)
$1,800 ($2.3 M)
$0
$200
$400
$600
$800
$1,000
$1,200
$1,400
$1,600
$1,800
$2,000
0
2,000
4,000
6,000
8,000
10,000
12,000
Discovered inRequirements/Specs
Discovered in TestExecution
Discovered in UAT Discovered inProduction
Defect Correction Cost (5 Project Sample)
Defects Cost to correct per defect
5
Mainframe Client Server Web 1.0 Web 2.0 / Mobile HTML 5 VR & Biometrics
1980 2000
1990
2020
2010 2030
Modern Day UI EvolutionAs the way applications are written changes. The way we test must also.
6
Business Analysts/QA
“Does my Business Work?”
Developers/Test Engineers
“Does my Software Work?”
UI Tests – The Bulk of Business Application Testing
6
End to End (UI)
Integration ( API)
Unit (Code)
UI Testing Remains the Most Challenging to Automate.
Last 10% takes 90% of the time!
7
Digital Business ProcessesComplex
8
1 Discover
Business Analysts/Super Users
2 Automate
Quality Assurance/Automation Professionals
3 Run
IT Operations/Project Management
Capture, visualize, document & analyze end-to-end business processes
Worksoft Analyze®
Build and maintain a reusable portfolio of business process test automation
Worksoft Analyze®, Worksoft Certify®
Run automation at industrial scale for projects & ongoing operations
Worksoft Certify®,Impact, Execution Manager, Business Process Procedure
The Worksoft Solution
9
Demonstration
10
Agenda
Effective automation testing
Cross-Platform End to End
• Creating on-demand automation
• The right tool for the job
11
12
• End to End Process
• Enter Order
• Outbound Delivery
• Post Goods Issue
• Invoicing
Test Data Concepts
13
• End to End Process
+ Add Inventory
• Enter Order
• Outbound Delivery
• Post Goods Issue
• Invoicing
Test Data Concepts
14
• End to End Process
+ Find or Generate Materials
+ Add Inventory
• Enter Order
• Outbound Delivery
• Post Goods Issue
• Invoicing
+ Cleanup for Repeatability
Test Data Concepts
15
Agenda
Effective automation testing
Cross-Platform End to End
Creating on-demand automation
Self-sufficient and repeatable
• The right tool for the job
16
• Hundreds of tests under management
across multiple applications
• Tests must often be scheduled in
sequence with complex dependencies
• Difficult to manage distributed and
diverse testing resources
• Central monitoring can be a challenge
• Testing time is strictly limited and
closely scrutinized
The Trouble with Automation
17
UI Automation Challenges
Access Orchestration Scale
18
1 Discover
Business Analysts/Super Users
2 Automate
Quality Assurance/Automation Professionals
3 Run
IT Operations/Project Management
Capture, visualize, document & analyze end-to-end business processes
Worksoft Analyze®
Build and maintain a reusable portfolio of business process test automation
Worksoft Analyze®, Worksoft Certify®
Run automation at industrial scale for projects & ongoing operations
Worksoft Certify®,Impact, Execution Manager, Business Process Procedure
The Worksoft Solution
19
Industrial Scale Intelligent Scheduling
• Distributable agents to [geographically distributed] physical or virtual workstations
• Mass parallel execution ability
• Automatic load balanced execution
• Dependencies to control intelligent execution
Lights Out Testing and Continuous TestingIndustrial Scale Intelligent Scheduling and Interoperable Execution
Interoperable Execution
Providing applications like those above the ability to trigger automation execution
• Rest API calls
• Command line interface (CLI)
And more…
20
Running Certify From CI - Jenkins
Jenkins builds and creates build number
Build Number
Call Execution Manager
Execution Manager
REST
Run Certify Tests
Get Status
Remote Test Lab 1CI/CD Server
EMAgent
EMAgent
EMAgent
On premise Test Lab 2
21
Demonstration
22
Lights Out Testing (LOT) provides the ability to
execute a battery of integrated tests at pre-
determined times. These tests will be re-runable
and will be based on exception reporting,
requiring manual intervention only if the test fails.
Tests can be scheduled at predetermined times
and leverage:
• Off-hour bandwidth
• Use of infrastructure and personnel for other
tasks
• The execution of large, data centric integrated
tests during off hours
• Elimination of manual validation and audit
What is Lights Out Testing (LOT)?“Because we’re covering all
of the business processes
in an automated fashion,
we’re finding defects that
also give us a snapshot of
the data that’s causing a
script to fail. Because we’re
now on a 24-hour testing
cycle, we can isolate the
cause more efficiently,
which also results in cost
savings.”
— Sada Krishnan, IT Project
Manager, Reliant
23
Cross Platform End to End Automation
Bidirectional Communication with Other Applications and capabilities
And more…
SOA Rest APIMock
Services
File
SystemRelationa
l Queries
Microsoft
OfficeExecutio
n
Execution Manager
Interoperability
24
Quality and Validation for DevOps and Packaged Application Testing SDLC
Requirements
• Test Cases Documented
Architect Develop
• Build Unit tests
CI &
Unit Test
• Run Unit Tests
CD to Staging Integration
• API Functional
• API Load Testing
• Application Functional
UAT
• End to End
• Requirement validation
• Exploratory Testing
CD to Prod Smoke
• Production Validation
Prepare
• Test Cases Documented
Discover
• Business process changes
• Integration
Realize
• End to End
• Requirement validation
• Exploratory Testing
Go Live Run
• Production Validation
• Developer based testing specific to
development of custom applications
• Application functional testing common to
both Testing SDLC – UI and end user
focus
Ide
a
Ma
jor
Developer
Testing
25
Supporting Continuous Integration and Continuous Delivery
Develop
Code Check in
BacklogGroom Stories
Sprint starts
CI Continuous Integration CD Continuous Delivery
Development
Sprints
Requirements
• Test Cases Documented
Architect Develop
• Build Unit tests
CI &
Unit Test
• Run Unit Tests
CD to Staging Integration
• API Functional
• API Load Testing
• Application Functional
UAT
• End to End
• Requirement validation
• Exploratory Testing
CD to Prod Smoke
• Production Validation
Worksoft
Execution
Manager
Application is deployed and connected
End-to-end is available
Development testing when
code is not connected
26
Digital Business ProcessesComplex
27
Agenda
Effective automation testing
Cross-Platform End to End
Creating on-demand automation
Self-sufficient and repeatable
The right tool for the job
Presented by Ty HensonWorksoft, Senior Solution Architect
28
Case Study:
US Federal Government Program
10 million+ participants; 400 core processes & 500,000
business process steps tested nightly
Virtual
Machines (150+)
29
Case Study:
Global Luxury Goods Manufacturer
Enterprise wide automation effort for end-to-end business
process validation with Worksoft Certify.
Significant reduction in production defects for SAP, Web
applications and more.
600 hours of automation
running daily on 30+ virtual machines
55,000 hours
of manual
labor annually
replaced
80% business
coverage of 500
core processes.
SCALE SAVINGS COVERAGE
30