Lights-Out Testing for Lights-On Business
Transcript of Lights-Out Testing for Lights-On Business
![Page 1: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/1.jpg)
1
Lights-Out Testing for Lights-On Business
Ty Henson
September 20, 2017
![Page 2: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/2.jpg)
2
Digital RiskIt’s not just about security, hackers, and data theft
![Page 3: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/3.jpg)
3
Agenda
• Effective automation testing
• Creating on-demand automation
• The right tool for the job
Presented by Ty HensonWorksoft, Senior Solution Architect
![Page 4: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/4.jpg)
4
Late Defect Discovery is Expensive and Disruptive!
0
9599
22781331
$18 ($0)
$180 ($1.7 M)
$360 ($820K)
$1,800 ($2.3 M)
$0
$200
$400
$600
$800
$1,000
$1,200
$1,400
$1,600
$1,800
$2,000
0
2,000
4,000
6,000
8,000
10,000
12,000
Discovered inRequirements/Specs
Discovered in TestExecution
Discovered in UAT Discovered inProduction
Defect Correction Cost (5 Project Sample)
Defects Cost to correct per defect
![Page 5: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/5.jpg)
5
Mainframe Client Server Web 1.0 Web 2.0 / Mobile HTML 5 VR & Biometrics
1980 2000
1990
2020
2010 2030
Modern Day UI EvolutionAs the way applications are written changes. The way we test must also.
![Page 6: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/6.jpg)
6
Business Analysts/QA
“Does my Business Work?”
Developers/Test Engineers
“Does my Software Work?”
UI Tests – The Bulk of Business Application Testing
6
End to End (UI)
Integration ( API)
Unit (Code)
UI Testing Remains the Most Challenging to Automate.
Last 10% takes 90% of the time!
![Page 7: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/7.jpg)
7
Digital Business ProcessesComplex
![Page 8: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/8.jpg)
8
1 Discover
Business Analysts/Super Users
2 Automate
Quality Assurance/Automation Professionals
3 Run
IT Operations/Project Management
Capture, visualize, document & analyze end-to-end business processes
Worksoft Analyze®
Build and maintain a reusable portfolio of business process test automation
Worksoft Analyze®, Worksoft Certify®
Run automation at industrial scale for projects & ongoing operations
Worksoft Certify®,Impact, Execution Manager, Business Process Procedure
The Worksoft Solution
![Page 9: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/9.jpg)
9
Demonstration
![Page 10: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/10.jpg)
10
Agenda
Effective automation testing
Cross-Platform End to End
• Creating on-demand automation
• The right tool for the job
![Page 11: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/11.jpg)
11
![Page 12: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/12.jpg)
12
• End to End Process
• Enter Order
• Outbound Delivery
• Post Goods Issue
• Invoicing
Test Data Concepts
![Page 13: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/13.jpg)
13
• End to End Process
+ Add Inventory
• Enter Order
• Outbound Delivery
• Post Goods Issue
• Invoicing
Test Data Concepts
![Page 14: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/14.jpg)
14
• End to End Process
+ Find or Generate Materials
+ Add Inventory
• Enter Order
• Outbound Delivery
• Post Goods Issue
• Invoicing
+ Cleanup for Repeatability
Test Data Concepts
![Page 15: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/15.jpg)
15
Agenda
Effective automation testing
Cross-Platform End to End
Creating on-demand automation
Self-sufficient and repeatable
• The right tool for the job
![Page 16: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/16.jpg)
16
• Hundreds of tests under management
across multiple applications
• Tests must often be scheduled in
sequence with complex dependencies
• Difficult to manage distributed and
diverse testing resources
• Central monitoring can be a challenge
• Testing time is strictly limited and
closely scrutinized
The Trouble with Automation
![Page 17: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/17.jpg)
17
UI Automation Challenges
Access Orchestration Scale
![Page 18: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/18.jpg)
18
1 Discover
Business Analysts/Super Users
2 Automate
Quality Assurance/Automation Professionals
3 Run
IT Operations/Project Management
Capture, visualize, document & analyze end-to-end business processes
Worksoft Analyze®
Build and maintain a reusable portfolio of business process test automation
Worksoft Analyze®, Worksoft Certify®
Run automation at industrial scale for projects & ongoing operations
Worksoft Certify®,Impact, Execution Manager, Business Process Procedure
The Worksoft Solution
![Page 19: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/19.jpg)
19
Industrial Scale Intelligent Scheduling
• Distributable agents to [geographically distributed] physical or virtual workstations
• Mass parallel execution ability
• Automatic load balanced execution
• Dependencies to control intelligent execution
Lights Out Testing and Continuous TestingIndustrial Scale Intelligent Scheduling and Interoperable Execution
Interoperable Execution
Providing applications like those above the ability to trigger automation execution
• Rest API calls
• Command line interface (CLI)
And more…
![Page 20: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/20.jpg)
20
Running Certify From CI - Jenkins
Jenkins builds and creates build number
Build Number
Call Execution Manager
Execution Manager
REST
Run Certify Tests
Get Status
Remote Test Lab 1CI/CD Server
EMAgent
EMAgent
EMAgent
On premise Test Lab 2
![Page 21: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/21.jpg)
21
Demonstration
![Page 22: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/22.jpg)
22
Lights Out Testing (LOT) provides the ability to
execute a battery of integrated tests at pre-
determined times. These tests will be re-runable
and will be based on exception reporting,
requiring manual intervention only if the test fails.
Tests can be scheduled at predetermined times
and leverage:
• Off-hour bandwidth
• Use of infrastructure and personnel for other
tasks
• The execution of large, data centric integrated
tests during off hours
• Elimination of manual validation and audit
What is Lights Out Testing (LOT)?“Because we’re covering all
of the business processes
in an automated fashion,
we’re finding defects that
also give us a snapshot of
the data that’s causing a
script to fail. Because we’re
now on a 24-hour testing
cycle, we can isolate the
cause more efficiently,
which also results in cost
savings.”
— Sada Krishnan, IT Project
Manager, Reliant
![Page 23: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/23.jpg)
23
Cross Platform End to End Automation
Bidirectional Communication with Other Applications and capabilities
And more…
SOA Rest APIMock
Services
File
SystemRelationa
l Queries
Microsoft
OfficeExecutio
n
Execution Manager
Interoperability
![Page 24: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/24.jpg)
24
Quality and Validation for DevOps and Packaged Application Testing SDLC
Requirements
• Test Cases Documented
Architect Develop
• Build Unit tests
CI &
Unit Test
• Run Unit Tests
CD to Staging Integration
• API Functional
• API Load Testing
• Application Functional
UAT
• End to End
• Requirement validation
• Exploratory Testing
CD to Prod Smoke
• Production Validation
Prepare
• Test Cases Documented
Discover
• Business process changes
• Integration
Realize
• End to End
• Requirement validation
• Exploratory Testing
Go Live Run
• Production Validation
• Developer based testing specific to
development of custom applications
• Application functional testing common to
both Testing SDLC – UI and end user
focus
Ide
a
Ma
jor
Developer
Testing
![Page 25: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/25.jpg)
25
Supporting Continuous Integration and Continuous Delivery
Develop
Code Check in
BacklogGroom Stories
Sprint starts
CI Continuous Integration CD Continuous Delivery
Development
Sprints
Requirements
• Test Cases Documented
Architect Develop
• Build Unit tests
CI &
Unit Test
• Run Unit Tests
CD to Staging Integration
• API Functional
• API Load Testing
• Application Functional
UAT
• End to End
• Requirement validation
• Exploratory Testing
CD to Prod Smoke
• Production Validation
Worksoft
Execution
Manager
Application is deployed and connected
End-to-end is available
Development testing when
code is not connected
![Page 26: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/26.jpg)
26
Digital Business ProcessesComplex
![Page 27: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/27.jpg)
27
Agenda
Effective automation testing
Cross-Platform End to End
Creating on-demand automation
Self-sufficient and repeatable
The right tool for the job
Presented by Ty HensonWorksoft, Senior Solution Architect
![Page 28: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/28.jpg)
28
Case Study:
US Federal Government Program
10 million+ participants; 400 core processes & 500,000
business process steps tested nightly
Virtual
Machines (150+)
![Page 29: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/29.jpg)
29
Case Study:
Global Luxury Goods Manufacturer
Enterprise wide automation effort for end-to-end business
process validation with Worksoft Certify.
Significant reduction in production defects for SAP, Web
applications and more.
600 hours of automation
running daily on 30+ virtual machines
55,000 hours
of manual
labor annually
replaced
80% business
coverage of 500
core processes.
SCALE SAVINGS COVERAGE
![Page 30: Lights-Out Testing for Lights-On Business](https://reader031.fdocuments.net/reader031/viewer/2022021419/5a676e537f8b9a656a8b5123/html5/thumbnails/30.jpg)
30