Real Application Testing
-
Upload
oracleonthebrain -
Category
Technology
-
view
2.609 -
download
0
description
Transcript of Real Application Testing
![Page 1: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/1.jpg)
Oracle Database 11g
Real Application Testing
and Enterprise Manager Virtual ClassroomLynn Ferrante and Michael Whelan
![Page 2: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/2.jpg)
Oracle Database 11g
Real Application Testing
and Enterprise Manager Virtual ClassroomLynn Ferrante and Michael Whelan
![Page 3: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/3.jpg)
3
Database Testing Agenda
• Overview
• Database testing
• Eliminate the risk of change by assessing the impact of
change on production databases
• Database masking
• Eliminate risk of moving data to non-production environments
![Page 4: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/4.jpg)
4
Database Testing
• Despite expensive testing success rate low
• Many issues go undetected
• System availability and performance negatively impacted
• Cause of low success rate
• Existing tools provide inadequate testing
• Simulate synthetic workload instead of replaying actual
production workload
• Provide partial workflow coverage
Database Replay makes real-world testing possible
![Page 5: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/5.jpg)
5
1 2 3 4 5
Typical Steps in Test Phase
![Page 6: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/6.jpg)
6
1: Install & Setup Test System
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
24 Days
4 Days
![Page 7: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/7.jpg)
7
2: Understand Application Usage
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
+20 Days
+0 Days
![Page 8: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/8.jpg)
8
3: Identify Key Transactions
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
+20 Days
+0 Days
![Page 9: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/9.jpg)
9
4: Develop Test Scripts
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
+80 Days
+2 Days
![Page 10: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/10.jpg)
10
5: Run Test Workload
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
+5 Days
+5 Days
![Page 11: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/11.jpg)
11
Innovate Faster
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
149 Days
11 Days
![Page 12: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/12.jpg)
12
0
20
40
60
80
Tim
e T
ak
en
(D
ays)
Install & Setup Understand
Application
Usage
Identify Key
Transactions
Generate
Workload
Run Test
DB Replay
LoadRunner
2
20
54
80
2420
DB Replay: 2 weeks
“Traditional” Testing: 30 weeks
Total Testing Time
500
Comparison of LoadRunner & Oracle Real
Application with e-Business Suite
![Page 13: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/13.jpg)
13
Load Testing andAccelerators for
Packaged Apps & Web Services
Functional Testing andAccelerators for
Packaged Apps & Web Services
Test Management
Real Application Testing
Data Masking
Real UserExperience Monitoring
Service Level Management
Packaged Application Management
SOA Application Management
Diagnostic and Tuning
Provisioning, Configuration Management
Change Tracking and Synchronization
Oracle Enterprise Manager
ApplicationOperations
Management
Top-Down Application Management
ApplicationQuality
Management
Application Testing Suite
![Page 14: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/14.jpg)
14
Real Application Testing
![Page 15: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/15.jpg)
15
Real Application Testing
Remediate
Test
Deploy
• Value
• Rapid technology adoption
• Higher testing quality
• Business Benefit
• Lower cost
• Lower risk
Change
![Page 16: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/16.jpg)
16
Ovum Summit
Oracle Real Application Testing – business agility
through superior testinghttp://www.oracle.com/corporate/analyst/reports/infrastructure/dbms/ovum
-real-application-testing.pdf
ROI
![Page 17: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/17.jpg)
17
Real Application Testing
SQL
PERFORMANCE
ANALYZER
DATABASE
REPLAY
Identify SQL statement with performance regressions
caused by system changes
Captures a production workload and allows replay of the workload on a test system
Execute SQL concurrently
Execute SQL serially
![Page 18: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/18.jpg)
18
Real Application Testing
Database Replay
![Page 19: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/19.jpg)
19
Database Replay
• Replay actual production database workload in test environment
• Identify, analyze and fix potential instabilities before making changes to production
• Capture Workload in Production
• Capture full production workload with real load, timing & concurrency characteristics
• Move the captured workload to test system
• Replay Workload in Test
• Make the desired changes in test system
• Replay workload with full production characteristics
• Honor commit ordering
• Analyze & Report
• Errors
• Data divergence
• Performance divergence
Analysis & Reporting
![Page 20: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/20.jpg)
20
Database Replay: Supported Changes
Changes
Unsupported
Changes Supported
•Database Upgrades, Patches
•Schema, Parameters
•RAC nodes, Interconnect
•OS Platforms, OS Upgrades
•CPU, Memory
•Storage
•Etc.
ClientClient
…Client
Middle Tier
Storage
Recording of External Client
Requests
![Page 21: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/21.jpg)
21
Common Usage Scenarios
• Database initialization/config parameter changes
• Schema changes
• Changes to operating systems and hardware
• Gathering statistics
• Database upgrades including patch deployments
• Implementation of tuning recommendations
• Partitioning
• Compression
• etc
![Page 22: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/22.jpg)
22
…
…
Database Replay Workflow
Production (10.2.0.4) Test (11.1)
Capture ReplayAnalysis &
ReportingProcess
StorageStorage
Mid-Tier
Replay Driver
Clients
![Page 23: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/23.jpg)
23
Step 1: Workload Capture
File 1
File 2
File n
…
Production System • All external client requests
captured in binary files
• System background, internal activity excluded
• Minimal performance overhead for capture
• For RAC, shared and local file system supported
• Specify interesting time period for capture, e.g., peak workload, month-end processing, etc.
• Can capture on 10.2.0.4 and replay on 11g
File System
ClientClient
…Client
Middle Tier
Storage
![Page 24: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/24.jpg)
25
Step 2: Process Workload Files
File 1
File 2
File n
…
File 1
File 2
File n
…
Metadata
Capture Files Replay Files
Test System• Setup test system
• Test DB is at same point in time
as before production capture
• Use RMAN to physically restore
production db from backup
• Use Snapshot standby
• Use imp/exp, Data Pump, etc.
• Processing transforms captured
data into replayable format
• Once processed, workload can be
replayed many times
• For RAC copy all capture files to
single location for processing
![Page 25: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/25.jpg)
26
Step 3: Replay Workload
• Replays workload preserving timing, concurrency and dependencies of the capture system
• Replay Driver is a special client program that consumes processed workload and sends requests to the replay system
• Replay Driver consists of one or more clients. For workloads with high concurrency, it may be necessary to start multiple clients to drive workload
Test System
File 1
File 2
File n
…
Metadata
Replay Files
Replay Driver
![Page 26: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/26.jpg)
27
Analysis & Reporting
• Comprehensive reports are provided for analysis purposes
• There (3) types of divergences are reported
• Data Divergence: Number of rows returned by each call are
compared and divergences reported
• Error Divergence: For each call error divergence is reported
• New: Error encountered during replay not seen during capture
• Not Found: Error encountered during capture not seen during
replay
• Mutated: Different error produced in replay than during capture
• Performance Divergence
• Capture and Replay Report: Provides high-level performance
information
• ADDM Report: Provides in-depth performance analysis
• AWR, ASH Report: Facilitates comparative or skew analysis
![Page 27: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/27.jpg)
28
Demo
![Page 28: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/28.jpg)
29
Real Application Testing
SQL Performance Analyzer
![Page 29: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/29.jpg)
30
• Enables identification of SQL performance regressions beforeend-users can be impacted
• SPA can help with any change that impacts SQL execution plan
• DB upgrades
• Optimizer statistics refresh
• New indexes, Materialized Views, Partitions, etc.
• Automates SQL performance tracking of hundreds of thousands of SQL statements – impossible to do manually
• Captures SQL workload with low overhead
• Integrated with SQL Tuning Advisor and SQL Plan Baselines for regression remediation
SPA Benefits
![Page 30: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/30.jpg)
31
……
ClientClient
…Client
Capture SQL
Middle Tier
Storage
Oracle DB
Re-execute SQL Queries
Production Test
Use SQL Tuning Advisor to tune regression
![Page 31: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/31.jpg)
32
SQL Performance Analyzer: Workflow
Production Test
Capture
SQL (STS)
Transport
STS
Execute SQL
Pre-change
Execute SQL
Post-change
Compare
Perf.
Steps (1) (2) (3) (4) (5)
(6) Reiterate
(7)
No
Yes(7)
Done?
Make Change
Production Change / Tuning Deployment
Tuned System
![Page 32: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/32.jpg)
33
Step 1: Capture SQL Workload
• SQL Tuning Set (STS) used to store SQL
workload
• STS includes:
• SQL Text
• Bind variables
• Execution plans
• Execution statistics
• Incremental capture used to populate
STS from cursor cache over a time
period
• SQL tuning set’s filtering and ranking
capabilities filters out undesirable SQL
• SQL workload captured in 10.2.0.1 and
higher can be used for SPA tasks in 11gProduction Database
Cursor Cache
SQL Tuning Set
Incremental Capture
![Page 33: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/33.jpg)
34
Step 2: Move SQL Workload to Test System
Production Database
Cursor Cache
SQL Tuning Set
Test Database
SQL Tuning Set
• Use Enterprise Manager
Export/Import
![Page 34: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/34.jpg)
35
Step 3: Execute SQL Before Making Change
• Establishes SQL workload performance
baseline
• SQL execution plan and statistics
captured
• SQL executed serially (no concurrency)
• Each SQL executed only once
• DDL/DML skipped
• Option to do Explain Plan only analysis
SQL Tuning Set
Fetch Next SQL
SQL Performance Analyzer
Execution Plan &
Statistics
Test Execute
Save Result
![Page 35: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/35.jpg)
36
Step 4: Execute SQL After Making Change
• Re-execute SQL after change
• Gathers new SQL execution plans
and statistics
SQL Tuning Set
SQL Performance Analyzer
Fetch Next SQL
Execution Plan &
Statistics
Test Execute
Save Result
Completed
• Manually implement the planned change
• Database upgrade, patches
• Optimizer statistics refresh
• Schema changes
• Database parameter changes
• Tuning actions, e.g., SQL Profile creation
![Page 36: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/36.jpg)
37
Step 5: Compare & Analyze Performance
Completed Completed
Compare
SQL Performance
SQL Performance Analyzer
Analysis Report
• Compare performance using
different metrics, e.g.,
• Elapsed Time
• CPU Time
• Optimizer Cost
• Buffer Gets
• SPA Report shows impact of
change for each SQL
• Improved SQL
• Regressed SQL
• Unchanged SQL
• Fix regressed SQL using SQL
Tuning Advisor or SQL Plan
Baselines
![Page 37: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/37.jpg)
38
SPA Report
![Page 38: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/38.jpg)
39
SPA Report
![Page 39: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/39.jpg)
40
Demo
![Page 40: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/40.jpg)
41
Real Application Testing Summary
• Provides cost-effective and easy-to-use solution of assessing impact of change on production systems
• Holistic and actual workload testing results in lower risk
• Brings down testing cycle from months down to days
• Lowers hardware costs by eliminating need for mid-tier and application setup on test system
• Maximizes ROI by leveraging Diagnostics and Tuning Pack to remediate regressions
• With Real Application Testing businesses can
• Stay competitive
• Improve profitability
• Be compliant
![Page 41: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/41.jpg)
42
Real Application Testing: Tools of the Trade
SQL Performance Analyzer Database Replay
What is it? • Predicts SQL performance deviations
before end-users can be impacted,
helps assess impact of change on
SQL response time
• Replays real database workload
on test system, helps assess
impact of change on workload
throughput
How it works? • Executes each SQL, stored in SQL
Tuning Set, in isolation using
production context and then
compares before and after execution
plans and run-time statistics
• Captures workloads and replays it
with production characteristics
including concurrency,
synchronization &
dependencies
When to use? • Unit testing of SQL with the goal to
identify the set of SQL statements
with improved/regressed performance
• Comprehensive testing of all
sub-systems of the database
server using real production
workload
SQLSQL
Dependency
Concurrency
Speed up/down
![Page 42: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/42.jpg)
43
Database Diagnostic Pack
![Page 43: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/43.jpg)
44
Database Diagnostic Pack
Introduction
• Part of the database management pack family
• Provides unique functionalities
• Industry’s first performance self-diagnosing engine
• Lightweight, Automatic, Out-of-box performance data warehouse
• Guided problem resolution
• Introduces a new, easy to use performance tuning methodology
• Not just another tool
• Core functionality built right into the database server
• Makes it most efficient, integrated and accurate
![Page 44: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/44.jpg)
45
Database Diagnostic Pack
Key Features
• Automatic Workload Repository (AWR)
• Active Session History (ASH)
• Automatic Database Diagnostic Monitor (ADDM)
![Page 45: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/45.jpg)
46
SQL
Advisor
High-load
SQL
IO / CPU
issues RAC issues
Automatic Diagnostic Engine
Snapshots in
Automatic Workload
Repository
Self-Diagnostic Engine
System
Resource
Advice
Network +
DB config
Advice
• Top Down Analysis Using
AWR Snapshots
• Throughput centric - Focus on
reducing time ‘DB time’
• Classification Tree - based on
decades of Oracle
performance tuning expertise
• Real-time results
• Don’t need to wait hours to
see the results
• Pinpoints root cause
• Distinguishes symptoms
from the root cause
• Reports non-problem areas
• E.g. I/O is not a problem
How Does ADDM Work?
![Page 46: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/46.jpg)
47
Database Tuning Pack
Introduction
• Part of the database management pack family
• Provides unique, automatic and deterministic SQL
tuning functionality
• Core functionality built in the Oracle Database 10g
kernel
• SQL Tuning Advisor
• SQL Access Advisor
• Requires diagnostic pack
![Page 47: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/47.jpg)
48
SQL Profiles
Contains auxiliary information specific to a given SQL statement
– Customized optimizer settings
Based on past execution history
– Compensation for missing or stale statistics
– Compensation for errors in optimizer estimates
Estimation errors occur due to data skews and correlations, complex filters and joins
Doesn’t freeze the execution plan
Doesn’t require any change to the SQL text– Ideal for Packaged Apps
Persistence: Works across shutdowns & upgrades
Transportable across databases (10.2)
![Page 48: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/48.jpg)
49
Data Masking Pack
![Page 49: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/49.jpg)
50
Oracle Data Masking Data Masking for Non-Production Environments
• Eliminate risk of moving data to
non-production environments
• Share data with partners
without exposing sensitive data
• No impact on production
environments
• Sensitive data never leaves the
database (masked in place)
• Automated masking process
preserves application integrity
LAST_NAME SSN SALARY
AGUILAR 203-33-3234 40,000
BENSON 323-22-2943 60,000
LAST_NAME SSN SALARY
ANSKEKSL 111—23-1111 40,000
BKJHHEIEDK 111-34-1345 60,000
Production
Non-Production
![Page 50: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/50.jpg)
51
Summary
• Real Application Testing for managing risk• Provides cost-effective and easy-to-use solution of assessing
impact of change on production systems
• Lowers hardware costs by eliminating need for mid-tier and application setup on test system
• Maximizes ROI by leveraging Diagnostics and Tuning Pack to remediate regressions
• Diagnostic and Tuning Pack for proactive
management
• Industry’s first performance self-diagnosing engine
• Data Masking for security of data
![Page 51: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/51.jpg)
52
0
20
40
60
80
Tim
e T
ak
en
(D
ays)
Install & Setup Understand
Application
Usage
Identify Key
Transactions
Generate
Workload
Run Test
DB Replay
LoadRunner
2
20
54
80
2420
DB Replay: 2 weeks
“Traditional” Testing: 30 weeks
Total Testing Time
500
Comparison of Traditional Approach &
Oracle Real Application with e-Business
Suite
![Page 52: Real Application Testing](https://reader034.fdocuments.net/reader034/viewer/2022042613/54625f38af7959b92a8b4e14/html5/thumbnails/52.jpg)