Benchmark Factory User Guide 6-8-1

Benchmark Factory forDatabases 6.8.1

User Guide

Copyright

2012 Quest Software, Inc.

ALL RIGHTS RESERVED.

This guide contains proprietary information protected by copyright. The software described inthis guide is furnished under a software license or nondisclosure agreement. This software may beused or copied only in accordance with the terms of the applicable agreement. No part of thisguide may be reproduced or transmitted in any form or by any means, electronic or mechanical,including photocopying and recording for any purpose other than the purchasers personal usewithout the written permission of Quest Software, Inc.

The information in this document is provided in connection with Quest products. No license,express or implied, by estoppel or otherwise, to any intellectual property right is granted by thisdocument or in connection with the sale of Quest products. EXCEPT AS SET FORTH INQUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FORTHIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMSANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTSINCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OFMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT,INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES(INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESSINTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE ORINABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THEPOSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respectto the accuracy or completeness of the contents of this document and reserves the right to makechanges to specifications and product descriptions at any time without notice. Quest does notmake any commitment to update the information contained in this document.

If you have any questions regarding your potential use of this material, contact:

Quest Software World HeadquartersLEGAL Dept5 Polaris WayAliso Viejo, CA 92656email: [email protected]

Refer to our Web site (www.quest.com) for regional and international office information.

Patents

Toad for Cloud Databases contains patent pending technology.

Trademarks

Benchmark Factory for Databases User GuideCopyright

3

Quest, Quest Software, the Quest Software logo, Benchmark Factory, Spotlight, LiteSpeed,Simplicity at Work, Toad, TOAD, T.O.A.D., and TOADWORLD and vToadare trademarks andregistered trademarks of Quest Software, Inc. For a complete list of Quest Softwares trademarks,see http://www.quest.com/legal/trademark-information.aspx. Other trademarks and registeredtrademarks used in this guide are property of their respective owners.

Benchmark Factory for Databases

User Guide

September 2012

Table of Contents

Copyright 2

Getting Started 17

Getting Started 17

Benchmark Factory Community 17

17

New in this Release 17

New for Release 6.8.1 17

New for Release 6.8 18

New For Release 6.7 18

Understanding Benchmark Factory 20

About Benchmark Factory 20

The Benchmark Factory Environment 20

Integration with Other Quest Software Products 21

Understanding the Benchmark Factory Environment 21

Understanding Benchmark Factory Terminology 22

How Benchmark Factory Works 25

Create a Script With Benchmark Factory 26

1. Set Up Benchmark Factory 27

Benchmark Factory Components 35

BenchmarkFactory Console 35

Agent(s) 36

Run Reports 37

Cutting and Pasting of Run Reports 38

Testing Server Throughput 38

Testing SQL Scalability 39

Testing Real Application Clusters 40

Benchmark Factory for Databases User GuideTable of Contents

5

Repository Manager 41

Views 44

Script View 44

Jobs Queue View 45

Properties View 47

Agents View 47

Output View 47

Properties 47

How Do I... 91

Overview 91

How do I... 91

User Scenario Wizard 92

Profile Creation Wizard 92

Overview 92

Edit a Profile 94

Submit Job Dialog 94

Run Reports 96

Cutting and Pasting of Run Reports 97

Testing Server Throughput 97

Testing SQL Scalability 98

Testing Real Application Clusters 99

Advanced Creation Objects (Object Mapping) 100

Benchmark Objects Wizard 101

Changing Scaling Factors 103

To Change Scale Factors 103

Execute File Wizard 105

BFScript Wizard 105


6

Script Wizard 105

Script Assist 107

SQL Scalability Wizard 107

Overview 107

Overview Tab 109

109

Transactions 109

Real-Time 110

Summary 110

Messages 111

Copy a Load Scenario to a Replay, Mix, Goal, or Scalability Test 112

Running SQL Preview 113

Replace Child Latencies 113

Goal Testing Quick Start 115

Overview 115

Actions Perfomred in This Quick Start 115

Use Global Search and Replace 124

Creating an ODBC Trace File 125

Real Application Clustering (RAC) Quick Start 125

Overview 125

Purpose of this Quick Start 126

Actions Performed in This Quick Start 126

Save the Selected Job to the Script View 136

Schedule A Job 136

About Latency 137

Set the Number of Users on an Agent 138

Stagger User Startup 140


7

Running Benchmark Factory with Multiple Agents 140

Benchmark Factory Console 140

Benchmark Factory Agent 141

Running a Job With Multiple Agents 141

Creating Load Scenarios 142

Load Scenario Wizard 142

Overview 142

Settings 144

About Settings 144

General Settings 144

Benchmarks Settings 146

Timing Settings 147

User Load (Measurement Intervals) 148

LatencySettings 150

Repository Settings 152

Agent Settings 154

Profiles 156

Overview 156

Edit Profiles 156

Drivers 158

Profile Properties 163

Overview 163

General Tab 163

Options Tab 165

Profile Properties Real-Time Counters Tab 166

Clustering Tab 169


8

Wizards 171

Overview 171



Data Repository Migration Wizard 173

Data Repository Migration Wizard 173





Capture and Replay Oracle Workload 176

Industry Standard Benchmark (TPC-C, TPC-H...) 192

Validate Oracle RAC Cluster Scalability 193

Test Database Scalability 195

Test SQL For Scalability 196

Replay Load from an Oracle Trace File 197

Test Stored Procedures for Scalability 200

Replay Load from SQL Server Trace Table 201

Import from Foglight Performance Analysis Repository 202

Import from a Delimited Text File 203

Replay Load From an ODBC Trace File 205

Import from Oracle Dynamic Performance View 206

Import from Quest Tools Export Files 207

Create a Custom Load Scenario (Scalability, Goal...) 208

Mix Test 208

Replay Test 209

Goal Test 210

Scalability Test 212


9



BFScript 214

BFScript Wizard 214





Transaction Wizard 224

Transaction Wizard 224

Test Stored Procedures for Scalability 225

Create SQL Statement Transaction 226

Create Prepared SQL Statement Transaction(s) 227

Create Execute Prepared SQL Statement Transaction 231

Import Transactions 232



Import and Replay Load Scenarios From an Oracle Trace File 233

Replay Load from a SQL Server Trace Table 234

Import from a Delimited Text File 235

Replay Load from an ODBC Trace File 235

Create a Custom User Scenario 236

Testing Results (Run Reports) 237

Benchmark Factory TestingResults 237

Benchmark Factory Testing Results Terminology 237

Exporting Test Results to Excel Using the Command Line 238

Real-Time Load Testing Results 239

Overview-Viewing Real-time Statistics Workloads 239


10

Change Graph Views 239

Summary Tab 242

Realtime Statistics Tab 242

Run vs.Userload Tab 244

Transaction Node 246

Benchmark Factory Run Reports 248

Run Reports 248


Run Reports Viewer 254

View Load Testing Results 256

Using Benchmark Factory Run Reports 257

View Multiple Run Results 270

Export BenchmarkFactory Test Results to Excel 274

Benchmarks 276

Overview 276

Realistic Expectations When Using Benchmarks 276

What Benchmarks Measure 276

Provided Benchmarks 276

AS3AP Benchmark 277

Best Practices 277

Scalable Hardware Benchmark 278

How the Scalable Hardware Benchmark Works 278

Scaling Factor 278

Best Practices 278

TPC-B Benchmark 279

Overview 279

Certification of Transaction Processing Council (TPC) Testing Results 279

Best Practices 279


11

Scaling Factor 280

TPC-C Benchmark 280

Overview 281

TPC-C Tables 281


Best Practices 282

TPC-D Benchmark 283

Overview 283


Best Practices 283

Scaling Factor 284

TPC-E Benchmark 284

Overview 284


Best Practices 285

Scaling Factor 285

TPC-H Benchmark 285

Overview 285

Certification of Transaction Processing Council Testing Results 286

Best Practices 286

Scaling Factor 286

BFScripts 288

About Scripts 288

Using Scripts in Username and Password Fields 290

Dates and Numbers 290

$BFCreditCardExp 290

$BFCurrentDate 290

BFCurrentDateTime 291


12

$BFDate 292

File Access 293

$BFFileArray 293

Global Variable 294

$BFGetGlobalVar 294

$BFSetGlobalVar 294

$BFSetGlobalVarRtn 295

Name and Address 295

$BFAddress 295

$BFAddress2 295

$BFCity 296

$BFCompany 296

$BFCountry 297

$BFEmail 297

$BFFirstName 297

$BFFullName 298

$BFLastName 298

$BFMiddleInitial 298

$BFPhone 299

$BFState 299

$BFZipCode 300

Numerical Manipulation 300

$BFFormat 300

$BFProd 301

$BFSum 301

Random Numbers 302

$BFRand 302

$BFRandRange 302


13

$BFURandRange 303

$BFURand 303

Random String 304

$BFCreditCard 304

$BFRandList 304

$BFRandMultiList 305

$BFRandStr 306

$BFURandList 307

$BFList 307

String Manipulation 308

$BFAsc 308

$BFChr 308

$BFConcat 309

$BFLeft 309

$BFLen 310

$BFLower 310

$BFMid 311

$BFRight 311

$BFTrim 311

$BFTrimLeft 312

$BFTrimRight 312

$BFUpper 313

Test Info 313

$BFGetVar 313

$BFMaxNode 314

$BFNode 314

$BFNumberOfIterations 314

$BFProfile 315


14

$BFRunID 315

$BFSetVar 316

$BFSetVarRtn 316

$BFUserCounter 317

$BFUserID 318

$BFUserLoad 318

$BFNextUserload 318

$BFPrevUserload 319

Quick Starts 320

About the Quickstarts 320

Real Application Clustering (RAC) Quick Start 320

Overview 320

Purpose of this Quick Start 321


Upgrading Your Database Server Quick Start 331


Oracle Trace Import-Database Replay Quick Start 339

Overview 339


Standard Benchmark Quick Start 346

Overview 346

Actions Performed Duirng this Quick Start 346

Goal Testing Quick Start 353

Overview 353

Actions Perfomred in This Quick Start 354

Troubleshooting 363

Agent Connection 363

Use Benchmark Factory with SQL Server 2005 Client 363


15

MySQL Initialization Settings 363

Appendix 364


Show Data/Show Graph 364

Graph Legend 365

Toolbar 366

Print 366

Copy Data to Clipboard 366

Copy Graph to Clipboard 366

Load Configuration 366

Save Configuration 366

Set as Default 366

Clear Configuration 367

Customize List Controls 367

Create a SQL 7 Trace Table Using the SQL Server Profiler 367



Oracle Instant Client Installation 370

Migrating Repository Data Using the DOS Command Line 371

Store Procedure Examples 372

Oracle Trace File Activation 373

Support Bundle 375

System Requirements/Upgrade Requirements/Supported Databases 376

System Requirements 376

Shortcut Keys 378

Copyright, Licensing, and Licensing Agreement 380

Copyright 380


16

Licensing and Virtual User Packs 381

Modify the Benchmark Factory License 382

Modify the Benchmark Factory Virtual User Pack 383

Legal Notice 383

384

384

Appendix:Contact Quest 396

Contact Quest/About Quest Software 396

About Quest 396

Contacting Quest Software 396

ContactingQuest Support 396

Index 398

Getting Started

Getting StartedBenchmark Factory is an intuitive, easy to use database load testing tool. The following linksprovide you with information to quickly start the testing process and gather results.

l Video: Take a Tour of Benchmark Factoryl About Benchmark Factory (page 20)

l Understanding the Benchmark Factory Environment (page 21)

l Understanding Benchmark Factory Terminology (page 22)

l How Benchmark Factory Works (page 25)

l Benchmark Factory Components

l Create a Script With Benchmark Factory (page 26)

l Benchmark Factory TestingResults (page 237)

l Quick Starts

l Real Application Clustering (RAC) Quick Start (page 320)

l Upgrading Your Database Server Quick Start (page 331)

l Oracle Trace Import-Database Replay Quick Start (page 339)

l Standard Benchmark Quick Start (page 346)

l Goal Testing Quick Start (page 353)

l Quest Benchmark Factory Web Site (contains additional videos and white papers)

Benchmark Factory Community

Get the latest product information, find helpful resources, and join a discussion with theBenchmark Factory for Databases team and other community members. Join the BenchmarkFactory for Databases community at http://www.benchmarkfactory.inside.quest.com.

New in this Release

New for Release 6.8.1

This release of Benchmark Factory for Databases provides bug fixes.

1

Benchmark Factory for Databases User GuideGetting Started

18

New for Release 6.8

New features in this release:

l New Oracle Workload Capture Method-This release of Benchmark Factory forDatabases provides a new method to accomplish Oracle Capture/Replay using Oracle'sFine Grained Auditing (FGA) objects.

l Customer Improvement Program -Benchmark Factory now tracks feature utilization inorder to provide feedback to improve Benchmark Factory for Databases.

l Automatic Partitioning-Automatic Partitioning of TPC-H tables and indexes for Oracletesting has been added.

l User Interface for SQLStatements-The user interface for SQLStatements has beenenhanced.

l Reports- A performance comparison report has been added to Run History to compareOracle runs.

l Wizards-The Oracle and SQL Server Capture wizards user interfaces have been streamlinefor easier use.

New For Release 6.7

New features in this release:

l Improved Run Reports User Interface-Benchmark Factory Run Reports has beenimproved to provide easier access to test results tables and graphs.

Benchmark Factory for Databases User GuideGetting Started

19

l Import and Export of Test Results-You can now import and export test results formRun Reports.

l Increased Benchmark Loading Speed-Benchmarks used in the database testing processnow load faster to reduce database testing time.

Understanding Benchmark Factory

About Benchmark FactoryBenchmark Factory is a database performance and code scalability testing tool that simulatesusers and transactions on the database and replays production workload in non-productionenvironments. This enables developers, DBAs, and QA teams to validate that their databases willscale as user load increases, application changes are made, and platform changes areimplemented. Benchmark Factory is available for Oracle, SQL Server, DB2, Sybase, MySQL, andother databases via ODBC connectivity. The following topics provides an overview of theBenchmark Factory Environment:

l The Benchmark Factory Environment (page 20)

l Understanding the Benchmark Factory Environment (page 21)

l Understanding Benchmark Factory Terminology (page 22)

l How Benchmark Factory Works (page 25)

l Create a Script With Benchmark Factory (page 26)

The Benchmark Factory EnvironmentBenchmark Factory places enormous stress on a database system, which is typically hard toachieve in a standard testing environment. A system typically breaks under extreme load. Byidentifying system capacity and performance bottlenecks before they occur, Benchmark Factoryfacilitates proactive testing, which in turn reduces downtime, development costs, and potentialloss of revenue. Benchmark Factory allows you to:

l Determine system throughput and capacity for database systems

l Simulate thousands of concurrent users with a minimal amount of hardware

l Find applications that do not scale well with an increase in the number of users

l Find breaking points, weak links, or bottlenecks of a system

l Quantify application or server performance

All test results are collected and stored in the repository for data analysis and reporting.Benchmark Factory collects a vast amount of statistics, including overall server throughput(measured in transactions per second, bytes transferred, etc.) and detailed transaction statistics by

2

Benchmark Factory for Databases User GuideUnderstanding Benchmark Factory

21

individual workstation producing a load. You use these statistics to measure, analyze, andpredict the capacity of a system.

Integration with Other Quest Software Products

Benchmark Factory integrates with other Quest Software products to allow IT departments toquickly measure the capacity and performance of their systems to ensure that users willexperience fast response times. For instance, Benchmark Factory can run multiple workloads inconjunction with Spotlight products. This provides the ability to detect and diagnose issues,allowing users to resolve bottlenecks, slow performance, and application flaws before anapplication is entered into production.

Understanding the Benchmark Factory EnvironmentNote: The Benchmark Factory Quick Starts provide procedures that introduce you to theBenchmark Factory environment. See "About the Quickstarts" (page 320) for more information.

The following provides a narrative on the terminology, application, and mechanics of usingBenchmark Factory.

Automated Testing, Virtual Users, and Agents

When you load test with Benchmark Factory, you are performing automated testing. Automatedtesting is the process of using software to test computer hardware or software implying that thesoftware is used instead of actual users. The software acts as a "virtual" user. For example, sayyou would like to test your database with two hundred virtual users over a given period of time.Benchmark Factory lets you select two hundred virtual users and the length of time you wish toperform the test.

Benchmark Factory uses Agents to deploy the virtual users. An Agent is a software routine thatwaits in the background and performs an action when a specified event occurs. One Agent cansimulate thousands of virtual users (limited by hardware and workload characteristics) at a time.Each virtual user has their own connection to the system under test.

Understanding Benchmarks

A benchmark is a performance test of hardware and/or software on a system-under-test.Benchmark Factory provides the option of using industry standard benchmarks during the loadtesting process. Benchmarks measure system peak performance when performing typicaloperations.

Benchmark Factory comes equipped with the following industry standard benchmarks:

l AS3AP Benchmark (page 277)

l Scalable Hardware Benchmark (page 278)


22

l TPC-B Benchmark (page 279)

l TPC-C Benchmark (page 280)

l TPC-D Benchmark (page 283)

l TPC-E Benchmark (page 284)

l TPC-H Benchmark (page 285)

Understanding Benchmark Factory TerminologyThe following provides a list of terminology required to understand the Benchmark Factory loadtesting process.

l A script is the user's workspace that allows you to create and save jobs.

l Scripts are created in the Script View. See "Script View" (page 44) for moreinformation.

l When ready to run, jobs are submitted to the Jobs Queue View. See "Jobs QueueView" (page 45) for more information.

l The Load Scenario Wizard is the starting point for the creation of scripts. See"Load Scenario Wizard" (page 175) for more information.

l A job contains load scenario, execution of a file, creation/deletion of benchmark objects,or a combination of each.

l A load scenario is an assembled test comprised of user scenarios and/or transactions.These tests can be run with multiple virtual users. A load scenario can be one of thefollowing:

l Mixed Workload: A Mixed Workload test runs for a specified time at eachpredetermined user load level. Each user will run a transaction mix based uponthe weights defined on the transactions tab. For example, if a test has twotransactions, A and B, with A having a weight of one and B having a weight offour, on average B will run four times for every time A is run once. The run orderwill be randomly generated for each user so they are not all running the sametransaction simultaneously. That run order is used for that user each time the test isperformed to ensure reproducible results.

l Goal Test: A goal test is used to find maximum throughput or response timegoals. A transaction mix is executed at user load levels, determined by setting abeginning, ending, and interval value. When run, the specified goal criterion isevaluated at the end of each iteration and the test ends once the goal or maximumuser load has been reached.

l Replay Test: A Replay Test runs multiple transactions with each one runningindependently on a specified number of users. The test will run until the definednumber of executions for each transaction or a specified time limit is reached.


23

l SQL Scalability Test: A SQL Scalability test executes each transactionindividually for each userload and timing or execution period. For example, atest could have two transactions, A and B, and two userloads of 10 and 20, withan iteration length of one minute. Transaction A would execute continually forone minute at userload 10, then B would do the same. Next A will run at userload20, followed again by test B, for a total time of 4 minutes.

Click here for load scenario creation procedures.

l A user scenario is a series of one or more transactions to be executed in order to make asingle transaction, normally associated with user behavior simulated against the system-under-test. An Execute SQL Statement from a file User Scenario will execute SQLstatements as defined in an XML file. See "User Scenario Wizard " (page 233) for moreinformation.

l A transaction is a single unit of work in the testing process, used in load scenarios anduser scenarios. For example, a SQL statement. Click here for procedures to create atransaction.

l Execute a file allows you to execute a file during the running of a job. See "Execute FileWizard" (page 174) for more information.

l Create or delete benchmark objects configures a system-under-test for industry standardbenchmark tests to measure system performance. Each standard benchmark has beendeveloped with specific system configuration requirements, that include tables, indexes,data, etc. See "Benchmark Objects Wizard" (page 171) for more information.


24


25

How Benchmark Factory WorksThe following steps provide an overview of how Benchmark Factory components interact duringthe load testing process.

1. The BenchmarkFactory Console (page 35) implements the workload testing process andcontrols one or more distributed agent machines. Each agent machine can simulatethousands of users. Each simulated user executes transactions and records statistics.

2. Benchmark Factory Agent(s) (page 36)machines simulate virtual users. The Agents sendtransactions to the system-under-test. The Agents record statistics that includes how muchdata the resulting transaction contained and how long it took to get the results. At theend of an iteration, each agent machine reports its findings back to Benchmark Factory.

3. A server (system-under-test) is the database Benchmark Factory connects to. BenchmarkFactory is server neutral and network protocol independent. Benchmark Factory usesvendor client libraries of the system-under-test. Any system that the client software cansupport, Benchmark Factory can support. Its only requirement is that Agent machinesmust be able to connect to a server through an appropriate manner and it supports thetest system.

4. The Repository Manager (page 41) stores all testing data.

5. Results are viewed from Run Reports (page 248).


26

Create a Script With Benchmark FactoryThe following provides a narrative discussion on how to create a script using BenchmarkFactory. This process consists of following steps:

1. Set up Benchmark Factory

2. Create a Script

3. Create a Profile

4. Create a Job

5. Run the Script

6. Analyze Test Results


27

Note: The Benchmark Factory Quick Starts provide procedures that introduce you to theBenchmark Factory environment. See "About the Quickstarts" (page 320) for more information.

1. Set Up Benchmark Factory

The Settings window displays default values used when creating new scripts. Changes to thesesettings affect only new load scenarios, not existing load scenarios.

To Display Benchmark Factory Settings

l Select Edit | Settings. The settings dialog displays.

The Settings window provides the following tabs:

l General Settings (page 144)

l Benchmarks Settings (page 146)

l Timing Settings (page 147)

l LatencySettings (page 150)

l Repository Settings (page 152)

l Agent Settings


28

2. Create a Script

Using the Benchmark Factory graphical user interface, you can create scripts in the Script View.You open one script at a time when using Benchmark Factory. See "BenchmarkFactory Console"(page 35) for more information.

All tasks required to complete a Benchmark Factory script can be accomplished using wizards.The first wizard you will use is the Load Scenario Wizard. The Load Scenario Wizard takes youthrough the steps required to create a script and complete a database load test. See "LoadScenario Wizard" (page 175) for more information.


29

The first step the Load Scenario wizard asks you to perform is to select a load scenario type fromthe load scenario dialog. Load Scenarios are an assembled load test comprised of:

l User scenarios: A series of one or more transactions to be executed in order normallyassociated with a behavior to be tested on the system-under-test. These are thecomponents used to build a load scenario.

l Transaction: A single unit of work in the testing process, such as retrieving a Web page,executing a SQL statement, writing a file, or sending an email.


30

After selecting the desired task type, the Load Scenario Wizard will guide you through the stepsfor script creation, depending on which type of task you select.

3. Create a Profile

Profiles are sets of information that Benchmark Factory uses to communicate with the system-under-test . This information includes, server name, IP address, and user name. You createprofiles using the:

l Profile Creation Wizard: (See "Profile Creation Wizard" (page 213) for more information.

l Edit Profiles (See "Edit a Profile" (page 94) for more information.


31

4. Create a Job

A job holds a load scenario, execution of a file, creation/deletion of benchmark objects, or acombination of each. Jobs are created in Benchmark Factory when a load scenario is added to ascript. A script can hold as many jobs as you chose to create.


32

Change Script Properties

Benchmark Factory script properties can be changed or adjusted to meet load testing requirement.You change properties by clicking on the node of the property you wish to change. For example,if you wish to change the user load test you click on the Load Scenario node. See About theScript Node for more information.


33

To change the User Load properties, select the User Load tab and select the desirednumber of users.

5. Run the Script

After completing the wizards and setting in the desired properties, you run the job by selecting

the Job Node and clicking the Submit Job icon , or right-clicking and selecting Submit Job.See "Submit Job Dialog " (page 222) for more information.


34

6. Analyze Test Results

Benchmark Factory provides a number of metrics to analyze a system-under-test. These include:

l Response, retrieval, and transactions times

l Transaction throughput in transactions per second (TPS)

l Amounts of data transferred in bites per second (BPS)

l Errors incurred

l Transaction Distribution

l Any user defined counters (Windows NT/2000 Performance Monitor) added using theBenchmark Factory Console.

Benchmark Factory provides the following testing results:

l Real time statistics (See "Realtime Statistics Tab " (page 242) for more information.)

l Run Reports (See "Run Reports " (page 248) for more information.)

l Testing results exported to Excel (See "Export BenchmarkFactory Test Results to Excel"(page 274) for more information.)

Benchmark Factory Components

BenchmarkFactory ConsoleThe Benchmark Factory console implements the database workload testing process. Thisinterface is where:

l Scripts are created and initiated. A script is the user workspace that allows you to designand save jobs.

l Jobs are created and submitted to the Jobs Queue View. See "Jobs Queue View" (page45) for more information.

l All script properties are set. See About the Script Node for more information.

l Settings can be made. See "About Settings" (page 144) for more information.AboutScripts (page 288)

l Users can access test results. See "Benchmark Factory TestingResults" (page 237) formore information.

l Agents are managed. See "Agent(s)" (page 36) for more information.

The Benchmark Factory Console contains the following views:

l Script View (page 44)

l Jobs Queue View (page 45)

l Properties View (page 47)

l Agents View (page 47)

l Output View (page 47)

3

Benchmark Factory for Databases User GuideBenchmark Factory Components

36

Agent(s)Benchmark Factory Agent(s) reside on each physical client machine and spawn multiple virtual-user sessions. Each Agentgenerates an information screen with tabs that contain a variety of Agent information. Benchmark Factory can controlhundreds of Agent machines. Each Agent can simulate up to 1500 virtual users, depending upon hardware and testconfiguration. Virtual users simulate the load against the system-under-test. Each virtual user is a separate thread, actingindependently of the other virtual users, with its own connection to the system-under-test. Each virtual user tracks its ownstatistics that includes transaction times and the number of times a transaction executes.

The Agent provides the follow tabs:

l Agent Test Status Tab

l Agent Virtual Users Tab

l Agent Transaction Tab

l Agent Output Tab


37

Run ReportsBenchmark Factory Run Reports is a separate executable that opens outside of BenchmarkFactory and provides a comprehensive and detailed collection of database load testing results.With Benchmark Factory you can drill down into a database to view a wide array of informationand statistics that gives you accurate insight into database performance. Run Reports Viewerallows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page254) for more information.

Note: Three instances of Run Reports can be viewed at one time.


38

Cutting and Pasting of Run Reports

From Benchmark Factory run reports you can cut and paste test results into the Clipboard. SeeCut and Pasting Reports for more information.

Testing Server Throughput

Server throughput is the measurement of database capacity. Benchmark Factory allows you toview a variety of server throughput statistics that include:

l Transaction/Second (TPS)

l Transaction Time

l Bytes/Second (BPS)

l Rows/Second (RPS)

l Total Bytes

l Total Errors

l Total Rows

l Response Time


39

Testing SQL Scalability

The Benchmark Factory transaction comparison graph provides users with a comparison of theindividual transaction performance as they compare to each other. This is most useful whenanalyzing a SQL Scalability test.


40

Testing Real Application Clusters

Benchmark Factory allows you to load test Oracle Real Application Clusters (RAC) to determinesystem capacity. The cluster performance graph allows users to view the performance ofindividual nodes.


41

Repository ManagerNote: If you create a new Benchmark Factory 5.5 repository, earlier versions of BenchmarkFactory will not work against this repository.

The Repository is a database where all of the test results are stored. Benchmark Factory insertstest results into the repository and provides an easy way to access the data. By default, theRepository is a MySQL database that resides on the same machine as Benchmark Factory forDatabases. The Repository can reside on another database server if required. Tto change thedatabase select the Data Source Name of the ODBC connection for the new database.

Note: If the database structure does not exist on the selected database, a prompt to create thestructure will appear when OK is clicked.

The supported repository database types are:

l Sybase Adaptive Server Anywhere

l Microsoft SQL Server

l MySQL

l Oracle


42

To launch the Repository Manager

1. Select Start | All Programs | Quest Software | Benchmark Factory for Databases |Repository | Repository Manager. The Repository Manager dialog displays.

Section Field/Icon Description

ConnectionParameters

Data SourceName

Data Source name of the ODBCconnection used to connect to therepository database.

ODBC Driver Current ODBC driver

User Name The User Name used to log into theselected database.

Password The Password associated with the username used to log into the database.


43


Displays the ODBC connectioninformation dialog for the selected datasource.

Displays the ODBC Data SourceAdministrator dialog, used to add andedit ODBC connections.

Tests the connection of the currentlyselected ODBC Data Source.

Maintenance Creates the repository objects on theselected database.

Deletes the repository objects on theselected database.Warning: This will delete all testresults stored in the Repository.

Launches the Data Migration Wizard.See "Data Repository MigrationWizard" (page 173) for moreinformation.


44

Views

Script View

The Script View displays the currently open script and provides user workspace that allows youto design and save jobs. When ready to run, jobs are submitted to the Jobs Queue. The ScriptView Toolbar provides user functionality when creating or editing load testing scripts.

Note: You must be in the script view to create jobs and edit jobs.

Script View Toolbar

The Script View toolbar provides user functionality when creating or editing load testing scripts.

Creates a blank job. You must click on the Script Node to active the icon.

Displays the Load Scenario Wizard (page 175)

Displays the User Scenario Wizard (page 233)


45

Displays the Transaction Wizard (page 224)

Displays the Benchmark Objects Wizard (page 171)

Displays the Execute File Wizard (page 174)

Displays the Submit Job Dialog (page 222)

Moves an item up in the Script View

Moves an item down in the Script View

Jobs Queue View

From the Jobs Queue, jobs are run, status can be viewed, and real-time testing results can beviewed. The Jobs Queue Toolbar (page 46) provides user functionality to save or delete jobs.The following job statues are provided:

l Scheduled: All jobs currently waiting to run or are scheduled to run a future time.

l Running: Job currently running.

l Completed: All completed jobs.


46

Jobs Queue Toolbar

Jobs Queue icons provides user functionality when running jobs.

Deletes the selected job from the Jobs Queue.

Saves the selected job to the Script View. See "Save the Selected Job to the Script View"(page 136) for more information.


47

Properties View

The Properties View allows you to monitor and managed scripts. Properties can beviewed for the:

l Script Node Properties (page 48)

l Jobs Node

Agents View

The Agent view displays the status of connected agents.

Output View

The Output window displays errors and messages that occur outside of running a job. Forexample, a .dll error or re-registering a benchmark.

Properties

About the Properties View

Benchmark Factory properties allow you to:

l Adjust script parameters

l View script properties


48

l Access Benchmark Factory Run Reports See "Run Reports " (page 248) for moreinformation.

l View realtime statistics (See "Realtime Statistics Tab " (page 242) for more information.

l Set and view latency See "About Latency " (page 137) for more information.

The following provides Benchmark Factory properties definitions:

l Script (See About the Script Node for more information.)

l Job (See Jobs Node for more information.)

l Latency (See "About Latency " (page 137) for more information.)

l Benchmark Properties See "Benchmark Properties" (page 87) for more information.)

Script Node Properties

Script Node

The Script node displays the Benchmark Factory Home page and Help options.

Jobs Node

Note: In the Jobs Queue view, not all properties are editable.

The Jobs node provides the following tabs:


49

l General

l Alerts

l Job Schedule

l Real-Time Counters

l Global Variables Node (page 72)

Load Scenario Node

Note: In the Jobs Queue view, no properties are editable.

The Load Scenario node provides the following tabs:

l Transaction Mix Tab (page 49)

l Transaction Tab

l Goal Test (page 210)

l User Load Tab (MeasJobs Nodeurement Intervals)

l Timing Tab

l Options Tab-Mixed Workload, Goal, and Scalability Test (page 54)

Transaction Mix Tab

Note: The Transaction Mix tab is only editable in the Script View.

This tab allows you to run a transaction mix based on the assigned weight. If a test hastwo transactions, A and B, with A having the weight of one and B having the weight ofsix, transaction B will run six for every time A runs once. The run order is randomlygenerated for users.


50

Settings Description

Name Name assigned to the transaction.

Weight Weight assigned to a transaction in a mix.

Percentage Percentage assigned to the transaction.

TotalWeight

Total weight of all the combined transactions.

Deletes a selected transaction from the mix.

User Scenario Node

The User Scenario node properties allows you to:

l Delete transactions

l Adjust transaction order

l Adjust latency (See "About Latency " (page 137) for more information.)


51

User Scenario From File

The User Scenario From File node properties allows you to:

l Executing a SQL Statements from a File (page 51)


Executing a SQL Statements from a File

When large Oracle trace files are imported into Benchmark Factory, they are converted toeditable XML files.

To Edit SQL Statements from a file

1. Click Edit.


52

2. Click File | Open.

The Open dialog displays.

3. Navigate to and select the required file XML file.

4. Click Open.

5. Under Session Details, click on the SQL statement you wish to edit. That statementdisplays in the SQL Statement view.


53

6. From the Prepare SQL tab, you can click in the upper right-hand corner to:

l Run SQL Preview See "Running SQL Preview" (page 113) for moreinformation.

l Launch the BFScript Wizard See "BFScript Wizard" (page 214) for moreinformation.

7. After you finish making the required edits, click File| Save to save your edits.


54

Transaction Node


The transaction node displays properties based on transaction type.

To view the Transaction node

Click the Transactions node in the Jobs Queue or Script View.

Options Tab-Mixed Workload, Goal, and Scalability Test

The Options tab specifies Repository, error handling, file execution, and databasecheckpoint settings.


55


Repository Settings Save Real-Time CounterInformationNote: To change real-timecounters please refer to ProfileProperties Real-Time CounterTab.

Saves counters andperformance monitorcounters to the Repositoryduring test. Sample rate isdetermined in job settings.

Save User Statistics Saves user level statistics

Save Response Distribution Saves transaction persecond and response timeper user distribution.

Log Transaction Results to disk Logs the transaction resultsof a job to disk.Note: Checking this optionimposes performancepenalties.

Error Handling Stop test after first error When checked, the test isstopped when an error is


56


reported.

Execute the followingprogram at thebeginning of eachiteration

File NameNote: This field acceptsBFScripts.

Name of the program toexecute at the beginning ofan iteration.

Enforce Timeout Enforces a timeout on thefile being executed.

Execute the followingprogram at the end ofeach iteration

File NameNote: This field acceptsBFScripts.

Name of the program toexecute at the end of aniteration.

Enforce Timeout Enforces a timeout on thefile being executed so thatif the program does notcomplete in the specifiedtime, it will be stopped, andthe load scenario willcontinue.

Database Checkpoints Perform checkpoint at start ofeach test iteration

Initiates a databasecheckpoint at the beginningof a test iteration.

Perform checkpoints during eachiteration

Initiates a databasecheckpoint during a testiteration.

Number of checkpoints Specifies the number ofcheckpoints to initiate.

Options Tab-Replay Test

The Repository Properties tab controls what information is saved in the Repository during therunning of a test, and allows you to set error handling properties. The database where allBenchmark Factory information is saved. It contains information about the transactions stored ineach benchmark DLL and is where all of the statistical information is stored about benchmarkexecutions.


57

To change Repository properties

1. Click the Load Scenario node.

2. Click the Options tab.

Setting Description

Repository Settings

Save Real-Time CounterInformation

Saves Benchmark Factory real time counters andperformance monitor counters to the repository duringtest.

Save User Statistics Saves user level statistics.

Save Distribution Saves transactions per second and response timedistribution.

Error Handling Stops the test when an error is returned.

Run Reports

Benchmark Factory Run Reports is a separate executable that opens outside of BenchmarkFactory and provides a comprehensive and detailed collection of database load testing results.


58

With Benchmark Factory you can drill down into a database to view a wide array of informationand statistics that gives you accurate insight into database performance. Run Reports Viewerallows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page254) for more information.







l Transaction Time


l Rows/Second (RPS)

l Total Bytes

l Total Errors

l Total Rows

l Response Time


59




60




61

SQL Statement Node

The SQL statement node displays individual SQL statements and allows you edit a SQLstatement or use the BFScript Wizard. See "BFScript Wizard" (page 214) for more information.


62

Create Objects Node


This node allows you to either create the benchmark objects or delete the benchmark objects.Benchmark objects are used by standard benchmark load scenarios. Benchmarks can be added toa job using the Benchmark Objects Wizard. See "Benchmark Objects Wizard" (page 171) formore information.

Field Description

Benchmark Name of the benchmark whose benchmark properties are beingedited.

Scale The factor used to increase the amount of data added to thedatabase.Valid benchmark scales are dependent upon the benchmark selected.For example, the AS3AP scales by a factor of 10 with valid scales of"1, 10, 20, 30..." and the TPC-C benchmark scales by a factory of 1,with valid scales of "1, 2, 3, 4..."


63

Field Description

Displays the Benchmark Properties page and allows you to edit thebenchmark scale and map database tables and indexes. See"Benchmark Properties" (page 87) for more information.

Always deleteobjects beforecreate

Checking this box deletes benchmark objects before creating newones.When unchecked, Benchmark Factory checks for the benchmarktables. If they exist, then the creation will be skipped.

Note: The scale factor is not checked.

Distribute loadusing BenchmarkFactory Agents

Distributes data generation across multiple agent machines to speedup the standard benchmark data load.

Number of virtualusers to performcreation/load

Number of virtual users used to perform the test.

Execute File Node


The Execute File node allows you to place a file execution in a job as a sibling to a loadscenario. The file executes during the running of a job.


64

Field Description

Execute File Type in or Select the File Name to execute:

Name of the file you wish to execute.

Note: This field accepts BFScripts. See "About Scripts" (page288) for more information.

Click to open a file.

Enforce Timeout Places a time constraint on the number of seconds a file has toexecute.

Enter a numberbetween 30 and 10000[n] Secs.

The number of seconds a file has to execute.

Jobs Queue Node Properties

Using the Jobs Node and Tabs

Note: Some of the nodes in the Jobs Queue are the same as the Script view. Not all nodes in theJobs Queue are editable and provide property viewing only.

The Jobs Node and tabs allow you to edit or view job parameters. These tabs can be usedwhen jobs are:

l Scheduled (page 64)

l Running (page 65)

l Completed (page 66)

Scheduled

When a job is scheduled to run, it is placed in the Scheduled Jobs Queue.


65

Note: Not all of the tabs are editable in the schedule section.

The Job Queue provides the following tabs:

l Job Setup Tab (page 66)

l Create Objects Node (page 67)

l Run Job Tab (page 68)

l See Result Tab (page 69)

Running

As a job runs, clicking on the Run Job tab allows you to view realtime testing results.

The following provides details on realtime testing viewing:

l Summary Tab (page 242)

l Realtime Statistics Tab (page 242)

l Run vs.Userload Tab (page 244)

l Transaction vs. Load Tab (page 82)


66

Completed

Jobs in the Completed Jobs Queue section allow you to perform limited test edits and completedtesting results using the See Result Tab (page 69).

Note: Not all of the tabs are editable in the schedule section.

The following provides tab details in the Completed tests section:

l Job Setup Tab (page 66)

l Create Objects Node (page 67)

l Run Job Tab (page 68)

l See Result Tab (page 69)

Job Setup Tab

This tab allows to view or edit job parameters and provides the following tabs:

l General

l Jobs Alerts

l Real-Time Counters

l Global Variables Node (page 72)


67

Create Objects Node


This node allows you to either create the benchmark objects or delete the benchmark objects.Benchmark objects are used by standard benchmark load scenarios. Benchmarks can be added toa job using the Benchmark Objects Wizard. See "Benchmark Objects Wizard" (page 171) formore information.


68

Field Description

Benchmark Name of the benchmark whose benchmark properties are beingedited.

Scale The factor used to increase the amount of data added to thedatabase.Valid benchmark scales are dependent upon the benchmark selected.For example, the AS3AP scales by a factor of 10 with valid scales of"1, 10, 20, 30..." and the TPC-C benchmark scales by a factory of 1,with valid scales of "1, 2, 3, 4..."

Displays the Benchmark Properties page and allows you to edit thebenchmark scale and map database tables and indexes. See"Benchmark Properties" (page 87) for more information.

Always deleteobjects beforecreate

Checking this box deletes benchmark objects before creating newones.When unchecked, Benchmark Factory checks for the benchmarktables. If they exist, then the creation will be skipped.

Note: The scale factor is not checked.

Distribute loadusing BenchmarkFactory Agents

Distributes data generation across multiple agent machines to speedup the standard benchmark data load.

Number of virtualusers to performcreation/load

Number of virtual users used to perform the test.

Run Job Tab

This RunJob tab allows you to schedule jobs and view job results. It provides thefollowing tabs:

l Schedule

l Summary Tab (page 242)


69

l Realtime Statistics Tab (page 242)

l Run vs.Userload Tab (page 244)

l Transaction vs. Load Tab (page 82)

See Result Tab

The see results tab allows you to:

l Select results of tests based on profile selection

l Sort test results by successful completion or those that completed with errors.

l View graphs of completed tests.

To use the See Results Tab:

1. From the Profiles drop-down, select the All Profiles or the required profile.


70

2. From the Status drop-down, select the required run type.

3. Click on the required requied test to view graphs.

4. To delete a test, select the test and click Delete in the upper right section of the dialog.


71

5. To view Run Reports, select the required test and click the Show Test Results button inthe upper right section of the dialog.

See "Run Reports " (page 248) for more information.

Load Scenario Node


The Load Scenario node provides the following tabs:

l Transaction Mix Tab (page 49)

l Transaction Tab

l Goal Test (page 210)

l User Load Tab (MeasJobs Nodeurement Intervals)

l Timing Tab

l Options Tab-Mixed Workload, Goal, and Scalability Test (page 54)

User Scenario Node

The User Scenario node properties allows you to:


72

l Delete transactions

l Adjust transaction order


Transaction Node


The transaction node displays properties based on transaction type.

To view the Transaction node

Click the Transactions node in the Jobs Queue or Script View.

Global Variables Node

From the global variable node, you can add or delete the required variables.


73

To add a global variable:

1. Right-click to display the global variable dialog.

2. Enter the name and value.

3. Click Add.

The new global variable dislays.


74

Run Reports

Benchmark Factory Run Reports is a separate executable that opens outside of BenchmarkFactory and provides a comprehensive and detailed collection of database load testing results.With Benchmark Factory you can drill down into a database to view a wide array of informationand statistics that gives you accurate insight into database performance. Run Reports Viewerallows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page254) for more information.







l Transaction Time


l Rows/Second (RPS)

l Total Bytes


75

l Total Errors

l Total Rows

l Response Time




76




77

Summary Tab

The Summary Tab properties displays information on the job when it is running. You canmonitor the status of the test, which userload its running, transactions per second (TPS), and avariety of other datapoints that give you insight into the actual performance your databaseprovides under the userload being tested.

Iteration Overruns

Iteration overruns occur at the end of an iteration to allow time for all transactions submittedwithin the test iteration cycle to complete, so that all transaction statistics can be collected. Forexample, an agent may execute a transaction during the last five seconds of test iteration, if thistransaction takes 15 seconds to complete, an iteration overrun of 10 seconds will occur.


78

Realtime Statistics Tab

The Real Time Statistics node provides you with real-time graphs and raw data. This data allowsyou to spot system-under-test issues that may be affecting server performance. The Real TimeStatistics node displays real-time counter data. Right-clicking inside the graph displays a drop-down that allows you to change graph settings and view.

To view Real-Time Statistic

1. In the Jobs Queue, click the job running.


79

2. Select Click here to Add Datapoints. The Add Datapoints dialog displays.

3. Select the desired datapoints to view.


80

3. Click OK. The graph displays with the selected datapoints. (To change graph views,right-click. See "Change Graph Views" (page 364) for more information.)

Run vs.Userload Tab

The Run vs. Load tab compares the user load transaction against the selected datapoint allowingyou to view how the various datapoints are being affected by the job running.

To view Run Vs.Userload tab testing results:

1. While the job is running, click the RunJob tab, then select the Run vs.Userload tab.

2. Click the datapoints Vs. Userload drop-down and select the desired datapoint.


81

The graph updates with the selected datapoint.

3. To change graph views, right-click. See Change Graph Views for more information.


82

Transaction vs. Load Tab

The Transaction vs. Load tab compares the user load transaction against the selected datapointallowing you to view how the various datapoints are being affected by the job running.

To view Transaction vs.Load tab testing results:

1. While the job is running, click the RunJob tab, then select the Run vs.Userload tab.

2. Click the datapoints vs. Userload drop-down and select the required datapoint.


83

The graph updates with the selected datapoint.

3. To change graph views, right-click. See "Change Graph Views" (page 364) for moreinformation.

Messages Node

The Messages node displays:

l Job status

l Any errors or warnings


84

Latency

About Latency

Latency is the speed in which transactions are sent to a server or the period between receivingresults of one transaction and sending the next transaction to a serve. Benchmark Factory allowsyou to select the following latency models and distribution models:

Latency Types Distribution Models

No Delay (page 84) Absolute Distribution (page 86)

Interarrival Time (page85)

Uniform Distribution (page 86)

Keying Time (page 85) Negative Exponential Distribution (page86)

Think Time (page 85) Normal Distribution (page 86)

Poisson Distribution (page 87)

No Delay

No Delay means that transactions execute as fast as possible. As soon as one transaction isprocessed, the next transaction is issued against the server. In the case of a mixed workload test,each virtual user issues transactions as fast as possible. The No Delay option is used when thegoal of the test is to stress the system to its limits, without concern for accurately simulating


85

users. With No Delay specified, a relatively low number of users can stress the system to itslimits. As such, there is no easy way to correlate N virtual users running with no delay to somenumber of real users.

Interarrival Time

Interarrival Time derives its name from achieving a specific transaction rate as seen by a server(the rate at which transactions are arriving at a server.) When specifying Interarrival Time,Benchmark Factory is instructed, that regardless of how long a transaction actually takes toexecute, to ensure that the transactions arrive at a server at a specific interval.

Keying Time

Keying Time is used to simulate an amount of time performing data entry, before executing atransaction (entering information). In many cases, Keying Time is used with Think Time toprovide a delay before and after a transaction execute. Selecting Keying Time inserts a fixed orvariable delay before executing a transaction.

Think Time

Think Time is used to simulate an amount of time "to think" about the results of a previoustransaction. This could be time spent performing analysis on the results of a database query.Selecting Think Time inserts a fixed or variable delay after executing a transaction.

Latency Distribution Models

About

Benchmark Factory provides the following distribution models

l Absolute Distribution (page 86)

l Uniform Distribution (page 86)

l Negative Exponential Distribution (page 86)

l Normal Distribution (page 86)

l Poisson Distribution (page 87)


86

Absolute Distribution

Absolute is a fixed length delay. If a 2000 ms delay is specified for Keying Time (page 85) and a3000 ms delay is specified forThink Time (page 85), each time the transaction should execute,Benchmark Factory waits 2000 ms, then fires the transaction, then waits an additional 3000 msbefore deciding which transaction to execute next. If an Inter-Arrival model is chosen with adelay type of Absolute set to 2000 ms, Benchmark Factory marks the time, executes thetransaction, and assuming the transaction finishes in less than two seconds, waits until twoseconds from the marked time has elapsed before determining which transaction to execute next.

Uniform Distribution

Selecting a Uniform delay instructs that random delay should be chosen, with an equalprobability of being the minimum value, the maximum value, or any value in between. Uniformdelays are chosen when it is suspected that the delay is highly random within a range or aminimal amount of statistical analysis has been performed to determine how the actual usersreact. Suppose a uniform Keying Time (page 85) is selected with a minimum value of 1000 msand a maximum value of 1500 ms. If the transaction is executed more than 500 times, there is ahigh probability that each possible delay has been selected at least once. With the other delaytypes, this is not the case. If the same 2000 ms to 2500 ms uniform delay is set for Interarrivaltime, the tester essentially is setting the test so that a server sees the transaction every 2 to 2.5seconds, instead of exactly two seconds as in the Absolute delay time.

Negative Exponential Distribution

Similar to the Normal distribution, Negative Exponential inserts a random delay based on amathematical model. A Negative Exponential distribution should be chosen when most usershave latency of the mean latency or less. Negative Exponential is weighted heavily towardsmaller latencies being selected relative to the mean and maximum latency.

Normal Distribution

Normal distributions differ from Uniform delays in that most of the delays chosen by BenchmarkFactory will be close to the average, but can vary by as much as 10% of the mean. While aUniform delay is used when users have latencies within equal likelihood of being anywherebetween two values, Normal distributions are chosen when all users fall within a range, but mostof the modeled users have a latency close to the average latency.


87

Poisson Distribution

A Poisson distribution is very similar to the Normal distribution and can be used most placeswhere a Normal distribution delay could be used. The biggest difference between a Normaldistribution and a Poisson distribution is that Poisson selects discreet values.

Benchmark Properties

Benchmark Properties

Benchmark scale factors are used to increase the size of the database testing tables during thecreation of benchmark objects, and to correctly execute the benchmark transactions as needed tofully utilize the database objects created.

The Benchmark Properties page displays information on:

l Scales (page 87)Scales (page 87)

l Advanced (page 89)

l History Tables (page 89)

Scales

The scale dialog specifies the benchmark scale used to create table data


88

Field Description

Benchmark Scale The factor used to increase the amount of data added to thedatabase.Valid benchmark scales are dependent upon the benchmarkselected. For example, the AS3AP benchmark scales by a factor of10 with valid scales of "1, 10, 20, 30..." and the TPC-C benchmarkscales by a factory of 1, with valid scales of "1, 2, 3, 4..."

Show EmptyTables

Displays any tables that will be created but not populated withdata.

Estimated Size The approximate size that the tables will use in the databasewithout indexes. Additional space is needed for indexes.


89

Advanced

Advancer mapping provides views of the tables and indexes that allows you to determine wherethere tables and indexes will be stored in the database. This allows your database to run moreefficiently. See "Advanced Creation Objects (Object Mapping)" (page 100) for more information.

History Tables

History tables allows you to set the number of history tables to create. The TPC-B benchmark ismade up of only one transaction that updates three tables and inserts a record into a history table.Inserting one record into one history table limits testing performance. The Benchmark Factoryproperties page allows the user to set the number of history tables to create during a test. Thebest ratio of history tables to virtual users is based on database configuration and hardware. Thenumber of history tables to use is determined by the tester.


90

Field Description

Number of Historytables

Allows you to set the number of history tables to create.

How Do I...

OverviewThe How Do I section points you to the procedures required to perform load testing.

How do I...l Create a Load Scenario - This wizard walks you through the steps to create and run

scripts and load scenarios.

l Create a User Scenario - This wizard walks you through the steps to create auser scenario.

l Create a Profile - This wizard walks you through the steps to hook up to the data baseyou are testing.

l Edit a Profile

l Submit a Job - This wizard walks you through the steps to submit a job to the JobsQueue for testing.

l Use Run Reports

l Use Advanced Creation Objects

l Create Benchmark Objects for Load Testing This wizard walks you through the steps tocreate benchmark objects for load testing.

l Execute a File - This wizard walks you through the steps to execute a file outside ofBenchmark Factory.

l Change Scale Factors

l Use Benchmark Factory Scripting Capabilities - This wizard walks you through the stepsto use Benchmark Factory built-in functions for use in scripts and the load testing process.

l Use SQL Scalability Wizard

l Copy a Load Scenario to a Replay Test, Mix Test, or Goal Test

l Run SQL Preview

l Replace Child Latencies

l Perform a Goal Test

l Use Global Search and Replace

l Create a Customized ODBC Trace File

l Run a Real Application Cluster (RAC) Workload

l Save the Selected Job to the Script View

4

Benchmark Factory for Databases User GuideHow Do I...

92

l Set Latency

l Set the Number of Users on an Agent

l Stagger User Startup

l Run Benchmark Factory with Multiple Agents

User Scenario WizardThe User Scenario Wizard allows you to build user scenarios. A user scenario is a series of oneor more transactions to be executed in order, normally associated with user behavior simulatedagainst the system-under-test. From the User Scenario Wizard you can:

l Import and Replay Load Scenarios From an Oracle Trace File

l Replay load from SQL Server trace table

l Import from a delimited text file

l Replay from an ODBC trace File

l Create a custom user scenario

Profile Creation Wizard

Overview

Profiles are used by Benchmark Factory to connect to the system-under-test. The profile containsinformation required to connect to the system-under-test, such as server name, IP address, anduser name. The Profile Creation Wizard walks you through steps required to create a BenchmarkFactory profile.

To create a profile

1. Select Wizards | Profile Creation.

2. Click Next.

3. From the Driver drop-down list, select the required driver type.

4. Click Next.

5. Enter the User/Schema.


93

6. Enter the Password.

7. From the TNSDriver drop-down, select the required database.

or

Click the Direct tab and enter the required fields.

8. From the Connect as drop, select the required connection, Normal, SYSDBA, orSYSOPER.

9. Select the installed client if applicable.

10. Click the Verify connection before proceeding check box to verify the connection to thespecified database.

11. Click Next.

12. Enter a profile name.

13. Click Finish.


94

Edit a ProfileTo edit a profile

1. Click the Edit Profile icon .

2. Double-click on a profile to edit or click View Properties in the upper right corner ofthe dialog.

The Edit Profiles dialog displays. The following options are provided.

l Adds a profile

l Deletes a profile

l Displays profile properties allowing you to edit the profile

Submit Job DialogA job contains load scenario, execute file, creation/deletion of benchmark objects, or acombination of each. The Submit Job Wizard walks you through the process of adding a job tothe jobs queue. See "Jobs Queue View" (page 45) for more information.

To submit a job to the Jobs Queue

1. Select the Job node.

2. Right-click.

3. Select Submit Job. The Submit Job dialog displays.


95

The Submit Job dialog allows you to tailor your job needs and provides thefollowing options:

Section Description

Profile Allows you to select profiles already created.

Displays the Profile Creation Wizard. See"Profile Creation Wizard" (page 213) for moreinformation.

Displays the Profile setting dialog. See"General Tab" (page 163) for more information.

Actions Allows you to:

Run Job: Submits the job to the queue and runs thejob.

Run Job when queue becomes idle: Submits the jobto the jobs queue and puts it in a ready state so thatit will run when the currently ready or running jobcompletes.


96

Section Description

Run Job per schedule: Submits the next jobscheduled.

Start Agent and Run Job: Starts the agent and runsthe job.

Start Agent and Run Job when queue becomesidle: Starts the agent and runs the job when allprevious jobs have finished.

Start Agent and Run Job per schedule: Starts theagent and runs the next job scheduled.

No action: Closes the dialog after you selectSubmit.

Comments Allows you to add a comment to the job.

Alerts Allows you to send an email, pager, or net-sendoperator alerts when the job completes.

Job Counters Allows you to add Job Counters to the job.

Stop job afterfirst error

When checked, the job will stop if an error isencountered.

4. After filling in the desired parameters, click Submit.

Run ReportsBenchmark Factory Run Reports is a separate executable that opens outside of BenchmarkFactory and provides a comprehensive and detailed collection of database load testing results.With Benchmark Factory you can drill down into a database to view a wide array of informationand statistics that gives you accurate insight into database performance. Run Reports Viewerallows you to access Benchmark Factory load testing results. See "Run Reports Viewer" (page254) for more information.



97






l Transaction Time


l Rows/Second (RPS)

l Total Bytes

l Total Errors

l Total Rows

l Response Time


98




99




100

Advanced Creation Objects (Object Mapping)The Advanced Creation Option allows you to set advanced properties for tablespaces anddatafiles, used for the creation of standard benchmark tables and indexes.

To create advanced objects

1. From the Database Type drop-down, select the desired database type.

2. Select the Tables or Indexes radio button.

3. Select the object(s) you wish to map.


101

4. Click Edit Options. The Mapping Options dialog displays.

5. Select the desired profile from the Load Tablespaces from Profile drop-down.

6. Select the desired options.

7. Click OK.

8. To save the mapping options, click Save. The Save Options File As dialog display.

9. Enter the desired name.

10. Click Save.

Benchmark Objects WizardThe Benchmark Objects Wizard creates or clears the database objects (tables/indexes, etc.)needed to run industry standard benchmarks. The Benchmark Objects Wizard is able to run ifyou have a root node job, load scenario, Create/Delete Benchmark object, or Execute File nodeselected. The Create/Delete Benchmark object node will be added just below the item selected oras a child of the job.

The Benchmark Objects Wizard:

l Selects the benchmark to create objects for

l Determines the scale of a benchmark

l Creates an instruction to create the benchmark objects

To create a Benchmark Object

Note: To create, delete, and modify database tables/indexes you must have the user loginrights required.

1. Select the Job or Load Scenario node.

2. Select Wizards | Benchmark Objects.


102

3. Click Next.

4. From the drop-down, select the desired benchmark.

5. Click Next.

6. Select the desired benchmark scale in Benchmark Scale. When setting up a load test,Benchmark Factory allows you to change the scaling factor of the standard benchmarksprovided. Scales factors increase the size of a database during the testing process allowingrealistic testing. By the scale factors, the number of rows added to the tables increases,allowing for larger userloads to be used, placing a greater stress on the system under test.During load testing, you can run various test with numerous scaling factors to ensure yourdatabase performs at various performance levels.

7. The Advanced Creation Option allows you to set advanced properties for tablespaces anddatafiles, used for the creation of standard benchmark tables and indexes. To add mappingoptions click Advanced Creation Option or proceed to step 18.

8. From the Database Type drop-down, select the desired database type.

9. Select the Tables or Indexes radio button.

10. Select the object(s) you wish to map.

11. Click Edit Options. The Mapping Options dialog displays.

12. Select the desired options.

13. Click OK.

14. To save the mapping options, click Save. The Save Options File As dialog display.

15. Enter the desired name.

16. Click Save.

17. Click OK.

18. Click Next. The Create/Delete benchmark objects dialog displays and provides thefollowing options:

l Create Benchmark Objects: Creates and initializes benchmark objects (tables, indexes,etc.) required to run the standard benchmark transactions.


103

l Always delete objects before create: Deletes existing benchmark objects (tables,indexes, etc.) before creating new ones.

Note: Checking the "Always delete objects before create" check box will delete allbenchmark objects in the selected database and then re-create the objects in thedatabases. When the checkbox is unchecked, Benchmark Factory checks for thebenchmark tables. If they exist, then the creation will be skipped. The scale factor isnot checked.

l Delete Benchmark Objects: Deletes existing benchmark objects (tables, indexes, etc.).

19. Select the desired option.

20. Click Next. The Benchmark Objects Wizard completion dialog displays.

21. Click Finish. The Benchmark Object node creates and displays in the Script View.

Changing Scaling FactorsWhen setting up a load test, Benchmark Factory allows you to change the scaling factor of thestandard benchmarks provided. Scales factors increase the size of a database during the testingprocess allowing realistic testing. By changing the scale factors, the number of rows added to thetables increases, allowing for larger userloads to be used, placing a greater stress on the system-under-test. During load testing, you can run various test with numerous scaling factors to ensureyour database performs at varying user load levels.

Benchmark Factory provides the following scalable benchmarks:

l AS3AP

l Scalable Hardware

l TPC-B

l TPC-C

l TPC-D

l TPC-H

l TPC-E Benchmark (page 284)

The Benchmark properties page allows you to change scale factors properties.

To Change Scale Factors

You can set the load test scale factor when:

l Creating a new script

l Creating/delete a benchmark object, using the Benchmark Object Wizard

l In Benchmark Factory settings

l From the Benchmark Object's property page


104

Setting the Scale Factor in Benchmark Factory Settings

Benchmark Factory settings allows you to set default scaling factors so you don't have to set iteach time you create a new load test.

To set a default scale factor

1. Select Edit | Settings. The settings dialog displays.

2. Click Benchmarks.

3. Double-click the desired benchmark. The Benchmark Properties dialog displays.

4. Set the desired scale in Benchmark Scale.

5. Click OK.

6. Click OK.

Changing the Scale Factory from the Benchmark Object Property Page

After creating the a benchmark object, you can edit that object to change the scale factor.

To edit the scale factor for a Benchmark Object

1. Click on the desired Benchmark Object in the Script view.

2. In the properties view for the Benchmark Object, select Edit. The Benchmark Propertiespage displays.

3. Set the desired scale in Benchmark Scale.

4. Click OK.


105

Execute File WizardDuring the running of a workload, you may want to add an executable file (.bat or .exe) as partof a job. This gives you the flexibility of setting up your database before running a workload.For example, you could run a .bat file to set up a prepare SQL statement before running aworkload.

To execute a file:

1. Select the Job or Load Scenario node.

2. Select Wizards | Execute File.

3. Click Next. The Execute File dialog displays.

4. Enter the path or navigate to the desired file.

5. Click the Enforce Timeout checkbox if desired. Enforce Timeout allows you to enter thedesired number of seconds Benchmark Factory waits for the execution to complete beforecontinuing the job.

6. Enter the desired number of seconds.

7. Click Next. The completion dialog displays.

8. Click Finish. The Execute File is created and displays in the Script View.

BFScript WizardNote: Controls with yellow background can be scripted.

Benchmark Factory scripting all

Benchmark Factory User Guide 6-8-1

Documents

Transcript of Benchmark Factory User Guide 6-8-1