Open Source Test Workshop for CIOs, CTOs, Managers

Open Source Test Workshop

Frank Cohen, +01 (408) 871-0122, [email protected]

June 8, 2011

Surface and Mitigate Performance Bottlenecks

For CIOs, CTOs, Senior Managers

Wednesday, June 8, 2011

Open Source Test Automation

Slides on Twitter‣http://twitter.com/fcohen

‣Please retweet

‣Please follow me

‣#ost

2



About PushToTest‣Our Mission: We Test, For A More Reliable World

‣Make It Easier To Move From Manual To Automated Testing

‣Support Advanced Testing, including Agile, Load, Integration

‣Open Source Testing (OST) For Everyone

3


XML, SOA, and Performance

Web 2.0 Success Driving Business‣Every Organization Wants Modern IT Functions and User Interfaces

‣Rich User Interfaces (RIA, using Ajax, Flash, Flex)

‣SOA Composite Applications

• Faster Time-To-Market

• Increase Revenue

• Cost Savings

‣Service Enabled Applications

• Deliver More Integration Faster

• Always-On

• Less Cost

4



The Reality of Web 2.0 Development‣We Build Applications Differently Now

‣Haze of New Protocols, Software Models, Data Formats

‣Agile Often Means Shorter Build-Test-Deploy ‘Waterfall-style’ Cycles

‣Many Still Trying “Test And Trash” Techniques

‣Uptime Depends on Business Service Management (BSM) Testingto Surface Functional and Performance Issues

5

of the Application Changes

Tests Re-recorded

30%Every 8 weeks

30%Wednesday, June 8, 2011


‣Proprietary Test Tools Offer:

‣QA Labs for 10,000 Users = $35,000 Per Year

‣Does Not Include: Cost of Test Employees, Root Cause Analysis Consultants

‣Does Not Guarantee Success

Testing Costs

6

Functional Test

Functional Test Platform $19,000

Functional Tests $8,000

Protocols: AJAX, SOAP $12,000

Support $5,000

$40,000

Load Test

Load Test Platform $20,000

Simulated User Licenses $120,000

Recoding The Test $35,000

Re-quality The Test $12,000

$187,000



Proprietary Testing Falling Behind‣Adobe Releases Testing APIs for Flex/Flash in 2009

‣JQuery, GWT, YUI, Extjs Each Make 5 Releases in 2010

‣Appcelerator Titanium Makes 3 Releases in 2010

‣Microsoft Draws 8,000 to Mix for StarLight in 2010

‣StarEast, Better Software, PyPy, JavaOne, Devoxx, OSBC, LinuxWorld Feature Keynote Talks On Ajax

7

‣ “HP Refunded Our Payment For The Ajax Protocol Handler”, a PushToTest customer in the automotive industry.



Lots Of Options

Open Source Testing Emerges

8

Selenium soapUI Sahi

HtmlUnit DBUnit Watir

Windmill Hudson TestMaker jMeter

‣Signs of Maturity: Record/Playback, Data Generation/Importer, Iteration Engine, Logger, Recovery Scenario Manager, Object Recognizer

Fitness FIT

FlexMonkeyTG4W



Depend On A Service Partner

9

‣Repurposing: One Test Script For Functional Tests, Load and Performance Tests, and Business Service Monitors

‣Components: Tests Run Even After Application Changes

‣Deployment: Run Tests In Your Lab, Ours, or Both!

‣Services: Add Expertise, Training, Tools To Your Organization

‣Enthusiasm: We participate and contribute to Open Source Test Projects



Manual Testing Process

10

Manual

Testing

Explore

1

2

Test Plan

Create Test Plans

Run The Test Steps

3

4

Results

Recommend Changes



Side Effects

11

‣Exhaustion

‣Fatigue

‣Blurry Vision

‣Night Sweats

‣ Insanity

‣Hair Loss

‣Acne

‣Divorce



OST Architecture

12

Functional

Test

Production

Monitor

Load Test

Continuous Integration

Repository Bug TrackerTest

ManagementMonitoring

Test Authoring

SoapUI (Pro)

Sahi (Pro)

JUnit Class Framework

Eclipse Java IDE

Fitness

Languages: Java, Python

Wiki Collaboration

Application Under TestWeb, RIA, BPM, SOA

PTTMonitorCPU, Net, Memory

Test Injector

BundlesMySQL, HtmlUnit

ScriptRunnersSahi, Selenium, SoapUI, Java

Data Production Librariescsv files, rdbms queries

Test Operation Console

RepositoryMySQL RDBMS

Results AnalysisBusiness Intelligence Tool

DesignerRecord/Playback

Grid and CloudTest Deployment



PTTMonitor

Load Injectors

An OST Cloud

13

Application Under Test

Central location from which a tester, developer, IT manager, CI service runs and monitors tests

TestMaker

Console

Thousands of virtual users perform real-life scenarios on the Application Under Test

TestNode

Stores results of every transaction, step, and monitored resource

Results DB

400+ charts identify the root cause to performance bottlenecks, functional issues

Analysis



PTTMonitor

Load Injectors

An OST Cloud

14


Central location from which a tester, developer, IT manager, CI service runs and monitors tests

TestMaker

Console

Thousands of virtual users perform real-life scenarios on the Application Under Test

TestNode

Stores results of every transaction, step, and monitored resource

Results DB

400+ charts identify the root cause to performance bottlenecks, functional issues

Analysis



Browser Compatibility Tests

15


TestMaker Console

TestNode

Selenese Test

Selenium RC SeleniumHtmlUnit



16

ScriptRunnersScriptRunner Type Name (langtype)

Designer Script IE, Htmlunit designerscript

Selenium HtmlUnit Browser selenium

Selenium RC IE, Firefox, Chrome seleniumrc

Sahi IE, HtmlUnit sahi

HTTP Archive Page Optimization HAR

Selenium RC IE, Firefox, Chrome seleniumrc

SoapUI SOAP and REST soapui

TestGen4Web Web 1.0 Tests testgen4web

Mozmill Mozilla XUL Tests mozmill

Unit Tests jUnit TestCase Java, Jython, JRuby, PHP, Perl, Groovy

DotNet .NET Tests dotnet

Visual Basic VB tests vb

Command Line Desktop and utils commandline



Component Approach

17

Data Production

Log-In

Component

setUp

runTest

tearDown

ApplicationTestScenario

Test Use Case

Product Search

Order Product

Log-Out

RDBMS DPL

CSV DPL

Custom DPL

Load Test

100 Users 500 Users 1000 Users

Log-In

Component

Component

Development

Database

QA Regression

Database

Load Test

Database Confirm Order

Production

Database



You’re Such A Geek!

Lots of Possibilities

18

Seattle, DatacenterTestNode

TestMaker

Dallas, TexasDatacenter

Generating Load

Test Sent



TestMaker Demonstration‣Fun And Easy Test Script Creation

‣Selenium and TestMaker

‣Repurposing A Single Test

‣Functional Test, Load and Performance Test, Production Monitor

‣Results Analysis

‣Chart Engine

19



Root Cause Analysis

20

‣PTTMonitor - Agent Based Monitor Gateway

‣Glassbox - Performance Monitoring for Java Applications

‣dynaTrace - Analysis In Application and DB Tiers



‣Correlate Test Operation To Resource Usage

‣Deploy as Web Application (WAR) and Service

‣Observes CPU, Network, Memory Utilization As Test Runs

‣Gateway to Glassbox, dynaTrace, Windows PerfMon

21

PTTMonitor



Results Analysis Engine

22 And 350 Additional ChartsWednesday, June 8, 2011


‣Correlate Test Operation To Application Tier Issues

‣Memory Leaks

‣Thread Deadlocks

‣Slow DB Connections

‣No Instrumentation of Applications

‣Live Performance Alert View

‣Run In Production

23

Glassbox



Glassbox Troubleshooter

24



‣PurePath from TestMaker Virtual Users To Application & DB Tier

‣Answers: What Are The Transactions and Why Are They Slow?

25

dynaTrace PurePaths

Application Server

dynaTrace monitors

applications for

functional issues

and performance

bottlenecks.

Test RunTime

PTTMonitor

dynaTrace Agent

TestNode

test

TestNode

test

TestNode

test

TestMaker Console

Database dynaTrace Agent

PurePath Token



Pinpoint The Problem‣Down to the Line of Source Code

26



‣Correlate PerfMon Statistics with Test Operation

‣PTTMonitor Gateways Data from PerfMon or MS SQL Server

.NET Monitoring

27

.NET Application Tier

Test RunTime

PTTMonitorTestNode

test

TestNode

test

TestNode

test

TestMaker Console

MS SQL ServerPerfMon

SOAP HTTP REST


Next let’s talk about


PushToTest Options

‣OnDemand

‣QuickStart

‣POC Option

28



Immediate Benefits‣Product Solutions

‣Better, Lower-Cost, More Efficient Test Tool

‣Business Service Monitoring at the Application Tier (SLA Compliance)

‣Methodology Solutions

‣ Improved Test Methodology for my Organization and Team

‣Services

‣Turnkey Load Testing

‣ Integrate TestMaker Into My Build Environment

‣Add A Protocol, Chart, Task to TestMaker To Meet My Needs29



Dual License

‣ Functional Testing with TestMaker Object Designer

‣ Repurposing Selenium, soapUI Sahi and unit tests into data-driven functional tests, load and performance tests, and business service monitors

‣ Load Testing To 50 Virtual Users

‣ Data-driven tests using CSV files.

‣ PTTMonitor Correlates CPU, Network, Memory Utilization As Tests Operate.

‣ 25 Charts and Reports

‣ Community Support and access to all documentation

30

‣ TestMaker Enterprise expands on the success of TestMaker Community

‣ Load Testing to Millions of Virtual Users

‣ Cloud and Grid Testing

‣ Data-driven tests using Data Production Libraries for CSV files, RDBMS, Data Generating Objects

‣ Root cause analysis and mitigation using Glassbox

‣ Mock services for agile service development

‣ Flex AMF Protocol Handler for Flex/Flash testing

‣ 1000+ Charts and Reports

‣ Business service monitors, including email notification

‣ Support, QuickStart, Training, Consulting


http://www.pushtotest.com/products/glassbox

http://www.pushtotest.com/products/glassbox


Pricing‣Annual Subscription Model Based On:

‣Seats for TestMaker Console Installation

‣Seats for SOAP and REST Test Authoring

‣Virtual User Levels for Load Testing

‣Count of Business Service Monitor

‣Technical support: 1 business day or faster response to emails, phone calls, instant messages

31



32

‣As-Needed, On-Demand Basis

‣Test In Your Environment, Or Our Cloud Test Environment, Or Both

‣ Ideal For Consulting Organizations Where Test Needs Are Unknown

‣AMI Distribution

‣Pay Only For What You Use

‣Annual Cap On Total Costs



The QuickStart Option

33

‣Your First Test, Built and Operated by PushToTest Experts

‣ Ideal For Organizations With Little or No Load Testing Experience

‣PushToTest Experts Build Your First Test

‣PushToTest Experts Operate The Test, Provide Root Cause Analysis and Mitigation

‣PushToTest Trains Your Team To Build Their Own Tests

‣$9,750 USD



The POC Option‣PushToTest Experts Help Your TestMaker Evaluation

‣Ensures TestMaker Works In Your Application/System

‣PushToTest Issues Price Proposal

‣Applies Engineers To Build and Operate Your Test

‣PushToTest Trains Your Team To Use TestMaker

‣Success of POC Confirms Intention To Purchase

‣Cost: $034



PushToTest Global Services‣Put Our Experts To Work For Your Projects

‣Methodology, Solutions, Support, Enthusiasm

‣Offices: USA (Texas and California,) Bangalore, London

‣The Near Shore Option

‣150 Test Experts

‣Costa Rica: CST, English, Inexpensive, On Site in 3-5 Hours

‣http://www.pushtotest.com/services

35

“PushToTest Uniquely Built Load Tests For Our Customers, Including Pepsi, And Worked With Our

Engineers To Surface and Solve Performance Bottlenecks.” Alan Rodriguez, Director of Testing, Tribal DDB Dallas



Slides on Twitter‣http://twitter.com/fcohen

‣Please retweet

‣Please follow me

‣#ost

36



Where To Go From Here‣Please Complete The Evaluation Form

‣And We Will Send You The Slides

‣Watch A Screencast and Tutorial

‣http://www.pushtotest.com/screencastcentral

‣Attend An Open Source Test Workshop

‣http://workshop.pushtotest.com

‣Ask For A Proposal on Licenses, Training, Support, Consulting

‣Call +01 408 871 0122, [email protected]

37


Open Source Test Workshop for CIOs, CTOs, Managers

Business

Transcript of Open Source Test Workshop for CIOs, CTOs, Managers