Open Source Test Workshop for QA Testers, Developers, IT Managers

Open Source Test Workshop

Frank Cohen, +01 (408) 871-0122, [email protected]

June 9, 2011

Surface and Mitigate Performance Bottlenecks

For QA Testers, Developers, Operations

Thursday, June 9, 2011

Open Source Test Automation

Slides on Twitter‣http://twitter.com/fcohen

‣Please retweet

‣Please follow me

‣#ost

2



About PushToTest

3



About PushToTest‣Our Mission: We Test, For A More Reliable World

3




‣Make It Easier To Move From Manual To Automated Testing

3





‣Support Advanced Testing, including Agile, Load, Integration

3





‣Support Advanced Testing, including Agile, Load, Integration

‣Open Source Testing (OST) For Everyone

3



Manual Testing Process

4

Manual

Testing




4

Manual

Testing

Explore

1




4

Manual

Testing

Explore

1

2

Test Plan

Create Test Plans

Run The Test Steps

3

4

Results

Recommend Changes



Side Effects

5



Side Effects

5

‣Exhaustion

‣Fatigue

‣Blurry Vision

‣Night Sweats

‣ Insanity

‣Hair Loss

‣Acne

‣Divorce



Proprietary Testing Falling Behind‣Adobe Releases Testing APIs for Flex/Flash in 2009

‣JQuery, GWT, YUI, Extjs Each Make 5 Releases in 2010

‣Appcelerator Titanium Makes 3 Releases in 2010

‣Microsoft Draws 8,000 to Mix for StarLight in 2010

‣StarEast, Better Software, PyPy, JavaOne, Devoxx, OSBC, LinuxWorld Feature Keynote Talks On Ajax

6

‣ “HP Refunded Our Payment For The Ajax Protocol Handler”, a PushToTest customer in the automotive industry.



Lots Of Options

Open Source Testing Emerges

7



Lots Of Options


7

Selenium soapUI Sahi

HtmlUnit DBUnit Watir

Windmill Hudson TestMaker jMeter

Fitness FIT

FlexMonkeyTG4W



Lots Of Options


7

Selenium soapUI Sahi

HtmlUnit DBUnit Watir

Windmill Hudson TestMaker jMeter

‣Signs of Maturity: Record/Playback, Data Generation/Importer, Iteration Engine, Logger, Recovery Scenario Manager, Object Recognizer

Fitness FIT

FlexMonkeyTG4W



OST Architecture

8

Application Under TestWeb, RIA, BPM, SOA



OST Architecture

8

Functional

Test

Production

Monitor

Load Test

Continuous Integration

Repository Bug TrackerTest

ManagementMonitoring

Test Authoring

SoapUI (Pro)

Sahi (Pro)

JUnit Class Framework

Eclipse Java IDE

Fitness

Languages: Java, Python

Wiki Collaboration




OST Architecture

8

Functional

Test

Production

Monitor

Load Test




Test Authoring

SoapUI (Pro)

Sahi (Pro)


Eclipse Java IDE

Fitness


Wiki Collaboration


Test Operation Console

RepositoryMySQL RDBMS

Results AnalysisBusiness Intelligence Tool

DesignerRecord/Playback

Grid and CloudTest Deployment



OST Architecture

8

Functional

Test

Production

Monitor

Load Test




Test Authoring

SoapUI (Pro)

Sahi (Pro)


Eclipse Java IDE

Fitness


Wiki Collaboration


PTTMonitorCPU, Net, Memory

Test Injector

BundlesMySQL, HtmlUnit

ScriptRunnersSahi, Selenium, SoapUI, Java

Data Production Librariescsv files, rdbms queries

Test Operation Console







TestMaker Architecture

9





TestMaker Architecture

9



TestMaker Console





TestNode

BundlesMySQL, HtmlUnit

ScriptRunnersSahi, Selenium, SoapUI, Java

Data Production Librariescsv files, rdbms queries



10

Test Script

1 Load Home Page

2

3

Enter Value

Validate Results

Selenium: Search Component



Selenium Script RunnerData Driven, Step Time,

HTMLUnit

10

Test Script

1 Load Home Page

2

3

Enter Value

Validate Results




Selenium Script RunnerData Driven, Step Time,

HTMLUnit

public class CheckEmail { public void setUp() {} ... public void runTest { selenium.setBaseUrl( "http://myurl" ); selenium.open("/login.srf”); selenium.type("search", "ctest"); selenium.click("//Item[6]"); selenium.assertTextExists("mytext"); }}

TestMaker Transformer: Java Unit Test

10

Test Script

1 Load Home Page

2

3

Enter Value

Validate Results




Browser Compatibility Tests

11

Application Under Test

TestMaker Console

TestNode

Selenese Test

Selenium RC SeleniumHtmlUnit



12

ScriptRunnersScriptRunner Type Name (langtype)

Designer Script IE, Htmlunit designerscript

Selenium HtmlUnit Browser selenium

Selenium RC IE, Firefox, Chrome seleniumrc

Sahi IE, HtmlUnit sahi

HTTP Archive Page Optimization HAR

Selenium RC IE, Firefox, Chrome seleniumrc

SoapUI SOAP and REST soapui

TestGen4Web Web 1.0 Tests testgen4web

Mozmill Mozilla XUL Tests mozmill

Unit Tests jUnit TestCase Java, Jython, JRuby, PHP, Perl, Groovy

DotNet .NET Tests dotnet

Visual Basic VB tests vb

Command Line Desktop and utils commandline



Your cloud or mine? Or both!

13

SOAP

TestNode

test

PTTMonitor

TestNode

test

PTTMonitor

Application or Service

SOAP SOAP SOAP

HTTP REST AMF AJAX

TestMaker Console

TestNode

test

PTTMonitor

TestNode

test

PTTMonitor

Test RunTime



Lots of Possibilities

14

Seattle, DatacenterTestNode

TestMaker

Dallas, TexasDatacenter

Generating Load

Test Sent



You’re Such A Geek!

Lots of Possibilities

14

Seattle, DatacenterTestNode

TestMaker

Dallas, TexasDatacenter

Generating Load

Test Sent



TestMaker Demonstration‣Fun And Easy Test Script Creation

‣Selenium and TestMaker

‣Repurposing A Single Test

‣Functional Test, Load and Performance Test, Production Monitor

‣Results Analysis

‣Chart Engine

15



Root Cause Analysis

16

‣PTTMonitor - Agent Based Monitor Gateway

‣Glassbox - Performance Monitoring for Java Applications

‣dynaTrace - Analysis In Application and DB Tiers



‣Correlate Test Operation To Resource Usage

‣Deploy as Web Application (WAR) and Service

‣Observes CPU, Network, Memory Utilization As Test Runs

‣Gateway to Glassbox, dynaTrace, Windows PerfMon

17

PTTMonitor



Results Analysis Engine

18 And 350 Additional ChartsThursday, June 9, 2011


‣Correlate Test Operation To Application Tier Issues

‣Memory Leaks

‣Thread Deadlocks

‣Slow DB Connections

‣No Instrumentation of Applications

‣Live Performance Alert View

‣Run In Production

19

Glassbox



Glassbox Troubleshooter

20



‣PurePath from TestMaker Virtual Users To Application & DB Tier

‣Answers: What Are The Transactions and Why Are They Slow?

21

dynaTrace PurePaths

Application Server

dynaTrace monitors

applications for

functional issues

and performance

bottlenecks.

Test RunTime

PTTMonitor

dynaTrace Agent

TestNode

test

TestNode

test

TestNode

test

TestMaker Console

Database dynaTrace Agent

PurePath Token



Pinpoint The Problem‣Down to the Line of Source Code

22



‣Correlate PerfMon Statistics with Test Operation

‣PTTMonitor Gateways Data from PerfMon or MS SQL Server

.NET Monitoring

23

.NET Application Tier

Test RunTime

PTTMonitorTestNode

test

TestNode

test

TestNode

test

TestMaker Console

MS SQL ServerPerfMon

SOAP HTTP REST



Dual License

‣ Functional Testing with TestMaker Object Designer

‣ Repurposing Selenium, soapUI Sahi and unit tests into data-driven functional tests, load and performance tests, and business service monitors

‣ Load Testing To 50 Virtual Users

‣ Data-driven tests using CSV files.

‣ PTTMonitor Correlates CPU, Network, Memory Utilization As Tests Operate.

‣ 25 Charts and Reports

‣ Community Support and access to all documentation

24

‣ TestMaker Enterprise expands on the success of TestMaker Community

‣ Load Testing to Millions of Virtual Users

‣ Cloud and Grid Testing

‣ Data-driven tests using Data Production Libraries for CSV files, RDBMS, Data Generating Objects

‣ Root cause analysis and mitigation using Glassbox

‣ Mock services for agile service development

‣ Flex AMF Protocol Handler for Flex/Flash testing

‣ 1000+ Charts and Reports

‣ Business service monitors, including email notification

‣ Support, QuickStart, Training, Consulting


http://www.pushtotest.com/products/glassbox

http://www.pushtotest.com/products/glassbox


Where To Go From Here‣Download TestMaker and Read The Tutorials

‣Please Complete The Evaluation Form

‣And We Will Send You The Slides

‣Watch the Screencasts

‣http://www.pushtotest.com/screencastcentral

‣Ask For A Proposal on Licenses, Training, Support, Consulting

‣Call +01 408 871 0122, [email protected]

25


Open Source Test Workshop for QA Testers, Developers, IT Managers

Technology

Transcript of Open Source Test Workshop for QA Testers, Developers, IT Managers