stp-2008-03

7/27/2019 stp-2008-03

1/40

A Publication

Manage Performance ByTesting Early and Often

Buildings Not in the Cards?Minimize Risk When Buying

Motivate a Team WithSome Spade Work

The FoundationOf Good Testing

BESTPRACTICES:

Change

Management

VOLUME 5 ISSUE 3 MARCH 2008 $8.95 www.stpmag.com
http://www.bzmedia.com/http://www.bzmedia.com/

7/27/2019 stp-2008-03

2/40

April 15-17, 2008San Mateo Marriott

San Mateo, CA

ABZMedia Eve

nt

SPRING
http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/

7/27/2019 stp-2008-03

3/40

Platinum Sponsors Gold Sponsors Silver Sponsor

www.stpcon.co

Register ByMarch 28 to GetThe Early-Bird RateSAVE OVER $200!

break your old testing habits

Learn the Latest Tips and TechniquesTry Out the Newest TechnologyAll at STPCon!

Great, informativeconference for soft-ware testers, leads

and managers alike.Useful tutorials andtechnical classes ofwide varietyA must-

attend for all seriousQA/SQE professionals!Alan Abar

Software QualityEngineering Manager,Covad Communications

SUPERB SPEAKERS

Michael Bolton, Jeff Feldstein,

Michael Hackett, Jeff Johnson, Bj Rollison,

Rob Sabourin, Mary Sweeney, Robert Walsh

AND DOZENS MORE!

TERRIFIC TOPICS

Improving Web Application Performance

Optimizing the Software Quality Process

Developing Quality Metrics

Testing SOA Applications

Charting Performance Results

Managing Test Teams

AND OVER 70 MORE TO

CHOOSE FROM!

Youll find information outside of your dailyactivities, and options/alternatives to think aboutnew approaches to testing.Alex Kang

Staff Engineer, Tellabs

It solidifies the total testing experience and opensyour eyes to alternative approaches and methodsthat you simply cannot get from books.John Croft

QA Manager, I4Commerce
http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/http://www.stpcon.com/

7/27/2019 stp-2008-03

4/40
http://www.seapine%2Ccom/qualityready2

7/27/2019 stp-2008-03

5/40

ContentsA Publication

VOLUME 5 ISSUE 3 MARCH 2008

14COVER STORYKeep Your Web App FromFalling Like a House of Cards

Dont wait until the end stages to discover that your applications archi-tecture doesnt scale well. Test early and often to keep your Web apps fromfalling apart. By Ernst Ambichl

Departments7 EditorialSelling software vulnerabilities to the

highest bidderfree market or shakedown?

8 ContributorsGet to know this months experts and the

best practices they preach.

9 FeedbackIts your chance to tell us where to go.

11 Out of the BoxNew products for testers.

36 Best PracticesFrom tulips to leaky levees: a comparative

study in change management.By Geoff Koch

38 Future TestUsage metering and software security join

forces for IP protection. By Kevin Morgan

29 Motivate YourTeam With AFew Simple Tricks

33 Testing FromThe Ground Up

24 Buy vs. Build:Minimize Risk

To find the best practices that chargeup your team, look for its motivators

and demotivators.

Shrink manage-ment, dumpgroupthink,e n c o u r a g e

c irculat ion ,and watch your

team get going.By Alan Berg

MARCH 2008 www.stpmag.com 5

Custom-developed and COTS software

bring a slippery slope of opportunity

and riskto system quality. Learn the

dangers of buying vs. building, andstrategies that can transform risk into

profit. By Rex Black

In software testing, as in construction,

a solid foundation is crucial. Ground

your project with comprehensible

requirements, a well-prepared test

strategy and continuous enhancement

of the test suite. By Kiran Vankatesh

7/27/2019 stp-2008-03

6/40

Empirix gives you the freedom to test yourway.

Tired of being held captive by proprietary scripting? Empirix offers a suite of

testing solutions that allow you to take your QA initiatives wherever you like.

Download our white paper,Lowering Switching Costs for Load Testing

Software, and let Empirix set you free.

HANDCUFFS OFF

4AKE THE

QUALITY ASSURANCE

www.empirix.com/freedom
http://www.empirix.com/freedomhttp://www.empirix.com/freedomhttp://www.empirix.com/freedomhttp://www.empirix.com/freedomhttp://www.empirix.com/freedomhttp://www.empirix.com/freedomhttp://www.empirix.com/freedomhttp://www.empirix.com/freedomhttp://www.empirix.com/freedomhttp://www.empirix.com/freedomhttp://www.empirix.com/freedom

7/27/2019 stp-2008-03

7/40

In a newsletter last Octo-

ber, I wrote about a Swiss

company with a name as

unusual as its mission.

WabiSabiLabi is one of a

growing number of compa-

nies to begin selling software

security vulnerabilities to

the highest bidder. As I re-ported at the time, the

model encourages security

companies, researchers and

others to capitalize their findings in an

open marketplace.

The idea was that buyers and sell-

ers would be vetted by the company,

and transactions would be limited to

legitimate organizations. After only

two months in business, the company

had logged 160,000 unique visitors,

1,000 registered sellers and 150 vul-

nerabilities.WSLabi attributed the quick success

to a security community anxious for an

opportunity to spread their experience

and research to an eager and ready audi-

ence of vetted buyers prepared to pay

for the latest information. Patrons of the

site (wslabi.com) include enterprises,

government agencies and major software

vendors in the IT security sector keen on

learning about the vulnerabilities as they

enter the world.

All that may sound good on paper, buttheres a dark side.

A Russian security research firm

called GLEG Ltd. is one of a number of

companies that analyze software for

security defects and offer the informa-

tion for sale to the softwares developer.

The company on January 1 announced

that it had identified a zero-day vulnera-

bility in RealNetworks RealPlayer 11

(build 6.0.14.74) that reportedly allows

for code execution when RealPlayer

opens a malicious song file. GLEG gives

this information to its customers andwants to be paid by RealNetworks before

revealing the exploit.

Security ShakedownAlthough this is perfectly

legal, it might seem more

like legalized extortion.

Somewhat akin to the local

locksmith, after youve pur-

chased a new lockset for

your home, shaking you

down so he wont sell copiesof your house keys.

A protocol that better

protects the security of our

software ecosystem would be for vul-

nerability finders to contract directly

with the vendor to find vulnerabilities,

says Chris Wysopal, CTO and co-

founder of Veracode, of the incident

on his blog. Veracode too offers securi-

ty testing solutions and services, but

operates a bit differently.

If a company is concerned about the

security of software its about to buy, it canhire Veracode to conduct an assessment.

We will contact the vendor and have

them upload their software binary exe-

cutable to our portal, Wysopal explains.

We analyze the software and deliver a

detailed report of the security issues we

find in the code. We also generate a sum-

mary report for the customer to under-

stand the security risks of the software.

This seems a more reasonable

approach; Veracode customers know

about the vulnerability and can weigh therisks of using the product, while the

applications developer gets what it needs

to fix the flaw.

Theres just one problem: The solu-

tion completely overlooks vulnerabili-

ties of the type found by GLEGin soft-

ware that is free. And for software thats

not free, Veracode serves only people

who ask for their services, leaving a lot

of software unchecked.

Im a firm believer in the free market,

as long as its solutions are fair to all sides.

I suppose that the simplest answer inRealNetworks case would be to become a

customer of GLEG.

Hackers and The

Free Market


VOLUME 5 ISSUE 3 MARCH 2008

Ed Notes

PresidentTed Bahr

Executive Vice PresidentAlan Zeichick

Software Test & Performance (ISSN- #1548-3460) ispublished monthly by BZ Media LLC, 7 High Street,Suite 407, Huntington, NY, 11743. Periodicals postagepaid at Huntington, NY and additional offices.

Software Test & Performance is a registered trade-mark of BZ Media LLC. All contents copyrighted2008 BZ Media LLC. All rights reserved. The priceof a one year subscription is US $49.95, $69.95 inCanada, $99.95 elsewhere.

POSTMASTER: Send changes of address to Software

Test & Performance, PO Box 2169, Skokie, IL 60076.Software Test & Performance Subscribers Servicesmay be reached at [email protected] or bycalling 1-847-763-9692.

Cover Photograph by Alexey Kashin

Director of Circulation

Agnes Vanek

+1-631-443-4158

[email protected]

EDITORIAL

SALES & MARKETING

READER SERVICE

Art Director

LuAnn T. Palazzo

[email protected]

Art /Production Assistant

Erin Broadhurst

[email protected]

ART & PRODUCTION

BZ Media LLC7 High Street, Suite 407Huntington, NY 11743+1-631-421-4158fax [email protected]

Editor

Edward J. Correia

+1-631-421-4158 x100

[email protected]

Copy Editor

Laurie OConnell

[email protected]

Editorial Director

Alan Zeichick

+1-650-359-4763

[email protected]

Contributing Editor

Geoff Koch

[email protected]

Publisher

Ted Bahr

+1-631-421-4158 x101

[email protected]

Associate Publisher

David Karp

+1-631-421-4158 x102

[email protected]

Advertising Traffic

Phyllis Oakes

+1-631-421-4158 x115

[email protected]

Director of Marketing

Marilyn Daly

+1-631-421-4158 x118

[email protected]

List Services

Lisa Fiske

+1-631-479-2977

[email protected]

Reprints

Lisa Abelson

+1-516-379-7097

[email protected]

Accounting

Viena Ludewig

+1-631-421-4158 x110

[email protected]

Customer Service/Subscriptions

+1-847-763-9692

[email protected]

Edward J. Correia
http://wslabi.com/mailto:[email protected]:[email protected]://www.bzmedia.com/http://www.americanbusinessmarketing.com/http://www.bpaww.com/mailto:[email protected]://wslabi.com/

7/27/2019 stp-2008-03

8/40

Were pleased to welcome ERNST AMBICHL, Borlands

chief scientist, to our pages. Ernst served as chief tech-

nology officer at Segue Software until 2006, when the

maker of SilkTest and other QA tools was acquired by

Borland. He joined Segue in 1998 and helped buildit into a leader in its field.

At Borland, Ernst is responsible for the architec-

ture of Borlands Lifecycle Quality Management prod-

ucts. In our lead feature, which begins on page 14,

Ernst will school you on methods of load testing ear-

ly in the development cycleeven when parts of an

application arent yet completedwith an eye toward

preventing downstream performance issues.

REX BLACKhas a quarter-century of software and sys-

tems engineering experience, and is president of

RBCS, a software, hardware and systems testing con-

sultancy.

In this issue, Rex lends his considerable expertise

to the practice of minimizing the risks of testing and

integrating outsourced application components.

Beginning on page 24, Rex mixes practical wisdom

with real-world experience from working with cor-

porations in dozens of countries to bring you an analy-

sis of the risk factors of integration, how to select a

component vendor and how to test its products and

processes.

We once again bring you the enjoyable style and wit

ofALAN BERG, the author of numerous articles andpapers on software development and testing. This

time, he draws from his experience on numerous

teams to enlighten us on motivating a development

team, beginning on page 29. And yes, bribery is one

of several techniques he espouses.

Alan is the lead developer of Central Computer

Services at the University of Amsterdam, a post he

has held for more than seven years. He holds a bach-

elors degree, two masters degrees and a teaching

certification.

KIRAN VANKATESHis test lead of the Testing Practiceat MindTree Consulting, an IT services and consult-

ing company with offices in the U.S., Europe and Asia-

Pacific. Beginning on page 33, Kiran offers a tutorial

covering the basics of good testing practice.

Kiran has been a software tester for four years, and

has a strong conceptual background in financial, health-

care and asset management systems. He is proficient in

functional testing, verification and general software test-

ing, and also has worked on real-time transactional appli-

cations. Kiran works in MindTrees Bangalore office and

holds a Software Test Engineer certificate from the

International Software Testing Qualifications Board.

Contributors

TO CONTACT AN AUTHOR, please send e-mail to [email protected].

8 Software Test & Performance MARCH 2008
http://www.itko.com/lisa

7/27/2019 stp-2008-03

9/40

The following letters refer to Edward J. Correias

editorial Defect Tracker for Politicians (Software

Test & Performance magazine, Feb. 2008; reti-

tled Track Politicians Like Bugs in the Feb. 5,

2008, edition of Test & QA Report newsletter;

seehttp://stpmag.com /retrieve/stp-0802.htm).

FROM FANTASTICJust received and read todays Test & QA

Report. I just wanted to say that was fan-

tastic.

Jo Compton

Los Angeles, CA

TO REFRESHINGA note to let you know how absolutely

refreshing your Ed Notes column was in

the February 2008 issue of Software Test &

Performance. Frankly, I did not even oncehave to mumble under my breath and

grind my teeth as I have been told that I

do when reading some of the liberal, pro-

gressive propaganda that always seems to

work its wayI am sure by no accident

into just about every issue of eWeek. Bravo!Michael Hyman

San Diego, CA

TO IGNORANT POOR TASTEI found this article to be in very poor taste.

First of all, it is probably a very bad idea

to inflict your political views on the read-ership of your publication. Second, many

of the statements you made were based on

faulty logic or ignorance of the facts, or

were just plain simplistic and/or not reflec-

tive of very high intelligence. You may want

to consider avoiding this kind of content

in the future.Steve Munger

Portland, OR

SPRINGTIME MEANS SUNSHINE,BASEBALL AND STPCON

Here they come again. No, not Derek Jeter and Barry Bonds. Im referring to Michael

Bolton, Hans Buwalda, Mary Sweeneyand Rob Sabourin, who also delivers the keynote

on testing in Scrum. These are just a few of the instructors youve told us are

your favorites, so weve brought them back to the Software Test & Performance

Conference in San Mateo, along with a few new faces too.

The San Mateo Marriott is where well break out of the box; the performancebox, that is. Youve told us you

wanted more performance class-

esand weve delivered. This

years conference will be loaded

(so to speak) with nearly a dozen

classes designed specifically to help

you find ways of improving the

performance of your applications.

Weve also brought Karen

Johnson to town, and shell offer

a two-part class on charting and

presenting performance results

using graphical analysisand proven storytelling

techniques.

If you were with us last

year, you might remem-

ber the Hands-On Testing

Showcase, a successful

event we introduced in

San Mateo and expand-

ed last fall in Boston. Well,

HOTS is back and will be

better than ever, with mul-

tiple vendors inviting youto test their latest prod-

ucts while enjoying copi-

ous quantities of fabulous

food and bottomless bins

of potent potables.

Well also be intro-

ducing Lightning Talks to

STPCon, where conference-goers can hear as many as 10 speakers in a single hour

give short, targeted lectures on the essence of a subject relevant to your job. Speakers

might test-drive a new topic, promote one of their classes or new pet project, or just

provoke thought among the audience with a brilliant concept.

So here it is, your ticket to advancing your testing skills, expanding your contact

base and broadening your mindall at the Software Test & Performance Conference.I hope to see you there, April 15-17, at the San Mateo Marriott. Edward J. Correia


FEEDBACK: Letters should include the writers

name, city, state and e-mail address . Send

your thoughts to [email protected] become the property of BZ Media and

may be edited for space and style.

STPCon in San Mateo this April will feature a demo hall

thats bigger than ever before and stocked to the rafters with the

newest products for software testers, and knowledgeable com-

pany reps to explain how to put them to use.

Feedback

SPRING
http://stpmag.com/retrieve/stp-0802.htmhttp://stpmag.com/retrieve/stp-0802.htm

7/27/2019 stp-2008-03

10/40
http://www.testcomplete.com/stp

7/27/2019 stp-2008-03

11/40

SOAPscope Server 6.1, the latest version

of Mindreefs SOA and Web services test-ing platform, now includes three desktop

modules aimed specifically at testers and

developers. The company also increased

support in the platform for OASIS WS-

Security specifications.

Among the new trio is SOAPscope

Architect 6.1, which the company describes

as a design-time governance and SOA

quality and testing platform for authoring

policy rules, design-time support, proto-

typing, change-time and runtime support.

The tool incorporates industry standardsand specifications for SOA applications and

enables design teams to build compliant

components in combination with their own

customized best practices.

Also new is SOAPscope Tester 6.1, which

brings load testing and test automation

to the SOA quality platform, and helps to

QA engineers, testers and consultants

identify quality problems and potential

performance bottlenecks early in the life

cycle.

SOAPscope Developer 6.1 integrates

tools for problem diagnosis and resolu-tion, unit testing and supporting service

customers. The tool allows teams to cre-

ate, test, deliver and support Web services

and SOA components, and automates

XML-oriented tasks. The three new mod-

ules are included with SOAPscope Server

6.1, a server-based solution intended for

use and collaboration by all members of

the SOA and Web services team, including

analysts and managers.

An Oasis of InteroperabilityAs OASIS and other specifications advance,

it becomes ever more important for com-

panies to remain compliant so their appli-cations continue to interoperate with those

of other organizations. According to the

company, all version 6.1 Mindreef prod-

ucts can be used to test Web services that

use WS-Security. They do this by invoking

and resending protected SOAP messages,

running scenario tests using the specified

X.509 Token Profile, signing and encrypt-

ing. Testers can use SOAPscope tools to

create working security profiles for differ-

ent WS-Security configurations and switch

between them for testing.Frank Grossman, president and CTO

of Mindreef, said, Project teams have been

lacking the ability to quickly and easilycheck for adherence to standards as serv-

ices are being created, tested and imple-

mented. The expanded line was designed

with this problem in mind, he added.

SOAPscope Server 6.1 introduces the

concept of the service space, a container

that allows teams to organize, collaborate

and share assets with other project teams

members, and run tests based on prede-

fined profiles, the company said in a state-

ment announcing the new products.

SOAPscope Server 6.1 is available now;pricing is based on project scope.

SOAPscope Trio Spots a Test-Team Oasis

Out of the Box

Break-out apps in SOAPscope Server 6.1 target application designers, testers and developers.


Aternity, which makes user experience

management tools, in late January began

shipping the Frontline Performance

Intelligence Platform, which it claims can

pre-emptively detect software problems,

monitor application usage and usability,

analyze end-user productivity, correlatebusiness performance and help with

capacity planning. Licensing starts at

US$75,000.

At the heart of the system is a series of

Microsoft Certified Agents, which gather

data about end-user activities and trans-

actions, and report back to an aggrega-

tion service. According to company

claims, the agents consume a maximumCPU utilization of 3 percent, and 0.1 per-

cent on average. Other services handle

data analysis and management.

By transforming every desktop into a

self-monitoring platform that is end-user-

experience aware, were enabling these

enterprises to harness the frontline intel-

ligence they need to make effective busi-

ness decisions that will drive increased

productivity, performance and usability,

said Aternity president and CEO Trevor

Matz in a statement introducing the prod-uct at the DEMO 08 Conference in Palm

Desert, Calif.

From Here to Aternity

7/27/2019 stp-2008-03

12/40

Talend updated its flagship Open Studio

data integration solution in February,

adding more than 30 new components,

connectivity to more databases and sup-

port for event triggering based on real-

time data conditions. The tool also now

can execute groovy scripts, dynamical-

ly load and execute Java classes, and gen-

erate graphs compatible with the

Portable Network Graphics (PNG) loss-

less compression specification.According to the company, the lat-

est version, Open Studio 2.3, now fully

supports the WSDL specifications,

enabling Talends data integration

processes to become data services com-

ponents of an SOA.

The company also claims perform-

ance gains of as much as 600 percent

over the previous versions, and major

enhancements to debugging and trace

modes for viewing data as it flows

through processes. These enhancements

add expand/collapse, pause/resumeand step-by-step viewing modes to the

viewing capabilities.

Connectivity in Open Studio 2.3 now

includes JasperSoft iReports, Microsoft

Dynamics and SQL Server 2008,

Mondrian, Palo, Sage CRM and Vertica,

all of which can be used for integration

as data targets or sources, the company

said. The release also expands support

for the data warehousing phenomenon

of Slowly Changing Dimensions (types

1, 2 and 3), adding IBM DB2, Ingres,

MySQL, SQL Server, Oracle,PostgreSQL and Sybase ASE to its sup-

ported list. Talend Open Studio 2.3 is

available now; pricing was not disclosed.

In related news, Talend in late

January struck a dea l under which

Microsoft will dedicate resources to help

the company optimize performance and

integration of Talends software prod-

ucts with Windows.

In a statement, Microsoft director of

platform technology strategy Sam Ramji

said that the companys motivation for

the move was expanding our cus-tomers options for data integration and

extending both Windows and SQL

Server.

Out of the Box


Open Studio

Goes Live

Open Studio 2.3 now supports event triggering based on real-time data conditions.

By now it would be a stretchto claim that software as a serv-

ice is a new thing, particular-

ly when companies are report-

ing half-billion-dollar fiscal

years, as Salesforce.com did in

2007.

But a scant few have

offered SaaS solutions for

testers, and none as complete

as promised by the forthcom-

ing Zephyr from D Software.

Zephyr consists of a series

of modern-looking, dynamicWeb pages centering around the con-

cept of desktops and dashboards.

Executives, managers and test team

members access the system through

desktops customized for theirspecific roles on the team.

All relevant applications are

contained in the desktop and

can open in multiple windows.

Managers might see project

and resource management

apps while testers see test case

creation and execution pro-

grams.

Changes to any data shared

among multiple team mem-

bers are updated on all screens

instantly, according to infor-mation on the companys Web site

(www.getzephyr.com).

Zephyrs sleek, dynamic interfaces take on the look of a high-end hi-fi

systemand present real-time data on project status.

The Credo of Zephyr QA Test Management:Of Desktops and Dashboards
http://salesforce.com/http://salesforce.com/

7/27/2019 stp-2008-03

13/40

VDI Spreads TheVirtual LoveVMware claims to have simplified the way

administrators using its tools can connectto and manage the virtual desktops under

their control. Virtual Desktop Manager

2 is an enhancement to VMware Virtual

Desktop Infrastructure (VDI) that the

company claims streamlines secure con-

nections to the data center and provides

continuity services that were previously

offered only for mission-critical applica-

tions.

VDI is available now starting at US$150

per concurrent user. Virtual Desktop

Manager 2 can connect from a PC or thin

client, can manage thousands of desktops

at once and reduces the time it takes to

provision a new desktop from hours to

minutes, according to a company news

release. The tool also is available in vari-

ous bundles.

Insight on Byte CodeAnalysisSource code analysis tool maker Klocwork

on Feb. 12 began shipping a new version

of Insight for Java, its automated analysis

tool that it claims now delivers accurate

bug and security vulnerability results from

byte code scans, regardless of the com-

piler and framework used to build it.

Insight for Java supports all versions

of Java up to and including 1.6, Java EE

and ME. It also works with AWT, GWT,

Hibernate and JavaMail, and integrates

with Eclipse, IBM Rational Application

Developer, IntelliJ IDEA and JBuilder

2007 IDEs, as well as ANT and Maven

build tools.

Springing Into .NETDevelopmentSpringSource has released Spring.NET

1.1, extending the Spring open source

framework for Java to the .NET environ-

ment. The tool is available now for free

download at www.springframework.net

/download.html.

According to a company news release,

features implemented or improved in ver-sion 1.1 include an inversion of control

container for configuring application

classes using dependency injection; an

ASP.NET framework for Web develop-

ment with bi-directional data binding and

improved localization support, data mod-

el and process management; externalizednavigation through result mapping; and

a UI-agnostic data validation framework.

We believe Sprint.NET will prove ben-

eficial to both the .NET developer com-

munity as well as the growing number

of developers who work on both [Java and

.NET] platforms, said Rob Johnson, CEO

and founder of SpringSource, which pri-

or to November was known as Inter-

face21. Johnson also founded the Spring

Framework for Java.

Also implemented are an aspect-ori-

ented programming framework, portableservice transactions, an aspect library, an

ADO.NET data access framework and

declarative transaction management via

XML configuration and attributes. It

reportedly integrates with ASP.NET AJAX,

NUnit and NHibernate 1.0 and 1.2, and

can mix ADO.NET and NHibernate oper-

ations in a single transaction.

Linux App? Now You

Can GuardITArxan Technologies has released a Linuxversion of GuardIT, giving Linux devel-

opers a solution for protecting their

applications from tampering.

According to Arxan, its solutions are

deployed using a binary solution that isnt

intrusive to application performance.

Through an interconnected mesh of

small security units called Guards scat-

tered across a compiled binary and then

dissolved into the application, Arxans

GuardIT fortifies the overall software

product against piracy, reverse engineer-ing, insertion of malware and other forms

of attack.

With the release in late January,

GuardIT now works with Linux desktop,

server and embedded platforms on x86

and PowerPC systems as well as on

Windows and .NET. GuardIT for Linux

offers feature parity with the Windows

version on both 32- and 64-bit architec-

tures. The new version also introduces

anti-tamper, anti-debug, obfuscation and

encryption technologies, the companysaid, as well as the ability to selectively

analyze and aim at specific portions of

the binary for targeted code protection.

GuardIT is available now; pricing was

not disclosed.

GlobalLogic: HeresVersion 1.0 Version 1.0No, its not a misprint. GlobalLogic last

month unveiled Version 1.0, a concep-

tualization and software development

service that it says is designed to help

startups and small shops get new soft-

ware applications or ventures off the

ground quickly and with relatively low

financial outlay.

With Version 1.0, GlobalLogic will

provide everything entrepreneurs need

to rapidly and qualitatively take an ideascribbled on a napkin to a product of

service in the market, said GlobalLogic

CEO Peter Harrison in a statement

announcing the new service. By pro-

viding early innovators with end-to-end

product engineering services, we let

them focus on strategy, marketing, cus-

tomer acquisition and go-to-market chal-

lenges.

Harrison compared the idea to what

has been common practice in the semi-

conductor industry for decades. We areseeing the emergence of a new breed of

fabless software company, and we are

excited to be an enabler of this new

trend.

For its part, GlobalLogic offers to pro-

duce early applications prototypes to

help companies attract customers and

investor feedback, and even fill in as

head of engineering or CTO when nec-

essary. The service also is offered to estab-

lished companies looking to overcome

the roadblocks they typically face when

launching an entirely new product, suchas slow internal procedures, lack of

domain experience and scarce software

engineering talent, according to a doc-

ument announcing the release. Though

pricing wasnt disclosed, the company

claims it can cut timelines and operat-

ing costs by as much 60 percent com-

pared with in-house development.

GlobalLogic employs nearly 3,000

people and has offices in the U.S., China,

India, and Ukraine.

Send product announcements to

[email protected]

http://www.springframework.net/download.htmlhttp://www.springframework.net/download.htmlhttp://www.springframework.net/download.html

7/27/2019 stp-2008-03

14/4014 Software Test & Performance MARCH 2008

Testing Early and Often Can Help

Prevent Web Applications From Crumbling

Under Pressure Like a House of Cards

7/27/2019 stp-2008-03

15/40

Photograph

byAlexeyKashin

to the discovery that the architecture

doesnt scale well, at a time when its too

late to do anything about it.

The earlier you start load testing dur-

ing the application life cycle, the earlier

the underlying infrastructures software

defects, design flaws and bottlenecks

will be found. A methodology that

establishes quality and perform-

ance-related activities early in the

application life cycle helps to miti-gate the risk of project failure, reduces

overall project costs, and increases

the applications quality and per-

formance.

Despite the well-known fact that the

cost of issue correction increases in

each downstream phase, project teams

often wait until the end of development

to set up and integrate load testing.

While its good practice to perform end-

to-end load tests on an application

shortly before going live with a new orupdated product to prove that the appli-

cation performs and scales as expected,

if the results dont meet the expecta-

tions, you cant do much to salvage

the project at such a late stage.

Usually these activities are lim-

ited to tuning the hardware or

software configurations, and

often, as a last resort, throw-

ing more or faster hardware

at the problem. If neither of

these activities is successful, its

back to development to find theroot cause of the problem in the

application code. In the worst-case

scenario, the core architecture isnt suit-

ed for scalability and performance, and

you have to redo core parts of the appli-

cation.

With the emergence of application

technologies such as SOA and the Web,

you also need to adapt your load testing

process to the new requirements and

challenges that new technologies bring.

What to Test EarlyDecisions about infrastructure and

application architecture are usually

done early in the application life cycle.

Both have a strong impact on applica-

tion design, implementation and opera-

tion. Reverting infrastructure and archi-

tectural decisions until late in the devel-

opment process can be painful. If you

want to prove your architectural con-

cept or different architectural alterna-

tives, you often start with a prototype

that implements your major concepts.By applying the prototype to the

planned hardware/software infrastruc-

ture early, you can test how well the cho-

sen architecture is suited to the infra-

structure it will run on.

Component load testing can be done

against business logic components as

soon as theyre ready, and without the

need of a fully developed UI or other

software components. With SOA-com-

ponent load testing, early load testing

becomes even more critical.

The earlier you start developing load

Ernst Ambichl is chief scientist at Borland.

By Ernst Ambichl

Many organizations wait until the end stages of applicationdevelopment to perform load testing. This practice often leads


7/27/2019 stp-2008-03


tests for components of your system, the

earlier you can start to find regressions

of performance when these compo-

nents change. By integrating load tests

as part of your regression test suite, you

can avoid detecting performance prob-

lems long after they are introduced.

Focus on Infrastructure AndArchitectureSome could argue that testing with a

focus on infrastructure is a classic bench-marking domain and doesnt have much

to do with load testing an application.

Basic hardware/software infrastructures

such as network switches, Web servers,

firewalls, application servers, DBMSs or

messaging middleware are already well

known and mature. Often you can even

find standard benchmarks for most

parts of your infrastructure. But be care-

ful: Standard benchmarks have down

sides, as they:

Ignore your applications individ-ual structure and workload

Exist only for discrete infrastruc-

ture parts, not for the specific com-

binationof infrastructure parts that

make up your application infra-

structure

Usually arent available for new

application technologies

The benefits of early load tests of

parts of the application within the tar-

get infrastructure are:

Early capacity assessment of the

application infrastructure Early check for scalability of your

architecture

Early identification of relevant per-

formance indicators and configu-

ration settings

Early information for infrastruc-

ture tuning

By load testing the infrastructure

early, youre able to learn about the con-

figuration settings and metrics that are

relevant to performance. Knowledge of

the relevant performance indicators

and configuration settings is highly

valuable, not only for later testing andtuning, but also for setting up the right

set of infrastructure monitors for your

live application.

For this kind of test, especially with-

in large IT organizations, two or more

groups often need to cooperate. The

first is IT operations, which is responsi-

ble for the infrastructure the applica-

tion will run on in production. The sec-

ond is the development team, which is

responsible for the application and the

scalability of the architecture. A dedi-

cated performance team (perhaps part

of the QA group, development or IT)

can greatly facilitate these efforts andact as the bridging group between

development and IT.

Load Testing a UI PrototypeLets assume you need to build a highly

scalable architecture for a Web-based

application with a high standard for

usability and speed of the user inter-

face. The application will be delivered

to all locations using the existing corpo-

rate intranet.

As part of the application develop-ment, youre designing a new HTML/UI

framework including third-party AJAX

components. You want to ensure early in

the process that the existing network

infrastructureas well as the companys

standard Web server infrastructurewill

deliver the required performance and

responsiveness for the new application.

To accomplish this goal, youll

load test a prototype of the applica-

tion UI using a new UI framework.

The prototype includes only a small

subset of the planned applicationsUI logic and is already using the

frameworks UI controls.

Since you dont yet have the business

logic in place, youre emulating the

business logic as hard-coded parts

WEB-APP LOAD TESTING

Web Server

UI Prototype

Web Server

UI Prototype

Database

Server

App. Server

App. Server

App. Server

Load Test Load Test Load Test

IntranetLoad

Balancer

FIG. 1: UI PROTOTYPE

Load Test

Web Server

(UI)

UI

Component 1

Database

Server

Data Access

Component 1

Data Access

Component 2App. ServerWeb Server

IntranetLoad

Balancer

UI Component

2...n

App. Server

(Business Logic)

BL

Component 1

BL Component

2...n

App. Server

FIG. 2:WITH FULL-TIER PROTOTYPE

7/27/2019 stp-2008-03

17/40MARCH 2008 www.stpmag.com 17

inside the UI prototype (see Figure 1).

Having this UI prototype in place, you

already can test how well the UI frame-

work performs on the planned infra-

structure. Stepwise, you can do tests

against a single Web server, load-bal-

anced Web servers, and across the cor-

porate intranet.

The idea of this type of early testing

is to determine whether some UI com-ponents might not be suitable for your

intranets network latency, for exam-

ple, or are consuming too much mem-

ory on the Web server to scale well.

This can and should be done beforeyou

base your whole application on these

components.

Load Testing a Full-Tier PrototypeIn another scenario, you may want to

verify that your applications planned

distributed architecture actually runsand scales as expected on the infra-

structure chosen for deployment.

To accomplish this, you can use anoth-

er prototype of your application for load

testing. The prototype needs only to con-

tain a small subset of the real application;

it doesnt need to be complete in terms of

the functionality it will deliver. Its impor-

tant that the prototype allows you to test

against a small set of use cases that already

touch all tiers of the application using the

proposed distributed architecture for the

application. For a typical Web-based appli-

cation, these tiers are Web server, applica-

tion server, database server and external

providers, if applicable.Load tests using a full-tier prototype

on the target infrastructure can help you

to get answers to the following questions:

What is the viability of the infrastruc-

ture?With a small subset of functionali-

ty touching all tiers, you can determine

whether the different infrastructure

components can work together to deliv-

er acceptable performance.

What are the design flaws that result in

bottlenecks?Your software architectures

scalability and performance, which

define how the different parts of the

infrastructure will work together, also

can be verified by a prototype that imple-

ments the architectural framework used

by the applications components. Even

different design alternatives (if availableas prototypes) can be tested for per-

formance, scalability and reliability.

Are there incompatibilities between the

different technologies used?Early detection

of incompatible parts of the infrastruc-

ture can be accomplished when a full-

tier prototype (as in Figure 2) exists that

touches all tiers of the application.

I ran into such a problem when test-

ing the servlet engine used in certain

Web-based products (in our case,

Tomcat) with one of the Web servers we

needed to support (in this case, IIS).

The problems occurred only under load

conditions. Testing our application onTomcat without IIS as the servlet con-

tainer never showed similar problems.

Component Load TestingModern multi-user applications are usu-

ally built with frameworks that allow for

modular, componentized design and

architecture. Componentizing your

application is the first and most impor-

tant step to enable you to begin your test-

ing earlier, when certain individual com-

ponents are getting ready. Especiallywith components that are accessible

remotely and/or concurrently from mul-

tiple clients, functional testing should be

expanded to component load testing as

soon as possible.

Often, functional tests for compo-

nents are already completed by devel-

opers with the help of standard unit-

testing frameworks like JUnit or NUnit.

With the right tools in place, its only a

small step to extend these JUnit/NUnit-

based tests to small component load

tests. My experience has shown thatmany elusive performance problems

can easily be found when exposing


UI Component

Load Test

Web Server

(User Interface)

UI

Component 1

Database

Server

Data Access

Component 1

Data Access

Component 2App. ServerWeb Server

IntranetLoad

Balancer

UI Component

2...n

App. Server

(Business Logic)

BL

Component 1

BL Component

2...n

App. Server

BL Component

Load Test

DA Component

Load Test

FIG. 3: COMPONENTS IN A MULTI-TIER APP

App 1 App 2

Service A Service B

App 2 App 3

Consumers

Services

Providers

Services Framework

FIG. 4: EXEMPLARY SOA

7/27/2019 stp-2008-03


remote and multi-instance components

to moderate load conditions.

What components should be load tested?

Its important to concentrate your load

testing on remote components and/or

components that are used concurrently

by multiple clients (Figure 3). From a

technology view, these are components

that expose their functionality via inter-

faces like RMI, RCP, CORBA and

(D)COM, .NET Remoting and, of

course, Web services. (SOA will be han-

dled in more detail later in this article).I also typically include SQL-based data

access components in my roster of can-

didates for load testing. Database per-

formance remains one of the critical

elements in a distributed application

architecture.

With the evolution of SOA technolo-

gy, there comes the need to adapt your

load testing approaches to SOAs new

requirements and challenges. First, lets

define SOA. As defined by XML.com:

SOA is an architectural style whose goalis to achieve loose coupling among interact-

ing software agents. A service is a unit of

work done by a service provider to achieve

desired end results for a service consumer.

Both provider and consumer are roles played

by software agents on behalf of their owners.

Loose couplingis the magic phrase in

this definition, and is the enabling fac-

tor that allows us to start testing as

soon as the services contract (or inter-

face) between the software agents is

defined. In theory, SOA architectures

are well suited for applying testingearly principles, as services should be

built with a high degree of autonomy

and with minimal dependencies to the

environment they run in. The termsWeb services and SOAP are purposely

omitted from this definition of SOA,

which is a much broader architectural

concept.

Factors that influence your load test-

ing approach for SOA applications

include:

A decreased predictability of use. The

agility SOA provides for building new

applications based on existing services

leads to more unpredictable usage pat-

terns and workloads compared to classic

n-tier applications. As a service

provider, you may not know who mightultimately consume your service at the

time youre developing it. Hence, test-

ing early for the scalability of your serv-

ices is important.

Increased complexity. Since applica-

tions based on SOA often consume mul-

tiple services (such as composite appli-

cations), the services call chain to fulfill

an application request can get quite

long, especially when using services that

themselves consume services.

Availability of service providers comeslate in the application life cycle. This is

especially true when your application

depends on a service provided by a

third party, such as a business partner. If

this is the case, you need to ensure that

you can test your application when not

all service providers are available.

Availability of service consumers comes

late in the application life cycle. You need

to ensure that you can test your service

before the service consumers begin con-

suming it.SOA facilitates distributed development.

Often, distributed teams or even differ-

ent organizations work on service

providers and service consumers. To

avoid finger pointing when perform-

ance problems are found during system

testing, its important to test the services

in isolation.

Complex root-cause analysis. Due to

the complexity and the distributed

nature of SOA applications, identifying

the root cause of SOA performance

problems is harder than in traditional n-

tier systems. The earlier you detect

problems in isolation, the easier it will

be to fix them.Impact of change increase. SOA-based

applications typically evolve over time

Consumers

Services

Providers

Services Test Framework

App 1

Service A

App 2

Simulator

App 2

Service B

App 3

FIG. 5:TESTING SERVICE B IN TEST FRAMEWORK

Consumers

Services

Providers

Services Test Framework

Simulator

App 1

Service A

App 2

App 2

Mock

Service B

App 3

FIG. 6:TESTING SERVICE A IN TEST FRAMEWORK


7/27/2019 stp-2008-03


and change constantly by adding new

applications on top of existing services

and new providers for existing services,

or by creating new services on top of

existing services. A simple change in a

service can impact multiple applications

consuming this service. This also intro-

duces the need to constantly retest and

carefully monitor your services whenev-

er you change the service.Different types of load tests can be

done in different stages of system devel-

opment. This depends on your testing

strategy and the availability of your SOA

components.

Isolation Load TestLoad testing should be done before you

integrate the service with your con-

sumer applications or integrate it into

the services framework.

Isolation load tests are the cheap-est load tests because you can do them

without having the whole infrastructure

in place. In addition, you typically wont

need a lot of virtual users to test a single

service behavior under load conditions

(synthetic workload in contrast to realis-

tic loads for end-to-end load tests).

This makes such tests good candi-

dates for regression testing. As soon as

the service changes, you can run isola-

tion load tests to check if the behavior

of the service has changed under load

conditions. Often, a fix of a defect relat-ed to the components functional

behavior just introduces a degradation

of performance.

Testing Without a ConsumerWhen developing services, you often

have no access to the client application,

or the application isnt ready for testing.

Also, if a service is consumed by multi-

ple applications, you wont reach suffi-

cient test coverage when testing is done

with only one client application.In the absence of a client applica-

tion, traditional test-script creation

techniques such as recording client

interactions arent possible. So, even if

you arent working in an agile develop-

ment shop, developing functional tests

as part of service implementation is a

good practice. You might even say its a

necessity. These functional tests can also

be reused for load testing.

Testing Without a Services Test

FrameworkDevelopers usually dont work within the

deployment infrastructure. They typically

use a small subset of the deployment

infrastructure or are developing within a

test framework (Figure 5) to execute and

debug their work, which is different from

the target framework.

Conducting small load tests as part of

developer activities (which can most often

be directly derived from unit tests) with-

out the burden to set up big infrastruc-

tures for testing helps to move load test-

ing nearer to the developer and earlier

into the application life cycle. You can dosmall load tests with your nightly develop-

er builds, which can signal changes in per-

formance as soon as possible.

Testing Without a ProviderAlthough SOA fosters loose coupling

between components and therefore

minimizes dependencies between

components, real dependency always

exists and cant be reduced. Real

dependencyis the set of features or serv-

ices that a system consumes from

other systems.

So how can you test a service that

depends on another service before that

service is available? In object-orientedprogramming, you use mock objects,

which are simulated objects that mimic

the behavior of real objects in con-

trolled ways. Similarly, you can create

mock servicesfor services that arent avail-

able or that you want to factor out of

your test (Figure 6).

Factoring out services by emulating

their behavior through mock services

offers the advantage of allowing testers

to control the behavior of the emulat-

ed service. This allows you to easilybuild load testing scenarios in which

you emulate the misbehavior of

dependent services such as service

calls that are tardy, time out or return

incorrect data.

Integration Load Test: ServicesFramework Integration TestAfter isolation testing, in which you

test the service in your services test

framework, you can replace your test

framework with the services frame-

work used for deployment. This letsyou test how well your service works in

the target environment. While this

usually adds the work of deploying

your services and providing a test envi-

ronment with the target services

framework, you can reuse the tests

Consumers

Services

Providers

Services Framework

App 1

Service A

App 2

Simulator

App 2

Service B

App 3

FIG. 7:TESTING SERVICE B IN SERVICES FRAMEWORK

You cancreate mock

services for

services that

arent available.


7/27/2019 stp-2008-03

20/40
http://ibm.com/takebackcontrol/innovate

7/27/2019 stp-2008-03

21/40
http://ibm.com/takebackcontrol/innovate

7/27/2019 stp-2008-03


youve already written.You wont perform integration load

testing (Figures 7 and 8) as often as

your isolation tests (as with every check-

in). But they should be done on a regu-

lar basis, such as every time develop-

ment passes a build to QA. This ensures

that QA isnt wasting time on testing

builds that dont pass the performance

criteria checked by your service frame-

work integration tests.

Youll also most likely increase work-

load by testing the scalability of the serv-

ices framework in combination withyour service. Extending your isolation

tests to services framework integration

tests helps to answers questions like:

How does the service scale within

the services framework?

How much overhead is the frame-

work adding to the service?

Does the framework correctly han-

dle the life cycle of the service?

What is the payload for enabling

security?

Integration Load Test: ServiceInteraction TestAs important as it is to test services in iso-

lation as early as possible to detect per-

formance problems, its equally crucial

to test the services in combination to

detect problems related to their interac-

tion with other services. No isolation test

will ever give you absolute certainty that

your system will pass even the most sim-

ple integration test, even if your isola-

tion tests cover almost all your code.

This is especially true of the per-formance, scalability and stability

aspects of your SOA-based application.

Establishing integration load tests assoon as two interacting services are

available helps to find integration prob-

lems early. Rerunning integration load

tests (regression testing) as soon as

dependent services change helps to

identify performance degradations at

the time theyre introduced.

With service interaction tests, youll

extend the test infrastructure to better

reflect the target system and extend the

workload patterns to more realistic sce-

narios (Figure 9). Also, your test scripts

will need to reflect that theyre testingthe integration aspect and not the isola-

tion aspect of the services.

System Load Test:End-to-End TestLoosely coupled architectural implemen-

tations such as those of an SOA create

additional complexities with end-to-end

load testing (Figure 10). Services that

share common a infrastructure or plat-

form require coordinated load testing to

truly replicate production-like states.

Providing the test infrastructure, cre-

ating and setting up these tests, identi-

fying production-like workloads, analyz-

ing results and finding the root cause

for performance problems is even moredifficult when compared with more tra-

ditional n-tier systems.

Everything that can be done to iden-

tify possible performance problems

before you actually perform your system

load test helps to lower the cost of fixing

performance problems and mitigate

the risk of project failure due to wrong

architectural decisions you cant redo at

the end.

Regression Load TestEvery change in a system might not onlyintroduce regressions in terms of func-

tionality, but also in terms of perform-

ance, scalability and stability. Focusing

only on functional test automation to

address regressions leaves performance

problems undetected until final system-

load tests.

Integrating load tests as part of

your regression test suite avoids the

danger of detecting performance

problems too long after they are intro-

duced. Because its expensive to set upand integrate load testing into a test

automation process, not all types of

load tests are suited for regression

load tests. Some good candidates for

regression load testing are:

Isolation load tests. Such load tests can

Consumers

Services

Providers

Services Framework

Simulator

App 1

Service A

App 2

App 2

Mock

Service B

App 3

FIG. 8:TESTING SERVICE A IN SERVICE FRAMEWORK

Simulator

App 1

Simulator

App 2

Service A Service B

App 2 App 3

Consumers

Services

Providers

Services Framework

FIG. 9: SERVICE A AND B TESTING IN SERVICES FRAMEWORK


7/27/2019 stp-2008-03


be done on a regular basis (ranging

from tests per check-in to nightly sched-

uled builds).

Services framework integration tests.

Isolation load tests also should be exe-

cuted regularly in the target services

framework.

Functional tests have simple success

conditions (usually pass/fail per test

case based on assertions in your testscript that make it easy to automate

your tests results analysis). This isnt

the case for load tests, which usually

require analysis of multiple metrics to

determine a pass or fail status. To auto-

mate that process and flag failed

load tests, you can use the following

methods:

Compare performance-relevant

metrics such as response times,

throughput rates and resource

consumption to defined baselines(static thresholds) that youve set

up for each individual load test.

Compare the change/delta of per-

formance-relevant metrics to his-

toric measurements of the same

test. In this case, you dont need to

set up thresholds for each test. Both methods have their advan-

tages and disadvantages. Decide

case-by-case which one best suits

your requirements.

Testing in Production:Application MonitoringLoad testing SOA applications under

real-life conditions is extremely com-

plex (Figure 11). Its therefore valu-

able to extend your testing approach

to the production phase of your appli-

cation to gather feedback for your test-ing.

Two techniques extend testing into

production and

both provide valu-

able feedback

about the accuracy

of your load testing:

Act ive service

monitors. By reusing

existing load-test-

ing scripts and exe-

cuting them on thelive system, you get

an accurate picture

of how the per-

formance of the

system under test

and the live system

compares. Leading

load-testing tools

have integrations

with application

performance monitoring frameworks,

which makes it easy to reuse your loadtesting assets for active monitors.

System and in-depth monitors. By using

system monitoring techniques, you can

keep track of services usage patterns.

Input/output data can be monitored with

in-depth monitoring techniques. Results

for service execution counts and inputcan be fed back into the testing process to

create more accurate workloads.

Early and IntegratedLoad testing can be done in early

stages of development and applied to

various components of an application

before the final end-to-end load test.

Early infrastructure load tests can mit-

igate the risk of investing in a specific

infrastructure that doesnt scale or

perform as needed.

By using prototypes of the applica-tion for load testing, you can proof

architectural concepts before you base

your whole application code on these

concepts. Component load tests helpto isolate performance problems

earlybefore they become difficult to

find and expensive to fix.

The integration of load testing

throughout the development process

has never been more important as,

due to increasing complexity, we face

less predictability of usage and more

dynamic changes in applications.

Because of SOAs loosely coupled

nature, unit and component testing

approaches can be adopted for load

testing, delivering early results aboutthe performance and scalability of

your services-based components.

Integrating load tests into your regu-

lar regression testing suite will help you

to detect performance regressions as

soon as theyre introduced. You can

extend your testing approach to the pro-

duction phase of your application by

reusing load testing assets for application

monitoring to gather feedback about

real usage and real performance.

For optimal success, load testingshould be conducted throughout the

project life cycle, started soon after an

application is conceived and continued

until its retired.

REFERENCES

What Is Service-Oriented Architecture?

Hao He,Sept. 30, 2003, OReilly xml.com,www.xml.com/pub/a/ws/2003/09/30/soa.html

W3.org, Web Services Glossaryhttp://dev.w3.org/2002/ws/arch/glossary/wsa

-glossary.html

Best Practices for Web Application Deploymentkeynote, Ernst Ambichl,Segue Software,Total

Performance Management Symposium,Mar. 18,

2004

Choosing a Load Testing Strategy Whitepaper,Ernst Embichl, Segue Software, 2005

Adjusting Testing for SOA, David S. Linthicum,SD Times,Aug. 15, 2007

App 1 App 2

Service A Service B

App 2 App 3

Consumers

Services

Services Framework

Real Users Real Users

Active MonitorApp 1

Providers

SystemMonitor

Service Performance Metrics(e.g. service response time)

Active MonitorApp 2

Service System Metrics(e.g. service execution count)

Service In-DepthMetrics(e.g. service input data)

In-depth Monitoring

FIG. 11:TESTING IN PRODUCTION


Simulator

App 1

Simulator

App 2

Service A Service B

App 2 App 3

Consumers

Services

Providers

Services Framework

Simulator

App 1

Simulator

App 2

FIG. 10: END-TO-END TESTING
http://xml.com/http://w3.org/http://dev.w3.org/20002/ws/arch/glossary/wsa-glossary.htmlhttp://dev.w3.org/20002/ws/arch/glossary/wsa-glossary.htmlhttp://dev.w3.org/20002/ws/arch/glossary/wsa-glossary.htmlhttp://w3.org/http://xml.com/

7/27/2019 stp-2008-03

24/40

than in-house development or

enhancement of software. In effect,

these two approaches constitute direct

or indirect outsourcing of some or all

of the development work for a system,

respectively.

While some project managers see

such outsourcing of development as

reducingthe overall risk, each integrat-

ed component can bring with it signif-

icantlyincreasedrisks to system quality.If your organization does or is plan-

ning to outsource, youll need to

understand the factors that lead to

these risks, and some strategies you

can use to manage them.

Ill illustrate the factors and the

strategies with a hypothetical project.

In this project, assume youre the proj-

ect manager for a bank that is creating

a Web application that allows home-

owners to apply for a home equity

loan.Youve purchased components

from two suppliers, including a COTSdatabase management system from

one of them. Youll hire an outsourced

custom development organization to

develop the Web pages, the business

logic on the servers, and the database

schemas and commands to manage

the data.

First, lets analyze how to recognize

the factors that create quality risks,

and identify strategies you can use to

manage those risks.


PhotographbyDavidFranklin

Rex Black is president of RBCS, a software,hardware and systems testing consultancy.

By Rex Black

More and more projects involve integration of custom-developedor commercial-off-the-shelf (COTS) components, rather

7/27/2019 stp-2008-03


Quality Risk Factors in IntegrationFigure 1 (page 27) shows four factorsthat lead to increased quality risk for a

system. Lets take a look at each, one at

a time.

One factor that increases quality

risk is component coupling, which creates

a strong interaction with the system

or consequence to the systemwhen

the component fails.

For example, suppose the customer

table on the Web application database

becomes locked and inaccessible

under normal load. In such a case,most of the other components of the

system, being unable toaccess customer information, also

would fail. The database is strongly

coupled to the rest of the system.

Another factor that increases risk isirreplaceability. This occurs when few

similar components are available or

the replacement is costly or requires a

long lead time.

If such a component creates quality

problems for your system, youre stuck

with them. For example, the database

package you choose might be replace-

able, provided that you dont do any-thing non-standard with it.

However, the development organi-zation will want to be paid for the cus-

tom-developed Web application. And

should you choose to try to replace it,

off-the-shelf products might not exist.

Yet another factor that increases

risk is essentiality, where some key fea-

ture or features of the system will be

unavailable if a certain component

doesnt work properly.

For example, suppose you planned

to include a pop-up loan planner on

the first page of your application to

allow customers to evaluate variouspayment scenarios. If that component

When You

Must Buy

Versus Build,

There Are

Ways To

Help You

Avoid Any

Slip-ups

7/27/2019 stp-2008-03


failed, you could still deliver most

of your applications major features,

since the planner is notessential to the

system.

But if the subsystem that accesses a

credit bureau to check customer cred-

it scores doesnt work, you cant

process loan applica-

tions. Checking credit

scores is essential to theapplication.

The final factor that

increases risk entails ven-

dor quality problems. This

factor can be compound-

ed if its accompanied by

slow turnaround on bug

fixes when problems are

reported.

If theres a high likeli-

hood of the vendor send-

ing you a bad compo-nent, the level of risk to

the quality of the entire

system is higher.

For example, if you

buy a commercial data-

base from a reputable,

established vendor, or if

you select a custom

development organiza-

tion with a proven track

record, then youll prob-

ably have fewer prob-

lems.If you use a new open

source database that has

never been used in

commercial applications

before, or if you use a

newly open custom devel-

opment organization, youll probably

have more problems, particularly if

there is poor technical support or if

its absent altogether.

Its obvious how these factors could

affect a typical data center application.Imagine a weapons system for which

defense contractors intend to develop

software to run on COTS platforms.

Here the situation is similar, though

the replaceability and vendor quality

problems could be exacerbated by

limited choices for components and

vendors.

How might these risks be mitigat-

ed? In my experience, Ive seen and

used four effective strategies.

Trust Your VendorOne strategy is simply to trust the ven-

dors component quality and testing,

and assume theyll deliver a sufficient-

ly good, more-or-less working compo-

nent to you. This approach may sound

nave on its face, but project teams do

it all the time. If you choose this

course, I suggest you do so with your

eyes open. Understand the risks youre

accepting. Allocate time

and finances as a contin-

gency for poor compo-nent quality. The more

coupled, essential and

irreplaceable the compo-

nent, the greater the

impact of such a situa-

tion.

To continue with our

example, you might

choose to trust both the

custom development

organization and the

database vendor. Youcould make such a deci-

sion rationally by check-

ing the development

organizations refer-

ences, assuming they can

provide references for

customers who used

them for projects that are

very similar to yours in

design and scale.

The same is true for

the database vendor,

though you might have todo your own research if

their sales and marketing

staff cannot or will not

provide references.

Relying solely on an

acceptance test is practi-

cally the same as trusting your partners

in the custom development situation.

For the COTS database, you could run

an acceptance test at the beginning of

the project for the database, using sim-

ple models to evaluate database per-formance, reliability and data quality

under your intended load conditions.

However, for the custom-developed

component, youll have to wait until

you receive the component before you

can acceptance-test it. And if the com-

ponent fails, what options do you

have?

Even if the contract stipulates that

you dont have to pay under these cir-

cumstances, you face a good chance of

a lawsuit, and you also have the actual

(and opportunity) costs of startingover with a new custom development

organization.

Manage Your VendorAnother strategy is to integrate, track

and manage the vendor testing of their

component as part of an overall, distrib-

uted test effort for the system. This

involves up-front planning, along with

sufficient clout with the vendor to insist

that they consider their test teams and

test efforts subordinate to (and con-

tained within) yours.To continue with our hypothetical

project, imagine that youre working

at a large bank and that the custom

development organization is a small

firm. Theyll probably be motivated to

get and retain your business. Theyll

be especially flexible if they think that

you have particularly good testing

processes and that they can learn

something from you.

In exchange for the effort you

expend managing their testing, youllhave early warning should quality

problems emerge, and therefore more

options to deal with such an outcome.

Conversely, though, if youre buy-

ing the database from a large COTS

vendor, they probably see your busi-

ness as a small part of their larger

product sales picture. They have their

own test processes, product road map

and target release dates. Its highly

unlikely that theyll be receptive to

offersmuch less insistencethat you

manage their testing operation.Even smaller COTS vendors, when

selling a COTS component, want to

sell you what theyre offering. Theyll

likely be averse to the possibility of an

open-ended situation under which you

might redefine the components

requirements through expansive test-

ing and ambiguous or evolving

pass/fail criteria for the tests. Ive seen

more than one COTS vendor get

burned by customers when they

allowed this to happen.Smart COTS vendors (large or

small) would probably insist that this

management of their testing, and any

resulting bug fixes and change

requests, be considered a customiza-

tion of their component subject to

time-and-materials billing.

The only likely exceptions to such a

condition would arise when the COTS

vendor saw a strong possibility that

working with you to fix problems and

change the product would benefit

their current or future customerssufficiently to justify the risks theyd be

taking.

REDUCING RISK

Run an

acceptance test

at the beginning

of the project

for the database,

using simple

models

to evaluate

database

performance.

7/27/2019 stp-2008-03


Fix Your VendorAnother option is to fix the compo-

nent vendors testing or quality prob-

lems. In other words, you go into the

situation expecting to either revamp

the vendors processes or build new

processes for them from scratch. Both

sides must expect that substantial

effort, including product modifica-

tions, will result. Once again, a keyassumption is that you have the clout

to insist that you be allowed to go in

and straighten out whats broken in

their test and quality processes.

This might sound daunting, but on

one project the client hired me to do

exactly that, and it worked out well.

The vendor was compensated for their

part of the work, including the modifi-

cations. And my client felt that the

vendor brought enough technical

innovation and capability to the proj-ect to justify their management of the

quality and testing problems. With

expectations aligned from the start,

both sides were happy.

Going back to our example, sup-

pose you assess the outsource develop-

ment organization before the project

and find their testing and quality

processes lacking. They accept your

assessment. You offer to help them fix

the issues that were identified, and

they accept that offer. If your assess-

ment identified the major problems,and if you and the vendor can resolve

those problems with the scope, budget

and schedule for the project, and if

continuing to work with that vendor

makes sense for other reasons, this can

succeed.

However, its difficult to imagine

that the database vendor would accede

to the request for an assessment of

their testing to begin with, not to men-

tion allowing you to come in and

implement changes to it. The very factthat a COTS vendor might agree to

such a request should set off alarm

bells in your mind. You should then

ask yourself if they actually have a

COTS product to sell or if youre deal-

ing with a prototype masquerading as

a product.

Test Your Vendors ComponentA final option, especially if you have

proof of incompetent testing by the

vendor, is to disregard their testing,

assume that the component is comingto you untested, and retest the compo-

nent. Youll have to allocate time and

effort for this, and realize that the ven-

dor will likely push to have every bug

report you submit reclassified as a

change request except in the most

egregious cases. You also have to ask

yourself if the vendor might decide, at

some point, to cut their losses and dis-

engage from the project. Youll want

to make sure you have contingency

plans in place should that happen.Ive had to do this for clients on sys-

tem testing projects. On one notable

project, a vendor sold my client a mail

server component that was seriously

buggy. We became aware of the prob-

lems by a series of misadventures in

which promised deliverables contin-

ued to show up late and with substan-

tive bugs, as well as fit-and-finish prob-

lems that gradually eroded our confi-

dence in them.

Eventually, the component didwork and was included in the system,

but the entire process took a few

months, not the one-week deliver-and-

integrate that was in the project plan.

Fortunately, slack elsewhere in the

schedule prevented this from becom-

ing a project-endangering episode.

Returning once again to our exam-

ple, suppose that you become aware of

serious quality problems in the early

prototypes delivered by the custom

development organization. You can no

longer trust their testing. Theres not

much point in managing a test process

that is clearly broken. Theres no time

remaining in your schedule to go inand fix their testing process. So, if you

intend to stick with this vendor, youll

need to start a serious testing effort to

take over where theyve failed.

Suppose you become aware of simi-

lar problems with the database vendor.

You can confront the vendor with the

problems. But if they delivered some-

thing to you with the assertion it

would work, can you really trust them

to resolve the problems now? Would

they be likely to let you manage theirtesting? If you try to do the testing

yourself, do you think theyll fix the

problems you find? If the component

isnt essential, youre best off omitting

it, or if itisreplaceable, youre best off

replacing it.

Whether for a COTS component or

a custom-developed component, these

are clearly nasty scenarios, and at

some point youd have to ask yourself

how you managed to get into such

trouble. If you ran acceptance tests ona COTS component, why werent the

problems identified?

If you thoroughly vetted your cus-

tom developer, why did they prove

incompetent? How should your quali-

ty risk-mitigation strategy for out-

sourced components change for

future projects? These are good ques-

tions, and should be saved for the

project retrospective. During the proj-

ect, the focus must remain on achiev-

ing the best possible outcome.

Implications,ConsiderationsAnd Success

All of these options can carry seriouspolitical implications. Should prob-

lems arise, the vendor is unlikely to

REDUCING RISK

Increased Risk to System

Quality Posed by Component

Component

Coupling

Component

Irreplaceability

Component

Essentiality

Vendor Quality

Problems

FIG. 1:THE FOUR CORNERS OF QUALITY RISK

7/27/2019 stp-2008-03


accept your assertion that their testing

staff is incompetent or their quality

unacceptable.

They might well attack your credi-

bility. If a senior manager made the

choice to use that vendorand

it might been an expensive

choicethat person might side

with the vendor against your

assertion.So, youll need to bring data

to the discussion about these

strategies if the triggering condi-

tions arise during the project.

Better yet, if youre dealing

with a custom-developed com-

ponent, see if you can influ-

ence the contract negotiations

up front to require the vendor

to submit their tests and their

test results, along with the offer

to let you perform acceptancetesting by your team prior to payment.

Build sufficient contingency plans

into your schedule, including an

allowance for replacement of the ven-

dor during the project if things start

looking bad. Make sure the vendor

understands that youre paying atten-

tion to quality and that payment

depends on delivery of a quality prod-

uct on time. Its amazing how motiva-

tional such clauses can be.

For COTS components, arrange a

careful component selection process,

including vendor research, talking to

references and acceptance-testing

using carefully designed tests. Identify

alternative sources if possible. Con-

sider the possibility and the conse-

quence of omitting the component if

it isnt essential.

Finally, DIYFinally, with the risks to system quality

managed at the component level, its

still possible to make a serious mistake

in the area of testing. Even the best-

tested and highest-quality components

might not work well in the particular

environment in which you

intend to use them. Plan on

integration-testing and system-

testing the integrated system

yourself.Integration of COTS and

outsourced custom-developed

software is a smart choice for

many organizations. Its a

trend that continues to grow

as organizations gain experi-

ence with it.

To ensure success on your

next integration project, con-

sider the factors that create

quality risk in such scenarios.

Select strategies that mitigatethose risks. Build risk mitigation and

contingency plans into your project

plan.

If you do these things and execute

the project carefully, with an eye on

testing and quality, you can control the

risks and reduce the likelihood and

impact of component quality prob-

lems.

REDUCING RISK

Plan on integration-testingand system-testing the integrated

system yourself.
http://www.checkpointech.com/BuildIT

7/27/2019 stp-2008-03

29/40

may not need to read this article.

However, if youre like most develop-

ment managers, your team has produc-

tivity highs and lows, feels down about

taking blame unfairly, or is frustrated by

any number of other problems common

to teams of all kinds.Developers come in all shapes and

sizes, ages and mentalities, and are

wrapped in many project experiences

and development methodologies. In my

years as a developer, Ive met a broad

range of interesting personalities

strong-willed and submissive alike. And

for as many types of developers, there

are probably as many specific tech-

niques for motivating them.

What f

stp-2008-03

Documents

Transcript of stp-2008-03