Fostering Connections to Success and Increasing Adoptions Act of 2008
Fostering Long-Term Test Automation Success
Transcript of Fostering Long-Term Test Automation Success
BT9 ConcurrentSession11/12/151:30pm
“Fostering Long-Term Test Automation Success”
Presented by:
Carl Nagle
SAS Institute, Inc.
Broughttoyouby:
340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·[email protected]·www.techwell.com
Carl Nagle SAS Institute, Inc.
At SAS Institute Carl Nagle is a principal software developer, focused for the past seventeen years on the development, integration, and evolution of enterprise software test automation tools and the open source SAFSDEV framework. For more than thirty years, Carl has been responsible for engineering both hardware and software testing systems including the design of advanced electronics for analog and digital subsystem testing with embedded software for high-end power generation and communication systems. This naturally evolved into the development of advanced software testing systems and frameworks. Carl and his team thrive on solving “impossible” test automation scenarios across multiple processes, platforms, and technologies.
Fostering*Long,term*Test*Automa2on*Success
!�Sustainable�!Test!Automa0on!
!with!Carl%Nagle,!!SAS!Ins0tute!
• Why%are%YOU%here?%
• Why%am%I%here?%(Has%it%really%been%9%years?)%
• How%did%we%achieve%ini@al%success?%
• Where%are%we%now?%
• Q%&%A%%!
In%A%Nutshell%
If!these!apply!to!YOU:!!• NEED!or!WANT!easy!test!automa0on!NOW,!• For!MY!current!team!or!project,!• Have!NO!concern!about!others,!or!even!future!projects,!• Hope!to!�fix�!a!failing!automa0on!strategy…!
Then,!you!might!be!looking!at!the!wrong!presenta0on!!!(!But!stay!!!We!might!be!able!to!change!your!mind!;Q)!
Why%Are%YOU%Here?%
If!these!apply!to!YOU:!!• NEED!test!automa0on!processes!that!last!• To!be!shared!and!leveraged!across!projects!• That!don�t!change!with!technology!or!tools!• That!might!�replace�!a!�struggling�!automa0on!strategy…!
Then,!you!might!be!looking!at!the!right!presenta0on!!!!
Why%Are%YOU%Here?%
!!!!!!!!!!!!!Over!18!years!ago,!SAS!Ins0tute!tasked!me!to:!!• Provide!a!common!test!automa0on!frameworkQQeliminate!silos!• Make!test!automa0on!accessible!to!developers!*and*!testers!• Handle!all!product!and!everQchanging!technology!domains!• Sustain!test!automa0on!success!over!years,!not!cycles!• Develop!it!in!an!openQsource!environment!
!!!!!!Today!I!report!that!this!prospers!s0llQQeven!aWer!18!years!!
Why%am%I%Here?%
1997% 2002% 2004% 2006% Today%
1. We!listened!to!the!industry!experts!�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�!
2. We!listened!to!poten0al!users!�We#need#to#leverage#and#share#our#test#exper4se#across#current#and#future#projects#regardless#of#the#technologies#or#tools#in#use.�!
3. We!collaborated!with!professional!peers!�Focus#the#test#automa4on#framework#to#deal#with#the#common#components#that#make#up#all#our#unique#applica4ons.�!!
4. We!implemented!it,!!shared!it,!!and!it!con0nues!to!evolve.!
How%did%we%achieve%ini@al%success?%
The!headlines!read:!!
!“In%today’s%environment%of%plumme@ng%cycle%@mes,%test%automa@on%becomes%an%increasingly%cri@cal%and%strategic%necessity.%%%Assuming%the%level%of%tes@ng%in%the%past%was%sufficient%(which%is%rarely%the%case),%how%do%we%possibly%keep%up%with%this%new%explosive%pace%of%webUenabled%deployment%while%retaining%sa@sfactory%test%coverage%and%reducing%risk?%%%The%answer%is%either%more%people%for%manual%tes@ng,%or%a%greater%level%of%test%automa@on.%AVer%all,%a%reduc@on%in%project%cycle%@mes%generally%correlates%to%a%reduc@on%of%@me%for%test.”%
The!headlines!read:!!!
%“With%the%onset%and%demand%for%rapidly%developed%and%deployed%web%clients,%test%automa@on%is%even%more%crucial.%%%Add%to%this%the%cold,%hard%reality%that%we%are%oVen%facing%more%than%one%ac@ve%project%at%a%@me.%%%Perhaps%the%team%is%finishing%up%Version%1.0,%adding%missing%features%to%Version%1.1,%and%prototyping%some%new%technologies%for%Version%2.0!”%
IBM!Ra0onal!Robot,!Func0onal!Tester!
HP!WinRunner,!QuickTest!Pro!
AutoIt!
“We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.”%
DIFFERENT!Technologies!
HTML!3.2,!4.0,!5,!SAP,!DOJO,!GWT!
Flex!
Java!
iOS,!Android!
VB,!.NET,!WPF,!Office!
DIFFERENT!Tes0ng!Tools!
SmartBear!TestComplete,!TestExecute!
EggPlant,!Sikuli,!ImageQBased!Tes0ng!
Mobile!SDKs,!Robo0um!
Selenium!1.0,!WebDriver!
Require%
C/C++!
• Separate!the!test!design!process!and!assets:!!• Completely!independent!of!the!test!execu0on!“engines”!• DataQdriven,!ac0onQbased,!keywordQdriven,!behaviorQdriven!• Support!testQfirst!principles!where!desirable!
!
�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%
PseudoQTypical!Example!Test!Records:!!
Google%%%%SignIn%%%%%%%%%%%Click%Explorer%%FolderTree%%Click%%"Coords=50,200�%%%%(tsk,%tsk,%tsk)%Explorer%%FolderTree%%Click%%My%Documents%%%%%%%%(beher)%
• Separate!the!test!design!process!and!assets:!!• Keep!vola0le!test!data!in!Maps!for!singleQpoint!of!maintenance!
�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%
[Google]%SignIn=�id=gb_70�%%[Explorer]%FolderTree=�Type=Treeview;Index=1�%%[FolderTree]%My%Documents=�Coords=50,200�%
[Google]%SignIn=�id=gb_70�%%[Explorer]%FolderTree=�Type=Treeview;Index=1�%%[FolderTree]%My%Documents=�Coords=50,200�%
The!Map:!
The!Test:!
Keep!vola0le!test!data!in!Maps!for!singleQpoint!of!maintenance.!
• Separate!the!test!design!process!and!assets:!!• Common%API!despite!years!of!evolving!technologies!and!tools!
�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%
• Doesn’t!maler!what!the!run0me!execu0on!engine!is:!!SQA%Robot,%IBM%Ra@onal%Robot,%XDE%Tester,%Func@onal%Tester%HP%WinRunner,%QuickTest%Pro,%SmartBear%Test%Complete,%AutoIt,%EggPlant,%Sikuli,%IBT,%Selenium,%Mobile%SDKs,%Robo@um%…%
Google%%%%SignIn%%%%%%%%%%Click%Explorer%%FolderTree%%Click%%My%Documents%
�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%
hlp://safsdev.sourceforge.net/sqabasic2000/SAFSReference.php!
• Separate!the!test!design!process!and!assets:!!• Test!IDE!can!be!different,!depending!on!skills!and!preferences!• CSV%Text%Files,!Spreadsheets,!Test!Composer!(CSV!Exporters)!• Java!IDE,!Eclipse,!SeleniumPlus!(Java!API!Drivers)!• Custom!IDE,!or!programs!(COM,!Ruby,!Perl…)!invoking!the!API!
�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%
Java!API!Examples:!!
Click(Map.Google.SignIn);%Click(Map.Explorer.FolderTree,%�My%Documents�);%
Click(Map.Google.SignIn);%Click(Map.Explorer.FolderTree,%�My%Documents�);%
Java!API!!Test:!
CSV!Test:!
The!Map:!
[Google]%SignIn=�id=gb_70�%%[Explorer]%FolderTree=�Type=Treeview;Index=1�%%[FolderTree]%My%Documents=�Coords=50,200�%
Then%do%safs%ac@on%Click%on%SignIn%in%Google%Then%do%safs%ac@on%Click%on%FolderTree%in%Explorer%using%�My%Documents�%
Cucumber!Test:!
• We!need!to!be!able!to!move!our!people!around!easily.!• A!person!tes0ng!.NET!can!easily!switch!to!tes0ng!HTML.!
!
• They!can’t!be!bogged!down!constantly!learning!new!tools.!• We!just!want!to!focus!on!test!design!using!the!Common%API.!
!
• They!want!to!support!each!other!across!divisions.!• Experience!applies!to!all!current!and!future!projects.!
“We#need#to#leverage#and#share#our#test#exper4se#across#current#and#future#projects#regardless#of#the#technologies#or#tools#in#use.”%
Applica@on%Map(s)%%(Vola0le!SPOM)!
SAFS%Services%and%Engine(s)!!
Java,!Web,!Flex,!Android!
Applica@on%Under%Test%%Java,!Web,!Flex,!Android!
Logs%
Func@onal%Tester%TestComplete%SeleniumPlus%Robo@um%ImageUBased%Tes@ng%
Executable%%Test%Designs%
%(CSV,!Cucumber,!
!Java,!SeleniumPlus)!
What%this%looks%like%in%a%complex%graph?%
• Implement!independent!Engines!for!test!execu0on:!!• Completely!independent!of!the!applica0on!being!tested!• Engines!interpret!and!implement!the!Common%API%
• Whether!the!domain!is!Java,!.NET,!Flex,!HTML,!SAP,!DOJO…!• Whether!the!tool!is!Adobe,!IBM,!Selenium,!SmartBear…!
• Engines!can!be!services!available!to!any!machine!process!• Engines!don�t!know!and!don�t!care!what!is!driving!them!• Mul0ple!engines!can!run!concurrently,!on!the!same!machine!• Support!engines!centrally!for!ALL!users,!and!ALL!applica0ons!
“Focus#the#test#automa4on#framework#to#deal#with#the#common#components#that#make#up#all#our#unique#applica4ons.”%
What%are%those%internal%Services?%
Logging!
Maps!
Variables!
Drivers!
Engines!
(But%you%really%don’t%have%to%care%about%this.)%
Where%are%we%now?%• We!have!grown!to!4!fullQ0me!inQhouse!framework!developers.!• Internally,!our!userQbase!grows!in!the!US,!Canada,!India,!and!China.!• We!just!completed!development!migra0on!to!SAFSDEV!on!GitHub.!• New!discussion!forums!at!safsdev.freeforums.net.!• Older!forum!archives!at!sourceforge.net/p/safsdev/mailman/!
!• We!have!doc!and!updated!binaries:!SAFSDEV!on!SourceForge.!
!• We!have!always!welcomed!contributors!for!new!engines!or!
features.!• Frankly,!we!have!been!VERY!BUSY!since!our!last!public!talk!in!2006.!
Fostering*Long,term*Test*Automa2on*Success
!�Sustainable�!Test!Automa0on!
!with!Carl%Nagle,!!SAS!Ins0tute!Q*&*A