M.A.S.Etesting - Let's Testlets-test.com/wp-content/uploads/2013/08/201306... · M.A.S.Etesting...
-
Upload
hoangtuyen -
Category
Documents
-
view
224 -
download
3
Transcript of M.A.S.Etesting - Let's Testlets-test.com/wp-content/uploads/2013/08/201306... · M.A.S.Etesting...
M.A.S.E testing (Manual Automated Scripted Exploratory)
Keith Stobie So*ware Quality Engineering Architect, TiVo
testmuse.wordpress.com
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
1
Keith Stobie
ASQ CSQE. BBST FoundaHons graduate. ISTQB FL.
Member of AST, ASQ, ACM, and IEEE, also SASQAG and PSQNC
Protocol Quality Assurance Process Model-‐Based Quality Assurance of Protocol DocumentaHon, STVR, 2011
Too Darn Big to Test – ACM Queue vol.3 (#1)-‐Feb.2005 TesHng for ExcepHons, STQE, July/Aug. 2000, 12-‐16
Keynoted at CAST 2007 and MBT-‐UC 2012
Test Architecture, Test Design, Test AutomaHon
Test Methodology
TiVo, Bing, Microso* servers & tools, BEA, Informix, Tandem
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
2
Context
• Scripted versus exploratory dimension and its interacHon with manual versus automated.
• Learn about the forces that influence when automaHon or manual tesHng is most appropriate and
• When confirmatory (scripted) or bug finding (exploratory) is most appropriate.
• The role and benefit of each type (manual scripted, automated scripted, manual exploratory, automated exploratory).
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
3
Perfect Software Testing
So#ware Tes+ng – Cem Kaner Perfect Tes+ng – James Bach an empirical technical
invesHgaHon conducted to provide stakeholders
with informaHon about the quality
of the product or service under test
TesHng is the infinite process of comparing the invisible
to the ambiguous so as to avoid the unthinkable happening to the anonymous
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
4
tes+ng. IEEE Standard for so*ware and system test documentaHon (IEEE Std 829-‐2008) (A) An acHvity in which a system or component is executed under
specified condiHons, the results are observed or recorded, and an evaluaHon is made of some aspect of the system or component.
(B) To conduct an acHvity as in (A).
Exhaustive vs. Pragmatic Testing Exhaus+ve / Complete Tes+ng
execute program on all possible inputs and compare actual to expected behavior.
• Could “prove” program correctness.
• Not pracHcal for any non-‐trivial program.
Comprehensive Tes+ng
• Usually some (coverage) criteria.
Pragma+c tes+ng
Select a vanishingly small % of all possible tests.
Goal: ExecuHng the Hny % of test will uncover a large % of the defects present.
A tes6ng method is essenHally a way to decide which Hny % to pick.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
5
Associations
Are you familiar with the term:
Manual Tes+ng?
Automated Tes+ng?
Scripted Tes+ng?
Exploratory Tes+ng?
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
6
It is tradiHonal for testers to be untrained in what I consider the basic cogniHve skills of excellent tesHng. It is tradiHonal, I suppose, for testers to have almost no ability to defend what they do, except through the use of clichés and appeals to authority and folklore. That tradiHon has poorly served our industry, in my opinion. I'd like to replace that with a tradiHon of context-‐driven, skilled tesHng -‐-‐ James Bach hnp://www.saHsfice.com/blog/archives/58
Manual testing What do you think of?
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
7
Automated testing What do you think of?
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
8
Scripted testing
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
9
What do you think of?
A test script in so*ware tesHng is a set of instrucHons that will be performed on the system under test to test that the system funcHons as expected. -‐-‐ hnp://en.wikipedia.org/wiki/Test_script Test script is a set of instrucHons (or steps) to follow in a prescribed sequence.
Exploratory testing What do you think of?
hnp://blogs.msdn.com/b/anunhara/archive/2011/10/20/exploratory-‐tesHng-‐introducHon.aspx
hnp://swtester.blogspot.com/2012/05/what-‐is-‐exploratory-‐tesHng.html
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
10
Agile testing
No test scripts
Adapt as you test Learn as you test Design as you test
Lightweight test Find new bugs all the time
No detailed test planning
Term was coined by Cem Kaner in 1983 in his book TesHng Computer So*ware : “Trust your insHncts” Updated descripHon by James Bach: “Exploratory tesHng is simultaneous learning, test design, and test execuHon, with an emphasis on learning”
Continuum
hnp://swtester.blogspot.com/2012/05/what-‐is-‐exploratory-‐tesHng.html
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
11
Test & Test Case (IEEE)
3.1.39 test:
(A) A set of one or more test cases.
(B) A set of one or more test procedures.
(C) A set of one or more test cases and procedures.
(D) The acHvity of execuHng (A), (B), and/or (C).
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
12
test case. IEEE Standard for so*ware and system test documentaHon (1) (IEEE Std 610.12-‐1990) A set of test inputs, execuHon condiHons,
and expected results developed for a parHcular objecHve, such as to exercise a parHcular program path or to verify compliance with a specific requirement.
(2) (IEEE Std 829-‐2008) DocumentaHon specifying inputs, predicted results, and a set of execuHon condiHons for a test item.
ScriptedóExploratory Continuum
hnp://www.qualitestgroup.com/media/presentaHons/PPT_Exploratory_TesHng_Explained.pdf
exploratory side of the con6nuum
pure scripted vague scripts
fragmentary test cases charters roles
freestyle exploratory tours
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
13
Itkonen, Juha; Mika V. Mäntylä and Casper Lassenius (2007). "Defect DetecHon Efficiency: Test Case Based vs. Exploratory TesHng" no significant differences in defect detecHon efficiency between {manually execu6ng} test case based tesHng (TCT) and {manual} exploratory tesHng (ET). [However TCT took more 6me – to prepare the test cases] no benefit in terms of defect detecHon efficiency of using predesigned test cases in comparison to an exploratory tesHng approach. – BJ Rollins, hnp://www.sigist.org.il/_Uploads/dbsAnachedFiles/Empirical_EvaluaHoness.pdf
Defect Detection Efficiency: Test Case Based vs. Exploratory Testing 79 advanced so*ware engineering students performed manual funcHonal tesHng on an open-‐source applicaHon with actual and seeded defects.
The distribuHons of detected defects did not differ significantly regarding technical type, detecHon difficulty, or severity.
TCT produced significantly more false defect reports than ET. Surprisingly, our results show no benefit of using predesigned test cases in terms of defect detecHon efficiency, emphasizing the need for further studies of manual tesHng.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
14
Phase Group 1 Group 2
Prepara+on Test cases for feature set A Test cases for feature set B
session 1 90 min
Test case based tesHng Feature set A
Exploratory tesHng TesHng Feature set A
session 2 90 min
Exploratory tesHng TesHng Feature set B
Test case based tesHng Feature set B
Empirical Evaluation of Exploratory Testing Effectiveness
No significant differences between the two approaches in terms of the detected defect types, severiHes, or detecHon difficulty. Difference in criHcality of defects between the 2 approaches is staHsHcally significant • ET more likely to idenHfy behavioral issues (e.g., “look and
feel”) • Scripted tests more likely to idenHfy technical defects
(computaHonal / code level) • Both approaches may miss criHcal defects The overall effecHveness of both exploratory tesHng and designing effecHve scripted tests depends heavily upon the individual tester’s professional knowledge of the system and tesHng!
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
15
Automated óManual Continuum?
If you read and hand-‐execute the code– if you do exactly what it tells you– then congratulaHons, you will have performed a poor manual test.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
16
Rule #1B: If you can truly automate a manual test, it couldn’t have been a good manual test.
Rule #1C: If you have a great automated test, it’s not the same as the manual test that you believe you were automa6ng.
Manual Tests Cannot Be Automated
hnp://www.saHsfice.com/blog/archives/58 -‐ James Bach
#1: A good manual test cannot be automated.
“human eyes, ears, hands, and brains are engaged in the execuHon of the test” – Michael Bolton
Technique vs. Approach
• A technique is a specific method that is used to accomplish a specific goal.
• An approach is the overall manner in which you act.
• Exploratory TesHng is an approach to tesHng. • All test techniques (i.e. manual, funcHonal) can be
done in an exploratory way or a scripted way (predefined test procedures, whether manual or automated)
• You can work in an exploratory way at any point in tesHng1
hnp://www.qualitestgroup.com/media/presentaHons/PPT_Exploratory_TesHng_Explained.pdf
1 hnp://www.tesHngeducaHon.org/BBST/exploratory/BBSTExploring.pdf
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
17
Random vs. Ad Hoc vs. Exploratory
hnp://qatestlab.com/services/No-‐DocumentaHon/ad-‐hoc-‐tesHng/
“To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory tesHng. “ – James Bach
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
18
Coder -‐ Domain Expert continuum
Manual
Coder Domain Expert
scriptless flowchart
Keyword AcHon word Gherkin
Rspec FIT
xUnit
test libraries
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
19
Automated
Automation goal
hnp://www.methodsandtools.com/archive/archive.php?id=94
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
20
Exploratory – find bugs does not guarantee any type of coverage
Numerous heurisHcs
Scripted – ConfirmaHon requires some knowledge of upfront results
Automated – requires hardware and so*ware may miss side-‐effect issues
Manual – requires human (even if computer assisted) may miss minor issues .
Continuum combinations
Automated Exploratory Manual Exploratory
Automated Scripted Manual Scripted
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
21
Keith’s History & Experience Mostly an automated script tester of Systems and Services (backend).
Wrote tools for exploratory API tesHng.
Did large scale system tesHng using random load generators.
Rare bouts of manual tesHng: Bing, Doyenz, TiVo
Three years of Model Based TesHng to automaHcally generate test scripts.
Limited GUI tesHng (manual or automated) [not my focus]
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
22 Manual Scripted: 5% Automated Scripted: 85%
Manual Exploratory: 5% Automated Exploratory: 5%
Manual Exploratory
Bing – Defects every week
Ken Johnston Single lener queries – inappropriate results
Bing log analysis : Big Data
Abandonment analysis – navigaHonal queries on images.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
23
Google “Z” safe off
Bing “s” (moderate)
Search Images Navigation
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
24
TiVo Manual Exploratory bugs
Take home builds
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie,
TiVo
25 Click
Charter Login
Outline Enter Username and Password
Specific
• Click mouse in username field to place cursor in username field. Type “testuser”.
• Click mouse in password field to place cursor in password field. Type “Pa$$w0rd”
• Click on login bunon
20 Ju
ne 2013
26
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
Manual Detail
20 Ju
ne 2013
27
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
AcHon LoginEnteringUsernameAndPassword( string username, string password)
Specific
• Mouse.Position(username).Click Username.Enter(“testuser”)
• Mouse.Position(password).Click Username.Enter(“Pa$$w0rd”)
• LoginButton.Click
Automated Details
TiVo Manual Script
Objec+ve
Verify that the TCD is able can make a recording from live TV Steps 1. Press the Live TV bunon 2. Press the Channel Up bunon to change channels and verify A 3. Wait awhile to build up some video in the cache 4. Press the Record bunon 5. Press select on the "Record this showing" menu item 6. Allow some Hme to pass 7. Press the Record bunon 8. Press the down bunon then select to select "Stop the current recording" 9. Press the TiVo bunon (you'll go to the main screen) 10. Press the Select bunon. You should be on My Shows (or what ever its name for that
product) 11. Verify B
Expected Results A. Verify the new channel has video that is correctly displayed B. Verify the recorded show is present at the top of the show list if you are sorted by date
Test Case 120527: Record from Live TV
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
28
TiVo Manual Script
Objec+ve
Verify able to direct tune cable channels.
Steps
1. key in a cable channel number from liveTV such as 201 2. verify A
Expected Results
A. Able to tune cable channel
Test Case 108864 Direct Tuning -‐ cable channels
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
29
TiVo Automated script
$controller-‐>gotoScreen('central’)
$controller-‐>gotoLiveTVChannel($channel);
$result = $controller-‐>{'screendump'} -‐>parseScreenDump(filefilter => "livetv.xsl");
$tuned_channel = $result-‐>{'channel-‐number'};
if ( $channel ne $tuned_channel ) {
$self-‐>setError("Not able to tune to Channel.");
return FAILURE;
}
return SUCCESS;
Test Case 108864 Direct Tuning -‐ cable channels
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
30
Automated Scripted Testing Pros • Low learning curve, no special tools • Low risk, done it for years • Reach, everyone can do it, no tools required • Predictable, repeatable • Success, proven mechanism to find bugs
Cons • Tests require a lot of code, every scenario • Expensive to maintain, cleanup actually lasts for years • Test issues, far surpass product bugs • Low coverage, out of the box • Rigid, difficult to improve, change, or just keep up to date • StaHc, stop finding bugs, same scenarios, missing bugs • Doesn’t scale, increasing feature sets, complexity, dependencies
Ed Triou
20 J
une
201
3
31
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
Copyright © 2013 Keith Stobie, TiVo 32
What’s a Model?
A model:
Is an abstrac+on or simplified representa+on of the system from a parHcular perspecHve
Supports inves+ga+on, discovery, explanaHon, predicHon, or construcHon
May be expressed as a descripHon, table, graphical diagram, or quanHtaHve mathemaHcal model
Is not necessarily comprehensive
20 Ju
ne 2013
Copyright (c) 2002 Elisabeth Hendrickson, Quality Tree So*ware, Inc.
33
Models in Everyday Life
Inspired by :“Modeling: A Picture's Worth 1000 Words” Copyright (c) 2002, Quality Tree So*ware, Inc.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
// Some WCF service interface [ServiceContract] public interface IService { [OperationContract(AsyncPattern = true)] IAsyncResult BeginGetCustomers(AsyncCallback callback , object state); List<Customer> EndGetCustomers(IAsyncResult result); }
Manual Model Exploration
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
34
Simple Finite State Machine (FSM) of an API.
End API
beginAPI
Not Started
endAPI
begin API
Proces-‐sing
Test Cases (input sequences) 1. begin; end; end 2. begin; begin 3. end; 4. begin; end; begin; end
hnp://www.codeproject.com/ArHcles/56175/A-‐Generic-‐Class-‐for-‐Wrapping-‐Asynchronous-‐Begin-‐En
Automated Exploratory
Accidental – thronled process creaHon
Dumb Monkey & Fuzzing – Random input
Random failures – similar to Chaos monkey
Random stress tesHng
Mine
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
35
Fuzz Testing
History – a dark & stormy night
Fuzz Revisited: A Re-‐examina6on of the Reliability of UNIX ... by BP Miller, et al
hnp://www.opensource.org/advocacy/fuzz-‐revisited.pdf
hnps://www.owasp.org/index.php/Fuzzing Tools list
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
36
providing invalid, unexpected, or random data to the inputs of a computer program
Fuzz Testing
Goal: push invalid data as far into system as possible. Unexpected input : length and content Both Dumb and Smart fuzzing valuable! • Dumb generates data with no regard to the format (data form of dumb Monkey tesHng)
• Smart requires knowledge of data format or how data is consumed. The more detailed the knowledge – the bener.
Generate -‐ creates new data from scratch
Muta+on -‐ transforms sample input data to create new input data : add/change/delete data Most fuzzing tools are a mix of each approach
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
37
Monkey Testing
Dumb / Wild / Unmanaged
banging on the keyboard by randomly generaHng input (key-‐presses; and mouse moves, clicks, drags, and drops)
Smart / Trained / Managed
detects available opHons displayed to the user and randomly enters data and presses bunons that apply to the detected state of the applicaHon
Chaos Monkey (ne�lix.com) hnp://techblog.ne�lix.com/2012/07/chaos-‐monkey-‐released-‐into-‐wild.html
Chaos Monkey is a service which runs in the Amazon Web Services (AWS) that seeks out Auto Scaling Groups (ASGs) and terminates instances (virtual machines) per group.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
38
hgp://www.ques+oningso#ware.com/2007/02/people-‐monkeys-‐and-‐models.html
Flavors of Stress Tests
Load Ensures that system can perform “acceptably” under Input and Resource load both Constant and Varying
Limit When capacity\limits are known. Equivalent to boundary testing
Breakpoint When capacity\limits are not known. Find heaviest load the component can handle, before failure
Torture Taking component beyond known or pre-defined min\max limits
Duration To run for some amount of time without undue load or failures. (aka “soak” testing)
Synchronization Flush out timing\synchronization\concurrency issues, e.g., multi-thread tests
Fault Injection Deterministically inject faults, simulate error conditions
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
39
40
Chat Slice where local consistency does matter
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
Spec Explorer
Move to a parHcular state (point), and then explore it (shoot).
Also “On The Fly” tesHng – conHnue generaHng inputs unHl Hme runs out. (Smart “state” monkey).
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
41
point and shoot hgp://msdn.microso#.com/en-‐us/library/ee620432.aspx
Manual Testing Advantages Disadvantages
Leverage experience of Testers to find Defects
Hme consuming
usable for majority of so*ware tesHng types
not suitable for Performance TesHng, Stress TesHng and Soak TesHng
performable in different phases of SDLC
ROI Is low for tests like Smoke or Regression Tests
performable even when applicaHon is not stable
Can lead to defect leakage due to human miss or error
EffecHve when requirements are volaHle
Not effecHve if tests require validaHon of large data sets
EffecHve when UI changes are frequent
Cannot perform most of the tests related to Security TesHng
Testers can lose interest running same tests over longer period of Hme
Success depends on experience and knowledge of testers
www.So*wareTesHngSo*ware.com
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
42
Manual/Automated Effectiveness
Manual Effec+ve Automated Effec+ve FuncHonal Sanity Unit Smoke Exploratory Database Ad-‐Hoc Regression IntegraHon API User Acceptance Web Services Usability Alpha / Beta Compliance
hnp://www.so*waretesHngso*ware.com/manual-‐tesHng/
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
43
Exploratory Tes+ng
• FuncHonality 1
• FuncHonality 2 • FuncHonality 3
• FuncHonality 4 Scripted Tes+ng
• FuncHonality 1
• FuncHonality 2
• FuncHonality 3
• FuncHonality 4
Differences Between Scripted and Exploratory Testing Scripted Tes+ng Exploratory Tes+ng
Directed from requirements Directed from requirements and exploring during
DeterminaHon of test cases well in advance
DeterminaHon of test cases during tesHng
ConfirmaHon of tesHng with the requirements
InvesHgaHon of system or applicaHon
Emphasizes on predicHon and decision making
Emphasizes on adaptability and learning
Involves confirmed tesHng Involves InvesHgaHon
Is about Controlling tests Is about Improvement of test design
Like making a speech — you read from a dra*
Like making a conversion — it’s spontaneous
The script is in control The tester’s mind is in control
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
44
hnp://tesHnginterviewsquesHons.blogspot.com/2013/01/exploratory-‐tesHng.html
BeneUits of Computer-‐Assisted Testing • Easy test case maintenance
• Reduced costs • More test cases
• Early bug detecHon • Increased bug count • Time savings
• Time to address bigger test issues
• Improved tester job saHsfacHon
Harry Robinson Exploratory Test AutomaHon slides from CAST 2010
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
45
“You can start with a small model and build in the direcHon you think will do the most good for you. “
Issues with Random Testing
VerificaHon of the result. Easier to have staHc tests with pre-‐computed outputs.
Very low probability sub-‐domains
• likely to be disregarded by random tesHng but cost of failures in those sub-‐domains may be very high
• good reason for using it as a complementary rather than the sole tes+ng strategy.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
46
Validation & VeriUication
ValidaHon (requirements confirmaHon) • Establishing the fitness of a so*ware product for its use. • “Are we building the right product?” • Requires interacHon with customers
VerificaHon (design confirmaHon) • Establishing the correspondence between the so*ware and its specificaHon.
• “Are we building the product right?” • Requires interacHon with so*ware
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
47 hnp://www.construx.com/PracHcing_Earl/Doing_JusHce_to_V_V/ … Confirmaiton
Analyze these claims:
You should write a test plan
It’s important that tesHng be repeatable
Each test case should have an expected result
Test automaHon saves money and Hme
All tesHng is based on a model of what is being tested
Good enough quality is not good enough
An undocumented test cannot be improved
Exploratory tesHng is a useful pracHce
It’s bener to use the term defect than bug
Ambiguity should be removed from requirements
Becoming a Test Expert – James Bach
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
48
ConUirmation & Bug Finding
Confirm Find bugs Quality isn’t worse than before Quality isn’t sufficient Support the team CriHque the product Repeatable Unpredictable
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
49
FuncHonal Tests Examples Story Tests Prototypes SimulaHons
Exploratory TesHng Scenarios
Usability TesHng User Acceptance TesHng
(UAT) Alpha / Beta
Unit Tests Component Tests
Performance & Load TesHng
Security TesHng “ility” TesHng
Supp
ortin
g th
e Te
am
Critique Product
Business Facing
Technology Facing
Q2 Q3
Q1 Q4
Copyrig
ht 2009: Lisa
Crispin, Ja
net G
rego
ry – Drago
nFire
Inc
used with
permission.
Agile Testing Quadrants
hnp://waHrmelon.com/2011/06/10/yet-‐another-‐so*ware-‐tesHng-‐pyramid/
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
50
Automated & Manual Manual
Automated Tools
51
Construx Validation Test list Level Test Type Description Notes
0 Unit Do individual code units (i.e., modules) work by themselves?
Typically done by developer to check the integrity and functionality of code units.
0 Code Integration
Do individual code units work well with one another?
Typically done by developer to insure code units interface successfully with one another.
1 Build Acceptance
Can the application be successfully installed?
Done by Tester
1 Smoke Does the build work well enough to be tested?
Done by Tester
1 Regression Have bug fixes or design enhancements in the code caused other parts of the program to malfunction?
Executed immediately after a successful Smoke Test. Tester reruns tests proved successful in the previous build.
2 Functional Are all the features and functions described in the Functional Specifications implemented and working properly?
Should cover both Front End (UI, GUI) as well as Backend (i.e., data validation). Conducted by QA as Black Box testing.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
52
Construx Validation Test list Level Test Type Description Notes
3 Desktop Configuration/Host Integration
Does the program: 1) Work in the communications, hardware and software environment described in the Requirements documents? 2) Run successfully with other typical programs loaded? 3) Interface successfully with remote network resources (e.g., database servers) and local peripherals (e.g., printers and modems)?
Integrates the program into the hardware environment, confirms that it can run concurrently with other typical software and that all necessary resources (local and remote) can be accessed. Conducted by QA as Black Box testing.
4 Load / Performance
Does the program: 1) Work acceptably fast for the user under typical conditions? 2) Work acceptably well when subject to extreme processor activity or volumes of inputs/outputs? 3) Work acceptably well with limited resources?
Finds out how well the program handles under less than optimal conditions. Managed by QA but often involves automation or volunteer ad hoc testers and network personnel to create atypical conditions.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
Cop
yrig
ht ©
201
3
Kei
th S
tobi
e, T
iVo
53
Construx Validation Test list Level Test Type Description Notes
4 Security / Compliance
Has end to end system security been confirmed? Does the program threaten to expose: 1) Corporate assets? 2) Private Customer Information? Does the product threaten government and organizational policy/procedure.
Insures protection of corporate assets and private user information from unauthorized access and assures compliance with governmental, industry and organizational policies and procedures.
5 Accept-ance
Is the service: 1) Comfortable to use? 2) Customizable? (if applicable) 3) Complete? Correct? (includes all user desired features and functions) 4) Control flow driven: efficiently handles typical user scenarios 5) Error flow driven: offering options and instructions for every error path.
Checks the program for ultimate user satisfaction.
6 Test in Production
1) Does the service operate correctly with real load
Monitors the system for continual satisfaction
20 Ju
ne 2013
Static / Dynamic V&V
Sta+c
• So*ware inspecHons • StaHc analysis of source code
• control/data flow analysis
Dynamic
• Defect tesHng – look for errors in funcHonality • Load tesHng – look for errors in scalability, performance,
reliability.
Verifica+on(design) & Valida+on(requirements)
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
54
Software Testing Techniques
StaHc
So*ware tesHng techniques
Dynamic
SyntacHc SemanHc Black-‐box White-‐box
Structural Random FuncHonal
StaHsHcal DeterminisHc
must execute the so*ware?
examine code syntax?
code examine?
How is test data selected?
Test data type generated?
Discrete ConHnuous Fault-‐based Error-‐based
Exit Criterion (reliability model)
Adequacy Criterion (control / data flow, …)
An EvaluaHon Scheme of So*ware TesHng Techniques, H-‐D. Chu, Univ. of Newcastle, Dept. of CS, TR 583, June 1997
Classifica3on Evalua3on
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
55
no
no no yes
yes
yes
What do you think?
• Is it “tesHng” if you simply run the program to see what it does?
• Is requirements confirmaHon done best with staHc or dynamic tesHng?
• Is design confirmaHon done best with staHc or dynamic tesHng?
A Beginners Guide to TesHng – Phillip Johnson, InformaHon & CS, U. of Hawaii
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
56
… Testing Acceptance Accessibility AcHve Agile Age Ad-‐hoc Alpha AsserHon API All-‐pairs Automated Basis Path Backward CompaHbility Beta Benchmark Big Bang IntegraHon Binary Portability Black box Boundary Value Bonom Up IntegraHon Branch
Breadth Business Logic Code-‐driven CompaHbility Comparison Component ConfiguraHon CondiHon Coverage Compliance Concurrency Conformance Context Driven Conversion Decision Coverage DestrucHve Dependency Dynamic Domain Error-‐Handling End-‐to-‐end Endurance Exploratory
Equivalence ParHHoning Fault injecHon Formal verificaHon FuncHonal Fuzz Gorilla Gray Box Glass box GUI so*ware GlobalizaHon Hybrid IntegraHon IntegraHon Interface Install/uninstall InternaHonalizaHon Inter-‐Systems Keyword-‐driven Load LocalizaHon Loop Manual Scripted Manual-‐Support
Model-‐Based MutaHon Modularity-‐driven Non-‐funcHonal NegaHve OperaHonal Orthogonal array Pair Passive Parallel Path PenetraHon Performance QualificaHon Ramp Regression Recovery Requirements Security Sanity Scenario Scalability Statement
StaHc Stability Smoke Storage Stress Structural System System IntegraHon TesHng in ProducHon Top Down IntegraHon Thread Upgrade Unit User Interface Usability Volume Vulnerability White box Workflow
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
57
Manual Exploratory
+ Cost effecHve one-‐shot bug discovery
-‐ No coverage guarantees, varies every Hme (for bener or worse)
Tester requires extensive background or knowledge
Numerous heurisHcs
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
58
Manual Scripted
for rapidly changing product + can guarantee some coverage
⬇ Manual exploratory doesn’t guarantee coverage
⬇ Automated scripted maintenance can be too high
+ Costly automaHon vs. cheap human Moravec's paradox : reasoning vs. sensorimotor,
e.g., sound quality, video quality, captcha -‐ May miss “minor” changes, e.g., typos, wrong font, …
• Requires upfront result knowledge Asperger syndrome highly talented specialist people
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
59
Automated Scripted
+ Guaranteed coverage for slowly or non-‐changing product
-‐ Misses many “side” observaHons, and/or very fragile
• Requires upfront result knowledge
⬇ Manual Scripted expensive vs. repeated automaHon
• Manual Exploratory complements large uncovered areas
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
60
Regression vs Latent
WriHng tests and running tests both have costs.
Run same tests, reduce regression defects. Regression: old code fails due to new code
AutomaHon mostly helps here
Create new tests, reduce latent defects (always there). Manual tesHng can most help here
Regression Latent
Old Code New Code
20 Ju
ne 2013
61
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
Regression Risk vs Code Base
0%
50%
100%
0
50
100
150
200
250
1 2 3 4 5 6 7 8 9 10 11
Old Code New Code Regression Risk
Example for 10 iteraHons when each new 20 units of code has 5% errors and old code 1% regression errors.
20 Ju
ne 2013
62
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
IteraHon / Sprint / Release
% of bugs due to regression
Units of Code
Automated Exploratory
+ Finds defects that other approaches miss
-‐ Incomplete, can’t cover most of products today
• Numerous heurisHcs
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
63
For each Type or Technique an approach to testing
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
64
Automated Exploratory Manual Exploratory
Pex (pexforfun.com) ModelJUnit, geneHc Modifying in the debugger
Many tools, e.g. xUnit Debugger script
Automated Scripted Manual Scripted Unit / Structural
Automated Exploratory Manual Exploratory
Broad, not deep Primary paths
(for slow changing) most basic use cases (for fast changing)
Automated Scripted Manual Scripted
Smoke
Automated Exploratory Manual Exploratory
Model Based TesHng heurisHcs
for most common/criHcal no ROI to automate
Automated Scripted Manual Scripted
Func+onal / Black Box
Automated Unit Testing Current automaHc unit generaHon test techniques: • Simple known failure modes (e.g., null parameters as inputs)
• analysis of the data structures used. Data structure generaHon can be based on Constraint Solvers, Models, and symbolic execuHon
You may see 100% code coverage, with no results checking! Be aware of results checking thoroughness; not just code coverage.
Eclat creates approximate test oracle from a test suite, then uses it to aid in generaHng test inputs likely to reveal bugs or expose new behavior while ignoring those that are invalid inputs or that exercise already-‐tested funcHonality
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
65
Manual Script Exactness
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
66
Avoiding Overkill in Manual Regression Tes+ng -‐ Lisa Shepard hnp://www.uploads.pnsqc.org/2012/papers/t-‐46_Shepard_paper.pdf
Document the Intention of the Test
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
67
Avoiding Overkill in Manual Regression Tes+ng -‐ Lisa Shepard
Jonathan Kohl, hnp://www.methodsandtools.com/archive/archive.php?id=65
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
68
Manual Exploratory
Marick hypothesis:
An applicaHon built with programmer TDD, whiteboard-‐style and example-‐heavy business-‐facing design, exploratory tes+ng of its visible workings, and some small set of automated whole-‐system sanity tests will be cheaper to develop and no worse in quality than one that differs in having minimal exploratory tes+ng, done through the GUI, plus a full set of business-‐facing TDD tests derived from the example-‐heavy design.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
69
hgp://www.exampler.com/blog/2008/03/23/an-‐alterna+ve-‐to-‐business-‐facing-‐tdd/
Automated Exploratory Testing
GeneHc Algorithms -‐-‐ global opHmizaHon vs random
Unit, FuncHonal, Concurrency
Guided -‐-‐
Kyle A. Larsen – coverage exploraHon of virus detecHon
Don Sleuts – SQL grammar & negaHve grammar
Large (e.g. nested) SQL legal inputs
SQL almost correct (SQL monkey, Fuzzing)
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
70
Basic outline of Genetic Algorithm
IniHalize (populaHon) -‐-‐ global opHmizaHon
Evaluate (populaHon)
While (stopping condiHon not saHsfied) { SelecHon (populaHon) -‐-‐ most fit => most likely to swap Crossover (populaHon) -‐-‐ swap “genes” Mutate (populaHon) -‐-‐ small changes to small group Evaluate (populaHon) -‐-‐ kill off less fit
}
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
71
Input vars == genes, set of inputs == chromosomes.
A Survey on So*ware TesHng Techniques using GeneHc Algorithm JCSI InternaHonal Journal of Computer Science Issues, Vol. 10, Issue 1, No 1, January 2013 ISSN (Print): 1694-‐0784 | ISSN (Online): 1694-‐0814 hnp://ijcsi.org/papers/IJCSI-‐10-‐1-‐1-‐381-‐393.pdf
Testing real-‐time systems using genetic algorithms use geneHc algorithms to find inputs with the longest or shortest execuHon Hmes automaHcally.
check whether they produce a temporal error.
The fitness func+on is the execuHon Hme measured in processor cycles.
Experiments using geneHc algorithms on a number of programs with up to 1511 LOC and 843 integer input parameters have successfully idenHfied new longer and shorter paths than had been found using random tesHng or systemaHc tesHng.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
72 So*ware Quality Journal, 1997, Volume 6, Issue 2, pp 127-‐135
Machine vs. Human
Machine Human Chess Play (deep search) Chess Play (panern match) Learning (training data) Learning Exploring Exploring
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
73
Advanced Chess is a form of chess developed in 1998 by Kasparov where a human plays against another human, and both have access to computers to enhance their strength. Argued by Kasparov to be stronger than a human or computer alone
What Is Exploratory Test Automation? “Computer assisted tesHng
that supports learning of new informaHon
about the quality of the so*ware under test”
-‐-‐ Douglas Hoffman & Cem Kaner
“automated” tests are enHrely scripted tests – Michael Bolton
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
74
For each Type or Technique an approach to testing
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
75
Automated Exploratory Manual Exploratory
(aka Stress!) Poke & Measure
Many tools “exercises”
Automated Scripted Manual Scripted
Performance
Automated Exploratory Manual Exploratory
Reachability, GUITAR, AUTO Works for me?
Automated checks Specifics to look for
Automated Scripted Manual Scripted
Usability
Automated Exploratory Manual Exploratory
Dynamic fuzzing Social anacks
Script kiddies Captcha
Automated Scripted Manual Scripted
Security
Memes about Test Automation
“Automated tests represent the automaHon of a manual process.” – Harold F. Tipton, Micki Krause
“The most important benefit of automated tesHng over convenHonal manual tesHng is the minimizaHon of costs over repeated tests.” – Markus Helfen, Michael Lauer
“I have never been convinced that finding ‘new’ bugs is a realisHc expectaHon for test automaHon.” – I.M. Testy
“A*er you have run your automated tests for the first Hme, you are done finding new bugs. Never again will you find a new issue. ” – Steve Rowe
“Running automated test scripts can not be used to find new bugs in the so*ware ...” – Cordell Vail
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
76
The Net=lix Simian Army
Chaos Monkey randomly disables producHon instances Freely available on github
Latency Monkey induces arHficial delays
Security Monkey finds security violaHons or vulnerabiliHes
10-‐18 Monkey (short for LocalizaHon-‐InternaHonalizaHon, or L10n-‐i18n) detects configuraHon and run Hme problems in instances serving customers in mulHple geographic regions, using different languages and character sets.
Chaos Gorilla simulates an outage of an enHre Amazon availability zone.
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
77
hgp://techblog.nenlix.com/2011/07/nenlix-‐simian-‐army.html
TestIstanbul Conferences 2012
Hybrid Approach
• Hybrid approach combines pure random tesHng with heurisHc search strategies in interleaving phases to increase overall coverage
• User can choose between various coverage goals, exit criteria and search strategies
Random Search Phases
Heuris+c Search Phases
An InnovaHve Approach to Model-‐based TesHng
Fake TrafUic (capacity test)
User Requests Service Backend
service Frontend service
Data Data Feed
ProducHon
Test
replayed Requests
Capacity test a single service, or enHre offering (via the frontend)
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
79
20 Ju
ne 2013
Modern Cap Generation Toolset
Controller
DC1 Monitoring system
Capacity clients
Bing Entry point VIP
DC3 Monitoring system
Capacity clients
Bing entry point
VIP
DC2 Monitoring system
Capacity clients
Bing Entry point VIP
DC…n Monitoring system
Capacity clients
Bing entry point VIP
From: Greg Veith – Workshop on Performance & Reliability 2010 (WOPR14)
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
80 1. Controller drives each of the clients across any/all DCs 2. Capacity is generated within each datacenter to avoid cross-‐DC network uHlizaHon 3. Controller pulls data from monitoring system auto-‐throgling based on SLA 4. 2 -‐3 engineers required for a typical full scale test
20 Ju
ne 2013
Harry Robinson
Consistency HeurisHc
Birthdates
Abraham Lincoln 2/12/1809
Abe Lincoln 12/2/1809
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
81
Rules of Thumb
Exploratory Test Automation Examples 1. Disk buffer size 2. Simulate events with diagnosHc probes 3. Database record locking 4. Long sequence regression tesHng 5. FuncHon equivalence tesHng (sample or exhausHve comparison to a
reference funcHon) 6. FuncHonal tesHng in the presence of background load 7. HosHle data stream tesHng 8. Simulate the hardware system under test (compare to actual system) 9. Comparison to self-‐verifying data 10. Comparison to a computaHonal or logical model or some other oracle 11. State-‐transiHon tesHng without a state model (dumb monkeys) 12. State-‐transiHon tesHng using a state model (terminate on failure rather
than a*er achieving some coverage criterion) 13. Random inputs to protocol checkers
www.kaner.com/pdfs/highvolCSTER.pdf hnp://www.associaHonforso*waretesHng.org/wp-‐content/files/Kaner_Hoffman-‐ExploratoryTestAutomaHon.pdf
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
82
For More Information
Keith Stobie
TiVo
hnp://testmuse.wordpress.com/
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
83
References
hnp://blogs.msdn.com/b/anunhara/archive/2011/10/20/exploratory-‐tesHng-‐introducHon.aspx hnp://tesHnginterviewsquesHons.blogspot.com/2013/01/exploratory-‐tesHng.html hnp://waHrmelon.com/2011/06/10/yet-‐another-‐so*ware-‐tesHng-‐pyramid/ hnp://swtester.blogspot.com/2012/05/what-‐is-‐exploratory-‐tesHng.html hnp://www.saHsfice.com/arHcles/what_is_et.shtml hnp://www.goforexam.in/2012/01/what-‐is-‐difference-‐between-‐scripted.html hnp://www.methodsandtools.com/archive/archive.php?id=65 hnp://www.tesHngeducaHon.org/BBST/exploratory/BBSTExploring.pdf
Exploratory Tes+ng
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
84
References
hnp://www.associaHonforso*waretesHng.org/wp-‐content/files/Kaner_Hoffman-‐ExploratoryTestAutomaHon.pdf
kaner.com/pdfs/VISTACONexploratoryTestAutma6on.pdf
www.kaner.com/pdfs/highvolCSTER.pdf
Harry Robinson Exploratory Test AutomaHon slides from CAST 2010
Automated Exploratory
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
85
References
Hendrickson, E. “A Picture’s Worth 1000 Words,” STQE Magazine, September/October 2002
PNSQC 2002
Winant, B. “Modeling PracHce and Requirements,” S6ckyminds.com. Available: hnp://www.sHckyminds.com/se/s3565.asp Harry’s Model-‐Based TesHng website:
hnp://www.geociHes.com/model_based_tesHng/
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
86
Model Based Tes+ng
References
hnp://www.so*waretesHngso*ware.com/manual-‐tesHng/
hnp://www.methodsandtools.com/archive/archive.php?id=94
hnp://qatestlab.com/services/We-‐Are-‐Professionals-‐in/automated-‐tesHng/
Manual vs Automated
20 Ju
ne 2013
Copyrig
ht ©
2013 Ke
ith Stobie, TiVo
87