Automated Exploratory Tests
-
Upload
zbyszek-mockun -
Category
Technology
-
view
1.821 -
download
0
description
Transcript of Automated Exploratory Tests
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Automated Exploratory Tests for Web Application
Zbyszek Moćkun
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
How Cognifide has developed automated
tests and where we are going
Objective
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
In short about process and environment
Introduction
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Methodologies, projects …
• Agile: SCRUM, Kanban
• Frequently releases
• No time for manual
regression
• Lot of work at the end of
sprint
• CI Environment
• Web application
• Migration
• Content Management
Systems (CMS)
• Short or medium term
projects
• Automation investment, no
time for money return
Methodologies Projects
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Functional tests
Common standards for automation
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Test Automation Lifecycle
Run
Manage
Update
Write automated
script
Write TC
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Advantages and disadvantages
• Coverage by
automated tests
• No duplications during
regression test
• Functionality is tested
• Time consuming
• Cost
• Invest and wait for return
• Test dependency
• Only coded functionality is
tested
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Can you believe your tests?
• Incorrect Xpath (locator) used in tests
• Not all data were verified (unable to write verifications for all
data)
• CSS/images or other files wasn’t load
• Frequently design changes require xpath (locator) update
• Incorrect CSS fixes (designs are not reflected by page
layout)
• Analytics doesn’t work
• Performance decrease
• Few thousands of pages (components variation – can’t test
all of them
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Migration (upgrade) projects
• No existing automated test for regression test
• Few thousands of pages
• Quick response needed
• Iterative development
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
New approach, theory part
Automated Exploratory Tests
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
We need automated tests
because ...
Source: http://www.testfacebook.com
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
What should we test?
Two perspectives:
User Client
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Aspects
CI Environment
Client
Analytics
User
Layout Functionality Performance
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Looking for a symptom
„Symptom is a departure from normal function or feeling
which is noticed by a patient, indicating the presence of
disease or abnormality. A symptom is subjective,
observed by the patient, and not measured.”
http://wikipedia.org
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Symptoms
Bug
Layout
Content
HTML code
JavaScript errors
Server response
(404)
Analytics
Validators (W3C)
Link checker
URL strategy
Performance
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Oracle Heuristics
It is used by comparing the output(s) of the system under test,
for a given test case input, to the outputs that the oracle
determines that product should have.
− other products (second program that uses a different algorithm to
evaluate the same mathematical expression as the product
under test)
− a consistency oracle that compares the results of one test
execution to another for similarity,
− a model-based oracle that uses the same model to generate and
verify system behavior,
− or a human being's judgment (i.e. does the program "seem" to
the user to do the correct thing?)
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Oracle Heuristic for GUI
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Reference symptoms
• Previous version of application
• Reference data saved on disc
• Other version of the same application (different servers)
• The same script takes pattern and compare
• One tool for different projects
Layout Content HTML code Performance monitoring
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Robots
• Scripts that are looking for symptoms
• The same script for all projects
• Open Source tools
JavaScript errors
Server response (404)
Validators (HTLM, CSS)
Analytics Link checker URL Strategy
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Exploratory testing
„Exploratory testing is an approach to software testing
that is concisely described as simultaneous learning, test
design, test execution and test result interpretation”
Cem Kaner
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
What are Automated Exploratory
Tests?
Meny different definition:
• Record and play manual exploratory test sessions
• Combined manual and automated tests
• Test steps automation
• Blackbox tests with input/output data generator
• Not at all possible due to human factor
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
What are Automated Exploratory
Tests?
„Tests supporting exploratory testing, not necessarily based on
the test scenario, searching areas where there may be errors
or a change in the application, allowing the examination of
applications faster and more deeply or not to test sites
available to man.”
Zbyszek Moćkun
„Automated Tests do not provide complete information. They
provide partial information that might be useful”
Cem Kaner
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Automated Exploratory Tests -
advantages
• One tool
− Gather all symptoms
− Use in all projects
− Quick to introduce to project (half a day)
− Resistant to appliciaton changes
• Do not require deep knowledge about application,
support the learning process
• Integrated with Continuous Intergration env
• Strong solution if combined with automated
functional tests
• Migration or upgrade tests
• Iterative development
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Track the propagation of
changes in the code - See what pages/components are infected by change
- Positive – false (code change affect all necessary places?)
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Practical part
Automated Exploratory Tests
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Report – example of use
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
URL management
• Sitemap
• Link crawler
• CSV file with url list
• Symptoms as parameter (on, off)
• Comments (bug id, ...)
• Hide element
• Other actions
• Keep in SVN
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Pattern management
• Easy to use (overwrite pattern)
• Checkbox (check all)
• Commit to svn
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Dynamic content
• Reference symptoms – need processing
• Use regexp
• Hide element
• Write generic function
• Robots – do not need any additional actions
• Dynamic components (counters, date, random numbers)
• Flash/video files
• Embedded content (twitter, facebook,...)
• Require functional test
• Static components – about 75%
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Mobile version
• Required (not nice to have), can’t avoid testing
• Mobile pages are more static than desktop version
• Do not use flash
• Not all symptoms are easy to check
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Narzędzia
• Selenium II/WebDriver
• Komparatory (ImageMagick, Pretty Diff)
• Walidatory (W3C)
• Performance (JMeter, Proxy)
• Serwery Proxy (BrowserMob)
• Firefox/Firebug (NetExport, ConsoleExport, JS Console
output redirector)
• HAR Viewer (http://www.softwareishard.com/har/viewer/)
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.
Q&A