Test Automation – Solving Common Challenges Through The Optical Layer
-
Upload
alliance-global-services -
Category
Technology
-
view
159 -
download
3
description
Transcript of Test Automation – Solving Common Challenges Through The Optical Layer
Test Automation:
Solving Common Challenges Through
“Black Box” Testing
A Primer on Using Image-Based Testing and
Device Control
July 8th, 2014
2 ©Alliance Global Services 2014
Today’s Presenter
As Director of Solution Engineering, Joshua Berry has 20 years
of industry experience, with more than a decade in leadership
and innovation. He is a leader within Alliance’s Solution
Engineering organization in developing cutting edge engineered
solutions. His focus includes test automation and tooling,
problem solving, product testing, and agile techniques in
software development.
Today’s Host
As Vice President, Sales, Bob Donaldson is responsible for
driving and overseeing Alliance’s sales and business
development activities while focusing on client value and client
service that translates into long-term partnerships and revenue
growth. Bob has over 20 years of experience in sales
management and operations. Bob holds a B.A. from
Elizabethtown College, Pa.
3 ©Alliance Global Services 2014
Overview
• Background
• Tool Agnostic Presentation
• Assumes Automation Exposure
• Agenda
• Problem Statement
• Black Box Testing Control and Feedback
• Image Analysis Best Practices & Tricks
• Solutioning Black Box Automation
• Real World Solutions
• Integrating IBT and Object-Based Testing
4 ©Alliance Global Services 2014
The “Textbook” Definition
What is Software Test Automation?
In other words,
Test Automation is using software to test software. One part test engineering + one part software engineering.
Software Test Automation is the application of computer software and electro-mechanical equipment to:
• Dynamically Control the execution of tests, • Compare actual outcomes to predicted outcomes, • Set up test preconditions and cleanup test modifications, • Perform other test control and test reporting functions.
5 ©Alliance Global Services 2014
Problem Statement
• Traditional Automation uses application Objects for feedback and
control.
• There are many cases where this does not apply:
– Device Automation
– The test is of video or image content
– Control’s object cannot be accessed
6 ©Alliance Global Services 2014
• All applications have physical input and output
• Primary User Interface
• “Black box” Automation interacts as a user would
– Image-Based Automation
• Reference Bitmaps and Optical Character Recognition (OCR)
• Real-time analysis
– A software or physical input harness
• Touch the application
• Real command format
Potential Solution: Black Box Automation
Input HarnessVideo Analysis Card
7 ©Alliance Global Services 2014
Object vs Image Based Testing
Object Based Testing (OBT)
• Relies on “under the hood” access to application controls
• Can quickly interact with an object
• Object can be hidden
• Can get highly detailed information about the object
• Can be highly dynamic using descriptive programming
• Some technologies, e.g. Flash, Silverlight difficult to automate
• Poor at testing “look and feel”
• Relies on good programming
• Tool or agent must be loaded on the System Under Test.
Image Based Testing (IBT)
• Relies on testing the graphical output as displayed on the screen
• Based on X,Y Coordinate system, Origin usually at upper left corner
• OCR analysis for text analysis
• Can enable cross-platform, touch-screen application testing
• Can be remotely installed
• Bitmap comparison takes time
• Fragile to slight UI changes
• Must capture images ahead of time
• Not recommended for enterprise or when object available
8 ©Alliance Global Services 2014
First Generation Image Analysis
• Analyze every pixel until failure
• Configurable tolerances for RGB variance
• Set the % match for pass / fail
Primitive Pixel Comparison
• Have to teach tool the font
• Unreliable / expensive OCR engines Primitive OCR
• Highly sensitive to changes in images
• Sensitive to video cards and screen settings
• Complex logic to build in basic robustness, such as location variance
Problems with First gen IBT:
9 ©Alliance Global Services 2014
Second Generation Image Analysis
• Treats image holistically
• Resolves images into primitives
• Can accommodate Size / Zoom, Rotation, Tilt / Skew, Partially Covered Images
• Facial Recognition and Pattern Template Analysis
Fuzzy Logic Image Analysis
• Reliable OCR Engines
• Open source near as reliable as commercial Advanced OCR
• Image still has to be visible
• Still sensitive to changes in base OS
• Greater chance of a false positive
Problems with Second Gen:
10 ©Alliance Global Services 2014
IMAGE ANALYSIS BEST PRACTICES
AND TRICKS
11 ©Alliance Global Services 2014
Back to Basics A Good Framework and Engineering Design
• All Automation Best Practices still apply
– Error and Exception Handling
– Logging and Reporting
– Scripting
– Test Data Management
• Temptation of Capture-Replay and Object-Action scripting
– Robust framework of reusable, modular test verbs to handle complex actions
– Don’t hard code navigation
– IBT is just another way of feedback
• Can still have common framework features
– Scriptless Automation
– Continuous Integration
– TCM Management Tools
12 ©Alliance Global Services 2014
Setup and Other General Guidelines
• Rule of Thumb: If it is hard for you to see it, it is hard for
the tool to see it.
• Set application desktop backgrounds to a standardized,
light color to provide contrast
• Standardize automation PC’s and device under test to a
standard video setting
– Best Resolution available
– Color Palette
• Avoid regions of the screen that change:
– Clocks
– Video
– Weather plug-ins
– Ads
• Be careful of “bleed through” of images near dynamic
sections of screen
13 ©Alliance Global Services 2014
Design a Framework that “Knows” Your Application
• Create a virtual map of the screen in your framework.
• “Know” where objects might be
• Limit verifications and actions to proper region
Logo Quick Navigate Live Video
Listings
Channel List
Time
14 ©Alliance Global Services 2014
Reference Images • Smallest Image that still has enough information
– Don’t always need the full image
– Example initial image: “Hello Kitty”
– Optimized reference image might look like:
– Is this reference image good enough to ensure Hello Kitty and only Hello Kitty is found?
– “Hello Stachie!!”
• Make use of Masking
– Most tools allow you to tell it to ignore regions
– A way of optimizing the bitmap
– Also a way of reusing a bitmap
• Pick images with high contrast
• Choose meaningful bitmap names
• Take variations of same image per OS
Masked Region
15 ©Alliance Global Services 2014
• Minimize the amount of bitmap / OCR verifications
• Avoid verifying images in exact X,Y Coordinates
• Search in a reasonable sized region
• Use OCR for text validation
• Avoid OCR over opaque / variable backgrounds
– OCR Engines need strong contrast between letter and background
– Text over video
Search Region Buffer
Exact X,Y Position
Searching for Bitmaps and Text
16 ©Alliance Global Services 2014
Use of Highlights
• Many Applications make use of a highlight, as shown below:
• For validation and navigation routines, your framework needs
to know where the highlight is.
– Map out where the buttons could be
– Create a bitmap of just the highlight with the inside masked.
– For variable width highlights, mask the ends.
– Use the tool to locate bitmap in icon region.
– To validate the highlight:
• Compare X,Y coordinates to the map,
• Dynamically determine what is at that position.
17 ©Alliance Global Services 2014
On-Screen Navigation in an IBT World
1. Determine X,Y Coordinate of Target Button.
2. Determine X,Y Coordinate of Highlight
3. Calculate Delta X,Y Navigation Vector from Highlight to Target
4. Move Up / Down / Left / Right appropriately
5. Recheck X,Y Coordinate of highlight
6. If within tolerance, Press <Enter>, else Step 3
18 ©Alliance Global Services 2014
Cross-Screen Navigation in an IBT world
• Develop Cross-Screen Navigation Routes
– Use Datums and Waypoints
– Map how to navigate to all screens from a common datum
– Find common screen between current and target screen
– <Back> to common screen, then follow Navigation to target screen
• Screen Identification
– Identify properties that optimally identify each screen
– Create an IdentifyScreen() function
• Example Navigation Map
– Language Setup: Menu, Setup, Language
– Video Setup: Menu, Setup, Video.
– Search: Menu, Search
19 ©Alliance Global Services 2014
SOLUTIONING BLACK-BOX AUTOMATION
20 ©Alliance Global Services 2014
• Direct Connection
• IP Stream
• VNC / RDP
• Webcam
• OBT / Feedback Agent
• Debug Port / Logs
• Open Loop Record
Image Capturing / Feedback Architectures
Test Automation PC
Automation Software
Device Under Test
App Under Test
App Under Test
Video Output
VNC Client
VNC Server
Video Output
OBT Feedback
Agent
Debug Data
21 ©Alliance Global Services 2014
• OBT
• Coordinate Click / Keys
• VNC / RDP
• Data / Diagnostic Port
• Device Agent / Instrumented Code
• Physical Test Harness – IR / RF Signals
– Robotics / Servos
– Interrupting Native I/O
– Voice Control
Application / Device Control Architectures
Test Automation PC
Automation Software
Device Under Test
App Under Test
App Under Test
VNC Client
VNC Server
Diagnostic PortControl Agent
Physical Test Harness
Clicks and Keys
22 ©Alliance Global Services 2014
• The means of control affects the test validity and accuracy
• It is up to the engineer to determine which type of architecture to use
• Real-world defects
– Key Queue
– Touch Screen
The Impact of the Solution
Input Harness
VNC / RDP
OBT
Debug Control
Physical Interrupt
23 ©Alliance Global Services 2014
Embedded Devices
• Software running in devices that can’t run a standard test tool Description
• Mobile, Cable STB, Game Consoles
• Video Conference, POS Terminals, Delivery Manifests
• Automotive Telematics / Sat-Nav Systems Examples
• Video Output
• VNC / RDP
• Webcam
• OBT Harness
Feedback
• Physical Hardware
• VNC / RDP
• Diagnostic Port
• OBT Harness
Control
24 ©Alliance Global Services 2014
Mobile and Touchscreen
• Specialized subset of Embedded Devices Description
• Jail breaking and instrumenting code often not allowed
• Getting a standard tool across Android / iOS / Win / Blackberry
• iOS5, iOS6 allows some VNC apps, iOS7 does not.
• iOS8 Health Check breaking “sandbox” paradigm.
• Standardizing tooling and testing with Web versions of the same application.
• Automating phone Interrupts and Interoperability
• A text or call interrupt
• Loss / Gain of WiFi
• Screen orientation change
Challenge(s)
• OBT Harnesses
• Some Tests manually, e.g. orientation, iOS7
• Harness on WiFi Router
• 3PA Text and Fake Text / Call Apps
Solutions
25 ©Alliance Global Services 2014
Video / Image Validation
• Testing the content of a picture or video independent of the platform application resides. Description
• Correct Image: User Authentication, UI Validation
• Streaming Video: eLearning, VOD, Video Ads
• Video / Image Quality: Streaming, Video Conference Examples
• Canned Pictures and Video Assets
• Short Loop for Video compare to partial still
• Canned High Bandwidth Videos
• “Moving Icon”
• Checksum validation
Successful Techniques
26 ©Alliance Global Services 2014
Custom Controls
• 3PA / non-standard controls OBT tools cannot recognize. Description:
• Not all parts of control visible
• Have to hunt for specific control
• Mapping can be difficult Challenges:
• Brute force click on parts of control and hope the hidden component appears
• Complex algorithm to determine region to search Solutions:
27 ©Alliance Global Services 2014
Some Current Tools Tool License Architecture Notes
QTP / UFT Commercial OBT tool with limited IBT added in 11.50 version, VNC to remote
Native mix of OBT and IBT
EggPlant Commercial VNC connection to remote machines
Pure IBT tool.
Ranorex Commercial Direct Connect IBT and OBT
Test Quest Commercial Direct Connect IBT
Sikuli Open-Source Work locally or through VNC IBT
Open.CV / EMGU Libraries
Open-Source C++ / C# image processing libraries
Powerful library to integrate into a tool
Robotium Open-Source Android OBT, Instrument
Frank Open-Source iOS OBT, Instrument
JAMO Commercial Mobile OBT
SeeTest Commercial Mobile OBT
Perfeco Mobile Commercial Cloud-hosted, Physical device input interrupt.
Integrates with UFT
28 ©Alliance Global Services 2014
SOLUTIONS FROM REAL-WORLD PROJECTS
29 ©Alliance Global Services 2014
Cable Set Top Boxes
• Control:
– PC-controlled IR Emitter taped to
front
– Key commands through diagnostics
– Power through PC control
• Feedback
– Diagnostic Port
– Real-Time Video Analysis
– Connected to TV’s for human
monitoring of test
• Both direct connection and IP-based
• Single STB and racks of 32
30 ©Alliance Global Services 2014
Robotic Tablet Automation
• Low cost high school robotics kit
• Webcam for feedback, Bitmap / OCR
• Custom control software: C#, EMGU
• CHALLENGES:
– Calibrating Webcam to tablet
– Virtual Coordinates of Webcam to Real Coordinates of Robot.
– Pressure / speed of stylus
• APPLICATIONS: Automotive Telematics, Point of Sale, Laptops
31 ©Alliance Global Services 2014
Visual Authentication
• User Security Image Problem Statement:
– Incorporate “Security Image” check into existing OBT automation test
bed
• Solution
– Create security images for each user in test environment
– Create meta data linking reference bitmaps to specific users
– Integrated open source comparison tool to framework
– VerifySecurityImage(UserName) to lookup reference image and
compare to screen.
• Video Asset Validation Problem Statement
– Ensure the correct video asset was streaming
• Solution
– Reduce assets to 15 – 20 second loops.
– Capture still at time = 5 seconds
– Crop reference bitmap and validate
32 ©Alliance Global Services 2014
Native Applications on Mobile and POS
• UFT 11.5x + SeeTest for OBT within mobile device
• Selenium + Siluki + VNC for IBT
• UFT 11.5x + VNC for IBT
33 ©Alliance Global Services 2014
Concluding Thoughts: Mixing IBT and OBT
• Image-Based Testing and Black Box Control are powerful and can solve many automation challenges.
– Object testers will have to adjust to changes such as Navigation
– Good Engineering still applies
– Use judgment with how you control the application
• A good automation engineer uses every means of feedback possible to suit to application and the test.
– There may be cases where one can mix IBT and OBT
– Image based feedback might only be 5% or 95%.
– Use IBT to augment OBT
– Look into tools like JAMO and SeeTest for mobile and devices
• Consider tools like UFT and Ranorex that can do IBT and OBT or integrate Image Analysis tools / libraries (e.g. Sikuli, EMGU) into OBT only tools (e.g. Selenium).
34 ©Alliance Global Services 2014
Questions?
Thank You
Joshua Berry
Director of Solution Engineering
www.allianceglobalservices.com