Automated Visual Testing - What, When & How
40
Adam Carmi (@carmiadam) Co-Founder and VP R&D at Applitools [email protected] AUTOMATED VISUAL TESTING WHAT, WHEN & HOW
-
Upload
adamcarmi -
Category
Technology
-
view
142 -
download
2
Transcript of Automated Visual Testing - What, When & How
Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools
AUTOMATED VISUAL TESTING
WHAT, WHEN & HOW
AGENDA
Why automated visual testing?Tools & TechnologyWhere does it fit?Q & A
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a
Graphical User Interface appears correctly to users
A VISUAL BUG
AND ANOTHER…
AND YET ANOTHER (IE 8 NOT OK)…
AND YET ANOTHER (IE 11 OK)…
AND THE LAST ONE…
WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY
Web browsers Devices Operating systems Screen resolutions Responsive designs L10n
WHY SHOULD IT BE AUTOMATED?
WHY SHOULD IT BE AUTOMATED?
NATIVE / HYBRID MOBILE APPS
Harder to roll back changesCan’t push dailyUpdates take battery and data
Higher quality bar
MANUALVISUALTESTINGISERRORPRONE
WHY SHOULD IT BE AUTOMATED?
AGENDA
Why automated visual testing?Tools & TechnologyWhere does it fit?
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
DRIVING AND CAPTURING
OTHER• Wraith (URLs)• PhantomCSS (CJS)• dpdxt (URLs)• Grunt PhotoBox
(URLs)• Snap And Compare
(URLs)• Specter (JS)• Gemini (JS)• Grunt-Vigo (URLs)• BackstopJS (URLs)• Applitools Eyes (All)
• WebDriverCSS (JS)• Fighting Layout Bugs
(Java)• Wraith-Selenium
(Ruby)• Selenium Visual Diff
(Java)• Needle (Python)• VisualCeption (PHP)• Huxley (RP)• Gemini (JS)• Vizregress (.NET)• Visualci (Java)• Viff (JS)• Pix-Diff (JS)• Applitools Eyes (All)
• CSS Critic (URLs)• FBSnapshotTestC
ase (XCTest)• Kobold (folders)• Applitools Eyes
(All)
DRIVINGAND CAPTURINGWITHSELENIUM
(WebDriverCSS)
MANUALLY WITH EYES EXPRESS
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
WHY NOT?
FALSE POSITIVES
ANTI ALIASING 1/2
ANTI ALIASING 2/2
IMAGE SCALING 1/2
IMAGE SCALING 2/2
AND MORE…1 pixel offsets in element positioning
Dynamic contentMoving elementsImages of different sizePerformance
DEMO
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
REPORT DIFFERENCESAs files on the file system (combined with source control)
REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
UPDATE THE BASELINE
Rename or commit individual image files
GUI (Gemini GUI)
UPDATE THE BASELINE
Overwrite mode
Automatic maintenance (Applitools Eyes)
AGENDA
Why automated visual testing?Tools & TechnologyWhere does it fit?
WHERE DOES IT FIT?
• Components
• Code review
• Developers
• Designers• QA
Visual testing of frontend components by frontend developers
WHERE DOES IT FIT?
• Pages• Page
sections
• Developers
• Designers• QA• Others
Full or partial validation of application screensA collaboration tool
WHERE DOES IT FIT?
• Staging vs. Production
• Ops• QA
Validate your staging deployment using your production deployment as a baseline
WHERE DOES IT FIT?
• Monitoring
• Ops• QA
No missing resources in productionNo corruption due to 3rd party dataNo corruption due to browser / OS upgrades
QUESTIONS?
Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools
THANK YOU
