Designing Software For Testability: A forgotten design pattern
-
Upload
rohitnayak -
Category
Technology
-
view
3.534 -
download
1
description
Transcript of Designing Software For Testability: A forgotten design pattern
![Page 1: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/1.jpg)
Designing Software For Testability
A forgotten design pattern
Rohit Nayak
Talentica
![Page 2: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/2.jpg)
Agenda Background and Motivation DFT in VLSI and hardware design Issues in Software Testing DFT for Software Development Automation and Testability Some Examples Cons, Review
![Page 3: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/3.jpg)
The Anti-Pattern
If it works, its the developer, if not blame QA!
![Page 4: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/4.jpg)
VLSI/PCB Testing Issues Mass produced Each piece needs to be validated Complexity
Number of subsystems Amount of logic
Access to internal logic Testing costs Cost of recall
![Page 5: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/5.jpg)
![Page 6: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/6.jpg)
Testability: “Effort required to test a product to
ensure that it performs its intended
function”
![Page 7: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/7.jpg)
DFT Principles Controllability Observability
Test Point Insertion Built-In Self Test (BIST) Fault Modelling Fault Simulation Test Pattern Generation
![Page 8: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/8.jpg)
![Page 9: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/9.jpg)
![Page 10: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/10.jpg)
Software Testing“Testability is a design issue and needs to
be addressed with the design of the rest of the system. Projects that defer testing to the later stages of a project will find their programmers unwilling to consider testability changes by the time testers actually roll onto the project and start making suggestions. Testability takes cooperation, appreciation and a team commitment to reliability.”
- Bret Pettichord
![Page 11: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/11.jpg)
Cost Of Bugs
Where Found
Where Introduced
Requirements Architecture Coding System Test Production
Requirements 1x 3x 5-10x 10x 10x-100x
Architecture 1x 10x 15x 25-100x
Coding 1x 10x 10-25x
![Page 12: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/12.jpg)
![Page 13: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/13.jpg)
Software Testability Suitability: clarity of specifications QA has Observability: we can only test what is visible Simplicity: easier design/UI makes testing easier Controllability: better the control, better the coverage
and automation Stability: how often it changes Performance: how fast it works Diagnosability: writing effective bug reports!
![Page 14: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/14.jpg)
Software DFT Patterns BIST
Automation suite w/ oracle
Unit Tests Assertions
Fault Simulation Mock/Stub Invalid params
Controllability Decoupling Bypassing Mock/Stub Set/Reset
Observability Logging Reporting Test Interfaces
![Page 15: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/15.jpg)
Good Automated Tests are
Repeatable Easy to Write Easy to Understand Fast Way Easier with a Testable Software
![Page 16: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/16.jpg)
When/How to Automate? Manager & Team are commited Testers/Dev either experienced or interested
in learning to script Product release cycles managed well Functionality / UI changes under control Early/Incremental approaches work best Build integration, reporting Each bug results in a automated test
![Page 17: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/17.jpg)
Automation As Coding Automation scripts and frameworks are CODE! Use tools and scripting languages Evaluate tools on real problems Follow development processes
Spiral/Agile
Use Source Control same as rest of code These will have bugs as well!
![Page 18: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/18.jpg)
Test “oracle” Expected Result
Generated once from previous run or Manually specified or Legacy system
Time adjustment Results
Database File Inline with code
![Page 19: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/19.jpg)
Test Environment Multiple VMs Automated installs/images Automation tools Bug reporting tools
![Page 20: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/20.jpg)
Unit Tests Each module has independent set Developer written/maintained setup, teardown xUnit, TestNG Runs on build Can run sub-set on install TDD
![Page 21: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/21.jpg)
Diagnosability All environments: Test / Staging / Production
Errors result in visual messages Errors are raised where they occur Errors can be localized Details are sufficient to fix issue Ability to send diag data to dev
![Page 22: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/22.jpg)
Logging Module Log levels (VERBOSE, INFO, WARNING, ERROR, CRITICAL, TIME) Define CRITICAL/ERROR levels well and use them! These should result in urgent notifications Lower levels in Production Time, App, Component, ThreadId, Message
8/2/2010 14:22pm: Notifier: Pop3: 8242: Sending welcome email to userid 334
Delimited columns for import Rotation based on Time, Size
![Page 23: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/23.jpg)
Built-in Self Test Inserting test code/interfaces Set/Reset to bring state to known value Reporting to get current state Assertions about values/state
assert(order.billed==true)
“oracle” based regression
![Page 24: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/24.jpg)
Some Examples Web Application
Software As A Service Browser Only UI Ajax / Dynamic HTML
Banking (Client-Server) Installed Application Desktop Client Server API
![Page 25: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/25.jpg)
Web Application HTML Page Title Id values for important divs/controls Hidden values (non-textual, graphs, tables) Measurable Ajax responses Tools: Selenium, Sahi, Watir, WebTest, curl Logging incl. browser/ip/session cookie Ability to simulate time zone, language
![Page 26: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/26.jpg)
Web Application - 2 Bypassing Captcha Mock TP APIs eg. Facebook, Google, OpenId SSL bypassing Transactions to be voided Multiple runs Always initialise controls
Browser autofill
![Page 27: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/27.jpg)
Desktop Unique identifiers to GUI controls Key Replay tools, SendKeys API test suites bypassing UI Client logs, ability to email Ability to run db queries in scripts
![Page 28: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/28.jpg)
Desktop - 2 Automation Friendly Third Party Controls
Ability to select cell Copy-enable text fields Key shortcuts to Forms UI for control focus,
clicking, navigation Access by value (tree/list controls)
OCR for images/text consoles
![Page 29: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/29.jpg)
Bypassing Credit card payment
Test cards Mock object Dev auto-approval bypass code
Order placement Dummy users, auto-fulfill Dummy vendors (email order) Admin screens move
![Page 30: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/30.jpg)
Dates Never use system date directly Config override
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<add key=“SystemDate" value=“$now"\>
</appSettings>
</configuration>
Nant: XMLPoke<xmlpoke file="App.config"
xpath="/configuration/appSettings/add[@key = 'SystemDate']/@value"
value="1-1-2009" />
![Page 31: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/31.jpg)
Sending Emails Email subsystem should be a stub in dev Bypass SMTP: log to file … Email should be sent only to a whitelist Use Gmail Id & Atom API Use Unique Id/Timestamp to distinguish
![Page 32: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/32.jpg)
Mocking / Stubbing Replace module during testing Different kinds:
Sub-system (Email, Print) Webservice (Credit Card, Flight Booking) Hardware device (Biometric, CNC Machine)
Return default values, implement simple/fixed logic
![Page 33: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/33.jpg)
Mocks - 2
Simulate Errors and Exceptions Provide more logging Jmock, Dependency Injection Con:
Code level Needs to change with API
![Page 34: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/34.jpg)
Anti-Testability Viewpoints Security compromise by testability interfaces,
logging
Remove or lock down in prod Extra coding time
Much lower testing costs, better quality Privacy issues in logging
Show only partial data Performance
Almost never an issue in practice!
![Page 35: Designing Software For Testability: A forgotten design pattern](https://reader033.fdocuments.net/reader033/viewer/2022061218/54b778914a7959e6038b45f5/html5/thumbnails/35.jpg)
Review BIST
Automation suite w/ oracle
Unit Tests Assertions
Fault Simulation Mock/Stub Invalid params
Controllability Decoupling Bypassing Mock/Stub Set/Reset
Observability Logging Reporting