Test and Verification Solutions 128 October 2009

Test and Verification Solutions

Improved time to market through automated

software testing

Mike Bartley, TVS

2Test and Verification Solutions 228 October 2009

Agenda

• The potential benefits & drawbacks of automation• Compare with HW verif• Can automation reduce TTM?• Automating static & dynamic testing• Deciding what to automate• Choosing your automation tools• Case study

3Test and Verification Solutions 328 October 2009

The potential benefits of automation

• Reduced costs• Improved quality• Improved employee motivation• Improved TTM

4Test and Verification Solutions 428 October 2009

The potential drawbacks of automation

• Shelf-ware!• Maintenance• Skills base• Missing failures

5Test and Verification Solutions 528 October 2009

HW verification The mechanics of complex test bench

Driver Stimulusgenerator

Test ScoreboardFunctionalCoverage

Design

Under

Test

Monitor

assert

constraint

addr data

Assertions

Coverage

Coverage

6Test and Verification Solutions 628 October 2009

The mechanics of finding a bug

Design Under Test

PropagateStimulate

…..01010101

…..01001101

…..10011010

…..01001101

01100101…..

11110101…..

00010101…..

Actual

Results

Expected

Results

Compare

Observe

7Test and Verification Solutions 728 October 2009

Can automation reduce TTM?

• Getting test results sooner• Catching bugs earlier• Stabilising the software sooner• Assessing the stability of the software

8Test and Verification Solutions 828 October 2009

Automating static testing

• Types of bug found• A quick win• Beware false failures• Introduce filters slowly• Developers not testers (check in checks, IDE)

• No testing of intent

9Test and Verification Solutions 928 October 2009

Automating dynamic testing

• Test plan• Test design specification• Test case specification• Test procedure specification• Test execution• Test log• Test incident report• Test summary report

10Test and Verification Solutions 1028 October 2009

Deciding what to automate

• Complexity of the test environment• The test level • Frequency of execution• Ease of automating pass/fail• Test stability and repeatability

11Test and Verification Solutions 1128 October 2009

Choosing your automation tools

• Who writes test test?• Matching skill set• Separate orthogonal concerns• Portability• Ease of adding/banning tests• Integration into configuration/build system• Cost of the tool?

12Test and Verification Solutions 1228 October 2009

Case Study – Background

• Includes chips, accelerator cards, and software

• All work with industry-standard x86-based systems

13Test and Verification Solutions 1328 October 2009

Case Study – Background

• Low level host driver code– Code on host computer communicates with the chip on the board in

the PCI slot

• Libraries of code– Runtime library– Mathematics acceleration libraries (ACML, MKL, …)

• Application code– Linpack, Montecarlo, ….

• Code running on the chip– This had parallel C extensions not supported by our chosen tool

• Software models of the hardware• In-house compiler for the code running on the chip• Mainly C and C++, some assembler

14Test and Verification Solutions 1428 October 2009

Case study – automation details

• Integrated into configuration/build system• Automated mostly lower level tests• Developed internally by SW Engineers• Offshored manual tests• Offshored some automated test development• Integration of static analysis• Test maintenance

15Test and Verification Solutions 1528 October 2009

Case study - results of automation

• Improved quality• Shortened release schedules• Release schedules hit• Reduced TTM• ROI?

16Test and Verification Solutions 1628 October 2009

Questions?

• Mike Bartley

• mike@tandvsolns.co.uk

• LinkedIn

• 07796 307958