TDD And A New Paradigm For Hardware Verification · 2017-07-18 · TDD And A New Paradigm For...

© 2011 XtremeEDA USA Corporation - Version 080721.10

TDD And A New Paradigm For Hardware Verification

Neil Johnson XtremeEDA

[email protected] http://www.AgileSoC.com

1


•  Neil Johnson – 13 years of hardware design and verification – Altera, Neterion, Flextronics, Nextwave Wireless

•  Principal Consultant XtremeEDA – Consulting services

•  Verification experts

– Clients are any size and many applications •  Telecom, networking, wireless, computer hardware, etc.

– We work remotely or onsite as part a client’s team

About Me

2


•  Neil Johnson – 12 years of hardware design and verification – Altera, Neterion, Flextronics, Nextwave Wireless

•  Principal Consultant XtremeEDA – Consulting services

•  Verification experts

– Clients are any size and many applications •  Telecom, networking, wireless, computer hardware, etc.

– We work remotely or onsite as part a client’s team

About Me

3

Hardware engineer that’s developing an increasingly unhealthy infatuation with Test-Driven Development


TDD And A New Paradigm For Hardware Verification

4

The Old Paradigm

What’s Wrong With This Picture?

A New Paradigm

“So... how’s that working out?”


•  ASIC – Application Specific Integrated Circuit – Static structure – Digital or mixed signal – High NRE/Low cost

•  FPGA – Field Programmable Gate Array – Reprogrammable structure – Primarily digital – No NRE/High cost

•  SoC – Either of the above + embedded processor(s) + software

What Do I Mean By Hardware

5


•  Typical SoC design flow – Specification – Design – Verification – Physical design – Fabrication – Validation –  Integration

SoC Development Basics – Agile2011

6

Pre-silicon

Documentation

Code

“Stuff”

Chip

Board

System

Production

OS

Drivers

Application

You Guys


•  Typical SoC design flow – Specification – Design – Verification – Physical design – Fabrication

Hardware Development Basics

7

Pre-silicon

Production

Documentation

Code

“Stuff”

Chip


1.  Verify each Block exhaustively in isolation

Hardware Verification in 2 Easy Steps

8

Block A

Block B

Block C

Block D

Block A

Block I

Block H

Block G

Block F

Block E


1.  Verify each Block exhaustively in isolation 2.  Integrate and verify at Top level

Hardware Verification in 2 Easy Steps

9

Block A

Block B

Block C

Block D

Block A

Block I

Block H

Block G

Block F

Block E


•  Simple testbench with lots of “simple” tests – manually created stimulus/manual checking – upside: focus and early results – downside: inadequate coverage

Directed Testing (aka: Old school testing)

10

DUT Generator Monitor

Pro

gres

s

Time


•  Complex testbench with fewer “smart” tests – Random stimulus/model based checking – upside: efficiency and rigor – downside: test harness complexity and long delay in results

Constrained Random Testing (aka: Cutting edge testing)

11

Pro

gres

s

Time

DUT Generator Monitor

Model


•  Directed testing – simple test harness – upside: focus and early results – downside: inadequate coverage

•  Constrained random testing – smarter test harness – upside: efficiency and rigor – downside: test harness complexity and long delay in results

Just to Repeat...

12


Development Timeline

13

Block Level design

Block Level Testbench

Block Level Testing

Top Level Testbench

Top Level Testing

Block Level

Top Level

DONE START


•  Long development cycles – 6-12 months is common is common in block testing – Time for testbench development is increasing

•  Used to take weeks... now it takes months

•  finish the code, then test it •  Unit testing is rare

– smoke testing of design – no testing-of-the-testbench

•  progress is based on code written

Observations From The Old Paradigm

14


What Are We Doing Wrong?

•  Brainstorm... in 5 minutes or less


Big Bang Progress

16

Way Too Much!!

DONE START


Prioritizing Sub-Systems Over Products

17

Block Level design


Top Level Testbench

Block Level

Top Level

Top Level Testing

Block Level Testing

DONE START

No Feedback


Subjective Definition of Done

18

Block Level design


Block Level Testing

Top Level Testbench

Top Level Testing

Block Level

Top Level

START Oops! NOT DONE


Emphasis on Bug Hunting


What Are We Doing Wrong?

•  The Old Paradigm – Big bang progress – Prioritizing block level over product level – Subjective definition of done – Emphasis on Bug hunting


How Do We Change?

•  Brainstorm... in 5 minutes or less


Incremental Progress

DONE

Incremental Progress START


Prioritize The Product

Block Level

Top Level

DONE START

Learn From The Product Much Earlier


Redefined Definition of Done

Block Level

Top Level

DONE START

DO

NE

DO

NE

DO

NE

DO

NE

DO

NE

DO

NE

DO

NE


Bug Prevention with TDD

Block Level

Top Level

Design/Testbench Code

Unit Level

DONE

Unit Tests Design

START


How Do We Change?

•  The New Paradigm –  Incremental progress – Prioritize the top-level (early integration) – Redefinition of the definition of done – Bug prevention with TDD


“So... How’s It Working Out?”

27


New Paradigm Case Study

•  Derived from a mid-project assessment of a recent project – what’s up?

•  project involved development of a new IP block •  I was in charge of verifying the IP block

–  I could... •  Plan incremental progress •  Redefine the definition of done •  Prevent bugs in the testbench with TDD

–  I couldn’t... •  Prioritize top-level •  Prevent bugs in the design


•  Split the entire test effort into a series of steps –  Step 1: Basic sanity of primary behavior –  Step 2: Sanity of first major feature set –  Step 3: Sanity of second major feature set –  Step 4: Sanity of third major feature set –  Step 5: Sanity of combined feature sets –  Step 6: Functional coverage of combined feature sets –  Step 7: Verify misc features –  Step 8: Clean-up/release

•  Incremental implementation of the testbench to support each step –  testbench never includes more than the minimum required –  choose techniques that suit each step

•  directed tests to start •  random tests later


29



30


Redefine The Definition Of Done

31


Redefine The Definition Of Done

32


•  Goal: – a bug free test harness

•  How: 1.  build component features using TDD 2.  integrate component features into the test harness as

they’re completed 3.  use the features against the design as soon as they’re

integrated 4.  repeat 1-3 until done

Bug Prevention With TDD


SVUnit Testing Framework

34

Testrunner

<ID_a>_testsuite <ID_n>_testsuite

<ID_x>_unit_test <ID_y>_unit_test

test_<a>() test_<b>()

Test Runner Object

Test Suite Objects

Unit Test Objects

Test Methods

Perl scripts and Makefiles

write code here


SVUnit In Practice - Mechanics

35

Write unit test for new test harness

feature

Ensure the new unit test passes

Ensure the new unit test fails

Write model code for new feature

START

Verify the Testbench

(TDD w/SVUnit)


SVUnit In Practice - Mechanics

36

Write unit test for new test harness

feature

Ensure the new unit test passes

Write test

Ensure the new unit test fails

Write model code for new feature

Run test

File BUG

START END

Verify the Testbench

(TDD w/SVUnit)

Verify the Design

(Directed Test)


•  Case Study Test harness –  3129 lines of code –  5222 lines of test code –  116 unit tests total –  41 tests against the design –  Found 30 design bugs –  Found 2 test harness bugs –  9 weeks of effort

SVUnit In Practice – Client Case Study

37

Case Study Model 740 lines of code

2801 lines of test code 62 unit tests

35 tests against the design 19 design bugs

1 model bug 5 weeks effort


New Paradigm Case Study – Summary

•  Incremental “step-by-step” milestones make sense –  fix the scope of the test harness to the goals of each step

•  Easy to convey progress with real results – very little ambiguity

•  Good code is easier to work with than garbage code – TDD is great for building a test harness – TDD is great for building reference models – TDD helped keep my bug rate very low


Summary

39

The Old Paradigm


A New Paradigm


?


Summary

40

The New Paradigm


A New Paradigm



Summary

41

For More Information: •  [email protected] •  @nosnhojn •  www.AgileSoC.com •  http://sourceforge.net/projects/svunit

TDD And A New Paradigm For Hardware Verification · 2017-07-18 · TDD And A New Paradigm For...

Documents

Transcript of TDD And A New Paradigm For Hardware Verification · 2017-07-18 · TDD And A New Paradigm For...