Assessment studies – FPGA Verification Tool

www.bitvis.no Your partner for Embedded SW and FPGA

Assessment studies –FPGA Verification Tool

ESTEC, Noordwijk, 4 December 2019TEC-ED & TEC-SW Final Presentation Days - December 2019

www.bitvis.no

Bitvis AS

ESA - Final Presentation Days Dec. 20192

Independent Design Centre for SW & FPGA/ASIC

23 designers (Embedded SW: 9, FPGA: 14)

Specification, Design, Implementation, Verification, Test

Methodology partner

Sparring and review partner

Verification IP provider (UVVM is used world-wide)

Course provider for FPGA Design and Verification

Merged with Inventas September 2019Part of CGI

The leading independent design centrein our field in Scandinavia

-------

-------

www.bitvis.no

What on earth is an FPGA?

FPGA

FPGA – the first impression An FPGA is like a gigantic, mostly digital PCB

• With uCtr, DSP, Memories, “Standard Comps.”, Filters, Peripherals, Glue logic, Customized modules, PHYs, …

• Very fast (high frequency) - as a complete system– due to internal connections

• Very flexible – as FPGAs are re-programmable in system

PCB: CPU/DSP/GPU

Memory

Peripherals

ADC conv.

PLL

Ethernet

USB

PCIe

etc...Anything may be switched or added


www.bitvis.no

Obvious benefits (cf PCB)


Compact system

High performance

Excellent MTBF

Extremely flexible

More deterministic than SW

Often lower power consumption

and LCC than other HW solutionsFPGA

www.bitvis.no

FPGA in embedded system


www.bitvis.no

One or more FPGAs in most advanced electronic systems

Testing the FPGA design


Tests FPGA

Must be properly tested– verified (simulated)

Project man hours

Verification Design Other

Quality: Late bug detections, Product faults, Break down & Loss, Safety

Verification Infrastructure

www.bitvis.no

The 2018 Wilson Research Group - Recap Functional Verification Study (1)

Half the project time is spent on verification


Typical space

www.bitvis.no

The 2018 Wilson Research Group Functional Verification Study (2)

Half the verification time is spent on debugging


We can definitely be more efficient! - structured!

www.bitvis.no

Assessment studies – FPGA Verification Tool

Based on UVVM (Universal VHDL Verification Methodology)

• Had a really good infrastructure and structured architecture

• Had standardised verification components and efficient reuse

• Open Source – Using the very permissive MIT license

VHDL based, because• VHDL is the most used HDL world-wide (Design + Verification)

• VHDL is used by 80-90% of FPGA designers in Europe• Most space and defense related companies world-wide use VHDL

ESA project purpose:• Add functionality to handle space related FPGA projects even better

ESA Project


www.bitvis.no

Quality and Efficiency enablers


Structure & ArchitectureStructure & Architecture

Simplicity

Overview, Readability

Modifiability, Maintainability, Extensibility

Debuggability

Reusability

Significantly affects:- Man hours / Cost- Schedule & TTM- Quality & MTTF- Product LCC- … Next project

Easily save 100-500 hoursSometimes 1000-2000 hours

Insufficient simulation willoften cause late problems

www.bitvis.no

clock_generator(clk, GC_CLK_PERIOD);

log(ID_LOG_HDR, "Started simulation of IRQC_TB");...check_value(irq2cpu, '0', ERROR, "irq2cpu default inactive");...check_stable(irq2cpu, now – v_reset_time);...gen_pulse(irqc_source(2), '1', clk_period, "Set source 2 for clock period");gen_pulse(irqc_source(3), '1', clk, 1, "Set source 3 for 1 period");...await_value(irq2cpu, '1', 0 ns, 2* C_CLK_PERIOD, ERROR,

"Interrupt expected immediately");...sbi_write(C_ADDR_ITR, x"AA", "ITR : Set interrupts");sbi_check(C_ADDR_IRR, x"AA", ERROR, "IRR");sbi_write(C_ADDR_ITR, x"55", "ITR : Set more interrupts");sbi_check(C_ADDR_IRR, x"FF", ERROR, "IRR");...report_alert_counters(FINAL);

Typical simple verif. scenario- a low complexity interrupt controller


irq_source(n)

IRQC

/n

clk

SBI (PIF)

arst irq2cpu

All procedures with:- Positive acknowledge

- If wanted

- Alert message- and mismatch report

- Alert count and ctrl

irq_source(n)

IRQC

/n

clk

SBI (PIF)

arst irq2cpu

clk gen

test sequencer

Testbench

www.bitvis.no

check_value() - To illustrate usage

Checks value against expected (or boolean)• Triggers an alert if fail – and reports mismatch + message

Overloads for different types (sl, slv, u, s, int, bool, etc)

With or without a return value (boolean OK)

-- E.g. inside the test sequencercheck_value(dout, x"00", ERROR, “dout must be default inactive");

BV: 60 ns irqc_tb check_value(slv x00)=> OK. dout must be default inactive

BV:==============================================================BV: ERROR:BV: 192 ns. irqc_tbBV: value was: 'xFF'. expected 'x00'.BV: dout must be default inactiveBV:==============================================================

check_value(val, exp, severity, msg, [scope]) -- Simple version


www.bitvis.no

log(), alert(), report_alert_summary()

check_stable(), await_stable()

clock_generator(), adjustable_clock_generator()

random(), randomize()

gen_pulse()

block_flag(), unblock_flag(), await_unblock_flag()

await_barrier()

enable_log_msg(), disable_log_msg()

to_string(), fill_string(), to_upper(), replace(), etc…

normalize_and_check()

set_log_file_name(), set_alert_file_name()

etc…

More in UVVM Utility Library


www.bitvis.noUVVM - The fastest growing ....14

Well Documented

www.bitvis.no

Simple data communication May use Utility Library

and provided BFMs

DUT (UART)

p_main (test-sequencer)

RX TXBFM BFM

uart_transmit(x"2A")sbi_check(C_RX, x"2A")

sbi_write(C_TX, x"B3")uart_expect(x"B3")

TB: 172 ns. uart_tb uart_transmit(x2A) on UART RXTB: 192 ns. uart_tb sbi_check(x1, ==> x2A) completed. From UART RX

TB: 192 ns. uart_tb sbi_write(x2, ==> xB3) completed. To UART TX

TB: ERROR:TB: 192 ns. uart_tbTB: value was: 'xB2'. expected 'xB3'.TB: (From uart_expect(xB3))TB:==============================================================

Free, Open source BFMs:

UART, AXI4-lite, SPI, I2C, Avalon MM, AXI4-stream, GPIO, SBI, ...

SBI


www.bitvis.no

More advanced DUT complexity


DUT (UART)


RX TX

SBI

DUT (UART)


RX TX

SBIUARTVVC

UARTVVC

SBI VVC

sbi_write(C_TX, x"B3")

uart_expect(x"B3")

sbi_write(SBI_VVCT,1, C_TX, x"B3")

uart_expect(UART_VVCT, 1, RX, x"B3")

Free, Open source BFMs and VVCs:

UART, AXI4-lite, SPI, I2C, Avalon MM, AXI4-stream, GPIO, SBI, ...

www.bitvis.no

SBI_VVC

VVC: VHDL Verification Component


TestcaseSequencer SBI_VVC

UART (DUT)

RXOther Ports

Clocks

Bus interfaceTX

Interpreter

- Is command for me?

- Is it to be queued?

- If not:Case on what to do

Executor

- Fetch from queue

- Case on what to do

- Call relevant BFM(s)& Execute transaction

Command Queue

Same main architecture in every VVC

• >95% same code - apart from BFM callsVVC Generation

UART BFM to UART_VVC:less than 30 min

Standard VVC internal architecture

www.bitvis.no

*_VVC

VVC: Easy to extend


Interpreter

- Is command for me?

- Is it to be queued?

- If not:Case on what to do

Executor

- Fetch from queue

- Case on what to do

- Call relevant BFM(s)& Execute transaction

Command Queue

Bit-rate checker

Frame-rate checker

Gap checker

Queue

Response-Executor

Standard Queuing system

Standard handling of multithreaded interfaces

Standard control of parallel checkers

- Easy to add local sequencers- Easy to add checkers/monitors/etc- Easy to handle split transactions- Easy to handle out of order execution

www.bitvis.no

Simultaneous activity on multiple interfaces

Encapsulated Reuse at all levels

Queue May initiate multiple high level commands

Local Sequencers for predefined higher level commands

Only in UVVM VVCs• May insert delay between commands – from sequencer The only system to target cycle related corner cases

• Simple handling of split transactions and out of order protocols

• Common commands to control VVC behaviour

• Simple synchronization of interface actions – from sequencer

• May use Broadcast and Multicast

VVC Advantages


Better Overview, Maintenance, Extensibility and Reuse

Even SW developerscan now write

FPGA test cases

www.bitvis.no

ESA Extensions• Scoreboarding

• Monitors

• Controlling randomisation and functional coverage

• Error injection (Brute force and Protocol aware)

• Local sequencers

• Watchdog (Simple and Activity based)

• Controlling property checkers

• Hierarchical VVCs - And Scoreboards for these

• Direct Transaction Transfer

• Specification Coverage

The new stuff


www.bitvis.no

Generic Scoreboard


Some func.

MP SBI

SBI_SB

Seq.

VVC?VVC?

SBI_VVCMP_VVC

MP_model

Compare

Statistics

Expecteddata

ActualdataQueue

Statistics

insert, delete, fetch

ignore_initial_mismatch

indexed on either entry or position

optional source element (in addition to expected + actual)

Counting:

entered

pending

matched

mismatched

dropped

deleted

initial garbage

generic data type

logging/reporting

flushing queue

clearing statistics

Configuration record:

allow_lossy

allow_out_of_order

mismatch_alert_level

etc...

www.bitvis.no

DUT

Some func.

MP SBI

Model• Models DUT behaviour

• Receives transaction on DUT input

• Generates expected data

• Passes expected data onto Scoreboard

Advanced verification- Using Scoreboards, VVCs and Models


SBI_SB

MP = My Protocol

Seq.

SBI

VVC?VVC?

SBI_VVCMP_VVC

MP_model

uart_transmit(UART_VVCT,1,TX, RANDOM_STIM, 256, "Transmit 256 rand bytes");uart_transmit(UART_VVCT,1,TX, FULL_COVERAGE_STIM, "Transmit until full cover");

sbi_read( SBI_VVCT,1, C_UART_RX_REG, TO_SB, 256, "Receive 256 B and send to SB");

uart_transmit(UART_VVCT,1,TX, x"42", "Transmit single byte");

sbi_read( SBI_VVCT,1, C_UART_RX_REG, TO_SB, "Receive byte & send to Scoreboard");

sbi_sb.check_actual(v_actual);

sbi_sb.add_expected(v_exp);

Compare

Statistics

Expecteddata

ActualdataQueue

Statistics

www.bitvis.no

Advanced built-in funct.

Some func.

UART SBI

SBI_SB

Seq.

VVC?VVC?

SBI_VVCUART_VVC

DUT model• Ctrl randomisation and functional coverage

• Protocol awareError injection

• Local sequencers

• Ctrl property checkers

Randomisation Inside BFMs and VVCs Ex. UART/SBI Q4

P.A. Error injection Inside BFMs Ex. UART/SBI Q4

Ctrl from VVC

Local Sequencers Inside VVC executors Ex. UART Q4

Ctrl from VVC

Ctrl. Property Chk Inside VVC arch. Ex. UART Q4


www.bitvis.no

"Brute force" Error & Monitor


Some func.

UART SBI

SBI_SB

Seq.

VVC?VVC?

SBI_VVCUART_VVC

DUT model

ErrorInjector

UARTMonitor

"Brute force" Error Injector Dedicated VIP Q4

Monitor Module inside VIP Ex. UART Q4

www.bitvis.no

Direct transaction transfer


Some func.

UART SBI

SBI_SB

Seq.

VVC?VVC?

SBI_VVCUART_VVC

DUT model

Direct Transaction Transfer Inside VVC Ex. UART/SBI Q4

www.bitvis.no

Watchdogs


Some func.

UART SBI

SBI_SB

Seq.

VVC?VVC?

SBI_VVCUART_VVC

DUT model

Simple WD Inside Util Q4

Activity WD VVCs and UVVM Ex. UART Q4

WD

Activity Watchdog.

www.bitvis.no

DUT

Ethernet Switch

Hierarchical VVCs and SBs


Ethernet_SB

Seq.

SBI

VVC?VVC?

GMII_VVCSBI_VVC

MP_model

ethernet_send(ETHERNET_VVCT,1,TX, v_mac_dest, v_mac_src, v_payload, “Send pkt”);

GMII

EthernetVVC

EthernetVVC

ethernet_receive(ETHERNET_VVCT, 1, RX, v_mac_dest, v_mac_src, TO_SB, <msg>);

ethernet_sb.add_expected(v_exp);

sbi_write( SBI_VVCT,1, C_IRQC_CLEAR, 0x"FF", "Clear complete interrupt reg");

End Q4

www.bitvis.no

Assure that all requirements have been verified• Specify all requirements

• Report coverage from test sequencer (or other TB parts)

• Generate summary report

Reporting that a testcase has finished is not sufficient

What if multiple requirements covered by same testcase?• E.g. Moving/turning something to a to a given position

Accel.req, speed req, decel req, pos.req, etc..

Specification Coverage


T1T2T3T4

R1R2

R3R4

T1R2 R3 R4R1VS

End Q4

www.bitvis.no

Simple variant


Req. Spec.(.doc)

Req. List(.txt)

Req_1; UART odd parity; TC_UART_1Req_2; UART 150 kbps; TC_UART_2Req_3; UART 100 kbps; TC_UART_1etc…

Test case

1(.vhd)

Inside test sequencer:start_testcase(req.file, result file); -- Read Req list.……..log_req_cov(Req_1, PASSED); -- Mark req. as passed……..end_testcase(VOID); -- Mark testcase as passed if alerts ok

Step 1: Manual (?)

Step 2: Run TC(s)

Step 3: Post process result files

Run run_spec_cov.py Generates report on compliancy for requirements (and TCs) Generates report on requirements vs testcase – if requested

www.bitvis.no

May specify required testcase for given requirement• If so – then may check compliance

May specify that at least one of multiple selected TCs must pass

May specify that a req. is tested in multiple TCs

May have multiple requirement files

May map reqs. in one file to reqs. in another• Thus allowing reusable TBs with coverage included

Spec. Coverage flexibility


www.bitvis.no

clock_generator(clk, GC_CLK_PERIOD);

log(ID_LOG_HDR, "Started simulation of IRQC_TB");...check_value(irq2cpu, '0', ERROR, "irq2cpu default inactive");...check_stable(irq2cpu, now – v_reset_time);...gen_pulse(irqc_source(2), '1', clk_period, "Set source 2 for clock period");gen_pulse(irqc_source(3), '1', clk, 1, "Set source 3 for 1 period");...await_value(irq2cpu, '1', 0 ns, 2* C_CLK_PERIOD, ERROR,

"Interrupt expected immediately");...sbi_write(C_ADDR_ITR, x"AA", "ITR : Set interrupts");sbi_check(C_ADDR_IRR, x"AA", ERROR, "IRR");sbi_write(C_ADDR_ITR, x"55", "ITR : Set more interrupts");sbi_check(C_ADDR_IRR, x"FF", ERROR, "IRR");...report_alert_counters(FINAL);

Recap - simple verif. scenario- Entry level first…? (UVVM Light)


irq_source(n)

IRQC

/n

clk

SBI (PIF)

arst irq2cpu

irq_source(n)

IRQC

/n

clk

SBI (PIF)

arst irq2cpu

clk gen

test sequencer

Testbench

www.bitvis.no

Summary


Project completed – with final milestone approved

Publishing on GitHub • Published: Scoreboard, Watchdogs

• Week 49/50: Error Inj, DTT, Monitor, Rand, SB with all VVCs

• End 2019: HVVC, Specification Coverage

Good feedback from users so far• Received several suggestions for improvements

• Lot of users asking for new extensions

Great cooperation with ESA throughout the project• Good to have technical contact with good technical insight

Successful «internal project» at Bitvis• All our FPGA designers have contributed – and proud of it

Assessment studies – FPGA Verification Tool

Documents

Transcript of Assessment studies – FPGA Verification Tool