Using Virtual Platforms To Improve Software Verification ... · Using Virtual Platforms To Improve...

© 2017 Arm Limited Taiwan 2017

Using Virtual Platforms To Improve Software

Verification and Validation Efficiency

Odin Shen | Staff FAE | Arm

Arm Tech Symposia

© 2017 Arm Limited

Software complexity and best practices

© 2017 Arm Limited 3

Software Costs Increasing

Primary Design Costs By Process Node

The cost of developing and qualifying software increases rapidly with higher complexity IC designs

High-performance SoCs include multiple processor cores and 100s of IP blocks

Source: IBS 2013


Software Verification and Validation

Traditional flows involve development followed by testing.

Modern practices such as Continuous Integration and Extreme Programming call for continuous development & test cycles.

• Develop -> Commit -> Test cycles daily, or several times a day

Frequent commits help reduce the number of conflicting changes.

Frequent testing gives clues about where and when the error was introduced.


Continuous Integration

Automation Server, e.g. Jenkins

• Commit in a version control system

• Test run scheduling

• Triggered after the other builds in the queue have completed

Scripting

• Configure target

• Connect to target

• Initiate tests

• Stop target and query state of memory, registers, etc

Control and Reporting

• Reset, results presentation, etc

Target


Automation is Key

Efficient Continuous Integration programs require good automation.

• Integration with common automation platforms such as Travis, Jenkins and AppVeyor

• Kick off jobs as soon as a commit is made

• Overnight testing

• Results collection and presentation

• Robust, stable, automatable target platforms


Targets For Your CI System

Continuous software integration enforces fail-fast, but you have to have the right infrastructure.

Key component of testing software on Arm is a fast, available, cheap, automatable target.

Let’s look at target options for a CI system…

© 2017 Arm Limited

Choosing a Target for your Software Testing


Software Test Platform Options

Emulator

FPGA

SoC

Virtual Platform


Variables to Consider

Accuracy – How much accuracy do you need? What is more important, accuracy or speed?

Performance – What kind of dependency do you have on performance? Can your testing finish within the desired time window?

Debug and Analysis – How stable is your design? How much debug do you expect? How much visibility into hardware/software interface do you need?

Availability – Are the targets available? Are there sufficient quantity to scale out to a large test environment?

Capacity – How big is your design? Will it fit within the target’s capacity? Does cost scale with the capacity required?

Cost – Have you considered scaled out cost? Infrastructure cost? Setup cost? Maintenance cost?

Flexibility – What is turnaround time for changes?


Platforms Based on Simulation

If you don’t need cycle accuracy, most will agree that a simulated target is preferable.

• Minimal infrastructure

• Easy to maintain

• Easy to deploy

• No capacity issues

• No expensive probes

• Easily automated

• Excellent debug capabilities

Some testing will require cycle accuracy, but testing functional correctness, software-software and software-hardware correctness don’t.


Standard Project Flow Without Virtual Platform

Manufac-turing

SoC Hardware Development

Spec Freeze Tape Out Project FinishedSilicon

Software Development, HW/SW Integration & System Validation

ArchDesign

Hardware–Software Co-design

Gained TTM

Manufac-turing

Standard Project Flow With Virtual Platform

SoC Hardware Development

Spec Freeze Tape Out Silicon

ArchDesign

Software Development, Integration & System Validation

Project Finished


Armv8 “Juno” test chip bring-up timeline

Development on virtual platform before siliconDevelopment on hardware

2 years 10 days

Full software stackand tools validated

on hardware

Virtual Prototype Availability

Fast Models

Arm DS-5 Arm TrustZone


The Real Value of Simulation Platforms

Better debug capabilities. Why? Visibility and control.

• Non-intrusive, deterministic debug

• Control, visibility and determinism are guaranteed

• Immediate halt of entire system on watchpoint or breakpoint

• No expiring watchdogs or transient states

• Capture state for testing, then carry on running as if the stop never occurred

• VP Debug callback

• Checkpointing

• Reverse execution

• Built in or 3rd party tools


Scripting Example

Model connection and configuration

Set breakpoints

Run the application

Get status of variables, registers, memory, etc.

Variable status determines test pass/fail


Debug Example

Virtual Platforms provide almost unlimited tracing capabilities. Virtually every signal, register, memory or CPU program flow can be traced.

• No need for special hardware or software instrumentation

A user can define his/her own monitors and query information about OS state and threads to give more context when printing out results.

A watchpoint can be set on a signal between a core cluster and a subsystem.

Callbacks from the model can non-intrusively report system state.

Stop on access


Virtual Prototype Availability

Cycle Accurate Debug Interface (CADI) defines debug interface to model.

Model Trace Interface (MTI) defines trace interface to model.

CADI and MTI are open and freely available.

– Enables a debug user experience similar to hardware debug.

– Scripting interface for advanced capability.

Source Level Debugger


Biggest Challenge with Virtual Platforms

Where’s the model?

• Companies like Synopsys, Arm, Intel have large modelling teams who create models of their IP

• But what about the rest of the system: UART, keyboard, display, speaker, etc?

• Someone has to model them, or, you exclude those peripherals from your testing

• As hardware evolves, the models must also evolve


Arm Models

Arm Models come from various places.

• Fixed Virtual Platforms (FVP)

• Roll your own with Fast Models, OVP, QEMU

• Ecosystem partners

• Silicon partners

The closer to the actual system, the better.

What if I don’t have a full system simulation?

• Test what you can

• Stub out code


Regression Testing

Regression farms also benefit from models.

Automated defect and performance regression testing.

Same advantages as CI scenario.

• Easy to setup and deploy

• Highly automatable

• Superior debug-ability

• Easy to maintain

Performance Regression Testing – Android Boot


Success Stories

Linaro

• System Guidance

• Armv8-A FVP

Customers

• Intel, Qualcomm, Hewlett Packard, Texas Instruments, Cirrus, Nokia, Ericsson, ST, HiSilicon, Samsung, and many more

Arm internal usage (compiler, models, system software, IP development and testing)

Arm System Guidance Platform


NN on Arm?• Deploy NN inference on Cortex v8.2 platform w/o real hardware

• How to select solution?

Accuracy PerformanceDebug and AnalysisAvailabilityCapacityCostFlexibility


Use MNIST as Example• The MNIST database of handwritten digits, available from this page, has a training set

of 60,000 examples, and a test set of 10,000 examples.

• It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.

http://yann.lecun.com/exdb/mnist/https://www.tensorflow.org/get_started/mnist/beginners


How To Deploy on Arm?• Purpose:

• Prototyping on Arm platform

• Early exploration NN engine design

• SW framework profiling

• Script

• Load image

• Load parameter

• Automation

• Profiling

.....

…

6

AMBA®4 ACE or AMBA5 CHI

SCU

Cortex-A55ARMv8.2-A

32b/64b CPU

L1 cache

Core 5

Shared L3 cache

7 8

ACP

Multicore Debug and Trace

Private L2 cache

Snoop Filter Async BridgesPeripheral Port

…

2

Cortex-A75ARMv8.2-A

32b/64b CPU

L1 cache

Core 13 4

Private L2 cacheDebugTrace

Streamline

Trace and Debug


MNIST at FVP• Debugging:

• Result: NN#1 NN#2


Let’s See• DS5 live demo

• Variables to Consider:

Accuracy PerformanceDebug and AnalysisAvailabilityCapacityCostFlexibility

.....

…

6

AMBA®4 ACE or AMBA5 CHI

SCU

Cortex-A55ARMv8.2-A

32b/64b CPU

L1 cache

Core 5

Shared L3 cache

7 8

ACP

Multicore Debug and Trace

Private L2 cache

Snoop Filter Async BridgesPeripheral Port

…

2

Cortex-A75ARMv8.2-A

32b/64b CPU

L1 cache

Core 13 4

Private L2 cacheDebugTrace

Streamline

Trace and Debug

VV

VV

VV

V


Summary

Growing software complexity is creating the need for new test strategies.

Continuous Integration enables “fail-fast”, greatly reducing software integration efforts.

Models are easy to automate and provide extended access and visibility into the system, are more reliable, and easier to maintain than hardware.

Freely available Arm system models at https://developer.arm.com/products/system-design/fixed-virtual-platforms with software stacks available from Linaro(www.linaro.org)

Looking for more about the demo?

Check the git and free feel to contact [email protected] for the real demo.

https://developer.arm.com/products/system-design/fixed-virtual-platforms

http://www.linaro.org/

https://github.com/odinshen/ArmV82_Keras_CNN_Inference

mailto:[email protected]

3131 © 2017 Arm Limited

The Arm trademarks featured in this presentation are registered trademarks or trademarks of Arm Limited (or its subsidiaries) in the US and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners.

www.arm.com/company/policies/trademarks

Using Virtual Platforms To Improve Software Verification ... · Using Virtual Platforms To Improve...

Documents

Transcript of Using Virtual Platforms To Improve Software Verification ... · Using Virtual Platforms To Improve...