METRICS Standards and Infrastructure for Design Productivity Measurement and Optimization

METRICS Standards and Infrastructure for Design Productivity Measurement and Optimization

Andrew B. Kahng and Stefanus Mantik

UCLA CS Dept., Los Angeles, CA

DAC 2000 Birds-of-a-Feather Meeting

June 7, 2000

Meeting AgendaMeeting Agenda

Welcome

Introduction and Motivations for METRICS

METRICS system architecture Discussion: Requirements for standard METRICS

system architecture

METRICS standards Discussion: Potential standard METRICS

names/semantics

Open Discussion

Conclusion: Action Items going forward

MotivationsMotivations

How do we improve design productivity ?

Does our design technology / capability yield better productivity than it did last year ?

How do we formally capture best known methods, and how do we identify them in the first place ?

Does our design environment support continuous improvement of the design process ?

Does our design environment support what-if / exploratory design ? Does it have early predictors of success / failure?

Currently, there are no standards or infrastructure for measuring and recording the semiconductor design process

Purpose of METRICSPurpose of METRICS

Standard infrastructure for the collection and the storage of design process information

Standard list of design metrics and process metrics

Analyses and reports that are useful for design process optimization

METRICS allows: Collect, Data-Mine, Measure, Diagnose, then Improve

Potential Data Collection/DiagnosesPotential Data Collection/Diagnoses

What happened within the tool as it ran? what was CPU/memory/solution quality? what were the key attributes of the instance? what iterations/branches were made, under what conditions?

What else was occurring in the project? spec revisions, constraint and netlist changes, …

User performs same operation repeatedly with nearly identical inputs tool is not acting as expected solution quality is poor, and knobs are being twiddled

BenefitsBenefits

Benefits for project management accurate resource prediction at any point in design

cycle up front estimates for people, time, technology, EDA

licenses, IP re-use... accurate project post-mortems

everything tracked - tools, flows, users, notes no “loose”, random data or information left at project end

(no more log files!!!) Management console

web-based, status-at-a-glance of tools, designs and systems at any point in project

Benefits for tool R&D feedback on the tool usage and parameters used improve benchmarking

Current StatusCurrent Status

Complete prototype of METRICS with industry standard components (Oracle, Java, XML, HTML, etc.)

Metricized place and route runs on 100+ designs; seeking access to Synopsys regression suite data

Complete metricization of Cadence system-level timing flow

Metricization of synthesis and Verilog simulation tools

Initial feedback from industry on METRICS standards

Attempting to spec Intel requirements for METRICS system

METRICS System ArchitectureMETRICS System Architecture

METRICS System ArchitectureMETRICS System Architecture

Inter/Intra-net

DBMetrics Data Warehouse

WebServer

JavaApplets

DataMining

Reporting

Transmitter Transmitterwrapper

Tool Tool Tool

TransmitterAPI

XML

TransmitterTransmitter

Wrapper-based Perl scripts that wrap log

files and STDOUT

Use existing log files (minor or no change in tool codes)

Completely dependent on log files

Metrics list is limited to the available reported data

Need extra process

API-based

C/C++ library that is embedded inside tools

Does not depend on log files

Data are obtained directly from tools

Require some changes in tool codes

METRICS PerformanceMETRICS Performance

Transmitter low CPU overhead

multi-threads / processes – non-blocking scheme buffering – reduce number of transmissions

small memory footprint limited buffer size

Reporting web-based

platform and location independent dynamic report generation

always up-to-date example: correlation plot – understand the relation

between two metrics and find the importance of certain metrics to the flow

Example ReportsExample Reports

hen 95%

rat 1% bull 2%

donkey 2%

% aborted per machine

% aborted per task

BA 8%

ATPG 22%

synthesis 20%

physical18%

postSyntTA13%

placedTA7%

funcSim7%

LVS 5%

LVS convergencetime

0 100 200 300 400 500 600

LVS

%

88

90

92

94

96

98

100

Current ResultsCurrent Results

CPU_TIME = 12 + 0.027 NUM_CELLS (corr = 0.93)

More plots are accessible at http://xenon.cs.ucla.edu:8080/metrics

COM-Based AlternativeCOM-Based Alternative

EDA tools: provide COM interface for capturing internal

information add information collections (“counters”) inside the

tools

METRICS transmitter: get information via the COM interface format the data in XML, encrypt the message, and send

it to the server

Benefit: allow independent development for transmitter and

tools

Discussion on METRICS ArchitectureDiscussion on METRICS Architecture

METRICS StandardsMETRICS Standards

METRICS StandardsMETRICS Standards

Standard metrics naming across tools same name same meaning, independent of tool

supplier generic metrics and tool-specific metrics no more ad hoc, incomparable log files

Standard schema for metrics database

Standard middleware for database interface

For complete current lists see: http://vlsicad.cs.ucla.edu/GSRC/METRICS

Examples of MetricsExamples of Metrics

tool_name stringtool_version stringtool_vendor stringcompiled_date mm/dd/yyyystart_time hh:mm:ssend_time hh:mm:sstool_user stringhost_name stringhost_id stringcpu_type stringos_name stringos_version stringcpu_time hh:mm:ss

Generic Tool Metricsnum_cells integernum_nets integerlayout_size doublerow_utilization doublewirelength doubleweighted_wl double

num_layers integernum_violations integernum_vias integerwirelength doublewrong-way_wl doublemax_congestion double

Placement Tool Metrics

Routing Tool Metrics

Partial list of metrics being collected now in Oracle8i

Levels of MetricsLevels of Metrics

Tool/Process level

Project level

Enterpriselevel

METRICS SiteMETRICS Site

http://vlsicad.cs.ucla.edu/GSRC/METRICS

Complete list of proposed metrics

Source codes for METRICS server and API

List of presentation on METRICS

Link to various sites related to METRICS

Discussion on METRICS StandardsDiscussion on METRICS Standards