1kuehl@eecs.berkeley.edu

Focus Focus group

BenchmarksBenchmarks

Andre Reis (UFRGS)Andre Reis (UFRGS)

Jarrod Roy (Univ. Michigan)Jarrod Roy (Univ. Michigan)

Vivek Shende (Univ. Michigan)Vivek Shende (Univ. Michigan)

Igor Markov (Univ. Michigan)Igor Markov (Univ. Michigan)

Fan Mo (UC Berkeley)Fan Mo (UC Berkeley)

Andreas Kuehlmann (Cadence)Andreas Kuehlmann (Cadence)

BenchmarksBenchmarks

2IWLS 2003

The “Benchmark” Focus GroupThe “Benchmark” Focus Group• The Benchmark discussion is a…

• after wasting many more IWLS benchmark chairs…

• we finally get companies to give us some circuits with more than 100 gates…• we have put all data nicely into a well defined open format…

• What would be left ?????

• Assuming that…

– Igor and Prabhakar (OUR IWLS BENCHMARK CHAIR!!!!)will talk about the IWLS benchmark effort

– forever ongoing mutual fingerpointing to assign blame for bad result sections in papers

3IWLS 2003

The “Benchmark” Focus GroupThe “Benchmark” Focus Group

Accountability

Reproducibility

Transparency

… for reporting research results

• So, we decided to rename ourselves into …

The “ART” Focus Group

4IWLS 2003

A DisclaimerA Disclaimer

• The following statements…

• do not apply to the • attendees of this year’s IWLS

• advisors or supervisors of members of this Focus group

• but to some of our colleagues who try to publish in the same area

5IWLS 2003

The The “ART”“ART” Focus Group Focus Group

• Long time ago, when CAD was still real CAD….

• Models were simple

• Few previous papers written on my favored topic

• Results could be reported in simple numbers

• Number of rows and columns in a PLA

• Number of literals in a ML circuit

• Runtime on a VAX 11/780 (IBM 360 ??)

• Area overhead of a floorplan

• Main focus was on point tools

• Results were intuitive and easily verifiable• And then there were simple focused benchmarks

• Deutsch’s difficult example for routing

• PLA benchmarks

• And then there was also SPICE

6IWLS 2003

The The “ART” “ART” ProblemProblem

• Today…

• Things got really messy…

• No hope for point tools anymore

• Model boundaries broke down many years ago…

… and we are in permanent denial about it

• Nobody wants to tell us the real technology and timing data

• Big divergence between academic research and industry use

• Results are not reproducible

• Assumptions are oversimplified

“Keep twisting the problem untill it finally fits the solution”

7IWLS 2003

The The “ART”“ART” Problem Problem

• Publication ethics versus effort …

• There are sooooo many papers out there and I need to get mine in.

• Citation cache is < 5 years

• Authors don’t read old papers ….

• Too hard to make results comparable…

• So, why bother?

• Using “MyMath” helps hiding that I reinvented the wheel

“To the best knowledge of the authors this work was not done before”

…. and neither do the reviewers

8IWLS 2003

The The “ART”“ART” Problem Problem

• There is no validation process of published results

• The papers just stay out there “uncommented”

• Poor developers suffer just to find out that it does not work …

… and don’t tell anyone.

• This is different in other communities

9IWLS 2003

Remember This “Breakthrough”?Remember This “Breakthrough”?

10IWLS 2003

Remember This “Breakthrough”?Remember This “Breakthrough”?

• Hendrik Schon does not work for Bell Labs anymore…

11IWLS 2003

What shall we do about this?

12IWLS 2003

The The “ART”“ART” Proposal Proposal

• Need publication channel for verifying published research

• Common practice in Physics and Medical community

• As “messy” environment as ours has become

• Encourage publishing scientific evaluations of previously reported algorithms/experiments

• Should confirm/refute results in a scientific manner

– Not just replicate

– Refutation requires proof, not just “doesn’t work”

» E.g. Sum of the block areas of some floorplan greater than chip area

• Currently a straight “reject” in conferences/workshops because contribution is not “novel”

13IWLS 2003

The The “ART”“ART” Proposal Proposal

• Multiple advantages….

• Graduate students can get familiar with an area and still get some publication out of it

• Tremendous help for industry and others to integrate solutions

• Independent evaluation, different point of view

• Authors will be more careful when submitting papers

• More experiments

– No “convenient” hiding of some benchmarks

• More emphasis on implementation details

– Heuristic choices

• Additional input for tenure evaluation

• Instead of Citeseer numbers

• And for our more senior academic fellows….

14kuehl@eecs.berkeley.edu

Thank YouThank You