Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

Challenges Promoting Theoretical EC Research

William E. HartOptimization Uncertainty/Estimation Dept

Sandia National Laboratorieswehart@sandia.gov

http://www.cs.sandia.gov/~wehart

EC Theory - GECCO 2002Slide 2

Overview

A reminder: EC theory is relevant!

Publishing theoretical conference papers is difficult

… some personal reflections

Some ideas/suggestions

EC Theory - GECCO 2002Slide 3

The Role of EC Theory

I. Modeling and Abstraction

• Goal: identify/validate essential features of successful EAs

• Simple EC methods allow us to explore their limitations/possibilities

• Analysis of abstract EC methods enables these results to be applied to many specific implementations

Q: What are effective abstract EA formulations?

EC Theory - GECCO 2002Slide 4

The Role of EC Theory

II. Unification and Linkage

• Goal: develop linkages between different disciplines

• Transferring ideas between disciplines is easier in simple EC algorithms

• Reformulating EC methods as more traditional algorithmic techniques clarifies their strengths and weaknesses

• Analytic tools from other disciplines can be validated on simple EC models

Q: Can analysis of direct search methods be applied to EAs?

EC Theory - GECCO 2002Slide 5

The Role of EC Theory

III. Assessing Fundamental Algorithmic Capabilities

• Goal: what EC methods can solve what classes of problems?

• Rigorous evaluation of EC methods is required

• Negative results must be formulated mathematically

• Mathematical analysis provides the strongest assurance of robustness in EC methods

Q: Is randomization an essential feature of EC methods?

EC Theory - GECCO 2002Slide 6

The Role of EC Theory

IV. Identifying New Algorithmic Directions

• Goal: anticipate and evaluate potential EC methods

• Can consider EC methods for which practical implementations are not readily available

• Abstract EC models can identify promising research directions and desirable EC capabilities

• The design of practical EC methods can be guided by our analysis of abstract EC models

Q: How can EC methods effectively exploit derivative information?

EC Theory - GECCO 2002Slide 7

What Theory (Probably) Cannot Do

Guarantee fast global convergence

Explain why your code does not seem to work on your application

Provide convergence guarantees for your new EC heuristic

Prove that a particular EC heuristics will not work in general

EC Theory - GECCO 2002Slide 8

Publishing Theoretical EC Results - Journals

EC journals generally encourage theoretical results

Journal articles are usually long enough to provide a complete analysis

Reviewers do not always appreciate the strategic value of theoretical results

Providing reasonable turnaround to authors of theoretical papers is challenging!

EC Theory - GECCO 2002Slide 9

Publishing Theoretical EC Results - Conferences

Conference papers have very tight space limitations!

Short presentation of theoretical results is quite difficult– What background can you assume of an EC audience?

– How can you explain, state and discuss the results?

Conferences typically have a very loose review process– Quick reviews needed

– Rough match of papers to reviewers

– Limited feedback to authors

EC Theory - GECCO 2002Slide 10

My Own Experiences

Publishing in:– ICGA, GECCO, CEC, FOGA– EC, IEEE TEC

General theme: local convergence on – Evolutionary pattern search EA framework– Self-adaptive EAs

Challenges:– Effectively meeting page limitations!– Poor reviewer feedback– Distinguishing between conference/journal publications

nR

EC Theory - GECCO 2002Slide 11

Meeting Conference Page Limits

Idea 1: State the results without proofs

– Usually can condense analysis to fit page limits

– Some reviewer always complains about missing proofs

– Note: any negative feedback can make it difficult to publish a conference paper!

EC Theory - GECCO 2002Slide 12

Meeting Conference Page Limits (cont’d)

Idea 2: Include proofs in an optimal appendix for the reviewers

– Reviewers get confused

– It is easy to ‘ding’ a paper for exceeding page limits

– Organizing and writing this appendix well can be challenging!

EC Theory - GECCO 2002Slide 13

Poor Reviewer Feedback

Typical reviewer comments:– I don’t believe these results

– I’m confused by these results…

– The analysis considers an algorithm/problem that is too simple

– The authors should rework their paper to …

Poor review of theoretical results:– A simple ‘bug’ in my ICGA97 paper was not caught

– Many reviewers fail to appreciate the difference between an approximate analysis of an EA’s search dynamics and an exact characterization of an EA’s search dynamics

EC Theory - GECCO 2002Slide 14

Publication in Journals

EC Publication Policy

Dilemma for EC theory– Can ‘simple’ theoretical conference papers be ‘republished’ in

journals?

– How about more extended conference papers? (e.g. FOGA?)

We should encourage theoretical analyses to be published in journals (they get more careful reviews there).

Current policies effectively discourage publication of theoretical papers in conferences!

Papers should describe mature work that is original in

nature and has not been published elsewhere.

EC Theory - GECCO 2002Slide 15

Some Ideas/Suggestions

Encourage the submission of “Extended Abstracts” for conferences– Idea: a paper that explicitly omits technical details to focus on

the ‘big picture’

– Commonly used in theoretical computer science conferencesSTOC, FOCS, SODA

Special Sessions/Minisymposiums– Idea: sessions of presenters who do not prepare a paper

– Organized about a specific topic or theme

– Approved by the conference organizing committee

– Commonly used within mathematical conferences (e.g. SIAM)

EC Theory - GECCO 2002Slide 16

Some Ideas/Suggestions (cont’d)

Incorporate EC Theory tracks into EC conferences– Idea: explicitly encourage theoretical papers

– This would hopefully help focus the reviewers

– Problem: this track probably cuts across other tracks

Work to clarify editorial policies for EC journals– Idea: allow ‘revised’ conference papers

– Example: SIAM J Computing

In general, only original papers will be considered. However, papers previously presented in conference proceedings of limited circulation may be submitted in revised form.

EC Theory - GECCO 2002Slide 17

Outstanding Issues

How do we ensure that qualified reviewers are available for theoretical papers?

What constitutes an adequate review of theoretical papers?

How can we best educate the EC community about the impact of EC theory?

EC Theory - GECCO 2002Slide 18

References

Challenges for Theoretical Computer Sciencehttp://www.research.att.com/~dsj/nsflist.html

SIAM: Society for Industrial and Applied Mathematicshttp://www.siam.org/

ACM SIGACThttp://sigact.acm.org/sigact

Advice to Authors of Extended Abstracts http://www.acm.org/sigsoft/conferences/pughadvice.html