BENCHMARK COMPUTATION: WHAT IS THE PURPOSE AND MEANING?I. BABusKA

Institute for Physical Science and Technology. University of Maryland. College Park. Maryland. USAJ.T. aDEN

Texas Institute for Computational Mechanics, The University of Texas at Austin, Austin, Texas. USA

IntroductionIn recent years, the solution of benchmark problems has

been strongly advocated as a means for 'validating'engineering software. The idea is classical: a softwareproduct, or perhaps simply an algorithm, is used to solveone or more specific model problems and results arecompared against a standard to determine reliability andperformance. Among well-known standard sets is theMacNeal-Harder set I. which provides a testbed forcertain finite element methods of structural analysis. Otherbenchmark problems are used to validate or 'anchor'computational fluid-dynamics software.

The idea of testing software on model problems is, ofcourse, a very natuml one that has. been part of developingsoftware tools since the inception of computationalanalysis. Indeed 'QA' (quality assurance) is a fundamentalstep in assessing the functionality of a code by exercisingit on at least a few cases. But the idea of 'Benchmarking'as it is now called, has steadily grown into an industryof great proportions in its own right. Benchmarking isbeing heralded within the software industry as a necessarybasis for ensuring the customer of the quality of a softwareproduct. The magazine Benchmark specializes in theperiodic publication of benchmark comparisons and is theorgan of the National Association for Finite ElemelllMethods (NAFEM) which endorses a formalization of thebenchmarking exercise. Today, 'certification' ofcommercial finite clement software is advocated by somespecialists, including commercial software developers, andthe proposed certification processes focus on benchmarktesting.

Our aim here is to examine some of the benchmarkingpractices now being used and to address inherentinconsistencies in the WdY some standard benchmarkingproblems are presented. Typically, benchmark problemsare analysed using different codes, the internal details andmethodologies of which are not available, and little is doneto explain differences in results that may be obtained. Inaddition, little justification of the recommended tests isgenerally given and exactly what properties of the subjectcode are being tested are frequently obscure. Under suchconditions, certification of software would seem to be self-serving for developers passing the tests and, at the least,unfair to both the developer and the user.

To address these concerns. we pose the followingquestions:

• What, in fact. are the goals of speci til' benchmarktests? What. in other 'words, are the properlies of thesoftware that the bem:hmark problem purporls tounveil?

• Are the aClual benchmark prohkms designedproperly to answer these questions?

• What can be inferred from the results or a fewbenchmark examples about more general classes orproblems?

• To resolve these and related issues lilr all cases of

benchmark problems is an impractical if not an impossibletask. Thus, we shall focus on a more modest goal ofdiscussing these issues for computational models of elasticplates.

• We presume that we are given the following task:Given a black-box finite element structural anatysiscode. determine its suitability for use by analyzing aspecified problem of bending of a linearly-elastic plate.The following questions arise:

1. What do we mean by a plate? There are many differentplate theories that can be used as a basis for developingfinite elements. Since the code under scrutiny is a blackbox, we presume that it must perform acceptably on ourbenchmark problems regardless of its basis. One shouldperhaps ask: What features of the plate behaviour are ofinterest? Displacements, moments, shear forces, reactions,etc. may be of interest, but other features can also beinvestigated. How can different plate elements becompared without establishing a meaningful theoreticalbasis for comparison? .

One possible resolution of this paradox is to prescribethe data for the plate as 'exact', these data (for example,moments) are computcd from the solution of a three-dimensional linear elasticity problem for a thin cylindricaldomain (the plate). Boundary conditions on the lateral(cylindrical) sides of the plate have to be preciselyformulated. For example, the notion of a 'simple support'has to be specified in a threc-dimensional setting becausethe dati of interest, for example moments and shear forcesin the neighbourhood of the boundary, as well as reactions,could be very sensitive to boundary conditions when lheplate is very thin.

Also. bounpary conditions have to be 'legal'. By this wemean that the solution has to exist and be unique. It isnot uncommon in engineering literature that an illegalboundary condition is used, such as, for example.specification that the (vertical) displacement on the center·line of the cylindrical side of the plate be zero. Then itis easy to show that, in general, a solution may not existto the plate problem.

The date of interest, for comparison purposes, have tobe finite. For example, referring to the benchmark problemof pinched shell when the load is a concentrated load.Belytschko 2 gives the exact (finite) value of thedisplacement under the load. This cannot be related to thethree-dimensional solution because the displacement isinfinite under the load. (It is also infinite when theReissner-Mindlin shell formulation is used).Nevertheless, the given value has been proposed inbem:hll1ark l'Olllputalions as the exact value. Similarsituations exist tll[ plates under line loading.~. Wh,ll prohlem does lhe tinite element method solve?Assuming. lhat \\'e successively rdine the mesh. areasonahle tinite clement method should converge to anexm:l solution of some mathematical problem. Usually aplate model. ror example. the Kirchhoff or a Reissner-

84

Mindlin model, is first formulated and the finite elementmethod is expected to converge to its exact solution.Nevertheless, it is frequently not clear from the descriptionof some codes exactly what is the solution to which thefinite element method converges.

It could also happen that the finite element solutionconverges to a limit, but the limit has little relation to themodel used as a basis for developing the element. Forexample, consider the problem of a bending of a circular,simply supported Kirchhoff plate characterized by abiharmonic equation. Use a triangulation for which thecircular boundary is approximated by straight-sideelements. Using classical conforming bending elementsand minimization of the total energy over the functionswhich are zero at the boundary, the solution will convergeto the solution of a biharmonic equation, but the boundaryconditions will be completely incorrect.

It may very well be that the only comparison principlefor assessment of codes for linear structural problems isto compare computed data (e.g., moments) with that ofthe three-dimensional elasticity solution.3. How should one compare the efficiency of various finiteelement codes? It is not uncommon in some benchmarkstudies to compare the efficiency of two or more codesall of which produce questionable or incorrect results. Itwould seem to be of little value if a given program wasten times faster than a competitor if the results were wrong.A more rational approach would seem to be to tie theaccuracy of calculation to the effort required to attain it.For instance, how much CPU time would a test cod'erequire to deliver a prescribed accuracy? Other factorswhich come into consideration are the relation betweenhuman cost to set up a calculation and computer cost forsolving the benchmark problems. Hence, any comparisonhas to be supported by concrete data and by convincinganalyses of their relevance.4. What does the benchmark computation show?Benchmark computations are often very complex and canbe naively designed to portray ccrtain deficiencies. Thebenchmark should be such that it isolates a specific andwell-defined aspect of the performance of a code and doesnot lead to false interpretations. For example, in Refercncc) the computation of a uniformly-loaded, simply-supported Kirchhoff plate of the form of a parallelogram(a "skewed plate") with interior angle 0 < a ~ 90° ispresented as a useful benchinark. The mesh is quadrilateral(or triangular with divided quadrilaterals) with sidesparallel to plates skewed sides. Typical benchmark resultsshow detcrioration of accuracy as a decreases. The dataas presented give the impression that the observed effectof deterioration is caused by the skewness of the elements,but this is not the case. There are two basic effectspresent-skewness of the elements and changing of theregularity of the solution (the singularity at the corners)by changing a. It is possible to show that the effect of theskewness of the elements is negligible in comparison to

that of the singularity. Without a careful analysis of theactual solution of the problem, a hastily designedbenchmark problem can lead to completely erroneousconclusions.

It should also be noted that the character of thesingularity associated with a depends, again. on the modelused to derive the plate element. Different convergencerates produced by different rcgularities could bemisinterpreted as different responses to skewness-again,an incorrect conclusion.5. What set of benchmark problems should be selected?It is essential that the purpose of the benchmarkcomputation be clearly defined together with thequantitative indicators that will be used for a givenpurpose. It has to be clear that the indicators depend ordo not depend upon, for example. whcther the regularityof the solution is essential or not in assessing theperformance of an element, etc. This can often beestablished theoretically or experimentally by an additionalset of examples.

Summary CommentsIn view of the deficiencies in many benchmark problems

described abovc, what guidelines could we lay down fortruly testing a given "black box" FEM code on a simpleclass of problems. such as bending of elastic plates?

We believe thi!' question is open and we solicit opinionsfrom membcrs of IACM. Certainly. some possibilitiessuggest themselves:

• Use a well-defined standard as a basis for comparisonof methods, such as the solution provided by three-dimensional linear elasticity theory or by a fine meshof elements based on 3D elasticity.

• Determine the CPU time required to produce a resultwithin an accuracy of five percent and then theadditional CPU's needed to bring the rcsult withinone percent.

• Provide rationale for selecting which "results" aremeaningful and exactly what aspects of thefunctionality, robustness, or overall performance ofthe code these results test.

Many other possibilities suggest themselves. We invitecomments and suggestions from the readership.

References1. R.H. MacNeal, and R.L. Harder. "A proposed standardset of problems to test finite element accuracy," Journalof Finite Elemellf Analysis and Design, I. pp. 3-20, 1985.2. K.J. Forsberg and H.H. Fong, (eds.), "Finite elementstandards forum," Proceedings of the 26th StructuresStructural Dynamics and Material Conference, April 15,1985, Orlando, Florida, PDA Engineering, Santa Ana,California, 1986.3. 1. Robinson, "An evaluation of skew sensitivity of thirtyplate bending elements in nineteen FE systems," F E.N.Finite Element News. Robinson and Assoc. Dorset,England, 1985.