GENERAUZED PENALTY-FUNCTION CONCEPTS IN

MATHEMATICAL OPTIMIZATION

M. Bellmore

The Johns Hopkins University, Baltimore, Maryland

H. J. Greenberg

Southern Methodist University, Dallas, Texas

and

J. J. Jarvis

Georgia Institute o/ Technology, Atlanta, Georgia

(Received June 17, 1968)

Given a mathematical program, this paper constructs an alternate problemwith its feasibility region a superset of the original mathematical program.The objective function of this new problem is constructed so that a penaltyis imposed for solutions outside the original feasibility region. One attemptsto choose an objective function that makes the optimal solutions to thenew problem the same as the optimal solutions to the original mathematicalprogram.

/CONSIDER the mathematical program:

PI: Maximize/(*) subject to

gi(x)^bi for i=l,---,k,

gi(x)=bi for i=k+\,---,m, and xin X (a subset of £").

DEFINITION 1. The feasibility region (FR) for PI is the set

FR = [x:gi{x)g6< for i=l, ••,k a n d

gi{x)=bi for i=k+l, • • • ,m]AX.

DEFINITION 2. The optimality region (OR) for PI is the set

OR=lx*:f(x*) ̂ f(x) for aU * in FR]AFR.

DEFINITION 3. A solution to PI is: (1) Ascertain that OR is empty be-cause (a) f(x) is unbounded over FR, (b) the supr«mum cannot be obtained,or (c) FR is empty (i.e., there is no feasible solution). Or, (2) find at leastone point in OR (and determine if it is unique, this uniquen^ determina-tion being optional).

230 M. Be/hnore, H. J. Greenberg, and J. J. ianis

We generally consider closed feasibility repons with an interior. Fur-ther, if the mathematical program models an economic problem, then therewill be at least one point in the optimality region. However, this assump-tion is not necessary to all of our discussion.

At times we shall be concerned with special classes of PI problems.Although our discussion applies to any mathematical program, we willpoint out some of the advantages that accrue when certain properties areknown.

PROPERTY 1. PI is a convex program if (1) /(*) is concave over FB,(2) fif.(*) is convex for f=1, •••,k, (3) gt.(*) is linear for i=k-\-\, • • -, m,and (4) Z is a closed convex set.

PROPERTY 2. PI is separable if/(«) =/i(*i)O/2(«i) and FR = FRlXFR2, where * is in FR if and only if xi is in FRl, and *2 is in FR2. Thesymbol O denotes a 'Mitten operator' {see Definition 4).DEFINITION 4. A Mitten operator satisfies the monotonicity relations(1) x'^y=>xOz^yOz and (2) x^y=^zOx^zOy for all x, y, zina, specifiedset.

The advantage that accrues when Property 1 is satisfied is deferreduntil we discuss a related penalty function problem. Mitten operatorswill also be discussed later. However, we can immediately see an ad-vantage of Property 2: it can easily be shown that 0R=ORl X0R2, where

0R\ = [x*:f{x*)^fixi) for aU xi in FRl]AFRl

and OR2 = [xi*:f(x2*)^f{xi) for aUx^ in FR2]AFR2.

In other words, we solve two reduced PI problems as (1) maximize/i(«i) subject to «i in FRl and (2) maximize/2(*2) subject to ««in FR2.

Now consider the mathematical program

P2: Maximize F\fix),g{x)] subject to * in X.

The nature of the function F is to penalize large values of gf,(*) for i^k andlarge values of |j?.(*)| for i>k. The choice of F (the penalty function) isvery general, but from a practical standpniint should be continuous in /(«)and g{x) in order to solve the P2 problem without augmenting more con-straints. There are four basic questions of interest:

(1) What is the relation of P2 to PI ?(2) Does there exist a function F such that the solution to P2 solves PI ?(3) Can P2 be solved more easily than PI?(4) Given that a function F exists, is there an efficient way to find it?Clearly one can 'force' the solution to P2 to be the same as PI if the

solution is known (we allow a discontinuous penalty function at this point).Hence, question (2), existence, is trivial in this context. However, ques-tion (4) raises the important aspect of search. We shall therefore use

Generalized Penahy-Function Concepts 231

parameters in order to achieve convergent search schema. That is, a classof penalty functions may be specified and parameters used to vary withinthat class. Consider the problem

P3: Maximize F[f(x) ,g(x); y\ subject to « in X,

where y is chosen from the parameter space Y and F is specified.The analogous four questions of interest are:(1) What is the relation of P 3 to P I ?(2) Does there exist a parameter y in F such that the solution to P3

solv^ P I?(3) Can P3 be solved more easily than P I?(4) Given that a parameter y in F exists, is there an efficient way to

find it?Question (2) in this new list is no longer trivial.

Before discussing each question individually, let us consider a wellknown class of parametric penalty functions known as Lagrangians. Theusual Lagrangian is F=f(x)-Y,^yigi(x) and Y=[y:yi^O for i=l.

KUHN-TUCKERP*' theory may be placed in this framework. Theirsaddle-point theorem answers question (1) for the class of P I problemsthey considered (i.e., which satisfies their qualifications: all convex pro-grams with X=E'^ and FR having an interior are among these—see alsoW. P. PiEBSKALLA,'̂ ' where monotonic constraint functions are con-sidered). The existence question in (2) is also answered. ARBOW, HUH-wicz, AND UZAWA'" give a search procedure for finding ** and y* thatprovides an answer for question (4) with respect to convex programs.

The use of the Lagrangian was extended to arbitrary P I problems byH. EvEHETT.w He partially answered question (1) by proving that whenP3 is solved for y=y* , P I is solved for b=g(x*).

Everett addr^sed question (2) by demonstrating that not every 6vector can be obtained no matter how y is chosen from F. These valuesof b Everett termed 'gaps.' We generalize this in our discussion of ques-tion (2).

BHOOKS AND GBOFFRIONW showed that, if y exists, then (**, y*) is asaddle point of the Lagrangian. They also we^ened Everett's maintheorem so that if 2/.=0 for i=l, , u (where u^k), then any P I de-fined wi th bi^gi(x*) for i=l, •••,u and bi=gi(x*) for i=u+l, • • •, TOhas the solution x*. This agrees with our intuition, since 2/,=0 oftenmeans that the constraint gi(x)Sbi is not binding (i.e., if removed fromP I , OR remains the same). Brooks and GeofFdon also showed a methodof searchii^ y for Lagrangians. This is reviewed in our discussion ofquestion (4).

232 M. MImore. H. J. Greenberg. and J. J. Jarvis

The nonparametric Lagrangian is defined as

where Gi is monotone nondecreasing in gdx) for i=l, • • •, fc.GOULD"" proved the existence of d such that the solution to P2 solves

PI by allowing discontinuous functions. He also proved that the selec-tion Gi from the class of monotone functions stated has the minimizingproperty of a saddle point. That is, (**, G*) is a saddle point of F. Ourwork is principally directed toward parametric penalty functions, but thereader is referred to Gould''" for further reading in nonparametric theory.

We define a class of parametric penalty functions as Lagrangians if

whereGo is continuous and monotone increasing in/(«) ,Gi is continuous and monotone nondecreasing in gi{x) for t= 1, • - , k,Gi is continuous for i=fc+1, • ••,m, andY =br ; i / ,^Ofor i=l , • • •, fclAS"".

NUNNI*" has reported that one may transform the PI problem from9i{x)^bi to -Gi\gi(x)]^-Giibi) = bi', where (?, is monotone decreasing,without changing the feasibility region. In addition, max Go[/(*)]=Go[max /(x)], so that the Lagrangians are really the usual Lagrangians of atransformed PI problem. The motivation for doing this is to eliminategap regions.

Example. Consider the problem PI: maximize fix)='E.iZi Xi^ subject to *'mX = [x:x^O]andg(x)='Eiz;xi^b. Using the Lagrangian, we have P3: maxi-mize F=/(«)-j/ff(,) = EJ=f(xi«-jfa:.) subject to x^O. The P3 problem is un-bounded so that all values of 6 lie in an Everett gap. Now consider another La-grangian as P3': maximize F'=/(«)-j/e"^*' =22i^" x,^-y exp(£*~r *.)i subjectto * go. For y =26 exp(—6), we obtain the solution as Xj*=b for anyj andz,* =0for ij^j. We thus have the solution for all values of 6 of interest (ije., positive).

The point here is that the weighted difference of /(«) and g{x) is in-sufficient penalty, since/(*) will always 'outweigh' g{x) with linear weight-ing.

Moreover, a restricted Lagrangian is a Lagrangian with its parameterspace Y restricted. For example, CAEBOLL'*' used the penalty function

F=/(*)+r£j-^ l/\gi(x) ~bi],

where r^O and m=k (i.e., no equality constraints). This is a restrictedLagrangian with yi=r for i=l,---,m. FIACCO AND MCCORMICK™generalized and formalized this into what is known as SUMT.

A more general class of penalty functions of interrat is a monotone penaity

Generalizmi PenaHy-funaion Conc^Os 233

function. I t is defined as one in which F is monotone increasing in/(*) andnonincreaang in gi(x) for i = 1, , k.

A aeparable-monotone penalty function is of the form

F = G4f(x)]OGi\gi(x)]O • • • OG™[^».(*)],

where F is monotone and O is a Mitten operator. Clearly, any Lagrangianis separable-monotone.

We shall now discuss each question individually. In each case, we areunable to answer the question completely, but some theorems are presentedthat address classes of penalty functions in an effort to add insight.

Questwn 1: What is the Relation of P3 to Fl?

THEOREM LI . Let F be a monotone penalty function for a given y. Let x*be an optimal solution to P3. Then, x* solves PI for b = g(x*).

The proof of Theorem 1.1 is in the appendix. I t basically follows theproof of Everett for the Lagrangian. We can weaken Theorem 1.1 with thefollowing:CoHOLLAKY 1.1.1. If F[/(«), g{x); y] = C (a constant) implies Flf(x),g(x)-\-aei; y] = C for all a, where i^k, then the solution to P3 (viz., «*) solvesPI for bj=gj(x*) forjy^i and bi^gi(x*).

The vector e, has components each 0 except the ith component, whichi s l .

Corollary 1.1.1. (see the appendix for a proof) extends Brooks andGeoffrion's weakening of Everett's theorem. Basically, if F is invariantwith gi(x) (as the hypothesis states), then no penalty is placed upon it sothat it cannot bind our solution. Therefore, it is as though the constraintwere not present, and any 6. that allows ** to be feasible (for P I ) is accepta-ble.

We thus have a relation of F3 to P I for all monotone penalty functions.Now we wish to prove some bounding theorems for a more general class ofpenalty functions.THEOREM 1.2. Let F be monotone increasing inf(x) and such that a contour,say F=K (a constant), implies a continuous function as f(x) = QK\a(x); ylFurther, let Q* be that function corresponding to F=F*(the maximum in PS).Then, foraUxwe havef(x) ^Q*\g(x); y].

For f(x) = QK\g(x);y], see COURANT,'" page 114, for sufficient conditions(viz., the implicit function theorem). Note that any L^rangian satisfiesthese conditions.

We prove Theorem 1.2 in the appendix. Geometrically, it says that allpoints in the space f(x) vs. g(x) lie below this contour resulting from themaximization. If we let /*(6) be /(«*), where x* solves P I , then we havea point of/*(b) for some h and an upper bound for all other b.

234 M. Mbnore, H. J. Greedberg, and J. J. Jarvis

As we swlve more P3 problems, these bounds may be strengthened.Corollary 1.2.1 addresses it^lf to this strengthening.CoHOLLAKT 1.2.1. Let Fi, ••,F,bes penalty functions satisfying the con-

ditions of Theorem 1.2 ioith maxima F*, •••, F*. Corresponding to each

F* there is a Q*, and

/(*)gminimum._x... . . .Q*\g{*);yl-

The proof of Corollary 1.2.1 follows dir«ctly from Theorem 1.2 and isomitted.

We thus have related PS to P I by showing that we are approximating/*( b) . (The usual Lagrangian is a linear approximation.) Each solutionto PS gives an exact point on f(b) and upper bounds for all other 6.

Ouestion 2: Does There Exist ay in Y tor PZ to Solve F17

Suppose all functions are differentiable. A necessary condition for x* tosolve PS is then

if ** is in the interior of X.Therefore, a first necessary condition for x* to solve PI is that there

exist a y in F such that the above equations hold at the solution to P I .For the Lagrangian, this is the usual Lagrange regularity, which usuallyrequires some constraint qualification on P I .

In this section we seek to be more specific and generalize the gap conceptdescribed by Everett. '"THEOREM 2.1. Let F be a Lagrangian as

Let problem PS for y=y* have a finite (or denumerable) nuniber of solutions,sayx*,---, X*, where t^ 1. Let the set B be defined as

B=[b:G,(bi) = ^Z{ wjGi\gi(xJ*)] for

i=l,---,m; £jZiu),= l and tr^Oj.Further, let B' be defined as

B-=B-[b:b=g{x;) for j=l,--,t].

Then, there exists no value ofymYwitha solution x** mch that g{x**) is inB~.

This theorem is proved in the appendix. Basically, it says that, ifseveral alternate maxima exist for PS, then the valura of b that can be ex-preraed as a convex combination of those generated c^mot be d)tained(except, of course, ihaae that have just been generated). f%ure 1 depictsan Everett gap in Hie one-constrfunt case.

Geaeralized Penahy-Fmdiwt Cmcepfs 235

There are several corollaries of interrat whose proofs follow directlyfrom Theorem 2.1.COROLLARY 2.1.1. A gap arises if some choice of y in Y produces at leasttwo solutions (and at most a denumerable number) to P3 with distinct g(x*).COROLLARY 2.1.2. If f*(b) is discontinuous at b*, then there exists d > 0svch that all h in the range b*—d^b<b* lie in a gap—i.e., no vcdves of yin Y have yet obtained all values of b in some left-sided neighborhood of b*.

f(x)ilty function (slope^y)

Fig. 1. Illustration of an Everett gap and its relation to Theorem 2.1.

Corollary 2.1.1 states a consequence that would help in anticipating gapre^ons. Corollary 2.1.2 is illustrated by Fig. 2. An important class ofP I problems tiiat have such discontinuities is discrete optimization—^i.e.,Z is a subset of integers (we thank G. Burch and W. Nunn for pointing thediscontinuous case out to us). Although there will always be gaps the-oretically, from a practical view some of these are not of interest. For ex-ample, consider a linear constraint of the form J^tZ' aiXi^b, where oi, •••,On are integers. The only values of b of interest are those that are multiplesof the greatest common divisor of Oi, • • •, a^.

Fig. 2. A discontinuity at 5*.

The more general question of the existence of y when X is a subset ofint^ers requires more investigation. If P I is a convex program and theinteger <x)netraint is not binding for some 6 of interest—i.e., the solutionto P I when * is allowed to be noninteger turns out to be int^er—thenthe y exists for the Lagrangian penalty function (NEMHAUSER AND ULL-

236 M. Bellmore, H. J. Greenbarg. and J. J. Jaryis

Fig. 3. Unbounded Lagrangian.

MAN'*" show the if-and-only-if case for the 0-1 linear integer program).When the integer constraint is binding, we cannot anticipate, in general,whether or not a gap will arise.

Gaps may arise even though no y in F yields alternate optima to P3.One example of this arises when the penalty function is unbounded. Con-sider a Lagrangian where the PI problem has a single constraint g(x)^b.Suppose there are points for which £?(«)—>— <» a,ndf(x) is finite. In this

f(x)s l o p e =y >0

Genwalized PenaHy-FuncHon Coneys 237

case, for any y>0, the penalty function is unbounded, and there is not amaximum {see Fig. 3).

A second example arises when h{b) has a discontinuity, where h{b) isthe envelope bounding the points of /(*) versus g{x) from above. Againconsider one constraint and the usual Lagrangian. Figure 4 shows a casein which a gap arises at bo, yet there are no alternative maxima to P3.

In addition, one might have a continuum of maxima to P3 and eitherhave a gap (see Fig. 5) or not have a gap (see Fig. 6).

However, the isolated point in Fig. 4 is pathological and does not arisein practice. If our P3 is bounded, gaps tend to arise when alternativemaxima exist for a given parameter. The gap then physically resemblesa dip in the response surface where no support contour of F exists. This isthe more common type of gap occurrence, t" and resolution of this type ofgap can be termed 'gap filling.' We shall discuss methods of gap resolu-tion heuristically. I t is important to refer to Question 3 to realize that theapproaches suggested here may be computationally infeasible at present.

We have seen that, from a practical view, there are two basic sources ofgaps: the unboundedness of P3 and alternative maxima to P3. The formeris more easily disposed of by the following rule of procedure. Let us sup-pose our objective in P3 is of the form Golf{x)]-\-Glg{x)], which is identicalto Ho{x)-H{x). Thus, selecting Go and 6? specifies Ho and H. Let Hoand H be monotone increasing in at least one component of x. In orderthat P3 be bounded, it is therefore necessary to select Go and G (recall thatGo and G are continuous) such that the order (see Courant,f«i page 189) ofHo is less than the order of H. Intuitively, we are saying that the penaltymust be 'sufficiently strong.'

Thus, the example in the introductory section has a gap of this type forthe Lagrangian. This is completely resolved by following the above ruleof procedure. I t is recognized that this resolution may generate computa-tional problems (see Question 3).

The second source of gaps is more common and requires more compli-cated methods to resolve. In theory, there is no problem, since penaltyfunctions can be chosen to 'dip' into the gap region. For example. Fig.1 shows a gap between g{xi*){=bi) and g{x2*){=b-d. If we seek b: bi<b<b2, then we have a gap using the usual Lagrangian.

Various alternatives are available (see Gould"")- One of these is touse a parabolic of the form

F=fix)-yi\gix)-b]^'", where yi^O, y^^l.The effect of this procedure is to probe the gap region near b and obtain anearer solution, or at least strengthen the upper bound on /*(6). As j / j - ^ «we force g{x*) = b. This, of course, may not be the solution if the envelope

238

f(x)

M. Mbnon. H. J. Groenberg, and J. J. Jaryis

> g{x)

IS not monotone nondecreasing. Even so, the ability to solve this problemIS lessened if certain properties (see Question 3) are not present

If we choose y,=l and y^ such that [/*(6i), fej and f/*(6i), 6] are on thesame contour (as pictured in Fig. 7, where, in regions A and B the La-

slope=y > 0

h(b)

Generalized Penalty-Fundhn Conceph 239

grangian yields a gap, region A being resolved by the parabola, and re^onB having stronger bounds to the point denoted by C), then the gap is re-solved in one step if f*(b)=f*(bi). Otherwise, a nearer feasible solutionand stronger bounds are obtained. This may be repeated until the gap isresolved.

In closing our discussion of Question 2, let us point to the future. Asophisticated penalty function program would use the techniques of arti-ficial intelligence to select the penalty function F and the parameter spaceY. As the search progresses, learning would dictate an 'optimal' selectionto generate as much information as possible about f*(b). Some of the

f(x)

Fig. 7. A parabolic gap filler.

known results in search under uncertainty may be applied in this 'optimalselection.'

However, before concluding our comments on Question 2, we wish toconsider a non-Lagrangian penalty function. Our purpose is to offer agap theorem that may prove beneficial in determining methods of resolu-tion.

We consider an exponential of the form ^ven in the following:THEOREM 2.2. L^t F(x;y) be given by

F(x;y) =G, I / (* )1- n < ^ Gi\g,(x)r,

where y is in F = [y:y^O]. We ossMmc Go and G, are monotone increasing,and Gi\gi(x)^ 1] for i=l,--,m. Then, if F(x;y) has alternate maximaXl*, • ••, X*, there exists noyinY SMCA thcU

240 A4. MImore, H. J. Greenberg, and J. J. Jarvis

Gi\gi{x*)]-= I i i . / Gi\gi{xJ*)r for

i=l,---,m where »^0 and

where J is the set of indices j satisfying

The proof of Theorem 2.2 is in the appendix. Note the similarity toTheorem 2.1 where F is linear over y (the Lagrangian). For the Lagran-gian any arithmetic mean of the fif(«,*) is in a gap; when F is geometricover y, any geometric mean of some of the gix*) is in a gap. Of course,the term 'gap' is used here in the Everett sense. Fory=0, Corollary 1.1.1may be invoked to reduce the gap region.

Question 3: Can PZ Be Solved More Easily than PI?

The very nature of the question suggests that a qualitative discussion ispresented. The authors wish to stress two desirable properties and citesome classes of PI problems that would benefit from them. The two prop-erties of interest are: (1) F is concave in * over X, and (2) F is separablein X over X, where X can be expressed as a cart^ian product of the par-titioned vector, (xi, xj).

Besides eliminating the gap problem pointed out in Question 2, theability to solve P3 is enhanced when F is concave. First, the local maxi-mum is also the global maximum. Second, schemes presently exist tomaximize a concave objective when X is convex.

Moreover, even if F is not concave in x, it is advantageous to have Funimodal in x over X. For example, consider the PI problem as: Maximize/(*) = 53<Zr x/, subject to gix) = ^iZi Xi^b and x^O. The objective/(*) is not concave. Suppose we use a monotone-separable penalty func-tion as:

P3: Maximize F=na.n[f{x), v/gix)]subject to «^0, where Y=\y:y^O].

Since/ and g are both monotone increasing ia *, the solution to PS satis-fies /(**)= y/g(x*). Therefore, if y = 6', then x* = b for any j and *,* = 0for tVj. We thus have the solution to PI for all nonn^ative values of b(note that for 6<0 there is no feasible solution).

The Lagrangian preserves concavity properties in that, if PI is a convexprogram, then P3 will have a concave objective. An example to illustratethe advantage of this is the quadratic program. Let Pi be: maximize-}>ixDx-{-cx subject to Axg 6 and X=E', where D is positive definite andsymmetric [there is no loss in generality from assuming D symmetric—ifdijT^dii, then replace D with D' given by d-,=d;.= (d,y+d^<)/2]. Thenthe P3 problem is: maximize {-}i)xDx-i-{c-yA)x, where Y=\y.y^O].

Generalized Penahy-FunctHM Concepts 241

Necessary and sufficient conditions for «* to be the solution to P3 arethat all first derivatives vanish. Hence, we can find ** by solving the equa-tions given by x* = (c-yA )D''\ Thus, solving P3 when y is specified re-duces to performing these matrix operations.

The second property of interest is separability. To illustrate its ad-vantage, consider the ahove quadratic program when D is the identitymatrix. The solution to P3 is then x*=e—yA, or x*=Cj—yaj for j = l• • •, n, where a, is the jth column of A.

If X restricts « to be nonnegative, we can trivially extend this to beX*=max (0, Cj-yjOj). One can verify that ** given above is the solutionby cheeking the Kuhn-Tucker conditions.""

Suppose further that X restricts « to he integer. The cell property issuch that we solve n univariate integer objectives. For a unimodal uni-variate function, the optimal int^er solution is one of the neighboring in-tegers of the optimal continuous solution. Hence, only 2n comparisonsneed be made (rather than 2").

In general, if the penalty function is Mitten-separable—i.e., if F =Hi(xi)OHi(xt), where O is a Mitten group operator, and Z = Z i X X ,where « is in X if and only if «i is in Xi and xj is in Zj; then P3 may besolved as two separate problems given by:

P3( 1): maximize Hi(xi), subject to *i in Xi;

P3(2): maximize Hi(xi), subject to JC2 in Xj.

Consider the Lagrangian applied to a PI problem that is sum-separableso that we have:

PI : maximize 2-*-"/.

subject to 23<-" 9

XiinXThe P3 problem is

P3: maximize

subject to X

where hi(xi; y) =fi(x

•XXi),

ii(xi)^

iiiXi) =

. for

ZJ-i"};, in X.

'.) - X

Ibj for

= bj for

i=l, ••

ii(xi;y),for i=

' ' •^ gi (x

j

j

1,

. ) •

= 1, ••.,*;,

= fc+l, •••,m

n.

• • •, n .

Everett terms this a 'cell problem,' where each h(xi; y) is a cell.Instead of solving P3 for a given y and repeating this until PI is solved

(or a gap is reached), we can use recursion as follows: Let Qt(y) be definedas Qi(y')='J^Pimaxhi(xi;y*). We thus have the recursion Q,(y')='Q«-i(y*)+max hi(xt; y*). For each value of y' there is an a^ociated »"*such that (x*, • • •, x*) is the solution to the subproblem:

242 M. fieffaiore, H. J. Grewiberg, and J. J. Jarvis

Maximize J^iZ[fi{xi),

subject to ^iZigij{Xi)^sy^ for j=l,---,k,

Z:-Ilgo(a;0 = si" for j=k-\-l, • • •,m,

XiiaXi for t = l , •• ,<.

We select values of y' to generate a 'useful' range of s'" values and con-tinue until t=n. Everett reports'" that this approach requires only a fewiterations of solving PS problems when the functions are monotone increas-ing-

Observe the similarity to dynamic programming, where stages areequivalent to cells. We are searching a parameter space Y rather than astate space containing the s*" vectors.

Thus, separability provides special structure which may be taken toadvantage. It is therefore desirable to select penalty functions where PS isseparable.

Question 4: Given that a Parameter y in K Exists, Is There anEfficient Way to Find It?

Let us first consider a restricted La^angian with a single parameter as

qFiacco and McCormick'"' use the following search scheme for Go[/(«)] =

/ ( * ) a n d Gi\gi{x)]= -l/\gi{x)-bi] for i = l , •••,m: ( 1 ) S e l e c t y>0 a n d

let xo be any feasible solution in the interior of FR (i.e., g{xo)<b). (2)Starting at *,_i, we generate «, as follows (for j = l , 2 , • • • ) : reduce y,where 0<y'<y'~^, and use a gradient search procedure to maximize P3;this new maximum is «,; repeat step 2 until the desired accuracy is obtained.

They prove convergence when PI is a convex program. Geometrically,the nature of the penalty function chosen prevents the gradient search fromcrossing the boundary by imposing an infinite penalty. That is, F divergesto negative infinity when gi{x) approaches bt from below for any i.

Other penalty functions having the same divergence property were in-vestigated by ZANGWILL.P" His results are somewhat more general thangeneral than those of Fiacco and McCormick, but still only apply to re-stricted Lagran^ans. He uses a single-parameter search, where the initialpoint is usually infeasible, and hence uses an 'outside-in' search.

Another approach (developed independently for m constraints byEverett and Furmani'i) is given in BELLMOBE, GBEENBEBG, AND JARVIS'''for m = l and F=f{x)-yg{x); they assume that F is bounded for all y^O:Step 1. Solve P3 for y=0 and y=Af, where Af is wbitrarily large; let *i*

GeneraUzed PenoHy-Function Cone«pte 243

and xi* be the two solutions, respectively; further, let bi*=g(xi*) for i—1,2. Step 2. Let bj^=^TaaXi\g(xi*):g(xi*)^b] and 6« = min, [j/(*y*):g(xn^b] if 6i=6», then stop; otherwise set 2/ = f/*(6i.)-r(6.)]/[6i-feJand solve P3; if the solution x* is such that g(x*)=bi. or 6., terminate,since 6 is in a gap; otherwise, repeat step 2.

Computational experience indicates that these search procedures eanbe very efficient for certain P i problems (see Bellmore, Greenberg, andJarvist")-

Now consider m-parameter Lagrangians. Everett uses a step-wiseprocedure whereby 2/.-'' = j/"''"(l+d,-'*), where d*" may contain positive,negative, and zero components. The sign of di'^ is determined by the fol-lowing rule: if gi(x*-i)>bi, then d\'^>Q, if gi(x*-i)=^bi, then d'i^=Q; or

j / ( ) ,

Arrow, Hurwicz, and Uzawa'" also used a step-wise procedure in find-ing saddle points. Brooks and Geoffrion'*' showed that we indeed aresearching for 2/=y*. where (**, y*) is a saddle-point of F.

Whenever ?.(**) 'crosses' 6.—i.e., when [ff<(**-2)-6.][^i(**-i)-6,]<0,then the magnitude of d<'' should be decreased. Everett'" reports that areduction of M has empirically proven successful.

Everett also reports that convergence may be accelerated by adding anextra parameter as:

E'where Y={(y,,y):y,^O for i=0, •

The value of j/o is adjusted by applying the rules to the redundant constraint

Brooks and Geoffrion'*' used linear programming BS another method ofsearching for y in F, where F is the Lagrangian. Recall that one couldsolve a convex program by linearizing the functions and solving the linearprogram (see, for example, HADLBY,'"' page 106). The Brooks-GeofFrionscheme may be viewed as such a device, except that the location of thebreak points need not be specified in advance. One approximates the PIproblem by solving the case:

Maximize ^'t^l f(x*)ut,

subject to X^jirgi(x*)ut^bi for t= 1, • • •, k,

Y^'^lgi(xt)ut=bi for t=fc+l, •••,m,

and (for normalization) '^'^lUt^l, u^O,where the vector xt solves the P3 problem for some previously choseny=y*. The new value of y has components equal to the m 'true' dualYuiabl^ to this Hnear program. That is, the dual is:

244 M. Bellmore. H. J. Gre^iterg, and J. J. Jaryis

Minimize X}'-!* ̂ i^.+Wo,

s u b j e c t t o «>o+ Z < ^ Wigi(x*) ^f(x*) for t = l , - , T ,

Wi^O for i=0, •••,k.

The value of y is w*. This choiee of y either gives the solution or gener-ates a new dual constraint. Hence, one uses the dual simplex, for example,to proceed with the search.

To illustrate, suppose we consider the problem: Maximize -}4(xi^+Xi^)-\-Xisubject to xi-a;2^0 and xj+xj^l. Using the Lagrangian, we have: MaximizeF = ~H xx'-i-(l~yi-yi)xi-}.i xi'-{-(i/i-yi)x2 subject to xi, x^^O. The solutionis: x,*=max(O, l-yi-y,), a;2*=max(0, yi-yi).

Suppose we start withy = (0,0). The solution is «* = (1, 0). This is not feasi-ble for PI, yielding b* = (1,1).

We solve the linear program: Minimize U?O+M'2, subject towo+wi+Wa^M andWo, tp g 0. An optimal solution is iCo* = wj* = 0 and wi* = H • Using y = {wi*, W2*)yields ** = (Ji, }i), which is the optimal solution to our original program.

We thus see that some results are available to search over Y, but onlywhen f is a Lagrangian. Some theorems are needed to gain insight intostructure. To this end, Theorem 4.1 is an extension of Everett's mono-tonicity theorem."'THEOREM4.1. LetF=Fi{f(x),g(x)]-{-ryilG\gi(x)]]''\ where rp^O, rG\gi(x)]is monotone nonincreasing in gi(x) and Y=ly:y'^O]. Then:

(1) Ify*=yt*>0 and ify*>yT, then gi(x*)^gi(x**).(2) / / y* = yt*>0, y*>yt*, G\gi(x*)]>l, and G^i(x**)]>l, then

gi(x*)^gi(x**).The proof of Theorem 4.1 is in the appendix. The result provides in-

formation on how to changey in order that gi(x*) converge to bi.An example of a monotone penalty function with Gi> 1 is F=Fi-\-yi\l +

exp[—gi(x)]] *', where Fi is monotone.It is hoped that this theorem will provide a method for searching yi and

2/2 analogous to the Lagrangian search schemes.Now consider the following:

THEOREM 4.2. Let Fi, •••,F,be s penalty functions with maxima F*, • • •,F*. Further, let F be defined as F=H(Fi, • • •, F,), where H is monotoneincreasing in Fi,---,F,. Then max F^H(F*, • • •,F,*). Further, ifxi*= • • • =«.* = x*, then max F = ff(Fi*, • • •, F*), with solution x*.COROLLARY 4.2.1. Let F=aFi+(l-a)Fi, where O^a^l. Further, let

^.=/(*)-E;2/fff,(*) for i=l,2.

Then, if xi*=Xi', values of y satisfying y=ay^^^-\-(I—a)y^^^ cannot yield anynew solution.

The proof of Theorem 4.2 is in the appendix. Corollary 4.2.1 follows

Generalized Penalty-Function Concepts 245

directly from Theorem 4.2. This result should help in searching parametersfor non-Lagrangian penalty functions.

In conclusion, it is useful to note when the y sought is a saddle point ofF{f, g—b;y). One could then, for example, employ the Arrow, Hurwicz,and Uzawa search scheme.

Brooks and Geoffrion'*' have shown thaty* is a saddle point of F when Fis the usual Lagrangian. Gould'"' also proved a saddle-point theoremdealing directly with functional. Our question is: "Under what circum-stances will we have F[/(x*), g{x*)-b;y*]^F\S{x'), g{x*)-b;y] for allyin r ? "

We shall consider F monotone, and (**, y*) satisfying the conditions inCorollary I.I.I. Further, let F be differentiable in y. From Taylor'stheorem, we have

In order for y* to minimize F*, we require iy-y*) \7yF*-\-R^0 for all yin F. A sufficient condition is:LEMMA 4.1. If VyF*=OandF isconvexiny, then y* is a saddle point of F.LEMMA 4.2. 7/(1) y*Fyt = O for t = l , ,m, and (2) yS/yF is monotonenondecreasing, then {x*,y*) is a saddle point of F.

In both lemmas, the saddle-point property holds if y* exists (i.e., nogaps). Note that the Lagrangian is a special case of both lemmas.

The presence of a saddle point then provides a dual program to FI.If F is concave in x, then a dual to PI is: Minimize, F{f, g;y): (1) ytY,(2) V,F=O, where we assume that 7 is a convex set. This relates toDANTZIG, EISENBBBG AND COTTLE'S'"' more general duality (i.e., not rely-ing on the Lagrangian) in that we can deal with programs which are notconvex. [For example, Pierskalla's monotonic case has a dual if F ischosen by the rule of procedure discussed in Question 2 and F is chosensuch that it is concave in x over X and convex in y over Y (a closed, con-vex set).]

FURTHER GENERALIZATIONS

THE PRECEDING discussion is a special case of the general concept of 'prob-lem transformation.' The general case allows transformations of variablesas well.DEFINITION 5. A problem transformation of PI, say P2, is an optimizationproblem where FR2=R{xl,fl, gl,Xl), f2=B{xl,fl, gl, XI), and x2 =V{xl,fl,gl,Xl), where FRi is the feasibility region for Pi. Similarly,fi, gi, Xi and xi denote the objective, subsidiary conditions, the set X andthe decision variables, respectively for Pi.DEFINITION 6. A vcUid problem transformation of PI (denoted VPT),say P2, is a problem transformation where ORl = T{0R2).

246 M. BeUnmre. H. J. Gramberg, and J. J. Jaryis

In other words, we solve a substitute problem in which the originaloptimality region is a transform of the new optimality region. Penaltyfunctions use the transformation given by: F i?2=Xl , f2=F(fl, gl), and*2 = * 1 .

Whether or not P2 is valid is not known in advance. Instead, we dis-cover that 0R2 is the optimality space for some P I problem, but not neces-sarily the one we wish to solve. However, information is obtained to de-termine our next choice of P2.

Classically, variable transformation has used elementary results. Forexample, if *2 = t(*l) and xl — ir\x1), then P2 becomes

maximize /2(x2)=/l[ t~ ' («2)] ,

subject to Sf2i(jt2) = ffl,[r*(»2)]g6. for i=\, • • • ,k,

g2i(x2)=gUt\x2)-\ = bi for z = fc-f 1, • • •, m.

where Q is given by: x2 is in X2, if and only if, t"^(*2) is in XI .

To illustrate, consider the problem given by Kuhn and Tucker"" as: maximizeXX, subject to - ( l -xO'-HaSO, x in Z=[«;*^0].

The solution, «• = (1, 0), is at a cusp so that the Kuhn-Tucker constraint quali-fication is not met. However, let us consider the variable transformation *=<(*)given by; zi=-(l-Xi)'-fa;j, «2=X2, with the inverse Zi = (2i-«j)"'-(-l, X8 = zj.The P2 problem is: maximize/2(») = (2i-«»)•"-Hi subject to Zi^O and * in Z =l*.-(«l-«j)'" + 1^0,Z2&0].

Further transformations discussed earlier give the equivalent problem: maxi-miae z\—z%, subject to zi^O, z\—zi^—\ and zj^O. This now is a linear programwith ** = (0,0). Using the inverse transformation we obtain x* = (1,0).

The point of this exercise is to illustrate how problem transformationsother than penalty functions may be applied to solve a given problemefficiently.

CONCLUDING REMARKS

THIS PAPER has placed the concept of penalty functions in a general theorythat unifies con^pts and presents some new theorems to gain i n s i s t intostructure.

Moreover, there is still the more general concept of problem transforma-tion in optimization theory, of which tlie penalty-function approach is aspecial case of interest. However, there are other transformations thatreduce the complexity of the problem.

GmeralLmd PentOfy-FuncHon Concepf*

APPENDIX

PROOFS OF THE THEOREMS

THEOREM 1.1. Let F be a monotone penalty function for a given y . Let xoptimal solution to PZ. Then, x* solves PI for b=g{x*).

Proof. Since x* maximizes P3, we have

for all X in X. Now, suppose we restrict x so that gi{x) g6. =?<(«*) for t = 1, , fcand gi{x) =6,- =?<(«*) for t = Jb-|-1, • • - , « . Suppose an x satisfying the above, sayx^, exists such that/(*') >/(«*). Then, by the monotonicity property:

[/(), 9();y]Fm''), g(**);y],which contradicts (1).CoKOLLABY 1.1.1. IfF\f{x),g{x);y] =C{aconstant)impliesF\f{x),g{x)-{-aei;y]=Cfor all a, where i^k, then the sdtiiion to P3 {viz., x*) solves PI for bj=g,ix*) forj^iandbi^gi{x*).

Proof. From Theorem 1.1 we have that ** solves PI for 6 =?(«). Further,

Therefore, x* maximizes F[/(*), g{x)-\-aei; y] over X. From Theorem 1.1, ** solvesPI for 6 =g{x*) -{-aei for aU o. Therefore, the only restriction on 6, is that ** befeasible—i.e., bi^gi{x*).THEOREM 1.2. Let F be monotone increasing in /(*) and such that a contour, sayF --K {a constant), implies a continuous function aa f{x) = Q*\g{x); y]. Further, letQ* be the function corresponding to F=F* {the mM;imum in P3). Then, for aU xwehavef{x)SQ*\g{x);y].

Proof. Suppose there exists x' such that /(»«) >Q*[g{x'); y]. Let F' =F[/(«>),?(*'); y], which then implies the existence of Q' such that f{x^) =Q%(«0; yl Wethus have

and since Q* is the contour for F=F*, we have

Hence, F> >F*. This contradicts the fact that F* is theTHEOREM 2.1. Let F be a Lagrangian as:

Let prMem PZ for y=y* have a finite {or denumerable) number of sdutions, sayxi*, •••, X*, where t^l. Let the set B be defined as

B=[b:Gi(bi)='£';Zlw,<li[gt(xi*)] for j = l , •• •, mEJH u),=l and wSOJ.Further, let R- be defined as B-^B-^[b:g{xj*)=b for j = l, ,t]. Then, thereexists no value otyinY with a solution «•* such that g{x**) is in B".

Proof. Assume there exists y** =y*+d in Y such that «** is the solution to

248 M. Bellmore. H. J. Greenberg. and J. J. Jarv/s

problem P3, and b**=g(x**). Clearly, if ***=«/ for any j = l, ••• ,« , thenb** = b* so that b** is not in B-. Hence, we consider *** ^xj* for any j = l, ,t.

Since x** solves F3 for y* =y**, we have

G.[/(***)l- Z * ^ yf*G.f{,.(***)]SG«[/(*y*)]2:5:7 yf*Gilg{x,*)]

for ;• = 1, •••,<. Substituting y*+d for y**, we have

for all j = l, •••,t. Further, since *,* is a solution to P3 for y=y* and sinceis not a solution to P3 for y=y*, we have

for all i = 1, • • •, L Thus, in order for «** to be a solution to P3 for y =y*-t-d andnot be a solution to P3 for y=y*, it is necessary that d satisfy the following in-equal i ty : T,t^di{Gilgi(,xj*)]-Gi\gi(x**)]}>0{oT&nj = l, • • - , ( .

Motzkin's transposition theorem'*' states that d exists to satisfy the aboveinequality if and only if there is no solution to the system:

E!-(«',|G.[ff.(*,*)]-(?.[(?i(***)]l=O, eSO,e^O for all i=l,---,m.

Setting M';=»y/Zt""i "•> we have that g(x**) cannot be in B~.THEOREM 2.2. Let F{x;y) be given by F(,;y) =Go[ / (« ) ] -n t r G.to.^,)]-., w^rey is in y = [y .-y ^0]. We assume Go and Gi are monotone increasing, and Gi^i(x)] & 1for i = l, • • •,m. Then, if F(x;y) has alternate maxima xi*, • • •, *,*, there exists noy in Y such that

where J is the set of indices j satisfying

Proof. Consider y»=y-f-d, and let ** maximize F(x-yi. For i*=*y* forsome J =1, • • •, < we have no new value of g(x*). Hence, we shall suppose thatX* does not maximize F(x;y).

Since ** maximizes F(*;y-l-d), we have

Go\r(.b)]-IlilT G.(6i)'"-".gGo[r(b'")!-!!'^ G . ( 6 . ' ) ' - « 'for i = 1, ,«. Let us subtract n H " C^ft.)" + Iand we obtain

for j" = 1, • • •, <. Since F(«y*;y) >F(«*, y), we have

Generalized Penalty-FuncHon Concepf* 249

For each j satisfying I l i i r G<(&.)''< S ntZT <?.(fr.O''<, we have [since (?,( )S1]lV::TG<ibi)''i<UtTGi{bi')% or Zdil\ogGi{bi)-logGi{bi')]<0. Such a dexists if and only if there is no solution to the system given by J^y u!,[Iog(?,(6<) —logG<(6.0]=0, wSO, ZiM'y = l. where the sum is over those j for whichTL^Gi{bi)''i^U.t^Gi{bi')»i. This becomes Gi{bi) = UiGi{bi')', for t>SOand E e , = l, where vy^wy/Etw*. Thus, if b satisfies n j l T G.^^.)"'^n Z * G K O j in / , then 6 cannot satisfy Gi{bi) = '[lj,jGi{bJ)'i for any

THEOREM 4.1. Le< F = Fif/(*), g{x)Hryi{G\gi{x)]]<'t where r?^0,tone nonincreasing in g,{x) and Y = ly:y^O]. Then:

(1) Ify2* =2/** >0 and yi* >y**, then gi{x*) ^g^x**)-(2) / / yi=y**>0, y*>yt , %.(**)]>l, and G\gi{x**)]>l, then gi{x*)S

9i{x**),where «* solves PZ for y =y*, and «** sclixs PZfory =y**.

Proof. Conclusion (1) follows directly from Everett's monotonicity theorem,but we present a proof for completeness.

We have F[f{x*), g{x*);y*]^F\f{x**), g{x**y,y*], since x* is the solution toPZ for y=y*. This implies

Let y* =yt*-\-d, where d>0. Substituting into the above yields

FiU{x*),g(x*)]-i-ryr{G[gdx*)]]'r-^rd{G[g,(x*)]]<>r

^F,\f{x**), g{x**)]+ryV{G[gi{x**)]\'r+rd[G

Since x** solves PZ for y=y**, we have that

Since d>0, we have

Now suppose r>0. Since j/**>0, we have (monotone nonincreasing in gi{x) and r >0, we have that G is monotone nonincreas-ing in gi{x). Hence, gi{x*) ^gi{x**).

If r<0, then G\gi{x*)]<G\gi{x**)] and G is monotone nondecreasing in gi{x),which gives the desired result.

Now consider conclusion (2). Again, since x* is a solution to PZ when y =y*,we have

We set yt* =y** -\-d, where d>0, and note that yi* =2/f* >0. Thus, we have

F,mx*),g{x*)]-{-ryr{G[gi(x*)]]''r^^F,lf{x**),g(x**)]-\-ryt*lG[gi(x**)]\H^.

Let us add

to both sides of the inequality. We thus have

A4. Mbnore, H. J. Grembero, ond J. J. Jarvis

Smee «** solves P3 for y =y**, we have

ryf{<?to<(«")]|''r-|-

which can be written as

ryt*{Glgi(x**)]]''ril-[G

Further, since yt* >0, we hav

Suppose r >0. We then wish to show that G[(/<(**)] SG[^,(«**)]. Upon so doing,the monotonicity of G then yields the desired result.

Consider the contrary—i.e., (%<(«*)]«?[».(***)]. Then, siand G\gi(x**)]>l, we have

0>l-{G[gi(x*W>l-{G{gi(x**)]\'',

0<-{l-[G\gi(,x*)\Y]<-{l-[G\jit(,x**)]]'\.Thus,

which is a contradiction.For r <0, G is monotone nondecreasing, and

Again, we need only show that Glp,(**)] ^G\gi{x**)]. We assume the contrary—i.e., suppose G\gi(x*)] >G[^.(«**)]. Then,

-{l-\G\gi{x*)]Y\>-{l-[G[gdx**)]]']>0,so that we have

which is a contradiction.T H E O R B M 4.2. Let F i , . , F , b e s penaUy functions with maxima Fi*, . , F.*.

Further, let F be defined as F = H ( F i , •••, F . ) , where H is mmuOone iruTeasing in

F i , • . . , F . . Then m & x F ^ H ( F i * , . • -, F . * ) . Further, t / « * = ••• = , . * = , * . thenmax F-H(F,*, • • .,F*) wUh sdution x*.

Proof. Since H is monotonic, we have H(Fi>, , F.») ^H(Fi', •••.F.') ifF ' g F ' . Hence, ^(F,*, •• . ,F.»)sF(Fi , • . • ,F . )=F, since F.SF<* for i = l,• • • » * •

Further, if *<*=,* fort = 1, .••,«, then, max F=ff(Fi», •• •,F.*).

T H B ATTTHOBS wish to express q>edal tiianks to H. EVEBSXT I I I for hismany helpful comments.

Generalized PenahyFunOiwt Conc^jfs 251

REFERENCES

1. K. J. AKROW AND L. HIJBWICZ, Studies in Linear and Non^Linear Programming,Stanford University Press, Stanford, CaKfomia, 1958.

2. AND , "Reduction of Constrained Maxima to Saddle-PointProblems," in J. NEWMAN (ed.). Proceedings of the Third Berkeley Symposiumon Mathematical Statidics and Pr(^>abaity, University of California Press,Berkeley, California, vol. 5,1-20,1956.

3. M. BELLMOHE, H . J . GEBENBERQ, AND J. J. JABVIS, "Optimal Attack of aNetwork," to be published.

4. R. BBOOKS AND A. GEOFPHION, "Finding Everett's Lagrange Multipliers byLinear Programming," Opns. Res. 14,1149-1153 (1966).

5. C. W. CAEBOLL, "The Created Response Surface Technique for OptimisingNonlinear Restrained Systems," Opns. Res. 9,169-184 (1961).

6. R. CouHANT, Differential and Integral Calculus, Vols. I and II, Interscience,New York, New York, 1937.

7. G. DANTZIG, E . EISENBEBG, AND R . W. COTTLE, "Symmetric Dual NonlinearPrograms," Operations Research Center, University of California, Berkeley(RR) 301.E.R. 172-35 (December, 1962).

8. H. EVERETT III, "Generalized Lagrange Multiplier Method for Solving Prob-lems of Optimum Allocation of Resources," Opns. Res. 11,399-471 (1963).

9. , private conversation.10. A. V. FiAcco AND G. P. McCoRMicK, "The Sequential Unconstrained Mini-

mization Technique for Non-Linear Programming, a Primal-Dual Method,"Mgt. Sci. 10, 360-364 (1964).

11. AND , "The Sequential Unconstrained Minimization Techniquefor Convex Programming with Equality Constraints," Research AnalysisCorp., RAC-TP-155, McLean, Vi i^ ia (1965).

12. G. E. FoRSYTHE, "Computing Constrained Maxima with Lagrange Multi-pliers," SIAM Rev. 3,173-178 (1965).

13. F. J. Goxju), "Extensions of Lagrange Multipliers in Non-Linear Program-ming," Ph.D. Thesis, University of Chicago, Graduate School of Business(1967).

14. H. J. GKEENBEKQ, Ph.D. Thesis, The Johns Hopkins University, Baltimore.Maryland (1968).

15. G. HADLEY, Non-Linear and Dynamic Programming, Addison-Wesley, Read-ing, Massachusetts, 1964.

16. F. JOHN, "Extremum Problems with Inequalities as Subsidiary Conditions,"in Studies and Essays, Courant Anniversary Volume, Interscience, New York,New York, 1948.

17. S. KAPLAN, "Solution of Lorie-Savage and Similar Integer Programming Prob-lems by the Generalized Lagrai^e Multiplier Method," Opns. Res. 14,1130-^11% (1966).

18. H. W. KTJHN AND A. W . TUCKEB, "Non-Linear Programming," in J. NEYMAN(ed.). Proceedings of the Second Berkeley Symposium in Mathematical Sta-tistics and ProbcMLUy, Univeraity of California Pr^ss, Berkeley, California,481-492,1951.

252 M. Mhnore. H. J. Grvenberg, and J. J. Jarvi*

19. L. G. MITTEN, "Composition Principles for Synthesis of Optimal Multi-StageProcesses," Opns. Res. 12,610-619 (1964).

20. T. S. MoTZKiN, "Two Consequences of the Transposition Theorem on LinearInequaUties," Econometrica 19, 184-185 (1951).

21. G. L. NBMHAUSER AND Z. ULLMAN, "A Note on the Generaliaed LagrangeMultiplier Solution to an Integer Programming Problem," Opns. Res. 16,450-453 (1968).

22. W. NtJNN, "The Lagrange Multiplieiv-A Heuristic Presentation," OEG Res.No. 76, CNA (now under University of Rochester), Washington, D.C. (1966).

23. W. P. PiERSKALLA, "Mathematical Programming with Monotonic ConstraintFunctions," Tech. Memo No. 74, Case Institute of Technology, Cleveland,Ohio (1967).

24. H. A. SPANG, III, "A Review of Minimization Techniques for Non-LinearFunctions," SI AM Rev. 4,34^-365 (1966).

25. W. I. ZANGWILL, "Non-Linear Programming Via Penalty Functions," Mgt.Sci. 13, 344-358 (1967).