Discrete search methods for optimizing stochastic systems

DISCRETE SEARCH METHODS FOR OPTIMIZING

STOCHASTIC SYSTEMS

MOHAMED A. AHMED,1 TALAL M. ALKHAMIS1* andDOUGLAS R. MILLER2

1Department of Statistics and Operations Research, Kuwait University, P.O. Box 5969, Safat, Kuwait2ORE Department, Mail Stop 4A6, George Mason University, Fairfax, VA 22030, USA

AbstractÐIn simulation practice, although estimating the performance of a complex stochastic systemis of great value to the decision maker, it is not always enough. For example, a warehouse managermay be interested in ®nding out the probability that all demands are met from on-hand inventoryunder a certain system con®guration of a ®xed safety stock and a ®xed order quantity. But he might bemore interested in ®nding out what values of safety stock and order quantity will maximize this prob-ability. In this paper we develop three strategies of a new iterative search procedure for ®nding the opti-mal parameters of a stochastic system, where the objective function cannot be evaluated exactly butmust be estimated through Monte Carlo simulation. In each iteration, two neighboring con®gurationsare compared and the one that appears to be the better one is passed on to the next iteration. The ®rststrategy of the proposed method uses a single observation of each con®guration in every iteration,while the second strategy uses a ®xed number of observations of each con®guration in every iteration.The third strategy uses sequential sampling with ®xed boundaries. We show that, for all of these threestrategies, the search process satis®es local balance equations and its equilibrium distribution gives mostweight to the optimal point (when suitably normalized by the size of the neighborhoods). We also showthat the con®guration that has been visited most often in the ®rst m iterations converges almost surelyto an optimum solution. # 1998 Elsevier Science Ltd. All rights reserved.

KeywordsÐStochastic optimization, Simulation, Time-homogeneous Markov chains

1. INTRODUCTION

In simulation practice, users are often interested not in the mean response but in the `extreme'response. Often, there is a chance that the response approaches in®nity; for example, in queuingsystems an individual waiting time might be extremely long. Usually such rare extremes are notof interest. Therefore, we may formalize the user's inquiry through estimating the probabilitythat the response variable exceeds a critical value, such as the probability that the waiting timeis longer than some speci®ed value [5]. We consider then the probability involving a perform-ance event of a stochastic system:

b � Pfperformance event of a stochastic system occursg, 0<b<1:

Examples of some applications are:

1. Queueing networks: the probability that the throughput is less than some value. Here theperformance measure may be throughput and the event of interest is de®ned as {throughputless than some value}.

2. Inventory systems: the probability that the demand can be met from on-hand inventory at alltimes. Here the event of interest is de®ned as {meeting all demands from on-hand inventory}.

3. Communication systems: the event of interest, is de®ned as {message delay less than somevalue}.

Hence estimating the probability b is equivalent to estimating the parameter of the Bernoullidistribution where b is de®ned as the probability of success.

Our search problem is to ®nd optimal points in S, where S= {1, 2, . . . , s} is a ®nite set ofsystem con®gurations, with maximum value of b. We will assume that S has multiple optimaand S* is the set containing them, i.e.

Computers ind. Engng Vol. 34, No. 4, pp. 703±716, 1998# 1998 Elsevier Science Ltd. All rights reserved

Printed in Great Britain0360-8352/98 $ - see front matter

PII: S0360-8352(98)00003-5

*Corresponding author.

703

S* � fi:bi � P�performance eventjconfiguration i � � maxj2Sbj,g:

If there is a unique optimum, then S*={i*}, where bi*>bj 8j $ S, j$ i*.

De®ne Xi to be the indicator function of performance event given con®guration i, i.e.

Xi � 1 with probability bi0 with probability 1ÿ bi, 0<bi<1:

�Our goal is to ®nd the optimum con®guration, i*, with max bi 8i $ S. Formally, our optimiz-

ation problem is to choose the optimal system, i.e. the system with the largest bi value.Therefore,

bi*�maxj2Sbj

�maxj2SP�performance eventjconfiguration j�

We focus on the case where the objective function is evaluated by simulation. In such a situ-ation, all the function evaluations will include noise, so conventional (deterministic) optimiz-ation methods cannot be used to solve this problem. Simulation optimization with discretedecision variables is currently a relatively underdeveloped ®eld [1, 2]. However, a fair amount ofliterature is available.

In this article we present and analyze three strategies of a new iterative search method for®nding the optimal Bernoulli parameter. In each iteration two neighboring con®gurations arecompared and the one that appears to be better is passed on to the next iteration. The ®rststrategy of the proposed method uses a single observation at each con®guration in every iter-ation, while the second strategy uses a ®xed number of observations of each con®guration inevery iteration. The third strategy uses sequential sampling with ®xed boundaries. At each iter-ation of the proposed strategies, the selection and acceptance of the next point does not dependon the iteration number, i.e. the search process is time homogeneous. We show that, for all ofthese three strategies, the search process satis®es local balance equations and its equilibrium dis-tribution gives most weight to the optimal point (when suitably normalized by the size of theneighbourhoods). We also show that the con®guration that has been visited most often in the®rst m iterations converges almost surely to a globally optimum solution.

2. SEARCH STRATEGY 1 (TIME-HOMOGENOUS SEARCH PROCESS WITH SINGLE OBSERVATION AND

21 MOVES)

In this section, we present the ®rst search strategy using a single observation of each con®gur-ation at each iteration and prove that it converges almost surely to a global solution. AssumeS= {1, 2, . . . , s} is a non-empty discrete ®nite set of con®gurations and the search is conductedby picking an initial point in S and then comparing a neighboring point according to the follow-ing de®nition.

De®nition 2.1. For each i $ S, the neighborhood of con®guration i is:

N�i � �fiÿ 1, i� 1g for i 6� 1 or i 6� s,f2g for i � 1,fsÿ 1g for i � s:

8<:Given i $ S, a candidate is selected from N(i) such that the probability of selecting a neighborj $ N(i) is equal to Gij, where

Gij �1

jN�i �j for j 2 N�i�,

Gij � 0 otherwise:

8<:A candidate neighbor j $ N(i) is accepted and a move is performed from i to j if Xj>Xi, where

M. A. Ahmed et al.704

Xi � 1 with probability bi,0 with probability 1ÿ bi:

�Accordingly the acceptance probability, Aij, the probability of accepting the con®guration j,once it is generated from con®guration i, is de®ned as follows:

Aij � P�Xi<Xj � � P�Xi � 0, Xj � 1� � �1ÿ bi �bjLet {Ik, k= 0, 1, . . .} be the stochastic process with state space S where Ik is the current stateof the search process at iteration k. The details of the algorithm for ®nding the con®gurationwith optimal b are given below. Note that for all i, l, Vl(i) is the number of times the sequence{Ik} has visited state i during the ®rst l iterations of the algorithm and Ik

* is the state that the al-gorithm has visited most often in the ®rst k iterations.

Algorithm 1:Step 0: Select a starting point I0$S. Let V0(I0) = 1 and V0(x) = 0 for all x $ S, x$I0. Let

k = 0 and Ik*=I0. Go to step 1.

Step 1: Given Ik=i, choose a candidate Jk from N(i) with probability distribution:P[Jk=jvIk=i] = Gij, j $ N(i).

Step 2: Given Jk=j, set

Ik�1 � Jk if Xi<Xj,Ik if XirXj:

�Step 3: Set k = k+ 1, Vk(Ik) = Vk ÿ 1(Ik) + 1 and Vk(x) = Vk ÿ 1(x), for all x $ S and x$Ik.

If Vk(Ik)/vN(Ik)v>Vk(Ik ÿ 1* )/vN(Ik ÿ 1

* )v then let Ik*=Ik; otherwise let Ik

*=Ik ÿ 1* . Go to step 1.

One of the following stopping criteria may be used for the above algorithm: Stop the searchprocess after performing a predetermined number of iterations or when the most visited statehas not been changed for a predetermined number of iterations.

The random process {Ik, k= 0, 1, . . .} produced by the above algorithm is a discrete time-homogenous Markov chain with states space S and its state transition probabilities are given by

Pi,j � P�Ik�1 � jjIk � i � � GijAij �

1

jN�i�j �1ÿ bi �bj j 2 N�i�,

1ÿXl2N�i �

1

jN�i �j �1ÿ bi �bl j � i,

0 otherwise:

8>>>>>><>>>>>>:Note that in algorithm 1, the random process {Ik} does not converge almost surely to an el-ement of the set S*. In fact the random process {Ik} is irreducible and may visit each element ofthe set S in®nitely often. Instead we will show in Theorem 2.1 that the sequence {Ik

*} convergesalmost surely to S*.

The digraph of the search process, using N(i) as de®ned in de®nition 2.1, is shown in Fig. 1.As can be seen from Fig. 1, the transition diagram for the search process is a tree (no cycles).Therefore the equilibrium distribution of {Ik} must have local balance de®ned as follows [6]:

Fig. 1. Transition digraph for the Markov chain {Ik} generated by algorithm 1.

Discrete search methods for optimizing stochastic systems 705

pipi,i�1 � pi�1pi�1,j i � 1, 2, . . . , sÿ 1,

which implies

pipi�1� Pi�1,j

pi,i�1� �1=jN�i� 1�j��1ÿ bi�1�bi�1=jN�i�j��1ÿ bi �bi�1

� jN�i�j�bi=1ÿ bi �jN�i� 1�j�bi�1=1ÿ bi�1�

,

therefore

pi � cjN�i �j�bi=1ÿ bi � i � 1, 2, . . . , s,

is the equilibrium distribution for the search process {Ik, k = 0, 1, . . .}, where c is normalizing

constant � 1Xj2SjN� j�j�bj=1ÿ bj �

:

Now we discuss the convergence of search strategy 1 as described in algorithm 1. Note that for

all i $ S and kr0, Vk(i) is the number of times that the Markov chain {Ik, k = 0, 1, . . .} has

visited state i in the ®rst k iterations and Ik* is the state that the search process has visited most

often after k iterations (normalized by size of neighborhoods). Letting VÄk(i) = Vk(i)/vN(i)v thenIk

*={i:VÄk(i)=Maxj2SVÄk(j)}. It is clear that the value of Ik*, say j changes only when the search

process visits point i and VÄk(i)>VÄk(j). Before showing that the sequence {Ik*} converges, we

need the following proposition and corollary (see Ref. [3]).

Proposition 2.1. (Strong law of large number (SLLN) for Markov chains). Let {Ik, k = 0, 1,

. . .} be a Markov chain with irreducible transition matrix P. Let g:S 4 R be a real valued function

on the ®nite state space S. Then limMÿ41�1=M�SMk�1g�Ik� � Si2Spig(i) almost surely, where

p = (p1, p2, . . . , ps) is the equilibrium distribution for P.

Corollary 2.1. Let g(Ik) = 1{j}(Ik) where

1f jg�Ik� � 1 if Ik � j0 otherwise,

�then P(limMÿ41(1/M)ak = 1

M 1{j}(Ik) = pj) = 1. In other words, if we observe the process {Ik},

the average number of visits to state j during the ®rst M iterations converges to pj for large M.

Note that VM( j ) = ak = 1M 1{j}(Ik(o)) which implies that VM(j)/M asymptotically equals pj. Now

we state and prove the convergence theorem for algorithm 1.

Theorem 2.1. The sequence {Ik*} generated by algorithm 1 converges almost surely to S*; i.e.

P{limkÿ41 1fS*g(Ik*) = 1} = 1.

Proof. Suppose that i $ S* and j ( S*, then bi>bj and bi/1ÿ bi>bj/1ÿ bj. Then by the de®nition

of pj, we have pi/vN(i)v>pj/vN( j )v. Since i $ S* and j ( S* are arbitrary, this shows that if pi/vN(i)v=maxl2Spl/vN(l)v, then i $ S* for all i $ S. Let Z denote the di�erence between the best and

the 2nd best of the equilibrium probabilities weighted by the size of the neighborhood, i.e.

Z=mini2S* (pi/vN(i)v)ÿmax

j2�S*(pj/vN(j)v) where S* [S denotes the set of global maximizers and

S*=Sÿ S*. Now de®ne A = {o: limkÿ41Vk(i, o)/kvN(i)v = pi/vN(i)v for all i $ S}; then P(A) = 1,

since P(limkÿ41Vk(i)/kvN(i)v = pi/vN(i)v) = 1 (by the SLLN for the regular Markov chains). So,

for almost all o, there exist ki(o) such that P(Vk(i)/kvN(i)vÿ pi/vN(i)v < Z/2, k>ki(o) 8i $ S) = 1.

Therefore, since vSv <1, P(Ik*(o) $ S*, k>ko

*=maxi ki(o)) = 1. Then

P(limkÿ411fS*g(Ik*) = 1) = 1 and this completes the proof. q

Corollary 2.2. If there is only one point in S*, i*, then Ik*4i* as k41 almost surely.

Illustrative example 2.1. Consider a test case with 4 points. In this example S= {1, 2, 3, 4},


S*={1, 4} and b1=0.9, b2=0.5, b3=0.1 and b4=0.9. For search strategy 1, the equilibriumprobability distribution (p1, p2, p3, p4) = (0.445, 0.099, 0.011, 0.445), i.e. points 1 and 4 receiveequal probability over an in®nite horizon.

3. SEARCH STRATEGY 2 (TIME-HOMOGENOUS SEARCH PROCESS WITH N OBSERVATIONS AND

GENERAL SYMMETRIC NEIGHBORHOODS)

In this section, we show how the method presented in Section 2 can be extended using a ®xednumber of observations of each con®guration at each iteration. Given we start our search at sol-ution point i $ S, a candidate j is selected from N(i) with probability Gij as de®ned before butwith more general neighborhoods. In this search strategy our neighborhoods satisfy the follow-ing assumption:

Assumption 3.1. The neighbor system N is symmetric i.e. j $ N(i)\ i $ N(j).

At each iteration a sample of n pairs of observations is taken from i and j, i.e. (Xi1, Xj1), (Xi2,Xj2), . . . , (Xin, Xjn), a candidate neighbor j $ N(i) is accepted and a move is performed from i toj if the observation from j dominates the observation from i in each pair, i.e. if \nm�1{Xim<Xjm}occurs. In this case the acceptance probability Aij is de®ned as follows

Aij � P

�\nm�1fXim<Xjmg

�� 1ÿ bi �bj �n:

The statement of the algorithm for search strategy 2 is as follows:Algorithm 2:Step 0: Select a starting point I0$S. Let V0(I0) = 1 and V0(x) = 0 for all x $ S, x ( I0. Let

k = 0 and Ik*=I0. Go to step 1.

Step 1: Given Ik=i, choose a candidate Jk from N(i) with probability distribution:P[Jk=jvIk=i] = Gij, j $ N(i).

Step 2: Sample n observations from i and j (Xi1, Xj1), (Xi2, Xj2), . . . , (Xin, Xjn) and set

Ik�1 � Jk if \nm�1fXim<Xjmg, Ik�1 � Ik otherwise:

Step 3: Set k = k+ 1, Vk(Ik) = Vk ÿ 1(Ik) + 1 and Vk(x) = Vk ÿ 1(x) for all x $ S and x$Ikif Vk(Ik)/vN(Ik)v>Vk(Ik ÿ 1

* )/vN(Ik ÿ 1* )v then let Ik

*=Ik; otherwise let Ik*=Ik ÿ 1

* . Go to step 1.The random process {Ik} produced by algorithm 2 is a discrete-time homogenous Markov

chain de®ned over states S and its state transition probabilities are given by

Pi,j � P�Ik�1 � jjIk � i � � GijAij �

1

jN�i�j ��1ÿ bi �bj�n j 2 N�i�,

1ÿXl2N�i �

1

jN�i �j ��1ÿ bi �bl�n j � i,

0 otherwise:

8>>>>>><>>>>>>:The digraph of the Markov process {Ik} produced by algorithm 2 is not necessarily a tree andlocal balance must be demonstrated. It will hold if there is a probability vector that satis®es thelocal balance equations; pi(n)pi,j(n) = pj(n)pj,i(n).

Consider

pi�n� � jN�i �j�bi=1ÿ bi �nXj2SjN� j�j�bj=1ÿ bj �n

i � 1, 2, . . . , s:

We will show in the next lemma that {pi(n), i $ S} is the stationary probability distribution forthe Markov chain Ik generated by algorithm 2.


Lemma 3.1. The vector p(n) consisting of pi(n) represents the stationary distribution for theMarkov chain {Ik} generated by algorithm 2.

Proof. It follows from assumption 1 that for every pair (i, j) of neighbors, j $ N(i),

pi�n�pj�n� �

jN�i �j�bi=1ÿ bi �njN� j�j�bj=1ÿ bj �n

� �1=jN� j��j��1ÿ bj �bi �n�1=jN�i��j��1ÿ bi �bj �n

� pj,i�n�pi,j�n�

and that pi(n)pi,j(n) = pi(n)pj,i(n) 8i, j $ S, hence pi(n) satis®es the local balance equations, there-fore local balance exists and p(n) is the equilibrium probability vector. Now we discuss the con-vergence of the search strategy 2. q


P{limkÿ411fS*g(Ik*) = 1} = 1.

Proof. Follows same argument as the proof for Theorem 2.1. q

Illustrative example 3.1. Consider example 2.1. Let ci=bi/1ÿ bi, i = 1, 2, 3, 4. Then for searchstrategy 2, the equilibrium probability distributions are as follows:

p1 � 1

1� 2�c2=c1�n � 2�c3=c1�n � �c4=c1�n, p2 � 1

1� 1=2�c1=c2�n � �c3=c2�n � 1=2�c4=c2�n,

p3 � 1

1� 1=2�c1=c3�n � �c2=c3�n � 1=2�c4=c3�nand p4 � 1

1� �c1=c4�n � 2�c2=c4�n � 2�c3=c4�n:

Table 1 gives the numerical values of the equilibrium distributions for di�erent values of n. Asn 41 the equilibrium distributions go to (0.5, 0, 0, 0.5).

Illustrative example 3.2. Consider the previous illustrative example with b4 perturbed up slightlyto be equal 0.91. Table 2 gives the numerical values of the equilibrium distributions for di�erent

Table 1. The equilibrium distributions for illustrative example 3.1 with di�erent values of n

n p1 p2 p3 p4

1 0.44506 0.09890 0.01098 0.445062 0.49383 0.01219 0.00015 0.493833 0.49931 0.00137 0.00001 0.499314 0.49992 0.00016 0.00000 0.499925 0.49999 0.00002 0.00000 0.49999

Table 3. The equilibrium distributions for illustrative example 3.2 (b4=0.89) with di�erent values of n

n p1 p2 p3 p4

1 0.46600 0.10356 0.01151 0.418935 0.63003 0.00002 0.00000 0.3699510 0.74361 0.00000 0.00000 0.2563920 0.89375 0.00000 0.00000 0.1062530 0.96063 0.00000 0.00000 0.0393750 0.99515 0.00000 0.00000 0.00485100 0.99998 0.00000 0.00000 0.00002

Table 2. The equilibrium distributions for illustrative example 3.2 (b4=0.91) with di�erent values of n

n p1 p2 p3 p4

1 0.42187 0.09375 0.01042 0.473965 0.35846 0.00001 0.00000 0.6415310 0.23792 0.00000 0.00000 0.7620820 0.08881 0.00000 0.00000 0.9111930 0.02953 0.00000 0.00000 0.9704750 0.00296 0.00000 0.00000 0.99704100 0.00001 0.00000 0.00000 0.99999


values of n. As n 41 the equilibrium distributions go to (0, 0, 0, 1). On the other hand,

Table 3 gives the numerical values of the equilibrium distributions for di�erent values of n with

b4 perturbed down slightly to be equal 0.89. As n 41 the equilibrium distributions go to (1, 0,

0, 0).

Two counter examples: the above results are very delicate and many search strategies will not

give desired results. We will illustrate this idea using two examples.

Counter Example 3.3. (Fixed sample size; move on ties.) For the previous illustrative example, if

we use the acceptance criteria such that: move from the current point i to a new solution point j

if \m = 1n {XimRXjm}, i.e. move on ties. In this case the search process will not give correct

answers; it will favor point 4. Using 21 neighborhood search, then

Aij=[P(XiRXj)]n=[1ÿ P(Xi>Xj)]

n=[1ÿ (1ÿ bj)bi]n and

Pi,j �

1

jN�i �j �1ÿ �1ÿ bj �bi�n j 2 N�i�,

1ÿXl2N�i �

1

jN�i�j �1ÿ �1ÿ bl �bi�n j � i,

0 otherwise:

8>>>>>><>>>>>>:The equilibrium distribution vector p with pP = p is given in Table 4 for di�erent values of n.

Note that the search process gives unit probability to point 4 as n41 and favors point 4 for

all values of n.

The value of b4 could be perturbed to be inferior point but it would still be visited proportion

of time equal to 1. Table 5 presents the numerical values of the equilibrium distributions for

di�erent values of n and b4=0.8 for the search process that moves on ties.

Counter Example 3.4. (Fixed sample size; not unanimous agreement.) Let

Yn �Xnm�1

Sm,

Table 4. The equilibrium distributions for counter example 3.1 with di�erent values of n

n p1 p2 p3 p4

1 0.21859 0.25311 0.14654 0.381765 0.05752 0.00747 0.00049 0.9345210 0.00377 0.00003 0.00001 0.9961915 0.00023 0.00000 0.00000 0.9997720 0.00001 0.00000 0.00000 0.9999925 0.00000 0.00000 0.00000 1.00000

Table 5. The equilibrium distributions for counter example 3.1 (b4=0.80) with di�erent values of n

n p1 p2 p3 p4

1 0.24991 0.28937 0.16753 0.291385 0.29757 0.03871 0.00252 0.6612010 0.16819 0.00142 0.00001 0.8303815 0.08354 0.00005 0.00000 0.9164120 0.03941 0.00000 0.00000 0.96059100 0.00000 0.00000 0.00000 1.00000


where

Sm � 1 if Xj > Xi

0 if XjRXi

�The search procedure of search strategies 1 and 2 move from point i to a new point j using a

unanimous agreement of n comparisons. Then using the unanimous decision, we move to pointj if Yn=n. Now, consider the following decision rule: move from point i to a new point j ifYn>n/2, i.e. move on majority. In this case the acceptance probability is given by,Aij=P(Yn>n/2). For the 4 point example with neighbourhood equal to 21, it is easy to calcu-late P and solve for pP = p. Table 6 presents the numerical values of the equilibrium distri-butions for di�erent values of n. It is clear that p does not give equal weight to point 1 andpoint 4. As n 41 the equilibrium distributions go to (1, 0, 0, 0), i.e. for large n, p gives unitprobability to point 1.

4. SEARCH STRATEGY 3 (TIME-HOMOGENOUS SEARCH PROCESS WITH SEQUENTIAL SAMPLING)

In this section, we present the third search strategy using sequential sampling with ®xedboundaries and prove that it converges almost surely to a global solution. For each i, j $ S gen-erate sample pairs sequentially, say (Xi1, Xj1), (Xi2, Xj2), . . . and let Sm denote the sign of (Xim,Xjm), such that

Sm ��1 if Xim<Xjm

0 if Xim � Xjm

ÿ1 if Xim > Xjm:

8<:Let

Yn �Xnm�1

Sm �Xnm�1

1fXim<Xjmg ÿXnm�1

1fXim>Xjmg

In each iteration of the search process we compare the current point i with an alternative point jas follows: conduct an experiment or a simulation to obtain samples from both i and j until the®rst time that Yn falls outside a given bounds 2b; if it falls out below, the alternative point isrejected where as if it falls out above, the alternative point is accepted.

The process {Yn, n = 1, 2, . . .}, Y0=0 is a random walk that moves a unit step in the positivedirection with probability p, moves a unit step in the negative direction with probability q, doesnot move with probability r = 1ÿ pÿ q and absorbing barriers at +b and ÿb (Fig. 2). Where,p = P(Xim<Xjm) = (1ÿ bi)bj and q= P(Xim>Xjm) = (1ÿ bj)bi.

Lemma 4.1. Using search strategy 3, the acceptance probability is given by

Aij ��bj=1ÿ bj �b

�bj=1ÿ bj �b � �bi=1ÿ bi �b

Table 6. The equilibrium distributions for counter example 3.4 with di�erent values of n

n p1 p2 p3 p4

4 0.54392 0.00218 0.00000 0.453906 0.58568 0.00040 0.00000 0.413938 0.62100 0.00007 0.00000 0.3789310 0.65358 0.00001 0.00000 0.3464112 0.68098 0.00000 0.00000 0.3190214 0.70689 0.00000 0.00000 0.2931116 0.73034 0.00000 0.00000 0.2696618 0.83057 0.00000 0.00000 0.16943


Proof. Using the theory of random walk (see Ref. [4]), the probability that the process starting

at zero will be absorbed at +b is given by

P�YT � �bjT � minfn:Yn � �b or ÿ bg� � 1ÿ �q=p�b1ÿ �q=p�2b �

1ÿ �q=p�b�1ÿ �q=p�b��1� �q=p�b� �

1

1� �q=p�b

Hence

Aij � 1

1� ��1ÿ bj �bi=�1ÿ bi �bj �b� 1

1� �1ÿ bj=bj �b�bi=1ÿ bi �b� �bj=1ÿ bj �b�bj=1ÿ bj �b � �bi=1ÿ bi �b

:

De®ne {Ik, k = 0, 1, . . .} to be the stochastic process where Ik$S is the current state of the

search process at iteration k. The details of the algorithm for ®nding the con®guration with op-

timal b are given below. Note that for all i, l, Vl(i) is the number of times the sequence {Ik} has

visited state i during the ®rst l iterations of the algorithm and Ik* is the most frequently visited

state. qAlgorithm 3: Algorithm 3 is the same as algorithm 1 and 2 except for step 2 which is as fol-

lows:

Step 2: Given Jk=j, set

Ik�1 � Jk with probability Aij,Ik with probability 1ÿ Aij,

�where

Aij � 1

1� ��1ÿ bj �bi=�1ÿ bi �bj �b

Our implementation of step 2 is as follows: ®rst we draw sample pairs from both current point i

and the new point j. We update the statistics Yn; if Yn equals +b, then Ik + 1=Jk. If Yn equals

ÿb, then Ik + 1=Ik.

Fig. 2. A typical sample path of the random walk {Yn}.


4.1. The stationary behaviour of algorithm 3

For each i $ S de®ne

pi�b� � jN�i �j�bi=1ÿ bi �bXj2SjN� j�j�bj=1ÿ bj �b

i � 1, 2, . . . , s:

We will show in the next Lemma that {pi(b), i $ S} is the stationary probability distribution for

the Markov chain Ik generated by algorithm 3.

Lemma 4.2. The vector p(b) consisting of p(b) represents the stationary distribution for the

Markov chain {Ik}.

Proof: It is clear that pi(b)>0 for each i $ S and that ai $ Spi(b) = 1. For every pair (i, j) of

neighbors, j $ N(i),

pi�b�pj�b� �

jN�i �j�bi=1ÿ bi �bjN� j�j�bj=1ÿ bj �b

� 1=jN� j�j��bi=1ÿ bi �b�=��bi=1ÿ bi �b � �bj=1ÿ bj �b�1=jN�i �j��bj=1ÿ bj �b�=��bi=1ÿ bi �b � �bj=1ÿ bj �b�

� GjiAji

GijAij� pji�b�

pij�b�

and that pi(b)pij(b) = pj(b)pji(b) 8i, j $ S, hence pi(b) satis®es the local balance equations, there-

fore local balance exists and p(b) is the equilibrium probability vector. q


P

�limkÿ41

1fS*g�I*k � � 1

�� 1:

Proof. Follows the same argument as the proof for Theorem 2.1. q

Illustrative example 4.1. Consider example 2.1. Then for search strategy 3, the equilibrium prob-

ability distributions are as follows:

p1 � 1

1� 2�c2=c1�b � 2�c3=c1�b � �c4=c1�b, p2 � 1

1� 1=2�c1=c2�b � �c3=c2�b � 1=2�c4=c2�b,

p3 � 1

1� 1=2�c1=c3�b � �c2=c3�b � 1=2�c4=c3�band p4 � 1

1� �c1=c4�b � 2�c2=c4�b � 2�c3=c4�b

When b = 1, the equilibrium probability distribution (p1, p2, p3, p4) = (0.445, 0.099, 0.011,

0.445), i.e. points 1 and 4 receive equal probability over an in®nite horizon. As b 41 the equi-

librium distributions go to (0.5, 0, 0, 0.5). Furthermore, If we perturb b4 slightly, say 0.91, then

as b 41 the equilibrium distributions go to (0, 0, 0, 1). On the other hand, if we perturb b4slightly, say 0.89, then as b 41 the equilibrium distributions go to (1, 0, 0, 0).

Counter Example 4.2. (Sequential sampling with ®xed boundary, move on ties.) For the previous

illustrative example, consider the probability of moving in the positive direction as

p = P(XiRXj) = 1ÿ P(Xi>Xj) = 1ÿ (1ÿ bj)bi, i.e, move on ties.

In this case the search process will not give correct answers; it will favour either point 1 or

point 4. In fact it will favor point 4. Using21 neighborhood search, then

Aij � 1

1� ��1ÿ bj �bi=1ÿ ��1ÿ bi �bj ��2

and


Pi,j �

1

jN�i�jAij j 2 N�i �,

1ÿXl2N�i �

1

jN�i�jAil j 2 �i�,

0 otherwise:

8>>>>>><>>>>>>:The equilibrium distribution vector p with pP = p is given in Table 7 for di�erent values of b.Note that the search process gives unit probability to point 4 as n 41.

The value of b4 could be perturbed to be inferior point but it would still be visited proportionof time equal to 1. Table 8 presents the numerical values of the equilibrium distributions fordi�erent values of b and b4=0.8.

Table 8. The equilibrium distributions for counter example 4.2 (b4=0.80) with di�erent values of b

b p1 p2 p3 p4

1 0.24991 0.28937 0.16753 0.293182 0.09394 0.12141 0.07845 0.706203 0.01556 0.02277 0.01666 0.945004 0.00207 0.00332 0.00267 0.991955 0.00028 0.00047 0.00041 0.998856 0.00004 0.00007 0.00006 0.999837 0.00001 0.00001 0.00001 0.999988 0.00000 0.00000 0.00000 1.00000

Fig. 3. Graph of bi.

Table 7. The equilibrium distributions for counter example 4.2 with di�erent values of b

b p1 p2 p3 p4

1 0.21859 0.25311 0.14654 0.381762 0.10162 0.12208 0.07333 0.702973 0.02785 0.03600 0.02326 0.912884 0.00620 0.00856 0.00591 0.979335 0.00132 0.00193 0.00141 0.995346 0.00028 0.00043 0.00033 0.998957 0.00006 0.00010 0.00008 0.999768 0.00001 0.00002 0.00002 0.999959 0.00000 0.00001 0.00000 0.9999910 0.00000 0.00000 0.00000 1.00000


5. COMPUTATIONAL RESULTS

In this example we implement each of the three search strategies discussed earlier to solve asimple discrete stochastic optimization problem with 50 con®gurations. Consider the followingoptimization problem, ®nd points in S with maximum value of b:

maxi2S bi,

where S= {0, 1, . . . , 49}

Fig. 4. Optimization trajectory for test case 1 using search strategy 1 based on 100 replicates.



bi � 0:95ÿ 0:05� i for i � 0, 1, . . . , 9,bi � 0:15� �0:8=49� � i for i � 10, 11, . . . , 49

�See Fig. 3 for the graph of the function bi. To make the search more di�cult, this objective

function has multiple optima. Note that we have two global maxima at i = 0 and i= 49 with

values equal to 0.95. We will apply search strategies 1, 2 and 3 to solve this optimization pro-

blem. Figures 4 and 5 show the results obtained by applying search strategy 1 and search strat-

egy 2. In particular, we use a ®xed sample of size 1 and a neighborhood of 21 for search

strategy 1. For search strategy 2, we use a ®xed sample of size 3 and neighborhood of 23.

Figures 4 and 5 show the average value of the objective function of the most visited con®guration


Table 9. The performance of search strategy 1 and 2 in terms of the number of observations

Number of observations 95% con®dence interval of the objective function value

search strategy 1 search strategy 2

200 0.62520.095 0.48420.098400 0.65020.094 0.52920.098600 0.67420.092 0.56820.097800 0.72320.088 0.58620.0971000 0.74320.086 0.62020.0952000 0.81120.077 0.66520.0934000 0.86320.067 0.77220.0826000 0.90420.058 0.80820.0778000 0.91620.054 0.81920.07510000 0.92620.051 0.82320.09815000 0.92620.051 0.83820.07220000 0.92920.051 0.84020.07425000 0.93020.050 0.87820.06430000 0.93020.050 0.90120.058


out of 100 replications that were used versus the number of iterations. Figure 6 shows the resultobtained by applying search strategy 3 using ®xed boundary b = 3 and neighborhood of25.

To analyze the relative rate of convergence of the proposed methods in terms of the numberof observations required to achieve various uniform levels of con®dence, we obtained 95% con-®dence interval of the objective function value of the most visited state versus the number of ob-servations of the objective value. Table 9 shows these results for search strategy 1 and 2. Weconclude that search strategy 1 converges faster than search strategy 2. For search strategy 3,we also obtained 95% con®dence interval of the objective function value of the most visitedstate versus the number of observations of the objective value. Strategy 3 requires very largenumber of observations to converge to an optimal solution. We conclude that search strategy 1and 2 converge much faster than search strategy 3 in terms of the number of observations.

6. CONCLUSION

We have presented new search methods for ®nding the optimal Bernoulli parameter. There isa considerable interest in this problem in ®elds of operations research and management science.Many times an experimenter is interested in the probability that a performance event of a sto-chastic system occurs. The methods presented in this paper have three di�erent strategies forsearching the optimal Bernoulli parameter. The ®rst and second strategies require only a ®xednumber of observations to be made at every iteration. The third strategy uses a sequentialsampling with ®xed boundaries. These strategies converges almost surely to a global solution ofthe underlying optimization problem. Computational experience shows the e�ciency of the pro-posed search methods.

This research was supported by the o�ce of the Vice President for Scienti®c Research,Kuwait University, under project number SS044. The authors wish to thank the referees, the as-sociate editors and the editor for their conscientious reading of this paper and their numeroussuggestions for improvement which were extremely useful and helpful in modifying the manu-script.

AcknowledgementsÐThis research was supported by the o�ce of the Vice President for Scienti®c Research, KuwaitUniversity, under project number SS044. The authors wish to thank the referees, the associate editors and the editor fortheir conscientious reading of this paper and their numerous suggestions for improvement which were extremely usefuland helpful in modifying the manuscript.

REFERENCES

1. Andradottir, S A method for discrete stochastic optimization. Manage Sci, 41(12), 1995, 1946±1961.2. Andradottir, S Global search for discrete stochastic optimization. SIAM J Opt, 6(2), 1996, 513±530.3. Cinlar E. Introduction to stochastic processes. Prentice-Hall, 1975.4. Feller W. An introduction to probability theory and its application, vol. II. New York: John Wiley and Sons, 1967.5. Kleijnen JPC. Statistical tools for simulation practitioners. Marcel Dekker, 1987.6. Ross SM. Introduction to probability models. 4th ed. San Diego: Academic Press, 1989.


Discrete search methods for optimizing stochastic systems

Documents

Transcript of Discrete search methods for optimizing stochastic systems