Testing Dynamic Adaptation in Distributed Systems1

Karun N. Biyani Sandeep S. KulkarniDepartment of Computer Science and Engineering

Michigan State UniversityEast Lansing, MI - USA

AbstractMany software systems undergo dynamic adaptation in

response to change in requirements or frequent changes intheir usage context. Before adaptation, the system satis-fies the behavior of the old program (i.e., program beforeadaptation), and after adaptation the system satisfies thebehavior of the new program (i.e., program after adapta-tion). During adaptation, the behavior of the old programand the behavior of the new program may overlap. Exist-ing algorithms to test distributed systems do not deal withadaptive systems. In this paper, we discuss testing of systemduring adaptation using predicate detection techniques.

1 IntroductionMany computing systems are required to provide unin-

terrupted service. These systems need to be upgraded asbugs are discovered, and adapt as environment or require-ment changes. If these systems are to provide continuousoperation, then any change in the system needs to be donewithout completely stopping the system. Such a change iscommonly referred to as dynamic adaptation [1, 2].

The verification and testing of these adaptive systemsimpose new challenges as the code and, in many cases thebehavior, of the program is changing. Consider an adap-tation from an old program A to a new program B. Typ-ically, during such adaptation the system is overlapped bythe behavior of both programs A and B. After adaptation,the system satisfies the behavior of program B. One of theaspect of verifying adaptation is to ensure syntactic compat-ibility, i.e., if one component is replaced by another duringadaptation, then both components should have compatibleinterfaces and types. However, in addition to syntactic com-patibility, one needs to verify the behavioral changes in thesystem due to adaptation.

In order to specify and verify the behavior of the systemduring dynamic adaptation, we presented an approach basedon adaptation lattice in [3]. Due to complexity of the adap-tive systems, the verification is often done on an abstractmodel of the system. To gain assurance about the adapta-tion implementation, verification is often supplemented bytesting of the concrete system.

Predicate detection is a common approach used in testingand debugging of distributed systems, as many problems

1 Email. {biyanika,sandeep}@cse.msu.eduTel. +1-517-355-2387. Fax. +1-517-432-1061.This work was partially sponsored by NSF CAREER CCR-0092724,

DARPA Grant OSURS01-C-1901, ONR Grant N00014-01-1-0744, and agrant from Michigan State University.

in distributed systems can be formulated as an instance ofGlobal Predicate Evaluation (GPE) [4]. Due to overlappingbehavior of the old program and the new program duringadaptation, the algorithms for predicate detection (e.g. [4–7]) cannot be applied directly. Specifically, these algorithmsdo not deal with the system whose code is being changed.

With this motivation, we extend the existing algorithmsto test the system during adaptation. In particular, we clas-sify the predicates to be detected during adaptation into twotypes: (i) adaptation-stable predicates, and (ii) adaptation-transient predicates. In this paper, we show how existingalgorithms can be modified to detect adaptation-transientpredicates. In the technical report of this paper [8] we dis-cuss detection of adaptation-stable predicates, and also de-scribe an approach to reduce the cost of testing.Organization of the paper. In Section 2, we discuss howwe model adaptive systems. In Section 3, we present al-gorithms to test adaptive systems. Finally, we conclude inSection 4.

2 Computation Model of AdaptationIn this section, we describe how we model the adaptive

programs.2.1 System Model

A (distributed) program P consists of a set of n pro-cesses {p1, p2, ..., pn} communicating via asynchronousmessages on interprocess channels. We do not assume thechannels to be FIFO (unless the application requires so). Aprocess pi is specified by a set of local constants, a set oflocal variables and a finite set of actions. A state of processis defined by the value of each variable of pi. The executionof a process consists of a sequence of events. An event isthe execution of some process action. An event is one of thethree types: local (or internal) event, send event or receiveevent. An action is an atomic and terminating statementthat updates zero or more variables, or sends zero or moremessages. To model adaptation, we introduce a special typeof action, adaptive action. The execution of adaptive ac-tion is a special type of event known as atomic adaptation.Each adaptive action is associated with some process, andthe statement of the adaptive action modifies the set of con-stants, variables and actions associated with the process.Distributed Computation. A distributed computation r ofa program P describes a single execution of P by a collec-tion of traces r[i] for each process pi. Each r[i] is a finitealternating sequence of states and events. For example, thetrace of process pi is s0

i e1i s

1i e

2i ..., where sk

i denotes the lo-cal state of pi immediately after event ek

i , and s0i denotes the

Second International Workshop on Automation of Software Test (AST'07)0-7695-2971-2/07 $20.00 © 2007

initial state before any actions are executed. A distributedcomputation is commonly depicted using a space-time dia-gram as shown in Figure 1.

2

1e 3

1e 4

1e

1

2e

1

3e 2

3e 3

3e

2

2e 3

2e 4

2e

1

1e

1p

2p

3p

Figure 1: Space-time diagram.

Formally, a distributed computation is a partially orderedset defined by the pair (E,→), where E is the set containingall events that are executed and → is the happened-beforerelation [9] that defines the causal precedence relation onevents (or states). The happened-before relation on states isdefined as follows:Causal Precedence. The state s causally precedes the statet (denoted as s → t) if and only if one of the followingholds: (i) if s and t are states on the same process and soccurred before t, (ii) if action following s is the send ofa message and action before t is the corresponding receive,(iii) there exists a state x such that s → x and x → t.

If for two events e1 and e2, neither e1 → e2 nor e2 → e1,then e1 and e2 are said to be concurrent.Global State and Cut. We defined a global state as a stateof all the processes and channels in the system. We, how-ever, ignore the channel states and represent global state asn-tuple of local states (sc1

1 , ..., scnn ).

A cut C associated with a global state g is a set of events,one event per process, such that event e ∈ C if and only ifthe process state immediately after event e is a part of g. Aglobal history H associated with the cut C (or correspond-ing global state g) of P is defined as the subset hc1

1 ∪...∪hcnn ,

where hci

i is the local history of process pi containing firstci (i.e., e1

i e2i ...e

ci

i ) events.Consistent Cut and Consistent Global State. A cut Cis consistent if for all events e in the corresponding globalhistory H , we have (e ∈ C) ∧ (e′ → e) ⇒ e′ ∈ H . Aglobal state g is consistent if the cut corresponding to it isconsistent.

Several possible global executions correspond to a givendistributed computation r. We call each total order of ras an observation. In other words, a sequence of consis-tent global states g0g1g2g3... is an observation, where g0

denotes the initial global state (s01, ..., s

0n), and each global

state gi is obtained from previous state gi−1 by some pro-cess executing a single event. For two such global statesgi−1 and gi, we say that gi−1 leads-to gi. The set of all con-sistent global states of a computation along with the leads-torelation defines a lattice of global states or computation lat-tice. A path in the lattice corresponds to an observation, andeach observation has a corresponding path in the lattice.

2.2 Adaptation Overview

The key to verifying adaptation in distributed programsis to ensure that the individual atomic steps in the adaptation

occur in an appropriate order and instances when they oc-cur are “safe”, i.e., the specification during adaptation [3] issatisfied. Towards this end, we divide this multi-step adap-tation into multiple atomic adaptations each occurring atonly one process.

We define a system before adaptation as old program anda system after adaptation as new program. Each atomicadaptation modifies the system into an intermediate pro-gram. The first atomic adaptation modifies the old programinto the first intermediate program. Similarly, other atomicadaptations modifies one intermediate program into the nextintermediate program. The last atomic adaptation resultsinto the new program.

Now, consider a distributed computation Ar that de-scribes an execution of the adaptive distributed program.Let σ = g0, g1, g2, ... be one of the observations corre-sponding to Ar. Let gi be the global state reached imme-diately after the execution of first atomic adaptation, andgj(j ≥ i) be the global state reached immediately after theexecution of last atomic adaptation. For all prefixes of σending in any state before state gk(k < i) the system sat-isfies the old program behavior, and for all suffixes of σstarting in any state gk(k ≥ j) the system satisfies the newprogram behavior. For the interval gi, ..., gj−1 the systemsatisfies specification during adaptation, which is specifiedin terms of the changing state space; i.e., as a set of specifi-cation for each intermediate program.

3 Testing Adaptation

To have assurance guarantees about the concrete imple-mentation of adaptation, in this section, we discuss testingof dynamic adaptation.

Examples of the properties that can be tested duringadaptation include deadlock detection, token loss detec-tion, and in general monitoring. We classify the proper-ties to be checked during adaptation into two categories: (i)adaptation-stable properties, and (ii) adaptation-transientproperties. If a property is to be satisfied for all states duringadaptation, then it is known as adaptation-stable property,and if a property is to be satisfied for some interval duringadaptation, then it is known as adaptation-transient prop-erty. A predicate used to specify adaptation-stable prop-erty is known as adaptation-stable predicate, and a predi-cate used to specify adaptation-transient property is knownis adaptation-transient predicate.

While testing for arbitrary predicates may be very ex-pensive, efficient testing is feasible if predicates have cer-tain characteristics. Specifically, in [4–7], authors haveidentified efficient algorithms for a variety of predicateclasses such as conjunctive predicates, disjunctive predi-cates, observer-independent predicates, stable or unstablepredicates. However, in context of testing adaptation, thesealgorithms cannot be employed directly as code of thesystem is changing during adaptation. Hence, we intro-duce adaptation vector to distinguish intermediate programstates. Furthermore, a vector clock system [9] is also main-tained.

2

29th International Conference on Software Engineering Workshops(ICSEW'07)0-7695-2830-9/07 $20.00 © 2007Second International Workshop on Automation of Software Test (AST'07)0-7695-2971-2/07 $20.00 © 2007

3.1 Adaptation Vector

Adaptation vector is used to identify the intermediateprogram states. An adaptation vector A is a vector of nelements, where n is the number of atomic adaptations.Each element of the adaptation vector, denoted by A[i],is boolean valued (1 represents true, 0 represents false);A[i] = 1 denotes execution of atomic adaptation ai in pastand A[i] = 0 denotes that atomic adaptation ai has not yetexecuted. For simplicity of discussion, we assume that theadaptation consists of n atomic adaptations a1, a2, ..., an,and atomic adaptation ai occurs at process pi. Our approachcan be easily extended if multiple atomic adaptations occurat a process.

A value of [0, ..., 0] is assigned to the old program whereno atomic adaptations have occurred. A value of [1, ..., 1] isassigned to the new program reached after all atomic adap-tations have occurred. If [a1, a2, ..., an] is an adaptationvector, then the value of [1, 1, 0, ..., 0] denotes an intermedi-ate program reached after execution of atomic adaptationsa1 and a2. Given an intermediate program and its corre-sponding adaptation vector, we can determine what atomicadaptations occurred in the past to reach that intermediateprogram.

3.1.1 Implementation of Adaptation VectorsEach process keeps its own local adaptation vector AV ,which is updated as the process learns about new atomicadaptation. The adaptation vector is maintained as follows:

• When process pi executes its own atomic adaptationai, it updates the adaptation vector AVi by AVi[i] = 1.

• When process pi sends message m to process pj , itattaches the current value of AVi to m. This value isdenoted by m.AV .

• When process pi receives message m from processpj , it updates its adaptation vector value as AVi =AVi ∨ m.AV , where the OR operation over vectorsis defined on a component-by-component basis.

3.2 Detecting Global Predicates DuringAdaptation

Given a global state of the system and the value of adap-tation vector in that state, we can identify the predicates thatshould be true in that state from the specification duringadaptation. Adaptation-stable predicates need to be checkedfor all global states constructed during adaptation. The in-termediate program in which the adaptation-stable predicateis detected can be easily identified from the value of adapta-tion vector associated with the state in which the predicatewas detected. On the other hand, adaptation-transient predi-cates need to be checked only in states of the correspondingintermediate program. Since testing of adaptation-transientpredicates is more interesting, we discuss that in this sec-tion.

Figure 2 shows how each process is divided into differ-ent sections based on the value of its adaptation vector AV .Whenever a local state sk

i of process pi is to be collected, the

12 2e a=

13e

23e

33 3e a=

11 1e a=

1p

2p

3p

2e′ 2

2e

000

000

000 010 110 111

010 111

100 110 111

31e

21e

Figure 2: Intermediate program states.

state ski is assigned an adaptation timestamp whose value

(denoted as s.AV ) is equal to the current value of AVi.We construct a global state as a n-tuple of local states

(s1, s2, ..., sn). This global state is a state of the interme-diate program whose adaptation vector is equal to s1.AV ∨s2.AV ∨ ... ∨ sn.AV , wheresi.AV ∨ sj .AV = [si.AV [0] ∨ sj .AV [0], ..., si.AV [n] ∨sj .AV [n]]

We now discuss an algorithm for detecting weak con-junctive [5] adaptation predicates. A predicate is calledweak if it is true for some observation of the distributedcomputation, and is similar to possibly predicate in [4].Conjunctive predicates are of the form C1 ∧ ...∧Cn, whereeach Ci is a local predicate. A weak conjunctive predicateis true if and only if there exists an observation in which allconjuncts are true in some global state. This type of pred-icate typically describes some bad or undesirable property.In [5] it is shown that to detect weak conjunctive predicate itis necessary and sufficient to find a set of concurrent statesin which local predicates are true.

3.2.1 Detecting Adaptation-Transient PredicatesConsider a weak conjunctive predicate wcp = C1 ∧ ... ∧Cn that is to be checked for some intermediate program Iduring adaptation. Let I be represented by the adaptationvector IAV . The goal is to detect wcp during the interval ofadaptation that corresponds to the intermediate program I .We discuss the centralized algorithm in which one processserves as a checker process and all other processes involvedin wcp are referred to as non-checker processes.

When detecting adaptation-transient predicates, the non-checker processes have to check for local predicates onlyfor some interval of adaptation as defined by the interme-diate program in which the adaptation-transient predicateis to be checked. Specifically, each non-checker processidentifies a set of local states that are potential states of in-termediate program I . Let the current value of adaptationvector in state s of process pi be AVi, which is denoted bys.AV . We first present the following lemma in the contextof identifying potential states of an intermediate program:Lemma. For state s of process pi, s.AV [i] = 0 ⇒ ∀j :j = i : AVj [i] = 0

Based on the above discussion, we present the extendedversion of algorithms for non-checker and checker pro-cesses from [5] in Figures 3 and 4 respectively. Each non-checker process maintains its own vector clock and adapta-tion vector. The value of adaptation vector is updated as dis-cussed earlier in this section. In the algorithm, plist main-

3

29th International Conference on Software Engineering Workshops(ICSEW'07)0-7695-2830-9/07 $20.00 © 2007Second International Workshop on Automation of Software Test (AST'07)0-7695-2971-2/07 $20.00 © 2007

process pi (non-checker)

var V : array {initially, ∀j : i �= j : V [j] = 0; V [i] = 1}/* vector clock */

AV : array {initially, ∀j :: AV [j] = 0}/* adaptation vector */

flag : boolean {initially, flag = true}/* first time the predd is true after any send event */

plist 〈IAV , pred, cp〉: array 〈 array, boolean expression, process id 〉/* predicates of intermediate programs to be tested */

� replace adaptive action {ai} withai; AV [i] = 1

� replace send statement {send (m) to pj} withsend (m, V,AV ) to pj ;V [i], flag := V [i] + 1, true

� replace receive action {rcv (m) from pj → stmts} withrcv (m, m.V, m.AV ) from pj → stmts;

∀j : V [j] = max(V [j],m.V [j]);∀j : AV [j] = AV [j] ∨ m.AV [j]

� add local predicate detection actionfor p in plist

if p.IAV [i] = AV [i] ∧ (p.IAV ∨ AV ) = p.IAV

∧p.pred = true ∧ flag thenflag := false;send (dbg, V,AV ) to p.cp (checker process)

fi

Figure 3: Algorithm for adaptation-transient predicates(non-checker process pi).

tains a list of - predicate, the corresponding intermediateprogram in which it needs to be checked, and the checkerprocess that is checking that predicate. Whenever the lo-cal predicate of a process becomes true for the first timesince the most recently sent message (or the beginning ofthe trace), it generates a debug message containing its vec-tor clock and adaptation vector and sends it to the checkerprocess.

We maintain one checker process algorithm for eachadaptation-transient predicate that needs to be checked. Thechecker process maintains a separate queue for each processinvolved in the wcp. Whenever a debug message is receivedfrom the process it is enqueued in the queue correspondingto that process. It is assumed that the checker process getsits message from any process in FIFO order. The algorithmcompares the vector clock values at the head of the queuesto determine if a consistent global state can be constructed.Specifically, when the algorithm terminates if all queues arenon-empty, then wcp was detected.

4 Conclusion

In this paper, we discussed testing of adaptation in dis-tributed systems using predicate detection. We identifiedtwo classes of predicates, namely adaptation-stable andadaptation -transient predicates, that occur during adapta-tion. We introduced adaptation vector to identify interme-diate program states in testing dynamic adaptation. Wediscussed algorithms for testing adaptation-transient pred-icates. We refer the reader to [8] where we have discussed

process p (checker)

var q1, ..., qn : queue of (V, AV )changed, newchanged : set of {1, 2, ..., n}IAV : array

� rcv (elem) from Pk →/* elem.V and elem.AV denotes vector clock and adaptation vector */

insert(qk , elem);if (head(qk ) = elem) then

changed := {k};while (changed �= φ) do

newchanged := {};for i in changed ∧ j in {1, 2, ..., n}

if (¬empty(qi) ∧ ¬empty(qj )) thenif head(qi).V < head(qj).V then

newchanged := newchanged∪ {i};fiif head(qj).V < head(qi).V then

newchanged := newchanged∪ {j};fi

fichanged := newchanged;for i in changed

deletehead(qi );od; /* end while */if ∀i : ¬empty(qi) then

int program AV := [Wn

i=1head(qi).AV [1], ...,Wni=1head(qi).AV [n]]

if IAV = int program AV thenfound := true

fifi

fi

Figure 4: Algorithm for adaptation-transient predicate(checker process).

the case study to show how predicate detection is used intesting adaptation of mutual exclusion protocols.

References[1] P. McKinley, S. Sadjadi, E. Kasten, and B. Cheng. Composing adap-

tive software. IEEE Computer, 37(7):56–64, 2004.

[2] S. Kulkarni, K. Biyani, and U. Arumugam. Composing distributedfault-tolerance components. In Workshop on Principles of DependableSystems - PoDSy, at DSN, pages W127–136, June 2003.

[3] S. Kulkarni and K. Biyani. Correctness of component-based adapta-tion. In International Symposium on Component-based Software En-gineering - CBSE, at ICSE, volume 3054 of Lecture Notes in Com-puter Science, pages 48–58, May 2004.

[4] O. Babaoglu and K. Marzullo. Consistent global states of distributedsystems: Fundamental concepts and mechanisms. In Distributed Sys-tems, pages 55–96. Addison-Wesley, 1993.

[5] V. Garg ang B. Waldecker. Detection of weak unstable predicates indistributed programs. IEEE Transactions on Parallel and DistributedSystems, 5(3):299–307, March 1994.

[6] M. Chandy and L. Lamport. Distributed snapshots: Determiningglobal states of distributed systems. ACM Transactions on ComputerSystems, 3(1):63–75, Feb 1985.

[7] S. Venkatesan and Brahma Dathan. Testing and debugging distributedprograms using global predicates. IEEE Transactions of Software En-gineering, 21(2):163–177, 1995.

[8] K. Biyani and S. Kulkarni. Testing dynamic adaptation in distributedsystems. Technical Report MSU-CSE-07-15, Michigan State Univer-sity, http://www.cse.msu.edu/pubs/techlib.php, Mar 2007.

[9] L. Lamport. Time, clocks, and the ordering of events in a distributedsystem. Communications of the ACM, 21(7):558–565, July 1978.

4

29th International Conference on Software Engineering Workshops(ICSEW'07)0-7695-2830-9/07 $20.00 © 2007Second International Workshop on Automation of Software Test (AST'07)0-7695-2971-2/07 $20.00 © 2007