L.I.A.R.: ACHIEVING SOCIAL CONTROL IN OPEN AND DECENTRALIZED MULTIAGENT SYSTEMS

This article was downloaded by: [Queensland University of Technology]On: 01 November 2014, At: 22:00Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Applied Artificial Intelligence: An International JournalPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/uaai20

L.I.A.R.: ACHIEVING SOCIAL CONTROL IN OPEN ANDDECENTRALIZED MULTIAGENT SYSTEMSLaurent Vercouter a & Guillaume Muller ba G2I Centre, École N.S. des Mines de Saint-Étienne , Saint-Étienne, Franceb MalA Team – LORIA , Vandoeuvre-lès-Nancy, FrancePublished online: 31 Aug 2010.

To cite this article: Laurent Vercouter & Guillaume Muller (2010) L.I.A.R.: ACHIEVING SOCIAL CONTROL IN OPEN ANDDECENTRALIZED MULTIAGENT SYSTEMS, Applied Artificial Intelligence: An International Journal, 24:8, 723-768, DOI:10.1080/08839514.2010.499502

To link to this article: http://dx.doi.org/10.1080/08839514.2010.499502

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of theContent. Any opinions and views expressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon andshould be independently verified with primary sources of information. Taylor and Francis shall not be liable forany losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use ofthe Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

L.I.A.R.: ACHIEVING SOCIAL CONTROL IN OPEN ANDDECENTRALIZED MULTIAGENT SYSTEMS

Laurent Vercouter1 and Guillaume Muller21G2I Centre, École N.S. des Mines de Saint-Étienne, Saint-Étienne, France2MalA Team – LORIA, Vandoeuvre-lès-Nancy, France

� Open and decentralized multiagent systems (ODMAS) are particularly vulnerable to theintroduction of faulty or malevolent agents. Indeed, such systems rely on collective tasks thatare performed collaboratively by several agents that interact to coordinate themselves. It istherefore very important that agents respect the system rules, especially concerning interaction,to achieve successfully these collective tasks. In this article we propose the L.I.A.R. model tocontrol the agents’ interactions. This model follows the social control approach that consistsof developing an adaptive and auto-organized control, set up by the agents themselves. Asbeing intrinsically decentralized and nonintrusive to the agents’ internal functioning, it is moreadapted to ODMAS than other approaches, like cryptographic security or centralized institutions.To implement such a social control, agents should be able to characterize interaction theyobserve and to sanction them. L.I.A.R. includes different formalisms: (i) a social commitmentmodel that enables agents to represent observed interactions, (ii) a model for social norm torepresent the system rules, (iii) social policies to evaluate the acceptability of agents interactions,(iv) and a reputation model to enable agents to apply sanctions to their peers. This articlepresents experiments of an implementation of L.I.A.R. in an agentified peer-to-peer network.These experiments show that L.I.A.R. is able to compute reputation levels quickly, precisely andefficiently. Moreover, these reputation levels are adaptive and enable agents to identify andisolate harmful agents. These reputation levels also enable agents to identify good peers, withwhich to pursue their interactions.

1. INTRODUCTION

The use of multiagent systems is often motivated in applications wherespecific system properties such as flexibility or adaptability are required.These properties can be obtained in multiagent systems thanks to theirdecentralization, their openness, and the autonomy of heterogeneousagents. Decentralization implies that information, resources, and agentcapacities are distributed among the agents of the system and hence a

Address correspondence to Laurent Vercouter, 158 cours Fauriel, F-42023 Saint-Étienne,France. E-mail: Laurent.Vercouter@emse.fr

724 L. Vercouter and G. Muller

agent cannot perform alone a global task. Openness can be defined as thepossibility of agents to enter or leave freely the multiagent system duringruntime. At last, heterogeneity and autonomy are the required propertiesof the agents to build open and flexible systems, and they rule out anyassumption concerning the way they are constructed and behave. Agentshave to take the autonomy of other agents into account while interacting.

The combination of these properties is the key to reach flexibility andadaptability as new agents implementing behaviors that were not predictedduring the design of the system can be added and can participate toglobal decentralised tasks. However, this increases the vulnerability of thesystem to the introduction of faulty or malevolent agents because it is notpossible to directly constrain agents’ behavior. Moreover, it is not possibleto perform a global control by a central entity.

An approach to tackle the problem of social control is theimplementation of a social control (Castelfranchi 2000). Social controlimplies that the agents themselves participate to the control of the system.Because agents only have a local and incomplete perception of the system,they have to cooperate to perform a global control of the system. The mostpromising way to perform social control is the use of trust and reputationmodels (Sabater-Mir and Sierra 2002; Huynh, Jennings, and Shadbolt 2004;Sabater-Mir, Paolucci, and Conte 2006; Herzig et al. 2010). Agents observetheir neighbors and should decide if another agent is trustful or not.Sharing this information with others could bring about a global control bythe way of ostracism, as agents getting bad reputations would be sociallyexcluded.

This article describes the L.I.A.R. (liar identification for agentreputation) model defined to perform social control over agents’interactions in decentralized and open multiagent systems. The foundationof L.I.A.R. is a reputation model built from an agent’s own interactions,from its observations, and from recommendations sent by other agents.L.I.A.R. also includes formalisms to represent interactions observed byan agent and norms to be enforced. There are three main specificcontributions of L.I.A.R.: (i) it is designed for decentralized systems,whereas most of existing models require a global complete view of asystem; (ii) it is focused on the control of agent interactions; and (iii) itprovides a full complete set of mechanisms to implement the behaviorof an agent participating to the social control from observations to thetrust decision. Experiments have been made to evaluate L.I.A.R. accordingto performance criteria defined by the ART-testbed group (Fullam et al.2005).

The sequence of this article is as follows. Section 1 gives a generaloverview of the L.I.A.R. model, which parts are detailed in the followingsections: the social commitment and social norm models in Section 2 andthe reputation model in Section 3. Section 4 presents experiments and

Achieving Social Control in Open and Decentralized MultiAgent Systems 725

a performance analysis of the L.I.A.R. model. Finally, Section 5 discussessome properties of the model, and Section 6 presents related works.

OVERVIEW OF THE L.I.A.R. MODEL

This section gives a global overview of the L.I.A.R. model. The socialcontrol approach followed by L.I.A.R. is first described. Then, the mainobjectives of the model and a global overview of its components arepresented.

Social Control of Open and Decentralized Systems

When designing a Open and Decentralized Multiagent System(ODMAS), one usually wants it to exhibit a global expected behaviorto achieve some global tasks. Even if there are heterogeneous agents,different agent deployers or if some emergent global behavior is expected,it is often necessary that the global system remains in a kind of stable statein which it continues to function correctly. This kind of state has beenidentified by Castelfranchi (2000) as the social order of the system. It hasalso been emphasized that the maintenance of social order is particularlyimportant in open and heterogeneous systems.

There exists several approaches to the control of multiagent systems.Security approaches (e.g., Blaze, Feigenbaum, and Lacy 1996) proposeseveral protections, but they do not bring any guarantee about thebehavior of the agents. For instance, a security infrastructure can ensureauthentication or confidentiality of agent communications, but it cannotguarantee that the sender of a message is honest.

Another approach is to use institutions (e.g., Plaza et al. 1998] thatsupervise the agents’ behavior and that have the power to sanction agentsthat do not follow the rules of the system. These works are interesting forsocial order because they introduce the use of explicit rules that shouldbe respected as well as the necessity to observe and represent the agents’behavior. However, institutions often implies the use of a central entity witha complete global view of the system. This makes this approach hard touse for ODMAS. Also, the sanctioning power of these institutions can beviewed as an intrusion into the agents’ autonomy.

Another approach is the social approach. It suggests that the agentsperform themselves an adaptive and auto-organized control of the otheragents. Castelfranchi (2000) uses the expression Social Control to referto this kind of control. Trust and reputation models (Castelfranchi andFalcone 1998; McKnight and Chervany 2001; Sabater-Mir 2002; Sabater-Mir, Paolucci, and Conte 2006) are often used for social control. Accordingto this approach, each agent observes and evaluates a small part of

the system—its neighborhood—to decide which other agents behave welland which ones behave badly, assigning to them a reputation value.Then, it can decide to “exclude” from its neighborhood the agents thatbehaved badly, by refusing to interact and cooperate with them anymore.The exchange of this kind of reputation by the way of gossip andrecommendations can fasten the reputation learning of other agents andgeneralize to the whole society the exclusion of harmful agents. Ostracismis then used to sanction agents exhibiting bad behaviors and, thus, itcreates an incentive to behave as expected.

The L.I.A.R. model described in this article proposes animplementation of social control for ODMAS that covers all the requiredsteps to achieve it, from the observation of interactions to the sanction.

Aims of L.I.A.R

L.I.A.R. is a model for the implementation of a social control ofagent interactions. The major contribution of L.I.A.R., regarding the stateof the art on reputation models, is that it proposes a complete model,going from observations of the interactions to making decisions to trust ordistrust. These characteristics make the L.I.A.R. model suitable for openand decentralized systems such as peer-to-peer networks or ad hoc networks,for which classical models are unsuitable.

The main assumption of the L.I.A.R. model is that the communicativerules are homogeneous and known by every agent. At least, they must beknown by the agents participating in the social control. However, agentsthat are not aware of these rules can still be deployed in the system,but they may be considered as harmful agents, if they do not behave asexpected.

L.I.A.R. Model Components

Because social control emerges from the combination of the activity ofseveral agents, L.I.A.R. proposes models to implement agents participatingto social control. Figure 1 presents the components that are used by aL.I.A.R. agent.

The arrows in Figure 1 define the behavior of a L.I.A.R. agent asfollows: An agent models the interactions it observes as socialcommitments. These social commitments are compared with the socialnorms by an evaluation process. Evaluations that result from thisprocess take the form of social policies. Reputations are setand updated by the punishment process using the social policies. Whenthere is no information, they are set by the initialization process.Reputations are used by the reasoning process, in conjunction withsome representation of the current context of reasoning. This process

FIGURE 1 L.I.A.R. model components.

fixes the trust intentions of the agent. Based on these trust intentionsand the context of decision, the decision process updates the mentalstates of the agent to build intentions about the sanctions to beapplied. An agent has a few possibilities for sanctioning: It can answernegatively to another agent (not believing what it said or refusing tocooperate with it), it can ignore its messages by not answering, or/andit can propagate information about the reputation of this agent toother agents, by the way of recommendations. Sanctions modify theway interactions occur (the dashed line represents influence). Inthe middle of Figure 1, two boxes provide inputs to the punishmentprocess. They represent recommendations received from other agents.The recommendations filtering process considers the reputationof the sender of the recommendation to keep only a set of trustedrecommendations. These recommendations are additional inputs forthe punishment process that can speed up the learning of accuratereputations.

SUPERVISION OF THE INTERACTIONS

This section describes the formalisms used to supervise interactions.First, the formalism used to represent interactions by social commitmentsis detailed. Then, the representation of social norms is described, as well astheir transcription into social policies. The last part of this section explainshow all these formalisms can be used in a process to detect the violationsof the social norms.

Social Commitment

There exists different approaches to enable agents to represent andreason about the interactions of their peers. Two main approaches to therepresentation of interactions are the cognitive approach and the socialapproach. The cognitive approach (Cohen and Levesque 1995; Labrouand Finin 1994) consists of representing a message by a speech act. Thesemantics of a speech act is defined subjectively, by referring to the mentalstates of the sender and receiver of the message. The social approach(Singh 1991, 2000; Fornara and Colombetti 2003; Bentahar, Moulin, andChaib-Draa 2003; Pasquier, Flores, and Chaib-draa 2004) proposes torepresent the occurrence of a message by a social commitment. In thiscase, there is no reference to agents’ mental state. A social commitmentrepresents the fact that a message has been sent and that its sender ispublicly committed on the message content.

The L.I.A.R. model uses this social approach to represent interactionsbecause it is necessary in ODMAS to have a formalism that is not intrusiveto the internal implementation or mental states of the agents. Interactionsshould be represented from an external point of view. Moreover, L.I.A.R.only requires that the utterance of the messages is recorded, and in thismodel there is no need to reason on the semantics of a message. Thus,we do not make any hypothesis about the language used by the agents tocommunicate. We consider that the agents are able to map speech actsfrom the language they use into social commitments, using mappings suchas those proposed by Fornara and Colombetti (2003) or Singh (2000).

Social Commitment DefinitionDefinition 1. A social commitment is defined as follows:

obSCom(db,cr,te,st, [cond, ]cont)where

• ob ∈ �(t) is the observer, i.e., the agent that represents an interactionby this social commitment. �(t) is the set of all agents in the system attime t . t models the instant when the system is considered. It refers to avirtual global clock;

• db ∈ �ob(t) is the debtor, i.e., the agent that is committed. �ob(t) is theset of all agents which ob knows at time t ;

• cr ∈ �ob(t) is the creditor, i.e., the agent toward which the socialcommitment holds;

• te ∈ � is the time of utterance of the message. � is the domain of time;• st ∈ �sc is the state of the social commitment. �sc =�inactive, active, fulfilled, violated, canceled� is theset of possible social commitment states;

• cond ∈ � is a set of activation conditions, i.e., a proposition that has tobe satisfied for the social commitment to become active. � represents thedomain of the first order terms that can be used to represent sentencesor propositions;

• cont ∈ � is the content of the social commitment, i.e., what thecommitment is about.

An agent ob stores all the social commitments it has observed untiltime t in a Social Commitment Set noted obSCS(t). We note obSCScr

db(t) ⊂obSCS(t) to refer to the set of all the social commitments from the debtordb toward the creditor cr.

The following example shows a social commitment from agent Alicetoward agent Bob, as modeled, by agent Oliver. Agent Alice committedat 1pm to the fact that Smith was the president of the United States from2002 to 2007.

Example 1. oliverSCom(Alice, Bob, 1pm, active, president(USA, Smith, 2002–2007))

Social Commitment Life CycleFigure 2 describes the life cycle of a social commitment using a UML

2.0 (OMG 2005) state diagram. The life cycle proceeds as follows:

• A social commitment is created in the inactive state.

FIGURE 2 Life cycle of a social commitment.

• When the activation conditions cond are true, the social commitmentbecomes active. The social commitment remains active as long as theobserver does not know whether it is fulfilled, violated, or canceled.

• The social commitment can be canceled either implicitly or explicitly.When the activation conditions do not hold true anymore, then thesocial commitment is implicitly canceled. The debtor or the creditor canalso explicitly cancel the commitment, when it is either in the inactiveor active state, for instance by sending a new message. In both cases,the commitment moves to the canceled state.

• The social commitment is moved to the fulfilled state if the debtorfulfils it, i.e., the observer believes its content is true.

• The social commitment is moved to the violated state if the debtordoes not fulfil it, i.e. the observer believes its content is false.

It is important to note that this is a representation of the socialcommitment life cycle from a “system” (i.e., centralized and omniscient)perspective. An agent only handles a local representation of the socialcommitments it has perceived. This incomplete knowledge of the overallsystem interactions leads an agent to ignore some social commitments butalso to have a wrong belief about the real state of a social commitment.For example, it is the case if an agent observed a message that created asocial commitment but didn’t perceive another message that changed thesocial commitment state.

Operations on Social Commitment ContentsWe assume that agents are able to interpret the content of messages.

More specifically, L.I.A.R. makes the assumption that agents can recognizethat two social commitment contents are inconsistent and that they candeduce a set of facets addressed by a content. Because these two capacitiesstrongly depend on a concrete application, L.I.A.R. do not propose anyspecific way to implement these operations. This task is left to the systemdeveloper that wants to adapt L.I.A.R. to a given application.

Inconsistent Contents. The operation ob.inconsistent-content:� × �(�) �→ �true, false� returns true if a set of contents isinconsistent at time t ∈ � , false otherwise. �(�) is the set of sets of �, i.e.,�( ) is the powerset operator. This inconsistency can be based on a logicalinconsistency in first-order terms of the concerned contents. It can alsotake into account some expert knowledge about the application domain.

Facets. The operation ob.facets :ob SCS �→ �(� ) takes as argumenta social commitment and returns a set of facets. � is the set of allexisting facets. The concept of facet correspond to a topic of a socialcommitment. It will mainly be used while building reputation values(Section 3) to assign different reputations according to different facets

of an agent (for instance, an agent can have a good reputation to giveinformation about the weather and a bad one to give information aboutthe currency exchange rates). L.I.A.R. uses also a specific facet namedrecommend attached to social commitments representing messages thatare recommendations about the reputation of another agent, such as theones described in Section 3.3.4.

Social Commitment InconsistencyDefinition 2. For an observer ob, a social commitment c is inconsistentwith a set of social commitments � ⊂ obSCS(t) if there exists some socialcommitments �c1, � � � , cn� in � that are in a “positive” state (active orfulfilled,) and which contents are inconsistent with the content ofc , but not with one another (the content of a commitment c is notedc.cont):

∀c ∈ obSCS(t),∀� ⊂ obSCS(t),∀t ∈ � ,ob.inconsistent(t , c ,�)

def= c �st ∈ �active, fulfilled� ∧ ∃�c1, � � � , cn� ⊆ � |¬ob.inconsistent_content(�t , c1,cont, � � � , cn �cont�)

∧∀c1 ∈ �c1, � � � , cn�, c1�st ∈ �active,fulfilled�

∧ob.inconsistent_content(t , �c �cont, c1�cont, � � � , cn �cont�)Example 2 illustrates an inconsistency between two social commitments

from two different agents, Alice and Dave, toward two different agents,Bob and Elen.

Example 2. oliverSCom(Alice, Bob, 1pm, active, president(USA, Smith, 2002–2007))

oliverSCom(Dave, Elen, 2pm, active, president (USA,Wesson, 2002–2007))

At time 2pm, agent Oliver can consider that the first socialcommitment creates an inconsistency with the set constituted by thesecond one, if it considers that it is not possible that two different personsare president of the United States during the same mandate.

Social Norms and Social Policies

Social norms define the rules that must be respected by the agentsduring their interactions. Besides, we introduce the concept of social policyto represent the situation, at a given instant, of a given agent, about agiven social commitment and a given social norm. For instance, we candefine a norm that prohibits social commitments to be in the violated

state. But if such a norm does not exist, violated social commitmentswould be accepted in the system and the agents implementing L.I.A.R. willnot consider them as malicious or unauthorised behavior. This separationincreases the generality of the L.I.A.R. model for the definition of norms.

Social Norm DefinitionDefinition 3. We define a social norm as follows:

snorm(op,Tg,Ev,Pu, cond, cont, st)

• op ∈ �I, 0, P� is the deontic operator that characterizes the socialnorm: I for prohibition, 0 for obligation, and P for permission;

• Tg ⊆ �(t) represents the entities under the control of the social norm(its “targets”). t is the time at which we consider the social norm. �(t)is the set of all the agents in the system;

• Ev ⊆ �(t) represents the entities that have judiciary power, i.e., whichdecide when the social norm is violated (the “evaluators”);

• Pu ⊆ �(t) represents the entities that have the executive power, i.e., thatapply the penalties (the “punishers”);

• cond ∈ � are the validity conditions of the social norm. The socialnorm is activated when these conditions are satisfied and deactivatedotherwise. It is a first-order term;

• cont ∈ � is the content of the social norm. It is a first-order term, toallow social norms to refer to social commitments;

• st ∈ �sn represents the state of the social norm. Possible states are:�sn

def= �inactive, active�.

Example 3 presents a social norm that represents the prohibition for amember of the MAS team (agent y) to talk to its Ph.D. supervisor (agentx) about politics, i.e., to create a social commitment which would have“politics” as one of its facets. The condition of the social norm statesthat agent x should be the supervisor of agent y, which is itself a memberof the MAS team. The agents that are targets of this social norm are themembers of the MAS team (��(t) ⊆ �(t)). The agents that can detectfulfilment or violation of the social norm are all the members of the MASteam. However, the agents that are enabled to sanction this social norm areonly the Ph.D. supervisors of the MAS team ��(t) ⊂ ��(t). Here,the social norm is active.

Example 3.

snorm(I ,��(t),��(t),��(t),

x ∈ ��(t) ∧ y ∈ ��(t) |supervisor(x , y),∀c ∈ SCSx

y (t),facets(c) ⊇ �politics�,active)

Where the predicate supervisor(x , y) is true if agent x is the supervisorof agent y, false otherwise. The function facets( ) takes as argument asocial commitment and returns a set of facets the content is about, fromthe system perspective (cf. Section 2.1.3).

Generation of Social PoliciesThe previous definition of social norm uses a. system perspective and

requires omniscient knowledge on the system (for instance, knowing theset of the overall agents in the system). To be used locally by agents, weneed to introduce the concept of social policy. This allows us to.

• Specify the content of a social norm from the point of view of eachevaluator. In fact, to enable evaluators to detect the violations of thesocial norms they are aware of, it is necessary that each of them adaptsthe content of the social norm, that is expressed with a general andomniscient point of view, into its own local and partial point of view onthe system.

• Cope with multiple violations of a given social norm by several targets.The rule described by a social norm can, at a given instant, be violated byseveral targets. Each social norm should then lead to the generation ofseveral social policies. Each social policy is directed to a single target toallow the evaluator to detect and keep track of each violation separately.

• Cope with multiple violations of a given social norm by a single target.A single agent can violate several times the same social norm. By(re)creating a social policy each time it is violated, an evaluator candetect multiple violations of the social norm by the same target. This alsokeeps track of each violation distinctly.

• Add penalties. Social norms do not indicate any sanctions to apply incase of violation. These penalties are associated to social policies so thateach punisher can decide if and what kind and amount of penalties itwants to attach to a given social norm. This subjectivity in the importanceof social norm violations follows the definition of social norms proposedby Tuomela (1995).

Social Policy DefinitionDefinition 4. A social policy is defined as follows:

evSPol(db, cr,te ,st, [cond,]cont)

• ev ∈ �(t) is the evaluator that generated the social policy from a socialnorm;

• db ∈ �ev(t) is the debtor, the agent committed to by this social policy;• cr ∈ �ev(t) is the creditor, the agent toward which the debtor iscommitted by this social policy;

• te ∈ � is the time of creation, the instant when the social policy hasbeen generated;

• st ∈ �sp is the state of the social policy. Possible states are {inactive,active, justifying, violated, fulfilled, canceled};

• cond ∈ � are the activation conditions. Here, it is a first-order term andit is optional (its omission corresponds to an always true condition);

• cont ∈ � represents the content of the social policy. It is also afirst-order term, because social policy contents may refer to socialcommitments (cf. Example 4).

As for social commitments, agents store social policies in Social PolicySets. evSPS(t) is the set of social policies created by agent ev before or attime t . evSPScr

db(t) is the set of social policies with debtor db and creditorcr as perceived by agent ev at instant t .

Example 4 presents a social policy that could have been generatedfrom the norm in Example 3 by agent Oliver. It expresses the positionof agent Alice (a member of the MAS team) with respect to the socialnorm: it made a social commitment that talks about politics with agentBob, one of its Pn.D. supervisors.

Example 4.

OliverSPol(Alice, Bob, 7pm, violated, Alice ∈ ��(t)

∧ Bob ∈ ��(t),Oliver.facets(sc) ⊇ �politics�)

Where sc ∈ oliverSCSBobAlice(t) is a social commitment of agent Alice

toward agent Bob as present in agent Oliver’s social commitment sets.For instance, this can be the one of Example 1.

Social Policy Life CycleFigure 3 describes the life cycle of a social policy using a UML 2.0 state

diagram. This life cycle proceeds as follows:

• A social policy is created in the inactive state.• When the activation conditions cond become true, the social policybecomes active.

• The social policy can be canceled, for instance if the social norm, fromwhich it has been generated, has been deactivated.

FIGURE 3 Life cycle of a social policy.

• The evaluator can move the social policy to the state fulfilled, if itbelieves the content of the social policy is true.

• The evaluator can move the social policy to the state justifying, if itbelieves that the content is false. In this case the evaluator suspects thesocial policy to be violated. It starts a justification protocol (see section2.3.2) to check whether a violation actually occurred or if its belief aboutthe violation is due to a false or incomplete state of its own local socialcommitment sets.

• According to the result of this justification protocol, the evaluator canconsider that the social policy has actually been violated if no proofhas been received (predicate proof_received not true) and movethe social policy to the violated state. In the opposite case, i.e, if aproof has been received (predicate proof_received true), it movesthe social policy to the canceled state and considers that its local socialcommitment sets were obsolete.

Operations on Social PoliciesAn agent should be able to perform a few operations on a social policy.

First, it must be able to associate a penalty with a social policy. It must alsobe able to deduce a set of facets and a set of dimensions addressed by asocial policy.

The concept of facet of a social policy is the same as the one introducedfor social commitments (i.e., a topic of conversation; see Section 2.1.3).Therefore, it refers to the social commitments addressed by the social norm.The concept of dimension refers to the social norm itself and to how itallows to judge the agent. According to McKnight and Chervany (2001),agents can be judged along four dimensions: integrity, competence,benevolence, and previsibility. Integrity corresponds to thesincerity and the honesty of the target. Competence refers to the qualityof what the target produces. Benevolence means caring and beingmotivated to act in one’s interest rather than acting opportunistically.

Finally, previsibility means that trustee actions can be forecasted in agiven situation.

The operations on social policies are defined as follows:

• pu.punishes: � × � �→ [0, 1]. � is the set of all social policies. Agentpu associates a penalty with a social policy and an instant. This penaltyis a float value in [0, 1] and represents the relative importance thatagent pu gives to the social policy, relatively to the other social policiesat time t . The way an agent associates a value with a social policy is notconstrained by the L.I.A.R. model. The model only allows an agent to doit. If an agent is not able to compare social policies to assign differentpenalties to them, the L.I.A.R. model can still be used by associating asame not null value (e.g., 1) with every social policy.

• ev.facets: evSPS �→ �(� ) takes as an argument a social policy andreturns the set of facets that agent ev associates with it. These facetsare deduced by agent ev from the facets of the social commitmentsreferenced by the content of the social policy.

• ev.dimensions: � �→ �() takes as an argument a social policy (∈�)and returns a set of dimensions. is the set of dimensions along whichit is possible to judge an agent.

def= �integrity, competence, benevolence, previsibility�

Evaluation Process

The evaluation process is used by an evaluator to detect social normviolations. It generates social policies the state of which corresponds to theresult of the evaluation.

Two distinct parts must be distinguished. First, detection of social normviolation can be done by an agent by considering its own perception ofsocial commitments. The second part, called justification protocol, is anoriginal contribution of L.I.A.R. because it has been defined specifically todeal with the decentralized nature of the system and the incompletenessof the agents knowledge. A violation can be detected as a consequence ofobsolete local social commitment sets. The justification protocol is used bythe evaluator to look for unknown or outdated social commitments thatcould cancel the violation.

Protocol for the Detection of Social Norm ViolationFigure 4 describes the social norm violation detection protocol using

an AUML sequence diagram (sp.st is the state of the social policy). This

FIGURE 4 Social norm violation detection protocol.

protocol proceeds as follows:

1. Some propagators transmit m observations to an evaluator. Note thatpropagator and evaluator are roles and therefore can be played by thesame agent.

2. These observations, modeled as social commitments, are filtered by theevaluator (see section 3.3.4 for details about the filtering process) and,eventually, added to the evaluator’s social commitment sets. Then, theevaluator generates social policies to represent the compliance of thenew social commitments with the social norms. (Appendix A presentsan implementation of such a process.) Finally, it tries to establish thestates of the social policies.

3. The social policies that are in a terminal state (fulfilled orcanceled) are stored and not considered anymore in the presentprotocol. All social policies whose content is not true are suspected to beviolated. They are moved to the justifying state and a justificationprotocol is started for each of them.

4. When the justification protocol ends with “proofs” that the violationdid not occur (predicate proof_received true), the local socialcommitment sets of the evaluator are updated with this newinformation. If some suspected violations remain (or if new ones havebeen created by the new information), the evaluator enters again inthe justification stage.

5. If no “proof” has been received (predicate proof_received is nottrue), then the social policies are moved to the violated state.

Example 5 details the progress of this process on a short scenario,where agent Bob transmits to agent Oliver a social commitment fromagent Alice that violates a social norm. Here, we consider that agentAlice did not cancel its social commitment.

Example 5. In the example below, Agent Bob plays the observer andpropagator roles and Oliver plays the evaluator role.

1. Bob transmits to Oliver its representation of a communication itreceived from agent Alice, as a social commitment. For instance thiscan be a social commitment similar to the one in Example 1, page 5,but with Bob as the observer.

2. Oliver decides whether to trust this message or not. If it decides totrust it, it interprets the communication into a social commitment whereit is the observer, as showed in Example 1. Oliver then instantiatesthe social norm given in Example 3 into the social policy given inExample 4. As the social commitment content does not respect thenorm content, the social policy is generated in the justifying state.

3. Because the social policy is generated in the justifying state, ajustification protocol is started.

4. This justification protocol ends with agent Oliver not getting anyproofs. Oliver therefore considers that a violation actually occurredand moves the social policy to the violated state.

Justification ProtocolThe justification protocol gives the chance to an agent suspected of

violating a social norm to prove that it is not the case. An agent can do soby showing that the violation has been detected because of an incompleteknowledge of the message exchanges.

This justification protocol is depicted in Figure 5 as an AUML sequencediagram. It proceeds as follows:

1. The social policy sp known by the evaluator ev is in the justifyingstate. The evaluator sends the content of sp to the debtor of the socialpolicy, as a justification request.

2. Agent ev waits during a delay �1 for an answer. If, at the end of thedelay �1, the debtor has not answered satisfactorily, i.e., the predicateproof_received is not true, then agent ev can widen its research-for “proofs”: It can send its request for justification to any agent that itbelieves could have observed the behavior of the debtor. Such agentscan be, for instance, the creditors of the social commitments referredto in the social policy. We label these agents the “PotentialObservers”. Ifthe evaluator still does not get any answer (after a delay �2), it can again

FIGURE 5 Justification protocol.

broaden its research to the set of all its acquaintances, with anotherdelay �3.

3. If the evaluator does not get any satisfactory answer, then it considersthe social policy has been violated and moves it to the violated state.If it does get an answer, then it updates its local representations of socialcommitments with the “proofs” obtained.

Agents that receive such justification requests can act in several ways:

• They can simply ignore the request.• If they also detect a violation, they can start a justification protocol too.In this case they play the same role as agent ev, but in another instanceof the justification protocol, which proceeds in parallel.

• If they do not detect a violation, then it means they have “proofs” thesuspected agents did not violate the social policy. This can happen,for instance, because the suspected agent canceled some of the socialcommitments referred to by the social policy. In this case they canprovide these “proofs” to agent ev. The message they send is of typeproof and gives a copy of a message p that proves that the content c ofthe social policy sp has not been violated.

The “proofs” that agent ev is waiting for are digitally signed messages(He, Sycara, and Su 2001). The digital signature is important becauseit guarantees the non-repudiation property: An agent that has digitallysigned a message can not pretend later that it did not do so.

Delays might depend on the underlying communication network andthe time needed by an agent to query its social commitment sets and take adecision. Therefore, we consider here that delays are fixed by the designerof the system, as well as the unit of time in which they are measured.

Example 6 details the progress of the justification protocol in the samescenario as Example 5.

Example 6. The justification protocol starts with a social policy like theone presented in Example 4, but which is in the state justifying.

1. Oliver plays the evaluator role and sends a request for proofs toAlice, the debtor of the social policy, which is also the debtor of thesocial commitment in the present case.

2. Alice has no proofs it has canceled its commitment and does notanswer the query.

3. When delay �1 expires, Oliver sends the same request for proofsto potential observers of Alice’s behavior. In the present scenario,Oliver sends its request to Bob, then waits �2.

4. Bob seeks in its social commitment sets and does not discover any proof.Therefore, it does not send anything to agent Oliver.

5. When �2 expires, Oliver sends the query to all its acquaintances andwaits for �3.

6. No agent answers the query, either because they ignore the query orbecause they do not. find any answer in their social commitment sets.

7. At the end of �3, Oliver considers the social policy as violated.

No agent can provide a “false” proof, as agent Oliver will ignore anymessage that is not digitally signed.

Any agent that has the ability to observe some messages and thatknows some social norms can play the role of evaluator in the protocolof detecting social norm violations. It is able to build an evaluation ofa target. This way, the first phase of the social control can be fulfilledby agents detecting violations. The processes described in this sectiondo not guarantee that every violation will be detected. But this is notpossible in decentralized, open, and large-scale multiagent systems becausewe can only use an incomplete view of the interactions occurring in suchsystems, based on the local perceptions of some agents. However, L.I.A.R.allows agents to reason on their local perception and to exchange themwith other agents to detect some violations. The justification protocolguarantees that an agent that did not violate any norms cannot be accusedof doing so, if it gives proofs of its correct behavior.

REPUTATION MODEL

The goal of the reputation model of L.I.A.R. is to provide anestimation, over time, of the compliance of other agents’ behavior withrespect to the social norms. Basically, the reputation model has two roles.First, it uses as inputs the results of the L.I.A.R. components presented in

the previous section—social policies—to compute reputations assigned toother agents. Second, it enables agents to reason and make decisions basedon these reputations.

Based on Quéré (2001) and McKnight and Chervany (2001)distinction of trust beliefs, trust intentions, and trust behaviors, wedefine the term “reputation” to refer to an agent’s beliefs about thetrustworthiness of another agent and “trust” as the act of taking a decisionto trust. In summary, reputation levels are the beliefs on which an agent makesits decision to trust.

In the first following subsection, the core concepts of the L.I.A.R.reputation model are defined. Then, the processes related to reputation(initialization, punishment, reasoning, decision, and propagation) aredescribed.

Reputation Types

Different reputation types can be considered according to thesourceand the kind of information used to compute a reputation value. Todistinguish these reputation types and their semantics, we need to considerthe roles that are involved in the reputation-related processes. We extendthe work of Conte and Paolucci (2002) to identify seven roles:

• target, the agent that is judged• participant, an agent that interacts with the target• observer, an agent that observes a message and interprets it as a socialcommitment

• evaluator, an agent that generates social policies from socialcommitments and norms

• punisher, an agent that computes reputation levels from a set of socialpolicies

• beneficiary, the agent that reasons and decides based on the reputationlevels

• propagator, an agent that sends recommendations: messages aboutreputation levels, but also about social policies or observed messages

According to the agents that play these roles, L.I.A.R. distinguishes fivereputation types:

• Direct Interaction based reputation (DIbRp) is built from messagesfrom the target to the beneficiary. The roles of beneficiary, punisher,evaluator, observer, and participant are played by the same agent. Thereis no propagator. For instance, if Alice directly interacts with Bob, shecan compute the level of DIbRp for Bob from her experience.

• Indirect Interaction based reputation (IlbRp) is built from messagesobserved by the beneficiary. The roles of beneficiary, punisher, evaluator,and observer are played by the same agent but in this case theparticipant is distinct. There is still no propagator. For instance, ifAlice observed interactions between Bob and Charles, she can use herobservations to update levels of IlbRp for Bob and Charles.

• Observations Recommendation based reputation (ObsRcbRp) is builtfrom observed messages propagated to the beneficiary by a propagator.An agent plays the roles of beneficiary, punisher, and evaluator, andanother distinct agent plays the roles of observer and propagator. Theparticipant can be any agent (except the agent that is the beneficiary).For instance, if Bob reports to Alice some interactions with Charles(without any evaluation regarding compliance to social norms), shecan use this observation reports to update the level of ObsRcbRp forCharles. This typically happens during the justification protocol.

• Evaluation Recommendation based reputation (EvRcbRp) is built fromsocial policies propagated to the beneficiary by a propagator. An agentplays the roles of beneficiary and punisher, and another distinct agentplays the roles of evaluator and propagator. The observer and theparticipant can be any agent. For instance, if Bob reports to Alice somenorm violations performed by Charles, she can use this violation reportsto update the level of EvRcbRp for Charles.

• Reputation Recommendation based reputation (RpRcbRp) is built fromreputation levels propagated to the beneficiary by a propagator. Anagent plays the role of beneficiary, and another distinct agent playsthe roles of punisher and propagator. The evaluator, observer, and theparticipant can be any agent. For instance, if Bob reports to Alice hisestimation of Charles’s reputation, she can use this value to update thelevel of RpRcbRp for Charles.

Each reputation type is formalized in L.I.A.R. as follows:

XbRptargetbeneficiary(facet, dimension, instant)

which represents the reputation of type X (X can be DI, II, ObRc,EvRc, RpRc) associated with agent target by agent beneficiary for the facetfacet and dimension dimension at time instant.

Computational Representation and Initialization Processes

Most researchers commonly agree that there is no standard unitin which to measure reputation (Dasgupta 1990) but that reputationsare graduated. A reputation model should then adopt a computationalrepresentation allowing the comparison of reputation levels.

L.I.A.R. uses the domain [−1,+1] ∪ �unknown� for reputation values.−1 represents the lowest reputation level and +1 the highest reputationlevel. A special value, unknown, is introduced to distinguish the case ofignorance, where a beneficiary has no information about a target.

The initialization process sets every reputation value at unknown at thebeginning.

Punishment Process

The punishment process consists for the punisher in computingreputation levels according to the sets of social policies it knows. Thecomputation depends on the type of the reputation as different inputs areconsidered according to the type of the reputation that is computed.

Social Policy SetsLet puSPS�

tg(t) be the social policy set known at time t by pu, wheretg is the debtor and � the set of all the creditors of the social policiesof the set. The subset puSPS�

tg(�, �, t) ⊆ puSPS�tg(t) contains only the social

policies of the set that are associated with a facet � and a dimension �.For computation, we consider the following social policy subsets,

according to their state:

• Fulfilled Social Policy Set

puFSPS�tg(�, �, t)

def= �sp ∈ puSPS�tg(�, �, t) | sp�st = fulfilled�

• Violated Social Policy Set

puVSPS�tg(�, �, t)

def= �sp ∈ puSPS�tg(�, �, t) | sp�st = violated�

• Canceled Social Policy Set

puCSPS�tg(�, �, t)

def= �sp ∈ puSPS�tg(�, �, t) | sp�st = canceled�

which are abbreviated by pu�SPS�tg(�, �, t), where � ∈ �F ,V ,C�.

The importance of a social policy set is defined as the sum ofthe penalties associated with each social policy of the set. We defineImp(pu�SPS�

tg(�, �, t)) as follows:

Imp(pu�SPS�tg(�, �, t))

def=∑

sp∈pu�SPS�tg(�,�,t)

pu�punishes(sp, t)

Direct Interaction Based ReputationDirect interactions are evaluated from social policies known by the

punisher pu and where pu is also the creditor. Direct Interaction basedreputation (DIbRp) is calculated as follows:

DIbRptgpu(�, �, t)

def=∑

�∈�F ,V ,C� � × Imp(pu�SPS�pu�tg (�, �, t))

∑�∈�F ,V ,C� |�| × Imp(pu�SPS

�pu�tg (�, �, t))

Where F , V , and C are weights associated with the states of the socialpolicies (respectively, in the fulfilled, violated, and canceledstate). These weights are floating values that the punisher is free to set.The only constraint is that F > 0 and V < 0.

Indirect Interaction Based ReputationIndirect interactions are evaluated from social policies known by

the punisher pu with the creditor being any agent except pu. IndirectInteraction based reputation (IIbRp) is calculated as follows:

IIbRptgpu(�, �, t)

def=∑

�∈�F ,V ,C� ′� × Imp(pu�SPS

�pu(t)\�pu�tg (�, �, t))

∑�∈�F ,V ,C� |′

�| × Imp(pu�SPS�pu(t)\�pu�tg (�, �, t))

The weights ′� ,

′ , and ′

� are similar to � , v , and c but they can havedifferent values. It is also required that ′

� > 0 and ′ < 0.

Recommendation Based ReputationsThe three other kinds of reputation are slightly different because they

are computed from recommendations (information communicated bypropagators) and not from observations made by the punisher. Therefore,the information used as an input is less reliable than for the two firstreputation types as propagators may lie in their recommendations.

The L.I.A.R. model is used recursively to decide if an agent should trusta recommendation or not, as well as the strength of this trust. The inputsof the reasoning process (which is detailed in section 3.4) are a target,a facet, a dimension, a set of threshold, and an instant. A specific facetnamed recommend is used to represent recommendations. The dimensionused to judge propagators is their integrity and the thresholds usedto filter out the recommendations are noted RcLev. The output of thereasoning process is twofold: On the one hand there is a Boolean part,trust_int, that indicates if the intention is to trust or not, whereas onthe other hand, a float value, trust_val, corresponds to the strengthassociates with this intention.

The set of trusted recommendation is denoted puTRc(t) and containsevery recommendation received by pu and for which the trust_int partof the output of the reasoning process is true. The recommendations forwhich the result is false are simply ignored.

The content of a recommendation can be a social commitment, asocial policy, or a reputation level. The reputation type that is builtdepends on the type of the recommendation content.

Observations Recommendation Based ReputationThe Observations Recommendation based reputation (ObsRcbRp)

is evaluated from recommendations containing social commitments.Propagated social commitments are involved in the generation of a setof social policies. The sets puObsRc�SPS(tg, �, �, t) with � ∈ �F ,V ,C�represent these social policies, extracted from puTRc(t) and grouped bytheir state. Observations Recommendation based reputation is calculatedas follows:

ObsRcbRptgpu(�, �, t)

def=∑

�∈�F ,V ,C� ′′� × Imp(puObsRc�SPS(tg, �, �, t))

∑�∈�F ,V ,C� |′′

�| × Imp(puObsRc�SPS(tg, �, �, t))

As in previous formulae, ′′� ,

′′ , and ′′

� are weights and ′′� > 0 and

′′ < 0.

Evaluation Recommendation Based ReputationThe Evaluation Recommendation based reputation (EvRcbRp)

is evaluated from recommendations containing social policies. Thesets puEvRc�SPS(tg, �, �, t) with � ∈ �F ,V ,C� represent these socialpolicies, extracted from puTRc(t) and grouped by their state. EvaluationRecommendation based reputation is calculated as follows:

EvRcbRptgpu(�, �, t)

def=∑

�∈�F ,V ,C� ′′′� × Imp(puEvRc�SPS(tg, �, �, t))∑

�∈�F ,V ,C� |′′′� | × Imp(puEvRc�SPS(tg, �, �, t))

As in previous formulas, ′′′� ,

′′′ , and ′′′

� are weights and ′′′� > 0 and

′′′ < 0.

Reputation Recommendation Based ReputationThe Reputation Recommendation based reputation (RpRcbRp) is

evaluated from recommendations containing reputation levels. Thecomputation formula is different from previous ones because thpbeneficiary has to merge reputation levels according to its degree oftrust toward the propagator. The set puRpRc(tg, �, �, t) contains thetrusted reputation recommendations. Each of these latter contains a

numerical value (recommendations with the unknown value are dropped).Reputation Recommendation based reputation is calculated as follows:

RpRcbRptgpu(�, �, t)

∑rc∈puRpRc(tg,�,�,t)

rc�level × pu�reasons(rc�pu, �′, �′,RcLev, t)�trust_val

∑rc∈puRpRc(tg,�,�,t)

pu�reasons(rc�pu, �′, �′,RcLev, t)�trust_val

where rc�level refers to the numerical value of reputation contained inthe recommendation rc, rc.pu refers to the agent that has computedthis value, �′ = recommend, and �′ = competence. pu.reasons is theprocess detailed in the next section. Its trust_val output is the weightof the trust intention, here associated with the trust in the punisher for itscompetence to recommend.

The RpRcbRp level is computed as the weighted average of thereputation levels received through recommendations. The weights usedin this computation are those extracted from the trust intention that thepunisher has in the other punishers for the facet �′, along the dimension�′. These values are considered to be positive.

In conclusion, the RpRcbRp is both part of the L.I.A.R. model and usesit for its computation. The benefit of the sharpness of the L.I.A.R. model isillustrated here. First, the recommendations are filtered according to theintegrity of the propagators for the recommend facet, then the weightgiven to each recommendation depends on the punishers’ competencefor this same facet.

Reasoning Process

The reasoning process consists, for a beneficiary bn, of deducing atrust intention based on the reputation levels associated with a target tg.It has as inputs: a target, a facet, a dimension, and a set of reasoningthresholds. These latter are positive float values labeled trust�bRp, distrust�bRp ,relevance�bRp where � ∈ �DI,II,ObsRc,EvRc,RpRc�. They are grouped in thestructure denoted Lev. An example of such thresholds has been presentedin section 3.3.4, where we discuss the recommendation filtering process.This constitutes the context of reasoning. The reasoning process outputis twofold: trust_int and trust_val, which respectively represent theintention to trust (boolean) and the strength of the intention (float value).Figure 6 shows the reasoning process.

Agent bn first tries to use the reputation level that it considers themost reliable (DIbRp in Figure 6). This type can be sufficient to fix theintention to trust (resp. distrust) the target. If the value associated with

FIGURE 6 Reasoning process.

the DIbRp is greater (resp. less) than the threshold Lev�trustDIbRp (resp.Lev�distrustDIbRp ), then agent pu has the intention to trust (resp. distrust) thetarget. If the DIbRp is in the state unknown, if it is not discriminant (i.e.,is between the thresholds Lev�trustDIbRp and Lev�distrustDIbRp ) or if it is not relevant(not enough direct interactions, threshold Lev�relevanceDIbRp ), then the DIbRpis not sufficient to set whether the intention is to trust the target or not.

In this case, the next reputation type (IIbRp in Figure 6) is used ina similar process with thresholds Lev�trustIIbRp and Lev�distrustIIbRp . If this processstill does not bring to a final state, the next reputation type is considered.The next reputation types (ObsRcbRp, EvRcbRp then RpRcbRp) areused with the corresponding thresholds (Lev�trustObsRcbRp and Lev�distrustObsRcbRp,Lev�trustEvRcbRp, and Lev�distrustEvRcbRp, Lev�trustRpRcbRp, and Lev�distrustRpRcbRp. Finally, ifnone of the previous values allows to fix the intention, then a GeneralDisposition to Trust (GDtT) is used.

The GDtT is a kind of default reputation. It is not attached to aparticular target. It represents the general inclination of the beneficiary totrust another agent, when it does not have information about it.

The relevance measure that we use for DIbRp (resp. IIbRp,ObsRcbRp, EvRcbRp, and RpRcbRp) in the cascading process isdefined by another threshold Lev�relevanceDIbRp ∈ [0,+∞) (resp. Lev�relevanceIIbRp ,Lev�relevanceObsRcbRp, Lev�relevanceEvRcbRp, and Lev�relevanceRpRcbRp), which represents thenumber of direct interactions (resp. indirect interactions and variousrecommendation types) from which the agent considers that its reputationlevel is relevant. The relevance 0 is associated with an unknown reputationlevel.

Finally, the weight of the output is set to the reputation level thatbrings about a trust intention. For instance, if DIbRp is sufficient to setthe intention, then trust_val is set to the value of DIbRp. The trustthresholds values are positive. The weight of a trust intention is alsopositive.

Decision Process

The decision process consists, for a beneficiary bn, in taking decisionsto act in trust or not in a given context. This process takes a target, acontext description, and an instant as inputs. As output, the mental statesof the beneficiary are modified according to the decision to act in trust ornot with the targets.

A beneficiary bn can take two kinds of decision:

selection: It can decide whether a given agent is trustworthy or not. It usesthe trust_int output of the reasoning process.

sort: It can compare different agents according to their trustworthiness. Ituses the trust_val output of the reasoning process.

Propagation Process

The propagation process is executed by a propagator and consists indeciding why, when, how, and to whom to send recommendations. Agentscan show various strategies of propagation, which are very dependant onthe application targeted by the decisions.

L.I.A.R. can be used to implement two different strategies:

push strategy: A propagator spontaneously sends recommendations tosome of its acquaintances.

pull strategy: A propagator receives requests to send recommendations.

The push strategy has the advantage of helping agents to speed uptheir learning of accurate reputations. However, in some cases it mayinvolve the sending of sensitive information about one’s partners. Also,such a process has a risk of flooding the network; Therefore, it willgenerally be used only by a limited set of agents and only in a limitednumber of situations.

The pull strategy is rather used by agents that seek information aboutunknown or insufficiently known agents. Such a process is less prone tosensitive information disclosure, because the requester agent selects itsprovider. However, the difficulty for the requester is to find an agent thatboth has the information it seeks, and that will provide this informationcorrectly.

In both processes the propagator has to decide whether to answer orto whom to send its recommendations. In L.I.A.R. we decided that theagent uses its reputation model to make its decision. In the push process, adecision of type “sort” is used to find to which subset of the acquaintancesto send the recommendations. In the case of the “pull” process, it is rathera decision of type “selection” that is used: The propagator decides whetherto answer or not.

EXPERIMENTAL RESULTS

This section presents experimental results obtained by simulating apure P2P network using the L.I.A.R. model. In such networks, agentsexchange information (e.g., file or resource locations) using a protocol.Different P2P protocols exist but they all require that agents propagatemessages through several hops to allow two agents to communicate.Such propagations will be exploited by L.I.A.R. as agents involved in apropagation chain will be able to observe some communications betweenother agents and to test whether they do or do not respect the norms.

The next subsection details how the simulation works. Thensubsection 4.2 lists the evaluation criteria that are used. The remainingsubsections show and discuss the L.I.A.R. performances regarding thesecriteria.

Simulation Settings

Several agents using the L.I.A.R. model are deployed. These agentscreate social commitments on facts selected within a set of propositionsand their negations: �A,B, � � � ,¬A,¬B, � � � �. For simplicity, we consider thatall these facts belong to the same facet called application and that onlya fact and its negation (e.g., A and ¬A) are inconsistent.

Each step of the simulation runs as follows:

1. Each agent generates NB_ENCOUNTER_BY_ITERATION socialcommitments. The content of each social commitment is randomlychosen in the set of propositions and their negations: �A,B, � � � ,¬A,¬B, � � � �. The debtors of the social commitments are randomly chosen.

2. These social commitments are added to the debtor and creditor’s socialcommitment sets. This starts a process of violation detection in thoseagents, which leads to the generation of some social policies.

3. For each social commitment, a number NB_OBSERVATION_BY_ITERA-TION of agents is selected. The selected agents perceive this socialcommitment as an indirect interaction. The social commitment is addedto these agents’ social commitment sets. These observers automaticallyrun violation detection processes and generate social policies.

4. The simulation uses a “push” propagation strategy (see section 3.6).Each agent sends NB_RECOMMENDATlON_BY_ITERATION recommen-dations to some other agents. The content of a recommendation is theDirect Interaction based reputation value about a randomly selectedtarget. The receiver of the recommendation is randomly selected.

5. Recommendations are represented by social commitments as for anymessage. Therefore, they are added to the debtor and creditor’s socialcommitment sets. The creditor of the recommendation uses the L.I.A.R.

decision process to decide if it trusts the recommendation. If it trustsit, the recommendation is used for the computation of the ReputationRecommendation based reputation.

6. As for other social commitments, recommendations are observed byNB_OBSERVATION_BY_ITERATION agents.

7. At last, each agent computes its reputation levels.

A simulation iterates these steps NB_ITERATIONS times. We testedseveral values for NB_ITERATIONS: 100, 200, 400, and 800. For mostof the simulations, a number of 400 steps is sufficient to obtain relevantresults.

The number of facts on which the agents can commit has an influenceon the time of convergence of the reputations. Indeed, the fewer factsthere are, the more chance there is that an agent contradicts itselfwhen it generates a new social commitment. We tested several values forNB_FACTS: 10, 20, 30, 50. We fixed the number of facts to 30 as it issufficiently high to allow agents not to commit too often on the same factsand also allows to show results in the fixed NB_ITERATIONS.

Behavior of AgentsEach agent is both a generator and a detector of norm violations. As a

generator, an agent is characterized by two parameters: VIOLATION_RATEand LIE_RATE. VIOLATION_RATE defines the rate of newly generatedsocial commitments on facet application that are inconsistent withprevious commitments of the agent. LIE_RATE has the same role,but for facet recommend. In this case, inconsistencies are generatedby false recommendations, which consists in sending a randomvalue selected in [−1,+1]� If not indicated otherwise, parametersVIOLATION_RATE and LIE_RATE are fixed as follows: each agenti ∈ 0, � � � ,N has a VI0LATI0N_RATE and a LIE_RATE set to i/N (whereN = NB_AGENTS − 1).

As detectors of violations, agents use two social norms to judge theirpeers’ behavior. These social norms forbid contradictions in emission andcontradictions in transmission. Figure 7(a), depicts the contradiction inemission, which is formalized in the Definition 5. In this case, a singleagent makes a social commitment that creates an inconsistency with someof its previous social commitments. Figure 7(b) depicts the contradictionin transmission, which is formalized in the Definition 6. In this case,an agent makes a social commitment that creates an inconsistency withsocial commitments it is the creditor of and that it has not disagreed with(disagreement should be explicit by changing the social commitment stateto canceled).

FIGURE 7 Contradictions in emission (a) and in transmission (b).

Definition 5 (Contradiction in Emission).

∀t ∈ � , snorm(I ,�(t),�(t),�(t), ∃x ∈ �(t) |cont_emission(t , x),active)

This social norm expresses that agents are forbidden to be in a situation ofcontradiction in emission. The predicate cont_emission(t , x) expressesthe fact that agent x is in a situation of contradiction in emission at instantt . It is formalized by (where SCS∗

x(t) = ⋃z∈�(t) SCS

zx(t)):

cont_emission(t , x) ≡ ∃y ∈ �(t), ∃c ∈ SCSyx(t) |

inconsistent(t , c , SCS∗x(t)\�c�)

This formula expresses the fact that agent x is debtor of a socialcommitment that is inconsistent with the overall set of the previous socialcommitments it was debtor of.

Definition 6 (Contradiction in Transmission).

∀t ∈ � , snorm(I ,�(t),�(t),�(t), ∃x ∈ �(t) |cont_transmission(t , x),active)

This social norm expresses that agents are forbidden to be in a situation ofcontradiction in transmission. The predicate cont_transmission(t , x)expresses the fact that agent x is in a situation of contradiction intransmission at instant t . It is formalized as follows (where SCSx

∗(t) =⋃z∈�(t) SCS

xz (t)):

cont_transmission(t , x) ≡ ∃y ∈ �(t), ∃c ∈ SCSyx(t) |

inconsistent(t , c , SCSx∗(t))

This formula expresses the fact that agent x makes a social commitmentc which creates an inconsistency with social commitments that have beenpreviously taken toward it.

As detectors, agents have to generate social policies from social norms.We define two generation strategies: forgiving and rancorous. In the forgivingstrategy, agents consider that a social policy is no longer violated if anagent cancels a posteriori one of the social commitments involved in theinconsistency so that the content of the social policy is no more false; in therancorous strategy, agents consider that when a social policy is violated, itremains violated forever. A parameter, NB_VI0LAT0RS_IN_P0PULATION,allows to fix the number of agents which violate the norms and another,NB_FORGIVER_IN_POPULATION, the number of these agents that areforgivers (the other being rancorous).

Finally, we set the parameters for the reasoning process as follows, foragents to be able to make decisions based on the reputations:

• trust�bRp = 0�8,∀� ∈ �DI,II,ObsRc,EvRc,RpRc�

• distrust�bRp = 0�5,∀� ∈ �DI,II,ObsRc,EvRc,RpRc�

• relevanceDIbRp = 10, relevanceIIbRp = 7, relevanceRpRcbRp = 5

Also, if it is not stated differently, the penalties associated with thesocial policies are fixed at 1.0 for every agent.

Description of the ExperimentsSimulations were run with NB_AGENTS = 11. This allows to run each

simulation configuration 10 times and to present results that are theaverage over these runs.

As the formulae defined to compute the ObsRcbRp and EvRcbRp aresimilar to those used for DIbRp and IlbRp, the results are similar. Themost prominent difference between the computations of these reputationsrelies in presence or absence of a filtering process. To illustrate theinfluence of this filtering process, we decided to present results forRpRcbRp, i.e., to restrict the recommendations to levels of reputation.

Also, we present results mostly for the application facet. Similarresults have been obtained with the recommend facet.

Experimentation Criteria

The ART-testbed group (Fullam et al. 2005) defined a set of propertiesthat a good reputation model should exhibit. We use these propertiesas evaluation criteria to estimate the performance of the L.I.A.R. model.

These evaluation criteria are as follows:

• Multi-�: A reputation model should be multi dimensional and multifaceted. By design, L.I.A.R. is multi-�, therefore we will not consider thiscriterion further.

• Quickly converging: A reputation model should enable an agent tocompute reputation levels that tend quickly to model the target’sbehavior.

• Precise: A reputation model should compute reputation levels thatmodel precisely the target’s behavior.

• Adaptive: A reputation model should be able to adapt the reputationlevels in case the target changes its behavior.

• Efficient: A reputation model must compute reputation levels withoutconsuming, top, much of the agent’s resources.

Finally, we show how agents using L.I.A.R. can identify and isolatemalevolent agents and decide with whom to interact.

Precision and Convergence

This section presents some experimental results about the precisionand convergence of L.I.A.R..

StrategiesThe two strategies (forgiving and rancorous) are compared in Figure 8.

This figure shows the ratio of the number of violations detected to thetotal number of evaluations along the y-axis. The simulation time steps arealong the x -axis. The dotted line represents the contradiction ratio of theconsidered agent (here 20%), i.e., the ratio of commitments that this agentdoes not cancel before making new commitments that are inconsistent.

The rancorous strategy converges toward a relevant value after abit more than 100 iterations. However, the forgiving strategy does notconverge. This is essentially because some violated commitments arecanceled a posteriori. An important change in the set of violated socialcommitments can then occur from one step to another. In the remainingexperiments, agents will always adopt a rancorous strategy.

Parameters Influencing Precision and ConvergenceFigure 9 shows the influence of the number of direct interactions on

the convergence of the Direct Interaction based reputation. As we canintuitively expect, it converges faster if it is computed from more inputs(direct interactions). The same results are observed for the influence ofthe Indirect Interaction based reputation.

FIGURE 8 Related precision and convergence with respect to the strategy.

In the case of reputations based on recommendations, it is slightlydifferent as propagators may lie in their recommendations (it is notthe case for interactions that are assumed to be correctly observed).L.I.A.R. provides a filtering mechanism to detect and ignore false

FIGURE 9 Direct Interaction based reputation with varying NB_ENC0UNTER_BY_ITERATI0N.

FIGURE 10 Convergence and precision of the Reputation Recommendation based reputation.

recommendations. However, the convergence speed and value of thesereputations depends on the number of agents that propagate falserecommendations. Figures 10(a) and 10(b) show the evolution of theReputation Recommendation based reputation (y-axis) over time (x -axis),respectively, then the filtering mechanism is not used and when it is used.

In the simulation presented in these figures, all violators have aVI0LATI0N_RATE of 0.8 meaning than in 80% of the cases they sendrecommendations with a random value. As the bad recommendations areuniformly randomly generated values in [−1,+1], their average tend to 0.This is the reason why when filtering is not used, the convergence valuesare attracted toward 0 by violators. The more violators there are, the closerto 0 is the reputation value (and the farther from a precise estimation ofthe target reputation).

Figure 10(b) shows the benefit of the filtering mechanism. Thedetection of violators allows an agent to ignore recommendations comingfrom them and then to compute a more precise reputation value. Therecan still be undetected violators that disturb the computation but theirimpact is lowered. There is no result (Figure 10(b) lacks a fourth line) inthe case of every other agent being a violator because the filtering processrejects all the recommendations. Therefore, the reputation stays unknownduring the entire simulation.

Adaptivity

Adaptivity is the capacity of a reputation model to react quickly toan important change in an agent behavior. Here, we consider an agentthat has a violation rate of 20% and that changes its behavior during thesimulation to a violation rate of 100% (it never cancels any commitment).

We study the adaptivity under two aspects: the inertia (i.e., the timethe model needs to adapt the reputation level) and the fragility (i.e, if

the decrease of the reputation of the target is more important when itsreputation was higher before the change).

InertiaThe formula denned in section 3.3.2 to compute the Direct Interaction

based reputation uses the overall set of direct interaction that the agenthas accumulated from its entrance in the system. It is therefore expectedthat the reputation level shows some inertia.

Figure 11 shows the level of the Direct Interaction based reputationcomputed by an agent for a given target along the y-axis. Time stepsare along the x -axis. The figure shows the inertia when a change in thebehavior of the target occurs at time step 50 (plain line), time step 100(dashed line or time step 150 (dotted line). The figure scope has beenextended to 800 time steps to confirm the decreasing of the reputationlevel. These results confirm that the model has some inertia as the updateof reputation levels is faster when the change occurred earlier.

This inertia can be lowered if agents use a time window to forgetsocial commitments that occured a long time ago. Figure 12 shows theinfluence of time windows. The plain line shows the Direct Interactionbased reputation with no time window (agents use all the evaluationsthey have), the dotted line shows a window of 50 social policies, and thedashed line shows a window of 10 social policies. Here, a “time window

FIGURE 11 Inertia of the Direct Interaction based reputation.

FIGURE 12 Influence of time windows on the inertia of the Direct Interaction based reputation.

of k” consists in considering only the k latest generated social policies. Ofcourse, the results show that the smaller the time window, the smaller theinertia.

However, the smaller the time window, the higher the weight of asingle social policy in the computation of the reputation level. As aconsequence, the sliding of the time window, which makes the systemforget about earliest social policy and consider a new one, can make asubstantial change in the reputation level. This consequence is particularlyhighlighted by the instability of the dotted line.

FragilityGambetta (2000) highlighted the fact that reputation should be fragile.

That means it is slow to get a good reputation but it is fast to get abad reputation with only a few bad actions, especially if the agent thatdeceives has, before its bad actions, a good reputation. Figure 13 showsthat reputations in L.I.A.R. are quite fragile.

Each line shows the Direct Interaction based reputation for a differenttarget: The plain line is for a target that had a “good” behavior before thechange (it did not cancel previous inconsistent commitments in 10% ofthe cases), the dashed line shows a “medium” behavior (the target did not

FIGURE 13 Fragility of the Direct Interaction based reputation.

cancel in 40% of the cases), and the dotted line shows a “bad” behavior(the target did not cancel in 60% of the cases).

The figure shows the fragility when a change in the behavior ofthe targets occurs at time step 100. The reputations of the agentsthat previously had a good behavior drop faster than the other agentsreputation. However, this result do not show an important fragility.

It is possible to configure fragility in L.I.A.R. while setting thepenalty values associated to social policies. Figures 14(a) and 14(b) showthe difference of evolution of the Direct Interaction based reputationaccording to the penalties.

FIGURE 14 Influence of the penalties of the social policies.

In the figures, agi denotes an agent (number i) that has aVIOLATION_RATE of i/(NB_AGENTS − 1). In Figure 14(a), the penaltyassociated with social policies is always 1.0. In Figure 14(b), it is set to1.0 when the social norm is fulfilled and to k when it is violated, where kis the number of social commitments involved in the contradiction. Thislatter way of setting penalties represents the fact that the agent considersworse a violation involving more social commitments. As the figure shows,increasing penalties associated with violated social policies augments thefragility.

Efficiency

The formulae proposed in this article for the computation of thereputations are linear combinations of simple computations on the setsof social policies or recommendations. As a consequence, the theoreticalcomplexity of the formulae is linear with respect to the size of the sets.

Figure 15 confirms that the progression of the time needed to computethe Reputation Recommendation based reputation is linear with theduration of the presence of an agent in the system, and therefore withthe size of the sets of recommendations it computes the reputation level.The model requires less than 3ms to update a reputation with a codewritten in Java and run on a 1.7G-Hz processor. In these experiments thetime is coarsely estimated by computing the difference between the time

FIGURE 15 Efficiency of the Reputation Recommendation based reputation computation.

before and after the call of the studied method. The parasitic peaks are theconsequence of the OS’s interruptions on the process running the methodexecution. The dashed line draws the linear equation y = 1/2000 · x .

The other reputations need less time to compute (about 1ms), as theydo not involve the filtering process.

Decision

Finally, the decision process of L.I.A.R. has been tested to check that itenables agents to identify and isolate harmful agents and to identify goodagents to reinforce their interactions with such agents. Figures 16(a) to16(c) show experimental results in the form of graphs of trust intention,i.e., graphs the nodes of which are the agents and edges the trust intentionrelations. To increase the readability of the graph, links are not orientedand represent reciprocal trust intentions. This way, a link will disappearwhen an agent A does not trust anymore another agent B (and there won’tbe a link to represent that B trusts A). The width of a link is emphasizedfor trust intentions with higher weights.

In this experiment, agents 8, 9, and 10 are “good” agents (i.e., they donot cancel their commitments 10% of the time) and the others, are “bad”agents (i.e., they do not cancel in 80% of the cases). At the beginning the

FIGURE 16 Graph of trust intentions at steps 1, 50, and 100.

graph is fully connected, meaning that the General Disposition to Trustis configured so that every agent trust every other agent (Figure 16(a) attime step 1).

The graphs show that the agents quickly (in around 50 time steps)detect which are the “bad” agents and do not have the intention totrust them. At time step 100, only the agents having a good behavior areconnected.

Similar results can be obtained with the forgiving strategy, with aconvergence in about 150 time steps, due to the higher inertia.

DISCUSSION

L.I.A.R. has been designed to implement social control in open anddecentralized MAS, such as peer-to-peer systems. We discuss here thespecific properties of L.I.A.R. that makes it suited to such networks.

One of the main specificity of ODMAS is that its decentralized naturemakes it impossible to have a global centralized view of everything thathappens in the system. It is thus impossible to control the system in a waysuch that every violation can be detected and sanctionned. L.I.A.R. takesinto account this decentralization by providing mechanisms to implementan agent that can reason locally on its partial observation of the behaviorof its neighbors. Propagation is then used to inform other agents thata violation occurred to speed up their learning process of other agents’reputation. Of course, some violations, especially at the beginning of thesimulations, remain undetected and are not sanctionned, but we believeit is impossible to obtain such a perfect control in ODMAS. However,experiments show that L.I.A.R. can compute agents reputation quitequickly and also update these reputations if agents change their behavior.

The justification protocol is an original contribution of the L.I.A.R.model. Some existing reputation models (Sabater-Mir and Sierra 2002;Huynh, Jennings, and Shadbolt 2004; Sabater-Mir, Paolucci, and Conte2006) propose that agents communicate to share their observations andexperiences with others to provide them some information they do notperceive, but they never consider the case where an agent is wronglyconsidered as a violator because of insufficient information. Thus, theL.I.A.R. justification protocol gives the chance to the suspected violatorto prove that it did not perform any violation by providing additionalinformation to its evaluator. This is also a feature of L.I.A.R. that makes itsuited to decentralized systems.

The decentralized nature of L.I.A.R. facilitates scalability because thereis no bottleneck in the system. Our experiments (Figure 15) confirmedthat the efficiency of L.I.A.R. depends linearly on the size of the sets ofevaluations or recommendations used to compute the reputations. Therisk is, then, that an agent accumulates more and more evaluations and

recommendations, and takes more and more time to compute reputations.If this case should be avoided, we proposed the use of time windows to puta limit to the maximum number of evaluations considered.

Finally, a difficult problem encountered in reputation systems is therisk of collusions. Sets of malevolent agents can be deployed tryingto fool the reputation mechanisms by recommending each other andsending false reputation values about other agents. L.I.A.R. proposes arecursive use of its reputation model on agents while they are acting aspropagators to assign a bad reputation in recommendation actions tomalevolent agents. Recommendations could then be filtered to ignorethe ones coming from malevolent agents. However, the bigger is theproportion of agents involved in the collusion, the harder it is to detectall of them and to reduce their influence. Experiments (Figure 10) haveshown the importance of their impact on reputation values according tothe size of the collusion. We also tested L.I.A.R. with a high number ofmalevolent agents (more than 70%) and showed (Figure 16) that goodagents managed after some time to identify and isolate malevolent agentsand that they formed a kind of trusted coalition.

RELATED WORKS

The objective of L.I.A.R. is to provide models that covers all thenecessary tasks that agents must perform to achieve social control. Thus,it includes a social commitment model to observe interactions, a socialnorm and policy model to represent the system rules and evaluate theagents’ behavior, a reputation model to judge and potentially excludeother agents. The contribution of L.I.A.R. regarding related works in eachof these field is discussed here.

Social commitments have been proposed (Singh 2000) to representcommunicative act in an objective way (as opposed to the mentalisticapproach adopted when using speech acts [Cohen and Levesque 1995])that makes it suited to perform external observations and control. Singh(2000) proposed a model for social commitments that includes a debtor,a creditor, a witness, and a content. Also, he proposed (as well asFornara and Colombetti 2003) means to convert speech acts into socialcommitments. These social commitments are intended to be stored inglobal commitment stores (Hamblin 1970). The concept of sanction hasthen been introduced by the model of Pasquier et al. (2004) accordingto the states of the social commitment (for instance, if it is violated).There are three main differences between the models presented aboveand the one defined in L.I.A.R. and that makes the latter particularlyadapted to open and decentralized systems. First, we added the notion ofobserver in the model, so that the commitment sets are localized; eachagent can have its own local representations of the social commitments

it has perceived. Second, because agents only have partial perceptionof the system, they can miss some of the speech acts that have beenuttered. Therefore, the state they believe the social commitment is in isnot necessarily its “true” state. This is the reason why the state of thelocal representation of the social commitments does not necessarily followthe life cycle given in section 2.1.2. Finally, because the detection that aviolation occurred generally happens a posteriori, it is important to considerthe time dimension into the model for an agent to get the state of a socialcommitment at a previous instant. These considerations are intrinsicallylinked to decentralized systems and have been integrated into the model.

The concept of social norm, introduced by Tuomela (1995)considering human societies, inspired several works in multiagent system.First, it is necessary to formally describe them. Two main formalisms areusually used: deontic logic (von Wright 1951) and descriptive models(Vàzquez-Salceda 2003; Kollingbaum and Norman 2005; López y Lópezand Luck 2004). In L.I.A.R., the second approach is used because it ismore expressive (Vàzquez-Salceda 2003) and less prone to computationallimitations, like undecidability or paradoxes (Meyer, Dignum, andWieringa 1994). Some other works are interested rather in the behaviorof agents and their situation regarding the satisfaction or violation ofsocial norms. This is represented by the concept of social policies (Viganò,Fornara, and Colombetti 2005; Singh 1999). L.I.A.R. considers these twoaspects and integrates both a social norm and a social policy model. As forsocial commitment, the originality of L.I.A.R. is to localize these models,assuming that only partial observations can be used and that it may benecessary to revise some previous decisions about norm violations.

The main research field related to L.I.A.R. is reputation in multiagentsystems. Marsh (1994) proposed what is considered to be the firstcomputational reputation model. He proposed both means to computereputations and to make decisions based on these reputations. The useof propagation to speed up the learning process of reputations has thenbeen integrated in the works of Schillo et al. (1999) and Sen and Sajja(2002). However, these reputation models remain quite simple as theydo not tackle the problem of detecting malicious behavior (they assumethat agents directly observe violations) and they assume that agents arehonest in their recommendation. More recent works proposed richermodels to obtain a finer evaluation of other agents. For instance, REGRET(Sabater-Mir 2002) and FIRE (Huynh, Jennings, and Shadbolt 2004) takesinto account several types of reputations according to the nature of theinformation: individual, social, or ontological in REGRET; interactions,roles, witnesses, or certification in FIRE. REGRET has also been extendedin REPAGE (Sabater-Mir, Paolucci, and Conte 2006) to integrate boththe concepts of reputation, as a collective judgement, and image (Conteand Paolucci 2002), as an individual judgement. The L.I.A.R. reputation

model can be classed in the same category as these last models evenif it exhibits a few differences. Here also, different reputation types areconsidered according to the information sources. But the reputations ofan agent also depend on facets and dimensions that allow to judge anagent according to a particular aspect of its behavior (its integrity, itscompetence, � � � ) or to a context of interaction. It is thus possible to usethe L.I.A.R. reputation model recursively to trust or distrust other agentswhen they send recommendations about a target. This trust decision aboutthe propagator is then naturally integrated in the trust decision processabout the target. The reasoning process of L.I.A.R. is also original becauseit does not require a final merge of all reputation types, which implies theloss of semantical fineness of reputation distinction, as it is usually done inthe models cited above.

CONCLUSION

Here, we focused on the problem of controlling agents’ interactionsin open and decentralized multiagents systems. Following Castelfranchi(2000) arguments, we considered it is better adapted in such contexts todeploy the social form of control, i.e, an adaptative and auto-organizedcontrol set by the agents themselves. As a consequence, we proposedL.I.A.R., a model that any agent can use to participate to the social controlof its peers’ interactions. This model is composed of various submodelsand processes that enable agents, first, to characterize the interactions theyperceive and, second, to sanction their peers.

A model of social commitment was defined first that allows agentsto model the interactions they perceive in a nonintrusive way. Then, weproposed models of social norm and social policy. The former, define,the rules that must be respected by the agents during their interactions.The latter evaluated the compliance of agents regarding these rules. Tocompute this compliance from the social commitments and the socialnorms into social policies, we also proposed an evaluation process thatagents can deploy to detect the violation or respect of the social norms.Based on these models and processes, agents build reputation models oftheir neighbors so that malicious ones can be excluded by ostracism.

The original contribution of L.I.A.R. is that it consists in a completeframework that integrate several models going from observation to trustdecision, required to participate to a social control. It has also beendefined to be deployed in open and decentralized systems. Specific aspectsof these systems—such as the fact that there is no possible centralizationof any task, that only partial observation can be performed, and that everyagent can lie in any kind of message—have been taken into account inL.I.A.R.

Experimental results are presented in a simulation of a peer-to-peernetwork to evaluate the model according to the criteria defined by theART-testbed group (Fullam et al. 2005). The results show that the L.I.A.R.model allows the agents to identify and isolate malevolent agents.

This modification of the neighborhood of agents to isolate malevolentagents can be seen as a kind of self-organization of the system to react tothe intrusion of malevolent agents. A preliminary version of L.I.A.R. hasbeen used in a previous work (Grizard et al. 2007) for a peer-to-peer systemself-organization considering norm violation but also agents’ preferencesto form coalitions of agents that work well together. In our future works,we will explore deeper the links between self-organization and reputation.Another improvement that could be useful would be to drop the necessaryassumption of L.I.A.R. that all agents participating in the social control usethe same reputation model. The use of heterogeneous reputation modelscauses a huge problem of interoperability. We are also working in thatdirection (Vercouter et al. 2007; Nardin et al. 2008) to bring solutions andto permit the use of L.I.A.R. jointly with other models.

APPENDIX A: GENERATION OF THE SOCIAL POLICIES

Below is an example of a process that a single agent (this) can useto generate the social policies from a norm sn) and for a specific target,agent x . It has to be executed for all the norms the agent knows and forall possible targets. It is written with a Java-like syntax.

The method getMatchingSComs is assumed to return a list of setsof social commitments that match the content of the norm. Finding thematching social commitments can be done by unification of the variablesoccurring in the content of the norm with the social commitmentsof thisSCS(t) or their composing elements (debtor, content, etc.). Forinstance, if the norm sn is the one of Example 3, then this method simplyconsists in finding single commitments which facets include politics. Ifthe norm is the one used in section 4, then the method must work on pairsof social commitments.

The method generateSPolContent generates the content of thesocial policy from the content of the norm and a set of matching socialcommitments. It consists of instantiating the variables of the content ofthe social norm sn with the social commitments or social commitmentcomposing elements. For instance, if the norm sn is the one of Example 3then this method simply consists of replacing ∀sc by a particular socialcommitment sc which content’s facets include politics.

REFERENCES

Bentahar, J., B. Moulin, and B. Chaib-Draa. 2003, July. Towards a formal frame work forconversational agents. In: Proceedings of the Workshop on Agent Communication Languages andConversation Policies at Autonomous Agents and Multi-Agent Systems (AAMAS’03), eds. M.-P. Hugetand F. Dignum, Melbourne, Australia.

Blaze, M., J. Feigenbaum, and J. Lacy. 1996, May. Decentralized trust management. In Proceedingsof the IEEE Symposium on Security and Privacy, 164–173. Oakland, CA: IEEE Computer Society,Washington, DC.

Castelfranchi, C. 2000, December. Engineering social order. In Proceedings of Engineering Societies inthe Agents World (ESAW’00), eds. A. Ominici, R. Tolksdorf, and F. Zambonelli, Volume 1972 ofLecture Notes in Computer Science. 1–18. Berlin, Germany: Springer-Verlag.

Castelfranchi, C. and R. Falcone. 1998. Principles of trust for MAS: Cognitive anatomy, socialimportance, and quantification. In Proceedings of the International Conference on Multi-Agent Systems(ICMAS’98), ed. Y. Demazeau, 72–79. Paris, France: IEEE Computer Society, Washington, DC.

Cohen, P. and H. Levesque. 1995. Communicative actions for artificial agents. In Proceedings of theInternational Conference on Multi Agent Systems (ICMAS’95), 65–72. Cambridge, MA: MIT Press.

Conte, R. and M. Paolucci. 2002. Reputation in Artificial Societies. Social Beliefs for Social Order.Dordrecht, The Netherlands: Kluwer Academic Publishers.

Dasgupta, P. 1990. Trust as a commodity. In Trust Making and Breaking Cooperative Relations, 49–72.New York: Basil Blackwell. (electronic edition, Department of Sociology, University of Oxford,Oxford, United Kingdom).

Fornara, N. and M. Colombetti. 2003, July. Defining interaction protocols using a commitment-based agent communication language. In Proceedings of Autonomous Agents and Multi-Agent Systems(AAMAS’03), 520–527. Melbourne, Australia: ACM Press, New York.

Fullam, K., T. Klos, G. Muller, J. Sabater, A. Schlosser, Z. Topol, K. S. Barber, J. S. Rosenschein,L. Vercouter, and M. Voss. 2005, July. A specification of the agent reputation and trust(ART) testbed: Experimentation and competition for trust in agent societies. In Proceedings ofAutonomous Agents and Multi-Agent Systems (AAMAS’05), eds. F. Dignum, V. Dignum, S. Koenig,S. Kraus, M. P. Singh, and M. Wooldridge, 512–518. Utrecht, The Netherlands: ACM Press,New York.

Gambetta, D. 2000. Can we trust trust? In Trust. Making and Breaking Cooperative Relations, 213–237.New York: Basil Blackwell. (electronic edition, Department of Sociology, University of Oxford,Oxford, United Kingdom).

Grizard, A., L. Vercouter, T. Stratulat, and G. Muller. 2007. A peer-to-peer normative system toachieve social order. In Post-Proceedings of the Workshop on Coordination, Organizations, Institutions,and Norms in Agent Systems at Autonomous Agents and Multi-Agent Systems (AAMAS’06), eds. V.Dignum, N. Fornara, and P. Noriega, 274–289. Volume LNCS 4386 of Lecture Notes inComputer Science. Berlin, Germany: Springer-Verlag.

Hamblin, C. L. 1970. Fallacies. London, United Kingdom: Methuen.He, Q., K. P. Sycara, and Z. Su. 2001. Proceedings of Trust and Deception in Virtual Societies,

Security infrastructure for software agent society. Kluwer Academic Publishers, pp. 139–156.Herzig, A., E. Lorjni, J. F. Hübner, and L. Vercouter. 2010. A logic of trust and reputation. Logic

Journal of the IGPL 18(1):214–244 (February). Special Issue Normative Multiagent Systems.Huynh, T. D., N. R. Jennings, and N. R. Shadbolt. 2004. FIRE: An integrated trust and reputation

model for open multi-agent systems. In Proceedings of the 16th European Conference on ArtificialIntelligence, ECAI’2001, eds. Ramon Lopez de Mantaras and Lorenza Saitta, 18–22. IOS Press.

Kollingbaum, M. J. and T. J. Norman. 2005, July. Informed deliberation during norm-governedpractical reasoning. In Proceedings of the Workshop on Agents, Norms and Institutions for RegulatedMulti-Agent Systems (ANIREM) at Autonomous Agents and Multi-Agent Systems (AAMAS’05), eds O.Boissier, J. Padget, V. Dignum, G. Lindemann, E. Matson, S. Ossowski, J. S. Sichman, and J.Vàzquez-Salceda, 19–31. Volume 3913 of Lecture Notes in Artificial Intelligence. Utrecht, TheNetherlands: Springer-Verlag, Berlin, Germany.

Labrou, Y. and T. Finin. 1994, November. A semantics approach for KQML—a general purposecommunication language for software agents. In Proceedings of the Conference on Information andKnowledge Management (CIKM’94), 447–455. Gaithersburg, MD: ACM Press, New York.

López y López, F. and M. Luck. 2004, January. A model of normative multi-agent systems anddynamic relationships. In Proceedings of the Workshop on Regulated Agent-Based Social Systems:Theories and Applications (RASTA) at Autonomous Agents and Multi-Agent Systems (AAMAS’02),eds. G. Lindemann, D. Moldt, and M. Paolucci, 259–280. Volume 2934 of Lecture Notes inComputer Science. Bologna, Italy: Springer-Verlag, Berlin, Germany.

Marsh, S. 1994, April. Formalizing trust as a computational concept. Ph.D. dissertation, Departmentof Computer Science and Mathematics, University of Stirling, Scotland, United Kingdom.

McKnight, D. H. and N. Chervany. 2001, May. Trust and distrust definitions: One bite at a time. InProceedings of the Workshop on Deception, Fraud and Trust in Agent Societies at Autonomous Agents andMulti-Agent Systems (AAMAS’01), 27–54. Volume 2246 of Lecture Notes fn Computer Science.Montreal, Canada: Springer-Verlag, London, United Kingdom.

Meyer, J.-J. Ch., F. P. M. Dignum, and R. J. Wieringa. 1994. The paradoxes of deontic logic revisited:a computer science perspective. Technical Report UU-CS-1994-38, Utrecht University, Utrecht,The Netherlands.

Nardin, L. G., A. A. F. Brandao, J. S. Sichman, and L. Vercouter. 2008, October. A service-orientedarchitecture to support agent reputation models interoperability. In 3rd Workshop on Ontologiesand Their Applications (WONTO20192008). Salvador, Bahia, Brazil.

OMG. 2005, August. Unified Modeling Language: Superstructure, v2.0. Technical Report, ObjectManagement Group. http://www.omg.org/technology/documents/vault.htm#modeling.

Pasquier, P., R. A. Flores, and B. Chaib-draa. 2004, October. Modelling flexible social commitmentsand their enforcement. In Proceedings of Engineering Societies in the Agents’ World (ESAW’04), 139–151. Volume 3451 of Lecture Notes in Computer Science. Toulouse, France.

Plaza, E., J. Lluìs Arcos, P. Noriega, and C. Sierra. 1998. Competing agents in agent-mediatedinstitutions. Personal and Ubiquitous Computing 2(3):212–220.

Quéré, L. 2001. La structure cognitive et normative de la confiance. Réseaux 19(108):125–152.Sabater-Mir, J. 2002, July. Trust and reputation for agent societies. Ph.D. dissertation, Artificial

Intelligence Research Institute, Universitat Autònoma de Barcelona, Barcelona, Spain.Sabater-Mir, J. and C. Sierra. 2002. Social ReGreT, a reputation model based on social relations.

SIGecom Exchanges 3(1):44–56.Sabater-Mir, J., M. Paolucci, and R. Conte. 2006. Repage: REPutation and ImAGE among limited

autonomous partners. Journal of Artificial Societies and Social Simulation 9(2):3.Schillo, M., P. Funk, and M. Rovatsos. 1999, May. Who can you trust: Dealing with deception. In

Proceedings of the Workshop on Deception, Fraud, and Trust in Agent Societies at Autonomous Agents(AA’99), eds. C. Castelfranchi, Y. Tan, R. Falcone, and B. S. Firozabadi, 81–94. Seattle, WA.

Sen, S. and N. Sajja. 2002, July. Robustness of reputation-based trust: Boolean case. In Proceedings ofAutonomous Agents and Multi-Agent Systems (AAMAS’02), eds. M. Gini, T. Ishida, C. Castelfranchi,and W. L. Johnson, 288–293. Volume 1. Bologna, Italy: ACM Press, New York.

Singh, M. P. 1991, November. Social and psychological commitments in multi-agent systems. InProceedings of the AAAI Fall Symposium on Knowledge and Action at Social and Organizational Levels(longer version), 104–106. Monterey, CA.

Singh, M. P. 1999. An ontology for commitments in multi-agent systems: Towards a unification ofnormative concepts. AI and Law 7:97–113.

Singh, M. P. 2000. A social semantics for agent communication languages. In Proceedings of theWorkshop on Agent Communication Languages at the International Joint Conference on ArtificialIntelligence (IJCAI’99), eds. F. Dignum and M. Gieaves, 31–45. Heidelberg, Germany: Springer-Verlag.

Tuomela, R. 1995. The Importance of Us: A Philosophical Study of Basic Social Norms. Stanford, CA:Stanford University Press.

Vàzquez-Salceda, J. 2003, April. The role of norms and electronic institutioi multi-agent systemsapplied to complex domains. The HARMONIA framework. Ph.D. dissertation, UniversitatPolitécnica de Catalunya, Barcelona, Spain.

Vercouter, L., S. J. Casare, J. S. Sichman, and A. A. F. Brandao. 2007. An experience on reputationmodels interoperability using a functional ontology. In Proceedings of the International JointConference on Artificial Intelligence (IJCAI’07). Hyderabad, India, pp. 617–622.

Viganò, F., N. Fornara, and M. Colombetti. 2005, July. An event driven approach to norms inartificial institutions. In Proceedings of the Workshop Agents, Norms and Institutions for REgulatedMulti-Agent Systems (ANIREM) at Autonomous Agents and Multi-Agent Systems (AAMAS’05), eds. O.Boissier, J. Padget, V. Dignum, G. Lindemann, E. Matson, S. Ossowski, J. S. Sichman, and J.Vázquez-Salceda, 142–154. Volume 3913 of Lecture Notes in Artificial Intelligence. Utrecht,The Netherlands: Springer-Verlag, Berlin, Germany.

von Wright, G. H. 1951. Deontic logic. Mind 60:1–15.

L.I.A.R.: ACHIEVING SOCIAL CONTROL IN OPEN AND DECENTRALIZED MULTIAGENT SYSTEMS

Documents

Transcript of L.I.A.R.: ACHIEVING SOCIAL CONTROL IN OPEN AND DECENTRALIZED MULTIAGENT SYSTEMS

A4M33MAS - Multiagent Systems Agents and their behavior … · 2018-09-19 · Multi-agent systems & Logic • Multi-agent systems – Complex decentralized systems whose behaviour

Decentralized Control Applied to Multi-DOF Tuned-Mass Damper Design Decentralized H2 Control Decentralized H Control Decentralized Pole Shifting Decentralized.

COLA: Decentralized Linear Learningpeople.inf.ethz.ch/ybian/docs/poster-cola.pdf · References [1] Nedicet al.16’Achieving geometric convergence for distributed optimization over

Agents and Multiagent Systems

Fundamentals of Multiagent Systems

INFRASTRUCTURES FOR THE ENVIRONMENT OF MULTIAGENT SYSTEMScse.unl.edu/~lksoh/Classes/CSCE475_875_Fall17/seminars/Seminar_… · Provide agents with services useful for achieving individual

LECTURE 6: MULTIAGENT INTERACTIONS

Developing Multiagent Systems: The Gaia Methodology · Developing Multiagent Systems: The Gaia Methodology FRANCO ZAMBONELLI ... Developing Multiagent Systems: The Gaia Methodology

Multiagent System Communication

Multiagent Cooperative Search for Portfolio Selectionparkes/pubs/gebfinal.pdf · Multiagent Cooperative Search for Portfolio Selection ... MULTIAGENT COOPERATIVE SEARCH 125 paper

Learning Policy Representations in Multiagent Systemsmshediva/assets/pdf/multiagent... · 2021. 1. 5. · Learning Policy Representations in Multiagent Systems Aditya Grover 1Maruan

Achieving Goals in Decentralized POMDPs

MULTIAGENT SYSTEMS IN MODULAR ROBOTICS50 modular robotics, multiagent systems, metamorphic structures of robots Rudolf JÁNOŠ * MULTIAGENT SYSTEMS IN MODULAR ROBOTICS Abstract The

From Multiagent Systems to Multiagent Societies Michael Berger Based on: 1) “Multiagent Systems and Societies of Agents” / Michael N. Huhns and Larry M.

Simulació multiagent de la Girona medieval emprant SDLPS 1 ... · Simulació multiagent de la Girona Medieval emprant SDLPS 3 DADES DEL PROJECTE Títol del Projecte: Simulació multiagent

MULTIAGENT SYSTEMS – AN INTRODUCTION

TRANSITION IN MULTIAGENT ORGANIZATIONS

Engineering Ethical Multiagent Systems

Chapter 16: Multiagent Systems

CSCE 875 Seminar: Multiagent Learning using a Variable ...cse.unl.edu/~lksoh/Classes/CSCE475_875_Fall11/seminars/Seminar... · CSCE 875 Seminar: Multiagent Learning using a ... Multiagent