CogSysI Lecture 11: Intrduction to Multi-Agent Systems · CogSysI Lecture 11: Intrduction to...
Transcript of CogSysI Lecture 11: Intrduction to Multi-Agent Systems · CogSysI Lecture 11: Intrduction to...
CogSysI Lecture 11: Intrduction toMulti-Agent Systems
Intelligent Agents
Ute Schmid (lecture)
Emanuel Kitzelmann (practice)
Applied Computer Science, Bamberg University
last change: July 1, 2008
Schmid, CogSysI-11, MAS – p. 1
The Idea of MAS
An agent is a computer system that is capable ofindependent action on behalf of its owner or user.
A multiagent system consists of a number of agentswhich interact, typically by exchanging messages viasome computer network infrastructure.
Different agents might represent users/owners withdifferent goals/motivations.
Therefore, to succesfully interact, agents require theability to
CooperateCoordinateNegotiate
with each other (similar to interaction of people ineveryday live)
Schmid, CogSysI-11, MAS – p. 2
Key Research Questions
Micro/Agent Design: how to build agents capable ofindependet, autonomous action
Macro/Society Design: how to build agents capable ofinteracting with other agents, esp. if they have differentgoals/interests?
Standard AI: focus on intelligent individual
MAS: Social abilitiesEmergence of cooperation in a society ofself-interested agentsLanguage to communicate beliefs and aspirationsConflict recognition and resolutionCoordination of activities to reach common goals
Schmid, CogSysI-11, MAS – p. 3
Example Scenarios
NASA Deep Space 1 mission (1998): space probe withan autonomous, agent-based control system which canmake some decisions by itself (before: control decisionswere completley done by a 300 person ground crew)
Autonomous air-traffic control systems: recognition offailure of other control systems and cooperation to trackand deal with attended flights (e.g. DVMT, Durfee;OASIS)
Last minute holiday package via PDA, using anegotiating agent
Schmid, CogSysI-11, MAS – p. 4
MAS is Interdiciplinary Research
Software Engineering: Agent paradigm(going beyond OO)
Social Sciences: Using theories, gaining insights bysimulation of artificial societies
AI: use planning, reasoning, learning technologies;study intelligent behavior in dynamic, interactiveenvironments
Game theory: use theories and techniques fornegotiation
Schmid, CogSysI-11, MAS – p. 5
Definition: Agent
An agent is a computersystem that is situated insome environment, and thatis capable ofautonomous action in this en-vironment in order to meetits design objectives.
Agent
Sensors Actuators
Action
OutputInput
Percept
"compute"
Environment
Compare to
Control systems: e.g. thermostate
Software demons: e.g. xbiff
Schmid, CogSysI-11, MAS – p. 6
Environments
Accessible vs. inaccesibleobtaining complete, accurate, up-to-date information about theenvironments state
Deterministic vs. non-deterministiceach action has a single, guaranteed effect, no uncertaintyabout the result of an action; note: highly complex deterministicenvironments must be handled as non-deterministic
Static vs. dynamicenvironment remains unchanged except by performance of theagent
Discrete vs. continuousfixed, finite number of actions and percepts
→ Open Env: inaccessible, non-deterministic, dynamic, continuous
Schmid, CogSysI-11, MAS – p. 7
Reactive Systems
Two sources of complexity of MAS: characteristics ofenvironment and nature of interaction between agentand environment
Reactive system: maintainance of interaction withenvironment, must be studied and described on abehavioral level (not a functional level, i.e. in classicalterms of pre- and postconditions)
Example: reactive planning systems
Local decisions have global consequences
Example: printer controllerSimple rule: first grant access to process p1 and atsome later time to process p2 is unfair, because it mightnever grant access to p2
Schmid, CogSysI-11, MAS – p. 8
Intelligent Agent
An intelligent agent is a computer system with theability to perform actions independently, autonomously,and flexible (on behalf of a user or an owner).
Flexibility means: beingreactivepro-activesocial
Schmid, CogSysI-11, MAS – p. 9
Demands and Examples
Performing a useful activity on behalf of humans ororganizations (cleaning roboter)
Coexist/interact with humans (cleaning roboter)
Be aware of social rules and norms (transportationrobot)
Coordinate activities (team of cleaning robots)
Cooperate or compete (RoboCup)
Entertain or educate people (games, tutor systems)
Schmid, CogSysI-11, MAS – p. 10
Acting Reactivly
If the environment of a program is static (known inadvance), the program cannot fail (Compile-Time,Runtime)
In the real world, changes occur, information isincomplete (dynamic system).
A reactive system continuously interacts with itsenvironment and reacts in time to changes
Example: Java-Listener, BUT: here reactions do NOTtake into account the current state of the environment,they are determined in advance
Reactive systems can be modelled relativestraight-forward: e.g. as stimulus-response rules
Schmid, CogSysI-11, MAS – p. 11
Acting Pro-activly
means: generate goals autonomously, try to reachgoals
not only event-driven behavior but: act on one’s owninitiative
Schmid, CogSysI-11, MAS – p. 12
Acting Socially
Real world is a multi-agent environment
When trying to reach goals, others must be taken intoaccount
Some goals can only be reached through cooperation
In some situations exit conflicts and competition (e.g.internet auctions)
Social skills of agents: ability to model goals of otheragents when trying to reach one’s own (local) goals,ability to interact (i.e. cooperate and coordinate)
Schmid, CogSysI-11, MAS – p. 13
Further Features
Mobility: ability to move in a computer net or in anotherenvironment
Adaptivity/learning: Improving performance over time
Rationality: do not act in a way which hinders to fulfillone’s goals
Schmid, CogSysI-11, MAS – p. 14
Example: Tileworld
T
T
T
T
T
T
H H
H
Dynamic enviroment: holes appear/disappear
Agent must recognize changes and modify its behavior
left: push up to H; middle: H disappears; right: better go to H to the right
Schmid, CogSysI-11, MAS – p. 15
Agents as Intentional Systems
Endowing agents with “mental” states: beliefs, desires,wishes, hopes
Folk psychology: attributing attitudes for predicting andexplaining other peoples behavior
Intentional systems (Dennett):First order: having beliefs, desires, etc.Second order: having beliefs and desires aboutbeliefs and desires of its own and others
Compare to physical systems: for predicting that astone will fall from my hand I do nat attribute beliefs anddesires but mass or weight
Schmid, CogSysI-11, MAS – p. 16
Abstract Architecture
Environment: finite set of discrete states E = e, e′, . . .assumption: continuous env can be modelled by adiscrete env to any degree of accuracy
Repertoire of possible actions of an agent:Ac = α, α′, . . .
Interaction of agent and environment: run r, as asequence of interleaved environment states and actions
R: set of all possible finite sequences over E and Ac
RAc: subset of R ending with an action
RE: subset of R ending with an environment state
Schmid, CogSysI-11, MAS – p. 17
Abstract Architecture
State transformer function: τ : RAc → P (E)
Termination: τ(r) = ∅
Environment Env = (E, e0, τ)
Agent: Ag : RE → Ac
Set of runs of an agent in an environment R(Ag,Env)
Behavioral equivalence: R(Ag1, Env) = R(Ag2, Env)
Schmid, CogSysI-11, MAS – p. 18
Purely Reactive Agents
No reference to their history, next state is onlydependent on the current state
Ag : E → Ac
Example: thermostate
Schmid, CogSysI-11, MAS – p. 19
Perception
see : E → Per
action : Per∗ → Ac
Agent Ag = (see, action)
Equivalence relation over environment states: e ∼ e′ ifsee(e) = see(e′)
If | ∼ | = |E|, the agent is omniscientIf | ∼ | = 1, the agent has no perceptual ability
Schmid, CogSysI-11, MAS – p. 20
Agents with State
Internal states I
action : I → Ac
next : I × Per → I
State-based agents as defined here are not morepowerful than agents as defined above.
Identical expressive power: Every state-based agentcan be transformed into a standard agent that isbehaviorally equivalent.
Schmid, CogSysI-11, MAS – p. 21
Utility Functions
Telling an agent what to do without telling it how to do it
Indirectly via some performance measure
Associate utility with states of environment, preferactions leading to states with higher utilities
Utility can be defined over states or over runsu : E → R or u : R → R
Schmid, CogSysI-11, MAS – p. 22
Task Environments
Ψ : R → 0, 1is 1 (true) if a run satisfies some specification and 0(false) otherwise
Task environment: 〈Env, Ψ〉
specifies the properties of the environment andthe criteria by which an agent will be judged to havesucceeded in its task
Definition of successpessimistic: ∀r ∈ R(Ag,Env) it has to hold Ψ(r)
optimistic ∃r ∈ R(Ag,Env) where it holds Ψ(r)
Two kinds of tasks:Achievement: relation to planningMaintainance
Schmid, CogSysI-11, MAS – p. 23
Deductive Reasoning Agents
The usual problems of knowledge engineeringTransduction problem: Translating world in adequatesymbolic descriptionRepresentation problem: providing a representationsuch that agents can reason with it for the results tobe in time and useful
Agents as theorem proversLogic-based agents: “deliberate agents”
Specialized languages, e.g. MetateM, based ontemporal logic
Schmid, CogSysI-11, MAS – p. 24
Deliberate Agents
D as a set of logical formulae, internal state of an agentis δ ∈ D
ρ as set of deduction rules
δ →ρ ϕ: formula ϕ can be proved from database δ usingdeduction rules ρ
Goal: Deriving a formula Do(α) either as best action oras action which is not explicitly forbidden
function action(δ : D)
for each α ∈ Ac doif δ →ρ Do(α) then return α
for each α ∈ Ac doif δ 6→ρ¬Do(α) then return α
return nullSchmid, CogSysI-11, MAS – p. 25
Vacuum World
0
1
2
0 1 2
Predicates:In(x,y)Dirt(x,y)Facing(d)(d ∈ north, east, south,west)
Navigation:In(0,0) ∧ Facing(north) ∧ ¬Dirt(0,0) → Do(forward)
Cleaning:In(x,y) ∧ Dirt(x,y) → Do(suck)
Schmid, CogSysI-11, MAS – p. 26
Concurrent MetateM
Michael Fisher, 1994
Language for direct execution of logical formulae(exacutable temporal logic)
Near the “ideal” of agents as deductive theorem provers
Concurrently executing agents, communication viaasynchronous broadcast message passing
Components of an agentInterface: defines interaction with other agents withid, set of environment propositions (acceptedmessages), set of component properties (messagesthe agent can send)stack(pop, push)[popped, full]
Computational engine (executable temporal logic)
Schmid, CogSysI-11, MAS – p. 27
Executable Temporal Logic
Agent specification as set of program rules of the formantecedent about past => consequent aboutpresent and future
Declarative past and imperative future paradigm
Schmid, CogSysI-11, MAS – p. 28
Temporal Connectives
Operator Meaning©ϕ ϕ is true ’tomorrow’⊙
ϕ ϕ was true ’yesterday’♦ϕ at some time in the future, ϕ
ϕ always in the future, ϕ
♦ • ϕ at some time in the past, ϕ
• ϕ always in the past, ϕ
ϕUΨ ϕ will be true until Ψ
ϕSΨ ϕ has been true since Ψ
ϕWΨ ϕ is true unless Ψ
ϕZΨ ϕ is true since Ψ
start nullary operator, true only at the beginning
Schmid, CogSysI-11, MAS – p. 29
Example
rp(ask1,ask2)[give1,give2]:⊙
ask1 ⇒ ♦give1;⊙
ask2 ⇒ ♦give2;
start → ¬(give1 ∧ give2).
rc1(give1)[ask1]:
start → ask1;⊙
ask1 → ask1.
rc2(ask1,give2)[ask2]:⊙
(ask1 ∧¬ ask2) → ask2.
Schmid, CogSysI-11, MAS – p. 30
Example cont.
Example Run:Time Agent
rp rc1 rc20 ask11 ask1 ask1 ask22 ask1,ask2,give1 ask13 ask1, give2 ask1, give1 ask24 ask1, ask2, give1 ask1 give 2. . .
Schmid, CogSysI-11, MAS – p. 31
Practical Reasoning Agents
What we want to achieve: deliberation
How to achieve a state: means-end-analysis
Schmid, CogSysI-11, MAS – p. 32
The Procedural Reasoning System
Georgeff and Lansky
Belief-desire-intention architecture (BDI)
Intentions
PlansBeliefs
Desires
Interpreter
data input
action output Schmid, CogSysI-11, MAS – p. 33
Rational Actions
Reasoning and planning agents:calculative/computational rationality
First step: delibaration – what goal should be achievedforming an intention
Second step: Means-End-Analysis – how should thegoal be achieved
Intention: agent commits itself to a task, inconstistentintentions can be blocked by an active intention
Agents belief that their intentions can be fulfilled and donot belief that they cannot fulfill their intentions(rationality)
Model: Belief, desire, intention Semantics (BDI)
In the following: Focus on multi-agent interactions (games)Schmid, CogSysI-11, MAS – p. 34
Rational Actions in MAS
Assume that environment can be influenced by bothagents
Utility function u(ω) with ω represented as action tuple:
ui(D,D) = 1; ui(D,C) = 1; ui(C,D) = 4; ui(C,C) = 4
uj(D,D) = 1; uj(D,C) = 4; uj(C,D) = 1; uj(C,C) = 4
Preference of agent i:
(C,C) ≥i (C,D) ≥i (D,C) ≥i (D,D)
C is the rational decision for i, because it prefers allstates where it selects C
This is the basic model of game theory.
Representation in payoff matrices Schmid, CogSysI-11, MAS – p. 35
Payoff Matrix
iCoopearate Defect
j Cooperate 4 14 4
Defect 4 11 1
Schmid, CogSysI-11, MAS – p. 36
Dominant Strategies
Strategy: Decision function by which an agent selectsan action
A strategy s1 dominates another strategy s2 if an agentprefers each possible outcome of strategy s1 over eachpossible outcome of strategy s2 (with respect to thepossible actions of the other agents)
is1 s2
j s1’ 4 14 4
s2’ 4 11 1
outcome of s1 ≥ outcome of s2 for s′1
and for s′2
Schmid, CogSysI-11, MAS – p. 37
Dominance and Optimality
A rational agent will never select a strategy which isdominated by another strategy (such strategies can beexcluded when actions are selected)
Unfortunatly, there exists not always a distinct dominantstrategy
Pareto-optimality: an outcome is Pareto-optimal, if thereis no other outcome all agents would prefer
Schmid, CogSysI-11, MAS – p. 38
Nash-Equilibrium
Two strategies s1 and s2 are in a Nash-equilibirum if:Under assumption that agent i plays s1, s2 is therational choice of agent j
Under assumption that agent j plays s2, s1 is therational choice of agent i
If we have two strategies in Nash-equilibrium, no agenthas an incentive to change its strategy
Obtaining a Nash-equilibrium is desirable, because it isan effort to switch strategies and strategy switchesmight danger stability of a system
Schmid, CogSysI-11, MAS – p. 39
Analysis of game theoretic scenarios
Nash-equilibrium is an important tool for analyzinggame theoretic scenarios
If each player has a dominant strategy, the combinationof those strategies is called dominant strategyequilibrium
Solution of a game: the rational strategy of each agent
Each dominant strategy equilibrium is a Nashequilibrium
Nash’s theorem: there are equilibrium strategies even ifthere there is no dominant strategy
Nash equilibrium is a necessary condition for an optimalsolution (but is it also sufficient?)
Schmid, CogSysI-11, MAS – p. 40
Coordination Games
Acme, a video game hardware manufacturer, mustdecide whether its next game machine will use CD orDVD
Best, a video software producer, must decide whetherto produce its next game on CD or DVD
AcmeDVD CD
Best DVD 9 -49 -1
CD -3 5-1 5
Schmid, CogSysI-11, MAS – p. 41
Example cont.
No dominant strategy
Two Nash equilibria: (CD, CD) and (DVD, DVD)if one player moves to a different strategy unilaterally,he will be worse off
One Pareto-optimal solution: (DVD, DVD)
If payoff for CD and DVD would be equal, there wouldbe two Pareto-optimal solutions
→ guess or communicate→ coordination game
Schmid, CogSysI-11, MAS – p. 42
Zero-sum games
If preference orders of two agents are diametral, wehave a stronly competitive scenario:
ω ≥i ω′ → ω′ ≥j ω
An interaction is called zero-sum interaction, if
ui(ω) + uj(ω) = 0 for all ω ∈ Ω
All zero-sum interactions are competitive!
Schmid, CogSysI-11, MAS – p. 43
Prisoner’s Dilemma
Invented by Albert W. Tucker, further studied e.g. byRobert Axelrod
Two people are accused of complicity in a criminal act
They are in two different prison cells and cannotcommunicate
The attorney guarantees:If one confesses the crime and the other not, the firstwill be free, the other goes 5 years to prisonIf both confess, both go for 3 years to prison
Both know that they go 2 years to prison if none of themconfesses
Schmid, CogSysI-11, MAS – p. 44
Prisoner’s Dilemma
iconfess deny
j confess 3 53 0
deny 0 25 2
Global utility is maximal if both cooperate (deny)
But: for each single agent the rational choice is not tocooperate but to testify
Is cooperation feasible in a society of rational, egoisticalagents?
Schmid, CogSysI-11, MAS – p. 45
Generalized Form of PDicooperate defect
j cooperate win win muchwin lose much
defect lose much losewin much lose
T : Temptation to defect, R: Reward for mutual cooperation,
P : Punishment for mutual defection, S: Sucker’s payoff
For PD scenarios it must hold: T > R > P > S
For iterated versions, in addition: 2R > T + S
If that condition does not hold, then full cooperation is not
necessarily Pareto optimal, as the players are collectively
better off by having each player alternate between cooperate
and defect. (Douglas Hofstadter)Schmid, CogSysI-11, MAS – p. 46
Iterative PD
Repeated playing, memory of earlier encounters
Studied in social psychology, competition of computedstrategies
greedy strategies tend to do very poorly in the long run,altruistic strategies do better
Best deterministic strategy: “Tit for Tat”developped by Anatol Rapoport: the simplest of anyprogram entered, containing only four lines of BASIC,won the contest!
cooperate on the first iteration of the game; after that,the player does what his opponent did on the previousmove
Schmid, CogSysI-11, MAS – p. 47
Iterative PD cont.
Best if mis-communication is introduced: “Tit for Tat withforgiveness”When the opponent defects, on the next move, theplayer sometimes cooperates anyway, with a smallprobability (around 1%-5%). This allows for occasionalrecovery from getting trapped in a cycle of defections.
Schmid, CogSysI-11, MAS – p. 48
Analyzing Top Scoring Startegies
Nice, retaliating, forgiving, non-envious
Utopian sounding: Nice guys can finish first!
e.g.: arms race
Schmid, CogSysI-11, MAS – p. 49
Dilemma of the Commons
“Allmende Klemme” (William Forster Lloyd, 1833;Garrett Hardin, 1968)
social trap that involve a conflict over resourcesbetween individual interests and the common good
free access and unrestricted demand for a finiteresource ultimately dooms the resource throughover-exploitation!
This occurs because the benefits of exploitation accrueto individuals, each of which is motivated to maximisehis or her own use of the resource, while the costs ofexploitation are distributed between all those to whomthe resource is available
Solutions?
Schmid, CogSysI-11, MAS – p. 50
Multiagent Communication
Competitive: mechanisms for collective decision makingVotingAuctionNegotiationArgumentation
Cooperative: communication for distributed problemsolving
Speech actsAgent Communication LanguagesOntologies
Schmid, CogSysI-11, MAS – p. 51
Collective Decision MechanismsDesign of a protocol
Guaranteed success: ensuring that eventually an agreement can bereached
Maximizing social welfare: total sum of utilities should be maximal
Pareto efficiency: no other outcome where at least one agent isbetter off and none is worse off
Individual rationality: following the protocols in the best interest ofnegotiation perticipants
Stability: providing all agents with an incentive to behave in aparticular way (e.g. Nash equilibrium)
Simplicity: a participant can easily determine the optimal strategy
Distribution: designed such that there is no ’single point of failure’and to minimize communication between agents
Schmid, CogSysI-11, MAS – p. 52
Auctions
Online auctions are very popular
simple interaction scenarios → easy to automate
good choice as a simple way for agents to reachagreements, allocating goods, tasks, resources
Auctioneer agent, bidder agents, a good
private value vs. public/common value of goods,
correlated value: value for private factors as well asother agents valuation
Schmid, CogSysI-11, MAS – p. 53
Dimensions of Auction Protocolls
Winner determination: first price (highest bid gets goodfor the bidded amount), second price (highest biddergets the good, but for price of 2nd highest bid)
Knowledge of bids: open cry, sealed-bid
Bidding mechanism: one shot, ascending/descendingbids in successive rounds
→ different types of auctions
Schmid, CogSysI-11, MAS – p. 54
English Auctions
“Mother of auctions” (Sothebys)
first-price, open cry, ascending (starting with areservation price)
Dominant strategy: successively bid a small amoutmore than the current highest bid until price reachescurrent valuation, then withdraw
Winner’s curse: uncertainty about the true value (e.g.land speculation), winner might have overvalued thegood
Schmid, CogSysI-11, MAS – p. 55
Dutch Auctions
open-cry, descending
auctioneer starts with an artificially high value
decreasing value, until someone makes an offer
no dominant strategy
also susceptible to winner’s curse
Schmid, CogSysI-11, MAS – p. 56
First-Price Sealed-Bid Auctions
first-price, sealed bid, one-shot
simplest form of auction
difference between second-highest and highest bid iswasted money
best strategy: bid less than true valuation, how muchless depends on the other agents, no general solution
Schmid, CogSysI-11, MAS – p. 57
Vickery Auctions
second-price, sealed bid, one-shot
dominant strategy: bid true valuation
Because truth telling is the dominant strategy, this formis discussed much in multiagent literature
BUT: counterintuitive for human bidders
Possibility of antisocial behavior: own valuation 90 $,guess that another agent will pay 100 $, therefore bid99 $ such that opponent needs to pay more thannecessary
Commercial situations: one company cannot competedirectly but foces other company into bankruptcy
Schmid, CogSysI-11, MAS – p. 58
Issues for Auctions
Expected revenue: Strategies for the auctioneer tomaximize his revenue (for risk-neutral bidders, expectedrevenue is provably identical in all auction types)
Lies and collusions: coalition between agents (bid smallamounts, share win afterwards), place bogus bidders, ...
Counterspeculation: costs time and money and is risky(compare with meta-level reasoning)
Schmid, CogSysI-11, MAS – p. 59
Negotiaton
Auctions are only concerned with allocation of goods
When agents must reach agreements on matters ofmutual interest, richer techniques are required
Negotiation techniques for artificial agents(Rosenschein and Zlotkin, 1994)
Schmid, CogSysI-11, MAS – p. 60
Object vs. Agent CommunicationObject o2 invokes a public method m1 of object o1
passing argument arg→ o2 communicates arg to o1
→ BUT: the decision to execute m1 lies only with o2
An autonomous agent has control over its state and itsbehaviorThere is no guarantee that another agent reallyperforms an action
An agent cannot force another agent to perform someaction or to change its internal state
An agent can try to influence another agent bycommunication
Communication can change the internal state (belief,desire, intention) of another agent
Communication as special case of action: speech actsSchmid, CogSysI-11, MAS – p. 61