Workshop on High Performance, Fault-Adaptive Large Scale Real-Time Systems

Vanderbilt University

The SRTA Agent Architecture as a Basis for Building Soft Real-Time

Multi-Agent Systems

Victor R. LesserComputer Science Department

University of Massachusetts, Amherst

November 15, 2002

Acknowledgements

• Byran Horling• Dr. Regis Vincent (SRI)• Dr. Tom Wagner (Honeywell

Research)

• URL:http://mas.cs.umass.edu/~bhorling/papers/02-14.ps.gz

Outline• Background/Motivation

• EW Challenge Problem • Approach to Soft Real-Time• Approach to Building MAS

• SRTA Agent Architecture• Experimental Evaluation• Summary

Long Term Motivation• Development of domain-independent

techniques (toolkits) for coordinating the soft real-time activities of teams of cooperative agents?

• Ease the construction of complex, multi-agent applications that operate in a coherent manner

• Avoid reproducing for each application the complex reasoning involved in soft real-time control and in coordinating the activities of agents

DARPA EW Challenge Problem: Distributed Sensor Network

•Small 2D Doppler radar units

–Scan one of three 120 sectors at a time

Commodity Processor associated with each radar

•Communicate short messages using radio

•Triangulate radars to do tracking

Approach to Soft Real-Time: Design-to-Time

• “Given a time bound, dynamically construct and execute a problem-solving procedure which will (probably) produce a reasonable answer with (approximately) the time available.” (D’Ambrosio)

• Involves elements of planning (deciding what to do) and scheduling (deciding when to perform particular actions).

TÆMS: A Domain Independent Framework for Modeling Agent Activities

• The top-level goals/objectives/abstract-tasks that an agent intends to achieve

• One or more of the possible ways that they could be achieved• abstraction hierarchy

(HTN) whose leaves are basic action instantiations, called methods

• A precise, quantitative definition of the performance (Qaf’s) • solution quality, cost, time

and resource usage.

Recommend a High-End PC SystemMake Decision

MoneyResource

Build ProductObjectsOutcomesNum Prod 1-4Num Prod 5-8Num Prod 9-12Num Prod ...

Get Basic ProductInformationQuery & ExtractVender mQuery & ExtractPossible Maker n

Gather ReviewsSearch & ProcessZDnet Reviews Search & ProcessPC World

Query & ProcessConsumers Reports

q_sum_all()q_sum() q_sum()

q_seq_last()

q (10% 0)(90% 10)c (100% 0)d (10% 2min)(10% 2.5min)(80% 3min)q (20% 0)(80% 8)c (100% 0)d (50% 1min)(50% 2min)Query & ExtractPC Connection Query& ExtractNECX q (25% 0)(75% 20)c (100% $2)d (90% 3)(10% 5)

q (15% 0)(75% 10)c (100% 0)d (30% 3min) (30% 4min) (40% 5min)q(..), c(..), d(..)q(..), d(..), c(..)

consumes $2limitsq multiplier (100% 0)c multiplier (x)d multiplier (x)q(..), c(..), d(..)

facilitates & hindersfacilitates & hindersq multiplier (100% +20%)d multiplier (100% +20%)

TaskMethodResource nleTask nleSubtask RelationKey

enables.........q_sum()

Soft Real Time Control -- Different Paths for Achieving Task -- “BUILD PRODUCT

OBJECTS”

• Schedule A - Client has no resource limitations; maximize quality• Query-and-Extract-PC-Connection, Query-and-Extract-PC-Mall, Search-and-Process-

ZDnet, Query-and-Process-Consumers-Report (Expected Q=55.3,C=2, D=11.5)• Schedule B - Client is interested in a free solution

• Query-and-Extract-PC-Connection, Query-and-Extract-PC-Mall, Search-and-Process-Zdnet (Expected Q=33.2,C=0, D=8.4)

• Schedule C - Client request an even trade-off between quality, cost and duration• Query-and-Extract-PC-Connection, Search-and-Process-Zdnet (Expected Q=22.4,C=0,

D=5.6)

• Schedule D - Client wishes to maximize quality while meeting a hard deadline of 7 minutes• Query-and-Extract-PC-Mall, Query-and-Process-Consumers-Report

(Expected Q=25.9,C=2, D=6)

Examples of Schedules Produced by the Design-To-Criteria (DTC) Scheduler

Representing Coordination Patterns Among Agents

MoneyResource

enables

enablesRecommend a High-End PC System

Make Decisionq_sum_all()q_seq_last()Build ProductObjects

Get Basic ProductInformationNon-local Task Gather ReviewsNon-local TaskGet Basic ProductInformation

Query & ExtractVender mQuery & ExtractPossible Maker n q_sum()Query & ExtractPC Connection BuyInformation

Gather ReviewsSearch & ProcessPC Worldq_sum()Search & ProcessZDnet ReviewsQuery & ProcessConsumers Reports

task contractingtask contracting

consumes / limitsconsumes/ limits

Task Agent

InformationAgentInformation Agent

Approach to Decomposing A Problem into Agents

• Sophisticated/Highly Competent Agents• Concurrent goals, goals are time and resource sensitive,

goals have varying utilities• Goals have alternative ways of being solved that produce

differing levels of utility and consume differing amounts of resources

• Not all goals necessarily need to be solved

• (Sub)Goals spread across agents are interdependent• Contention for scarce resources• Contributing towards the solution of a higher-level goal• Hard and soft constraints

• Sufficient computational/communication resources to do “some” reasoning about coordination• Medium granularity domain tasks

What about simpler agents?• Activities of simple,single-threaded agents become the goals of

sophisticated agents with dedicated processing resources • Sophisticated agents do the selection, multiplexing, scheduling,

coordination and distribution of goals• Contrast with O.S. doing the scheduling without context

Sector Manager

Tracking Manager

Tracking Agent

Scanning Agent

Approach to Soft, Real-Time Distributed Coordination/Resource Allocation

• Structured as a distributed optimization problem with a range of “satisficing” solutions

• Adaptable to available time and communication bandwidth

• Responsive to dynamics of environment• Organizationally constrained — range of agents

and issues are limited• Can be done at different levels of abstraction• Does not require all issues to be resolved to be

successful — resource manager agents able to resolve some issues locally

Layered Agent Architecture

• Domain analysis and goal formulation• Organization-level resource allocation

• Agent-level resource allocation• Constraint discovery and satisfaction

• Intra-agent organization and communication

• Environmental access points

Problem Solver/ Negotiation

Soft Real Time Architecture

Java Agent Framework

JAF: Java Agent Architecture• Component-based agent design• Attempt to maximize code reuse.

Execute Control CommunicatePulse Actions Scan Scheduler

Problem Solver

Execute Control Communicate

Resource Modeler Directory ServiceLog SchedulerStateSensor Observe

• Interfaces are hidden by JAF.Interfaces are hidden by JAF.•Radsim/ RF communication/ sensorRadsim/ RF communication/ sensor

SRTA: Soft Real-Time Agent Architecture

• Facilitates creation of multi-resource management agents

• Basis of building complex “virtual” agent organizations

• Allows for abstract negotiation — maps abstract assignment into detailed resource allocations

• Ability to resolve conflicts locally that are not resolved through negotiation

These are key to building soft real-time distributed allocation policies

Soft Real-Time Control Architecture

Resource Modeler

Conflict Resolution Module

Task Merging

Problem solverPeriodic Task Controller

TÆMS Library

Cache Check

DTC-Planner

Partial Order Scheduler

Parallel Execution Module

LearningUpdateCache

CacheHit

Linear Plan

TAEMS-Plan Network/Objective

Goal Description/Objective

Parallel ScheduleScheduleFailure

Results

Update Expectations

Schedule Failure

OtherAgents

Schedule

Resource Uses

MultipleStructures

Negotiation(e.g. SPAM)

Commitments/Decommitments

Schedule failure/Abstract view

• Operates at 50 to 100ms cycle time• Written in JAVA except for Planner in C++

• Uses domain-independent, quantitative representation of agent activities -- TÆMS• Scheduling of multiple activities that have deadlines

and are resource sensitive• Can choose among alternative ways of achieving

activities that trade off decreased utility for lower resource consumption

• Responds to uncertain conditions without the need for complete re-planning/scheduling of activities

Characteristics of Soft Real-Time Control Architecture

Addressing Real Time – Direct

• Direct technologies - making it possible• DTC (Design-To-Criteria) planner

• TÆMS HTN for representing alternative plan options using quantitative information

• Create appropriate plan given time, resource costs and quality constraints • Partial order scheduling creates “loose” schedules which can be

quickly shifted to real-time constraints.• Avoids constant re-planning• Allows parallel execution and resource usage.

• Modeling of some meta-level activities (e.g. negotiation) permit more direct reasoning of time allocation.• We do not model scheduling costs yet.

• Learning component discerns actual execution characteristics so future actions can be better modeled.

Addressing Real Time – Indirect

• Indirect technologies - making it easier• Periodic commitments reduce the need

for re-negotiation.• Scheduled caching reduces the need to

call DTC• Piecemeal addition and removal of tasks

eliminates the need for constant dramatic rescheduling and re-planning.

Scheduling• Partial-order Scheduler uses a “sliding”

mechanism, coupled with a resource modeler, to quickly shift scheduled tasks.• Action start time uncertainty - real time.

• Duration uncertainty.

• Commitments have a “window” of time in which the agent can perform them.• Precise action scheduling is left to the discretion of

the performing agent.

Set-parameters

First task to achieve

enables2 enables3

lock

Track LowTrack MediumTrack High

Send Results

Sensor

Task 1Q_min

TaskQ_max

Deadline: 3000

RF

2 Other tasks to achieve

Enables1

Init Calibrate Send-Message 1

lock1

Q_minEnables4Task2

Q_min

Task3

Negotiate-Tracking Send-tracking-Info

Sensor

Enables4

RF

Reacting to Unexpected Changes

500 1000 1500 2000 2500 3000 3500 4000

InitCalibrate

Send Msg

Set-Parameters Track-Medium Send-Results

Send-Info-TrackingNegotiate-Tracking

500 1000 1500 2000 2500 3000 3500 4000

InitCalibrate

Send Msg

Set-Parameters Track-Medium Send-Results

Send-Info-TrackingNegotiate-Tracking

500 1000 1500 2000 2500 3000 3500 4000

InitCalibrate

Send Msg

Set-Parameters

Negotiate-Tracking

time

time

timeSend-Info-Tracking

Send-Results

•Analogous reactions also take place within the periodic task controller• Slot-based scheduler used to facilitate repetitive actions

Track-Medium

Meta-Level Costing

• Typical scheduling reasons about primitive actions.• This only accounts for some percentage of the

agent’s time.• So called meta-level activities (e.g. negotiation,

scheduling, planning) use significant resources but are usually not accounted for directly.

• Without accountability, these activities can interfere with the actions and commitments currently scheduled over.

Meta-Level Costing (cont’d)• To completely reason about all the agent’s

actions, we must: • Directly and Indirectly incorporate the activities in

plans. • Derive expected costs for these activities.• Use this information when generating schedules.

• We are currently using a representation of negotiation activities in some our task structures• Future goal is to more directly account for

activities like planning and scheduling.

Reducing Scheduling Overhead

• Activity parallelism learning• Anticipation of converting linear plan to

parallel plan if resources are available

• Schedule caching

• Adjustable time granularity• Responsiveness vs. meta-level overhead

Activity Parallelism Learning

• The plan is first scheduled…• The schedule is analyzed for parallel actions• These are used to form sets of “hints,”

associated with the current resource context• A hint is actually just a mutual facilitates

relationship• These hints are later applied to the task

structure before planning• The facilitates tells DTC that if A is run

first, B will have a duration of zero (or vice versa)

• This technique required no changes to DTC

• This can result in better plan selection.

q_min

Plan B

A B C1 C2

q_max

AB

C1

E

Deadline

A B C1

Deadline

DTC Schedule

Actual Schedule

q_min

Plan B

A B C1 C2

q_maxE

AB

Deadline

A B

Deadline

DTC Schedule

Actual ScheduleC2

C2

F F

Original Plan

Plan With Hints

Schedule Caching

• Agents in repetitive environments must frequently address goals which have previously been seen.• Ex: In sensor environment, “Scan-Sector” or “Perform-Track-

Measurement”• Results from prior planning can be reused

• A key is generated for each task structure• Incorporates method names and expectations, interrelationships,

normalized deadlines, etc.• Works correctly with activity parallelism hints

• If results exist from a task structure with the same key, that plan is used instead of calling DTC

• DTC is an external C++ binary, requiring file reads and writes, so savings are significant

Adjustable Time Granularity• The millisecond time line is divided into coarser increments.

• Does not significantly degrade the agent’s ability to meet “wall clock” time deadlines• (assuming success within a reasonable grain size)

• It does decrease the number of rescheduling events which are needed

• Since the agent was already operating effectively at a coarse timeline, this technique has no new drawbacks.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5

Real Time

Perceived Time

Real Time

Perceived Time

Deadlines

Actions

Deadlines

Actions

Effects of Schedule Caching

• Periodic tasks and methods with deadlines are more achievable

• As a result, track updates increase

Effects of Changing Granularity

Rescheduling attempts decrease, reducing overheadMethods with deadlines and periodic tasks are satisfied more often30-40 seems ideal; above that, the coarse granularity decreases agent

responsiveness

Summary

• SRTA architecture is a powerful tool for building soft real time agent organizations

• Sophisticated soft real time agent control is practical by exploiting a variety of mechanisms• to speed up the planning, scheduling and

rescheduling cycles

Future SRTA Work• Continue to speed up architecture

• Simple Planner for time-critical situations

• Additional work on conflict-resolution strategies

• Meta-level control component to balance control and coordination costs and domain problem solving