ACI©

Data Solutions for Irregular Warfare Simulations

Deborah Duong, David Makovoz, and Hyam Singer

Augustine Consulting and Impact Computing Corporation

ACI©

The Biggest Issue in Agent Based Modeling is DATA

• Agent Based Models (ABM) capture social processes through experimentation– Agents are entities that perceive their environment and act based on motivations– Patterns in agent behaviors, social institutions, arise from the motivated interactions of

individual agents, and thus from first principles– Social Institutions are emergent, endogenously generated, dynamic structures – Institutions feedback to agent motivations, creating new social structures dynamically– Analysts examine the effects of DoD interventions on a population’s motivations and social

institutions by looking for changes in the dynamic structures generated by the ABM• Social Systems, and good Agent-Based Models of Social Systems, are characterized

by Path Dependencies– As the symbolic species, humans have randomness built into their systems

• Anything can mean anything: symbols are arbitrary– Although there are regularities, the real world is but an instance of many possible worlds

• Problem: If the structures of interest that describe agent societies are dynamic and generated from first principles, and the real world data is a single possible path, how do we emulate real world data dynamically and from motivation?– Real world data is a “snap shot”– To generate the data in the same way that agent based models represent it, with motivation,

is to explain the data– We can not use the agent based model to explore consequences if it can not emulate the

baseline dynamic structures. It needs to generate “this step” before it can generate the next

ACI©

There is a Divide between Theory and Data in ABMs

• Two classes of agent models emphasize one or the other aspect of modeling but not both– Theory-centric Models

• Elegant models that capture the essential processes of a theory, with little, often random initialization data

• Lots of data is generated endogenously• Relations in accordance with theory, not actual values, are important• Example: Schelling's Model of Segregation

– Data-centric Models• Large models that ingest lots of real world data that describe a scenario well• When they try to capture relations that exist, they usually don’t do so in a causal manner,

often double counting phenomena• The DoD shows more interest in these because they want to explore particular situations• But this type can’t do a good job at that because theory doesn’t derive the existing state

of the simulation, and so it cant get to the next state either… • ABMs that are both Data-Centric and Theory-Centric are needed

– ABMs need to generate the dynamic structures of existing scenarios – ABMs need to make static data dynamic by explaining it

ACI©

Learning is Essential to the Solution

• We want to generate new Dynamic Structures from first principles– We are trying to explore *how* individual agent behavior may bring the agent

society to a particular state. We don’t know how to begin with• We cant tell them how to adjust their behavior in particular circumstances, they must do

it based on motivation • That is learning. It is how cognitive agents and reactive agents differ

– Agents may learn new behaviors based motivations, creating various dynamic structures

– The trick is to get them to emulate the feedback between private incentive and social institution that constitute the dynamic structures of the particular scenario of interest

• When agents adapt to each other based on motivations to form a social system, its called “coevolution”

• Coevolution can also be used to help agents adjust towards data– Static initialization data– Data arriving from other simulations in a composition of simulations

ACI©

Bringing Data and Theory together through Co-evolutionary Feedback

• Co-evolution: the process of mutual adaptation– Competitive Coevolution

• Biological: Predator and prey– The Cheetah and the Gazelle make each other fast

• Social: enemy adaptation to our technologies in warfare– Cooperative Coevolution

• Biological: ecological systems with roles for species– Gaia

• Social: The division of labor into different trade roles– The Formation of Social Systems

• Properties of Coevolutionary Systems– “Spandrels”

• How structures come to be can be completely different than why they stay– Biological: Wings used to be used for “scuttling”– Social: The little fish with the bones are given to Eskimo women and the big fish to men. And they are the only

source of Vitamin D/Calcium for pregnant women. They don’t know that• Coevolutionary Systems are like a “House of Mirrors”

– Biological: If you seed a predator into an ecosystem, the prey will adapt, and then a similar predator will arise within the ecosystem

– Social: Reflexivity. Self fulfilling prophecy. Kilcullen: “Perception IS Reality.” Japan developing its own form of capitalism.

• Similarly, we can seed coevolutionary ABMs with data…– …. Establishing the dynamic structures we wish to explain

ACI©

Coevolutionary Programs that use Feedback to Adapt to Data

• Nexus Network Learner (OSD/SAC)– ABM of Social Role Network Development – Agents learn from and adapt to each other– Seeded by Data: Gradually pushes dynamic structures in simulation

towards the setup of the data, thus explaining it• Indra (Army/I2WD, Army/INSCOM)

– Named after the Indian God’s net of diamonds, each of which reflect all other diamonds (a house of mirrors)

– Statistical Natural Language Processing program– Inputs a large corpus of freetext (news reports, email messages, etc.)– Extracts an Ontology of Roles and Role-relations– Uses Feedback: Nouns classified by verbs they use, and verbs

classified by nouns they use– Seeded by Data: Gradually pushes development of ontologies towards

an existing ontology, filling in the gaps

ACI©

What is Nexus Network Learner ?

• One of the two Nexus Cognitive Agent models that Debbie Duong wrote at the OSD/CAPE/Simulation Analysis Center

• A Simulation of Social Role Networks in which Agents learn – To choose role partners to perform transactions with

• Choice based on signs, social markers and communications on past transaction behaviors

– Transaction behaviors and signs• Choice based on signs and social markers

– Based on Cultural Values • Social markers, roles, transaction behaviors, signs, role-

based communications and cultural values are all input to the program– Population data determines the initial population tendencies– Utility and motivation determines how they change

ACI©

What does Nexus Network Learner Simulate?

• Social Structure– Its Formation, Maintenance and Dissolution– Of Social Groups, from Terrorist Networks to Larger Societies

• Social Theory: Classical Interpretive Social Science– Roles and Signs from Symbolic Interactionism– Institutional Economics

• Macro-Level Social Effects of Micro-Level Actions– All actions take place at the individual, tactical level– All agents learn to react based on goals– These reactions form vicious and virtuous cycles that make patterns at

the macro level, called social institutions– The existence of institutions affects motivations and actions again

• What are the second and third order effects, based on motivations and changes of motivations– Agents don’t just react to us, they react to other agent’s reactions

• What may happen to institutions after we leave

ACI©

How is Social Structure Simulated ?

• Social Structures form as Cognition Separates Intention from Action– Agents Plan to act through learned strategies – Social Plans must correspond to work

• A plan to accept a bribe will not take place without someone offering a bribe

• A plan to reject employers who steal paychecks will not work if all employers steal

• Larger forces, such as unemployment and international intervention, matter

• The environment may not give them opportunity to act, so new plans are selected for– Transactions may be resource constrained, simulating flow of

funds – Resource constraints may be turned off with exogenous

injection of funds

ACI©

How Does Nexus Network Learner Work?

• Artificial Intelligence Technologies represent Cognition– Rule Based

• An ontology of roles with crisp rules for roles – represents general social structures, that can be used in many scenarios– defines utility of transactions

– Machine Learning• Bayesian networks initialize social markers , signs/transaction

behaviors, and role choice behaviors• The Bayesian Optimization Algorithm (BOA) changes those

behaviors based on the utility of transactions– BOA can be seeded with initialization data and injected data– A form of Evolutionary Computation using reinforcement Learning

optimizes (satisfices) utility– As conditionals change, the equilibrium point moves (in accord with the

New Institutional Economics)

ACI©

What Happens in Nexus Network Learner?

• Individual Agents Choose Network Partners– Ontology tells who may choose and how many

• Example: an “Employer” may choose an average of 5 employees with a standard deviation of three

– Bayesian network tells how the choices are ranked– Passive role may have an option to reject offer

• Example: an “Employee” may reject an employer because a role relation has told her he steals paychecks

– Ontology may include a chance occurrence of natural attrition• Individual Agents engage in transactions

– Account distributions send funds through networks according to rules in ontology and transaction behaviors in Bayesian networks

– Probability of observing, reporting, and knowing about behaviors are role-based

– Agents may go to jail, and not be allowed to participate in transactions for a time

• Every N cycles, they judge their learned strategies by utility based on transactions that their valued role partners engaged in– Ontology determines culturally valued individuals and transactions– After testing all strategies agents recombine them

ACI©

Performing Tests with Nexus Network Learner

• A wide variety of tests relevant to IW may be performed• For example, new network formations and behaviors may be tested based on

many different things…– The effect of different utility functions

• For example, make agents care only for self rather than larger social network– The effect of different penalties

• For example, a penalty attribute that encodes different jail terms or different chances of getting caught

– The effect of different exogenous resources• For example, test resource rents or foreign aid

– The effect of different abilities to observe• For example, the effect of a media agent

– The effect of removing different agents• For example, measure how long it takes to replace a terrorist leader

• Monte Carlo methods reveal if new structures are the result of different CONOPS– Bayesian Networks make Nexus Stochastic

ACI©

How Nexus Agents Learn

• As each agent learns, all the agents coevolve, making them very adaptive– Every agent has its own private learning algorithm– Their behaviors affect the larger social structure and the larger social

structure affects their behaviors• Micro-Macro Integration is modeled

– They can adapt to data from other simulations and to initial country data as well

• The learning algorithm in each agent makes the adaptation to data flexible– BOA (Bayesian Optimization Algorithm) can start learning from initial data

• In the calibration phase, agents to adapt to initial data, so that they generate it though their perceptions and motivations

• Thus they “explain” the data, going from correlation to cause– This greater ability to ingest data also allows them to meld with other

simulations in a composition• Together, composed simulations create a coherent picture of the social

environment• Conflicts are resolved through mutual adaptation

ACI©

How Data Ingestion Works

• Every Agent of a class starts out with the same variety of the behavior propensities of his class– For Example, if an agent is a Elder Female of the Mongo tribe, she may

have a 50 percent propensity to steal• Every agent its own “population” of behavior strategies in its head,

initialized to correct proportions of its class• Agents use a Bayesian Optimization Algorithm to modify those

behaviors based on utility• Agents of one class interact with others based on the expected

probabilities, putting selective pressure on the agents to have those behaviors in those proportions

• Agents find niches of corresponding behaviors based on utility which come to match those proportions, thus explaining the proportions within the population classes

ACI©

Use Case: Modeling Corruption• Corruption in African Society is said to be the result of conflicting social networks

– Kinship Network vs. Bureaucratic Network• Kinship Network has patron-client roles and many obligations• Bureaucratic Network has merit-based impersonal roles• Rules on how choose network relations (based on merit?) and how to distribute

resources (based on kin obligations?) differ– Nexus Network Learner could easily model Kinship, Bureaucratic, and Trade

Networks• Nexus Network Learner could easily represent the roles and role relations of both

networks– Includes 65 roles, including roles and role relations important to matrilocal and

patrilocal ethnicities• Nexus Network Learner could easily represent the corrupt behaviors which change

distribution of resources in the networks– There are said to be eight basic types of corruption, that Nexus models with basic

stealing and bribing behaviors, that occur in different sectors, and during both network choice and transactions

– Nexus models the moving of funds from Bureaucratic to Kinship networks based on behavior traits

ACI©

Nexus Network Learner Summary

• Nexus Network Learner is a robust, general, flexible tool for modeling Social Role Networks, resource flows, and transactions through those networks– Usable in a wide variety of IW scenarios

• Nexus Network Learner can get to the crux of the issues in IW because it models agent motivations and interpretations

• Nexus Network Learner easily ingests real world data because it can adapt to it

• Nexus Network Learner works well in a composition of other simulations because it can adapt to them

• Nexus Network Learner offers Monte Carlo exploration of the effects of DIMEFIL actions on Social Structure

ACI©

Indra

– Statistical Natural Language Processing program that can “read” and interpret text, and make the information in that text available to IW simulations

– Inputs a large corpus of freetext (news reports, email messages, etc.)

– Extracts an Ontology of Roles and Role-relations– Uses Feedback: Nouns classified by verbs they use,

and verbs classified by nouns they use– Seeded by Data: Gradually pushes development of

ontologies towards an existing ontology (such as needed by a simulation), filling in the gaps

ACI©

What is unique about Indra ?

• Uses Mutual Information to choose parse, assign word sense, and form ontologies based on context

• Iterative feedback finds global consensus on meaning, for accurate role discovery

• Flexible emergent ontologies form, combining data driven with hypothesis driven approaches

• Feedback facilitates data fusion with other modalities• A way to feed higher level information back to lower

level extraction, introducing feedback to data fusion

ACI©

Language is Context Dependent

• Language is deeply context dependent, but natural language programs complete each stage before the next starts in “pipelines”

• Indra uses a feedback loop to let the parse, word sense assignment, and ontological assignments inform each other

• The result is a flexible data driven ontology that can be aligned with other models

ACI©

Making “Sense” of Text • “Word sense” of entities and their actions • Inter-Document Coreference Resolution•Many ways of Naming a Person• Different Persons may have the same

name• Link Normalization •Many ways of referring to a Behavior• Different Behaviors referred to with the

same words

ACI©

General Roles and Role Relationships

• Indra extracts general Role and Role relationships from text

• These Role and Role relationships are arranged in ontological groupings

• Iterative feedback allows different parts of the ontology to influence each other

• Iterative feedback makes system deeply adaptive so outside data can have widespread influence

ACI©

Global Consensus on Sense

• Grouping of entities and links increases the information with each iteration

• With each iteration, the unsupervised scatter-gather finds the “sense” of named entities, finding which individuals they are based on their role

• As information corrects senses of links and entities, and neighbors correct their neighbors, a global consensus on sense forms.

• As links and entities are grouped, an emergent ontology is formed

ACI©

Iterative Feedback introduced in stages

• Stage 1: Upper-lower feedback *Implemented• Larger clusters and smaller clusters influence each other

• Stage 2: Side-to-side feedback *Implemented• Node clusters and link clusters influence each other

• Stage 3: More Upper-lower feedback • Ontology and parse influence each other

• Stage 4: Feedback with external systems • Seed hypotheses from analysts and inference engines have

wide influence

ACI©

Stage 1: Upper –Lower Feedback

• Roles are clustered according to link contexts, and Role relations are clustered according to entity contexts• Two separate ontologies form

• Clusters at higher levels split clusters at lower levels• Essential for word sense (and “entity sense”)• For example, clusters for factories and autotrophs split the

word “plant”• Clustering algorithms are either agglomerative or

divisive: “unsupervised scatter gather” is both• Clusters split and divide until convergence

ACI©

Stage 2 : Side to Side feedback

• Stage 1 was clustering entities based on links and links based on entities

• Stage 2 is clustering entities based on link *clusters* and links based on entity *clusters*

• The separate Role and Role relationship ontologies of stage 1 become intertwined

• Needed for data smoothing and more consensus

ACI©

Stage 3: More upper-lower feedback

• Choose parse based on ontology (parse already influences ontology in feedforward)

• Choose parse based on how common it is for similar words to be attached in that way.

• Example:• Jane ate the salad with a fork

• “with” modifies “ate” because tools such as “forks” and “knives” are typically found to be used to “eat” or “consume”

• Jane ate the salad with croutons• With modifies salad, because things that are “eaten” or “consumed” are

typically foods such as “croutons” or “tomatoes”• Later, instead of using rule based parser, use mutual

information to parse (Yuret), making Indra purely statistical• Can be used with any language

ACI©

Stage 4:Feedback with External Systems

• Purpose of feedback is deep adaptivity, so external data can influence and be easily fused

• Hypothesis Driven AND Data Driven Ontologies • If an analyst groups concepts:

• Collocated paths found• These help develop analyst’s concept• More consonant concepts and paths found• RELATIVELY FEW points of correspondence needed

ACI©

Example Cluster

• p:35805,n:34540.fes // morocco city• p:35805,n:37114.tenerife //spanish city• p:35805,n:37344.zaragoza //spanish city, with football club• p:35805,n:37548.boavista //portugese island, with football club• p:35805,n:38590.maritimo //portugese sports club known for football team• p:43243,n:39997.• p:39997,n:29474.saccoh• p:39997,n:29612.spaho //bosnia small town• p:39997,n:33375.spartak //Moscow football club• p:39997,n:34467.environmentastrit• p:39997,n:34721.haxhi //Albanian football player• p:43243,n:40629.tenerife• p:43243,n:41043.boavista• p:43243,n:42049.maritimo• p:46477,n:44423.bilbao //basque city• p:46477,n:44563.centreleft //football position• p:49912,n:48979.oviedo //spanish city

ACI©

Example Cluster • p:49224,n:50682.tenerife• p:56352,n:53348.• p:53348,n:46799.• p:46799,n:40301.rayo //football club in madrid• p:46799,n:41027.bilbao• p:53348,n:47751.bilbao• p:56352,n:53354.• p:53354,n:47225.shelling• p:56352,n:53766.shelling• p:56352,n:53814.spartak• p:56352,n:54104.youridjourkaeff• p:56352,n:54108.zaragoza• p:56352,n:54460.colo //chile football club• p:56352,n:55076.kickoff //football term• p:65663,n:62554.• p:62554,n:60508.youridjourkaeff• p:83660,n:85323.youridjourkaeff• p:86579,n:84114.• p:84114,n:81134.• p:81134,n:75091.• p:75091,n:73692.deportivo //spanish football club

ACI©

Ontologies Problematic

• Indra will approximate most likely (highest mutual information) ontology

• BUT, analysts want their own ontologies• Different experts look at same data• Data stored in primitive entities and paths• Indra to make semantic model on the fly tailored

to ontology of who is looking at it• Tailored Ontologies towards ontologies of

particular simulation models

ACI©

Hypothesis Driven AND Data Driven

• Indra can flexibly take in analyst input• Indra can align its ontology to another with

very few points of correspondence• Indra can fill in the gaps• Feedback gives Indra advantage over other

systems that generate ontologies:• Global consensus• Ability to adapt to any amount of user input

ACI©

Summary

• Complex Adaptive Systems techniques of Feedback and Coevolution can make data available to simulations

• Indra works on the data side to make data in free text, such as we find on the internet, available

• Nexus Network Learner works on the simulation side to make sure data can be ingested and used by a simulation

• Together, Indra and Nexus Network Learner have the potential to form a live connection to dynamic data, so that events may be analyzed as they occur