Post on 30-Apr-2020
MIT Lincoln Laboratory2007 CBIS --1
Jerome J. Braun 12/1/2006
Multi-Purpose Machine-Intelligence-based Information Fusion (FLASH)
Dr. Jerome J. Braun,Mr. Yan Glina,
Ms. Laura J. Brattain, and Dr. Timothy J. Dasey
Lincoln Laboratory, Massachusetts Institute of Technology
2007 Chemical Biological Information Systems (CBIS) Conference
10 January 2007
This material is based upon work supported by the National Science Foundation under Grant No. 0329901. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
MIT Lincoln Laboratory2007 CBIS --2
Jerome J. Braun 12/1/2006
Overview
• Problem– Fusion, recognition and reasoning with uncertain and disparate data (from sensors
and other information sources) for demanding decision-support applications, including complex phenomenology and absence of reliable models
• Relevance– Exploitation of multiple imperfect data sources is key to robust decision support
for CBD systems
• Solution– Advanced algorithmic methods for decision support– FLASH: novel cognitive-processing oriented machine-intelligence
decision support architecture and methods– Broad applicability to CB attack detection and characterization, facility protection,
course-of-action guidance and other decision-support tasks
Uncertain and Disparate Multisource Data (sensors and other sources)
FLASHDecision Support Architecture
Machine-Intelligence Multisource Information Fusion
Decision Support for diverse CBD applications
MIT Lincoln Laboratory2007 CBIS --3
Jerome J. Braun 12/1/2006
Motivation for FLASH Machine Intelligence Approach
• CBD domain challenges– Complex phenomenology– Limited reliability of models and processes– Data inexactness and uncertainty
• Machine-intelligence character of FLASH methods– Particularly suitable for demanding decision-support tasks such as CBD– Generality of algorithmic methods offers potential of multi-purpose functionality– Alleviate redevelopment problems and costs associated with proliferation of
single-purpose approaches lacking scalability and migration capability– Enable augmented cognition for warfighter, reduce information overload and
enhance decision quality
MIT Lincoln Laboratory2007 CBIS --4
Jerome J. Braun 12/1/2006
Machine Intelligence and Machine Learning
• Paradigm shift from coding pre-known tasks to coding learning process itself• Machine intelligence must include learning
– Automatically learn from information embedded in data, to recognize/decide cases not-seen-before
– Learning implies adaptability• Cognitive systems – high-end machine-learning systems
Training data
Automatic Learning
Classifier (Recognizer)
Military transport Civilian transport
Training
Military or civilian vehicle?
Decision
Generalization: correct decision on data significantly different from previously seen
For harder cases, classifier must generalize better
Classifier (Recognizer)
Training data
Recognition (deciding)
Training
Feature Extraction
Feature Extraction
Data to recognize
Features: salient aspects of inputs to recognizer
Decision
For CBD: “backgrounds”and “releases”
For CBD: “attack Y/N”
Machine intelligence and machine learning alleviate need for precise models, adapt better to unanticipated conditions and different applications
Machine intelligence and machine learning alleviate need for precise models, adapt better to unanticipated conditions and different applications
MIT Lincoln Laboratory2007 CBIS --5
Jerome J. Braun 12/1/2006
Hybrid Approach
• Information fusion = inference, recognition, reasoning, and decision-making– Algorithmic process from multiple data sources to decisions– Not about networking, data collection and storage, etc.
• Existent approaches to information fusion all have limitations– Sample of method areas shown below– Relative strengths and weaknesses shown do not imply method applicability or
inapplicability for particular task
We are developing hybrid approach to alleviate limitations of individual methodsWe are developing hybrid approach to alleviate limitations of individual methods
Classical signal processingBayesian methodsNon-Bayesian methodsEstimation theoryNeural networks
RelativeStrength
RelativeWeakness
Perfo
rm
infer
ence
Cope w
ith
unce
rtain
data
Cope w
ith
dispa
rate
data
Suitab
le fo
r FF -
featur
e fus
ion
Suitab
le fo
r DF -
decis
ion fu
sion
Suitab
le fo
r bot
h
FF an
d DF
Suitab
le fo
r
time-s
eries
MIT Lincoln Laboratory2007 CBIS --6
Jerome J. Braun 12/1/2006
Conceptual View of FLASH(Fusion, Learning, Adaptive Super-Hybrid)
• FLASH integrates diverse machine-intelligence supervised and unsupervised learning and reasoning methods (subsystems, agents) at multiple levels
• Cognitive-processing architecture – inspired by neuroscience, cognitive studiesDetection, characterization, tasking, COA
Multi-agent cognitive-processing architectureMulti-agent cognitive-processing architecture
Context-scope recognition hybrid
Local-scope recognition
hybrid
Feature Selection
Event simulations
Feature Extraction
Local-scope fusion
Context-scope fusion Approximate
reasoning
Action planning
Planning and goal
satisfaction
Uncertainty estimation and
adjustmentEvent
clustering State-space estimation
Shared information
Lower levels
Higher levels
Multisource data
MIT Lincoln Laboratory2007 CBIS --7
Jerome J. Braun 12/1/2006
FLASH-1 Simplified Block Diagram
InstanceLevel
Recognition(ILR)
Hybrid
Time-Series Recognition
(TSR) Hybrid
Feature Extraction
and Selection
Instance Level
Fusion
Uncertainty Management
Background clustering
Release simulators
Training set generation
TemporalFusion Reasoning
ResultsEvent detection, characterization, CoA
Multisensor and context data
MIT Lincoln Laboratory2007 CBIS --8
Jerome J. Braun 12/1/2006
Multisensor Fusion Testbed
• Source of data for CB information fusion and decision support studies
• Portable for deployments in various settings• Multiple multisensor nodes, easily reconfigurable, seamless
addition of sensors• Foundation for CB “common operating infrastructure”• Multiple deployments within MIT LL and multi-month data collection
campaigns– Office building environment– Cafeteria with complex background, including cooking facilities
operation and proximity to outdoorsNode contents: • Multiple optical particle counters• MIT LL BAWS III• Airflow sensor• Space for additional sensors• Cluster processor• Power supply
Lower levels
Higher levels Context -scope
recognition hybrid
Local – scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local -scope fusion
Context -scope fusion Approximate
reasoning
Action planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering
State -space estimation
MIT Lincoln Laboratory2007 CBIS --9
Jerome J. Braun 12/1/2006
Testbed Data Management System
Sensor interfaces
Periodic updates
Web ServerSensorML Data Services
Multisensor Node Software
Web Server SensorMLMonitoring
Remote Server
Access via Web Services
Central Data Warehouse
Local storage
Node Web
Server
Short-term Storage
Multisensor data model
• Extensible and modular• Encapsulate and insulate sensor specifics• Data integrity and security• Data model and warehouse for sensor-independent
multisensor data management
Infrastructure suggested as blueprint for CB testbed effortsInfrastructure suggested as blueprint for CB testbed efforts
Lower levels
Higher levels Context -scope
recognition hybrid
Local – scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local -scope fusion
Context -scope fusion Approximate
reasoning
Action planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering
State -space estimation
MIT Lincoln Laboratory2007 CBIS --10
Jerome J. Braun 12/1/2006
Background Data
• Extensive indoor-background measurement campaigns conducted • Collected data used for development of training datasets for machine learning• Enable background characterization
06/21 06/22 06/23 06/24 06/25 06/26
Diurnal Pattern Node 3
06/21 06/22 06/23 06/24 06/25 06/2606/21 06/22 06/23 06/24 06/25 06/26Time (month/date)
Diurnal Pattern Node 3
0
100
200
Part
icle
C
once
ntra
tion
(a.u
.)
21:10 21:20 21:30 21:40Time (hr:min)
Inter-node Temporal Shifts
Node 1
Node 3
Node 4
21:10 21:20 21:30 21:4021:10 21:20 21:30 21:40
Node 1
Node 3
Node 4
Part
icle
Con
cent
ratio
n (a
.u.)
0
50
100
0
50
100
0
20
40
60
Background data excerpts
Jan Feb Mar0
500
0
500
0
500
0
500
1000
Time (month)
Part
icle
Con
cent
ratio
n (a
.u.)
00:00 06:00 12:00 18:00 00:00Time (hr:min)
Multi-month campaign
Lower levels
Higher levels Context -scope
recognition hybrid
Local – scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local -scope fusion
Context -scope fusion Approximate
reasoning
Action planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering
State -space estimation
MIT Lincoln Laboratory2007 CBIS --11
Jerome J. Braun 12/1/2006
Release Data
• Stochastic simulation approach for efficient generation of multiple release-cases to support development of training and testing datasets
• Span parameter-space of plausible releases
• Computationally efficient, rudimentary T&D (transport and dispersion)
Release simulation example
Node 1 Node 2 Node 3 Node 4 Node 5 Node 1 Node 2 Node 3 Node 4 Node 5
Lower levels
Higher levels Context -scope
recognition hybrid
Local – scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local -scope fusion
Context -scope fusion Approximate
reasoning
Action planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering
State -space estimation
MIT Lincoln Laboratory2007 CBIS --12
Jerome J. Braun 12/1/2006
Multisensor Feature Extraction
• Feature design “from first principles” challenging in CB– Non-specific sensing modalities, complex T&D
• Diverse set of features, unconventional types and uses
Dynamic modeling
Virtual sensor features
Kalman or Particle filters estimate signal at locations without sensors
2 0 0 4 0 0 6 0 0 8 0 0 1 0 0 0 1 2 0 0 1 4 0 0
1 0 2
1 0 3
Am
plitu
de (a
u)
time
2 0 0 4 0 0 6 0 0 8 0 0 1 0 0 0 1 2 0 0 1 4 0 0- 1 5 0
- 1 0 0
- 5 0
0
5 0
1 0 0
1 5 0
2 0 0
5 0 0 1 0 0 0 1 5 0 0 2 0 0 0 2 5 0 00 . 5
1
1 . 5
2
2 . 5
3
3 . 5
x 1 0 - 3
TKE
(au)
Poly
(au
)
OPC signal
Temporal localization
features
Uncertainty management
features
Indicate events in sensor signal
Gauge sensor uncertainty
• Signal derivatives
• Entropy• Histogram
differences• Covariances• Fourier• Wavelet• Polynomial fit• Turbulent
kinetic energy• Eddy
dissipation• and other…
Lower levels
Higher levels Context -scope
recognition hybrid
Local – scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local -scope fusion
Context -scope fusion Approximate
reasoning
Action planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering
State -space estimation
MIT Lincoln Laboratory2007 CBIS --13
Jerome J. Braun 12/1/2006
Feature Selection
• Classification task difficulty increases with number of features• Feature selection automatically decides feature efficacy• Classifier-independent feature selection methods developed• Exploitable for dynamic feature space selection
Information-theoretic approach (mutual information maximization) ranks features according to how well they represent the data
Classifier-independent feature selection approachClassifier-independent feature selection approach
0 5 10 15 20 25
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2x 10-3
# Features Removed From Feature Set
Mut
ual-
Info
rmat
ion-
Base
d Fe
atur
e Se
t Qua
lity
(a.u
.)
Automatic feature ranking designates features that can be removed without significant feature-space quality loss
Feature space matched to present data characteristics
Feature space matched to past data characteristics
Lower levels
Higher levels Context -scope
recognition hybrid
Local – scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local -scope fusion
Context -scope fusion Approximate
reasoning
Action planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering
State -space estimation
MIT Lincoln Laboratory2007 CBIS --14
Jerome J. Braun 12/1/2006
Background Clustering
• Background vs. release discrimination difficult due to background dynamics
• Find background subtypes by clustering, and recast problem as set of easier tasks
– Subtype vs. release
• Temporal clustering methods developed– Enhancements over Adaptive Resonance
Theory (ART) based clustering
Automatically discover plausible and non-obvious background typesAutomatically discover plausible and non-obvious background types
1
1a1c 1b
22a
2b06/21 06/22 06/23 06/24 06/25 06/26 06/27 06/28 06/29 06/30 07/01
Cluster ID
Time
1
1a1c 1b
1
1a1c 1b
22a
2b
22a
2b06/21 06/22 06/23 06/24 06/25 06/26 06/27 06/28 06/29 06/30 07/0106/21 06/22 06/23 06/24 06/25 06/26 06/27 06/28 06/29 06/30 07/01
Cluster ID
Time
Diurnal effectsBackground clustering result Non-obvious background types
Notional 2D view of multisensor data
Feature A
Feat
ure
B
Feature A
Feat
ure
B
Backgrounds Backgrounds and releases
Temporal progression
Per-subtype discrimination and fusion
Point to classify
Per-subtype Decision boundaries
f f
Lower levels
Higher levels Context -scope
recognition hybrid
Local – scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local -scope fusion
Context -scope fusion Approximate
reasoning
Action planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering
State -space estimation
MIT Lincoln Laboratory2007 CBIS --15
Jerome J. Braun 12/1/2006
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
PFA
P DET
PFA
P DET
Computational experiment ROCs
Computational Experiments with FLASH-1
• Preliminary results of relative gains with respective aspects of FLASH– Simple non-FLASH fusion scheme provided for reference
• Suggestive of relative benefit of hybrid approach elements, not any specific system performance
A: Simple multisensor fusion algorithm (learning of thresholds)
B: FLASH-1 feature selection, background clustering and ILR recognition
C: FLASH-1 as in (B), plus ILR fusion and TSR hybrid
D: FLASH-1 as in (C) plus fuzzy context modification method (threat data simulated)
Preliminary results of relative gains illustrate potential of FLASHPreliminary results of relative gains illustrate potential of FLASH
More details on A, B, C, and D, follow
MIT Lincoln Laboratory2007 CBIS --16
Jerome J. Braun 12/1/2006
Local-Scope Recognition
• Instance-Level Recognition (ILR) Hybrid currently based on multiple Support Vector Machines (SVM)
• ILR outputs are functions of distance from decision boundaries of multiple SVMs
• Computational experiments on datasets from deployment 1
• Comparison with “non-FLASH”simple fusion algorithm, based on estimating thresholds
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
PFA
P DET
PFA
P DET
Computational experiment ROCs
A: Simple multisensor fusion algorithm
Lower levels
Higher levels Context-scope
recognition hybrid
Local –scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local-scope fusion
Context-scope fusion Approximate
reasoningAction
planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering State-space
estimation
Performance results showing ILR hybrid outperforming simple fusion method
Performance results showing ILR hybrid outperforming simple fusion method
B: FLASH-1 feature selection, background clustering and ILR recognition
MIT Lincoln Laboratory2007 CBIS --17
Jerome J. Braun 12/1/2006
Local-scope Fusion and Context-scope Recognition
• ILR fusion based on Dempster-Shafer theory of evidence
• Each recognizer in ILR hybrid constitutes separate evidence source
• Time-Series Recognition (TSR) operates on sequences of ILR fusion stage outputs
• TSR based on Hidden Markov Models– Continuous Density HMMs (CD-HMMs)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
PFA
P DET
PFA
P DET
Refuting evidence
Belief Plausibility
Supporting evidence
1
Supporting or unknown
0
Computational experiment ROCs
B: FLASH-1 feature selection, background clustering and ILR recognition
C: FLASH-1 as in (B), plus ILR fusion and TSR hybrid
Performance results showing relative gains of FLASH-1 ILR fusion and TSR recognition
Performance results showing relative gains of FLASH-1 ILR fusion and TSR recognition
Lower levels
Higher levels Context-scope
recognition hybrid
Local –scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local-scope fusion
Context-scope fusion Approximate
reasoningAction
planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering State-space
estimation
A: Simple multisensor fusion algorithm
MIT Lincoln Laboratory2007 CBIS --18
Jerome J. Braun 12/1/2006
Fuzzy Logic Methods in FLASH-1
• Fuzzy logic methods convenient for representing descriptive/vague data
• FLASH-1 fuzzy outcome modification exploits certain data as modifiers rather than inputs to classification
• Example: Perceived threat assessment– Modulate TSR outcome by perceived
threat assessment– Context simulated by Gaussian
distributions correlated with truth-values– Mamdani-type fuzzy inference
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
PFA
P DET
PFA
P DET
Computational experiment ROCs
B: FLASH-1 feature selection, background clustering and ILR recognition
C: FLASH-1 as in (B), plus ILR fusion and TSR hybrid
FuzzyReasoning
ContextModifier
Feature or ILR/TSR level belief
membership functions
Perceived threat context membership
functionsContext-modified belief membership functions
D: FLASH-1 as in (C) plus fuzzy context modification method (threat data simulated)
Performance results showing further improvements with simulated “context” data
Performance results showing further improvements with simulated “context” data
Lower levels
Higher levels Context-scope
recognition hybrid
Local –scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local-scope fusion
Context-scope fusion Approximate
reasoningAction
planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering State-space
estimation
A: Simple multisensor fusion algorithm
MIT Lincoln Laboratory2007 CBIS --19
Jerome J. Braun 12/1/2006
Belief Network Reasoning
• Knowledge representation with FLASH semantic nets• Initial results with Bayesian net, indoor protection example
TSR-level alarmTrueFalse
76.223.8
AirflowEastWestLV
90.05.005.00
Risk level categoryVery lowLowHighVery high
0100
Close Fire DoorsNoneEastWestBoth
23.14.0568.84.05
Sensor network statusOKPerturbedInop
90.05.005.00
Known interferenceTrueFalse
1.0099.0
TSR release indicationTrueFalse
90.010.0
Set HVAC ModeNormalHigh
24.375.7
Issue Warning AlertYesNo
7.6292.4 Immediate evacuation
YesNo
7.8692.1
Conditions
Actions
Lower levels
Higher levels Context-scope
recognition hybrid
Local –scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local-scope fusion
Context-scope fusion Approximate
reasoningAction
planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering State-space
estimation
MIT Lincoln Laboratory2007 CBIS --20
Jerome J. Braun 12/1/2006
Course-of-Action Decision Support
• Course-of-action (COA) guidance – recognizing relevant battlespace conditions and reasoning to generate available options and tradeoffs
• Cognitive processing character makes FLASH approach promising for demanding COA guidance applications
– Machine learning, uncertainty provisions, approximate reasoning
• Further options include machine learning from scenarios, past cases, “lessons learned”, desktop exercises, after-action reviews, and information acquired from tacticians and other subject matter experts
MIT Lincoln Laboratory2007 CBIS --21
Jerome J. Braun 12/1/2006
Potential of FLASH for Battlespace Management Systems
• Potential as information-fusion and machine-intelligence algorithmic backbone for CBRN Battlespace Management systems such as JWARN, JEM, and JOEF
• Intelligent decision-support technology with multi-purpose applicability– Attack detection/characterization, prediction, impact assessment, course-of-action, others
• Augmented cognition for warfighters
Point, Standoff, Biological, Chemical, etc.
Intel,Weather,Commerce,Healthcare,Other MSB, etc.
Transport,Dispersion,Effectiveness,Epidemiology,Population,etc.
Disparate Sensor & Context Data: Uncertain, inconsistent, conflicting,
incomplete, ambiguous, vague
JWA
RN
, JEM, JO
EFFLASH Cognitive-Processing Machine-Intelligence Backbone
for Information Fusion, Recognition and Reasoning
Multiple sensors Multiple systems Multiple models
Results: Event detection and characterization, effect predictions, course-of-action guidance, tradeoffs, …
Lower levels
Higher levels Context -scope
recognition hybrid
Local –scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local -scope fusion
Context -scope fusion Approximate
reasoning
Action planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering
State -space estimation
MIT Lincoln Laboratory2007 CBIS --22
Jerome J. Braun 12/1/2006
Summary
• FLASH: novel cognitive-processing machine-intelligence decision-support architecture and methods
• Performance results to date show potential of FLASH
• Broad multi-purpose applicability, including CB detection, characterization, Course-of-Action guidance, and other CBR defense decision-support applications
Lower levels
Higher levels Context-scope
recognition hybrid
Local –scope recognition hybrid
Feature Selection
Event simulations
Feature extraction
Local-scope fusion
Context-scope fusion Approximate
reasoningAction
planning
Planning and goal
satisfaction
Uncertainty estimation and adjustment
Shared information
Multisource data
Event clustering State-space
estimation